Ask a data team what they are building this quarter and the answers come quickly: a new lakehouse layer, a streaming job, a fresh set of dashboards. Ask what they are maintaining and you often get a pause. That pause is where most DataOps trouble starts. Pipelines that were delivered months ago keep running on trust, and trust is not a monitoring strategy. So let us go through six myths about looking after existing data work, in the friendliest way we can manage, because every one of them sounds perfectly reasonable until the Monday a report is wrong.
Myth 1: “If the job still turns green, the pipeline is fine”
A green status only says that the job ran to the end without an error. It says nothing about whether the numbers it produced are right. A source system can change a column’s meaning, a currency field can start arriving in a different unit, a partner can quietly stop sending one region, and the pipeline will cheerfully load whatever shows up. Success is a process fact; correctness is a data fact, and they are checked in different places.
What helps is a small set of content checks that live next to the job: row counts that should be in a sensible range, columns that should never be empty, values that should stay inside a plausible band. The earlier article on stopping streaming analytics pipelines from snapping goes deeper on the checks themselves. For this post, the point is simpler: nobody should treat a green light as a reason to stop looking.
Myth 2: “Old tables are harmless, so leave them alone”
Old tables cost something even when nobody queries them. They consume storage, they show up in catalog searches and mislead newcomers, and they sometimes keep feeding a downstream job nobody remembers. A person joining the team will find three tables with almost the same name and pick the wrong one, which is how stale numbers end up in a board deck.
A friendly rule of thumb: every table in the warehouse should have an owner and a reason. If neither can be found after a reasonable search, mark it deprecated, wait through a notice period, then archive it. Deleting is the last step, not the first, and the archive step means nobody has to be brave.
Myth 3: “Documentation is a one-time deliverable”
Documentation written at launch describes the pipeline as it was imagined, not as it evolved through a year of patches. Joins were added, filters changed, a source got swapped. The page still says what it said on day one, and it carries an air of authority that makes it dangerous.
The fix is not heroic. Put the page review into the same cadence as everything else you maintain, and when someone changes a pipeline, the change request touches the page. A short line at the top with the last confirmed date lets readers judge freshness at a glance.
Myth 4: “Links in our docs and dashboards will take care of themselves”
This one catches almost every team. A data catalog entry links to a definitions page. A dashboard footer links to the methodology note. A README links to an upstream vendor’s schema reference. Over time, wikis get reorganised, vendors restructure their sites and internal tools are renamed, and those links lead to error pages. Readers lose confidence, and a person trying to understand a metric gives up and asks around in chat.
You can keep a plain list of the important addresses and test them periodically, though honestly that is a chore people skip. A link-management service such as URLSnow can hold the addresses you publish and run its own automatic health checks, noting when one stops responding, so the dead ones announce themselves. It will not read the page and decide whether its content is still accurate, but it saves you from learning about a broken reference from an annoyed analyst.
Myth 5: “Maintenance slows down delivery”
It feels that way because the cost of upkeep is visible and the cost of neglect is deferred. Yet the teams that ship fastest are usually the ones whose foundations are tidy. They spend less time debugging mysterious lineage, less time arguing about which table is right and less time reviving ancient jobs that fail in the middle of a quarter close.
A practical compromise is to reserve a fixed slice of each cycle for existing assets and treat it as protected. When you write a plan for a modern multi-environment setup, as discussed in the thinking in handling data work spread over multiple cloud providers, the number of things that can drift multiplies, which is an argument for upkeep time, not against it.
Myth 6: “We will know when something is broken”
Sometimes you will, loudly. But the failures that hurt most are the quiet ones: a metric drifting slowly away from reality, a feed that delivers a stale file because the producer’s schedule changed, a dataset whose permissions no longer match the team. Nobody complains because nobody knows. The first sign is usually an outsider, such as an auditor or a customer, asking why two reports disagree.
The remedy is to go looking on purpose. A regular tour of what already exists, with a few simple questions, finds problems while they are cheap.
A gentle routine that survives busy weeks
Here is a short routine that data teams tell us they can sustain. It is built around three questions about each asset: does it still do its job, does anyone still depend on it, and does everything it refers to still exist.
- Sample, do not boil the ocean. Pick one domain each round, such as finance data or product events.
- Check the freshness and content of its most important tables and compare against a trusted figure from another source.
- Walk the references. Open the documentation, dashboard notes and catalog links tied to that domain and note any that fail.
- Ask for owners. Where an asset has none, either assign one or schedule it for retirement.
- Write down the outcome. One line per asset: kept, repaired or retired.
You will notice that nothing in this routine requires new tooling or a long project. It requires a calendar slot and a willingness to be slightly bored for an afternoon. Boredom is a decent sign that nothing is on fire.
Before the next new build
The next time a proposal lands for something exciting, try asking what existing asset it will rest upon and when that asset was last examined. If the answer is a shrug, spend the first week on the foundation. The new work will go better, and so will your sleep. Myths are comforting because they let us skip chores; the truth is that a little regular attention to the old pipes is what makes the shiny new ones trustworthy.
