It was a Monday morning, and the finance dashboard was empty again. The data engineer on call opened the orchestrator, found a red task near the end of the nightly run, reran it, and watched the numbers fill in. Coffee was poured. The incident was closed in nine minutes. Three weeks later the same thing happened, and then again. Each time the fix was a rerun, and each time it felt like bad luck. This is the story of how a team stopped treating those mornings as luck and started tracing recurring pipeline failures back to one stubborn cause.
If your own data platform has a ritual of reruns, the approach below should feel familiar. It is a problem-to-solution diagnosis: describe the symptom honestly, collect the repeats, look across them, and only then choose a remedy.
The symptom: a failure that always looked different
On the surface the failures did not match. One week the error message mentioned a missing file. Another week it was a schema mismatch. The third time a downstream join produced zero rows and nothing technically failed at all, yet the report was wrong. Each incident had its own ticket title, its own responder and its own explanation in the chat thread.
That is the first trap of data work: the visible error is often the last domino, not the first. A missing file, a changed column type and an empty join can all be consequences of a single upstream event. If you diagnose each one on its own terms, you will conclude that the pipeline is simply fragile and move on.
Step one: keep a plain tally
The team’s turning point was unglamorous. After the third Monday, one engineer opened a shared sheet and began logging every failed run with five facts: the day and time, the task that went red, the error text, what the responder did, and what had changed in the previous twenty-four hours. No dashboard, no new tool, just rows.
Within a month the sheet held eleven entries. The rows looked scattered until someone sorted by weekday. Nine of the eleven came on a Monday-type morning, the first shift after a break in the calendar. That alone was a strong hint that the cause had something to do with the calendar rather than the code.
Step two: look across the incidents, not inside one
Reading eleven incident write-ups side by side is a different activity from reading one. Questions that never occur during a single firefight become obvious in a table.
- Do the failures cluster around a time of day, a day of the week or a month-end?
- Does the same upstream source appear in the lineage of every broken output?
- Was a deploy, a vendor change or a schedule shift recorded just before each one?
- Does the fix always look the same, such as “rerun,” “backfill” or “restart”?
For this team the answers converged. Every broken output depended on a file drop from an external partner. On weekends and holidays that partner’s export ran late, sometimes by two hours, sometimes by six. The nightly job started on a fixed clock, so it sometimes read a half-written file or no file at all. The error text varied because the pipeline happened to fail at different places depending on how much of the file had arrived.
Step three: separate the cause from the symptoms
With the pattern visible, the team wrote the diagnosis in one sentence: “the pipeline assumes a complete input at a fixed time, and the supplier does not guarantee either.” That sentence changed the conversation. The earlier tickets blamed parsing, schema drift and join logic. None of those was wrong as a description, but none was the cause.
A useful test is whether the proposed cause predicts the incidents you have not looked at yet. This one did. When the team checked older runs, two silent data-quality complaints from the previous quarter also lined up with late deliveries.
Step four: choose the fix that works once
There were several options on the table, and they differ in how lasting they are.
- Retry automatically. Cheap, but it only hides the lateness and still risks reading a partial file.
- Add a bigger buffer. Start the job later. It helps until the next slow delivery, and it delays everyone’s reports to cover a few bad days.
- Wait on the signal. Make the job begin only when the partner’s completion marker and a row-count check both arrive, with a clear timeout and an alert that says what is missing.
The team chose the third. It took an afternoon to build and removed the whole category of failure. Over the next quarter the Monday reruns stopped, and the sheet gained a final column recording that the pattern had been retired and how.
Where to read further outside your own team
Teams often ask where to read more on habits like these. One option is the site at DevOpsSchool.org, an open community space for software engineers. As the site tells it, visitors find study paths, role-based roadmaps, practical labs, a glossary, and a catalogue of tools to browse. A data engineer can skim the glossary next to a failure tally and borrow plain, agreed names for the problems that keep returning.
More reading on this site
Several earlier articles here connect to the same theme. The piece on keeping real-time analytics from snapping under load looks at the engineering side of preventing breaks. For teams spread over several providers, the guide to coordinating data work spread over several clouds explains why handoffs between systems are such frequent trouble spots. And the article on speeding up finance reporting without losing control shows the kind of workload where an empty Monday dashboard hurts most.
Why the pattern hid for so long
It is worth asking why a weekly problem escaped notice for weeks. Three reasons stand out, and each is common on data teams.
First, the person who fixed each incident was different, so nobody saw more than one or two. Second, every fix worked, which made the incident feel closed. A rerun that succeeds is rewarding, and reward discourages curiosity. Third, the alerts named the failing task, not the missing input, so the wording of each page pointed in a new direction. Anyone reading only the alert text would have sworn they were looking at three unrelated bugs.
Recognising those traps matters because they recur in other places. A flaky test suite, a nightly warehouse load and a monthly reconciliation export can all hide an upstream timing assumption in just the same way.
What the team kept doing afterward
The tally did not disappear once the Monday problem was solved. It became a standing habit with three small rules. Every failed run gets a row, even if the fix took ten seconds. Once a fortnight the sheet is sorted by task, by source and by weekday, and the team reads the groupings aloud. Whenever three rows share an attribute, somebody writes a one-line hypothesis and an owner is named to test it.
The habit also changed how reruns are viewed. A rerun is now treated as a data point rather than a cure. If a rerun fixes something, that fact goes in the sheet, because “fixed by rerun” is itself a clue: it usually means the failure came from timing, ordering or an input that was not ready, not from the logic.
There were softer benefits too. The on-call rotation became less lonely, because the sheet carried the memory that individual engineers used to hold in their heads. New joiners could read the last quarter’s rows and learn which parts of the platform deserved respect. Planning meetings gained a concrete input: instead of asking for time to “improve reliability,” the team could point at a pattern and show how many reruns it had caused.
A short checklist for your own diagnosis
- Log every failed run with the same five fields, however minor.
- Sort by time, source and responder before arguing about causes.
- Write the suspected cause as a plain sentence and test it against older incidents.
- Prefer the fix that removes the category over the one that speeds the rerun.
- Close the loop: record that the pattern was retired and what changed.
One bad Monday is an accident. Three of them is a message from the system about an assumption that no longer holds. The work is to read that message, and the cheapest instrument for doing so is still a table that someone keeps honestly.