Picture a data engineering team that is invented for this telling, yet familiar to anybody who has looked after a warehouse for a few years. They owned a nightly pipeline feeding finance dashboards, and one Tuesday morning the revenue table was wrong for a stretch of six days. What follows is how a decision journal changed the way they handled the repair, and later the way they handled almost everything else.
The Tuesday the numbers split
An upstream service had changed the format of a timestamp field. The ingestion job kept running, silently parsed the new values as nulls, and the downstream joins dropped those rows. Nobody noticed until an analyst asked why a regional total had halved.
The team gathered around a shared screen. There were two real options for fixing the damage. They could rerun the full history from the raw landing zone, which was thorough but would hold the warehouse busy for most of a day. Or they could backfill only the six affected days, which was fast but relied on their belief that nothing older was touched.
The lead engineer, for years the keeper of that kind of judgement, leaned toward the narrow backfill. A junior colleague wanted the full rerun. They argued for twenty minutes, settled on the narrow option, and moved on to the next fire.
Two months later, a second discrepancy appeared in an older month. Nobody could say why the narrow option had been chosen, what evidence had supported the claim that older data was clean, or whether anyone had planned to double-check it. The conversation had evaporated.
The notebook that started it
The next sprint, the team agreed to try something small. Whenever they made a call that was costly to reverse, they would write a few lines in a shared file called the decision journal. No template at first, just a date, the choice, the alternatives, the reasons, and a sentence about what they expected to see.
The first entry was written retroactively for the backfill, labelled as such so nobody would mistake it for a contemporaneous record. It said that the narrow option had been chosen because the full rerun would block quarter-close reports, and that the team had assumed older data was safe because the upstream change shipped within the week. It also noted, with some embarrassment, that nobody had actually checked the ingestion logs for the previous month.
Seeing that sentence on the page was uncomfortable and useful. It named an unverified assumption sitting under a decision. The team added a rule of thumb: any entry resting on a belief about data they had not queried must say so plainly, and must schedule a check.
What the journal looked like after a month
Thirty days in, the file held eleven entries. A few were small, such as choosing a partitioning scheme for a new events table. A few were bigger, like deciding to move one transformation from a scheduled script into the orchestration tool the rest of the stack used.
Each entry followed the same loose shape:
- What we decided, in a single sentence.
- The other options on the table and why they lost.
- What we believe to be true, and what we are assuming.
- What we expect to see by a stated date.
- A blank line reserved for the review.
The blank line mattered most. Every other Friday, whoever held the rotating facilitator role opened the file and read out any entries whose dates had arrived. The team filled in what had actually happened, without erasing the original expectation.
The review that changed their minds
One entry concerned a decision to keep a legacy batch job rather than rewrite it as a streaming pipeline. The reasoning at the time was sound: the downstream consumers only read once a day, the rewrite would take weeks, and the existing job had run reliably. The expected outcome was no complaints about freshness.
At review time, the team found the opposite. A new reporting team had started reading the tables hourly, and freshness complaints had begun. But the journal also showed that the original reasoning had been reasonable given what was known. The call had not been wrong; the context had shifted. That distinction let them revisit the question without blame, and the rewrite was scheduled on its merits.
Another entry went the other way. They had chosen a heavier validation step on a high-volume feed, expecting it to slow ingestion modestly. The review showed the cost was negligible and the step had caught two malformed batches. That result earned the pattern a place in their default setup.
If you enjoy stories about how teams gain time through disciplined data operations, the account of a DataOps playbook that gave a tyre manufacturer hours back each day shows the same appetite for writing things down and checking later.
A side conversation about shared platforms
During one review, a team member wondered aloud about decisions that sit outside the data stack entirely, such as how consumer marketplaces handle identity. She had been reading about a site called MotoShareGo, a place to rent or share two-wheelers, which according to its own pages lets visitors sign in through a central account service. Whatever the merits, she said, a choice like that is a trade between convenience and a shared dependency, and the people who made it must have had reasons worth keeping on file. It was a throwaway remark, but it got the team to add an entry for their own single sign-on decision.
Rules they settled on
Write before you know the outcome
The entry gets written the same day as the decision, ideally in the same pull request. Late entries are allowed but marked as such, so no one mistakes reconstruction for memory.
State a check you could actually run
Expectations like “this should be better” were rejected in review. The team asked for something queryable: a row count, a latency percentile, an error tally, a count of tickets. If there was no way to check, the entry said so and the team treated the decision as a gamble, and flagged it.
Never rewrite the past
Review notes are appended below the original text. If a later realisation changes the picture, a fresh entry links back to the old one. The sequence tells the story of how the team learned.
Make it light enough to last through a hectic week
A few lines per entry. Anything larger became a separate design note that the entry simply linked to. The team protected the habit by keeping the cost of one entry very low.
What changed in the next incident
Four months on, another upstream format change caused another silent failure. This time the team opened the journal first. They found an earlier entry noting that schema-drift detection had been deferred, with a stated reason and a review date that had already passed. The reason no longer held, because the cost of adding the check had shrunk. They added it that afternoon.
The repair itself went differently too. They wrote the decision down before acting: full rerun versus narrow backfill, why they preferred one, what evidence they had pulled from ingestion logs, and what they would check afterwards. The review note two weeks later was brief. The call held up, and they had the receipts to prove it.
Colleagues from a neighbouring group, who had been reading about stopping pipeline breaks while real-time analytics keep flowing, asked to copy the file format. It spread without any mandate.
Takeaways you can borrow
- Start with one file and the five fields above. Resist building tooling around it.
- Mark every assumption as an assumption, and schedule a way to test it.
- Hold a short, regular review slot. Without it, the journal turns into a graveyard.
- Treat a reversed decision as new information, not as a mistake to hide.
- Judge the quality of a call by its reasoning at the time as well as its outcome.
The team still has the Tuesday incident in their records. It is no longer a story about who argued the loudest. It is an entry with a date, a reasoning trail and a review note, and it is the one they point newcomers to first.