The Backfill Call: A Data Team’s Decision Journal Story

Picture a data engineering team that is invented for this telling, yet familiar to anybody who has looked after a warehouse for a few years. They owned a nightly pipeline feeding finance dashboards, and one Tuesday morning the revenue table was wrong for a stretch of six days. What follows is how a decision journal changed the way they handled the repair, and later the way they handled almost everything else.

The Tuesday the numbers split

An upstream service had changed the format of a timestamp field. The ingestion job kept running, silently parsed the new values as nulls, and the downstream joins dropped those rows. Nobody noticed until an analyst asked why a regional total had halved.

The team gathered around a shared screen. There were two real options for fixing the damage. They could rerun the full history from the raw landing zone, which was thorough but would hold the warehouse busy for most of a day. Or they could backfill only the six affected days, which was fast but relied on their belief that nothing older was touched.

The lead engineer, for years the keeper of that kind of judgement, leaned toward the narrow backfill. A junior colleague wanted the full rerun. They argued for twenty minutes, settled on the narrow option, and moved on to the next fire.

Two months later, a second discrepancy appeared in an older month. Nobody could say why the narrow option had been chosen, what evidence had supported the claim that older data was clean, or whether anyone had planned to double-check it. The conversation had evaporated.

The notebook that started it

The next sprint, the team agreed to try something small. Whenever they made a call that was costly to reverse, they would write a few lines in a shared file called the decision journal. No template at first, just a date, the choice, the alternatives, the reasons, and a sentence about what they expected to see.

The first entry was written retroactively for the backfill, labelled as such so nobody would mistake it for a contemporaneous record. It said that the narrow option had been chosen because the full rerun would block quarter-close reports, and that the team had assumed older data was safe because the upstream change shipped within the week. It also noted, with some embarrassment, that nobody had actually checked the ingestion logs for the previous month.

Seeing that sentence on the page was uncomfortable and useful. It named an unverified assumption sitting under a decision. The team added a rule of thumb: any entry resting on a belief about data they had not queried must say so plainly, and must schedule a check.

What the journal looked like after a month

Thirty days in, the file held eleven entries. A few were small, such as choosing a partitioning scheme for a new events table. A few were bigger, like deciding to move one transformation from a scheduled script into the orchestration tool the rest of the stack used.

Each entry followed the same loose shape:

  • What we decided, in a single sentence.
  • The other options on the table and why they lost.
  • What we believe to be true, and what we are assuming.
  • What we expect to see by a stated date.
  • A blank line reserved for the review.

The blank line mattered most. Every other Friday, whoever held the rotating facilitator role opened the file and read out any entries whose dates had arrived. The team filled in what had actually happened, without erasing the original expectation.

The review that changed their minds

One entry concerned a decision to keep a legacy batch job rather than rewrite it as a streaming pipeline. The reasoning at the time was sound: the downstream consumers only read once a day, the rewrite would take weeks, and the existing job had run reliably. The expected outcome was no complaints about freshness.

At review time, the team found the opposite. A new reporting team had started reading the tables hourly, and freshness complaints had begun. But the journal also showed that the original reasoning had been reasonable given what was known. The call had not been wrong; the context had shifted. That distinction let them revisit the question without blame, and the rewrite was scheduled on its merits.

Another entry went the other way. They had chosen a heavier validation step on a high-volume feed, expecting it to slow ingestion modestly. The review showed the cost was negligible and the step had caught two malformed batches. That result earned the pattern a place in their default setup.

If you enjoy stories about how teams gain time through disciplined data operations, the account of a DataOps playbook that gave a tyre manufacturer hours back each day shows the same appetite for writing things down and checking later.

A side conversation about shared platforms

During one review, a team member wondered aloud about decisions that sit outside the data stack entirely, such as how consumer marketplaces handle identity. She had been reading about a site called MotoShareGo, a place to rent or share two-wheelers, which according to its own pages lets visitors sign in through a central account service. Whatever the merits, she said, a choice like that is a trade between convenience and a shared dependency, and the people who made it must have had reasons worth keeping on file. It was a throwaway remark, but it got the team to add an entry for their own single sign-on decision.

Rules they settled on

Write before you know the outcome

The entry gets written the same day as the decision, ideally in the same pull request. Late entries are allowed but marked as such, so no one mistakes reconstruction for memory.

State a check you could actually run

Expectations like “this should be better” were rejected in review. The team asked for something queryable: a row count, a latency percentile, an error tally, a count of tickets. If there was no way to check, the entry said so and the team treated the decision as a gamble, and flagged it.

Never rewrite the past

Review notes are appended below the original text. If a later realisation changes the picture, a fresh entry links back to the old one. The sequence tells the story of how the team learned.

Make it light enough to last through a hectic week

A few lines per entry. Anything larger became a separate design note that the entry simply linked to. The team protected the habit by keeping the cost of one entry very low.

What changed in the next incident

Four months on, another upstream format change caused another silent failure. This time the team opened the journal first. They found an earlier entry noting that schema-drift detection had been deferred, with a stated reason and a review date that had already passed. The reason no longer held, because the cost of adding the check had shrunk. They added it that afternoon.

The repair itself went differently too. They wrote the decision down before acting: full rerun versus narrow backfill, why they preferred one, what evidence they had pulled from ingestion logs, and what they would check afterwards. The review note two weeks later was brief. The call held up, and they had the receipts to prove it.

Colleagues from a neighbouring group, who had been reading about stopping pipeline breaks while real-time analytics keep flowing, asked to copy the file format. It spread without any mandate.

Takeaways you can borrow

  • Start with one file and the five fields above. Resist building tooling around it.
  • Mark every assumption as an assumption, and schedule a way to test it.
  • Hold a short, regular review slot. Without it, the journal turns into a graveyard.
  • Treat a reversed decision as new information, not as a mistake to hide.
  • Judge the quality of a call by its reasoning at the time as well as its outcome.

The team still has the Tuesday incident in their records. It is no longer a story about who argued the loudest. It is an entry with a date, a reasoning trail and a review note, and it is the one they point newcomers to first.

Related Posts

Reusing Proven Pipeline Parts: Questions Data Teams Ask

A data team stops rewriting the same cleansing logic. Straight answers to common questions on keeping, storing, adapting and trusting reusable pipeline pieces.

Read More

How to Give Every Data Engineering Week a Theme

A step-by-step method for assigning each data engineering week one theme, using last month’s pipeline failures as evidence and a parking list for the rest.

Read More

Data Pipeline Basics: A Seven-Day Row-Count Habit

New to data engineering? Try one pre-merge row-count check for a week, keep a simple log, and use a mini-glossary to make sense of what you find.

Read More

Why Data Teams Are Drawing Pipelines by Hand Again

A friendly look at a quiet trend in data engineering: sketching the flow of tables and jobs on paper first, and what that habit catches before a single line runs.

Read More

Wiki, Repo or Ticket Thread? Where DataOps Notes Are Easiest to Find

Three common homes for pipeline notes compared on one test: how quickly a tired engineer can find the answer later. Plus four rules to make any of them searchable.

Read More

Five Myths About Parking Side Ideas in DataOps Work

Five common misconceptions about keeping a not-now list during pipeline releases, and what a data team can do instead.

Read More
Subscribe
Notify of
guest
0 Comments
Oldest
Newest Most Voted
0
Would love your thoughts, please comment.x
()
x