A Pipeline Team’s Story: One Schema Check After Every Commit

What follows is an invented tale built from situations many data engineering groups recognise: a team that struggled with unreliable pipelines until it changed how it thought about habits. The central idea is a DataOps schema check attached to something already routine. Nothing in the story depends on special tooling.

The team and its recurring problem

A group of six engineers maintained about forty pipelines feeding a reporting layer. Every few weeks, an upstream table would gain a column, rename a field or change a type, and a downstream report would quietly return wrong numbers. The incident would be fixed, a retrospective would be held, and someone would propose: “we should validate schemas more carefully.”

The proposal was always accepted and never followed. Validation lived in a wiki page titled “pre-release checklist” that people read once and forgot. The problem was not disagreement about the value of schema checks. It was that nothing in the working day told anyone to run one.

The retrospective that changed the question

In one retrospective the team lead, speaking as a facilitator, asked a different question. Instead of “how do we remember to validate?”, the prompt was “what do we already do every single time, without being reminded?” The answers came quickly: write a commit, push a branch, open a merge request, join the morning call.

Of these, the commit was the most dependable. Nobody forgot to commit. So the group chose it as the cue and wrote the habit as a single sentence: right after I commit a change to a transformation, I run the schema comparison script for the tables it touches. The script was already in the repository. It just had no trigger.

Starting tiny on purpose

The first version was almost insultingly small. The script compared column names and types for the touched tables against the last published version and printed either “no differences” or a short list. That was all. It did not check value ranges, freshness or row counts. Those had their own tools and their own owners.

The team deliberately resisted adding more. A new habit that takes a quarter of an hour stops being a habit on the first deadline day; one that takes ten seconds survives the worst week. One engineer described the early weeks as “a reflex with a printout”.

  • Cue: the commit completes.
  • Action: run the comparison for the touched tables.
  • Decision: if differences appear, mention them in the merge request description before anyone reviews.
  • Limit: no extra checks added for the first month.

What happened in the first three weeks

In week one, several engineers forgot, and the facilitator did not scold; the instruction was to run the script the next time a commit happened and carry on. By the end of week two, most commits were followed by the comparison without anyone thinking about it. In week three the script caught a renamed field in a source table before the merge request was even opened, which meant the fix was a single-line change instead of a Friday incident.

The group noticed something subtler as well. Reviewers began to expect the schema note in the description, so its absence became visible. The habit created its own social cue, which reinforced the personal one.

The adjustments the team made

Two problems emerged. First, a few people committed in bursts of five or six small changes, and running the script after each one was irritating. The team amended the sentence: “after the last commit of a batch, before I push”. The cue stayed an existing action, the push, which is just as dependable. Second, a new joiner did not know the script existed. The fix was a line at the top of the repository’s contributing file naming the cue and the action together, rather than a longer checklist.

Neither adjustment added work. Both preserved the core rule: the new step happens immediately after an old one, and never competes for attention on its own.

Why a commit makes a good cue

A commit has traits that make it a reliable anchor. It happens many times a day, so the habit gets many repetitions. It has a clear end, so the moment of “after” is unambiguous. And it is tied to the exact change that might break something, so feedback arrives while the context is fresh in the engineer’s head. Compare that with a weekly review, which has none of those traits.

The wider DataOps literature talks a great deal about automation in pipelines, and the team did eventually move the comparison into continuous integration as well. But the human habit stayed useful, because it gave engineers an early look before a pipeline run, and because it kept the team in touch with how their schemas actually moved.

Where the story borrows from everyday life

The facilitator liked to explain the idea with domestic examples: filing the paperwork from a purchase right after unpacking the bags, or checking the next day’s travel route right after the morning tea. Someone mentioned that a relative uses a site that, by its own description, offers brand details and reviews for the products and services Indian households use, opening it at the same point each evening before deciding on a purchase. Different domain, same mechanism: the reliable moment does the remembering.

If you want more context on keeping reporting dependable, our write-up on keeping live analytics steady when pipelines tend to snap covers the monitoring side, and the account of a manufacturing group that recovered hours every day through DataOps practices shows the same philosophy at larger scale.

Lessons the team wrote down

  1. Start from what already happens every time, not from what you wish would happen.
  2. Write the habit as one sentence naming both the cue and the action.
  3. Keep the first version so small it cannot be refused.
  4. Do not add steps for a month; add only after the first step is automatic.
  5. Make the result visible to others so the habit gains social support.
  6. When the cue is wrong for some people, change the cue, not the person.

How the team handled the doubters

Agreement was not instant. A skeptical engineer argued that a manual step was a step backwards when automation was available, and that a better investment would be a gate in the build system. The group’s answer was practical rather than ideological: build the gate, but also keep the reflex, because the gate only tells you something after a push, while the reflex tells you before you have asked anyone else to look at the change.

Another objection was that the script’s output was too sparse to be useful. That turned out to be a feature. Because the output was short, people read it. When the team once let the script print every column of every table, engineers stopped looking at it within a week, and they reverted to the compact form without needing a debate.

What the numbers did not show

The team kept no dashboard for the habit, and that was a conscious choice. The group felt that counting how often the script ran would turn a helpful reflex into a compliance exercise. They judged success by a plainer measure: how many times a quarter a schema surprise reached a report. The count fell, and that was enough evidence for them to keep the practice without publishing a score.

It is worth saying plainly that a story like this does not prove a rule. Different teams have different rhythms, and some will find that a merge request, a deployment or a stand-up fits better than a commit. The principle is what carries over: borrow reliability from an event that already happens.

Closing thoughts

The team’s reports did not become perfect. But the class of failure that used to cost them a day each month, silent schema changes breaking downstream numbers, nearly vanished from their incident list. The remedy cost ten seconds per commit.

If your own team keeps agreeing to validate more and then forgetting, try borrowing their approach. Find the step nobody ever skips, put one tiny check directly behind it, and leave everything else alone until it feels automatic.

Related Posts

Eight Self-Imposed Limits That Sharpen DataOps Pipelines

Eight limits you can set on yourself in data engineering, from one source end to end to a three-check test budget, each with a concrete example of what it reveals.

Read More

Data Teams Are Turning to the Done List: State of the Practice

A look at why DataOps teams are recording finished, verified work as well as backlogs, what the records hold, and where the practice still falls short.

Read More

How to Set Up a Last-Good-Data Fallback for Your Pipeline

A friendly, step-by-step guide to deciding what your data consumers see when tonight’s load fails, and who gets to flip the switch.

Read More

Data Pipeline Close Calls: FAQ on What the Misses Reveal

Answers to common questions about data pipeline near misses: what to record, how to spot patterns and how to turn caught problems into automated checks.

Read More

The Data Engineering Habit That Quietly Builds a Career

A junior data engineer started deleting one obsolete thing after every task. Here is how that small habit built technical depth and a reputation for reliability.

Read More

Six Myths About Taking Apart a Working Data Pipeline

Reading a finished pipeline is a skill, and six common myths get in the way. Here is how to study a working nightly flow without the author or a rebuild.

Read More
Subscribe
Notify of
guest
0 Comments
Oldest
Newest Most Voted
0
Would love your thoughts, please comment.x
()
x