Picture a familiar headache. You spend a fortnight building a data pipeline on your own, the tests pass, the job runs on schedule, and then the analyst who actually needs the table tells you the grain is wrong. One row per order line, when the analyst needed one row per order. Nothing is broken in the technical sense. The work simply answers a question nobody was asking. If that sounds familiar, this post diagnoses why it keeps happening in data engineering and offers a fix that costs far less than a rewrite: show the pipeline sketch to its consumers before you build it.
The symptom: a pipeline that works and still disappoints
Data work has a special way of hiding its flaws. Code that moves data from A to B can be perfectly correct and still be useless, because correctness here depends on meaning, not syntax. Is a “customer” someone who signed up, or someone who paid? Does “daily” mean the calendar day or the business day in a particular time zone? Do late-arriving records overwrite earlier ones or sit beside them?
These questions rarely surface while you are heads-down in the build, since every answer you picked feels reasonable to you. They surface when somebody downstream opens the output, squints and says “hm, that is not what I meant.” By then the transformation logic, the backfill and the scheduling have all been wired around the original assumptions.
The diagnosis: private polish postpones the pain
Why do capable engineers keep walking into this? Mostly because of a perfectly human instinct. We want to present something that works. A half-finished pipeline feels unprofessional, so we keep refining until it feels safe to share, and the sharing slips from Tuesday to Friday to next sprint. Meanwhile the flaws that matter most, the semantic ones, sit untouched because nobody who could spot them has looked.
There is a second contributor: the output of data work is often abstract until the very end. A diagram of boxes and arrows does not look like a decision, but each arrow hides several. Which source is the master for an address? What happens when two systems disagree? Those choices are invisible in the code review and obvious to the person who owns the business process.
The fix: share the rough sketch, and say what you want back
The remedy is simple to state and slightly uncomfortable to do. Ahead of writing any job, set a rough draft before one or two of the people who will read the result. The draft can be tiny:
- A hand-drawn flow of sources, steps and the final table.
- The proposed column list with a one-line meaning beside each name.
- Five sample rows, typed by hand, showing what a typical record and an awkward one will look like.
- A short list of the choices you made on the reader’s behalf, flagged as “tell me if wrong”.
That last item is the most productive. When you write down “I treated cancelled orders as excluded” in plain words, a reader can disagree in ten seconds. When the same decision is buried in a WHERE clause on line 212, it takes a production incident to find it.
Be specific about the feedback you want
Vague requests get vague answers. “Thoughts?” tends to produce “looks fine”. Try these instead, matched to what the draft is for:
- “Is one row per order the right grain for what you do with this?”
- “Which of these column names would you misread?”
- “What is the oddest record you have seen in this data, and does my sample cover it?”
- “How fresh does this really need to be: hourly, daily, or does it not matter?”
Questions like these are easy to answer from knowledge people already have. Nobody needs to study your design for an hour. They just need to react to something concrete.
A small walk-through
Let me replay the opening story with the fix applied. The engineer writes a half-page: sources are the order system and the payments feed, output is a table of orders with status and total, refreshed nightly. The page also carries five sample rows and the line “I am assuming one row per order, with line items rolled up into a single total.” It goes to the analyst and to the person who runs finance reconciliation, with a note saying nothing is built yet.
The analyst replies within the hour: line-level detail for product mix is needed. The finance colleague replies by the afternoon: refunds arrive days later and must stay linked to the original order. Neither answer was in the original plan. Both are easy to absorb at the sketch stage, where they mean adding a second table and a refund column. After a build, they would have meant reworking joins and rerunning history.
If you are weighing how a whole data-operations practice fits together, a few articles on this site are worth a look, including a piece about keeping real-time analytics steady and avoiding pipeline breakage and a guide to coordinating data work spread over multiple cloud platforms. For a story of time recovered through process change, there is also a case study of a tyre maker reclaiming daily hours through a DataOps playbook.
Timing: how early is early enough?
A handy test: imagine the bill for having to bin the current draft. If the bill is “an afternoon”, you are in the right zone. If the answer is “I would have to redo the backfill”, you waited too long. For pipelines, the natural moments are before the first transformation is written, again when the first real sample lands in a scratch table, and a third time just before scheduling. Each showing can be shorter than the last, because the earlier answers already shaped the work.
It also helps to vary who you ask. The person who owns the source system spots problems at the input end, such as fields that are blank for legacy records. The person who consumes the output spots problems at the other end. Asking only one of them leaves half the pipeline unreviewed, and the gaps usually sit exactly where the two meet.
Common objections, answered
“It will look sloppy.” Only if it is presented as finished. Label it a sketch and it reads as care, because it shows you wanted to get it right.
“People are busy.” True, which is why the draft should be short and the question narrow. A five-minute read with one focused request is easier to fit into a day than a forty-minute design review later.
“I will get contradictory answers.” Sometimes, and that is useful information. Two consumers who want different grains tell you the output may need two tables or a clearer contract, and finding that out at sketch time is a gift.
“What if nobody replies?” Silence after a clear request, with a deadline, is also a signal. State that you will proceed on your stated assumptions by Thursday, then do it. The assumptions are now written down, which helps when questions come later.
What good looks like afterwards
After a round of early feedback, the draft turns into something with more weight: a short design note that records the agreed grain, the definitions, the refresh pattern and the open questions. Keep it beside the code. When a new teammate asks why refunds are modelled the way they are, the reasoning is already on file.
A tiny habit makes this stick: before starting any new table, write the sketch and name your two reviewers on day one. It becomes as routine as creating a branch.
A note from outside the data world
The instinct to show early is common wherever plans meet audiences. Anyone who has helped arrange a weekend programme knows how a rough schedule passed around to a few friends exposes clashes quickly. Sites that gather such plans in one place, for example KeralaOrbit, an events website for Kerala whose sections include listings, venues, a forum and a blog, show how much information a good plan has to juggle before anyone turns up.
Wrapping up
When a pipeline disappoints despite working, suspect meaning rather than mechanics, and suspect that nobody who knew the meaning saw the work in time. The cure is cheap: sketch it, mark your assumptions, ask a pointed question, and wait for a reply before you build. You will still ship the pipeline. It will just land on the first try more often, and your analyst will stop sighing at your tables.