Here is a trend that does not show up in any vendor roadmap: data engineers are picking up pens again. Walk past a few teams’ whiteboards and you will see rough boxes, arrows and scribbled table names long before anyone opens an editor. It looks old-fashioned, which is half its appeal. Still, there is a practical reason behind the revival, and it has to do with how expensive a wrong assumption becomes once a pipeline is running.
The state of the pipeline, in plain terms
Most data platforms today are made of many small pieces: ingestion jobs, a warehouse or lakehouse, transformation layers, a scheduler, a catalogue and a handful of consumers. Each piece is easy to reason about alone. The trouble lives in the joins between them, and those joins are exactly what a quick sketch makes visible.
Teams tell a similar story when they look back at projects that went sideways. The code was fine. The schema was fine. What nobody had noticed was that two upstream feeds disagreed about what a “customer” is, or that the finance report needed yesterday’s numbers by seven in the morning while the load finished at eight. A drawing forces those conversations early, while they still cost a few minutes of argument instead of a rewrite.
What a hand sketch catches that code review does not
Code review looks at what exists. A sketch looks at what is missing. Here are the gaps teams report finding most often when they draw first:
- A source that has no named owner, so nobody can approve a schema change.
- A table that two jobs both write to, with no agreement on who wins.
- A consumer, often a spreadsheet, that quietly depends on a column about to be renamed.
- A freshness promise that no step in the flow can actually keep.
- A backfill path that nobody planned, so every correction becomes a manual favour.
None of these show up in a linter. All of them show up when somebody draws the arrows and asks, out loud, “and then what happens here?”
Why now?
A few forces are pushing the trend. Pipelines have become cheaper to build, which paradoxically makes it easier to build the wrong thing quickly. Self-service tools let analysts spin up transformations without a formal design step. And teams are spread across locations, so a shared picture has become the thing that keeps everyone arguing about the same system.
There is also a cultural shift. After a few years of platform sprawl, plenty of data leads have grown wary of the phrase “we will figure it out in the build.” The sketch is a polite way of saying: let’s figure out as much as we can while it is still erasable.
A friendly recipe for your next sketch
You do not need a method with a trademark. Try this on your next feature request:
- Draw the data’s journey from the first source to the last reader, left to right, as plain boxes.
- Write the name of an owner under every box. A blank is a finding.
- Mark the moment of truth for each box: when should the data be ready, and who notices if it is late?
- Pick one awkward real-world record, such as a refund, a duplicate or a late-arriving event, and trace it through.
- Circle the smallest slice that would still serve one real reader.
Fifteen minutes with those five steps often exposes more than a week of unit tests, simply because the questions are different. Tests answer “does this code do what it says?” while a sketch asks “is this the right thing to say?”
A worked example: the late-arriving order
Take a retail team that wants a daily sales table. The sketch has three boxes: orders from the shop system, refunds from the payment side, and a summary table for the weekly review. Tracing a refund that arrives two days after its order exposes a problem within a minute. The summary has already been published, so does it change retroactively or show up as a correction on the later day? Nobody had decided. Settling that on paper took one short conversation. Settling it after the dashboards went live would have meant explaining why last Tuesday’s figure moved.
The same exercise tends to surface a naming problem too. If one box says “orders” and another says “sales,” ask whether they mean the same thing. Often they do not, and the drawing is the cheapest place to find out.
Where teams get stuck
Two habits spoil the exercise. The first is polishing: spending the session making the diagram pretty in a drawing tool, which turns a cheap experiment into a deliverable nobody dares to change. Keep it ugly. The second is drawing only the happy path. Force at least one box to fail, and write down what the reader of the final table sees when it does.
Another quiet trap is treating the sketch as a contract. It is a question list. When a reviewer says “this looks right,” the better reply is “what would make it wrong?”
How to make it stick on a busy team
Habits spread when they are easy to repeat. Some teams keep a roll of paper by the stand-up area; others add a single line to their ticket template asking whether a sketch exists. A sketch need not be reviewed formally. Its job is to be looked at by one other person who asks a naive question, and naive questions are the whole value.
Keeping the human side in view
Data work is full of devices, accounts and logins that engineers juggle without noticing: laptops, tablets, phones for two-factor prompts, headsets for the daily call. When a colleague asks which guide to hand a new teammate for sorting out their gear, it helps to know that Osnivo, an independent site with checkers, troubleshooting flows and a community forum for everyday gadgets, is one such place. That small habit of pointing people at a reliable reference keeps the data conversation focused on data.
What good looks like six months later
Teams who keep the habit describe a few visible changes. Design meetings get shorter, because the first ten minutes are spent pointing at a page rather than describing a system from memory. New joiners ramp up faster since the old sketches, photographed and filed, tell the story of why things look the way they do. And surprises shrink. A surprise in data engineering is rarely a clever bug; it is usually a mismatch that somebody could have spotted with a marker.
For the operational side of the same lesson, an earlier post on stopping real-time analytics from snapping mid-stream is a good follow-up, and another on running a data estate across several providers shows why a picture of the whole estate matters once pieces are scattered.
The short version
Draw it small, draw it early, draw it ugly. Put an owner under every box, trace one awkward record through the whole path, and circle the smallest useful slice. Then photograph the page and start building with a clearer head. The trend may look nostalgic, but it is really about cost: an arrow moved on paper is free, and the same arrow moved in production is a ticket, a meeting and a postmortem.