Imagine a data engineer named Lucía who inherits a nightly pipeline that everyone agrees is too slow. She has three ideas for speeding it up and a free week. By Friday the job finishes sooner, her manager is pleased, and then someone asks the awkward question: how much sooner, compared with what? Lucía realises she never wrote down the old figures. She remembers the job “finishing around breakfast”, but not whether that meant six in the morning or nine. Her good work cannot be shown, only described.

That small embarrassment is the whole case for taking a baseline before you touch a data pipeline. This explainer keeps the idea in everyday terms, so a newcomer to DataOps can use it on the very next job they own.

What a baseline means for a data pipeline

A baseline is a dated snapshot taken while the thing is still untouched, before any edit lands. For a pipeline, that snapshot answers a few plain questions. How long does a normal run take from start to finish? Which step is the slowest? How often does a run fail or need a restart? How much compute does it use, and what does that cost over a month? And what do the people who depend on the output say about it?

You are not judging anything yet. You are photographing the room before you rearrange the furniture. Anyone looking at the room afterwards can then see exactly what moved.

Why memory is not good enough

People are poor at recalling the early state of a system once it has changed. We remember the dramatic nights, the failed loads and the angry messages, and we forget the ordinary Tuesdays. After an optimisation, the new behaviour becomes the normal one within days, and the old one fades. If improvement is the story you want to tell, you need the old chapter written down while it is still true.

There is a second trap. Data volumes change on their own. A pipeline can slow down in March simply because the input doubled, and speed up in April because a source system was retired. Without a baseline that includes input size, you may credit your own tinkering for something the calendar did.

Lucía’s second attempt

On the next pipeline she inherits, Lucía does it differently. Before making any change, she lets the job run untouched for two ordinary weeks and saves the facts it already produces: start and end times from the scheduler, row counts from each stage, and the retry log. She adds a short note in her own words about what the downstream analysts complain about, and another about the one step she suspects is the bottleneck.

She also writes down what she does not know. She does not yet know whether the slow step is slow because of its query or because of the storage underneath it. Putting the doubt on paper stops her from quietly rewriting history later.

A checklist you can reuse

• Record the typical and the worst run time over at least a couple of normal weeks.
• Save the size of the input for each run, so volume can be separated from speed.
• Note the failure and restart count, and what usually causes them.
• Write down the monthly bill for the compute and storage involved.
• Collect two or three sentences from the people who use the output.
• Store everything with a date, somewhere nobody will tidy it up.

If you are still choosing where to begin on the wider craft, a good companion is a look at keeping live analytics feeds from snapping. And for a sense of what a very large time saving can look like when it is measured properly, see the account of a manufacturer that won back hours of daily effort.

Where else this habit shows up

The same instinct applies far outside engineering. Someone planning treatment overseas might use the BestEyeHospitals website, a place for international patients to browse hospital and specialist listings and sketch out a journey before they commit to anything. The site is careful to say that what it publishes is general information, not professional medical advice, and it tells readers to confirm details with the providers themselves. Treat any directory the way Lucía treats a pipeline: useful as a starting map, never a replacement for facts you gathered yourself.

Keeping the snapshot honest

One caution: do not alter the job while you are observing it. A quiet fix on day nine, however tempting, contaminates the whole record. If something breaks badly enough that you must act, note the date and the action so the later comparison can allow for it.

Comparing after the change

Once Lucía has made her changes, she repeats the same measurements, in the same way, over a similar stretch of time. She compares run time per unit of input rather than raw run time, since volume may have shifted. She re-reads the notes from the analysts and asks whether the complaints have changed or merely moved.

Some of her ideas turn out to help a great deal. One does nothing at all, and she is glad to know it before it spreads to other jobs. A third helps the average but makes the worst night worse, a result she would never have seen without the old worst-case figure.

The takeaway in one sentence

Before you make a pipeline faster, cheaper or tidier, write down how it behaves today. The record takes a couple of weeks of patience, costs almost nothing, and turns every later claim from an opinion into something anyone can check.