A nightly pipeline stalls at 02:10. The on-call engineer opens the alert, scrolls through the logs and realises that the fix is something only one colleague has ever done: a partial reload that skips the already-loaded partitions. That colleague is on leave. Forty minutes pass, two dashboards stay stale, and someone eventually improvises a full reload that doubles the load on the warehouse. Nothing in this story is a technical failure in the strict sense. It is a runbook failure, and it turns up in data engineering more often than any exotic bug.

Think of what follows as a post-incident review turned into advice: symptoms first, root cause second, a fix third. The subject is the data pipeline runbook, a short written procedure for every recurring task and every fix that worked, so that good work is repeatable by anyone on the team.

The symptoms: how a missing runbook shows up

Teams rarely say “we lack runbooks”. They describe symptoms, and the symptoms are quite consistent:

• The same question appears in chat repeatedly. “How do I re-run the load for yesterday?” is asked by a different person each month, and answered by the same one.
• Recovery time depends on who is on shift. The identical failure takes ten minutes on one rota and ninety on another.
• Fixes are not reproducible. After a failure is resolved, nobody can state with confidence which action resolved it.
• New starters shadow for weeks. Onboarding is a long apprenticeship rather than a reading list plus supervised practice.
• Holiday cover is anxious. The person leaving writes a hurried hand-over note and checks their phone anyway.

Any two of these appearing together is a reliable signal. They have a shared cause: operational knowledge is stored in people, not in the system of work.

The diagnosis: why the knowledge stays in heads

Three mechanisms account for most cases.

Fixes are made under pressure and never revisited. During an incident the goal is to restore service. Writing documentation feels like a luxury, and once the pressure lifts, the next task takes its place. The knowledge exists for a few hours and then decays.

Tasks feel too small to document. Re-running a failed partition, rotating a service credential or backfilling a late-arriving file can each be done in a few commands. Because they are small, they seem obvious, and so they are never written down. Obvious to the person who has done them twenty times is not obvious to a colleague meeting them for the first time.

Documentation is separated from the work. A wiki page that lives far from the alert, and was last edited in a different quarter, will not be opened during a failure. People reach for the nearest human instead.

The fix: a runbook for every recurring task

The remedy is deliberately modest. For each task you perform more than twice, and for each fix that worked, write a short numbered routine. Keep it close to the place where the problem is noticed. Here is a template that has held up well for pipeline work.

1. Name and trigger. Use the words people speak aloud: “Morning orders table is empty”. State which alert or symptom brings you here.
2. Safe first checks. Read-only actions: the job status page, the last successful run time, the row count of the landing table.
3. Decision point. A plain branch such as “If the upstream file is missing, go to step 6. If the file is present but the job failed, continue.”
4. Action steps. One command or click per line, with the exact job name, parameter or environment. Avoid prose that hides the action.
5. Verification. The specific check that confirms recovery, such as a row count compared against the previous day, and the point at which you stop.
6. Escalation. The condition that should make you hand over, and the name or channel to hand over to.
7. Last-time notes. A line or two on anything unexpected. This is where the page accumulates real experience.

The structure is intentionally boring. A runbook is not an essay; it is closer to a recipe card. Its strength lies in being the same shape every time, so that a reader under stress knows where each answer will be.

Worked example: the partial reload

Return to the 02:10 failure. The runbook for it might say:

1. Open the orchestrator and find the failed task. Note the partition date in its parameters.
2. Check that the source extract for that date is present and complete. If it is not, stop and use the late-file routine.
3. Confirm that the target table contains no rows for the failed date, or that the rows are partial. If rows are complete, do not reload.
4. Trigger the task for that single date with the “skip existing partitions” option enabled.
5. When the task finishes, compare the row count for that date with the previous day’s count. A wide gap needs a second look before anyone announces recovery.
6. Post the date, the action taken and the result in the incident channel.

Notice step three. That is the piece of judgement that the absent colleague carried in her head: do not reload what is already correct. Writing it as an explicit check converts expertise into a repeatable decision.

Where runbooks fit with validation and automation

Written routines and automated checks are partners, not rivals. Runbooks do not compete with them. They sit next to them. Our article on how continuous validation is put together in a DataOps setup describes the automated side: tests that detect bad data as it moves. The runbook covers what a human does once a test has fired, or when automation cannot decide on its own.

There is also a natural progression. A task performed by hand and written down is a candidate for automation, and the written steps become its specification. We discuss that route in our notes on sound practice for automating DataOps tooling. Skipping the written stage tends to automate guesses; documenting first tends to automate decisions.

Keeping the page true

A stale runbook is worse than none, because it carries unearned authority. Four disciplines keep them honest:

• Edit immediately after use. The person who followed the page fixes any step that was unclear, while the details are still fresh.
• Owner and date on every page. Readers can see who maintains it and how recently it was confirmed.
• Link from the alert. The notification carries the address of its runbook, which removes the search.
• Fresh-eyes test. Hand the page to a colleague new to that job and let the author sit on their hands while it is followed. Every hesitation marks a gap.

Measuring whether it worked

You do not need elaborate metrics. Three informal observations tell you most of what you need: whether the same how-do-I question still circulates in chat, whether a recovery on an unfamiliar rota looks like one on a familiar rota, and whether people take leave without a hurried hand-over. When those three improve, the runbooks are doing their job. If you keep a record of incidents, you can also note how often the responder followed a page rather than asking for help, though that is a rough signal and should not be turned into a target.

Edge cases and limits

No procedure survives contact with every failure, and that is acceptable. The point is to remove avoidable improvisation, not all judgement.

Runbooks cover repeatable situations. Novel failures still need an engineer who understands the system, and the pages should say so by defining where routine ends and investigation begins. Security-sensitive procedures, such as credential rotation, deserve extra care about where the page is stored and who can edit it. And very rare tasks, such as a yearly migration, benefit from a rehearsed checklist rather than a page that is read cold once a year.

For a broader view of how teams put these habits into practice, the case-style piece on real-world DataOps implementations is a useful companion, and the TheDataOps.org home page collects our writing on pipelines, reliability and the journey from DevOps to DataOps in one place.

A first step you can take today

Open your team chat and search for “how do I”. Take the question that has appeared most often, ask the person who usually answers to perform the task once while someone records the steps, and turn that record into the template above. Place it next to the alert, ask a third colleague to try it, and revise. The first runbook takes an hour; the tenth takes twenty minutes; and the day your most experienced engineer is unavailable, the pipeline recovers anyway.