Every data team has a story about the morning a nightly load did not finish. The dashboards were still there, but the numbers were from yesterday, or worse, half of today. Somebody noticed at ten, somebody else noticed at noon, and by the afternoon a finance colleague was asking why the report disagreed with the spreadsheet. The fix is rarely clever. It is a DataOps fallback that you decided on while everything was calm. Here is a simple, step-by-step way to set one up, written for people who have a pipeline to look after and not much spare time.
Step 1: Pick the one output that hurts most when it breaks
Do not start with the whole warehouse. Start with a single table or feed that other people lean on every day. A good test is to ask, “If this were stale for six hours, who would phone me?” The answer points at the output worth protecting first. Jot that name on a fresh sheet. Everything else in this guide gets applied to that one item until it feels routine.
It helps to be honest about what depends on it. List the dashboards, exports and scheduled emails that read from it. Five minutes with a lineage view or even a grep through your repository is enough. You are not building a map of the universe, only finding the people who would be affected.
Step 2: Decide what “good enough” looks like if the fresh data never arrives
The plan B for a data product is usually one of three things: serve yesterday’s version, serve a partial result clearly marked as partial, or serve nothing and show a plain message. Each is a legitimate choice. The point is that you choose it now, not in the middle of an incident.
- Yesterday’s snapshot suits reports where a one-day lag is tolerable, such as weekly planning views.
- A flagged partial load suits operational views, where some information beats none, as long as the label is unmissable.
- A deliberate blank with a note suits anything feeding money movement or compliance, where a wrong number is worse than a missing one.
Talk to the people who use the output. Their answer is often different from your guess, and hearing it takes a single short conversation.
Step 3: Keep a last-good copy where you can reach it quickly
A fallback needs something to fall back to. For most teams that means retaining the previous successful version of the table next to the current one, using whatever mechanism your platform offers: a snapshot, a time-travel feature, a dated clone or simply a renamed copy. Keep it simple enough that a colleague who has never touched the job can find it.
Check two practical details. First, how long does it take to swap the copy in? If the answer is “an hour of manual SQL,” your plan B is slower than the problem. Second, does the copy carry the same permissions as the live one? Nothing is more irritating than a successful rollback that nobody is allowed to read.
Step 4: Define the trigger as something you can measure
“If the load looks wrong” is not a trigger. “If the row count differs from the trailing average by more than the tolerance we agreed” is. So is “if the job has not reported success by the hour the first user logs in.” Pick one or two signals that are cheap to compute and unlikely to cry wolf.
Teams that already check freshness and volume have done the harder part. The article on live analytics and pipeline breakage covers the watching half. What this step adds is a decision attached to the check: when it fires, the response is already chosen.
Step 5: Name the role that can flip the switch
Write down a role, not a person: the on-call data engineer, the platform owner, the team lead on duty. Then give that role explicit permission to swap in the fallback without a meeting. Many teams skip this and lose an hour to “should we?” threads. The authority is the cheapest part of the plan and the most often forgotten.
Add a single line about who gets told afterwards. A short message to the affected users, saying “you are seeing the previous day’s data while we resolve a load issue,” keeps trust intact far better than silence does.
Step 6: Write it on one page and put it with the job
Keep the whole plan to a page: the output, the chosen fallback, the trigger, the owner, the swap steps and the message to send. Keep it beside the pipeline code in version control, or link it from the job’s alert. If you run workloads across several providers, the page should also say where each copy lives; a related read, running data work across several clouds, reminds us that where a copy sits matters too.
Finally, try the swap once on an ordinary afternoon, using a test copy if you prefer. Time it, fix anything awkward and update the page. You are not performing a grand exercise, just proving the card is not fiction.
A small outside-the-data-world comparison
The mindset is not unique to pipelines. If you have ever planned a night out with a few options lined up, you have used the same thinking. A site like MyEventNow, where upcoming gigs and venue pages sit beside a blog and a discussion forum is handy when the first choice falls through, because alternatives are only a browse away. Data teams get the same comfort by having the second option ready before the first one stumbles.
Common snags and how to avoid them
The fallback quietly goes stale
A last-good copy that refreshes along with the failed job is no copy at all. Make sure the snapshot is only replaced after a load has passed its checks, never before.
Everyone assumes someone else knows the steps
Pair up for the first test. If a second person can follow the page without asking questions, it is clear enough. If not, rewrite the confusing line, not the person.
The trigger fires too often
Loosen the threshold slightly, or add a short delay before the alert. A fallback that is used every week signals a deeper problem, and that is worth fixing at the source rather than normalising.
Downstream jobs are not told
If other pipelines consume your table, they will pick up the swapped data without comment. Decide whether they should pause, continue or tag their own outputs. A single note in the page is enough to start.
Nobody remembers the card exists
Mention it in onboarding, link it from the alert text and glance at it during any review of the pipeline. A plan that lives only in one engineer’s head leaves with that engineer on their next holiday, and the gap tends to show itself on the worst possible day.
The fallback hides a real problem
Serving older data keeps people productive, but it can also mask a fault that keeps growing. Log every swap, review the log weekly and treat repeated entries as a ticket for the underlying cause.
What a calm morning looks like afterwards
When the nightly load next stumbles, the alert goes to the owner, the owner checks the trigger, swaps in the previous snapshot and posts a two-line notice. Users see slightly older numbers with a label. The engineer investigates the root cause over coffee instead of in a panic, and the thread afterwards is short.
That is all a plan B is meant to deliver: a hiccup that stays a hiccup. Pick the output, choose the fallback, define the trigger, name the owner and keep one page. Once the first pipeline has its card, the second takes half the time, and before long the whole team is doing it without being asked.