Questions Data Teams Ask About MLOps After a Model Goes Odd

A data engineer at a mid-sized subscription company told me about the afternoon her pipelines were perfect and the business was still unhappy. Every table landed on time, every quality test passed, and yet the churn predictions that marketing relied on had quietly gone odd. Nobody had broken anything in the data layer. The trouble lived one step further downstream, in the territory MLOps is meant to cover. Her story is the thread through the questions below, because most data teams meet MLOps this way: by surprise, from the side.

Where does DataOps stop and MLOps begin?

Her first instinct was to ask whose problem it was. The Ops Atlas has a clear answer in its MLOps entry. Upstream ingestion, data contracts and quality work for every consumer of a dataset belong to DataOps. MLOps takes over where a specific model needs its own training snapshots, its feature definitions and checks that training and serving agree with each other.

That division felt generous to her at first, because it meant she did not have to own everything. It also meant that a clean warehouse could not vouch for a model. Passing data tests answers the question “is this table sound?” and says nothing about “is this model still right?”.

Why did good data still produce a bad model?

She found the culprit after two days: the way a feature was computed during training differed subtly from the way the live service computed it. The Atlas calls this skew and treats it as its own category. When two pipelines disagree, the remedy is to repair the pipeline, not to retrain. A retrain would simply have baked the disagreement into a fresh model.

The entry recommends writing one definition of each feature and reusing the same code on both paths, then proving the match with a replay test whose acceptable skew rate is zero. It also nods to the public Rules of Machine Learning, which suggest logging the features actually used at serving time so that you can compare them later. Her team adopted the replay test the following week and found two more features that had drifted apart.

Is drift the same thing as a data problem?

Not quite, and this was the distinction that changed how the team spoke in meetings. The Atlas separates three situations that look alike on a dashboard. Skew, as above, means the pipelines disagree. An upstream defect means the source data itself is wrong, and the fix belongs at the source, which is squarely her territory. Real drift means the world changed.

Within drift there are further splits. If the inputs shift, retraining may help provided fresh labels exist. If the base rate of the outcome shifts, recalibrating thresholds can be enough. If the relationship between inputs and outcome changes, new labels, new features or a redesign are needed. A single “drift score” cannot tell these apart, which is why the entry insists that diagnosis come before any retraining job is launched.

How does she know a drift number deserves attention?

She used to treat a population stability index above a published cut-off as a verdict. The Atlas takes a more sceptical line: those familiar cut-offs rest on convention rather than research, and the figure shifts with bin choices and sample size. In its executed lab, an empty reference bin made the index wander between roughly 0.146 and 0.425 as a tiny smoothing constant was varied, while a Jensen-Shannon distance stayed in a narrow band near 0.13 to 0.14. Her takeaway was to calibrate alerts on her own history and to check more than one distance measure.

What should a data team version besides tables?

Her pipelines already tracked schema changes. What they lacked was a record binding a model to the precise data it learned from. The habit the Atlas describes is stamping each training run with four facts: which commit ran, which exact data went in, what settings were used and what software stack hosted it. A tool such as DVC is named as one way to keep the dataset fingerprint, with a note that at data-lake scale its own documentation points toward lakeFS.

The same entry is careful about what “reproducible” means. It does not require identical bits; it asks for metrics within an agreed tolerance, since training itself can be non-deterministic. That relaxed definition made the practice far easier to sell to her colleagues.

When labels arrive late, what can she watch instead?

Her churn labels take a month to mature. Waiting thirty days to learn the model has slipped is not a monitoring plan. The Atlas proposes watching three layers: the serving service, the incoming data and the model’s quality. For quality during the waiting period it recommends stand-ins: how predictions are distributed, how far their average sits from an outcome rate that has since matured, and action limits that say in advance what to do when a proxy crosses a line.

It also warns that some label-free performance estimators stop working under concept drift, so the proxy approach must be treated as an early signal and never as proof.

Did the team need a new platform?

They considered buying one. The Atlas steers away from that reflex: it describes MLOps as a practice rather than a product, and observes that a platform without ownership and gates amounts to the earliest maturity level with a nicer interface. Google’s ladder has three rungs, starting with hand-run steps and ending with fully automated build-and-release; Microsoft’s scale has five, numbered zero to four. The team placed itself honestly at the first rung and set a goal of one rung up.

Their first-month list was modest: one reproducible training pipeline, one registry entry with lineage, and a rehearsed way back to the previous model. The entry’s own advice is nearly identical, and it quietly reassured her that small was fine.

What did the team already own that helped?

More than they realised. Their earlier work on keeping real-time analytics steady and catching pipeline breaks had given them an alerting habit. A past project described in a playbook that returned hours to a manufacturer’s day taught them to measure effort saved. And the discipline of reading numbers carefully, as discussed in our piece on faster, safer financial analytics, carried straight over to judging model metrics.

These habits matter more than any tool choice. A team that already treats data as something to be tested has crossed most of the cultural distance.

How should a new model be introduced?

Another conversation she had in that first month was about confidence. In the Atlas, gradual release has three steps. First comes a shadow stage: the newcomer is exercised on real requests in the background and customers keep getting the incumbent’s replies. Next a canary routes a small share of requests to the newcomer. Last, an online experiment checks whether the business outcome actually improves. The entry cites a travel-booking company’s experience that better model metrics do not automatically translate into business gain, a lesson that stung her team’s analysts, who had assumed the two moved together.

Before any of that, a gate pits the challenger against the current model using a fixed test set cut by time. The lab example in the Atlas rejected a candidate whose overall score rose by 0.009 because customers in their first six months fared worse, then promoted it after a repair. She printed that example and pinned it above her desk.

What should never be automated?

The temptation after a rough month is to automate everything: retrain nightly, promote automatically, never be surprised again. The Atlas lists exactly that among its anti-patterns, because a loop that promotes itself can end up learning from its own outputs and trigger outages. Automation is a level you pick deliberately. Her team keeps a human decision at the promotion step and automates the scheduling and the checks around it.

Rollback follows the same philosophy. The entry is firm that going back means redirecting an alias or traffic share, never retraining, and that an ordinary rule-based fallback should be kept ready for the worst day. It suggests rehearsing quarterly. Her team booked the first rehearsal for a quiet Thursday and discovered that nobody remembered where the fallback rules lived.

What is the human lesson underneath all this?

When she finished telling me the story, she did not talk about tools. She talked about the relief of having words for what had gone wrong. Skew, defect and drift are three words that turn a vague complaint into an assignment. Each points to a different owner and a different remedy, which spares a team the ritual of blaming the model, then the data, then each other.

If your own data platform is spotless but a model somewhere is acting strangely, start with the Atlas entry, ask the three diagnostic questions in order, and write down who owns each answer. That small act of naming is where MLOps usually begins in a data team, long before anyone buys anything.

Related Posts

A Friendly Naming Audit for Data Pipelines, Tables and Columns

A conversational checklist for auditing how your pipelines, tables, columns and jobs are named, so people and schedulers can find what they need.

Read More

One-Page Summary for Data Pipelines: A Beginner Primer and Glossary

New to data engineering? This primer explains the one-page summary for a pipeline, with a short glossary of the terms you will meet along the way.

Read More

How to Score Alert Grouping: A Data Team’s AIOps Evaluation Guide

A step-by-step method for judging AIOps alert grouping with labelled incidents, pairwise precision and recall, topology ablations and a shadow-mode gate.

Read More

FinOps in 2026: A Friendly State-of-Play for Data Engineers

A relaxed tour of what shifted in FinOps this year, from the FOCUS billing format to survey numbers, and what it means for people who run data pipelines.

Read More

Pipeline, Consumer or Ticket: Pick Your Weekly DataOps Review

Three ways to start a weekly DataOps review compared: read the pipeline runs, the reader complaints or the ticket board. Here is when each one earns its place.

Read More

Six PeopleOps Myths Data Teams Should Stop Repeating

Six common PeopleOps myths, from rebrand claims to eNPS and the four-fifths rule, corrected with careful sourcing and small worked numbers.

Read More
Subscribe
Notify of
guest
0 Comments
Oldest
Newest Most Voted
0
Would love your thoughts, please comment.x
()
x