Let me tell you about Tobias and Ines, two data engineers at a regional bakery chain with a mildly chaotic reporting setup. Their nightly jobs ran on an old scheduler that everyone distrusted, and the plan was to move them to a newer one. It should have taken a month. It took four, and the reason is the lesson in this story: they treated every behaviour of the old system as something the new one had to reproduce. A data migration only moves quickly when the must-have behaviours are listed first, and everything else is honestly labelled a tie-breaker.
The first week: an inventory that grew teeth
Tobias started the way most of us would. He exported the list of jobs, opened each one and wrote down what it did. Sixty-three jobs became a spreadsheet with a column called “behaviour to keep.” By Friday that column held more than two hundred entries, ranging from “loads the sales table” to “writes a log line with a smiley face when done.”
Ines looked at it over coffee and laughed, but kindly. “If we have to keep every one of these, we are rebuilding the old scheduler inside the new one.” The spreadsheet was complete, and that was exactly the trouble. It described everything the old system did, with no signal about which things anyone depended on.
They agreed on a new column and a rule. The column was called “who breaks if this stops?” The rule was that an entry could only stay in the must-have group if a named person or a named downstream report would fail without it.
What went into the must-have pile
Working through the jobs with that question, the pile shrank dramatically. After two afternoons it held a handful of items, and each one fit on a single line.
- The sales and inventory tables must be refreshed before the stores’ managers open their morning report.
- If a load fails, a human must hear about it before the report is opened, not after.
- Reruns must be safe: running the same job twice must not double the numbers.
- Each table keeps the same column names and meanings, because other people’s queries point at them.
- Credentials stay out of job files.
Notice that none of these mention the scheduler’s features. They describe results that the business could feel. That is a good habit for any must-have list: phrase each item as something that a person would notice breaking, rather than as a capability a tool might have.
What got relabelled as a tie-breaker
The other hundred-plus entries did not vanish. Tobias created a second list called “nice, in this order” and let the team vote lightly. Pretty run history, a nicer retry interface, tagging jobs by team, a chart of run durations, the smiley-face log line: these all landed there. Some were genuinely valuable and ended up in the second release. Some turned out to be habits that nobody could justify.
The change in tone was instant. When a team member raised “can we keep the weekly summary email that comes from the old system?”, nobody had to defend it or attack it. Ines simply asked whether a human would be stuck without it. The honest answer was no, because the summary was duplicated by a message in the team channel, so it moved to the second list, where it could wait its turn.
Why the second list matters for morale
People resist trimming when they feel their preference is being deleted. A visible second list fixes this. It tells everyone that the idea was heard, ranked and parked, which is quite different from being dismissed. In our story, the bakery’s reporting analyst, Priya, admitted that she had been quietly worried about a certain export format. Once she saw it listed as a tie-breaker, ranked third, she stopped worrying and offered to test the first release.
Where the outside world fits in
While they worked, Ines mentioned a relative who was planning care in another country and had a similar problem, with a long wish list and no sense of order. The relative had been reading an educational website for international patients that lays out how to compare heart-care options. Ines noted that the site describes its material as educational rather than medical advice, so the comparison framework is for organising your own questions and not for deciding anything clinical. She found the structure oddly familiar: gather what you need, review options, compare. She took it as a reminder that ordering your requirements is a skill that transfers anywhere.
Back at work, the lesson was concrete. Tobias also revisited the team’s earlier reading on how other data teams carried out their DataOps projects, noticing that the stories that went smoothly all started with a small agreed core. And a write-up on stopping live analytics feeds from snapping mid-pipeline confirmed the same instinct about reruns and alerts.
The cutover, in plain terms
With the short list in hand, the migration became a series of small checks rather than a leap of faith. For each job they ran the old and the new version side by side for three nights, comparing the table contents by hand for the first few and by a simple query afterwards. Only the must-have behaviours were tested at this stage. The nice-to-haves were scheduled for after the cutover.
Two jobs failed the rerun check on the first night, because they appended rows instead of replacing them. That was a real must-have violation, fixed in an afternoon. Under the old way of working, those same two issues would have been buried among two hundred other notes and would probably have surfaced in a store manager’s inbox.
The cutover itself took one evening. The old scheduler stayed paused, not deleted, for two weeks. That detail also came from the must-have list, in a way: “we can go back” was not on the first draft, but Ines added it after remembering how much she hated one-way doors.
What they would do again, and what they would skip
Asked afterwards what they would repeat, Tobias gave three answers. First, write the non-negotiables before opening the old system, because looking at it first makes everything seem equally sacred. Second, attach a person’s name to every must-have, so that the list means something to someone. Third, keep the tie-breaker list public and ordered, so nobody has to wonder whether an idea was lost.
What would they skip? The two-hundred-row inventory, mostly. The inventory was useful as a reference, but the first exercise should have been the question “who breaks if this stops?” asked of each job rather than of each behaviour. Asking it per job would have produced the short list in an afternoon.
A version you can try this week
You do not need a migration to borrow this. Pick any piece of data work that feels heavy, such as a new report, a table redesign or a switch of tools, and take twenty minutes.
- Write the one-line outcomes without which the work is a failure. Keep it to a number you can say in one breath.
- Next to each, write a name or a downstream consumer who would notice.
- Dump every other wish into a second list and put them in a rough order.
- Build and test only the first list. Pull from the second list when the first is done and there is time.
If something from the second list turns out to be a real must-have after all, promote it openly and say why. Moving items between lists is not failure, it is the method working. What you are avoiding is the silent state in which everything is critical and so nothing can be finished.
As for Tobias and Ines, their second release shipped six weeks after the first. The run-history page came with it, and so did the smiley face, because Ines insisted. It sat at the very bottom of the tie-breaker list, where a smiley face belongs.
