Fixing a Stalled Warehouse Migration With Fifteen-Minute Merges

Symptom: the warehouse migration has been “ninety percent done” for six weeks. The branch is enormous, the review queue avoids it, and every attempt to rebase produces a fresh pile of conflicts. Nobody is idle, yet nothing is landing. If that sounds familiar, the problem is almost certainly not your SQL skills or your tooling. It is the size of the unit of work, and the fix is to slice the migration into fifteen-minute steps that can each be merged on their own.

This is a practical diagnosis, in the order you would run it on a real team: what the stall looks like, why it happens, how to cut the work, and how to keep the cuts from creeping back to monster size.

Diagnosis: what a stalled data migration looks like

Three signs usually appear together. First, one long-lived branch holds every change, from renamed columns to a rewritten loader. Second, the person carrying it says things like “I just need a clear day to finish”. Third, nobody can answer the question “what is the smallest piece that could go live this week?” with a straight face.

These signs share a root. The task was framed as a single destination, “the new warehouse layout”, with no visible first step. A destination with no first step is easy to postpone, and each week of postponement makes the diff bigger and the eventual review scarier.

Root cause: the task has no seams

Data work resists casual slicing because everything seems connected. A table feeds a view, the view feeds a report, the report feeds an export. Pull one thread and the whole jumper moves. So people conclude the job must be done in one go, and they wait for the day they can do it in one go.

In practice the connections are almost always narrower than they feel. You can usually find seams, places where an old and a new thing can exist side by side for a while. The whole technique is finding those seams and then cutting along them in quarter-hour pieces.

Solution: cut along four kinds of seam

1. Seams between read and write

Start with the side that cannot break anything. A fifteen-minute step might be “add the new table, empty, with its final column names”. Nothing reads it, nothing writes to it, and yet it is merged and visible. Another: “write a query that reproduces last month’s totals from the old table and save it beside the migration notes”. It will become your yardstick later.

2. Seams between old and new

Let both versions run. One step creates a job that copies a single day of data into the new layout. The next compares row counts for that day. The next extends the copy to a week. Each is small, each is reversible, and each produces a number you can look at.

3. Seams between consumers

A migration is not finished until every reader has moved, and readers can move one at a time. Make a list of who reads the old structure, one line per consumer, in a single sitting. Then the plan writes itself: one quarter-hour per consumer to point it at the new table and check its output against the yardstick.

4. Seams between decide and do

Mixed-up thinking and typing are a classic slowdown. Give contentious choices their own step: “write down which timestamp column wins and why”. That takes fifteen minutes of thought and leaves a sentence others can argue with. The follow-up steps are mechanical and do not reopen the debate.

A sample slice list

Here is what the first week of a repaired plan might look like for a team moving an orders dataset into a cleaner layout.

  1. Inventory every reader of the old orders table and paste the list into the ticket.
  2. Create the empty replacement table with final names and types; merge.
  3. Write the yardstick query for one closed month and store it with the notes.
  4. Copy one day of rows into the new table by script; check the counts.
  5. Add a nightly step that copies the previous day; leave it running quietly.
  6. Point the least important report at the new table; compare it to the yardstick.
  7. Repeat the previous step for the next report, one report per sitting.
  8. Remove the old table only after a final step confirms no reader remains.

None of those steps needs a clear day. Each fits in the gap before a stand-up. Taken together they form a migration that goes live gradually instead of in a single nervous cut-over.

What good review looks like at this size

Small changes are easier to review, but only if the author helps. Put the intent as a single sentence leading the description, link the yardstick result, and say what is deliberately left for the next step. Reviewers who see a tiny, tidy change tend to answer within hours, which in turn keeps the author from drifting into a side branch. A fast review loop is half the benefit of slicing, and it is usually the half that people forget to count.

Keeping slices from growing back

Monster changes return by stealth. A step that should take a quarter of an hour picks up “while I am here” extras, and a week later you are carrying a second giant branch. Three habits guard against that. Write a done-line before you begin, and refuse to expand it. Park extras on a list instead of fixing them on the spot. And if a step overshoots, split the remainder into a new step rather than pushing through; the second half is usually clearer after a break anyway.

When the yardstick disagrees

Sooner or later a comparison step will report a mismatch, and this is where sliced work earns its keep. Because the copied range is small, the search space is small: one day, one table, one transformation. Spend a quarter of an hour narrowing it down. Split the day in half and compare each half. Check whether the difference is in a single column or spread across rows. Most mismatches turn out to be boring, such as a timezone boundary or a trailing space, and they are far cheaper to find in a one-day sample than in a month of production data. Write the cause on the ticket, fix it in its own step, and rerun the comparison before moving on.

If the mismatch is not boring, you have learned something valuable early, while the old table is still serving everyone and nothing is at risk. That alone justifies the slow-looking approach.

Related reading on this site

If your migration touches streaming data, the piece on stopping pipeline breaks in live analytics gives useful context for what each slice should protect. And for teams spread over more than one platform, this overview of data operations spanning several cloud providers pairs well with the consumer list when you decide who moves first.

One non-engineering analogy

It can help to see the same logic somewhere else. Consider a motorbike and scooter hire-and-share website that hands sign-in to a separate account service. A platform like that is made of distinct parts that can each be handled separately, and that independence is exactly what you are trying to create inside your own pipeline before you migrate it.

Checklist for the first sitting

  • Name the destination in one sentence and park it above everything else in a fresh note.
  • List every reader and writer of the thing you are moving.
  • Find one seam: a place where old and new could coexist.
  • Write the first step with a single verb and a visible outcome.
  • Do only that step, then write the second before you stop.

The bottom line

A stalled migration is a planning problem wearing a technical costume. Diagnose it by looking for the giant branch and the missing first step. Treat it by finding seams and cutting the work into pieces that fit in fifteen minutes and merge on their own. You will still need discipline, but you will no longer need a clear day, and clear days were never coming anyway.

Related Posts

Data Engineering Career Skills: The Three-Line Lesson Learned

A practical look at why a three-line lesson learned after each pipeline job builds judgment, interview stories and a record of growth that tools alone cannot.

Read More

Six Myths About Urgent Pipeline Fires and Important Data Work

Six beliefs about urgent versus important data work, from loud failures to calm periods, each taken apart with a plain fix for DataOps teams.

Read More

A Gentle No: How a Data Team Declines Without Burning Bridges

A plain-language explainer on how data engineers can say no politely to one-off extract requests and surprise deadlines while still being a team people want to ask.

Read More

The Data Engineer Skill Missing From Job Ads: Rereading Your Own Work

Job ads list tools, but one habit separates steady data engineers from the rest: reviewing your own pipeline change the morning after you write it.

Read More

Slow Data Pipeline? Diagnose It by Changing One Thing at a Time

Treat a slow nightly pipeline as a diagnosis problem: isolate the stage, change one variable, test the fix on a slice and keep the evidence.

Read More

Limiting Choices in Data Pipelines: A DataOps FAQ

Answers to common questions about shortening the menu of pipeline decisions so data teams keep attention for the choices that carry real risk.

Read More
Subscribe
Notify of
guest
0 Comments
Oldest
Newest Most Voted
0
Would love your thoughts, please comment.x
()
x