Data engineering plans fail in a very particular way. The sprint says a new ingestion job will take two days, the transformation logic is straightforward, the tests are small, and a week later the job still has not reached production. The delay is rarely a mystery in hindsight. A schema drifted, a backfill ran longer than anyone assumed, a stakeholder changed a column definition. What is missing is not skill but a record that compares predicted effort with real effort. This checklist helps a data team build that record and turn it into better forecasts for pipeline work.
Think of it as a self-inspection of how your team forecasts. Work down the items one by one, and wherever a box stays empty, that gap deserves attention before anything else.
Part one: capture the guess properly
- The estimate is written down before work starts. A forecast spoken in standup does not count. It must sit in the ticket, with a date, so nobody can quietly revise it after the outcome is known.
- The unit is focused hours or person-days, stated once. Mixing elapsed days with effort hours is the most common reason data teams cannot compare numbers across months.
- The scope sentence is attached. One sentence describing what “done” means, for example “table loads nightly, row counts checked, alert wired”. Without it, the actual figure measures a different job than the guess did.
- Known unknowns are listed. If you do not yet know the shape of the source data, say so in the ticket. Later you can see whether jobs with listed unknowns overrun more often.
Part two: capture the actual honestly
- Hours are logged at close, not reconstructed at month end. Reconstructed numbers are pulled toward the original guess by memory and politeness.
- Waiting time is separated from working time. A pipeline that sits for six hours on a queue did not take six hours of effort. Record both if you can, since waiting is a scheduling problem and effort is an estimating problem.
- Rework is counted in the same entry. If a transformation was rewritten after review, those hours belong to the original task, not to a new ticket that conveniently resets the clock.
- Cancelled and abandoned tasks are recorded too. Dropping them from the log flatters the numbers and hides the cases where the guess was most wrong.
Part three: compute and read the ratio
For every closed task, divide actual hours by estimated hours. Keep the figure to one decimal. A job guessed at five hours that took eight gives 1.6. A job guessed at ten that took nine gives 0.9. The point of using a ratio is that it travels across sizes: a 1.6 on a small fix and a 1.6 on a month-long migration describe the same optimism.
- You have at least ten closed entries before drawing a conclusion. Fewer than that and a single odd task dominates the median.
- You use the median, not the mean. One disastrous backfill can drag an average far from what is typical.
- You look at the spread as well as the centre. A median of 1.3 with ratios scattered between 0.6 and 3.5 means your problem is unpredictability, and a single correction factor will not fix that.
- You compare the ratio by task type. Source onboarding, modelling work, quality-rule authoring and incident repair behave differently. A data team often finds that onboarding a new feed runs well over, while writing a validation rule is close to its guess.
A practical note on tooling: none of this needs a platform. A shared spreadsheet with columns for ticket, type, estimate, actual and ratio is enough for the first quarter. Teams that jump straight to automated exports often discover that the export captures timestamps rather than effort, and the numbers then describe how long tickets stayed open instead of how long the work took. Start manual, learn what you actually want to measure, and automate only after the definitions have settled.
Part four: interpret the pattern
A ratio consistently above 1 is usually optimism, and that is the normal case. But a few other readings are worth checking before you simply multiply every guess by 1.4.
- Ratios that climb with task size point to compounding integration surprises. The remedy is to split large work into stages and estimate each separately.
- Ratios that are high only for tasks with external dependencies suggest the guess ignores waiting on other teams. Estimate the dependency chain explicitly.
- Ratios near 1 for tasks done by the same person repeatedly indicate genuine familiarity. Do not pad those.
- Ratios that improve for a month and then slip often follow staffing changes or a new tool. Note the event in the log so the shift has an explanation.
Part five: close the loop on the next estimate
- The next guess shows two numbers: the raw gut figure and the corrected one. Both are visible to the whole team.
- The factor is taken from the same task type, not from the blended number, wherever you have enough entries.
- Corrections are reviewed after each batch of ten tasks. If the factor has converged toward 1, reduce it. Do not keep an old correction alive out of habit.
- Someone owns the log. Ownership can rotate, but at any moment one named role is responsible for making sure entries are complete.
A short illustration from a pipeline team
Consider a team that keeps this log for a quarter. Their first twelve ingestion tasks produce a median ratio of 1.8, while twelve validation-rule tasks come in at 1.1. Before the log, they believed they were “a bit optimistic” across the board. After it, they know the optimism lives in ingestion, mostly because source teams change field formats without warning. They respond by adding a short discovery step at the start of every ingestion ticket and by applying a 1.8 correction until the discovery step proves it can lower the figure. The next quarter’s ingestion ratio falls to 1.3. They did not work faster; they predicted better and discovered problems earlier.
The illustration is invented for the sake of the example, but the pattern appears in plenty of real engineering groups once someone bothers to measure it.
Wider reading for people who like to measure
Teams that enjoy this sort of evidence-led habit sometimes look beyond engineering blogs for ideas on shared learning. The Cotocus Foundation site presents itself as a nonprofit’s home for technology, innovation and education, and its program, research, community and blog sections make a decent browsing stop for anyone interested in how knowledge gets passed around.
Related reading on this site
Two earlier pieces connect well with the checklist above. One explains how disciplined data practices keep streaming analytics from breaking, a good companion when you are deciding what counts as “done” in the scope sentence. Another recounts a manufacturing case where better data workflows returned hours to the working day, which shows why tracking time, honestly and in numbers, tends to reveal where effort actually goes.
Final pass: the one-page audit
Print the lists above and score your team: one point per ticked item, out of the total. Anything missing from part one or two means your data is untrustworthy, so fix those first. Anything missing from part three or four means your data is fine but unread. Anything missing from part five means you are collecting numbers and not using them, which is the most wasteful state of all. Revisit the audit in a month. Estimating well is a trainable skill, and the evidence for improvement should appear in your own ratios within a couple of cycles.