Good Enough Data Quality, Explained Without the Jargon

Ask a data team whether their tables are clean and you will usually get a nervous laugh. Nobody’s data is spotless, and anyone who claims otherwise has not looked closely. So the useful question is not “is it perfect?” but “is it good enough for what we are about to do with it?” That one change in wording turns data quality from an endless chore into a decision you can actually make, and this explainer walks through how.

What “fit for purpose” really means

Think about a recipe card taped inside a cupboard. It does not need to list the calorie count of every grain of rice. It needs to be accurate enough that dinner turns out right. Data works the same way. A table used for a rough weekly trend can tolerate a few missing rows, while a table feeding an invoice run cannot. The data is identical in both cases; the purpose changes what counts as acceptable.

When teams skip that step, they end up applying one giant standard to everything. Every column gets the strictest checks, every pipeline gets the same review, and the whole organisation slows to the speed of its most demanding dataset. Good enough is not a lower standard. It is a standard that has been matched to the job.

Why chasing perfect data never finishes

There are three reasons the finish line keeps retreating. Source systems change without warning, so a field that was always filled in starts arriving blank. People enter things inconsistently, so the same city shows up in four spellings. And every cleanup reveals the next layer of oddities underneath. You fix duplicates and discover the timestamps disagree; you fix timestamps and discover a unit mix-up.

None of this means cleaning is pointless. It means cleaning needs a stopping rule, otherwise the work swells, and the people waiting on the data get nothing meanwhile.

Three plain questions to set the bar

Before any pipeline gets polished, sit down with whoever will use its output and ask three things in ordinary language.

  1. What decision will this data feed? A decision about a monthly budget tolerates more fuzz than one about a customer’s balance.
  2. What goes wrong if a value is off? If the answer is “someone shrugs and re-runs it”, you are probably done. If it is “we send a wrong letter”, keep going.
  3. How soon is it needed? An answer delivered after the meeting is worth very little, however immaculate it is.

Write the answers down in a couple of lines next to the pipeline. That note becomes the acceptance test, and it is far easier to defend than a vague feeling that the data “needs a bit more work”.

The small set of checks worth keeping

Once the bar is set, resist the urge to build a wall of tests. Keep it short enough that people really read the results. For most pipelines a handful of checks does the heavy lifting: did the expected number of rows arrive, are the key fields filled in, do the values fall in a sensible range, and did the job finish on time. If those pass, the output is good enough for the common cases, and your attention is free for the odd cases.

If you want a sense of how failures actually surface in a live system, the article on keeping fast analytics running when pipelines snap is a good companion. It shows why a few well-placed checks catch most of the trouble that people expect to need hundreds of rules for.

A worked example, in words

Imagine a small retail team that wants a daily sales summary. Version one pulls yesterday’s orders, adds them up by region and emails a table. The team notices that about one order in a few hundred has no region. A perfectionist would stop everything to trace every missing region back to its source. A pragmatist asks what the summary is used for. It guides stock ordering, and a handful of unassigned orders do not change a stock decision.

So the pragmatist adds a line at the bottom of the email reading “unassigned orders” with the count, ships the summary, and schedules the root-cause hunt for later if the count grows. Readers get their numbers on time and can see the limitation for themselves. That is good enough in the best sense: honest about its gaps and ready when needed.

Where outside examples help

It helps to look at how other kinds of sites structure choices, because they have the same trade-off. A legal-services listing site such as BestLawyerNow lays out its area-by-area listings next to an articles page and appointment buttons, a tidy way of letting a visitor stop browsing once they have enough to act. A data consumer deserves the same courtesy: enough clarity to proceed, with the deeper detail available on request.

How to keep “good enough” from sliding into “sloppy”

There is a real risk here, and it is worth naming. Good enough only works when the bar is explicit and revisited. A few habits keep it honest:

  • Record the known gaps in the dataset’s description so nobody is surprised later.
  • Review the bar when the purpose changes, for example when a report meant for rough planning starts feeding something financial.
  • Track which checks have actually caught a problem, and retire the ones that never fire.

The same thinking applies when data crosses several platforms. The piece on running data work across more than one cloud touches on how quickly rules multiply when every environment has its own opinion, which is one more reason to keep the checks lean and the bar clearly written.

The takeaway for your next pipeline

Start by naming the decision. Decide what error is tolerable, keep a short list of checks, publish with a visible note about what the data does not cover, and revisit only when the purpose shifts. You will ship sooner, argue less, and trust the numbers more, because everyone knows precisely what they are worth. Polish is always available later, and it is cheaper once you know where the real problems live.

Related Posts

Why Data Teams Are Asking Why Five Times After Pipeline Failures

A state-of-practice look at root-cause questioning in DataOps: how repeated why-chains move teams from patching symptoms to fixing handoffs.

Read More

Three Ways to Narrate a Broken Data Pipeline, Compared

Speaking, writing and single-record replay each expose a silent data bug differently. A side-by-side look at three solo narration methods on one short-revenue case.

Read More

How to Tally and Trim Task Switching on a Data Pipeline Team

A step-by-step method for data engineers to log every mid-task interruption for a week, sort the log, and cut the switches that cost the most focus.

Read More

A Friendly Checklist: Ask What Would Make Your Pipeline Easier

Before rebuilding a data pipeline, ask the people who touch it one question. This checklist shows how to run that audit and turn answers into one small change.

Read More

DataOps Maintenance Schedules: Questions Teams Ask Most

Answers to the common questions about putting pipeline upkeep on a fixed calendar, from cadence and ownership to maintenance windows.

Read More

A DataOps Team Looks Back on a Season: A Fictional Case Study

A fictional story of a small DataOps team that paused at season’s end, counted the quiet frictions in its pipelines and chose three finishable fixes.

Read More
Subscribe
Notify of
guest
0 Comments
Oldest
Newest Most Voted
0
Would love your thoughts, please comment.x
()
x