On her second week as a junior data engineer, Priya was handed what sounded like a boring task: every Monday, check that the files a partner sends us match what ended up in our tables. She assumed it would take ten minutes. It took her most of the morning, found three problems, and became the part of her job she now explains to every new joiner. This primer walks through the idea the way she learned it, with a small glossary at the end, because the routine she stumbled into is the simplest form of data reconciliation and one of the best habits in DataOps.

What reconciliation means for data

Reconciliation is just comparing two records that are supposed to describe the same thing and chasing down every difference. In a data team, the two records are usually “what we were promised or told to expect” and “what our systems actually hold”. A partner announces a delivery of files; the pipeline reports a successful load; a report says everything is fine. Three statements, none of which has been compared with the others.

Records drift apart quietly. A file arrives twice and the second copy is skipped. A schema change drops a column and the loader fills it with blanks. A job retries after a timeout and loads half a batch. Nothing crashes, so no alert fires, and the gap sits there until a stakeholder asks why last month does not add up. The cure is not smarter detection. It is a fixed, regular, boring comparison, done while each gap is still small enough to explain.

Priya’s first Monday

Her mentor, Arjun, gave her three things: the delivery notes from the partner, the load log from the pipeline, and read access to the landing tables. His instruction was to take each delivery and ask only one question: what should be here, and what is here?

She started with the simplest comparison, how many files were announced against how many were loaded. One delivery was short. The partner’s note listed a file the pipeline had never seen, because it had been uploaded to the wrong folder. Then she compared record counts per file against the totals in the note. One file loaded fewer records than announced, and the log showed a quiet warning about malformed rows that had been skipped. Finally she looked at a handful of values, the sort of fields a business would care about, and found a date column that had been read in the wrong format for one source.

None of those problems would have set off a pipeline failure. All of them would have surfaced eventually, probably as an awkward question in a meeting. Found on Monday morning, each took under an hour to fix.

Turning a chore into a routine

Arjun’s lesson for her was about rhythm. He pointed out that the first morning was long precisely because it was the first. Had the comparison been running weekly from the start, there would have been little to find. So they wrote the routine down:

1. Pick the day and time. Same slot every week, no negotiation.
2. List what was expected: files, record counts, key fields.
3. List what arrived and what was loaded.
4. Mark every difference as explained, unexplained or accepted.
5. Give each unexplained item an owner before moving on.

Over the next couple of months the unexplained column got smaller, and something more interesting happened: the team started noticing patterns. Wrong-folder uploads clustered around one partner contact. Skipped malformed rows came from one source system. Patterns like these let you fix causes instead of symptoms, which is the point where reconciliation stops being a chore and starts feeding the broader practice of making pipelines dependable. The team’s own writing on stopping pipeline breakage while analytics run live covers the monitoring side; reconciliation is its slower, more human companion.

Where money data fits in

The partner files Priya checks include payment and invoice data, and she found that financial records are where gaps matter most. A transaction that is announced but not loaded, or loaded twice, becomes somebody’s real problem. That is why finance analytics teams take reconciliation seriously, and why the ideas in speeding up and safeguarding financial analytics with DataOps lean on matching source records to loaded records at every hop.

On the finance team’s own side, the matching happens against invoices and settlements. Invoicing platforms like PayManagerPro, which covers invoices and payment links for finance and accounts staff, build in a reconciliation step so people can see which billed items have been settled. When Priya’s tables carry data from a system like that, she knows the source side is already being matched, and her job is to make sure the copy in the warehouse matches too.

Beginner mistakes worth dodging

Priya made a few missteps early on, and they are common enough to list. She first tried to compare everything at once, which produced a wall of differences nobody could prioritise; starting with counts and only then moving to values kept the list readable. She also trusted the load log as proof of the load, when the log only records what the pipeline believed it did. And she once marked a difference “accepted” without writing down why, then could not remember the reason a month later. A one-line note beside each accepted item costs seconds and saves an argument.

Mini-glossary

Here are the terms Priya wishes someone had handed her on day one.

• Reconciliation: comparing two records of the same thing and resolving each difference.
• Manifest: a list sent with a delivery that says what the delivery should contain, such as file names and record counts.
• Row count check: the simplest test, comparing the number of records before and after a step.
• Checksum: a short fingerprint of a file’s contents. If two copies have the same fingerprint, they are identical.
• Late arrival: data that shows up after the period it belongs to has been processed.
• Duplicate load: the same data loaded twice, often after a retry.
• Tolerance: the size of difference you agree to accept without investigation, for example for rounding.
• Drift: the slow, quiet divergence of two records that started out matching.

Getting started yourself

You do not need new tooling to try this. Choose one delivery you depend on, whether from a partner, a vendor or an internal system, and write down its manifest. For the next four weeks, compare that manifest to what landed, in a fixed slot. Keep a plain list of differences and who resolved each. If your platform spans several environments, remember that the same comparison applies between them; the guide to running data operations across several clouds shows how quickly copies multiply and why matching them early matters.

Priya now runs the Monday check in about twenty minutes. When a newcomer asks why it exists, she gives the same answer Arjun gave her: nobody is accusing the pipeline of lying. It is just that two records written at different times by different systems will always, left alone, disagree a little more each week. A short habit of asking them to agree keeps that disagreement cheap.