Verdan Markets runs 214 supermarkets and, until this spring, one data platform whose production state lived in six engineers’ heads. The Monday that proved it was the fourth of the month: the wholesale pricing feed arrived intact, yet forty-one downstream reports still broke, because an overnight “small adjustment” — applied live, by hand, over a contractor’s phone call — had pointed the ingestion job at the staging bucket. The data was fine. The configuration, unwritten and unrecorded, was not. Eight weeks of GitOps later, the same month-end ran without a single manual touch, and the distance between those two Mondays is the cleanest statement of the practice’s value this team can offer.
Before GitOps: An Estate That Lived in Six Heads
Ines, who leads the platform group at Verdan, did the unglamorous audit first. Her team of five maintained a warehouse, two stream processors, an orchestration layer, a feature store, and roughly ninety scheduled jobs. When she asked each engineer to write down, from memory, the production settings of the components they owned, the reconstruction disagreed with reality in eleven places. Two of those eleven were load-bearing: a retention window nobody remembered changing, and a credential rotated during an incident and never written back to the runbook.
The audit produced an uncomfortable conclusion. The estate had no authoritative description. Every repair, every tuning pass, every 2 a.m. improvisation had edited the world directly while the documentation aged in place. Analytics engineering work — the pipelines themselves — was peer-reviewed and versioned; the ground the pipelines stood on was folklore. Closing that gap is the whole proposition of GitOps: describe the desired state of the infrastructure in a repository, let an automated controller reconcile the live environment against that description continuously, and route every change through review before it lands.
Week One and Two: Deciding the Shape Before the Tooling
The team resisted the urge to install anything for the first fortnight. Instead they settled five questions on paper, in this order:
- Where does the declaration live — one repository, or several?
- How are environments separated from each other?
- Who is allowed to approve a change, and to what?
- How do secrets reach the runtime without entering history?
- What happens when reality and declaration disagree?
The answers set the pattern for everything after. One repository, structured by purpose: a section per environment, a section for shared infrastructure, a section per product domain. Environments became directories sitting side by side, not long-lived branches — the team had tried branch-per-environment once, in another company, and the merge conflicts had poisoned the well. Promotion meant copying a tested change from one directory to the next, with the artifact referenced by an immutable digest rather than a moving tag, so that what ran in staging was provably what ran in production. Every value change travelled alone; bundling a timeout tweak with a schema migration had, in the past, made rollbacks impossible to reason about.
Rendering moved into CI. The manifests that reached a directory were reviewed as rendered output — a human read the diff of what would actually change, not the template that might produce it. For anyone mapping this onto data tooling, the group’s earlier notes on keeping real-time analytics stable when pipelines break cover the monitoring half of the same discipline; this story is about the half that decides what gets deployed in the first place.
Week Three: The Controller Arrives
With the layout agreed, the team installed a reconciliation controller and pointed it at the repository directory that described the platform itself — deliberately, so the tool administered its own installation before it touched anything else. Then the migration proper began, one component at a time, with destructive reconciliation switched off. The controller observed and reported; it did not delete. Only after a component had sat in agreement with its declaration for a week did the team grant it the authority to prune.
Cloud resources followed the same route. The warehouse’s storage tiers, the network paths, the managed queue — these were captured as declarations too, since GitOps has never been an exclusively Kubernetes affair; provisioning tools exist that let a reconciler drive cloud APIs the same way it drives cluster state. Engineers sceptical about “yet another layer” were won over by the drift reports: on day nine, the controller flagged that someone’s console session had doubled a stream shard count that morning. Before GitOps, that change would have surfaced as a cost variance weeks later. Now it surfaced as a pull request’s worth of evidence within minutes of the next poll.
The Drift Policy, Written Down at Last
Not every difference between declared and live state is a fault, and the team’s policy spells out which is which. Autoscalers own replica counts, so the controller tolerates their writes. Certain provider-populated fields are ignored by agreement. Genuine unexplained differences raise an alert. And a short list of resources — the namespaces that hold stateful stores, the persistent volumes, the schema registry — is protected from automated deletion entirely, because a reconcile loop with free rein to prune is a loaded instrument in a platform full of data.
That last point deserves its own paragraph, because data people feel it most. A Git repository holds descriptions, not datasets: table contents, object stores, and volumes stay outside it, which means version control is emphatically not a backup strategy. The declaration can rebuild the warehouse’s shape; the records inside are a separate recovery discipline. Conflating the two is the most expensive misunderstanding the team encountered in interviews with peers.
Week Eight: The GitOps Numbers, Before and After
Ines keeps the before-and-after on one slide. Median time from a merged change to a healthy, serving component fell from half a day to minutes, because approval and application stopped being a scheduling problem. The count of unexplained configuration differences dropped from eleven at audit to zero tolerated — every difference now has a ticket, an owner, and a policy entry. The month-end close ran with no emergency console access at all; the access log showed exactly two production sessions in the whole period, both rehearsed break-glass drills.
The honest costs are on the same slide. Review time is now the bottleneck — the team spends more minutes reading diffs than it used to spend clicking, and that is a real salary line item. The controller’s polling adds a small fixed delay to every change. And writing declarations is a skill the five of them are still sharpening; the reading list started with a guide to GitOps with Argo CD and the CAPA certification path and grew from there. A fuller treatment of the operating model, including the reconciliation mechanics summarised loosely here, lives in xOps School’s GitOps guide, which the team used to settle arguments about definitions during week one.
The finding in one sentence: the estate stopped depending on memory the day its description moved into review — every number above moved because the configuration acquired the same scrutiny the code always had.
Verdan’s data organisation still has pipelines that fail and queries that regress. What it no longer has is uncertainty about what is running, where a setting came from, or which written sentence authorises it. For a multi-cloud footprint, the same declaration pattern extends outward — the team’s next quarter extends it to the second provider, a step their notes on running data operations across several clouds already anticipate. The Monday feed arrives, the reports inherit a known state, and the six heads that used to hold the platform together are free to think about something harder than recall.
