Spend a few months near any data team and you will notice a quiet trend in how they talk about their week. Fewer people brag about the heroic overnight fix, and more people proudly describe the boring thing they stopped doing by hand. That shift is the story of DataOps right now, and it is worth a friendly look: how teams are getting better at deciding which manual chores deserve a script, which deserve a template, and which should stay human forever.
The chores that quietly pile up
Every data team has a short list of tasks that began as a one-off and never went away. Somebody checks that yesterday’s load finished. Somebody eyeballs a few rows after a schema change. Somebody reruns a failed job with a different date. Somebody pastes a summary into a chat channel so stakeholders know the numbers are fine.
None of these is difficult, and that is exactly why they linger. Each one costs ten minutes, nobody complains, and the total never appears on any planning board. Add them up across a team and a surprising slice of the week is spent repeating motions rather than thinking about data.
The common mistake: automating on day one
The obvious reaction is to automate everything immediately, and plenty of teams have tried it. The trouble is that a task you have done only once or twice is a task you do not yet understand. The script you write encodes your first guess about which parts are fixed and which vary, and the guess is often wrong. You end up with a brittle job that needs more babysitting than the manual version.
The healthier trend runs the other way. Teams are leaning into a short manual phase on purpose. They run the chore by hand a handful of times, jot down every step, and note where they hesitated. Only then do they decide what to write down as code.
What stays the same, and what never does
The useful exercise is a simple sort. After a few manual rounds, put each step into one of two columns. The first column holds steps that are identical every time: the query that counts yesterday’s rows, the comparison against the expected range, the formatting of the summary message. The second holds steps that need a person: deciding whether a dip is a real change in the business or a broken feed, deciding whether to backfill or wait, deciding who needs to hear about it.
- Identical every time: script it, test it, and let it run unattended.
- Mostly identical with a few variations: turn it into a template with blanks to fill in.
- Different every time: leave it to people, but give them a checklist of what to look at.
That middle option is underrated. Plenty of data work does not need code at all, only a decent template. A reusable message format for reporting a late load, with space for what happened and what is being done, removes a lot of friction without a single line of automation.
Why this feels like the direction of travel
Several things push teams this way. Tooling for scheduling and testing is easier to adopt than it used to be, so the cost of scripting a repeated step keeps falling. Teams are also larger and more distributed, which means a chore that lived in one person’s head now has to be shared. And everyone has been burned at least once by a pipeline that nobody dared to touch.
Writing the repetitive step down as a script is a form of sharing. It turns a private habit into something a colleague can read, run, and improve. If you want a feel for how data groups put these ideas into practice, the overview of how data teams have actually put DataOps to work is a good companion read, and the piece on keeping live analytics from snapping at the seams shows where the unglamorous automation tends to pay off.
A tale of two Mondays
Picture two teams. On Monday morning the first team opens a spreadsheet, runs through twelve checks by hand, and finds one problem forty minutes later. The second team has scripted nine of those checks, runs them with a single command, and reads a short report in five minutes. They still do the other three checks manually, because those need context. When the report flags a problem, they spend their time investigating, not hunting.
The second team did not get there by buying something. They got there by noticing, over several weeks, which checks never changed. That patience is the whole trick.
Where a reusable template beats a script
Not every industry that handles data lives in pipelines. Consider an intake desk that fields enquiries from far away. Take the overseas-patient enquiry desk behind a site like HipHospitals, which by its own account lets people abroad browse hospitals and doctors country by country and submit quote and appointment requests. A desk handling that kind of traffic faces the same decision a data team does: which parts of each reply are always the same, and which need a human to read the person’s actual question. You can browse how the way that site lays out its listings to see why a repeatable structure matters there.
Practical habits worth borrowing
- Keep a running notes file next to any chore you do more than twice.
- After about five manual runs, mark each step as fixed or variable.
- Script the fixed steps first, and make the script report what it did.
- Give the variable steps a template or a short checklist instead.
- Add a dry-run option so teammates can see the effect before trusting it.
- Revisit the script after a month and retire anything that no longer earns its keep.
The takeaway
The trend in DataOps is not a rush toward automating everything. It is a growing respect for the order of operations: do it by hand until you understand it, spot the unchanging core, hand that core to a script, and leave the decisions with people. Teams that follow the order tend to end up with fewer scripts, but better ones, and with more time for the questions that actually need a human mind.
So if your own week is full of ten-minute chores, try this. Pick one, run it by hand three more times with a notes file open, and see what refuses to change. That stubborn core is your first candidate for a script, and the rest can wait.
