Data teams get asked about telemetry more often than they expect. A pipeline slows down, someone wants a trace; a warehouse load fails, someone asks why the alert came forty minutes late. Behind those questions sits a quieter one that this FAQ tries to answer: who runs the tooling that records all of this, and how do you keep it healthy? That job has an informal name in the Ops Atlas, ObservabilityOps, and what follows is a set of the questions that keep coming up whenever data engineers compare notes over coffee. Each answer stays brief, and a few point to a longer read.
What does ObservabilityOps actually mean here?
It is not a product and not a standard. The Atlas states that openly and uses the label for operating the observability system itself: the instrumentation rules, the pipelines that carry logs and traces, the stores that keep metrics, the dashboards and the cost of all of it. Using observability to investigate an outage is a different activity. If you can say whose job it is to keep your telemetry accurate, affordable and alive, you already have the idea, whatever your org chart calls it.
Why should a data engineer care about label cardinality?
Because data platforms love dimensions. Table names, partitions, tenant identifiers, job run numbers: every one is tempting as a metric label. The trouble is arithmetic: every fresh pairing of label values is stored as its own series, and the series total is what your metrics store pays for in memory and disk. Ten tables times fifty tenants times a hundred job runs is already fifty thousand. The Atlas guidance is to avoid unbounded values in labels entirely, and as a rough guide to keep a metric’s cardinality below about ten, investigating anything that grows past a hundred.
The same discipline shows up in log stores. Loki, for example, wants stream labels limited to tens of values, with the high-variety detail left in the log line. A run identifier belongs in the line, not on the label.
What happens when a scrape carries too many series?
This one surprises people. In Prometheus, a configured sample limit or label limit that is exceeded fails the entire scrape for that target, not just the offending metric. You do not lose a few series; you lose the whole picture for that service until someone fixes it. Cloudflare described this problem at very large scale in 2023, running 916 Prometheus servers holding around 4.9 billion series. Their answer was to patch Prometheus so that a server caps series instead of rejecting scrapes. Their line was that if CI checks pass, the capacity is there for you.
The takeaway for a smaller shop is to catch cardinality changes before they ship. A check in the pipeline that estimates new series is cheaper than a midnight page.
Do we really need sampling for traces?
Often later than you think. The OpenTelemetry guidance the Atlas relies on suggests considering sampling once you are producing around a thousand traces per second. Before that, storing everything is usually simpler. When you do sample, there are two broad choices. Deciding at the very first span is the inexpensive route, yet it cannot promise that traces ending in an error survive. Deciding once the trace is finished does let you retain the slow or failed ones, but it costs you state: every span of a given trace must be routed to one Collector instance, something the guidance calls difficult to run. Many teams start with the first approach and graduate later.
One tip that helps data teams: calculate your rate metrics from all spans before sampling, so that the numbers on your dashboard are not skewed by whatever fraction you kept.
How long should we keep telemetry?
There is no universal answer, but there is a useful default to question. Prometheus keeps fifteen days of data unless you change it. The Atlas recommends thinking in tiers, with a hot tier for fast queries, a warm one for occasional digging and an archive for the rarely needed. For a back-of-the-envelope disk estimate, multiply the retention in seconds by the samples per second and then by one or two bytes per sample.
A sizing exercise in the entry makes the point: a fleet that would need roughly 84 to 168 gigabytes under a careless label scheme drops to about 6 to 13 gigabytes for fifteen days once the problem label is gone. Retention and cardinality multiply each other.
Where does data quality monitoring fit?
Next to this, not inside it. Freshness checks, row counts and schema tests describe your data, while ObservabilityOps looks after the systems that watch the platform. They meet at one useful place: if your freshness canary stops reporting, you need to know whether the data is late or the watcher is down. The Atlas calls for a heartbeat for exactly that reason, an alert that always fires, with a separate alarm if it ever stops.
For a hands-on look at the data side, our earlier practical guide to live data monitoring in DataOps pairs well with this one, and a beginner walk-through of monitoring tool choices for data teams across clouds covers the tool landscape.
Who looks after this when the team is small?
The Atlas offers a plain heuristic. For a company with only a couple of teams, the sheet says a dedicated group is probably overkill; a named on-call owner earns its keep mainly when you host the stores yourself or operate under regulation. Once three or more teams are fighting rising cost or noisy alerts, the answer depends on hosting. Running stateful stores yourself argues for a platform team; buying them as a service argues for something lighter, a small group that writes the standards. The Atlas also cautions against dumping every team’s alerts on one central crew, and it cites DORA’s view that this kind of monitoring work should not sit with just one person or team.
Can a dashboard be wrong without anyone noticing?
Easily, and data people feel this one personally. A panel built on a metric that silently changed meaning keeps drawing a confident line. The Atlas ties dashboard design to alert design: page on symptoms that users can see, and treat signals that nobody opens as candidates for removal, a point it borrows from the Google SRE book. A ninety-day unused-signal ratio, the share of dashboards and rules that nobody touched in that window, gives you a number to shrink. Fewer, better-kept views are easier to trust.
How do we know the work is paying off?
Measure a few things and keep them boring:
- Delivery success of the pipeline itself. A worked example in the entry divides 9,950,000 successful exports by the 10,000,000 attempted (successes plus 40,000 plus 10,000 failures). The answer is 99.5 percent, short of a 99.9 percent goal, and the lesson is that retries should be counted only after they run out.
- Actionable pages. Fifteen of forty-two is 35.7 percent, a number that tends to motivate alert cleanup quickly.
- Cost per unit of traffic. Take what you spend in a month, divide by the requests served and state it per million.
- Unknown-service share. It should sit near zero if instrumentation sets service names properly.
If you want the long version, the Atlas write-up of ObservabilityOps and its maturity levels walks through all of these, along with a thirty, sixty and ninety day plan.
Is there anything that speeds up root-cause hunting?
Consistent identifiers. If every log line carries a trace and span identifier as a proper field, and propagation headers survive your queues and proxies, a failed batch job can be followed through the whole chain. Queues are the classic break point and need span links; proxies that drop the standard header, or services mixing two propagation formats, also cut traces in half. The Atlas adds a privacy warning: propagated baggage can end up in third-party API calls, so permit only named keys and remove them as requests leave your network. For deeper reading on tracing a broken pipeline, see our piece on speeding up pipeline root-cause analysis with telemetry.
One last question: where do I begin on Monday?
Pick the noisiest dashboard, find the label with the widest range of values, and ask whether it needs to be a label at all. Then add a heartbeat alert, and write down who gets paged if it goes quiet. Those two steps take an afternoon and cover most of what the discipline asks of a beginner. The rest is repetition: standards, limits, a regular alert review and a monthly cost number that someone owns.