Stuck Shard, Growing Lag: A ServerlessOps Fix for Poison Records

The symptom is always the same. Somebody on the analytics side asks why yesterday’s numbers have not arrived, and you open the dashboard to find nothing red. No error spikes, no crashed jobs. Yet the newest data is hours behind and the gap is widening. If your pipeline runs on event-driven functions reading from a stream, there is a good chance a single malformed record is holding an entire shard hostage. This is a ServerlessOps problem in its purest form, and it has a short list of causes and an even shorter list of fixes.

Symptom: the quiet backlog

Start by describing what you actually see, because the description narrows the cause quickly.

  • Freshness of output tables drifts, but only for some partitions or keys.
  • The function’s error count is flat, or even low.
  • Invocation counts remain high, as if work were happening.
  • The oldest unprocessed record keeps getting older minute by minute.

That last item is the tell. Stream consumers report how old the record at the head of the line is, and the Ops Atlas lists this “backlog age” among the core signals for the discipline, next to the share of throttled calls and the depth of failure queues. When age climbs while errors stay flat, the function is not failing to start. It is starting, choking on one item, and starting again on the same item.

Diagnosis: why a poison record repeats forever

Streams guarantee order within a shard. That guarantee is the reason a bad record is so destructive: the platform cannot skip ahead, because skipping would break ordering. So when your function throws on record number 4,812, the whole batch containing it is handed back and tried again. Nothing behind that record moves.

Now add the default. The Atlas draws attention to the fact that, for stream event source mappings, the retry limit defaults to minus one, which means unlimited attempts. Put an endless retry together with ordered delivery and you get the stuck shard. The shard will wait politely until the record ages out of retention, which for one common database stream is twenty-four hours. By then the damage to your freshness targets is done, and the record that caused it may be gone too.

Before you change anything, confirm the diagnosis in three checks:

  1. Pull the age metric for the stream consumer and look at its shape. A straight diagonal climb means a stuck shard; a sawtooth means ordinary load.
  2. Look at the logs for the same record position or sequence number repeating across invocations.
  3. Check whether the shard count matters: one stuck shard among many will hurt only the keys that map to it, which explains the “only some partitions are late” report.

Fix one: make retries finite

Set a maximum number of attempts and a maximum record age on the mapping. With both in place, a record that cannot be processed is eventually cut loose instead of blocking the line. The Atlas names these two settings, together with a batch-splitting option, as the minimum set for stream consumers. Pick the numbers by asking how stale your downstream tables are allowed to become. If an hour of delay is tolerable, a maximum age of a few minutes plus a small attempt count is plenty.

Be honest about the trade. Finite retries mean you will drop the occasional record from the main path, so you need somewhere to send it. That brings us to the next fix.

Fix two: bisect the batch and keep the evidence

Instead of retrying the whole batch, ask the platform to split a failing batch in half and retry each half. Repeat that and the poison record ends up isolated in a batch of one, while its healthy neighbours go through. In the Atlas this is the batch-bisect option on stream mappings, and it is the single most effective way to shrink the blast radius of one bad row.

The isolated record then needs a destination. One more wrinkle trips people up: for these stream sources, the failure destination receives only a small metadata document, containing the shard and the sequence position, not the record itself. The Atlas is explicit about this. The record’s actual content stays in the stream until retention removes it, and for the database stream mentioned above that is a day. If you want the content for later analysis, you have to copy it somewhere durable yourself, for example by writing the raw payload to object storage before processing, and then use the metadata to find your copy.

I would go further and call this the data-engineering twist on the standard advice. In a transactional system the failed message sits in a dead-letter queue with its body intact. In a stream it does not, so “we will look at the DLQ tomorrow” quietly turns into “we will look at a pointer to something that has expired.”

Fix three: report partial success properly

Batches introduce a second trap. Suppose a batch holds a hundred messages from a queue and ninety-nine succeed. If the function throws because of the one bad message, the platform may treat the entire batch as failed and redeliver all hundred. Ninety-nine records are now processed twice. With a loader that appends rows, that is how duplicates reach your warehouse.

The remedy is to answer with a partial result: the function hands back the identifiers of just the items that failed, and only those go back on the queue. The Atlas lists this practice alongside idempotency as basic hygiene, and it adds a related warning that many teams miss. If you do this, alarms built on the function’s error count stop telling the truth, because item failures reported through the response are not counted as errors. Watch the age of the oldest message and the volume entering the failure queue instead.

Fix four: make the loader safe to repeat

Even with every setting tuned, duplicates will still reach you occasionally, since delivery in these systems leans toward “at least once.” Your loading step should therefore be safe to run twice. In data terms, that means upserting on a business key instead of blindly appending, so that a repeated record overwrites itself. The Atlas cautions against using the platform’s message identifier for this purpose, because redriven messages receive new identifiers. Use a key that comes from the real event, such as an order number paired with an event version, and carry it all the way to the final table.

Check the timeouts while you are there

A stuck shard often has a sibling problem: a downstream call that is slower than the function’s own timeout, so each attempt ends by being killed rather than failing cleanly. The nesting rule in the Atlas is that anything the function calls should time out sooner than the function does. When the order is reversed, the function dies mid-write and you can never be sure what landed in the table. Align them, then retest with a deliberately slow dependency in a staging environment.

A verification routine

Once you have changed the mapping, prove the fix instead of trusting it. Push a handful of deliberately malformed records into a test stream among healthy ones. Watch three things: healthy records should keep flowing within seconds, the bad record should land in the failure destination after the capped attempts, and the metadata there should let you find your durable raw copy. Then repeat with a batch where a few items fail halfway and confirm the partial response returns only those items.

For background reading on why this kind of break shows up late, see our notes on keeping real-time analytics from breaking when pipelines misbehave. When the lag has a less obvious source, the write-up on tracing pipeline failures back to their cause with telemetry shows how to follow a symptom upstream.

What to put on the wall

Teams that live with this class of problem tend to keep a small panel on a shared screen: age of the oldest record per consumer, failure-destination inflow, the percentage of calls throttled, and the share of runs that reached the end without a duplicate side effect. The Atlas names that last measure, the duplicate side-effect rate, and it deserves its place. If you want the wider context for these signals, the Ops Atlas page covering ServerlessOps practice lays out the full list.

The takeaway is blunt. Unlimited retries plus ordered delivery plus a missing alarm equals a silent stall. Cap the attempts, split the batch, report partial success, and make the loader forgiving. Do those four things and a single malformed record becomes a line in a log instead of a missed morning report.

Related Posts

The Ledger Lane Crew and the Registry That Ate a Disk

A fictional data team learns ContainerOps the hard way: a vanished base image, a bloated registry and a log file nobody rotated, and what they changed.

Read More

SysOps for Data Engineers: A Primer With a Mini-Glossary

A beginner primer on host-level SysOps for data teams: schedulers, journals, log rotation, runway and clocks, with a seven-term glossary.

Read More

Self-Run vs Managed Control Planes: A KubernetesOps Comparison

Kubeadm or a managed service? A side-by-side look at ownership, upgrades, rollback and extended-support cost, written for teams that run data platforms.

Read More

InfraOps Explained for Data Engineers: Code, State and Reality

What InfraOps means for data teams: three records of an estate, plan review, small state files and a drift check you can start this month.

Read More

Six DisasterRecoveryOps Myths Data Teams Still Believe

Replicas, zones and root-cause hunts feel like safety. Six comfortable beliefs about recovery, and what the evidence says to do instead.

Read More

Restore Skills: The BackupOps Career Edge for Data Engineers

Why recovery craft sets data engineers apart, which BackupOps competencies can be shown with evidence, and a ninety-day plan to build a portfolio.

Read More
Subscribe
Notify of
guest
0 Comments
Oldest
Newest Most Voted
0
Would love your thoughts, please comment.x
()
x