Your agent failure rate depends on which traces you kept

An agent dashboard reports no failed runs. The trace store contains ten successful examples. That sounds reassuring until someone asks how many runs the application started, and which ones the tracing pipeline discarded.

I would refuse a release decision based on a raw failure percentage from a selectively retained trace store. The denominator needs its own evidence. A sampling rule can change the apparent failure rate without changing the agent at all.

This becomes awkward when the observability configuration does exactly what the team intended. Keeping every detected failure and a small selection of successes gives engineers a useful investigation queue. Divide failures by retained traces, though, and that queue makes the service look worse than the complete run population. A different selection can hide every failure.

The example below constructs both outcomes. It uses invented run records, executes locally and makes no model calls. It demonstrates selection bias, not the measured reliability of an agent or an installed OpenTelemetry pipeline.

Decide what the percentage describes

Start with the unit you intend to count. A user task can produce several agent attempts. Each attempt can make several tool requests, and each request can produce multiple spans. A percentage calculated over spans cannot silently become a task failure rate.

For this example, one record represents one completed agent run with a known outcome. There are 100 records: 95 successes and five failures. The complete run failure rate is 5%.

Now consider two retained subsets:

The first subset is deliberately chosen to miss the failures. It is a counterexample, not a claim that random sampling always misses errors. The second deliberately favours failures. Its percentage answers “what fraction of the records we kept failed?” That is a different question from “what fraction of all runs failed?”

Keep the view names visible in dashboards. A panel called Failures among retained traces is defensible. Relabelling the same value Agent reliability conceals the selection policy and the counting unit.

For retrying agents, also retain the task-to-attempt relationship. The earlier repeated-trial evaluation example explains why eventual success and first-attempt success need separate denominators.

Head sampling cannot inspect a future failure

OpenTelemetry’s sampling guide describes head sampling as an early decision that does not inspect the whole trace. Its explicit limitation is that head sampling alone cannot ensure retention of every trace containing an error.

An early decision may be entirely reasonable for cost control. The problem begins when a team expects the retained data to contain every late tool failure, policy violation or failed final answer. Those outcomes may not exist when the decision is made.

Tail sampling considers all or most of the spans available for a trace before deciding what to retain. That allows outcome-aware rules, such as retaining traces with error status or particular attributes. It also introduces buffering and operational costs. OpenTelemetry documents those trade-offs rather than promising that tail sampling is free or complete.

Putting a tail sampler after an upstream discard cannot reconstruct the discarded evidence. If the SDK or an earlier collector never sends a failed run’s spans, the later policy has nothing to inspect. Audit the whole path, including SDK settings, collector routing and backend retention, before writing “all failures retained” in an operating procedure.

For agents, decide what counts as failure separately from transport status. A tool can return successfully while the final answer cites an unauthorised document. Conversely, a correctly refused operation may be a successful safety outcome. An HTTP error rule will not implement that application judgement. Emit a bounded, application-owned outcome classification when the judgement becomes available, and document how late classifications reach the investigation record.

Run the selection-bias fixture

Download agent-trace-sampling.py, then run:

python3 agent-trace-sampling.py
python3 -O agent-trace-sampling.py

The program uses Python’s standard library and writes no files. It prints four summaries and runs ten unit tests. The tests check the complete inventory, both subsets, a tail-after-preselection chain, duplicate identities, invalid records and unknown outcomes. They also check that an empty inventory does not produce a reassuring zero failure rate.

The summary names are intentionally explicit:

full_inventory:          100 runs, 5 failures
preselected_subset:       10 runs, 0 failures
failure_enriched_subset:  15 runs, 5 failures
tail_after_preselection:  10 runs, 0 failures

The final scenario applies an outcome-aware retention rule only to the ten records that survived preselection. It still contains no failures. That result follows from the input set; no later policy can recover the other records.

This fixture uses a fixed list of identifiers to make the example reproducible. It does not implement an OpenTelemetry probability sampler, generate statistical confidence intervals or exercise a collector. Its complete_failure_fraction field means complete for the supplied inventory only. Passing a sampled subset to the function cannot establish that the subset contains every production run.

The function rejects duplicate run IDs instead of silently counting them twice. It preserves unknown outcomes and withholds a complete failure fraction when any are present. The fraction among known outcomes remains available under a separate name. That makes a useful local contract, but a production collector must still prove that it discovered the expected runs.

Keep outcome accounting independent of trace retention

My recommendation is to maintain a compact run inventory alongside diagnostic tracing. Register the run when work starts. Record its final outcome, or leave it visibly unresolved. Reconcile starts against terminal records rather than treating the absence of an error span as success.

For a bounded evaluation, the scheduled case manifest can provide the expected inventory. For a production workflow, use a durable admission or task record where the architecture supports one. Trace IDs can link records to diagnostic detail, but a missing trace must not remove the run from the outcome count.

OpenTelemetry’s metrics documentation distinguishes aggregate statistical measurements from request traces. Counters can support operational outcome totals independently of trace sampling. They still need correct instrumentation and a monitored export path. An unsampled metric is not a durable audit ledger, and a process crash can prevent an outcome measurement from being emitted.

Keep the instrumentation bounded. Outcome class, workflow version and environment are useful dimensions when their possible values are controlled. Putting every run ID into metric labels creates a cardinality problem. Keep run-level identities in the inventory or restricted event store, then use trace links for investigation.

If the scheduled inventory contains unresolved runs, show their count beside the known success and failure totals. Do not divide by only the completed runs and present the result as the complete experiment. A report can say “five failures among 97 known outcomes, three unresolved” without deciding that the unresolved runs succeeded.

Verify the sampler under late and missing evidence

The Collector’s tail sampling processor documentation requires all spans for a given trace to reach the same collector instance for effective decisions. It also describes the processor’s stateful buffering, decision timing and capacity controls. A policy file alone does not verify those deployment conditions.

Before trusting the production path, inject a known failure and follow its identity from the task record to the backend. Then repeat with the outcome arriving late, an exporter interruption and a collector under pressure. Record which evidence survives each case. Test these conditions in a disposable environment; the downloadable fixture does not perform those integration tests.

For a long-running agent, a final assessment may arrive after the tracing system has made its retention decision. Choose deliberately whether to keep the run trace open, use linked traces for separate phases, or store the later assessment independently. Each choice changes how an investigator finds the complete run. Do not assume a late assessment can change an already completed sampling decision.

Finally, retaining failure evidence is not permission to store whole prompts, credentials or customer documents indefinitely. Keep the outcome and safe diagnostic references; apply access controls and retention rules to sensitive payloads. A failure-enriched store can concentrate exactly the material an attacker would want.

For the wider process of defining outcomes, testing graders and building release evidence, see AI Agent Evals. Get the book on Leanpub.

Take one current dashboard panel and write down its counting unit, source inventory and retention policy. If the denominator comes from retained traces, label it accordingly before using it to approve the next release.