Claude Code investigates a duplicate payment, reads the deployment state, patches a retry worker, and runs the focused tests. During the same run, it calls an MCP tool that resets a production queue offset.
The reset succeeds. The telemetry exporter crashes before it flushes that span.
Later, the team replays the trace. Every recorded event reproduces cleanly. The patch looks sensible, the tests pass, and the review packet contains no production write. The replay is faithful to the trace and false about the run.
Observability cannot help when the evidence pipeline silently omits the event that matters. Before a Claude Code trace can support replay, incident review, or approval, the runtime needs to prove that the expected events arrived and that each consequential tool call reached a terminal state.
Give every run an event contract
A trace usually grows from whatever each component happens to emit. That is useful for debugging, but weak as production evidence. Define the event classes a run must produce before it starts.
trace_contract:
contract_id: tc-payments-repair-v3
run_id: claude-run-6042
required_lifecycle:
- run.started
- task.contract_loaded
- policy.loaded
- run.completed
tool_call_rule:
for_each_attempt:
require:
- tool.requested
- policy.decision
- tool.dispatched
- tool.terminal
state_change_rule:
for_each_successful_write:
require:
- effect.observed
- effect.bound_to_operation
sequence:
scope: run
contiguous: true
The contract does not predict how many reads or test runs Claude Code will need. It defines the relationships that must hold. A requested tool call needs a policy decision. A dispatched call needs a terminal outcome. A successful external write needs an observed effect tied to the operation that caused it.
Keep this contract in the runtime, not in the prompt. The model may explain the run, but it cannot decide that a missing event is optional after the fact.
Number events before asynchronous export
Assign the sequence number where the event occurs, before buffering or export. If the collector assigns numbers after receipt, a dropped event disappears without leaving a gap.
agent_event:
run_id: claude-run-6042
sequence: 87
event_id: ev-0087
parent_event_id: ev-0085
type: tool.dispatched
operation_id: op-queue-reset-19
tool: queue.reset_offset
target: payments-production-eu
emitted_at: 2026-08-24T09:16:42.201Z
previous_event_digest: sha256:4c98...
event_digest: sha256:71af...
The next persisted event arrives with sequence: 89. That gap is evidence. It does not tell you whether event 88 was a terminal result, an effect observation, or something else, but it proves that the collector cannot present this trace as complete.
A digest chain adds protection against deletion and reordering after collection. It does not rescue an event that was never persisted, so pair it with durable local buffering and collector acknowledgements.
Track tool calls as lifecycles, not loose log lines
Sequence continuity catches a dropped event. Lifecycle accounting catches a missing outcome even when a faulty emitter never reserved a sequence number.
tool_lifecycle_accounting:
requested: 14
policy_allowed: 11
policy_denied: 3
dispatched: 11
terminal_success: 9
terminal_failure: 1
terminal_unknown: 0
unresolved_dispatched: 1
unresolved_operations:
- operation_id: op-queue-reset-19
tool: queue.reset_offset
target: payments-production-eu
last_event: tool.dispatched
For a dispatched external call, absence is an unknown outcome. Do not label it failed and do not replay it. Reconcile the operation with the target system first, using its idempotency key, operation journal, or observed state.
This is the same boundary that applies to a timed-out MCP write. The difference here is that the response may have existed inside the running process and vanished on the way to storage. The recovery rule stays the same because the reviewer cannot prove the outcome.
Require the sinks to account for what they accepted
A local sequence can look complete while the durable sink is still behind. Record acknowledgements from each required destination.
trace_sink_acknowledgements:
run_id: claude-run-6042
emitted_through_sequence: 96
sinks:
- name: durable-event-store
acknowledged_through_sequence: 96
acknowledged_digest: sha256:22de...
- name: security-audit-store
acknowledged_through_sequence: 96
acknowledged_digest: sha256:22de...
exporter_pending_events: 0
exporter_dead_letters: 0
Do not let run.completed permit review before required sinks acknowledge the final sequence. If the audit store is unavailable, the run may finish its scoped work, but the review packet should remain blocked or explicitly degraded. A green telemetry dashboard is not an acknowledgement protocol.
Be careful with sampling. Diagnostic spans can be sampled. Policy decisions, tool dispatches, terminal outcomes, approvals, external effects, and run boundaries cannot. If cost forces you to sample those records, you no longer have an evidence trace.
Issue one trace completeness receipt
The reviewer should not have to inspect thousands of events to discover whether the collection process worked. Generate a small receipt from the contract, sequence, lifecycle, and sink checks.
trace_completeness_receipt:
receipt_id: tcr-20260824-031
run_id: claude-run-6042
trace_contract: tc-payments-repair-v3
sequence_check:
first: 1
last: 96
gaps: [88]
digest_chain_valid: false
lifecycle_check:
dispatched: 11
terminal: 10
unresolved: [op-queue-reset-19]
effect_check:
successful_writes: 2
observed_effects: 1
sink_check:
required_sinks_acknowledged: true
dead_letters: 0
decision: block_replay_and_review
recovery:
- reconcile op-queue-reset-19
- recover or explain sequence 88
- regenerate receipt
A passing receipt should have no unexplained sequence gaps, no dispatched call without a terminal state, one verified effect for every successful state change, and acknowledgements from every required sink. Bind the receipt digest to the review packet so the packet cannot quietly point at a different trace later.
This receipt complements an evidence-linked tool trace. Evidence links explain why an action happened. Completeness checks prove that the trace did not omit part of what happened.
Test the telemetry failure path
A happy-path replay proves the replay engine can consume a trace. It says nothing about whether the trace is whole. Break the evidence pipeline on purpose.
trace_completeness_evals:
- case: exporter_crashes_after_external_write
expect: block_review_with_unresolved_operation
- case: collector_drops_one_numbered_event
expect: reject_sequence_gap
- case: tool_dispatch_has_no_terminal_event
expect: classify_unknown_and_reconcile
- case: successful_write_has_no_effect_observation
expect: block_completion_receipt
- case: audit_sink_acknowledges_only_prefix
expect: wait_or_mark_trace_degraded
- case: required_security_event_is_sampled
expect: fail_trace_contract
- case: duplicate_event_arrives_after_retry
expect: deduplicate_by_event_id_without_hiding_gap
- case: events_arrive_out_of_order
expect: reorder_then_validate_sequence_and_digest
Run these tests at the exporter and collector boundaries. In-memory unit tests that append every event to a perfect list cannot expose a flush failure, dead-letter queue, partial acknowledgement, or duplicate delivery.
Also test the review user interface. A blocked receipt should be impossible to mistake for a warning that can be clicked away. Show the unresolved operation, target, last known event, and required recovery action.
Refuse a clean replay of incomplete evidence
Replay can reproduce only what the trace contains. That makes completeness a prerequisite, not a property you infer because replay succeeded.
Define the event contract before the run. Number events at emission. Account for every tool lifecycle and external effect. Require durable sink acknowledgements, then issue a receipt that either permits replay or names the gap that blocks it.
A clean replay of an incomplete trace is still incomplete evidence.
Claude Code: Building Production Agents That Actually Work contains production patterns for observability, replay, MCP boundaries, evals, rollback, and review packets around coding agents.