At 14:06, a reviewer approved a Claude Code run that would rotate one staging credential. The MCP gateway checked the approval, dispatched the write, and recorded a successful commit. Four seconds later, the reviewer revoked the grant because the wider deployment had been cancelled.

The incident query sorted every event by timestamp and produced this order:

14:06:11.042  credential committed
14:06:11.317  approval granted
14:06:15.204  approval revoked

The write appeared to happen 275 milliseconds before approval. Security treated it as an unauthorized action. The gateway team treated it as a logging bug. Neither side could prove its position from the records they had.

The cause was ordinary clock skew. The credential service clock ran 430 milliseconds behind the policy gateway. Every event was real. Sorting their timestamps invented an execution history that never happened.

All identifiers and records below are fictional examples.

Causal order receipt for a Claude Code MCP operation

A timestamp is evidence from one clock

Distributed traces make wall time look more precise than it is. A timestamp with six decimal places still came from a clock that may be early, late, stepped by time synchronization, or sampled after an event was buffered.

That matters once Claude Code can trigger work across an MCP gateway, a queue, and a downstream service. You may receive four plausible times from four different machines:

events:
  - type: approval_granted
    service: policy-gateway
    observed_at: 2026-09-10T14:06:11.317Z
  - type: operation_dispatched
    service: mcp-gateway
    observed_at: 2026-09-10T14:06:11.401Z
  - type: credential_committed
    service: credential-store
    observed_at: 2026-09-10T14:06:11.042Z
  - type: approval_revoked
    service: policy-gateway
    observed_at: 2026-09-10T14:06:15.204Z

The timestamps help operators search a time window. They do not establish causality across those services. The commit can carry an earlier wall time even when it was caused by the dispatch.

This is a narrow but nasty gap in an otherwise sensible agent effect evidence packet. Independent receipts prove more than a transcript, but reviewers still need a trustworthy way to order those receipts.

Put the ordering decision at the enforcement point

The gateway that checks authority and releases the effect already sits on the critical path. Give it a durable, monotonic sequence for security-relevant transitions within one control domain.

gateway_sequence:
  control_domain: staging-credential-writes
  operation_id: op-example-771
  entries:
    - sequence: 88410
      event_id: evt-example-approval
      type: authority_accepted
      approval_id: approval-example-39
      effect_hash: sha256:example-effect
    - sequence: 88411
      event_id: evt-example-dispatch
      type: effect_dispatched
      parent_event_id: evt-example-approval
    - sequence: 88412
      event_id: evt-example-commit-observed
      type: durable_commit_observed
      parent_event_id: evt-example-dispatch
    - sequence: 88413
      event_id: evt-example-revocation
      type: authority_revoked
      approval_id: approval-example-39

Now the review does not depend on comparing the policy gateway’s clock with the credential store’s clock. Sequence 88410 allowed the exact effect. Sequence 88411 released it. The gateway observed the durable commit at 88412. Revocation entered the same ordered control stream at 88413.

The sequence does not claim that every event in the company has one global order. It covers the transitions that must be ordered to answer this security question. A single global counter would add a fragile bottleneck and still would not explain causal relationships outside its domain.

The credential service should return its own commit evidence. That receipt needs a link to the dispatch that caused it.

commit_receipt:
  operation_id: op-example-771
  event_id: evt-example-store-commit
  caused_by_event_id: evt-example-dispatch
  dispatch_sequence: 88411
  resource_id: credential-example-12
  previous_version: 47
  committed_version: 48
  committed_value_hash: sha256:example-value
  service_observed_at: 2026-09-10T14:06:11.042Z
  acknowledged_by_gateway_sequence: 88412

The service timestamp remains useful. Keep it. The parent event and gateway acknowledgement carry the ordering proof.

Bind these fields cryptographically or store them in an append-only log. If a worker can replace caused_by_event_id after the incident, the chain is decoration rather than evidence. The operation ID, intended effect hash, approval ID, dispatch sequence, and committed version should agree across the authority and effect records.

This also closes a retry trap. A second attempt may reuse the stable operation ID, but it must receive a new attempt and dispatch event. Otherwise two attempts can claim the same place in the sequence and a reviewer cannot tell which one produced the effect.

Record how much to trust each clock

Wall time still matters for expiry, latency, and correlation with systems outside the gateway. Record clock quality beside the event instead of quietly assuming perfect synchronization.

clock_evidence:
  event_id: evt-example-store-commit
  clock_source: ntp-example-2
  offset_estimate_ms: -430
  uncertainty_ms: 35
  last_sync_at: 2026-09-10T14:05:54Z
  monotonic_elapsed_ms_from_dispatch: 124

The offset estimate explains why the commit’s wall time appears early. The uncertainty stops investigators from inventing an exact cross-service order when two events fall inside the error bound. A monotonic duration is useful within one process because wall-clock corrections do not make it run backwards.

If clock uncertainty exceeds the gap between two independent events, label their wall-time order indeterminate. Do not pick the prettier narrative. Use the causal links and enforcement sequence for decisions that depend on which action authorized another.

Fail closed when the chain breaks

A causal chain can be incomplete for mundane reasons. The gateway may crash after dispatch but before recording the downstream acknowledgement. The queue may deliver an old message after retention removed its parent event. A service may emit a receipt without the expected operation ID.

Treat those cases as unresolved effects:

causal_order_check:
  operation_id: op-example-771
  approval_sequence: 88410
  dispatch_sequence: 88411
  commit_ack_sequence: null
  revocation_sequence: 88413
  chain_complete: false
  effect_state: unknown
  automatic_retry_allowed: false
  required_action: reconcile_authoritative_state

Do not infer failure from a missing acknowledgement and dispatch the write again. Reconcile through the authoritative read path first. A gap in the audit chain is an observability failure, not proof that the external effect did not happen.

If approval is revoked while the effect is in flight, the sequence makes the race visible. Your policy still needs to define the outcome. Some operations can be cancelled before commit. Others must finish and trigger compensation. The audit record should state which boundary applied rather than letting a timestamp sort make that decision after the event.

Test the order, not the formatting

An eval for this control should alter clocks and delivery order while preserving the causal history. Useful fixtures include:

  1. The downstream clock runs 500 milliseconds behind the gateway.
  2. Time synchronization steps the gateway clock backwards during the run.
  3. The commit receipt arrives after the revocation event even though commit happened first.
  4. A retry uses the operation ID but presents an unknown dispatch event.
  5. A valid commit receipt names the wrong parent event.
  6. The gateway restarts and continues with a lower sequence number.
  7. Clock uncertainty is larger than the apparent interval between approval and commit.
  8. The effect commits, but the gateway never records its acknowledgement.

Assert that the verifier follows sequence and parent links, rejects broken chains, and reports uncertainty. A test that only checks chronological log output will reproduce the original mistake.

My review rule is simple: use timestamps to find the run. Use a durable enforcement sequence and causal event links to prove its order.

Claude Code: Building Production Agents That Actually Scale covers the operating controls that turn Claude Code and MCP workflows into bounded, reviewable production systems.