Evals

A review packet has passing tests but a missing MCP write receipt. Further agent writes are blocked while an authorised operator reconciles the outcome.

Claude Code review packets: unknown is not a pass

Passing tests cannot settle a missing MCP write receipt. Give unresolved effects a status that the approval gate cannot ignore.

An agent writes revision 41, an engineer writes revision 42, and conditional recovery refuses to overwrite the engineer's repair

Claude Code rollback must not erase the next engineer's fix

A saved snapshot can undo the agent’s change and somebody else’s repair. Make recovery conditional on the revision the agent wrote.

One Claude Code MCP call fans out to paid workers while a shared cost capability blocks work beyond the 25 dollar limit

One Claude Code MCP call spent the whole agent budget

The Claude Code run had a $25 limit and made one MCP call. Hidden fan-out behind the server produced a $286.40 bill.

A Claude Code incident replay forks into a faithful path using captured MCP evidence and a misleading path that calls today's live tools

Your Claude Code incident replay used today's tool output

The same prompt and repository commit produced a clean replay because an MCP tool returned newer evidence. The team had rerun the task, not reproduced the incident.

Claude Code plans against resource version 41, but a state-bound write gate blocks execution after the resource changes to version 42

Claude Code approved one state. Its MCP write changed another

A reviewer approved a Claude Code change against resource version 41. By execution time, version 42 existed, but the MCP write replaced it anyway.

Two Claude Code eval cases are separated by a reset gate that verifies a clean sandbox before the second case starts

Claude Code passed the eval because the previous case cleaned up for it

A Claude Code eval can pass only because an earlier case left the sandbox in a helpful state. Give every case a fresh environment, a state fingerprint, and a reset receipt.

Claude Code receives the first 100 of 8,742 records, follows every cursor against one snapshot, and produces a collection completeness receipt

Claude Code checked the first page and called the job done

An MCP tool returned 100 clean records, so Claude Code declared a migration complete. The other 8,642 records were still waiting behind a cursor.

A Claude Code patch enters an independent evidence review before the agent explanation is revealed in a second pass

Do not let Claude Code explain its patch to the evaluator first

A polished Claude Code handoff can anchor the reviewer before the evidence is checked. Give an independent evaluator the task contract, patch, and raw test results first, then reveal the agent’s explanation.

A Claude Code trace with a missing MCP tool event is blocked by a completeness gate before replay and review

Make Claude Code prove its trace is complete before replay

A clean replay can be dangerously reassuring when the original trace dropped the tool call that changed production. Add sequence checks, expected-event accounting, and a completeness receipt before review.

Claude Code compacts its conversation while an external control-state checkpoint preserves authority, denied actions, pending effects, and evidence bindings

Keep Claude Code's safety state outside its context window

Context compaction can preserve the task while losing the decision that blocked a dangerous tool call. Store authority, denials, and unresolved effects in a control checkpoint outside the transcript.