Evals

Claude Code plans against resource version 41, but a state-bound write gate blocks execution after the resource changes to version 42

Claude Code approved one state. Its MCP write changed another

A reviewer approved a Claude Code change against resource version 41. By execution time, version 42 existed, but the MCP write replaced it anyway.

Two Claude Code eval cases are separated by a reset gate that verifies a clean sandbox before the second case starts

Claude Code passed the eval because the previous case cleaned up for it

A Claude Code eval can pass only because an earlier case left the sandbox in a helpful state. Give every case a fresh environment, a state fingerprint, and a reset receipt.

Claude Code receives the first 100 of 8,742 records, follows every cursor against one snapshot, and produces a collection completeness receipt

Claude Code checked the first page and called the job done

An MCP tool returned 100 clean records, so Claude Code declared a migration complete. The other 8,642 records were still waiting behind a cursor.

A Claude Code patch enters an independent evidence review before the agent explanation is revealed in a second pass

Do not let Claude Code explain its patch to the evaluator first

A polished Claude Code handoff can anchor the reviewer before the evidence is checked. Give an independent evaluator the task contract, patch, and raw test results first, then reveal the agent’s explanation.

A Claude Code trace with a missing MCP tool event is blocked by a completeness gate before replay and review

Make Claude Code prove its trace is complete before replay

A clean replay can be dangerously reassuring when the original trace dropped the tool call that changed production. Add sequence checks, expected-event accounting, and a completeness receipt before review.

Claude Code compacts its conversation while an external control-state checkpoint preserves authority, denied actions, pending effects, and evidence bindings

Keep Claude Code's safety state outside its context window

Context compaction can preserve the task while losing the decision that blocked a dangerous tool call. Store authority, denials, and unresolved effects in a control checkpoint outside the transcript.

Three individually valid Claude Code MCP reads are rejected because their source versions form a mixed production snapshot

Reject a Claude Code plan built from a mixed snapshot

Claude Code can read three correct facts and still build a plan for a system state that never existed. Record a coherent read set and reject evidence that crosses a consistency boundary.

A Claude Code test denominator records expected, discovered, selected, executed, skipped, and missing tests before a review can pass

Make Claude Code report the tests it did not run

A green test result says little when the agent selected the wrong suite or skipped the risky path. Add a test denominator to every Claude Code review packet.

MCP tool output is labelled untrusted, reduced to evidence, checked against the original Claude Code scope, and recorded in a reviewable trace

Do not let MCP tool output rewrite Claude Code's instructions

An MCP response can contain instruction-shaped text that pushes Claude Code beyond its approved task. Put a trust boundary between tool evidence and agent authority.

A Claude Code run pins an MCP capability manifest and stops when the live server schema, implementation, target scope, or side effects differ

Pin the MCP tool contract before Claude Code uses it

An MCP method can keep the same name while its schema, target scope, or side effects change. Pin a capability manifest so Claude Code cannot inherit a different tool after approval.