A Claude Code eval asks the agent to repair a failed deployment. The case creates a temporary feature flag, writes a deployment record, and then removes both during cleanup.
The next case checks whether Claude Code refuses to deploy without an approved flag. It passes. The agent finds the flag missing and stops exactly as expected.
Run that second case by itself and it fails.
The shared sandbox already contained an approved flag from a developer’s earlier test. In the full suite, the first case happened to delete it. The refusal looked like agent judgment, but the result depended on test order.
This kind of false confidence is easy to miss. The prompt is identical. The evaluator score is green. The trace for the passing run may even look sensible. What changed was the state outside the transcript.
Give each eval its own environment
A serious eval case should own its repository checkout, MCP namespace, credentials, queues, caches, database records, and clock assumptions. Sharing one long-lived sandbox because provisioning is slow turns the suite into a sequence of hidden dependencies.
The case manifest should name what it expects before Claude Code receives the task:
eval_case:
id: deploy-without-approved-flag
fixture_version: deploy-fixtures-v12
environment_id: eval-7f31
repository_commit: 81ac932
principal: eval-agent-7f31
mcp_namespace: eval-7f31
expected_start:
feature_flags: []
deployment_records: []
pending_jobs: 0
cache_generation: 441
forbidden_targets:
- staging-shared
- production
An environment identifier in a log is not isolation by itself. The database schema, object-store prefix, service identity, queue topic, and MCP resource scope must all derive from it. A global cache or shared administrator credential can quietly reconnect supposedly separate cases.
Fresh infrastructure is the cleanest option. If the platform cannot afford that for every case, use namespaced resources and prove that the namespace is empty before the run. Do not treat a cleanup command returning exit code zero as proof.
Fingerprint the starting state
The evaluator needs evidence about the world that produced the answer. Record a state fingerprint after provisioning and immediately before dispatch.
start_fingerprint:
environment_id: eval-7f31
observed_at: 2026-09-05T09:42:08Z
repository_tree: sha256:ad41...
database_snapshot: sha256:8be2...
feature_flag_set: sha256:e3b0...
mcp_capabilities: sha256:17cc...
queue_depths:
deploy-jobs: 0
audit-events: 0
cache_generation: 441
foreign_canaries: []
status: clean
Use hashes for comparison, but keep a structured inventory behind them. When a fingerprint changes, the operator needs to know whether the difference came from a flag, a queue message, a stale MCP session, or a new tool contract.
Include the fingerprint in the run record and bind the result to it. This follows the same principle as binding test evidence to the code it actually tested. A score without its starting state is evidence about an environment you may never be able to recreate.
Use canaries to catch cross-case leakage
Give every case a unique canary, such as eval-case-7f31, and place it in the resources that should remain private to that case. Search for foreign canaries before the run and search for the current canary after cleanup.
The checks should cover more than database rows:
- MCP sessions and connection pools
- prompt and tool-result caches
- retry queues and dead letter records
- temporary cloud roles and tokens
- object-store prefixes
- background jobs still running after the agent stops
A foreign canary at startup means the case is contaminated. The right result is invalid_environment, not pass or fail. Otherwise the scoring system blames Claude Code for state it did not create, or gives it credit for help supplied by another case.
This also catches work that outlives the visible run. Stopping Claude Code does not cancel queued MCP work. A delayed job from case A can mutate case B after its clean-state check unless the queue and worker lease belong to the case namespace.
Randomise order, then run failures alone
Run the suite in several shuffled orders. Any case whose result changes with order has found a dependency, not a model capability.
Then rerun every failure, and a sample of passes, in a newly provisioned environment. This catches the opposite problem: a failure caused by leaked state that disappears when isolated. It also catches passes that relied on a previous case to prepare or remove something.
Keep concurrency as a separate mode. A suite that passes serially can fail when two cases share a rate limit, account budget, cache key, or worker pool. The concurrency test should use deliberately colliding fixtures as well as supposedly isolated ones.
isolation_matrix:
serial_original_order: pass
serial_shuffled_orders: 20
order_dependent_results: 0
isolated_reruns: 12
isolated_result_mismatches: 0
concurrent_runs: 8
foreign_canary_hits: 0
Do not average away a mismatch. One order-dependent result invalidates that case until the leaked dependency is understood.
Require a reset receipt
Cleanup is part of the eval, not background housekeeping. Destroy the namespace, revoke the case identity, stop its workers, drain or delete its queues, invalidate its sessions, and verify that its canary no longer appears in shared systems.
reset_receipt:
case_id: deploy-without-approved-flag
environment_id: eval-7f31
resources_destroyed: 14
credentials_revoked: 1
workers_stopped: 2
pending_operations: 0
canary_search:
value: eval-case-7f31
remaining_locations: []
shared_baseline_restored: true
verified_at: 2026-09-05T09:47:51Z
decision: safe_for_next_case
If cleanup cannot prove that pending work stopped, quarantine the environment. Do not release it to the next case. An unresolved external write remains an unknown outcome, just as it would in a production MCP workflow.
The receipt should sit beside the prompt, tool trace, output, evaluator decision, model version, fixture version, and start fingerprint. That gives a reviewer enough evidence to distinguish a Claude Code failure from a broken test harness.
A green eval is useful only when the case started where you think it started. Isolate the environment, fingerprint it, hunt for foreign canaries, shuffle the suite, and make cleanup prove its work before another case runs.
Claude Code: Building Production Agents That Actually Work covers evals, MCP boundaries, observability, permissions, rollback, cost control, and review packets for teams moving beyond Claude Code demos.