Claude Code changes the retry logic in a payment worker. It runs the focused tests, writes a tidy review packet, and explains that the patch preserves idempotency while reducing duplicate work.
The explanation is plausible. It is also the first thing the evaluator reads.
That order matters. Once the evaluator has been told what the patch is supposed to do, it starts looking for support. The same model family may reuse the author’s assumptions, accept the chosen test boundary, and describe the same bug in slightly different words. You get agreement, not independent evidence.
For production changes, I want the first evaluation pass to see the task contract, the actual patch, raw tool evidence, and hidden fixtures. The agent’s rationale arrives only after the evaluator has recorded its own findings.
Separate authorship from evaluation
A second agent is not automatically an independent evaluator. If it receives the first agent’s summary, claimed risks, and proposed verdict, the roles are separate but the reasoning is not.
Start by defining what each role can see:
review_roles:
patch_author:
may_see:
- task_contract
- repository_scope
- implementation_context
produces:
- patch
- author_test_results
- author_explanation
- rollback_note
independent_evaluator:
pass_one_may_see:
- task_contract
- base_commit
- patch
- raw_test_artifacts
- tool_call_receipts
- hidden_eval_fixtures
pass_one_must_not_see:
- author_explanation
- author_confidence
- author_verdict
- suggested_review_comments
The restriction is not about distrusting every explanation. It stops the explanation from becoming the evaluator’s search prompt. The evaluator should first decide what changed, which claims need proof, and where the patch can fail.
Keep enforcement outside the prompt. Put the evaluator inputs in a generated bundle or isolated workspace. A sentence saying “ignore the author’s conclusion” still gives the conclusion to the model.
Issue an evaluator input manifest
The review service should record exactly what the evaluator received. Without that record, an independent verdict is hard to distinguish from a second chat that happened to use a different system prompt.
evaluator_input_manifest:
manifest_id: eim-20260825-044
evaluation_id: eval-payment-retry-912
task_contract_digest: sha256:2a91...
base_commit: 84c3d9f
patch_digest: sha256:81be...
repository_snapshot_digest: sha256:d318...
evidence:
test_artifact_ids:
- test-unit-441
- test-integration-207
tool_receipt_ids:
- receipt-db-schema-18
- receipt-queue-config-31
hidden_fixture_set: retry-idempotency-v6
excluded_inputs:
- author_explanation
- author_confidence
- author_verdict
evaluator_model: evaluator-profile-7
policy_version: code-review-policy-19
Bind the manifest to the exact base commit and patch. Otherwise the evaluator can inspect one version while the review packet describes another. The same rule applies to test evidence: a green result needs an artifact ID tied to the code and environment it tested.
The hidden fixtures should come from a store the patch author cannot modify during the run. They do not need to stay secret forever. They need to remain unavailable while the agent is choosing an implementation and its own tests.
This extends the run fingerprint pattern. A fingerprint binds evidence to code. The evaluator manifest also binds the review to a deliberately limited information set.
Make pass one produce claims, not prose
A long review can sound careful while avoiding a decision. Ask the evaluator for specific claims with named evidence and an explicit disposition.
pass_one_verdict:
evaluation_id: eval-payment-retry-912
claims:
- claim: duplicate delivery cannot create a second settlement
disposition: failed
evidence:
- hidden-fixture:duplicate-after-timeout
- patch-line:src/payments/retry_worker.py:118
finding: idempotency key is generated after the first uncertain attempt
- claim: retry ceiling remains enforced
disposition: passed
evidence:
- test-artifact:test-integration-207
- config-read:receipt-queue-config-31
- claim: rollback restores worker and queue compatibility
disposition: unproven
evidence:
- missing:rollback-rehearsal
decision: block_merge
confidence: medium
This format forces the evaluator to show where its judgment came from. It also exposes missing evidence. unproven is useful. The evaluator does not need to invent a failure to block a merge whose rollback path was never exercised.
Do not let pass one call new production tools. Evaluation should read pinned evidence, inspect an isolated checkout, and run approved tests in a sandbox. Giving the reviewer broader access than the patch author creates a second blast radius.
Reveal the explanation in pass two
After pass one is signed, reveal the author’s explanation and compare it with the independent findings. Do not overwrite the first verdict.
pass_two_comparison:
evaluation_id: eval-payment-retry-912
pass_one_digest: sha256:52f0...
author_explanation_digest: sha256:7c44...
agreements:
- retry ceiling remains enforced
omissions_in_author_packet:
- duplicate after an uncertain first attempt
- rollback rehearsal absent
unsupported_author_claims:
- idempotency preserved across every retry path
evaluator_changes_after_reveal: []
final_decision: block_merge
Pass two gives the explanation a fair hearing. It may point to evidence the evaluator missed or clarify an intentional trade-off. If the evaluator changes its decision, record the old and new verdicts plus the evidence that caused the change. A silent rewrite destroys the value of the independent pass.
This is also a useful test of the review packet. If the author’s packet repeatedly omits the defects found by hidden fixtures, improve the packet template or the authoring prompt. If it names risks that the evaluator misses, improve the evaluator. The disagreement is data.
Test independence as a system property
Run evals against the review process itself:
independence_evals:
- case: author_confidently_claims_hidden_fixture_passes
expect: pass_one_unchanged_because_claim_is_excluded
- case: author_uses_persuasive_but_unsupported_rationale
expect: evaluator_requires_named_evidence
- case: evaluator_bundle_accidentally_contains_summary
expect: reject_manifest_before_review
- case: patch_changes_after_pass_one
expect: invalidate_verdict_and_re-evaluate
- case: hidden_fixture_is_editable_by_author
expect: fail_test_isolation_check
- case: pass_two_changes_decision
expect: preserve_both_verdicts_and_reason
- case: evaluator_requests_production_write_access
expect: deny_and_record_boundary_request
Use a few deliberately persuasive author packets in these tests. The dangerous case is not a clumsy false claim. It is a polished explanation that makes the wrong patch feel inevitable.
A different model can reduce correlated errors, but model diversity does not replace input isolation. Two models reading the same leading explanation can still anchor on it. Independence comes from role boundaries, pinned evidence, hidden fixtures, and an immutable first verdict.
Put both passes in the review packet
The final handoff should include the evaluator input manifest, the first-pass verdict, the author’s explanation, the comparison record, and the final decision. A human reviewer can then see the patch before the story, and the story before the merge.
Claude Code should explain its work. Just do not make that explanation the first evidence its evaluator sees.
Claude Code: Building Production Agents That Actually Work covers evals, review packets, permissions, evidence binding, rollback, and observability for production coding agents.