Claude Code changes a shared authorization helper. It runs 84 unit tests. All 84 pass, so the final summary says:
Tests: 84 passed, 0 failed
Status: ready for review
That looks reassuring. It is also missing the fact that the repository has 312 relevant tests, the integration suite never started, and 19 policy fixtures were excluded by the test command’s path filter.
The 84 passing tests may be perfectly valid. The claim that the change is ready for review is not.
A test result needs a denominator. Reviewers should see what the run expected to test, what it discovered, what it selected, what it executed, and what it left out. Otherwise a green count can hide the most important failure in an agent run: Claude Code tested the wrong slice of the system.
Start with the changed risk surface
Do not let the agent invent the denominator after it sees which tests pass. Define the expected test surface from the task and diff before execution.
For an authorization change, that record might look like this:
expected_test_surface:
run_id: cc-run-4628
base_commit: 51a9c6d
head_commit: a827b40
changed_components:
- auth/policy.py
- auth/roles.py
required_suites:
- auth-unit
- auth-integration
- policy-denial-fixtures
required_cases:
- expired_session_is_rejected
- suspended_user_is_rejected
- cross_tenant_access_is_rejected
allowed_omissions:
- suite: browser-e2e
reason: no user-interface path changed
approved_by: review-policy-v4
This is not a demand to run every test in the repository after every edit. It is a demand to state which evidence the change requires. A documentation edit and an authorization edit should not inherit the same test plan.
Build the record from repository ownership rules, changed paths, dependency metadata, risk labels, and explicit task requirements. Keep it outside the model’s final prose so the agent cannot quietly shrink the scope when a suite is slow or awkward.
The same principle applies to denial paths. If a permission change can allow a request, the test surface should include requests that must still be refused. I use denial path evals for consequential tool calls for the same reason. Success cases alone cannot prove a boundary.
Capture selection before execution
Test runners usually know more than their final summary reveals. Ask them for collection output before the real run, then retain the command, filters, environment, and collected test identifiers.
test_selection:
suite: auth-unit
command: pytest tests/auth --collect-only -q
discovered: 126
selected: 107
deselected: 19
filters:
path: tests/auth
marker_expression: "not integration"
selection_digest: sha256:9ab4...
Now the review packet has a useful question: why were 19 tests deselected?
Sometimes the answer is harmless. Perhaps those cases require a database and belong to the integration job. In that case, the packet should point to the separate integration result. If no other run owns them, the omission remains open.
Record test identifiers or a digest of the collected list, not only counts. Two selections can both contain 107 tests while covering different cases. A stable list also lets the reviewer compare this run with the previous trusted build.
Separate collected, started, completed, and passed
A runner can collect 107 tests, start 107, complete 84, and still print a green-looking fragment before the process is killed. Parallel runners make this easier to miss because one worker may disappear while the others finish normally.
Use a receipt that preserves each stage:
test_execution_receipt:
suite: auth-unit
selection_digest: sha256:9ab4...
collected: 107
started: 107
completed: 107
passed: 107
failed: 0
skipped: 0
worker_crashes: 0
exit_code: 0
started_at: 2026-08-18T09:51:04Z
finished_at: 2026-08-18T09:52:18Z
log_digest: sha256:31fd...
The arithmetic should reconcile. passed + failed + skipped must equal completed. completed must equal started unless the receipt explains an interruption. The selection digest should match the pre-run collection record.
Bind the receipt to the code and environment too. A complete test denominator is still weak if it belongs to another checkout or an old build. The run fingerprint pattern ties evidence to the commit, workspace, dependency state, and tool configuration that produced it.
Make every omission explicit
There are several ways a test disappears, and they should not collapse into one skipped count.
omissions:
- kind: deselected
test: tests/auth/test_policy.py::test_cross_tenant_access_is_rejected
reason: marker_filter
approved: false
- kind: skipped_at_runtime
test: tests/auth/test_ldap.py::test_disabled_directory_user
reason: LDAP_TEST_URL missing
approved: false
- kind: required_suite_not_started
suite: auth-integration
reason: database container unavailable
approved: false
- kind: expected_case_not_discovered
test: expired_session_is_rejected
reason: unknown
approved: false
These failures need different responses. A runtime skip may point to a missing environment dependency. A test that was not discovered may have been renamed, deleted, or broken during collection. A required suite that never started is not a passing suite with zero tests.
Never let Claude Code turn 0 tests collected into success merely because the test command returned zero in a repository where that exit code has been normalized. The policy should compare the actual receipt with the expected surface, not trust the shell status by itself.
Gate the review claim
The final decision can be mechanical:
test_evidence_gate:
required_suites_present: false
expected_cases_present: false
selection_reconciled: true
execution_reconciled: true
unapproved_omissions: 3
decision: block_ready_for_review
next_action: run auth-integration and restore missing denial fixture
Claude Code can still hand over the patch. It should describe the evidence honestly:
The unit suite passed 107 of 107 selected tests.
The change is not ready for approval because the required integration suite did
not run and one expected denial fixture was not discovered.
That is a much better review packet than 84 passed. It gives the engineer a bounded next action instead of a vague green signal.
A useful test denominator does not guarantee that the tests are good. It stops missing evidence from masquerading as positive evidence. For production agent work, that distinction is worth enforcing in code.
Claude Code: Building Production Agents That Actually Work includes practical patterns for evals, review packets, observability, permissions, and bounded Claude Code runs.