Your agent's eval score improved. Its permission checks got worse.

A support agent gets better at answering ordinary questions. It also gets worse at refusing account changes that the caller is not allowed to make. The evaluation dashboard reports an improvement.

Both statements can be true. If ordinary questions dominate the dataset, enough extra correct answers will outweigh the failed permission checks in an average.

I would not let that average authorise a release. A successful password-reset explanation cannot compensate for changing somebody else’s account. Some failures need their own release gate, regardless of the overall score.

The example below is synthetic. It demonstrates an accounting failure and an application-owned gate, not measured behaviour from a particular model or a report of a customer incident. You can run the complete fixture without an API key.

The average is correct; the decision is wrong

Consider a fixed evaluation set with 900 routine support cases and 100 permission-boundary cases. A routine pass means that the answer meets the task’s rubric. A boundary pass means that the attempted unauthorised action is refused without the forbidden state change. The grader must inspect that state, not merely accept a polite refusal in the final answer.

Here are the deliberately constructed results:

The overall pass rate rises from 91% to 93.5%. The candidate gains 45 routine passes and loses 20 boundary passes. Nothing is wrong with the addition. What is wrong is using the sum as permission to ship.

Google’s guidance on classification metrics explains why accuracy can mislead when classes are imbalanced or different mistakes carry different costs. This agent example uses task pass rates rather than a binary classifier’s accuracy, but the decision problem is similar: a count treats every passed case as an equal contribution, even when the consequences are very different.

This is not Simpson’s paradox. The case mix stays fixed, and only one family gets worse. Calling every misleading average a statistical paradox obscures a simpler problem: the release rule has allowed one kind of success to buy permission for another kind of failure.

Define the slices before seeing the result

A slice is a named subset of the evaluation cases. Start with distinctions that change a release decision. For a support agent, ordinary explanations and unauthorised account changes clearly belong in different groups. Depending on the product, unsupported languages or high-value refunds may need separate treatment too.

Do not build a hundred dashboard categories because the metadata makes it easy. Each required slice needs a reason, an owner and a defined response when the result is unacceptable. Otherwise slicing becomes another way to display bad news without acting on it.

OpenAI’s evaluation best practices recommends task-specific evaluations, production-representative data, edge cases and adversarial cases, with human feedback to calibrate automated scoring. The gate here is my implementation choice, not an OpenAI setting or a vendor-prescribed threshold.

For this demonstration, the policy is intentionally simple:

• Require a verdict for every expected case, and reject unexpected case identifiers.
• Require at least 90% overall passes.
• Require every permission-boundary case to pass.

The last rule is a zero-observed-failure requirement on this fixed regression suite. It does not mean that passing 100 cases proves a zero production failure rate. A release owner still needs to judge whether those cases cover the threats, whether the grader is trustworthy and whether repeated agent trials reveal instability.

Freeze the required inventory and policy with the comparison. If the candidate changes the cases or the grader, that is a different experiment. Do not present its score as a like-for-like improvement without explaining the change.

Run the release-gate fixture

Download agent-eval-slice-gate.py, then run:

python3 agent-eval-slice-gate.py
python3 -O agent-eval-slice-gate.py

The script uses Python’s standard library. It creates synthetic Boolean verdicts in memory, checks the exact expected case identifiers and calculates the results. It does not execute an agent, contact a service or modify an account.

The comparison prints:

baseline: overall=91.0%, slices={'boundary': (100, 100), 'routine': (810, 900)}, release=ALLOW
candidate: overall=93.5%, slices={'boundary': (80, 100), 'routine': (855, 900)}, release=BLOCK

Each tuple is (passed, expected). The script then runs nine unit tests. Both ordinary and optimised Python execution passed those tests before publication. These are nine distinct tests executed in two modes, not eighteen distinct tests and not live-model trials.

The tests cover the improved-average regression, a single boundary failure, the overall threshold and missing evidence. An extra case, an unknown verdict and the integer 1 in place of a Boolean verdict also block the gate. That last check matters because Python treats bool as a subclass of int; an overly permissive input check can quietly accept a malformed result.

The fixture uses explicit conditionals and unittest assertions, so running with -O does not remove the release checks. It also exits successfully when all its unit tests pass, even though the deliberately bad candidate prints BLOCK. To integrate the policy into CI, your adapter must turn release_allowed(candidate) == False into a nonzero job exit. Do not use the demonstration script’s process exit as the candidate’s release verdict.

Keep missing evidence out of the success column

A timeout is not a successful refusal. Nor is a missing trace proof that the forbidden write never happened.

The fixture rejects an absent case or a verdict of None. In a real harness, retain richer outcomes such as execution error, grader error and incomplete observation. Preserve their causes for investigation, but do not silently drop them from the expected denominator.

That distinction is the same one behind review packets where unknown is not a pass. A gate needs enough evidence to make the decision it claims to make.

The dictionary in this fixture cannot detect duplicate rows that an upstream loader has already overwritten. A production adapter should reject duplicate case identifiers before constructing the mapping. If trials repeat, identify each observation with both a case identifier and a trial identifier. Keep retries as separate observations under a declared retry policy; replacing a failure with the last successful attempt changes what the pass rate means.

Similarly, the fixture trusts its Boolean inputs. It cannot tell whether a grader marked a forbidden write as safe. Test the grader with deliberately corrupted traces and state snapshots. The result store should retain the tested implementation, case-set and grader versions so a reviewer can reconstruct what produced the verdicts.

A blocking slice still needs an investigation

Once the candidate is blocked, inspect the failed boundary cases before tuning the prompt. Did a tool adapter skip an authorisation check? Did a refusal appear in the text while a queued action continued? Did the expected state come from the wrong tenant?

Those are different defects. A new prompt may conceal them in a small sample without repairing the execution path.

Use isolated initial state for each trial. The earlier article on an eval passing because the previous case cleaned up for it addresses that separate problem. Clean accounting cannot rescue contaminated observations.

For stochastic runs, preserve the paired case comparisons and trial outcomes rather than reporting only a rounded average. Decide in advance how much evidence the release requires. A small slice with no observed failures remains weak evidence about rare failures. Repeated tests on near-identical prompts do not automatically provide broad coverage.

Keep a representative workload score for understanding ordinary service quality. Maintain a separate challenge set for dangerous behaviours. Report their denominators and purposes separately instead of pretending that an oversampled attack suite predicts the production incident rate.

Put the decision in the release record

A useful release record says which policy was applied, which slices were required and what stopped the candidate. For this fixture, the explanation is short: the overall floor passed, but 20 of 100 permission-boundary cases failed. The candidate stays blocked.

That wording gives an engineer something to investigate. “Quality improved by 2.5 percentage points” does not.

For the wider process, my book AI Agent Evals covers task contracts, evaluation harnesses and the evidence behind release decisions. Get AI Agent Evals on Leanpub.

Before approving your next agent update, ask which failures the average is allowed to hide. If the answer includes a permission violation, change the gate before changing the model.