A parallel evaluation run returns every answer. The result count matches the case count. The grader reports perfect accuracy.
It can still be grading the wrong pairs.
If you compare the first received answer with the first scheduled case, completion order has quietly become part of your scoring contract. My rule is simple: establish which case produced each answer before letting the answer influence a release decision. Matching list lengths is insufficient.
This article uses a deliberately small synthetic fixture. It makes no model calls and describes no production incident. Both answers are wrong for their actual cases. Reversing their delivery order makes a positional grader report that both passed.
How two wrong answers become a perfect score
Suppose an authorization evaluator has these expected decisions:
cases = [
{"case_id": "allow-own", "expected": "ALLOW"},
{"case_id": "deny-other", "expected": "DENY"},
]
These labels are teaching inputs, not a real authorization policy. An actual access-control test would also specify the principal, resource and policy context.
The observed answers arrive in the opposite order:
results = [
{"run_id": "run-1", "case_id": "deny-other", "answer": "ALLOW"},
{"run_id": "run-1", "case_id": "allow-own", "answer": "DENY"},
]
The first result allowed the case that should have been denied. The second denied the case that should have been allowed. An exact-match grader should record zero correct answers.
Here is a broken implementation:
pairs = list(zip(cases, results, strict=True))
accuracy = sum(
case["expected"] == result["answer"]
for case, result in pairs
) / len(pairs)
It reports 1.0. The first expected value is ALLOW, and the first received value is ALLOW. Then it compares DENY with DENY. Neither comparison checks case_id.
Python’s documentation for zip specifies positional pairing. By default, zip also stops at the shortest iterable. The strict=True option, available since Python 3.10, raises an error for unequal lengths when the iterator is consumed. That is useful protection against truncated batches. It does not establish that two equally long sequences contain corresponding records.
There is a separate reason to check the collector. Python’s thread-pool example keeps a future_to_url mapping when consuming futures through as_completed. The result is associated with the URL that produced it, rather than the next URL in a separate list. Carry the same principle into an evaluation runner: retain the case identity at submission and use that association on completion.
Do not infer that every parallel API returns unordered results. Check the interface you use. Even where ordering is promised, explicit identity makes filtered results, retries and stored batches easier to validate.
Validate the batch before scoring it
The repair has two parts: reject a batch that cannot be reconciled, then compare each answer with its own expectation.
The complete Python fixture builds checked indexes before reaching this scoring section:
missing = sorted(set(expected) - set(observed))
unexpected = sorted(set(observed) - set(expected))
if missing or unexpected:
raise ValueError("case mismatch: " + json.dumps({
"missing": missing, "unexpected": unexpected}))
# Validate the entire batch before computing or emitting any score.
decisions = {
key: observed[key]["answer"] == expected[key]["expected"]
for key in sorted(expected)
}
Here, expected and observed are dictionaries indexed by validated case_id. The complete implementation rejects malformed records, empty string fields, duplicate case IDs and an empty manifest. It also requires every result to carry the requested run_id.
Do not replace the checked index with an unchecked dictionary comprehension:
# Unsafe when duplicate IDs have not already been rejected.
observed = {row["case_id"]: row for row in results}
A repeated ID would overwrite an earlier record. If the duplicate represents a retry, that could silently choose the last received attempt. Arrival order would be making a policy decision again.
For this fixture, any duplicate is invalid. A production runner that permits repeated attempts needs an explicit selection policy and an attempt identifier. Decide before the run whether you will grade the first attempt, the last allowed attempt or every attempt separately. Keep the rejected and failed attempts available for analysis. Do not pick the best answer after looking at the scores.
The batch validation completes before this implementation returns any aggregate. This matters when adapting the idea to streaming code. An iterator can produce several valid-looking pairs before a later record fails validation. Publishing a partial score and later discovering a duplicate is different from returning one validated report.
Run the counterexample and the repair
Download the fixture from the link above. It needs Python 3.10 or later and only the standard library. It does not write files or contact a service.
python3 --version
python3 agent-eval-result-identity.py
python3 agent-eval-result-identity.py --test
python3 -O agent-eval-result-identity.py --test
The demonstration prints:
positional_accuracy=1.0
{"accuracy": 0.0, "assessed": 2, "cases": {"allow-own": false, "deny-other": false}, "passed": 0, "run_id": "run-1"}
I executed this fixture with Python 3.14.7 before publication. Its 12 test methods passed in normal and optimised mode. These are local harness tests, not evidence of any model’s quality.
The regression I would keep is permutation invariance: shuffling the manifest or the result batch must not change the ID-based report. The fixture checks every ordering of its two cases and two results. It also tests a correct batch, a missing case, duplicates on both sides, a wrong run ID and an unknown case replacing an expected one without changing the row count.
Notice that the broken positional implementation remains in the download. One test requires it to produce the false perfect score. Keeping the counterexample makes the defect reproducible and shows what the repair prevents.
Identity is more than a model-generated label
In an actual runner, bind the case identifier to the submitted work outside the model response. A model returning case_id: allow-own does not prove that it received that case. Likewise, a caller-supplied run label is not an authenticated record of execution.
For a thread pool, a future-to-case mapping can preserve the orchestrator’s association. For persisted or remote jobs, bind the provider’s job identity to your manifest and retain the mapping. The mechanism depends on the execution interface; the fixture does not implement either transport.
A useful manifest also fixes what the case meant. If an expected answer or prompt changes while retaining the same ID, an ID join alone cannot detect that semantic change. Version the dataset and bind the run to the selected version. Where tampering is a concern, protect the manifest and result provenance using controls appropriate to that threat model.
Keep these checks distinct from numeric score validation and coverage accounting. A finite score can belong to the wrong case. A complete set of IDs can carry invalid scores. Passing either check does not waive the other.
What I would put in the release gate
Before accepting the quality score, require a report that names the run and dataset version, records the expected case set and reconciles it with the results. Reject unexpected cases and unexplained duplicates. If the execution produced incomplete coverage, report that state separately instead of silently reducing the evaluation set.
Then test the join itself. Reverse the result order. Remove a record. Replace a case with an unknown ID while keeping the count unchanged. Insert a duplicate. Your harness should either produce the same valid report or refuse the batch for a specific reason.
For the broader design of agent quality gates, AI Agent Evals covers evaluation as an engineering responsibility. Get the book on Leanpub.
Start with the shuffled batch. If it changes your reported accuracy, the next model comparison can wait until the join is fixed.