A team compares two versions of an agent. The evaluator receives the old answer first and the new answer second. It chooses the new answer, and the release report records an improvement.
What happens if the evaluator sees those same answers in the opposite order?
I would require that check before treating a pairwise judge as release evidence. An evaluator that changes its preference when two unchanged answers trade places needs investigation. Averaging its verdicts can conceal the problem.
This is a hypothetical release scenario, not a reported incident. The downloadable example below uses invented judgements to test the accounting. It makes no model calls and does not measure any current model’s bias.
A judge is another component to test
In Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena, Zheng and colleagues examine position bias alongside verbosity bias, self-enhancement bias and limits in reasoning. Their results concern the models and tasks studied in that paper. They do not establish the error rate of a judge you deploy today.
OpenAI’s evaluation guidance also identifies response order and answer length as challenges for model grading, and recommends calibrating automated evaluation against human judgements.
Pairwise comparison can still be useful. Asking which answer better satisfies a specific requirement is often easier to audit than interpreting an unexplained score of 8.4. But a pairwise label is only evidence of the judge’s preference under the supplied conditions. It does not establish that the preferred answer is correct, safe or authorised.
For a refund agent, test whether the ledger contains the permitted refund and no duplicate effect. For a code agent, execute the relevant tests. Use a model judge for criteria that need interpretation, such as whether an explanation accurately describes the observed failure. Keep those checks separate in the report.
Freeze the answers and change only their positions
Give each answer a stable internal identity: A for the baseline output and B for the candidate output. Keep those identities out of the judge’s visible labels if they would reveal which version you want to win. The judge can see neutral labels such as first and second.
Run two fresh judging contexts:
- AB: present answer A first and answer B second.
- BA: present answer B first and answer A second.
Keep the task, reference material, rubric and answer bytes fixed. Do not regenerate the answers for the second judgement. Otherwise you have changed both the answer and its position, and cannot interpret the comparison as an order check.
Likewise, do not give the second judge call the first verdict. You want a fresh assessment, not an invitation to defend an earlier choice. Record the judge model version and sampling settings. Preserve the rendered prompts and raw responses in storage appropriate for the data’s sensitivity.
The collector must map position labels back to stable answer identities. A vote for first in AB means A. A vote for second in BA also means A. Counting those as contradictory votes would create an accounting defect before you even examined judge behaviour.
Keep disagreement separate from a tie
The fixture contains four synthetic cases:
| Case | AB judgement | BA judgement | Mapped result |
|---|---|---|---|
| c1 | First | Second | A wins in both orders |
| c2 | First | First | Winner changes from A to B |
| c3 | Tie | Tie | Stable tie |
| c4 | Second | Unknown | Incomplete pair |
Case c2 deserves review. The judge chooses the first position twice, so it prefers a different answer after the swap. That observation is consistent with a position effect, but a single pair cannot separate position bias from ordinary sampling variability or other execution differences.
Case c3 is different. Both judgements explicitly say that the rubric does not distinguish the answers. Case c4 provides too little evidence for an order-consistency decision. A timeout, parser failure or missing response must not become a tie.
The audit reports one disagreement among three complete pairs. It also reports one unknown pair out of four scheduled cases. Those denominators belong next to each other. If the collector drops incomplete pairs entirely, a judge service with frequent failures can appear more dependable than it is.
The first position wins three of the five decisive individual judgements. Ties and unknowns are excluded from that particular denominator, but remain in the other counts. This fraction is a descriptive diagnostic, not a statistical finding. The fixture is tiny and deliberately constructed; one of those five judgements belongs to an incomplete pair.
Run the accounting fixture
Download judge-order-audit.py. It uses only Python’s standard library.
python3 judge-order-audit.py
python3 -O judge-order-audit.py
No-argument mode prints the synthetic report and runs twelve unit tests. The tests cover identity remapping, reordered records, duplicate and missing judgements, invalid labels, malformed inventories, unknown results and a tie changing to a decisive verdict. Optimised Python runs the same validation because the input checks do not rely on removable assert statements.
The fixture reports these counts:
{
"cases": 4,
"complete_pairs": 3,
"stable_A": 1,
"stable_B": 0,
"stable_tie": 1,
"order_disagreement": 1,
"unknown_pairs": 1,
"first_position_wins": 3,
"decisive_judgements": 5
}
The full output includes the disagreement and first-position fractions. When a denominator is zero, the corresponding fraction is null. Zero would incorrectly suggest that the audit observed no disagreement or no preference after assessing usable results.
For your own records, supply a manifest:
{
"cases": ["refund-explanation"],
"judgements": [
{"case": "refund-explanation", "order": "AB", "winner": "first"},
{"case": "refund-explanation", "order": "BA", "winner": "second"}
]
}
python3 judge-order-audit.py manifest.json
A valid manifest exits with status 0, even if every pair disagrees. That status means the accounting input is valid; it is not permission to release an agent. Invalid input exits with status 1. Define any release threshold separately, before examining the candidate’s results.
The manifest requires one record per case and order. A failed collector must write an explicit unknown or leave an input-validation failure. The program cannot verify that both calls received identical answers, that a model call really happened, or that the declared case list includes every scheduled case. Bind the manifest to the experiment inventory and raw call evidence in your actual harness. This download is an accounting example, not an evaluation service.
Separate order sensitivity from random variation
A paired swap is a useful first diagnostic. It is not enough to estimate a systematic position effect in a noisy judge.
For a larger experiment, predeclare repeated judgements in both orders. Randomise the execution order across cases so that AB calls do not all run before BA calls during a service or configuration change. Also repeat some calls without swapping the answers. Those unchanged-order repeats help show how much instability occurs without the intervention.
Keep the frozen answer pair as the analysis unit. Repeated judgements of the same pair are not additional independent tasks. If you calculate uncertainty intervals or compare judge versions, use a method that respects that grouping and the paired design. The downloadable fixture handles one pair of judgements per case; it deliberately does not calculate confidence intervals or significance tests.
Agree on adjudication before the run. A defensible policy for a consequential release is to send order disagreements and unknowns for review rather than converting them into candidate wins. Preserve the original verdicts after adjudication, so the final report distinguishes raw judge behaviour from the reviewed decision.
Do not repeatedly call the judge until it returns a stable preference. That creates a selection policy which can hide inconvenient observations. The earlier guide on retries and agent reliability explains why successful later attempts do not erase the earlier outcomes.
Stable preference still needs calibration
A judge can choose the same wrong answer in both positions. Both answers can also violate a requirement while one sounds more persuasive. Order stability therefore belongs beside agreement with reviewed reference cases, rather than replacing it.
Build calibration cases from failures your team understands. Include a fluent explanation that contradicts the ledger, a concise correct answer beside a longer incorrect one, and cases where both answers are unacceptable. Ask reviewers to apply the same rubric and inspect the evidence behind their decisions. Investigate disagreements between reviewers instead of treating an unexamined human label as infallible.
Track those results by task and failure type. A judge that handles everyday explanations well may still miss a permission violation. The slice-level release-gate example shows how an improved overall score can hide a regression in a consequential subset.
If you are building that evaluation workflow, AI Agent Evals covers task contracts, graders and release evidence. Get AI Agent Evals on Leanpub.
Before the next comparison run, freeze one answer pair, execute both orders and check the identity mapping by hand. Add a deliberate missing response and a first-position-only judge to your collector tests. The report should expose both defects without awarding the candidate an improvement.