An evaluation adapter returns {"score": NaN}. The release script checks whether the score is below 0.8. It is not, so the script approves the candidate.

That result follows from a small, reproducible Python bug. It does not require a malicious model, a compromised judge or a sophisticated prompt. A non-finite value reaching a poorly written gate is enough.

I would block an automatic release on an invalid score and preserve the evaluation as invalid evidence. Silently treating it as a pass is unsafe; quietly converting it to zero also loses information about what broke. The score contract needs its own tests before the team uses it to approve an agent change.

The example below is a synthetic component fixture. It demonstrates Python parsing and comparison behaviour, not a vulnerability reported in a particular evaluation platform or a measured failure rate for an AI model.

Run the counterexample

Download the standard-library Python fixture, inspect it, then run:

python3 agent-eval-nan-gate.py
python3 agent-eval-nan-gate.py --test
python3 -O agent-eval-nan-gate.py --test

The file makes no network requests and invokes no model. On a phone, swipe horizontally inside a code block to read long lines. Local execution on Python 3.9.6 produced:

{"score":NaN} -> naive=PASS, strict=INVALID
{"score":1e400} -> naive=PASS, strict=INVALID
{"score":0.9} -> naive=PASS, strict=PASS
{"score":0.7} -> naive=FAIL, strict=FAIL

The vulnerable gate is short enough to inspect in full:

def naive_gate(raw):
    score = json.loads(raw)["score"]
    if score < 0.8:
        return "FAIL"
    return "PASS"

NaN means “not a number”. For the floating-point NaN in this example, both score < 0.8 and score >= 0.8 evaluate to false. The script only checks the first expression. Falling through to PASS therefore approves a value that never satisfied the acceptance condition.

Changing the last two lines to return "PASS" if score >= 0.8 else "FAIL" prevents this particular NaN pass. It still approves positive infinity, and it still collapses a broken result into the ordinary failure category. Validate the input before comparing it.

JSON parsing does not establish the score contract

RFC 8259, section 6 excludes NaN and Infinity from the JSON number grammar. Python’s default JSON decoder nevertheless accepts those constants as an extension. The Python JSON documentation describes this behaviour and provides parse_constant so callers can reject them.

The fixture installs a rejecting callback:

def reject_constant(token):
    raise ValueError("non-standard numeric constant: " + token)

record = json.loads(raw, parse_constant=reject_constant)

That closes the literal-constant path. It does not close every route to a non-finite float. The string 1e400 uses valid JSON number syntax, but in the executed Python fixture it decodes to positive infinity. The callback is not invoked for that ordinary numeric token.

RFC 8259 explicitly names 1E400 as a potential interoperability problem and allows implementations to limit numeric range and precision. A producer and consumer can disagree about representable magnitude even when the wire text follows the number grammar.

For a score bounded between zero and one, accepting such a value has no useful interpretation. Check the decoded value as well as the input syntax. If your application uses Decimal, NumPy scalars or another numeric representation, define and test that representation’s contract separately; the supplied validator deliberately accepts only built-in Python integers and floats.

Validate before making the threshold decision

The download requires one object containing exactly one score field. It also rejects duplicate members using object_pairs_hook, so two conflicting scores cannot silently become one. The earlier duplicate JSON member example covers that parsing boundary in more detail.

After checking the record shape, the validator applies these rules:

score = record["score"]
if type(score) not in (int, float):
    raise ValueError("score must be numeric, excluding bool")
if type(score) is float and not math.isfinite(score):
    raise ValueError("score must be finite")
if not 0 <= score <= 1:
    raise ValueError("score outside [0, 1]")

The exact type check is intentional. In Python, Boolean values behave as integers in many contexts. A broader isinstance(score, (int, float)) check accepts True, which can then satisfy a threshold of 0.8. This fixture rejects Booleans, numeric strings and null instead of coercing them into scores.

Python documents math.isfinite as false for infinity and NaN. The validator calls it only for floats. Built-in integers are finite, and converting a very large integer to a float merely to check it can introduce an avoidable overflow. The range comparison rejects those oversized integers directly.

The accepted interval is closed: zero and one are valid scores. The release threshold is fixed at 0.8, with equality accepted. These are teaching-policy choices, not a recommendation that every agent benchmark use the same threshold. A configurable threshold needs equivalent numeric validation and a versioned policy record.

The download then returns PASS, FAIL or INVALID. Its caller must map those states explicitly. An enum or nonempty string used as a Boolean would be a separate integration bug: even the string "INVALID" is truthy in Python. An automatic release should proceed only on an exact accepted state, with any exception or unrecognised state blocking that path.

Keep invalid evidence visible in the report

A failed task and an invalid evaluation require different follow-up. A valid score below the threshold can support a conclusion about the evaluated run. An invalid score says the harness lacks usable evidence for that decision. It may indicate a parser defect, missing observations, a calculation problem or an incompatible producer version. The fixture does not diagnose which cause applies.

Do not drop invalid rows and report the average of what remains without explaining the missing coverage. That changes the assessed population. For a release report, retain the scheduled trial count, valid result count and invalid result count, alongside the metric’s actual denominator. Preserve the original trial identity when investigating, and record a rerun as a new observation rather than overwriting the unusable one.

A different policy may allow human review when some evidence is missing. If so, make that an explicit review state with a named decision owner. Do not implement it as an exception handler that returns success to keep CI moving.

At the producer boundary, json.dumps(result, allow_nan=False) rejects non-finite float values instead of emitting Python’s non-standard constants. The fixture tests that setting too. It is a useful output check, but consumers still need their own validation: another producer, cached artifact or later transformation can bypass the encoder you control.

The validator is intentionally small. It does not impose transport byte limits, nesting limits, authenticated artifact provenance or a full trial schema. Those controls belong around it. A passing score from an unauthorised or mismatched evaluation file remains unsuitable release evidence.

Test malformed evidence as carefully as low scores

The fixture has twelve test methods. They cover the vulnerable NaN path, comparison behaviour, rejected constants, exponent overflow, threshold equality and endpoints. Further cases reject out-of-range values, wrong types, unexpected record shapes, duplicate scores and malformed JSON; the final test checks the strict output encoder. All twelve passed normally and under python3 -O. The validator uses explicit exceptions rather than removable assert statements.

These tests establish the local component behaviour only. They do not prove that a deployed CI job uses this function or maps INVALID to a blocked release. Exercise that integration separately with a disposable candidate: substitute an invalid result, run the actual decision command and verify that no release action occurs. Keep the invalid artifact and decision receipt so reviewers can inspect the path.

For the broader work of testing graders and making release decisions from retained evidence, see AI Agent Evals, with the current edition on Leanpub. This download is a separate teaching fixture, not a replacement for the book’s companion labs.

Add NaN, 1e400 and true to the next score-contract test. Verify both the validator’s result and the release caller’s response before using that gate to approve a candidate.