Your retrieval migration returns the same documents in the same order. The smoke test passes. Then the answer service starts accepting passages that the old service rejected.
That can happen without changing an embedding or losing a neighbour. A cosine-similarity cutoff of 0.8 cannot be copied unchanged into a squared-L2 distance filter. For unit vectors, the equivalent squared-L2 cutoff is 0.4, with the comparison pointing the other way.
I would make score semantics part of the retrieval adapter’s tested contract. A field named score is too vague to govern whether an agent has enough evidence to answer.
The example below uses synthetic two-dimensional vectors. It reproduces the threshold error locally; it is not a benchmark of embedding quality or a claim that a particular RAG framework has this defect.
The same ordering can hide a different decision
Faiss’s metric documentation says that METRIC_L2 reports squared Euclidean distance. It omits the square root because that transformation preserves distance ordering. The same page explains how normalising both database and query vectors makes inner-product search equivalent to cosine-similarity search.
pgvector’s query documentation exposes a different set of named operators: <-> gives L2 distance, <=> gives cosine distance, and <#> gives negative inner product. Its cosine-similarity example explicitly computes 1 - cosine_distance.
These are documented conventions, not defects. An adapter creates the defect when it strips away the metric name and leaves downstream code to guess what a floating-point number means.
For unit vectors, let s be cosine similarity. The relationships are:
cosine distance = 1 - s
squared L2 distance = 2 - 2*s
L2 distance = sqrt(2 - 2*s)
Therefore an inclusive acceptance rule of s >= 0.8 becomes:
cosine distance <= 0.2
squared L2 distance <= 0.4
L2 distance <= sqrt(0.4), approximately 0.632456
The displayed decimals describe mathematical cutoffs. Floating-point implementations can land on opposite sides of an exact boundary through rounding. Choose a comparison policy, test values near the boundary in the actual backend, and retain full precision when calculating the converted threshold. Rounding the L2 cutoff to 0.63 changes the rule slightly.
The L2 conversions require unit-length query and document vectors. The relationship between cosine similarity and cosine distance does not require you to store unit vectors, provided the backend actually computes cosine distance. Do not apply the L2 formula to arbitrary raw embeddings.
Run a migration that passes the ranking test
Download rag-score-contract.py, inspect it, then run:
python3 rag-score-contract.py
python3 rag-score-contract.py --test
python3 -O rag-score-contract.py --test
The fixture uses only the Python standard library. It makes no network requests and calls neither Faiss nor Postgres. It tests the arithmetic behind the documented metrics, leaving backend integration as a separate check.
The query is (1, 0). Three synthetic document vectors have cosine similarities of 0.9, 0.7 and 0.3. Each vector is normalised before scoring. The demonstration prints:
document cosine cosine_distance squared_l2 l2
A 0.900 0.100 0.200 0.447
B 0.700 0.300 0.600 0.775
C 0.300 0.700 1.400 1.183
cosine >= 0.8: A
wrong squared_l2 <= 0.8: A,B
correct squared_l2 <= 0.4: A
wrong cosine_distance >= 0.8: none
Every correctly directed sort gives A, B, C. A test that checks only the top document, or even the complete ordering, misses the error.
Copying 0.8 into the squared-L2 filter admits B. Copying both the number and the higher-is-better comparison into a cosine-distance filter rejects all three. The first mistake broadens the evidence set; the second can make a working corpus appear unable to answer.
The script has twelve unit tests. They cover rank agreement, converted acceptance, the deliberately incorrect cutoff, invalid inputs and the effect of unnormalised inner product. Both ordinary and optimised Python runs pass. The tests establish behaviour for this fixture, not retrieval accuracy for a real corpus.
In particular, A, B and C have no human relevance labels. Calling B an irrelevant document would go beyond the experiment. B is a document that the original mathematical rule excluded and the faulty migration admitted. Whether the original rule was useful is a separate evaluation question.
Normalisation belongs in the contract too
Inner product can reward vector magnitude. In the fixture, A has coordinates (0.9, sqrt(0.19)). Another vector points in a direction with cosine similarity 0.7 but has length ten: (7, sqrt(51)).
Against query (1, 0), their raw inner products are 0.9 and 7. Raw inner-product search ranks the longer vector first, despite its lower cosine similarity. Normalising both vectors restores the cosine ordering. This follows the distinction documented by Faiss: database-vector norms affect inner-product ranking.
Normalising only the query is insufficient. Nor should you assume that every embedding model emits unit vectors because a previous model did. Check the provider’s contract and measure the vectors reaching the index after your preprocessing. If magnitude is intentional for the retrieval method, changing it is a retrieval-design decision that needs evaluation.
Reject zero vectors before a cosine calculation. Define what happens to non-finite values and dimension mismatches rather than letting them produce an empty result set that looks like an ordinary no-answer outcome. The fixture rejects these inputs. Its validation is deliberately small and is not a production ingestion library.
For an application-owned adapter, I would record fields such as:
{
"metric": "cosine_similarity",
"higher_is_better": true,
"normalisation": "query_and_document_unit_l2",
"threshold": 0.8,
"threshold_inclusive": true,
"embedding_version": "fixture-v1",
"score_transform": "none"
}
This is an illustrative application contract, not a native Faiss or pgvector configuration. Keep the original backend metric and transformation in the adapter’s versioned definition. If the application standardises on cosine similarity, test the conversion at that boundary and reject an unknown metric instead of silently treating it as cosine.
A cross-encoder reranker, reciprocal-rank fusion score or vendor-defined relevance score needs its own interpretation. None of the equations above grants permission to convert an arbitrary score into cosine similarity.
Test the acceptance set before switching traffic
Freeze a small corpus and a query set that exercise your actual answer gate. Include queries whose best candidate sits above the threshold, below it and near it. Record document identifiers and raw scores before any adapter transformation. Use the same corpus snapshot and query embeddings on both paths so a metric change is not confused with a simultaneous model upgrade.
First run exact search where the backend supports it. Compare ranking and the set of accepted document identifiers independently. For the unchanged-vector migration demonstrated here, a correctly converted threshold should preserve the acceptance set apart from explicitly handled numerical boundary effects.
Then run the intended approximate index. pgvector documents that approximate indexes trade recall for speed and can return different results. Matching metric semantics does not guarantee matching candidate retrieval. A document absent from the candidate set cannot pass a later threshold, however accurate the conversion is.
Keep authorisation filters unchanged during this comparison. A retrieval migration must preserve permission checks as well as relevance behaviour. The article on authorisation filters and top-k retrieval covers why a permitted document can disappear when filtering happens after a limited candidate search. That is a separate failure from a score being interpreted incorrectly; the two can occur in the same request.
For each test query, save enough evidence to distinguish them: the adapter version, metric, normalisation policy, index configuration, candidate identifiers, raw scores and accepted identifiers. Avoid putting sensitive passage text into broad-access logs merely to diagnose a numeric conversion.
Inspect the final no-answer decision too. A threshold migration can change answer coverage while a dashboard reports accuracy only for the remaining answers. Risk and coverage need separate denominators; otherwise a stricter accidental gate can appear to improve the system.
A cosine cutoff is not a confidence estimate
A cosine similarity of 0.8 does not mean an 80% probability that a passage supports the answer. The fixture chooses that number to make the conversion visible. It does not recommend it as a production setting.
Choose a cutoff using labelled questions and passages from the intended workload. Evaluate relevant evidence that was rejected as well as unsupported evidence that was accepted. Repeat that work when you change the embedding model, chunking or corpus, even if the metric name stays the same. Algebra can preserve a rule across equivalent representations; it cannot establish that the rule still separates useful evidence from misleading matches.
For the broader workflow around testing and reviewing agent behaviour, my book Claude Code: Building Production Agents That Actually Scale on Amazon Kindle covers production controls around agent work. The fixture here is a focused retrieval-adapter check, not an excerpt or a claim that the book documents these backend APIs.
Before your next retrieval migration, add one assertion for the accepted document set beside the existing ranking assertion. Run it through the real adapter, with the metric and normalisation policy recorded in the test receipt.