You generate questions from a policy manual, shuffle the rows and reserve some for testing. The test questions have different identifiers. Nobody has pasted the test answers into the prompt.

That sounds like a clean evaluation. It may still answer a much narrower question than the one you intend to ask.

Suppose the development set asks when a customer can request a refund. The test set asks how long a customer has to return a purchase. Both derive from the same paragraph. After several rounds of prompt changes and retrieval tuning, success on the second question is weak evidence that the system handles a different policy family.

My rule is simple: split at the level of the generalisation claim, not merely at the level of the CSV row. If the claim is that a RAG system handles previously untuned document families, keep each family on one side of the development/test boundary.

This article includes an offline manifest checker. It tests declared source lineage and duplicate question/answer pairs. It does not measure a model, discover semantic similarity or prove that a dataset is uncontaminated.

Decide what the test is supposed to establish

There are legitimate reasons to test new questions against familiar documents. A support assistant may serve the same small handbook every day. You might want to measure whether it handles fresh wording, ambiguous questions or multi-turn follow-ups against that fixed corpus.

Call that a fixed-corpus question-generalisation test. Shared source documents are not automatically a defect in that experiment.

A different claim needs a different split: “Our retrieval configuration works across policy families that we did not use during development.” Randomly separating questions from the same documents does not establish that. You need held-out source families, with enough variety to make the comparison useful.

The principle comes from grouped evaluation. The scikit-learn cross-validation guide explains that samples within a group can be dependent. When the goal is generalisation to unseen groups, the validation groups must not appear in the paired training set. Its examples use patients and experiments. Applying that principle to RAG source families is an evaluation design choice here, not a scikit-learn feature that understands documents.

For a prompt-based application, “development” includes more than weight training. It includes inspecting failures and choosing the prompt, chunking policy, reranker settings or retrieval thresholds in response. Those choices can adapt to a source family even when the model’s weights never change.

Retrieving the document is not the same as leaking the answer

A RAG system is supposed to retrieve source material. Removing the held-out family’s documents from the test-time index would change the task if production users are allowed to retrieve them.

For the experiment described here, freeze the system configuration using development families. At evaluation time, make the held-out documents available through the normal authorised ingestion and retrieval path. Keep the held-out questions and reference answers out of prompt selection and tuning. Do not place the generated answer key in the retrieval corpus.

This distinction matters. A correct test can let the system read the travel policy while preventing the developer from repeatedly tuning against the travel-policy test answers.

If your experiment instead measures temporal generalisation, define the information cutoff explicitly. A document published after the simulated query time must not slip into the index. If it measures access control, use the caller’s permitted document set. One split cannot establish all these properties at once.

The scikit-learn guidance on data leakage warns that using test data to make model choices produces optimistic estimates. The RAG equivalent is often less visible than a misplaced fit call: someone reads the held-out failures, adjusts the prompt and continues calling the same set independent evidence.

Track the family before creating variations

Start with a source register. A useful record has a stable family identifier, a version identifier and the source location. Carry that lineage into every generated question, paraphrase and expected answer.

A new PDF filename should not automatically create a new family. Nor should exporting the same handbook to HTML. For a family-generalisation test, related revisions normally stay together. Otherwise a minor wording change can turn last month’s development document into this month’s supposedly unseen test document.

That is a policy decision, not something a hash can settle. Different chapters of a large manual may be independent enough for one experiment and tightly related for another. Define the grouping rule before looking at the scores, and record exceptions rather than quietly renaming difficult cases.

If a question depends on several documents, connect their families before assigning splits. For example, a question that combines a refund policy with a regional exception links those sources for this purpose. Group the connected component or exclude that cross-family question under a declared rule. Giving the question only its first source’s identifier hides the dependency.

Large connected components can make a balanced split impossible. Report that limitation. Breaking the component merely to obtain a neat ratio defeats the reason for grouping.

Run the manifest audit

Download rag-eval-split-audit.py. The script uses Python’s standard library and makes no network requests.

python3 rag-eval-split-audit.py
python3 -O rag-eval-split-audit.py

The synthetic demonstration prints:

shared family: BLOCK
disjoint families: PASS

It then runs nine distinct unit tests. They cover a valid split, family overlap across the split, duplicate question/answer pairs despite different family labels, duplicate case identifiers, malformed records, missing or unknown splits, padded identifiers and a documented detection limit. The tests passed in both normal and optimised Python before publication.

To check a manifest of your own, save a JSON array in this shape. These are deliberately simple example policies, not actual customer terms:

[
  {
    "case_id": "refund-001",
    "family_id": "refund-policy",
    "split": "dev",
    "question": "When can a customer request a refund?",
    "answer": "Within 30 days."
  },
  {
    "case_id": "travel-001",
    "family_id": "travel-policy",
    "split": "test",
    "question": "What is the travel allowance?",
    "answer": "The example allowance is 100."
  }
]

Run the checker before executing an expensive evaluation:

python3 rag-eval-split-audit.py manifest.json

In manifest mode, a clean result exits with status 0. A rejected manifest exits with status 1. The demonstration mode instead exits according to its unit-test result, so do not use the demonstration command as your dataset gate.

Change the second record’s family_id to refund-policy and the audit blocks it. That is intentional even though the question and answer differ: this particular gate enforces disjoint declared families. It does not decide whether the family labels themselves are correct.

A passing audit has a narrow meaning

The checker compares question/answer pairs after whitespace normalisation and hashes the resulting pair. It preserves case and punctuation. This catches an exact duplicate with changed spacing even if somebody assigned it a different family label.

It misses paraphrases with false lineage. One unit test deliberately demonstrates that limit: a differently worded refund question, labelled as a separate family, passes. That passing test documents a blind spot; it is not evidence that the case is independent.

For a real dataset, review suspicious near-duplicates and trace them back to the source register. Similarity search can help find candidates, but a similarity score does not decide whether two questions test independent reasoning. Shared boilerplate can create false positives, while a substantial paraphrase can conceal the same answer.

This fixture also lacks an expected case inventory. It requires both splits to be present, but cannot detect that an upstream exporter omitted half the intended test set. Bind the manifest to an approved inventory and check exact membership before scoring. The separate release-gate article demonstrates why missing results must not disappear from a denominator.

Keep dataset integrity and performance interpretation separate. A clean family split cannot prove that the grader is correct, that the tool boundary is safe or that repeated trials are stable. It cannot inspect a foundation model’s pretraining data either.

Retire the holdout when you start learning from it

Once engineers inspect the held-out failures and tune against them, those cases are useful development evidence. Keep them as regression tests. Stop presenting them as untouched confirmation of the next candidate.

Use a separate validation set for repeated selection, and reserve a final holdout for a declared decision point. If the final test exposes a defect that needs repair, make the repair, but obtain fresh confirmation evidence rather than pretending the earlier boundary survived.

Report the number of source families as well as the number of questions. A thousand questions derived from one policy provide different evidence from questions spanning many independent families. Do not treat every paraphrase as an independent observation when calculating uncertainty.

For the wider evaluation workflow, my book AI Agent Evals covers task contracts, harnesses and release evidence. Get AI Agent Evals on Leanpub.

Before trusting the next RAG score, name what was held out: a row, a wording, a document family or a future period. The score only supports the experiment you actually ran.