Your RAG filter blocks leaks and still hides the right answer

A retrieval service can exclude every unauthorised document and still fail the person asking the question. If it selects the nearest three documents first, then removes the ones the caller cannot read, it may return nothing. In the example below, the useful, authorised documents are seventh and eighth.

I would reject a RAG release gate that tests only whether restricted text escaped. It also needs to test whether an authorised caller can retrieve the evidence required to answer. Otherwise, an access-control test can pass while the assistant repeatedly claims it cannot find information that the user is entitled to see.

This is a retrieval-order problem. It does not require a prompt injection, a stale permission cache or a model behaving badly. You can reproduce it without calling a model.

A permission-safe result can have zero recall

Consider a synthetic corpus ordered by a query’s similarity score. Six restricted documents occupy the highest ranks. Below them are two authorised documents containing the answer, followed by one authorised but irrelevant document. Request three results.

The restricted documents are relevant to the question, but they are not available to this caller. They must not enter the answer context. The two authorised answer documents are the evidence against which we assess retrieval success.

Every row excludes restricted documents from the returned set. Only the last two recover the authorised evidence. Increasing the initial budget from three to six achieves nothing in this example because all six candidates are still restricted.

The scores and relevance labels are invented teaching inputs, not measurements of a search service. The example uses exact sorting so the effect of filter order is visible without approximate-nearest-neighbour behaviour, sharding or reranking.

Run the counterexample

Download the standard-library Python fixture, inspect it, then run:

python3 rag-filter-order.py
python3 rag-filter-order.py --test
python3 -O rag-filter-order.py --test

The first command prints the result IDs and authorised relevant recall for each row above. The tests check candidate starvation, recovery with a larger budget, exclusion of denied documents, partial recall, deterministic tie handling and invalid inputs. They also check the empty-corpus case and reject duplicate document IDs.

The two procedures differ at one boundary. These are the core operations, with ranked already sorted by descending score:

# Filter within the candidate population, then take k.
allowed_first = [doc for doc in ranked if doc.allowed][:k]

# Take a bounded candidate population, then filter and take k.
candidates_first = [
    doc for doc in ranked[:candidate_limit] if doc.allowed
][:k]

The fixture’s allowed flag is a supplied test label. It performs no authentication and does not implement a policy service. It also assumes the whole corpus fits in memory. Treat it as a regression example for the retrieval contract, not a production search implementation.

For evaluation, the denominator is the number of documents that are both relevant to the query and authorised for that principal. In this fixture it is two. If that set is empty, the script returns null for recall rather than manufacturing a perfect score. Evaluate those cases separately: the correct behaviour may be an explicit insufficient-evidence response.

Check what your engine means by postfiltering

Microsoft’s vector-filter documentation distinguishes three Azure AI Search modes. Their placement in the search pipeline differs:

• preFilter applies the predicate during HNSW traversal on each shard, before selecting the filtered results to aggregate.
• postFilter selects local candidates on each shard, filters them, then aggregates the surviving results.
• strictPostFilter, documented as preview, filters after selecting the unfiltered global top-k results.

The exact-sorting fixture resembles the global candidate-cutoff problem in the third mode. It does not simulate Azure’s sharded postFilter algorithm. Calling every filtering operation after some retrieval step “postfiltering” obscures where candidates are lost.

Microsoft documents false negatives for selective filters and small candidate counts in both postfilter modes. For strict postfiltering, the result can be empty even when matching documents exist. The documentation recommends prefiltering when recall matters, while noting that selective predicates can increase traversal work, CPU use and latency.

Those costs need measurement on your index. A filter accepting a small fraction of the corpus can change search behaviour substantially. Do not use this local list example to predict service latency or infer that approximate search has perfect semantic recall. Returning three allowed documents establishes a count; relevance labels are still needed to establish whether they answer the question.

Record the actual engine, API version and filter mode with the experiment. In Azure, index age can also affect the default mode. Set supported behaviour deliberately and verify the request reaching the service rather than relying on the application’s option name.

Keep permission enforcement ahead of disclosure

Improving recall does not justify sending restricted candidates to a model and asking it to ignore them. An external reranker, prompt trace or debug log may disclose those candidates before the final answer filter runs.

Decide which components are authorised to process the full corpus. Apply the permission boundary before content leaves that trusted scope. Where a trusted search service handles a shared index, its internal traversal is different from returning restricted document bodies to an application or third-party model.

For a custom security-filter design, Microsoft’s security-trimming pattern explicitly distinguishes identity strings in a filter from authentication and authorisation. A supplied group identifier is only a string. Your trusted application must establish the caller’s identity, derive the permitted scope and apply the filter on every applicable query. A browser-supplied group list or model-authored tenant filter cannot establish authority.

When too few results survive, retry only within that same authorised scope. You can increase the candidate budget, use an engine-supported filtered search, or perform a bounded exhaustive search over the allowed subset. Each option needs latency and cost limits. Never respond to an empty result by silently dropping the permission predicate.

If policy resolution fails, report the retrieval as blocked or unavailable. Do not turn it into “no matching documents”, and do not fall back to unfiltered retrieval. These states need different operational responses.

Build an evaluation set around the caller

A question alone is an incomplete RAG test case when access varies by user. Bind the question to a test principal, a policy snapshot and reviewed evidence labels. The same question can have different legitimate evidence sets for two callers.

Include a case like the fixture, where restricted near-neighbours outrank accessible evidence. Add a caller with broad access, one with a narrow allowed set, and a caller with no authorised relevant evidence. Test a missing policy response separately from a valid deny decision. Use synthetic documents or an approved private evaluation store so the evaluation itself does not copy restricted material into a less protected system.

For each case, retain enough evidence to distinguish failures:

1. The intended authorised relevant set under the test policy.
2. The returned document IDs and the stage at which filtering occurred.
3. The requested candidate budget and final result count.
4. Whether any denied content reached generation, reranking or logging outside its authorised scope.
5. Retrieval latency and the reason for an empty or blocked result.

Keep these records access controlled. Even document identifiers and aggregate counts can reveal information that a caller should not receive. Internal evaluation evidence is not automatically suitable for a user-facing explanation.

Measure authorised relevant recall alongside prohibited disclosure. Also measure answer support: retrieving a relevant document does not prove that the generated answer used it correctly. If the assistant declines to answer, distinguish a justified absence of accessible evidence from retrieval starvation. The latter can otherwise look like cautious behaviour in an abstention and coverage report.

Permissions can change after retrieval too. The separate RAG cache-revocation example covers reuse and answer-release checks. Correct filter placement does not resolve those later boundaries.

Make the empty-result path part of the release test

Before increasing a global top-k setting, run the same authorised test cases through the current filter mode and a proposed alternative. Compare evidence recovery, prohibited disclosure, latency and bounded retry cost. Keep a held-out set so tuning the candidate budget does not become tuning to the examples you already inspected.

For a broader method of building these tests and turning failures into release decisions, see AI Agent Evals on Leanpub. The immediate task is smaller: add a case with accessible evidence below the unfiltered candidate cutoff, then verify that the real retrieval endpoint finds it without widening the caller’s permissions.