A policy used to contain an exception. The owner removes it, the ingestion job processes the revised document, and every chunk write succeeds. A week later, the RAG assistant still retrieves the exception.

That can happen without a model inventing anything. If the job only upserts chunks present in the new version, an old chunk can remain available under its existing ID. I would block a document-replacement release until a test proves that removed passages disappear from the retrieval path.

The example below is a constructed SQLite fixture. It executes real database statements against an in-memory table; it does not call an embedding model, vector database or search service. The failure concerns which rows remain eligible for retrieval. Ranking and answer generation are separate tests.

An upsert only handles the rows you send

SQLite’s UPSERT documentation describes an insert that becomes an update or a no-op when a uniqueness constraint conflicts. For a multi-row insert, SQLite makes that decision separately for each inserted row. It does not infer that rows absent from the request should be deleted.

Suppose an ingestion adapter assigns chunk identities using (tenant, document, ordinal). The original policy has these passages:

0  Current policy introduction
1  Manager approval required
2  Retired exception: verbal approval is enough

The replacement has two:

0  Current policy introduction
1  Written approval required

This statement updates ordinals 0 and 1 when the job submits the replacement:

INSERT INTO chunks VALUES (?, ?, ?, ?)
ON CONFLICT(tenant, document, ordinal)
DO UPDATE SET body=excluded.body;

Nothing in that statement touches ordinal 2. A query scoped to the tenant and document still returns the retired exception. All submitted writes can be correct while the resulting document is wrong.

Content-derived chunk IDs change the shape of the problem, but they do not supply deletion semantics. Edited text may receive a new ID while the previous text remains under its old one. The ingestion contract needs to define the complete active chunk set, whatever ID scheme it uses.

Reproduce the stale tail

Download and inspect rag-stale-chunks.py. It uses Python 3.9 or later and the standard library:

python3 rag-stale-chunks.py
python3 rag-stale-chunks.py --test
python3 -O rag-stale-chunks.py --test

The demonstration first inserts the original document, then upserts its shorter replacement. It prints three rows: the two current passages followed by the retired exception. The repaired path prints only the two current passages.

Nine tests passed locally, both normally and with Python optimisation enabled, using SQLite 3.51.0. They cover the stale-tail reproduction, successful replacement, rollback after an injected failure, tenant and document isolation, explicit empty replacement, incomplete-input rejection, invalid bodies, repeated replacement and refusal to take over an existing transaction. The tests use unittest checks rather than removable Python assert statements.

The fixture deliberately selects all rows for a document. That makes the stale row visible without relying on an embedding score. A vector-service integration test should additionally query for the removed passage through the same search endpoint and filters the application uses. Absence from a local table is insufficient evidence about a separate index.

Replace one complete document in one transaction

For this small relational fixture, the replacement operation owns a dedicated, idle connection. It validates the input, begins a write transaction, deletes the old document’s rows, inserts the replacement and checks the exact resulting rows before committing.

The central statements are:

BEGIN IMMEDIATE;
DELETE FROM chunks WHERE tenant=? AND document=?;
-- Insert every validated replacement chunk here.
-- Compare stored ordinals and bodies with the input.
COMMIT;

The downloadable implementation supplies the inserts and comparison, and rolls back on an exception. The comment-only SQL above is an outline, not a complete replacement script.

SQLite’s transaction documentation explains that BEGIN IMMEDIATE starts a write transaction immediately and may fail with SQLITE_BUSY if another writer is active. The fixture does not add an automatic retry policy. A production caller needs to handle contention without relabelling a failed attempt as success.

The failure test raises an exception after deletion and before insertion. After rollback, the original three rows are still present. This tests an application exception inside a transaction. It is not a power-loss, disk-failure or distributed-consistency test.

Both the deletion and verification bind the tenant and document parameters. Another tenant’s policy and another document belonging to the same tenant remain unchanged. Those parameters must come from an authorised ingestion job; accepting a caller-supplied tenant string is not authentication. The fixture demonstrates data scoping, not access control.

Empty and incomplete must mean different things

An empty chunk list is dangerous when deletion is part of replacement. It could mean the owner deliberately removed all content. It could also mean the fetch timed out, a parser failed or the connector returned only part of a paginated result.

The fixture requires complete=True before it changes anything. A complete empty list removes the document’s chunks. An incomplete empty list raises an error and preserves the original. It also rejects non-string or empty chunk bodies, oversized fixture inputs and ambiguous completeness values such as the string "true".

That flag is an application precondition, not evidence by itself. The caller has to establish completeness from the connector’s actual contract: the source version fetched, successful parsing, pagination exhaustion where relevant, and an explicit distinction between deletion and failed acquisition. The earlier inventory-completeness article covers why reading one page cannot justify a whole-inventory decision.

Keep the prior version when acquisition is uncertain. For content that must be withdrawn immediately, use a separately authorised document-denial or tombstone path rather than pretending a failed fetch produced a valid empty snapshot. Serving the old document while investigating may be unacceptable; availability and withdrawal policy need an explicit decision.

A remote vector index needs its own replacement protocol

The transaction here covers one SQLite database. Copying the same sequence into calls to a remote vector service does not make those calls atomic. A delete may become visible before its replacement, and acknowledged writes may not yet be searchable. The bulk-ingestion receipt article separates request success from item-level success.

One application design is to stage chunks under a new generation, verify its expected membership, then change the active-generation reference. Retrieval must consistently use that reference. Old generations can be retired afterwards, subject to retention rules. This is a design option, not a claim that every vector service supplies an atomic generation switch.

There are further conditions to test before adopting it:

  • An older ingestion job must not activate its generation after a newer source revision. Use a source-version precondition or equivalent compare-and-set rule at activation.
  • Search, neighbour expansion and citation lookup must resolve the same active document generation. Updating only the initial search filter leaves another path to old text.
  • Acknowledged writes need a service-specific search-visibility check. Membership verification through a point-read API may exercise different consistency guarantees.
  • Cached answers may retain the removed passage after index replacement. Invalidate or revalidate them according to the document’s withdrawal policy.

The local fixture has no source-version guard, cache or concurrent-reader test. Its repeated-replacement test establishes the same final contents for the same input; it does not establish safe ordering between competing versions. Keep those limits visible when moving the function into a service.

For the wider engineering approach to agent tools and review evidence, see my Claude Code book, available as a Kindle edition on Amazon UK.

Add a shrink test to the ingestion pipeline before the next release: ingest a document with a distinctive retired passage, replace it with a shorter complete version, then search for that passage through the production retrieval adapter. Record the active source version and the returned chunk IDs. A successful write count alone cannot pass that test.