The other jurisdiction's clause reached the answer
RolesCustomer AI DeploymentLLM Product EngineeringAI Evaluation and Reliability
The question
A regulated-domain assistant serves analysts in two jurisdictions from one index. In a red-team probe, a question asked under one jurisdiction returned a contract clause that belongs to the other. The reranker and the generator behave as designed. Where is the leak, how do you close it, and how do you prove that it stays closed?
Answer it first, aloud or on paper. Then open the answer.
The answer
Show short answer
The leak is in retrieval: the jurisdiction tag was not applied as a pre-filter before the similarity search, so a semantically close chunk from the other jurisdiction entered the candidate pool, and nothing downstream is built to remove it. Close it by tagging every chunk with role and jurisdiction at ingestion and filtering on both before hybrid retrieval runs. Prove it with a cross-jurisdiction probe set in the red-team suite, scored pass or fail with a severity, and run before every release.
Show full answer
Why downstream stages cannot save you
The reranker orders the candidates it receives by relevance, and the leaked clause is relevant. The generator cites what it is given. The output guard scrubs personal data and blocks forbidden topics; it does not know which jurisdiction a clause belongs to. Access control has to happen before the text reaches the prompt, and the only stage that sees the whole corpus is retrieval.
The capstone order
Filter by role and jurisdiction first. Then run dense and BM25 retrieval in parallel, fuse by rank, send the top 20 to the reranker, the top 5 to the generator, then the guards, then cite and return. The tags come from ingestion, where each chunk receives a summary, a role label and a jurisdiction tag.
Closing it
- Tag at ingestion, and make ingestion refuse a chunk that has no tag.
- Filter before retrieval, not on the top k after it; a post-filter can leave an empty result and still lets an untagged chunk through.
- Treat the caller's jurisdiction as a request parameter, never as a hint in the prompt.
Proving it
The capstone runs a pre-release red team of 50 adversarial prompts that include cross-jurisdiction leaks, scored pass or fail with a severity. Its exercise adds a second corpus slice under another jurisdiction and runs a 20-question cross-jurisdiction probe. Put that probe in the release gate for every change to ingestion or retrieval, and carry the leak count on the drift dashboard as a hard zero.
Follow-ups
L3SeniorWhy can the reranker not catch the leaked clause?
Show expected answer
it only orders the candidates it receives, and a clause from the wrong jurisdiction is still relevant to the question.
L3SeniorWhy is the output guard not the fix?
Show expected answer
it scrubs personal data and blocks forbidden topics; it cannot tell which jurisdiction a clause belongs to.
L4StaffThe tags exist and the leak persists. What do you check?
Show expected answer
whether the filter runs before retrieval or on the top k after it, and whether ingestion lets an untagged chunk in.
L4StaffHow do you keep the proof current?
Show expected answer
run the probe set in the red team on every change to ingestion or retrieval, and show the leak count on the drift dashboard with a zero threshold.
Wrong answers
Show the wrong answers that interviewers hear
“Add a sentence to the system prompt about the user's jurisdiction.”
The model cannot verify which jurisdiction a chunk belongs to; access control belongs in retrieval.
“Separate indexes solve it, no tags needed.”
Role filtering still needs labels, shared documents get duplicated, and the probe is still needed to prove the split holds.
“The red team passed once, so it is closed.”
One pass proves one build; the probe runs on every ingestion or retrieval change.
Taught in
The capstone production RAG applies its access filter ahead of retrieval and probes it in its red team, and the lesson advanced RAG explains metadata pre-filtering.