Private RAG that survives information governance: eight decisions

“Ask questions of our own documents” is the most requested internal AI capability in almost every organisation we work with. The technique behind it is retrieval-augmented generation, or RAG: the system finds relevant passages in your documents and has a model answer from them. It is also the capability most likely to be quietly dropped nine months later.

It is rarely dropped because the technology did not work. It is dropped because the pilot was built without the constraints the production system would have to meet. By the time information governance, the DPO or the security team looked at it properly, the fixes were structural.

These are the eight decisions that decide whether it gets approved.

The short version

1. Permissions must be resolved at query time

This is the one that kills the most projects.

Suppose your index contains documents that not every user may see, and your retrieval layer does not enforce that per user and per query. You have built a system that will one day surface an HR investigation to the person it concerns. The model does not know the document was restricted. It only knows it was relevant.

The wrong approaches, in rough order of how often we see them:

• Indexing everything and hoping the prompt discourages disclosure.
• Filtering results after retrieval, in application code, as a bolt-on.
• Building one index per team, which fragments knowledge and drifts out of sync at once.

The right approach is to carry the source system’s access control into the index as metadata, and filter before the model sees a candidate document, using the identity of the user asking. This is harder than the demo. It is also the difference between a prototype and a system.

Design it in from day one. Retrofitting per-user authorisation into a system built on a service account is close to a rewrite.

2. Answers must cite, and the citations must be checkable

An answer without a source is not usable in a regulated context. Nobody can verify it and nobody will take responsibility for it.

Every response should cite the documents it drew on, with links that open the source at the relevant place. That lets users verify, it makes failure visible, and it changes behaviour. People check a cited answer far more often than an uncited one.

It also gives you the audit trail. When someone asks in six months why a decision was made, “here is the answer and the four documents behind it” is a different conversation from “the system said so”.

3. The corpus needs an owner and a lifecycle

Retrieval quality is mostly a content problem. The largest single source of confident wrong answers we see is superseded documents left in the index. The 2019 policy and the 2026 policy are both retrieved, the model has no reliable way to know which governs, and it answers from whichever is more similar to the question.

You need someone accountable for what is in the corpus, a rule for what gets removed, and at minimum effective dates carried as metadata so recency can be weighted and staleness surfaced.

This is unglamorous and it matters more than the model choice.

4. Decide the data boundary before you choose components

Suppose patient data, privileged material or special category data will reach the system in production. That constrains everything: which embedding model you can use, where the vector store lives, which inference endpoint is permitted, and what your logs may contain.

Choosing components first and discovering the constraint later is the expensive path, because embeddings are not portable between models. Changing your embedding model means re-indexing the entire corpus.

One part gets missed constantly: the embedding step sends your document text to a model too. Teams that carefully self-host inference sometimes route the whole corpus through a hosted embedding API without noticing they have just exported everything.

5. Logs are a data protection problem

Prompt and response logs are very useful for improving the system. They are also a growing store of exactly the sensitive material you were being careful about.

Decide what is retained, for how long, who can read it, whether it is in scope for subject access, and whether it is covered by your DPIA and your retention schedule. A logging store nobody classified is a common and avoidable finding.

6. Evaluate before you launch, and keep evaluating

Build a set of 100 to 200 real questions with known-correct answers, drawn from your corpus and reviewed by people who know the subject. Run it before launch and after every material change: new model, new chunking, new content source.

Without this you cannot tell whether a change helped, and you cannot answer “how accurate is it?” when someone whose approval you need asks. “It seems good” is not an acceptable answer.

Include questions the system should refuse: things outside the corpus, things the user is not entitled to see, things where the correct answer is “the documents do not say”. Refusal is a feature, and it is the first thing to degrade when someone tunes for helpfulness.

7. Design the failure mode

The system will be wrong sometimes. Design for it.

Show uncertainty plainly. Make “I could not find this in the documents” a normal outcome. Put a visible route to report a bad answer next to every response. Make clear in the interface which decisions must not be made on the system’s output alone.

If the assistant can take actions as well as answer, treat retrieved text as untrusted input, because a document can carry instructions. Limit what the assistant can do, and contain it. Our guide to NVIDIA OpenShell and Sentry covers the controls the NCSC now asks for.

An assistant that admits uncertainty is trusted more, and it is far easier to defend to an assurance panel than one that never says it does not know.

8. Plan the handover from the first week

If the system only runs because one contractor understands it, you have bought a dependency.

That means:

• infrastructure as code
• a documented re-index procedure
• the evaluation set in the repository alongside the code
• a runbook for the common failure modes
• named internal owners for the corpus, the infrastructure and the evaluation

We treat this as a deliverable. It is the part clients tell us later mattered most.

The pattern behind all eight

None of these are model problems. They are systems, governance and content problems, which is why a pilot built to demonstrate capability tells you almost nothing about whether the production system will be approved.

If you are commissioning this work, the useful question is not “can you build a chatbot over our documents”. Almost anyone can, and the demo will be impressive. The useful question is “how will this enforce per-user permissions, and what will you show information governance?”

That one separates suppliers quickly.

Frequently asked questions

What is RAG?

Retrieval-augmented generation. The system searches your documents for passages relevant to a question, then a language model writes an answer from those passages, with citations. It lets a model answer from your material without being retrained on it.

Can RAG respect document-level permissions?

Yes, but only if you design for it. Carry the source system’s access control into the index as metadata and filter on the asking user’s identity before the model sees any candidate document. Filtering afterwards, in application code, is the common failure.

Where should embeddings be generated?

Wherever you would be happy for the documents themselves to go. The embedding step sends your document text to a model, so a hosted embedding API is a data export. Embeddings are also not portable between models, so changing the model means re-indexing the corpus.

How do you evaluate a RAG system?

Build 100 to 200 real questions with known-correct answers, reviewed by people who know the subject. Include questions it should refuse. Run the set before launch and after every material change.

---

This is the design standard we build to in AI implementation, on the infrastructure decided in private and self-hosted AI. To choose between hosted and private models first, read self-hosted or hosted LLMs.