Architecture
Self-hosted or hosted LLMs? A decision framework for regulated teams
A practical comparison of self-hosted open-weight models and hosted frontier APIs: real costs, capability gaps, and the test that decides between them.
On this page
This decision gets made badly in both directions.
Some organisations reach for self-hosting on reflex, on the assumption that sovereignty demands it, and end up maintaining GPU infrastructure to summarise press releases. Others rule it out on the assumption that open-weight models are hopelessly behind, using a mental benchmark that was accurate two years ago.
Here is how to decide properly.
The short version
- Classify first. List each use case with the highest data classification it will touch in production. That table often decides most of the architecture.
- The capability gap matters less than you expect. For retrieval, extraction, classification, summarisation and drafting, test on 100 to 200 of your own examples before you choose.
- Hosted wins on low, unpredictable volume. Self-hosted wins on high, steady volume, provided someone owns operations.
- Most organisations end up hybrid, routing by classification and defaulting to the private path.
- Agents change the decision. An agent that takes actions needs a containment boundary as well as a data boundary.
Start with classification, not capability
The instinct is to begin by comparing model quality. That is the wrong end.
Begin with a table. List every use case you are seriously considering, and for each one write down the highest data classification that will touch the system in production. Not in the pilot, not in the demo, but in the version people use.
That table usually decides most of the architecture on its own:
| Highest classification in production | Realistic options |
|---|---|
| Public / already published | Anything, including hosted frontier models |
| Internal, non-personal | Hosted under enterprise terms, or private UK deployment |
| Personal data, moderate sensitivity | Private UK deployment; hosted only with a completed DPIA and enforced controls |
| Special category, privileged, or safety-critical | Self-hosted, or private deployment with strong isolation and a documented legal position |
Only once you know which row you are in does the capability comparison mean anything, because you are now comparing options that are available to you.
The capability gap is smaller than you think
Frontier hosted models remain ahead of open-weight models on the hardest reasoning tasks. That gap is real, and it is not closing as fast as enthusiasts claim.
It is also mostly irrelevant to the work regulated organisations want done. The bulk of real demand is:
- retrieval and question answering over an internal corpus
- extraction of structured fields from unstructured documents
- classification and routing
- summarisation against a defined template
- drafting in a house style, for human review
On these, well-configured mid-size open-weight models perform close enough to frontier models that the difference is usually small. It is smaller than the variance introduced by your retrieval quality, your prompt design and your document preparation. Teams routinely attribute to model capability what is a chunking problem.
The test that matters: build an evaluation set of 100 to 200 real examples from your own corpus, with known-correct answers, before you choose anything. Run both options against it. In our experience this reverses the team’s prior assumption about half the time, in both directions, and it costs a few days.
Anyone who tells you which model is best for your use case without having seen your data is guessing.
The real cost comparison
The cost conversation is distorted in both directions, because the two options are usually compared on different bases.
Hosted costs scale with usage and start near zero. Cheap at low volume. The failure mode is a successful rollout. Per-token costs that were trivial in a pilot with 20 users become material at 2,000, and your only lever is restricting usage, which undermines the thing you built.
Self-hosted costs are mostly fixed and mostly up front: accelerators or dedicated cloud GPU capacity, plus the engineering time to run it. Expensive at low volume, and increasingly favourable as usage grows, because marginal cost per request approaches the cost of electricity.
The crossover depends on your workload, but the shape is consistent. Hosted wins on low and unpredictable volume. Self-hosted wins on high and steady volume. Model your projected volume at full adoption, and compare there.
Two costs consistently get left off the self-hosted side:
- Operational burden. Someone has to own capacity, updates, monitoring and incident response. If that person does not exist and you are not hiring them, self-hosting costs more than your model says.
- Model refresh. Open-weight models improve and you will want to move. Budget for periodic re-evaluation and migration.
And one cost consistently gets left off the hosted side:
- Architectural lock-in. Building tightly around one provider’s API, tooling and quirks makes leaving expensive. Keep an internal interface you control with the provider behind it, and this cost stays manageable.
The hybrid pattern most organisations land on
In practice the mature answer is rarely one or the other. It is a routing layer that sends requests to different backends based on data classification:
- A classification step determines what kind of data is in the request.
- Sensitive material routes to the private or self-hosted model.
- Everything else can use the hosted frontier model.
- The routing decision is logged, so you can prove the boundary held.
This gets you frontier capability where it is safe and sovereignty where it is required, and it produces the audit evidence that a boundary existed and was enforced by a control.
The weak point is the classifier. If it is wrong, sensitive data goes to the wrong place without anyone noticing. Build it conservatively: default to the private path, require positive evidence to route externally, and sample the decisions for review. A routing layer that fails open is worse than no routing layer, because it produces confident evidence of a boundary that was not held.
What decides it
After the classification table, four questions.
Do you have, or will you hire, someone who can own inference infrastructure? If not, self-hosting is a commitment you cannot keep. This is the most common cause of abandoned private deployments.
Is your volume predictable and material? High steady volume favours self-hosting. Spiky or uncertain volume favours hosted, at least until the pattern is clear.
Does your regulator, insurer or largest customer ask where the data goes? If that question is already in your tenders or your assurance reviews, the answer has commercial value beyond compliance. That value belongs in the business case.
Will it act, or only answer? An assistant that answers is a data problem. An agent that can call tools and take actions is also a containment problem, and self-hosting the model does not contain it. The NCSC’s August guidance on agentic AI asks for sandboxing, monitoring and an emergency stop. Our guide to NVIDIA OpenShell and Sentry shows what that looks like on a DGX or HGX estate.
In short
Do not choose an architecture because it sounds appropriately serious about security. Choose it because the classification table demands it, the evaluation set supports it, and the cost model works at the volume you expect.
Then keep the boundary clean enough that you can change your mind in a year. You probably will.
Frequently asked questions
Is self-hosting an LLM cheaper than using an API?
Not at low volume. Hosted costs start near zero and scale with usage, while self-hosted costs are mostly fixed and up front. Model your projected volume at full adoption and compare there: self-hosting tends to win on high, steady volume.
Are open-weight models good enough for regulated work?
For retrieval, extraction, classification, summarisation and drafting, often yes. The gap to frontier models is real on the hardest reasoning tasks. Test both on 100 to 200 real examples from your own corpus before you decide.
What is a hybrid LLM architecture?
A routing layer that sends each request to a private or a hosted model based on data classification. It should default to the private path, require positive evidence to route externally, and log every decision.
Do we need to buy GPUs to keep data private?
Not always. A private deployment on dedicated capacity in a UK cloud region gives in-country processing and contractual control without owning accelerators. Our guide to sovereign AI sets out the tiers.
We do this analysis as part of an AI readiness assessment, and build the result in private and self-hosted deployment. For what the pieces cost, see what AI consultancy costs in the UK.