Local LLM deployment is justified when data cannot leave your boundary, and it is feasible with open-weights models on modest hardware if you accept a capability gap. Size the hardware from model size and concurrency, quantise to fit, and treat the deployment as a production service with access control, monitoring and evaluation.
When local deployment is the right answer
The primary justification is data residency or confidentiality: the content cannot lawfully or contractually be sent to an external provider. In that case local deployment is not a performance decision, it is a constraint, and the question becomes how to make it good enough.
Secondary justifications include predictable latency without network dependency, cost predictability at very high sustained volume, and freedom from provider changes. Each is valid but weaker than a hard confidentiality requirement.
Local deployment is rarely cheaper for occasional use. The hardware, engineering time and operational burden usually exceed API costs at low volume, so a purely financial case is usually wrong.
Hardware sizing and serving
Memory requirement is dominated by model weights plus key-value cache for in-flight sequences. Weights scale with parameter count and precision; the key-value cache scales with context length multiplied by concurrent requests, which is why concurrency rather than model size alone determines hardware.
Quantisation reduces weight memory substantially with a modest accuracy cost, and modern formats make it the default choice for self-hosted inference. Validate the quantised model on your own evaluation set rather than assuming benchmark parity.
For serving, use an established inference server rather than a bespoke harness. Continuous batching, paged attention and speculative decoding are significant engineering achievements that determine throughput, and reimplementing them is not a good use of effort.
| Factor | Effect | Guidance |
|---|---|---|
| Model parameter count | Weight memory and quality | Choose from your own evaluations |
| Precision and quantisation | Memory and throughput | Quantise, then validate accuracy |
| Context length | Key-value cache size | Long context is expensive at concurrency |
| Concurrency | Total memory and latency | The real sizing driver |
| Batch size | Throughput and latency | Larger batches raise throughput, add latency |
| Hardware type | Throughput per cost | Fit the model to available accelerators |
Operating inference as a service
An inference endpoint is a production service. It needs authentication, per-team rate limits, request logging with retention limits, health monitoring with latency and error-rate objectives, and a versioned model registry so a change is traceable and reversible.
Model versions should be deployed with the same progressive approach as application code: a new model goes to internal traffic, then a small cohort, with an evaluation gate before full rollout. Silent quality regressions after a model swap are common and are only caught by evaluation.
Capacity planning must account for the fact that a single long request can occupy substantial memory for its duration. Concurrency limits based on measured memory per sequence prevent the failure mode where one large request causes out-of-memory errors for others.
- Authenticate and rate-limit the inference endpoint per consumer.
- Version models in a registry with rollback capability.
- Roll out model changes progressively behind an evaluation gate.
- Set concurrency limits from measured memory per sequence.
Accepting and measuring the capability gap
Open-weights models are excellent at summarisation, drafting, extraction and constrained question answering. They are weaker at long multi-step reasoning and at tasks requiring broad world knowledge, and they hallucinate more confidently when unsure.
Design around that profile rather than fighting it. Constrain outputs with schemas, ground answers in retrieved content, require the model to indicate insufficient information, and keep a human in the loop for anything consequential.
Measure on your own tasks. A generic benchmark score says little about whether a model can extract the right fields from your invoices, so build a test set from real examples and re-run it for every model or configuration change.
- Prefer extraction, summarisation and drafting over open-ended reasoning.
- Constrain output with schemas and require citations.
- Allow the model to answer that information is insufficient.
- Evaluate on real task data, not public benchmarks.
Frequently asked questions
Is running a local LLM cheaper than using an API?
Rarely at low volume once hardware, engineering time and operations are counted. Local deployment is usually justified by confidentiality or residency requirements rather than by cost, though it can become cost-effective at very high sustained usage.
How much GPU memory does a local LLM need?
Model weights plus key-value cache for concurrent sequences. Weights scale with parameter count and precision, while the cache scales with context length multiplied by concurrency, which is why concurrency is the practical sizing driver.
Can open-weights models match hosted frontier models?
For summarisation, drafting and extraction they are often close enough. For long multi-step reasoning and broad world knowledge a gap remains, so design workflows around the strengths and keep humans in the loop for consequential decisions.
References
Engineer your next strategic platform
Estimate your project investment and delivery timeline instantly, or schedule an architectural discovery call with our engineering leadership.