Run a model locally when the data cannot leave your control, the task is narrow enough for a smaller model, or you need offline use. Use a hosted model when you need the highest capability, have bursty demand, or lack people to operate inference. Many teams do both and route by task.
Six questions
- How sensitive is the input? If the prompt contains source code, customer data or credentials that policy forbids sending to a third party, local is the cleaner answer. Ollama's FAQ states that when run locally, the company does not see your prompts or data. That statement covers local use, and cloud hosted models are a separate service with their own terms.
- How hard is the task? Narrow, repetitive tasks such as classification, extraction or summarizing short text often fit a smaller model. Open ended reasoning over large inputs is where quality gaps show first. See when a small model is enough.
- How much volume? Steady high volume can favor owned hardware. Occasional use rarely repays buying a GPU. Work out the real costs for both before deciding. I have no universal break even figure, because it depends on your hardware and prices.
- What latency and availability do you need? A local model has no network round trip and works on a plane. Hosted services scale on demand and handle spikes you would have to over provision for.
- Who runs it? Inference servers need updates, monitoring, access control and capacity planning. If nobody owns that, a managed service may be safer than a neglected local one.
- What do contracts and regulators say? Some customers or rules limit where data may be processed. Check the actual text for your case.
What local does not solve
Running locally keeps prompts off third party services. It does not remove other risks:
- Downloaded model files are a supply chain input. See model file formats.
- A model server can be exposed by mistake. See binding and exposure.
- You own patching of the runtime.
- Output still needs review, particularly code.
What hosted does not solve
A hosted API gives you the provider's controls and their terms, which you need to read: retention, training use, regional processing and subprocessors. You also take on an availability dependency and per token spending that grows with use.
A hybrid pattern
A common pattern is to send sensitive or high volume simple tasks to a local model and hard or low volume tasks to a hosted one, with one internal interface in front so applications do not care which answered. Keep a log of what was routed where, and review that routing rule when policies change.
Decision shortcuts
- Sensitive data and simple task: local.
- Sensitive data and hard task: local with the largest model your hardware allows, or an arrangement your security team has approved.
- Public data and hard task: hosted.
- Occasional use and no operator: hosted.
- Offline requirement: local.
For deployment risks in more depth, see local LLM deployment risks. Safeguard's free scan at https://gold.safeguard.sh/scan can check the dependencies of whatever you build around either choice.
Quick check
Is local always cheaper?
No. Hardware, power, and the time to operate it count. Work it out for your volume.
Is local always more secure?
It keeps data in your hands, which helps. It also makes you responsible for patching, access control and file provenance.
Sources
Last reviewed: 19 October 2026.
