What happens when a self-hosted model hits its limits?

There is no plan to run out of, but there are hard limits: memory, context window, concurrent requests and queue depth. When you reach them, requests slow, queue or fail. Know the settings, watch for the signs and decide ahead of time whether to add hardware, shrink the model or route overflow elsewhere.

Does self-hosting remove limits?

It removes a vendor's quota and replaces it with your own hardware. We are not quoting prices, because they depend on your hardware and we did not fetch any provider page for this post. The runtime limits below come from the Ollama FAQ, fetched on 18 October 2026. Other runtimes differ, so check yours.

What limits will you hit first?

Context window. Ollama's documentation says the default context window is 4096 tokens, set higher with the OLLAMA_CONTEXT_LENGTH variable. Long source files or logs get cut off if you leave the default. Raising it uses more memory.

Memory. If a model does not fit in GPU memory, part of it runs from system memory. The ollama ps command shows a Processor column that reads 100 percent GPU, 100 percent CPU or a split such as 48 percent CPU and 52 percent GPU. A split usually means slower answers.

Parallel requests. OLLAMA_NUM_PARALLEL defaults to 1 per model, and memory use scales with that value multiplied by the context length. The FAQ gives an example: a 2K context with 4 parallel requests needs about 8K.

Queue depth. OLLAMA_MAX_QUEUE defaults to 512 requests. Past that, the server returns an overloaded error (503).

Model churn. OLLAMA_MAX_LOADED_MODELS defaults to 3 times the GPU count, or 3 for CPU. If a new model does not fit, requests wait until idle models unload. Models stay in memory for 5 minutes by default, controlled by keep_alive.

What is the security limit?

By default the server binds to 127.0.0.1 on port 11434, so only the local machine can reach it. Setting OLLAMA_HOST to 0.0.0.0:11434 accepts network connections. That is the moment a personal tool becomes a service. Put authentication and a network boundary in front of it before you do.

What to do

1. Write down your target: number of users, typical prompt length, acceptable wait time.
2. Set the context length on purpose, and test with your longest realistic input.
3. Run ollama ps under load and confirm the model sits fully on the GPU.
4. Set parallelism from measured memory, not hope.
5. Alert on 503 responses and on queue growth.
6. Keep the server on loopback, or behind an authenticated proxy on a restricted network.
7. Decide the overflow path now: a smaller model, a second node, or a hosted model for non-sensitive work.
8. Record the model name, version and settings, so results are repeatable.

How does this relate to Safeguard?

Safeguard supports Full AI, Zero AI or bring your own model, including local and on-prem models. Zero AI means scanners run without an LLM, so if your local model is saturated, scanning does not stop.

What does a sizing example look like?

Take a team of ten engineers who each send a request every few minutes during the day. Peaks matter more than averages, so imagine four requests landing at once.

With the default parallelism of 1, three of them wait. Raise OLLAMA_NUM_PARALLEL to 4 and all four run at once, but memory use rises with it. The FAQ says parallel requests effectively enlarge the context: a 2K context with 4 parallel requests needs about 8K. Now add a request that pastes a long file and you raise the context to cover it, and the memory need grows again.

So write the arithmetic down: context length times parallel requests, plus the model itself. Compare it with your GPU memory, and leave headroom. If it does not fit, you have three choices: a smaller model, a shorter context, or a second machine. The queue limit of 512 is a backstop, not a plan.

Quick check

Sources

• Ollama FAQ

Last reviewed: 18 October 2026.