Continuous batching and PagedAttention explained for team servers

A GPU is used well only when it works on many requests at once. Continuous batching adds and removes requests from the running batch as they arrive and finish, and PagedAttention manages the KV cache in pages so less memory is wasted. Together they let a shared server handle more users.

The serving problem

A single chat on a local model leaves a GPU mostly idle between small matrix operations. Generation is sequential for each user, so adding users one after another is slow. The efficient move is to batch: process the next token for many users in the same pass.

The vLLM paper states the memory side of this directly. High throughput serving needs large batches, but the KV cache for each request is huge and changes size as the request runs. Poor management wastes memory through fragmentation and duplication.

Continuous batching

A naive batch waits for a group of requests, runs them together and returns when the slowest finishes. Short requests then sit idle. Continuous, also called dynamic, batching lets a new request join and a finished one leave between steps.

The vLLM docs list continuous batching among its features. The llama.cpp server README documents continuous batching as an option that is enabled by default, described as dynamic batching, and a parallel slots setting with an automatic default.

PagedAttention

The paper proposes an attention algorithm inspired by virtual memory and paging in operating systems. Instead of reserving one big contiguous block per request for the maximum possible length, the cache is split into blocks allocated as needed. The authors report near zero waste of KV cache memory and flexible sharing of the cache between requests.

In their evaluations, they report roughly 2 to 4 times higher throughput than the systems they compared against, at similar latency, with larger gains for longer sequences, larger models and more complex decoding. Treat that as the authors' result on their setup, not a promise for your hardware.

What this means for a small team

• Memory still bounds you. Each active request needs cache. Ollama's FAQ says memory scales with parallel requests times context length. See KV cache sizing.
• Defaults differ. Ollama's default parallelism is 1. Raise it deliberately and watch memory.
• Latency per user rises as the batch grows. Throughput and individual speed trade against each other.
• Prefix sharing helps when prompts share a long system prompt. The vLLM docs mention prefix caching among features.

How to test your own setup

1. Write a script that sends N concurrent requests with realistic prompt lengths.
2. Record time to first token and total tokens per second at N of 1, 2, 4 and 8.
3. Watch GPU memory as N grows.
4. Choose the highest N where latency is still acceptable.

For the choice of runtime see Ollama, llama.cpp or vLLM. A shared server also needs the exposure controls in binding and exposure.

Quick check

Sources

• Efficient Memory Management for Large Language Model Serving with PagedAttention (arXiv)
• vLLM documentation
• llama.cpp server README
• Ollama FAQ

Last reviewed: 19 October 2026.