LLM throughput metrics: tokens per second, prefill, decode, batching

Tokens per second can mean the speed one user sees or the total a whole server produces, and the two move in opposite directions as load rises. Reading a prompt (prefill) is compute heavy, while writing the answer one token at a time (decode) is limited by memory speed. Batching many users together is how GPU cost per token falls.

This article defines the metrics buyers meet in quotes and benchmarks, explains why text generation behaves the way it does, shows a worked example with real published numbers, and turns throughput into a cost per million tokens. Every figure has its source and date, and derived arithmetic is labelled as such.

Key numbers at a glance

• Continuous batching, Anyscale blog post of 22 June 2023: up to 23x throughput over naive static batching, measured with vLLM on Meta's OPT-13B on a single 40 GB NVIDIA A100, with 1,000 requests of 512 input tokens each.
• Other results in that post, over the same naive baseline: about 8x for continuous batching on Ray Serve and text-generation-inference, and about 4x for FasterTransformer.
• vLLM paper (arXiv 2309.06180): 2x to 4x higher throughput at the same latency than FasterTransformer and Orca, from near-zero KV cache memory waste.
• DistServe paper (arXiv 2401.09670): separating prefill and decode onto different GPUs can serve 7.4x more requests or meet a 12.6x tighter latency target than state-of-the-art systems, while keeping more than 90% of requests within latency limits. These are the authors' reported results in their test settings.
• Worked decode example from the inference arithmetic post at fergusfinn.com: about 12.75 milliseconds per decode step for 64 users on an H100 setup with FP8 weights, which the post turns into 5,018 tokens per second across the two GPUs used (2,509 per GPU).
• Same post: compute-bound threshold of 295 tokens in that configuration, so a decode batch of 64 is well below it.
• dstack benchmark (22 July 2025): Llama 3.3 70B on one AMD MI300X, with both a VM and bare metal saturating at concurrency 256, which the author attributes to KV cache pressure on GPU memory.
• NVIDIA benchmarking concepts post: tokens per second per system is total output tokens across all simultaneous requests; tokens per second per user is output length divided by end-to-end latency.
• Rental price used for cost examples: $3.99 per H100 GPU-hour, Lambda 8-GPU list price per GPU, checked 10 October 2026 by IntuitionLabs and shown on Lambda's pricing page on 19 October 2026.

What a token is

A language model does not read letters or words. It reads tokens, which are chunks of text, often a short word or part of a word. The exact split depends on the model's tokenizer, so a "token" is not the same size across models. Speeds quoted in tokens per second are comparable only for the same model family or with that difference in mind.

Two phases make up a request. First the model reads your prompt. Then it writes the answer, one token at a time, each new token depending on all the earlier ones.

The metrics, one by one

NVIDIA's developer post on LLM benchmarking sets out definitions that most tools follow. Here they are in plain words.

Time to first token (TTFT). The time from sending the request to getting the first token back. NVIDIA defines it as the time it takes to process the prompt and generate the first token. It includes waiting in a queue, the prompt processing itself and network delay, and it grows with longer prompts. If TTFT feels slow, either the prompt is long, the system is busy, or both.

Inter-token latency (ITL), also called time per output token (TPOT). The average time between consecutive tokens once output begins. The NVIDIA post says it reflects only the decoding phase and warns that different tools calculate it differently, so compare results only when definitions match.

End-to-end latency. The time from submitting the request to receiving the complete answer. It equals TTFT plus the time spent generating.

Tokens per second per user. Output length divided by end-to-end latency. As answers get long, it approaches one divided by ITL. This is the speed a person feels when reading a streamed answer.

Tokens per second per system, the total throughput. Total output tokens divided by the measurement time, counting every request running at the same time. This is the number that sets your cost, because it tells you how much work the GPUs did.

Requests per second. Completed requests divided by time. Useful, but it depends on request length, so it is hard to compare.

NVIDIA's post also stresses that tools measure differently. One of the two tools it discusses includes extra overheads in the time, which at a single concurrent user can reach 33% of the benchmark duration. Always ask which tool and which definition produced a number.

The central trade-off: per user speed against total speed

NVIDIA's post says total tokens per second rises as you add concurrent users until the system saturates, while tokens per second per user falls as latency rises. When the number of simultaneous requests exceeds what the system can batch, requests queue and TTFT goes up. It recommends testing concurrency from one up to slightly above the maximum batch size, and measuring concurrency instead of request rate, since waiting requests can pile up without limit if arrivals exceed capacity.

In practice, this means every quoted throughput figure has a hidden second number. "5,000 tokens per second" might be one fast user or a hundred slow ones. Ask for both, and ask at what load.

Prefill and decode: two very different jobs

Prefill is the phase where the model processes the whole prompt in one go. Many tokens are handled in parallel, so the GPU's arithmetic units stay busy. For most realistic prompt lengths, it is compute heavy. A longer prompt means more work and a larger TTFT.

Decode is the phase where tokens are produced one by one. Each step produces one token per active request, and each step must read the model's weights and the stored memory of the conversation from GPU memory. The Anyscale post says LLM inference is memory-IO bound, not compute bound: moving data to the compute cores takes longer than the computing, so throughput is largely set by how large a batch fits in high-bandwidth GPU memory. The fergusfinn.com post makes the same point for decode specifically and puts a number on it, as the worked example below shows.

That memory of the conversation so far is the KV cache, short for key-value cache. It stores intermediate results for every earlier token so the model need not recompute them. It grows with each token and with each user, and it takes real GPU memory. The vLLM paper says KV cache memory for each request is huge, grows and shrinks dynamically, and when poorly managed is wasted through fragmentation and duplication, which limits batch size.

Worked example: why batching changes everything

The inference arithmetic post at fergusfinn.com works through a decode example on H100 GPUs with 8 bit (FP8) weights. These are the post's figures:

• Model weights: 6.93e10 bytes, which is about 69 GB.
• KV cache per token: 1.64e5 bytes. For an average sequence length of 1,536 tokens, that is about 252 MB per user.
• Users in the batch: 64, so the KV cache read each step is about 16.1 GB.
• Total data read per decode step: about 8.55e10 bytes, which is about 85.5 GB.
• Memory bandwidth for the setup: 6.70e12 bytes per second. The post uses two GPUs working together.
• Time per step: 12.75 ms, which is bytes divided by bandwidth.

Sixty-four users each get one token per step, so the system makes 5,018 tokens per second in total, or 2,509 per GPU, according to the post.

Now the important part. The weights are read once per step no matter how many users share the step. Only the KV cache grows with each user. The post notes that weight transfers are constant with batch size while KV cache transfers are not. So with 64 users, each user's share of weight traffic is a small fraction of what a single user would carry alone.

The same figures give the single user case. With a batch of one, the data per step is about 6.93e10 plus 2.52e8 bytes, divided by 6.70e12, which is about 10.4 ms. One token per 10.4 ms is about 96 tokens per second in total. So the same two GPUs produce about 96 tokens per second for one user, or about 5,000 for 64 users. Step time barely changed, from about 10.4 ms to 12.75 ms, while throughput rose by about 52 times. That is the entire economic case for batching. The single user figure is calculated from the post's published inputs, not a measured result.

The price you pay is latency for each user. A step takes longer as the KV cache grows, so each person waits a bit longer per token, though in this example only from about 10.4 to 12.75 ms.

From tokens per second to cost per million tokens

Take two H100 GPUs at $3.99 each per hour, which is $7.98 per hour. Using the ceiling figures above, which assume the GPUs do nothing else and ignore prompt processing, scheduling and idle time:

• At 96 tokens per second, the setup makes about 346,000 tokens per hour, so cost is about $23 per million output tokens.
• At 5,018 tokens per second, it makes about 18.1 million tokens per hour, so cost is about $0.44 per million output tokens.

These are illustrations from one published worked example and one list price. Real systems lose some of this to prompt processing, queueing, uneven traffic and idle time, which is why the companion explainer on GPU utilization matters. But they show the shape: fill the batch and cost per token collapses. An underused GPU running a few users is expensive per token no matter how cheap the hourly rate looks.

How the industry raises throughput

Continuous batching. Older static batching waits for every request in a batch to finish before starting new ones, so early finishers leave the GPU underused until the longest one is done. Continuous batching, also called iteration level scheduling, slots a new request in as soon as another finishes. The Anyscale post of 22 June 2023 reports up to 23x throughput over naive static batching in its test (OPT-13B on an A100 40 GB, 1,000 requests, 512 input tokens, generation lengths with a mean of 128 tokens), and says vLLM more than doubles performance compared with naive continuous batching in the same benchmark. The baseline matters: 23x is against the naive approach, not against other modern servers.

Better memory management. The vLLM paper proposes PagedAttention, an attention algorithm inspired by virtual memory and paging in operating systems, and reports near-zero KV cache waste and 2x to 4x throughput at the same latency compared with FasterTransformer and Orca. More KV cache memory saved means bigger batches.

Separating prefill and decode. Because the two phases stress hardware differently and have different latency goals (TTFT for prefill, TPOT for decode), running them together causes interference. The DistServe paper assigns them to different GPUs and reports 7.4x more requests served or 12.6x tighter latency targets than state-of-the-art systems in its tests, with more than 90% of requests within limits. These are the authors' results for their workloads and settings.

Hitting the memory ceiling. The dstack benchmark found that a 70 billion parameter model on one MI300X saturated at concurrency 256 on both bare metal and a VM, which the author attributes to KV cache pressure on GPU memory. Batching has a limit set by memory.

How to read a vendor's throughput claim

Ask these questions:

• Which model, at what precision (8 bit or 16 bit), on how many and which GPUs?
• What were the prompt and output lengths? Long prompts shift work to prefill, and long outputs to decode.
• At what concurrency? Total tokens per second at 256 users and per user speed at one user are very different numbers.
• Is the figure total throughput, per user speed or per GPU?
• What were TTFT and ITL at that load?
• Which benchmarking tool and which definition were used?
• Is the number a measured result or a theoretical ceiling?

If a quote does not answer these, you cannot turn it into a cost.

What is firm and what is uncertain

Firm: the metric definitions in NVIDIA's post, the results as reported in the Anyscale post, the vLLM and DistServe abstracts, and the worked example as published on fergusfinn.com.

Uncertain: how any of this carries to a newer model, chip or serving stack, since most figures here come from 2023 to 2025 and specific setups; whether DistServe's reported gains hold in your workload; and the real cost per token of any provider, which depends on utilization that providers do not publish. The cost per million tokens figures are calculated from a single worked example and a list price, and they exclude every real world loss. The fergusfinn.com post is an individual's analysis, not a vendor benchmark, and it does not name the model.

What to watch next

• Independent benchmarks that report TTFT, ITL and total throughput together at several load levels.
• Wider adoption of prefill and decode separation in production servers.
• Memory improvements, such as more GPU memory per chip, that allow larger batches.
• Inference MFU numbers at scale, which the companion utilization explainer notes have not been published in the sources it cites.

For related reading, see GPU utilization explained, GPU rental prices in 2026, Anthropic compute commitments explained and Anthropic revenue and margin, reported versus estimated.

If you run model servers in containers, Safeguard offers SBOM and dependency scanning and a free scan at gold.safeguard.sh/scan.

Quick check

Sources

• LLM benchmarking: fundamental concepts, NVIDIA Technical Blog
• How continuous batching enables 23x throughput in LLM inference, Anyscale, 22 June 2023
• Efficient memory management for large language model serving with PagedAttention, arXiv 2309.06180
• DistServe: disaggregating prefill and decoding for goodput-optimized LLM serving, arXiv 2401.09670
• How fast can an LLM go? Inference arithmetic, fergusfinn.com author
• Benchmarking AMD GPUs: bare-metal and VMs, dstack, 22 July 2025
• Lambda pricing, read 19 October 2026
• H100 rental prices compared, IntuitionLabs, first-party prices checked 10 October 2026

Research results are as reported by their authors in their own test settings. Cost per million tokens figures are calculated from the cited worked example and list price.

Last reviewed: 19 October 2026.