Tokens per second, TTFT and ITL: reading LLM speed numbers

Three numbers describe how fast a language model feels. Time to first token (TTFT) is the wait before anything appears. Inter-token latency (ITL) is the gap between later tokens. Tokens per second is throughput, for one user or a whole server. A speed claim means little until you know which one it is and how it was measured.

Key numbers at a glance

• 450 ms and 40 ms: the TTFT and time per output token limits for Llama 2 70B in MLPerf Inference's interactive scenario; the conversational limits are 2,000 ms and 200 ms (MLCommons inference rules, as of 19 October 2026).
• 1,500 ms and 15 ms: the interactive limits for the DeepSeek-R1 reasoning benchmark in MLPerf; the server limits are 2,000 ms and 80 ms (MLCommons inference rules, as of 19 October 2026).
• 5.7x: improvement in the best per accelerator DeepSeek-R1 server result in MLPerf Inference v6.1 compared with v5.1 a year earlier (MLCommons, 16 September 2026).
• 30 organizations: submitted to MLPerf Inference v6.1, a record for the benchmark (MLCommons, 16 September 2026).
• Up to 2.5x: higher output tokens per second from Anthropic's fast mode, which the documentation says does not target TTFT (Anthropic documentation, as of 19 October 2026).
• 2x price: fast mode on Claude Opus 5.5 is listed at 8 and 40 dollars per million input and output tokens, against 4 and 20 dollars at standard speed (Anthropic pricing page, as of 19 October 2026).
• 8 times a day: how often Artificial Analysis runs its 1,000 and 10,000 input token tests per endpoint, reporting the median of the past 72 hours (Artificial Analysis methodology, as of 19 October 2026).
• 2 to 4 times: throughput gain reported for vLLM's PagedAttention over FasterTransformer and Orca at the same latency (Kwon and co-authors, September 2023).
• 2 to 3 times: speedup reported for speculative decoding on T5-XXL, with identical outputs (Leviathan and co-authors, November 2022).
• 19 billion tokens per minute: processed by Google's model APIs, up from 16 billion stated three weeks earlier (Google I/O, 19 May 2026; Alphabet Q1 call, 29 April 2026).

Why one speed number is never enough

"Fast" can mean three different things. It can mean how long you wait before the answer starts. It can mean how quickly text streams once it starts. Or it can mean how many users a server handles at once. A system can do well on one and badly on another.

A restaurant analogy helps. The time until your first plate arrives is one measure. How quickly the rest of the courses follow is another. How many tables the kitchen serves per hour is a third. A kitchen that serves a thousand diners an hour while keeping each one waiting scores well on the third and poorly on the first.

Language model serving works the same way, and the terms have largely settled. The definitions below follow NVIDIA's benchmarking documentation for its inference microservices and the MLPerf Inference rules from MLCommons, the industry group that runs the most widely cited public inference benchmark.

The terms you will see most

Time to first token (TTFT). NVIDIA defines it as the time from query submission to the first received token, if the response is not empty. It includes queuing, network time and prefill. Prefill is the step in which the model reads the whole prompt before writing anything. A longer prompt means more prefill, so TTFT usually grows with prompt length.

End to end latency. The time from submitting a query to receiving the full response. NVIDIA gives it as TTFT plus generation time.

Inter-token latency (ITL). NVIDIA defines ITL as the average time between consecutive tokens, computed as end to end latency minus TTFT, divided by output tokens minus one. MLPerf uses the name time per output token (TPOT) for a similar measure, defined as the average interval between all the tokens generated. ITL and TPOT describe the decode phase, where the model writes one token after another.

Tokens per second (TPS). This has two meanings, and mixing them up causes most confusion. Total TPS for a system is output throughput across all simultaneous requests: total output tokens divided by the time from the first request to the last response. TPS per user is one request's output length divided by its end to end latency. For long outputs, per user TPS approaches one divided by ITL.

Requests per second (RPS). NVIDIA defines it as the average number of requests the system completes successfully in one second. It counts whole requests, not tokens.

NVIDIA's documentation warns that tools differ on whether TTFT is included in the ITL average, and that values should be compared only when the definitions match. That one sentence explains many disagreements between published speed claims.

What the public benchmarks treat as "fast enough"

The most concrete public reference for acceptable latency is MLPerf Inference. Its rules set hard limits that a submission must meet in each scenario, so the limits show what the benchmark's designers consider interactive. As listed in the MLCommons inference rules on 19 October 2026:

• Llama 3.1 8B (summarization): TTFT 2,000 ms and TPOT 100 ms in the conversational scenario; 500 ms and 30 ms in the interactive scenario.
• Llama 2 70B (question answering): 2,000 ms and 200 ms conversational; 450 ms and 40 ms interactive.
• Llama 3.1 405B (text generation): 6,000 ms and 175 ms server; 4,500 ms and 80 ms interactive.
• DeepSeek-R1 (reasoning): 2,000 ms and 80 ms server; 1,500 ms and 15 ms interactive.
• GPT-OSS-120B (text generation): 3,000 ms and 80 ms server; 2,000 ms and 20 ms interactive.

Two things stand out. First, interactive limits are far tighter than server limits, and the gap is widest on TPOT. A TPOT of 15 ms works out to about 67 tokens per second for each user; 80 ms is about 12.5 tokens per second. Second, the reasoning benchmark has the tightest decode limit of all. The rules do not give a reason, but one plausible reading is that a reasoning model writes many tokens before its answer, so each one has to come quickly for the whole response to arrive in reasonable time.

MLPerf rules also apply tail latency. The general server scenario uses the 99th percentile, so most requests must meet the bound, not just the average one.

The pace of hardware and software improvement shows in the results. MLCommons reported on 16 September 2026 that the best per accelerator DeepSeek-R1 result in the server scenario in v6.1 was 5.7 times better than in v5.1 a year earlier. The round drew a record 30 submitting organizations, added speculative decoding in the interactive scenario for two benchmarks, and included preview results for NVIDIA's Vera Rubin systems. These are vendor submissions under fixed rules, so they show what is possible on tuned systems, not what a given cloud API delivers.

How public leaderboards measure API speed

For hosted APIs, the most cited independent numbers come from Artificial Analysis, which publishes its method. Its "output speed" is the average number of tokens received per second after the first token arrives, which is a per user decode measure. Its TTFT is the time from sending the request to the first token of the response; for reasoning models it counts the first reasoning token, and it reports a separate time to first answer token measured after any thinking. Its end to end response time includes input processing, reasoning and the answer.

The method matters for comparison. Artificial Analysis tests 1,000 and 10,000 input token workloads 8 times a day, runs a 100,000 token workload once a week, and reports the median over the past 72 hours (14 days for the 100,000 token case). Its website default is the 10,000 token prompt. A speed figure from another source with a 100 token prompt, a single test or a best case run is not comparable with these medians.

Why speed and capacity trade off

NVIDIA's documentation states the core relationship plainly: as the number of concurrent requests rises, total system TPS increases while TPS per user decreases. More requests in flight let the GPU do more useful work per step, but each request waits longer for its turn.

That is the trade every operator makes. Packing more requests onto a GPU lowers cost per token and slows each user. Serving fewer requests per GPU gives each user a faster stream at a higher cost per token. A voice assistant needs quick first tokens and steady streaming. An overnight batch job only cares about total throughput.

Providers now price that trade openly. Anthropic's documentation describes fast mode as running the same model with a faster inference configuration, giving up to 2.5 times higher output tokens per second on Claude Opus 5.5 and two earlier Opus models. It says the benefit is focused on output tokens per second, not TTFT. Its pricing page lists fast mode on Opus 5.5 at 8 dollars per million input tokens and 40 dollars per million output tokens, double the standard 4 and 20 dollars. The same page lists a 50 percent discount for the Batch API, which trades latency for price in the other direction. See the batch API and when half price is worth the wait.

What the serving techniques change

Each technique acts on a different metric.

Batching and memory management. The KV cache is the memory a model keeps for each request's attention state. The vLLM paper by Kwon and co-authors (September 2023) says this memory is large, grows and shrinks dynamically, and is wasted through fragmentation when managed poorly, which limits batch size. Its PagedAttention method borrows virtual memory paging from operating systems to get near zero waste, and the authors report 2 to 4 times higher throughput than FasterTransformer and Orca at the same latency. That raises system TPS. See continuous batching and PagedAttention.

Speculative decoding. A small, fast model drafts several tokens and the large model checks them in parallel. Leviathan and co-authors (November 2022) reported a 2 to 3 times speedup on T5-XXL with identical outputs and no retraining. That mainly improves ITL and per user TPS. MLPerf added it to the interactive scenario in v6.1. See speculative decoding explained.

Prompt caching. If the start of a prompt repeats, a provider can skip part of the prefill. That mostly helps TTFT and input cost for long, repeated prompts. See how prompt caching works.

Quantization. Storing model weights at lower precision shrinks memory and can speed decoding, with possible quality trade offs. See what quantization is.

All of these figures come from the authors' own experiments on specific models and hardware. Results vary with model, hardware, prompt length and traffic.

Thinking changes the picture

With reasoning models, the first visible token can arrive after a long hidden thinking phase. Anthropic's documentation on extended thinking says higher thinking budgets enable more thorough reasoning, with diminishing returns that depend on the task, at the cost of increased latency. It also recommends batch processing for thinking budgets above 32,000 tokens, because such long requests can hit system timeouts.

That is why Artificial Analysis reports both TTFT (first reasoning token) and time to first answer token. A model can have a fast decode rate and still feel slow, because the person sees nothing until thinking ends. When you read a speed claim for a reasoning model, check which clock it uses.

Streaming, agents and tails

Most chat products stream tokens as they are made. A person starts reading after TTFT, then text arrives at about the ITL rate. Once text arrives faster than someone reads, extra decode speed is barely noticed, while a long TTFT still shows a blank screen. Interactive products therefore care most about TTFT and a steady ITL.

Agents behave differently. An agent that calls a model many times in sequence pays each call's TTFT in turn, and it usually parses the full response rather than streaming it to a person. End to end latency and the number of sequential calls dominate. Shorter prompts, prompt caching and running independent calls in parallel all reduce the total.

Under load, a request may wait in a queue before prefill begins, and NVIDIA's TTFT definition includes that wait. A rising TTFT at high concurrency usually points to queuing. A rising ITL at high concurrency points to each decode step sharing the GPU with more requests. Reading the two together shows where the bottleneck sits. Averages hide the slowest requests, which is why MLPerf uses the 99th percentile.

Five ways speed claims mislead

1. Different definitions. Whether ITL includes TTFT, and whether TTFT counts thinking, changes the number.
2. Missing concurrency. One request at a time says nothing about behavior under load.
3. Missing prompt and output lengths. TTFT depends on prompt length; per user TPS depends on output length.
4. System versus user speed. A large system TPS can hide a poor experience per user.
5. Averages hide tails. Ask for percentiles, and for how many runs and over what period.

How to measure your own workload

Test with traffic that looks like yours: your usual prompt sizes and expected answer lengths. Record TTFT, ITL, end to end latency and per user TPS at several concurrency levels. Plot per user TPS against system TPS and pick the point that meets your product's target. Repeat after any change of model, hardware, driver or serving software. Keep raw logs, and do not log prompt contents that hold sensitive data; see logging LLM interactions without leaking sensitive data. For how serving speed connects to chip choices, see AI chip market share in 2026, and for the capacity behind it, OpenAI compute commitments explained.

What is firm and what is uncertain

Firm:

• The definitions of TTFT, ITL or TPOT, and system versus per user TPS are published by NVIDIA and MLCommons, and the concurrency trade off is documented.
• MLPerf's latency limits for each benchmark and scenario are public, and its v6.1 results were published on 16 September 2026.
• Providers now sell speed and latency as priced options: fast mode at double the standard rate, batch at half, per Anthropic's pricing page as of 19 October 2026.

Uncertain:

• How a given hosted API performs for your traffic. Public medians and vendor benchmarks use fixed prompts and may not match your load.
• How much of MLPerf's year on year gain reaches cloud APIs, and when. The results show tuned systems, not service levels.
• How reasoning time will be reported. Tools differ on whether TTFT ends at the first reasoning token or the first answer token.

What to watch next

• The next MLPerf Inference round, and whether new agentic and multi turn benchmarks add latency limits that reflect agent workloads.
• Whether more providers publish priced speed tiers like fast mode, and how they define the gain.
• Changes to independent leaderboards' methods for reasoning models.
• Whether API token volumes, such as Google's 19 billion tokens per minute in May 2026, keep rising faster than capacity, which would push TTFT up through queuing.

If you run the serving stack yourself, Safeguard's SBOM and dependency scanning covers the inference server and its packages, with a free scan at https://gold.safeguard.sh/scan.

Quick check

Sources

• NVIDIA, NIM benchmarking: LLM metrics
• MLCommons, MLPerf Inference rules (accessed 19 October 2026)
• MLCommons, MLPerf Inference v6.1 results (16 September 2026)
• MLCommons, MLPerf Inference: Datacenter benchmark page
• Artificial Analysis, Performance benchmarking methodology
• Anthropic, Fast mode documentation
• Anthropic, Pricing documentation (accessed 19 October 2026)
• Anthropic, Extended thinking documentation
• Kwon et al., Efficient Memory Management for Large Language Model Serving with PagedAttention, arXiv 2309.06180
• Leviathan et al., Fast Inference from Transformers via Speculative Decoding, arXiv 2211.17192
• Google, I/O 2026: Welcome to the agentic Gemini era (19 May 2026)
• Alphabet, Q1 2026 earnings call (29 April 2026)

Last reviewed: 19 October 2026.