Fastest LLM Inference: Lowest Latency and Most Tokens per Second

The fastest LLM inference comes from three moves made together: spread the model's weights across as much memory bandwidth as you can buy, keep the batch per replica small, and generate several tokens per weight pass with speculative decoding. Decode speed is set by how fast the weights can be read, so cost per token rises as latency falls.

That is the short answer. The long answer is a chain of physics, hardware and software choices, each with a measurable trade-off, and this post walks through all of it. We start with what "fast" means (time to first token, time per output token, tokens per second per user), then show why decode is limited by memory bandwidth and prefill by compute, then work out in Python how fast a 1B, 10B, 100B and 1 trillion parameter model can go in principle on different hardware. After that come the speed-first chips (wafer-scale SRAM, deterministic dataflow LPUs, reconfigurable dataflow units and GPU racks with 72 GPUs on one NVLink domain), the measured numbers from MLPerf Inference v6.1, the software techniques (speculative decoding, FlashAttention and Flash-Decoding, CUDA graphs, tensor parallelism, disaggregated prefill and decode, KV cache placement), and finally a "speed at any cost" configuration guide.

This post is part of our AI chips and cost series. Its sibling posts cover the money side: GPU rental prices in 2026, cost per million tokens when self-hosting and whether AI is getting cheaper. Here cost is secondary, but we still show it, because every speed decision moves it.

A note on method. Every chip specification comes from the vendor's own page or paper, every measured result comes from MLCommons MLPerf Inference or a peer-reviewed or preprint paper, and every vendor speed claim is labelled "vendor claim" with its date. Every computed number in this post came out of a Python script that we ran on a Mac; the script is reproduced below. The serving commands and the client script follow vendor and project documentation but were not executed, because the machine this was written on has no data center GPU.

Key takeaways

• Two latencies, two bottlenecks. Time to first token (TTFT) is dominated by prefill, which is compute-bound. Time per output token (TPOT) is dominated by decode, which at small batch is memory-bandwidth-bound: every new token requires reading every active weight once.
• The bandwidth bound is simple. Per-user decode speed at batch 1 is at most aggregate memory bandwidth divided by bytes read per token (active weights plus KV cache). A 100B dense model in FP8 on eight H100 SXM GPUs tops out near 264 tokens per second per user even in theory.
• Batching buys throughput with latency. Larger batches share each weight read across more users, so cost per token falls fast, but every user's tokens arrive more slowly as KV cache reads grow.
• Speed-first hardware moves weights closer to compute. Wafer-scale and LPU designs keep weights in on-chip SRAM with petabytes per second of bandwidth (vendor figures). On those machines the bound shifts from memory to compute pipelines and interconnect hops.
• GPU racks win by aggregation. A GB200 NVL72 rack puts 72 GPUs and 576 TB/s of HBM bandwidth on one NVLink domain (NVIDIA figures). Tensor parallelism across it cuts per-token time until communication latency dominates.
• Measured results exist, but they measure throughput under a latency limit. MLPerf Inference v6.1 (published 16 September 2026) has interactive scenarios with strict TTFT and TPOT limits. In our examples, systems deliver 37 to 82 percent of their offline throughput when held to them.
• Speculative decoding is the biggest software lever. With a draft acceptance rate of 0.8 and four drafted tokens, the Leviathan et al. formula gives about 2.8 times faster decode, at extra compute per token.
• Speed costs money. On public on-demand prices, serving a 100B model to one user at a time can cost over 20 times more per token than serving 32 users per replica.

What "fastest" means: the metrics

"Fast" has at least four meanings in LLM serving, and they pull against each other. Pick the one your users feel before you buy anything.

MLPerf's rules define the two latency metrics the same way: TTFT "measures the latency of the first token" and TPOT "measures the average interval between all the tokens generated" (MLPerf Inference rules). Those rules also show how much the latency targets matter. For Llama 2 70B the interactive limits are 450 ms TTFT and 40 ms TPOT, against 2,000 ms and 200 ms in the looser "conversational" category. For a typical Llama 2 70B answer of about 294 tokens (the reference length in the rules), the interactive limits allow an end-to-end time of 0.45 + 293 × 0.04 = 12.17 seconds. For DeepSeek-R1 the interactive limits are 1,500 ms and 15 ms, which means at least about 67 tokens per second per user; a 1,000 token reasoning answer at those limits takes 16.49 seconds.

The rest of this post optimises for low TTFT and low TPOT at the 99th percentile, with throughput and cost as constraints rather than goals.

Latency anatomy: prefill versus decode

Every autoregressive request has two phases.

 request arrives
   |
   v
 +--------------------------- PREFILL ----------------------------+
 | all prompt tokens processed in parallel                        |
 | big matrix-matrix multiplies, high arithmetic intensity        |
 | writes K and V for every prompt token into the KV cache        |
 | ends by emitting token 1                    => TTFT ends here   |
 +----------------------------------------------------------------+
   |
   v
 +---------------------------- DECODE ----------------------------+
 | step 1: read ALL active weights + KV cache -> token 2           |
 | step 2: read ALL active weights + KV cache -> token 3           |
 | ...                                                            |
 | step n: read ALL active weights + KV cache -> token n+1         |
 | each step = one TPOT; steps are strictly sequential            |
 +----------------------------------------------------------------+

The DistServe paper names the two latencies after these two phases, noting that applications "emphasize individual latency for each phase: time to first token (TTFT) for the prefill phase and time per output token (TPOT) of each request for the decoding phase" (Zhong et al., DistServe, 2024). The Splitwise paper from the same period found the prompt phase "compute-intensive" and token generation "memory-intensive" (Patel et al., Splitwise, 2023).

Why decode is bandwidth-bound

A transformer layer's matrix multiplies cost about 2 FLOPs per weight per token (one multiply, one add). In FP8 each weight is 1 byte. At batch size 1, each decode step therefore performs about 2 FLOPs for every byte of weights it reads. That ratio is called arithmetic intensity.

A chip can only keep its math units busy if the work's arithmetic intensity exceeds the chip's ratio of peak FLOPs to memory bandwidth (the roofline "ridge point"). For an H100 SXM, NVIDIA lists 3,958 TFLOPS of FP8 with sparsity, so about 1,979 TFLOPS dense, and 3.35 TB/s of memory bandwidth (NVIDIA H100). The ridge point is 1,979 ÷ 3.35, about 591 FLOPs per byte. For a GB200 NVL72 GPU, NVIDIA's rack figures (720 PFLOPS sparse FP8 over 72 GPUs, dense half of sparse, and 576 TB/s over 72 GPUs) give 5,000 TFLOPS dense FP8 and 8 TB/s per GPU, a ridge point of 625 (NVIDIA GB200 NVL72).

So batch 1 decode at 2 FLOPs per byte uses well under 1 percent of the math units. The weights stream through, the tensor cores wait. The Medusa authors describe the same bottleneck: each decoding step "necessitates moving the full model parameters from High-Bandwidth Memory (HBM) to the accelerator's cache" (Cai et al., Medusa, 2024).

Batching is the standard fix for throughput: one weight read serves every request in the batch. It is the enemy of latency, because a bigger batch means more KV cache to read and more compute per step, and because the scheduler holds requests to fill it. The fastest possible decode therefore runs at small batch, with the hardware mostly idle in FLOP terms. That is the root reason fast inference is expensive.

Why prefill is compute-bound

Prefill processes every prompt token at once, so each weight byte is reused across hundreds or thousands of tokens. A 8,192 token prompt has an arithmetic intensity on weights in the thousands of FLOPs per byte, far above the ridge point. TTFT for long prompts is therefore set by peak FLOPs, model FLOP utilisation (MFU), and how many chips share the work. We work through the numbers below.

How fast can a model go in principle? Worked numbers

This is the core calculation. For each model size we compute memory for weights, KV cache, and an upper bound on decode speed from memory bandwidth. All figures come from the script below.

The script

# fastest_llm_calc.py: the bandwidth bound on decode speed, in plain Python
GB, TB, PB = 1e9, 1e12, 1e15

# Vendor peak bandwidth and memory (see Sources). Peak, not sustained.
hw = {
    "H100 SXM":           (3.35 * TB, 80 * GB),
    "H200":               (4.8 * TB, 141 * GB),
    "B200 (GB200)":       (8 * TB, 186 * GB),
    "8x H100 SXM":        (8 * 3.35 * TB, 8 * 80 * GB),
    "8x H200":            (8 * 4.8 * TB, 8 * 141 * GB),
    "GB200 NVL72":        (576 * TB, 13.4 * TB),
}

# Illustrative shapes: params, active params, layers, KV heads, head dim
models = {
    "1B dense":            (1e9,  1e9,  16, 8, 64),
    "10B dense":           (10e9, 10e9, 40, 8, 128),
    "100B dense":          (100e9, 100e9, 80, 8, 128),
    "1T dense":            (1e12, 1e12, 128, 16, 128),
    "1T MoE (32B active)": (1e12, 32e9, 61, 8, 128),
}

def kv_bytes_per_token(layers, kv_heads, head_dim, bytes_per_val=1):
    return 2 * layers * kv_heads * head_dim * bytes_per_val   # K and V

CTX = 8192          # tokens of context per user
W_BYTES = 1         # FP8 weights
for name, (P, A, L, kvh, hd) in models.items():
    read = A * W_BYTES + kv_bytes_per_token(L, kvh, hd) * CTX
    for hname, (bw, mem) in hw.items():
        fits = P * W_BYTES + kv_bytes_per_token(L, kvh, hd) * CTX <= mem
        print(name, hname, round(bw / read), "" if fits else "(does not fit)")

The model shapes are illustrative, chosen to look like common dense transformers with grouped-query attention. The 1T MoE shape borrows its total and active parameter counts and layer count from Moonshot AI's published Kimi K2 card (1T total, 32B activated, 61 layers, 384 experts with 8 selected per token, block FP8 checkpoints) (Kimi K2). Kimi K2 itself uses multi-head latent attention, which stores a smaller KV cache than our generic formula, so our MoE KV numbers are conservative.

Weight memory

The MoE row is the trap people fall into: a trillion-parameter MoE model reads only its active experts per token, but all experts still have to be resident somewhere fast, because the next token may route to any of them.

KV cache

KV cache per token is 2 (keys and values) × layers × KV heads × head dimension × bytes per value. With an FP8 KV cache:

At batch 1 the KV cache is small next to the weights. At batch 32 and long context, it becomes a large share of every decode step's memory traffic, which is why per-user speed falls as batch grows.

The decode bound per user, batch 1, FP8 weights, 8K context

This is the theoretical ceiling: aggregate peak bandwidth divided by bytes read per token. Real systems reach a fraction of it.

Values are output tokens per second per user. "Does not fit" means weights plus one user's KV cache exceed the configuration's HBM.

Three things stand out.

First, the 1B and 10B rows on a full NVL72 rack are not meaningful. Long before you reach half a million tokens per second for one user, other limits take over: each layer's kernels run in sequence, each tensor-parallel step needs a collective, and kernel launches cost microseconds. We model that floor in the tensor parallelism section.

Second, the 1T MoE row shows why MoE is the speed architecture of this generation. It needs the memory of a 1T model but reads like a 32B model. On an NVL72 its bandwidth bound is about 30 times the 1T dense bound (17,442 versus 574).

Third, at a realistic 50 percent of peak bandwidth, the numbers halve: about 132 tokens per second per user for a 100B dense model on eight H100s, about 2,842 for the same model on an NVL72 (if you could use all 72 GPUs efficiently for one user, which you cannot, as shown below), and about 8,721 for the 1T MoE on an NVL72.

Prefill: the TTFT side

Prefill FLOPs for the linear layers are about 2 × active parameters × prompt tokens (attention adds a term that grows with the square of prompt length, which we leave out here, so these are lower bounds). We assume 40 percent MFU on dense FP8 peak: 1,979 TFLOPS per H100 and 5,000 TFLOPS per GB200 GPU, both derived from NVIDIA's sparse figures as explained above.

Two lessons. Long prompts on a dense 100B model on one 8-GPU node already push TTFT past a second, so prompt length is a first-class latency input. And the full-rack column assumes perfect scaling of one prompt over 72 GPUs, which real systems do not achieve; it is an ideal, useful only to show that more chips per request shorten TTFT.

The batch-size trade-off, with prices

Cost is secondary in this post, but you should see what speed costs. We use Microsoft Azure's public retail price API for East US 2, Linux, on demand, checked 25 October 2026: the ND96isr H100 v5 size (8 H100 GPUs) at $98.32 per hour and the ND96isr H200 v5 size (8 H200 GPUs) at $84.80 per hour (Azure Retail Prices API). That is $12.29 and $10.60 per GPU-hour. These are estimates from stated assumptions: 50 percent of peak bandwidth achieved, FP8 weights and KV cache, 8,192 tokens of context per user, decode only, and the replica fully busy.

For the 100B model on H100s, going from batch 32 to batch 1 raises per-user speed by about 40 percent (94 to 132 tokens per second) and raises cost per token about 23 times ($9.11 to $206.55). That is the shape of the whole problem: the last increments of speed are the most expensive. Note also that the H200 node is both faster and, at these list prices, cheaper per token than the H100 node, because it has 43 percent more bandwidth per GPU at a lower hourly price.

Simplifications to keep in mind: real decode also reads activations and does some compute that does not shrink with batch; we ignore prefill time; and we assume every second of the replica is used. The self-hosting cost post goes deeper on utilisation.

Hardware built for speed

The fastest systems attack the bandwidth bound in two different ways. GPU racks add more HBM stacks and join more GPUs with a fast fabric. SRAM-centric designs keep the weights on the chip itself. The figures below are vendor figures from vendor pages and papers. Treat them as peak, not sustained.

GPU rack scale: GB200 NVL72 and tensor parallelism over NVLink

NVIDIA's GB200 NVL72 joins 72 Blackwell GPUs and 36 Grace CPUs in one rack with a single NVLink domain. NVIDIA lists 130 TB/s of total NVLink bandwidth (1.8 TB/s per GPU), 13.4 TB of HBM3E with 576 TB/s of bandwidth, and 720 PFLOPS of dense NVFP4 (1,440 sparse) for the rack (NVIDIA GB200 NVL72). Per GPU that is 8 TB/s and about 186 GB.

Vendor claim, as shown on NVIDIA's GB200 NVL72 page on 25 October 2026: "30x faster real-time LLM inference performance for trillion-parameter language models" versus H100. The footnote conditions matter: a 50 ms token-to-token latency treated as real time, 5 s first token latency, 32,768 input and 1,024 output tokens, against HGX H100 scaled over InfiniBand, and marked "Projected performance subject to change." It is a projection under one latency target, not a measurement.

The reason a rack matters for speed is tensor parallelism. Megatron-LM's intra-layer parallelism splits each layer's weight matrices across GPUs, so each GPU reads only its shard; the cost is collective communication, two all-reduces per layer in the forward pass in the Megatron design (Shoeybi et al., Megatron-LM, 2019). For decode at small batch, those all-reduces move tiny messages, so their cost is mostly fixed latency, not bandwidth. That gives a simple model:

time per token  =  weight bytes / (TP x per-GPU bandwidth x efficiency)
                 + layers x 2 x all-reduce latency

We do not have a vendor-published small-message all-reduce latency for NVLink at each TP size, so the latency values below (5 µs and 10 µs per all-reduce) are illustrative assumptions, not measurements. The model is a 100B dense FP8 model on B200-class GPUs at 8 TB/s and 70 percent bandwidth efficiency, 80 layers.

Doubling TP from 8 to 16 still helps a lot. Going from 32 to 72 helps little, because the fixed communication time is now larger than the memory time. That is the general rule: tensor parallelism shortens TPOT until per-layer synchronization dominates, and the cheaper your collectives, the further you can push it. This is also why a single NVLink domain of 72 GPUs matters more for latency than the same 72 GPUs connected over a network, where each hop adds more latency per all-reduce.

For MoE models, expert parallelism spreads experts across GPUs, so each token's routed experts are read in parallel on different chips. The 1T MoE row above needs only 1,000 GB at FP8, about 7.5 percent of an NVL72's 13.4 TB, leaving most of the HBM for KV cache.

Wafer-scale with on-chip SRAM: Cerebras WSE-3

Cerebras announced the WSE-3 on 13 March 2024: 4 trillion transistors, 900,000 AI cores, 44 GB of on-chip SRAM, 125 petaflops of peak AI compute, built on TSMC 5 nm (Cerebras WSE-3 press release). Its inference launch post (27 August 2024) gives "21 petabytes/s of aggregate memory bandwidth" (Cerebras inference launch). Cerebras's current chip page describes a WSE-3 Turbo variant with 250 petaflops; it does not say whether that figure is sparse or dense, and does not state SRAM or bandwidth for it (Cerebras chip page).

The argument for SRAM is exactly the bandwidth bound. The launch post makes it with numbers: a 70B model at 16 bits is 140 GB, and 1,000 tokens per second would need about 140 TB/s, versus 3.3 TB/s on an H100. The post claims the WSE-3 has 7,000 times an H100's bandwidth; our arithmetic from the two published figures (21 PB/s and 3.35 TB/s) gives about 6,269 times, so treat 7,000 as rounded.

Vendor claims, dated:

Sources: Cerebras inference launch and Cerebras Llama 3.1 405B post.

What the SRAM capacity means in practice: 44 GB per wafer is small next to the models. A 70B model at 16 bits does not fit on one wafer, and Cerebras says larger models are split across multiple CS-3 systems. Our arithmetic: 405B at 16 bits is 810 GB, or at least 19 wafers if every weight sits in SRAM (Cerebras's 405B post does not state the system count). A 100B FP8 model needs at least 3 wafers, and a 1T FP8 model at least 23.

On such a system, the bandwidth bound almost disappears. Reading 810 GB at 21 PB/s takes about 39 µs, while 969 tokens per second is about 1.03 ms per token. The real floor is now the time to push one token through every layer's compute and across wafer-to-wafer links. In other words, SRAM machines move the bottleneck rather than remove it. The trade-offs: SRAM per wafer limits how many models and how much KV cache fit per system, so very long contexts and very large models need many systems, and you depend on one vendor's compiler and runtime.

Deterministic dataflow LPUs: Groq

Groq's design, described in its ISCA 2022 paper, is a tensor streaming processor whose execution is scheduled entirely by the compiler: there are no caches or dynamic arbitration to make timing vary, and on-chip SRAM across many chips is treated as one logically shared memory reached over a software-scheduled network (Abts et al., ISCA 2022). Determinism matters for latency: if every step's timing is known at compile time, tail latency stays close to average latency, which is what a p99 TPOT target rewards.

On 24 December 2025 Groq announced a "non-exclusive licensing agreement with Nvidia for Groq's inference technology"; founder Jonathan Ross, President Sunny Madra and other team members joined NVIDIA, Groq stayed an independent company under CEO Simon Edwards, and GroqCloud continued to run (Groq newsroom). Groq's architecture page, checked 25 October 2026, now describes an "NVIDIA Groq 3 LPX" rack that "connects 256 next-generation LPU accelerators to NVIDIA's Vera Rubin," with "128 GB of on-chip SRAM per rack," "40 PB/s SRAM bandwidth" and "315 PFLOPS of FP8 inference compute," phrased as figures for "when released" (Groq LPU architecture). Per LPU that works out to 0.5 GB of SRAM and about 156 TB/s. These are vendor figures for a product the page does not describe as shipping.

Capacity again sets the limit: 128 GB per rack holds a 100B FP8 model with room for some KV cache, but a 1T FP8 model would need at least 8 racks of SRAM (about 4 at FP4). Groq's earlier generation also kept weights in SRAM spread over many chips; we could not retrieve its per-chip product brief on 25 October 2026, so we do not quote per-chip figures for it.

Vendor claim, checked 25 October 2026: Groq's model documentation lists openai/gpt-oss-120b at 500 tokens per second, priced at $0.15 per million input tokens and $0.60 per million output tokens, and llama-3.3-70b-versatile at 280 tokens per second (GroqCloud models). Those are listed speeds, not measurements under a stated percentile.

Reconfigurable dataflow: SambaNova SN40L

SambaNova's SN40L paper describes a dataflow chip with a three-tier memory: 520 MiB of on-chip SRAM, 64 GiB of on-package HBM (1.8 TB/s in the paper's spec table) and up to 1.5 TiB of DDR, with 638 BF16 TFLOPS of peak compute per socket, two dies on TSMC 5 nm (Prabhakar et al., SN40L, 2024). The design fuses whole decoder layers into one dataflow kernel so that intermediate results stay on chip, and uses the DDR tier to hold many models at once.

The paper reports decode speeds on 16 SN40L sockets with BF16 weights, 8K sequence length and speculative decoding: 129 output tokens per second per user for Llama 3.1 405B, 457 for 70B, and 1,042 for 8B. That is a self-reported measurement in a paper written by the vendor, not an independent benchmark. It is a useful sanity check of the bandwidth bound: 16 sockets × 1.8 TB/s is 28.8 TB/s, and 810 GB of 405B BF16 weights gives a no-speculation bound of about 36 tokens per second. Reaching 129 means each weight pass produced several tokens, which is what speculative decoding does.

Vendor claim, 10 September 2024: SambaNova Cloud launched with Llama 3.1 405B at 132 tokens per second and Llama 3.1 70B at 461 tokens per second, both at full precision; the release quotes Artificial Analysis as having independently benchmarked the 405B endpoint at 132 output tokens per second (SambaNova press release).

The speed-first designs side by side

No row is "the fastest" without conditions. A wafer-scale system can post the highest per-user speed on a model that fits its SRAM budget; a GPU rack can serve far larger models and contexts at high per-user speed while also delivering high aggregate throughput; an SRAM system with tight determinism can hold a strict p99 more cheaply in some cases. Your model size, context length and traffic shape decide.

What MLPerf actually measured

MLPerf Inference is the only cross-vendor, peer-reviewed benchmark suite for these systems, so it is where measured numbers in this post come from. MLCommons released the v6.1 results on 16 September 2026; the results repository lists every submission with system, software stack and precision (MLPerf Inference v6.1 results).

How the interactive scenario works

The Server and Interactive scenarios send queries according to a Poisson process, and the reported metric is the maximum throughput the system sustains while meeting the benchmark's latency constraints at the 99th percentile (with an early-stopping correction) (MLPerf Inference rules). The LLM limits in the current rules document:

The rules allow speculative decoding only for specific workloads, including DeepSeek-R1 Interactive (EAGLE-style decoding with the model's own MTP head) and GPT-OSS-120B Interactive (EAGLE3-style with a named reference head), with a fixed configuration of three speculative steps, and they ban tricks that "artificially manipulate acceptance rates." That is a useful sign of where the field is: the benchmark owners consider speculative decoding a normal part of fast serving.

One caution: the rules document on GitHub is the current version, and limits can change between rounds. Check the round-specific rules before you compare results across rounds.

Measured results: the price of a latency guarantee

The most informative comparison is the same system across scenarios. Offline has no latency limit, so it shows maximum throughput; Interactive holds every request to the tight limits. All results below are closed division, from the v6.1 summary file.

Source: MLPerf Inference v6.1 summary. "Preview" means the system was not yet available at submission time, under MLPerf's availability rules.

What these numbers do and do not tell you:

• They measure throughput under a speed floor, not maximum per-user speed. A GB200 NVL72 at 240,274 tokens per second on DeepSeek-R1 Interactive is serving thousands of concurrent streams, each at no worse than about 67 tokens per second at p99. With throughput equal to streams divided by average TPOT, that is at most about 3,600 concurrent streams. MLPerf does not report the fastest single stream.
• Tighter latency costs throughput, and more so for bigger models. The 8B model keeps 82 percent of its offline throughput under interactive limits; DeepSeek-R1 on GB300 keeps 37 percent. Bigger models and tighter TPOT limits force smaller effective batches.
• Per accelerator, the interactive numbers are high. GB300 NVL72 at 253,506 tokens per second over 72 GPUs is about 3,521 tokens per second per GPU on DeepSeek-R1 Interactive; 8x MI355X on Llama 2 70B Interactive is about 9,149 per GPU. These are aggregate rates, not per-user rates.
• The SRAM-centric vendors are absent. The v6.1 summary contains no submissions from Cerebras, Groq or SambaNova. Their speed figures in this post are vendor claims, and there is no MLPerf result to compare them on equal terms.

What the latency guarantee costs per token

No public per-GPU list price exists for every system above, so here is the arithmetic at assumed rates of $5, $10 and $15 per accelerator-hour. These are estimates from stated assumptions, not quotes.

The interactive guarantee costs roughly 1.2 to 2.7 times the offline rate in these examples. That is far less than the 23 times in our batch 1 example, because interactive serving still batches heavily; it just caps how far. Pure batch 1 serving is a different, much more expensive regime.

Software that makes inference faster

Hardware sets the bound. Software decides how close you get, and some techniques change the bound itself.

Speculative decoding: more tokens per weight pass

Speculative decoding was introduced by Leviathan, Kalman and Matias, who showed "2X-3X acceleration" on T5-XXL with "identical outputs" (Leviathan et al., 2022). DeepMind's concurrent work reported "a 2-2.5x decoding speedup" on the 70B Chinchilla model (Chen et al., 2023). The idea: a cheap draft proposes γ tokens, the big model scores all of them in one forward pass (which costs about the same as scoring one token, because decode is bandwidth-bound), and a modified rejection sampling step accepts a prefix of the draft so the output distribution matches the big model exactly.

The Leviathan paper gives the maths. If each drafted token is accepted with probability α (assumed independent), the expected number of tokens produced per big-model pass is:

E[tokens per step] = (1 - alpha^(gamma + 1)) / (1 - alpha)

If the draft costs c times the big model per token, the expected wall-time improvement is:

speedup = (1 - alpha^(gamma + 1)) / ((1 - alpha) * (gamma * c + 1))

Worked values from our script:

The best γ depends on α and c: for α = 0.6 and c = 0.05 our script finds γ = 4 is best (1.92x); for α = 0.8 it finds γ = 8 (3.09x). A slow draft (c = 0.2) wipes out much of the gain, which is why the field moved to drafts built into the model.

• Medusa adds extra decoding heads to the model that predict several future tokens, verified together with tree attention. The paper reports over 2.2x speedup for Medusa-1 (heads trained on a frozen model) and 2.3 to 3.6x for Medusa-2 (heads trained jointly) (Cai et al., 2024).
• EAGLE drafts at the feature level (the second-to-top layer) rather than the token level; for LLaMA2-Chat 70B it reports a 2.7x to 3.5x latency speedup while keeping the output distribution (Li et al., 2024).
• EAGLE-3 switches to direct token prediction with multi-layer feature fusion; it reports speedups up to 6.5x, about 1.4x over EAGLE-2, and a 1.38x throughput gain at batch 64 in SGLang (Li et al., 2025).

Two caveats. Speculation spends extra compute to save bandwidth, so it helps most where compute is idle: low batch and memory-bound decode. vLLM's documentation says it targets "medium-to-low QPS (queries per second), memory-bound workloads" (vLLM speculative decoding). And acceptance rates depend on the text: code and structured output tend to accept more than open-ended prose, so measure α on your own traffic. For local setups, see our speculative decoding explainer for local inference.

FlashAttention and Flash-Decoding

FlashAttention makes attention "IO-aware," tiling the computation so it reads and writes HBM less, and computes exact attention (Dao et al., 2022). FlashAttention-2 improved work partitioning (Dao, 2023); FlashAttention-3 uses Hopper's asynchronous tensor cores and TMA, reaching up to 740 TFLOPS in FP16 (75 percent utilisation) and close to 1.2 PFLOPS in FP8 on H100 (Shah et al., 2024). These mostly help prefill and long-context TTFT.

Decode needs something else. With a query length of 1, FlashAttention parallelises only across batch and heads, so at batch 1 most of the GPU sits idle; the Flash-Decoding authors wrote that "With a batch size of 1, FlashAttention will use less than 1% of the GPU!" Flash-Decoding splits the KV sequence into chunks processed in parallel, then combines them with a reduction using the log-sum-exp of each chunk. The authors report up to 8x faster end-to-end decoding for very long sequences on CodeLlama-34B (Dao, Haziza, Massa and Sizov, Stanford CRFM, October 2023). For low-latency, long-context decode, a split-KV attention kernel is essential.

CUDA graphs and kernel fusion

At small batch, a decode step for one layer can take only tens of microseconds of GPU time, so the CPU cost of launching each kernel becomes visible. CUDA graphs fix this by separating definition from execution: a graph is "defined once and then launched repeatedly," and instantiation "performs much of the setup" ahead of time; stream capture records work into a graph instead of executing it (CUDA Programming Guide, CUDA Graphs). The catch: parameters can be updated in place, but "major changes to graph structure (such as topology or node types) require re-instantiation," so servers capture one graph per batch size bucket.

Kernel fusion has the same goal from the other side: fewer kernels, fewer round trips to HBM. Fusing the normalisation, the projections and the activation into fewer kernels cuts both launch overhead and memory traffic. SambaNova's paper takes this to the extreme with whole decoder layers fused into a single dataflow kernel.

Here is what a captured decode step looks like in PyTorch. This follows the PyTorch and CUDA documentation but was not executed for this post.

import torch

# model is already on the GPU, in eval mode, with a static KV cache
static_input = torch.zeros((1, 1), dtype=torch.long, device="cuda")
static_pos = torch.zeros((1,), dtype=torch.long, device="cuda")

s = torch.cuda.Stream()
s.wait_stream(torch.cuda.current_stream())
with torch.cuda.stream(s):
    for _ in range(3):                      # warm up allocators and autotuners
        model.decode_step(static_input, static_pos)
torch.cuda.current_stream().wait_stream(s)

g = torch.cuda.CUDAGraph()
with torch.cuda.graph(g):
    static_logits = model.decode_step(static_input, static_pos)

def fast_step(token_id, pos):
    static_input.fill_(token_id)            # update inputs in place
    static_pos.fill_(pos)
    g.replay()                              # one launch for the whole step
    return static_logits

Tensor parallelism degree versus latency

We modelled this above. The rule of thumb: raise TP until per-token memory time is comparable to per-token communication time, then stop and add replicas instead. Keep TP inside one NVLink or equivalent scale-up domain. Pipeline parallelism adds a sequential hop per stage and does not shorten TPOT for a single stream, so latency-first deployments prefer TP and expert parallelism; vLLM's docs also note pipeline parallelism is not composable with speculative decoding as of vllm 0.15.0 and earlier.

Disaggregated prefill and decode

When prefill and decode share GPUs, a long prompt arriving mid-stream stalls everyone's decode steps, which shows up as TPOT spikes at p99. DistServe assigns the two phases to different GPUs, choosing parallelism for each separately, and reports serving "7.4x more requests or 12.6x tighter SLO" while meeting latency constraints for over 90 percent of requests (DistServe). Splitwise reports 1.4x higher throughput at 20 percent lower cost, or 2.35x more throughput at the same cost and power (Splitwise).

The cost of disaggregation is moving the KV cache from the prefill GPU to the decode GPU. For our 100B example at 8,192 tokens, that is 1.342 GB: about 0.75 ms over 1.8 TB/s NVLink, but about 27 ms over a 400 Gb/s (50 GB/s) network link. So disaggregation adds to TTFT unless the transfer runs over fast fabric or is overlapped with the first decode steps.

KV cache placement

KV cache is the second stream of bytes every decode step reads, so where it lives sets TPOT:

• In HBM, paged. PagedAttention manages KV memory in blocks like virtual memory pages, with "near-zero waste," which lets a server hold more sequences without fragmentation (Kwon et al., 2023). For speed it matters because wasted memory forces smaller batches or recomputation.
• Quantised. An FP8 KV cache halves bytes read per step versus FP16. Check accuracy on your evaluation set.
• Shared prefixes. Prefix caching reuses the KV of a common system prompt across requests, which cuts TTFT for every request that shares it.
• Offloaded to CPU memory or storage. Good for capacity, bad for latency: reading KV over a host link is far slower than HBM. For a latency-first deployment, keep active sequences' KV in HBM or SRAM and offload only idle sessions.

Low-batch serving

The last technique is a policy, not a kernel: cap the batch. Every server has a limit on concurrent sequences per replica (in vLLM, max_num_seqs). Set it low enough that your p99 TPOT target holds at peak load, and add replicas for capacity. Combine that with admission control, so new requests queue at the edge rather than slowing every in-flight stream.

Worked examples: how fast can each model size go?

Pulling the pieces together. All figures are theoretical bounds or simple estimates from the script, at 50 percent of peak bandwidth where stated, before speculative decoding.

1B dense. Weights are 1 GB in FP8. On one H100 the batch 1 bound is about 2,954 tokens per second at full peak bandwidth (about 1,477 at 50 percent). Here bandwidth is rarely the limit; kernel launch overhead and per-layer latency are, so CUDA graphs and fused kernels matter more than more chips. Cost on an 8x H100 node at batch 1 is about $2.31 per million tokens at our assumptions, and a single GPU replica would be about an eighth of that.

10B dense. 10 GB in FP8. One H200 gives about 450 tokens per second per user at peak, about 225 at 50 percent. TP 8 on one node raises the bound to about 3,599 (about 1,799 at 50 percent). With speculative decoding at the 2.8x example above, that becomes several thousand tokens per second per user in principle. Batch 1 on 8x H200 costs about $13.09 per million tokens at our assumptions; batch 32 costs about $1.21.

100B dense. 100 GB in FP8. The bandwidth bound on 8x H100 is about 264 tokens per second per user at peak. On an NVL72 the ideal bound is 5,684, but our TP model says per-layer collectives cap a single replica far lower, around 540 to 950 tokens per second with TP 72 under our assumed latencies. Speculative decoding at 2.8x would lift a 330 tokens per second TP 8 replica to around 900. The SN40L paper's 405B result (129 tokens per second on 16 sockets with speculation) and Cerebras's vendor claim (969 tokens per second on 405B) bracket what purpose-built systems reported for a model four times larger.

1T MoE with 32B active. The model needs 1,000 GB in FP8, so it does not fit on one 8x H100 node (640 GB) but does fit on 8x H200 (1,128 GB) with little room for KV cache. On 8x H200 the batch 1 bound is about 1,163 tokens per second, about 581 at 50 percent. On an NVL72 the bound is about 17,442, and reading 32 GB at 576 TB/s takes about 0.056 ms, so here expert routing, all-to-all communication and per-layer latency set the real limit. TTFT for an 8,192 token prompt is about 83 ms on 8x H100-class compute at 40 percent MFU, against 2,587 ms for a 1T dense model, which is the other reason MoE dominates speed-first deployments.

1T dense, for contrast. 1,000 GB read per token. Even on an NVL72 the ideal bound is 574 tokens per second (about 287 at 50 percent, or about 3.5 ms per token), before communication. No hardware in this post serves a 1T dense model at the per-user speeds MoE models of the same size reach.

A "speed at any cost" configuration guide

If latency is the goal and the budget is not, this is the order to work in.

Step 1: write the target as numbers

Pick p99 TTFT and p99 TPOT at a stated prompt length, output length and peak concurrency. For example: "p99 TTFT under 300 ms for prompts up to 4,000 tokens, p99 TPOT under 10 ms (100 tokens per second per user), at 200 concurrent users." Without numbers you cannot tell whether a change helped.

Step 2: choose the model architecture before the hardware

At a given quality, MoE with few active parameters is faster per token than dense. FP8 weights read half the bytes of FP16; FP4 halves again, if your evaluation shows the quality holds. Check that the model ships with a speculative head (an MTP head or an EAGLE head) or that one exists for it, because that is your largest single software multiplier.

Step 3: pick hardware by bandwidth per replica, then capacity

Divide your active bytes per token by your TPOT target to get the effective bandwidth you need per replica, then double it for realistic efficiency. Example: a 32B active MoE in FP8 with a 10 ms TPOT target needs 32 GB in 10 ms, which is 3.2 TB/s effective, about 6.4 TB/s of peak, before KV cache. One B200-class GPU's bandwidth would cover the weights if the model fit, but a 1T MoE needs about 1 TB of fast memory, so capacity drives you to a multi-GPU node or rack, or to many SRAM chips.

Step 4: shard for latency

Use tensor parallelism inside one scale-up domain, sized with the memory-time versus communication-time model above. Use expert parallelism for MoE. Avoid pipeline parallelism for decode. Separate prefill and decode onto different GPUs if long prompts arrive while others are streaming.

Step 5: turn on every decode-side optimisation

Speculative decoding with a trained head, split-KV attention for long contexts, CUDA graphs for each batch bucket, FP8 KV cache, prefix caching for shared system prompts. In vLLM, a latency-first launch looks like this (flags from the vLLM documentation; not executed for this post, and check your installed version's vllm serve --help):

vllm serve <your-moe-model> \
  --tensor-parallel-size 8 \
  --enable-expert-parallel \
  --kv-cache-dtype fp8 \
  --enable-prefix-caching \
  --max-num-seqs 16 \
  --speculative-config '{"method": "eagle3", "model": "<eagle3-head-for-this-model>", "num_speculative_tokens": 3}'

vLLM captures CUDA graphs by default unless you pass --enforce-eager, so do not pass that flag in production. --max-num-seqs is the batch cap from the low-batch section; start low and raise it until p99 TPOT starts to slip.

Step 6: measure TTFT and TPOT yourself

Do not trust a single average. This client measures both at the percentile you care about against any OpenAI-compatible streaming endpoint. It follows the openai Python library's documented streaming interface and was not executed for this post. Stream chunks may hold more than one token; for exact TPOT, count tokens with the model's tokenizer or use the usage field if your server returns it.

import time, statistics
from openai import OpenAI

client = OpenAI(base_url="http://localhost:8000/v1", api_key="unused")

def one_request(prompt, max_tokens=256):
    t0 = time.perf_counter()
    first, chunks = None, 0
    stream = client.chat.completions.create(
        model="served-model",
        messages=[{"role": "user", "content": prompt}],
        max_tokens=max_tokens, stream=True)
    for event in stream:
        if event.choices and event.choices[0].delta.content:
            chunks += 1
            if first is None:
                first = time.perf_counter()
    end = time.perf_counter()
    ttft = first - t0
    tpot = (end - first) / max(chunks - 1, 1)
    return ttft, tpot

def p99(xs):
    xs = sorted(xs)
    return xs[int(0.99 * (len(xs) - 1))]

results = [one_request("Explain KV caching in two paragraphs.") for _ in range(200)]
ttfts, tpots = zip(*results)
print("p50 TTFT", statistics.median(ttfts), "p99 TTFT", p99(ttfts))
print("p50 TPOT", statistics.median(tpots), "p99 TPOT", p99(tpots))

Run it at your real concurrency (with threads or an async client), with your real prompt length distribution. Latency at one concurrent request says little about latency at 200.

Step 7: protect the tail

Cap batch per replica, use admission control, keep prefill off the decode GPUs, and keep spare replicas so a node failure does not double load on the rest. A system that is fast at p50 and slow at p99 will feel slow.

Decision table

For a broader view of serving engines, see our comparison of Ollama, llama.cpp and vLLM. For memory and interconnect basics, see the GPU memory and interconnects explainer, and for what happens inside each GPU, inside an NVIDIA streaming multiprocessor.

Common mistakes

• Optimising average latency. Users feel p99. MLPerf measures at the 99th percentile for a reason.
• Comparing a vendor's single-stream speed with another vendor's throughput. A tokens-per-second figure means nothing without "per user" or "aggregate," the batch size, prompt length and precision.
• Treating vendor claims as benchmarks. The speeds from Cerebras, Groq, SambaNova and NVIDIA's projections in this post are vendor claims with dates. Only the MLPerf rows are cross-vendor, peer-reviewed measurements.
• Forgetting that MoE needs all experts in fast memory. Active parameters set bandwidth per token; total parameters set capacity.
• Raising tensor parallelism forever. Past the point where collective latency matches memory time, more GPUs per replica barely help.
• Running speculative decoding at high batch and expecting the same gain. Speculation trades compute for bandwidth; when the GPU is already compute-bound, gains shrink.
• Measuring with one request at a time. Latency at concurrency 1 is a best case.
• Ignoring prompt length. TTFT grows with prompt length; a benchmark with 100 token prompts says little about 32,000 token prompts.
• Quantising without an evaluation. FP4 and FP8 speed decode, but check task quality on your data.
• Pulling draft heads and serving engines without tracking them. A speculative head is another model file from another repository, and the serving engine is a fast-moving dependency tree. Pin versions and record them.

FAQ

What is the fastest way to run an LLM?

Put the model's active weights in the highest-bandwidth memory you can, split each layer across chips with tensor parallelism inside one fast interconnect domain, keep batch size per replica small, and use speculative decoding. On GPUs, that means a rack-scale NVLink system and an MoE model with few active parameters; SRAM-centric systems from Cerebras and Groq report higher single-stream speeds on models that fit their memory (vendor claims).

Why is LLM decoding memory-bandwidth-bound?

Each output token requires reading every active weight once, and at batch 1 each weight byte is used for only about 2 FLOPs in FP8. GPUs need hundreds of FLOPs per byte to keep their math units busy (about 591 on an H100), so at small batch the chip waits on memory.

How many tokens per second can an H100 generate?

For one user at batch 1 with FP8 weights, the upper bound is H100 bandwidth (3.35 TB/s) divided by bytes read per token. That is about 314 tokens per second for a 10B dense model and about 2,954 for a 1B model, before overheads; real systems reach a fraction of that. A 100B FP8 model does not fit on one 80 GB H100.

What is the difference between TTFT and TPOT?

TTFT is the delay before the first token, set mostly by prefill compute. TPOT is the average gap between later tokens, set mostly by decode memory bandwidth. MLPerf limits both at the 99th percentile; for Llama 2 70B Interactive the limits are 450 ms and 40 ms.

Does speculative decoding change the output?

Not when implemented as in the original papers. Leviathan et al. and Chen et al. use a rejection sampling scheme that preserves the large model's output distribution; Leviathan et al. report identical outputs on T5-XXL. Some variants trade exactness for speed, so check which mode your server uses.

Is Cerebras or Groq faster than GPUs?

Both report higher single-stream output speeds than GPU clouds on models that fit their SRAM, for example Cerebras's vendor claim of 969 tokens per second on Llama 3.1 405B (18 November 2024). Neither appears in MLPerf Inference v6.1, so there is no cross-vendor measurement under the same rules. GPU racks lead on capacity per system and on measured throughput under latency limits.

What does MLPerf's interactive scenario measure?

Maximum throughput while every request meets tight TTFT and TPOT limits at the 99th percentile, with Poisson arrivals. It does not report the fastest single stream. Interactive results in v6.1 ranged from about 37 to 82 percent of the same system's offline throughput in our examples.

How much does fast inference cost?

It depends on batch. In our estimate for a 100B dense model on 8 H100s at Azure's on-demand list price, batch 1 costs about $206 per million output tokens and batch 32 about $9, while per-user speed changes from about 132 to 94 tokens per second. Interactive MLPerf results imply a much smaller premium, roughly 1.2 to 2.7 times offline cost, because interactive serving still batches.

Does a mixture of experts model run faster than a dense model?

Per token, yes, if all experts fit in fast memory: decode reads only the active experts. A 1T MoE with 32B active parameters has a bandwidth bound about 30 times higher than a 1T dense model on the same hardware, and its prefill FLOPs are about 31 times lower.

Should I disaggregate prefill and decode?

If long prompts arrive while other users are streaming and your p99 TPOT suffers, yes. DistServe and Splitwise both report large gains. Budget for the KV cache transfer: for a 100B model at 8,192 tokens it is about 1.3 GB, under a millisecond over NVLink and tens of milliseconds over a 400 Gb/s network.

Where Safeguard fits, and where it does not

Safeguard does not design, program or speed up chips, and it does not make inference faster or cheaper. Everything above is about hardware and serving software; Safeguard's role is the software supply chain underneath it.

A latency-first stack is a long dependency chain: GPU drivers and firmware, CUDA or ROCm, communication libraries, a serving engine such as vLLM, SGLang or TensorRT-LLM that releases often, kernel libraries, Python packages, container images, and model files including draft and EAGLE heads pulled from public model hubs. Each of those is a versioned component that can carry vulnerabilities. What applies, with each capability's maturity:

• SBOM generation and analysis (GA) in SPDX and CycloneDX, from repositories, container images and uploaded SBOM files, so a serving image's engine, CUDA libraries and Python packages are inventoried.
• Software composition analysis (GA) across language ecosystems and container layers, including transitive PyPI dependencies of serving engines.
• Vulnerability management (GA) with EPSS and CISA KEV enrichment, and the ability to ask which projects contain a given component and version.
• Policies and gates (GA) that can block a build carrying a component below a version you set.
• Autonomous remediation (GA, off by default) that opens tested pull requests to bump pinned dependencies.
• AI-BOM discovery (Early Access), which inventories AI assets and produces an ML-BOM, and AI-SPM (Early Access), which scans model artifact files in formats such as pickle, safetensors, GGUF and ONNX. Ask about scope before you plan around either.

Where Safeguard does not help: it does not tune kernels, pick parallelism, install drivers or manage GPU fleets, and a driver fix still has to come from the GPU vendor and be rolled out by your platform team.

Speed-first serving stacks change constantly: teams upgrade serving engines for each new speculative decoding method, pull new draft heads and rebuild images every week, and each rebuild pulls open source dependencies again. The hardware fixes come from hardware vendors, but most of what ships in a serving image is open source software you build yourself, and that is where the steps below apply.

More in this series

Other posts in our AI chips and costs series:

• AI Accelerator TCO: Buy vs Rent, Capex, Power, Cooling and Utilization
• AI Chips Compared 2026: B200, H200, MI355X, Gaudi 3, TPU, Trainium
• Cheapest LLM Inference: How to Cut Cost per Million Tokens
• How Many GPUs Does a 1 Trillion Parameter Model Need?
• How to Estimate AI Training and Inference Cost: Formulas and a Calculator
• LLM Inference Cost by Chip and Model Size, 1B to 1 Trillion Parameters
• LLM Training Cost by Chip and Model Size: 1B to 1T, Worked

Fix it without waiting for upstream

Whatever the vulnerability, most of the risk sits in two gaps. The first is the time between a fix being published and that fix actually running in your production builds. The second, and worse, is the time before any fix exists at all, while a maintainer works out a patch and a release schedule. Waiting through both gaps is a habit, not a requirement. Safeguard gives teams three ways to close them.

1. Autonomous remediation: pull requests, not tickets

When a fixed version exists, Safeguard's Griffin models open a tested, policy gated pull request that moves the affected dependency to the safe version. You can run it on one repository or in bulk across every project that carries the vulnerable version. Remediation covers eight categories, including dependencies, licenses, security posture and findings. Reachability analysis ranks the vulnerable code paths your application actually calls first, so the first pull requests you merge are the ones that matter most. Autonomous remediation ships switched off, with every category set to Manual, so your team decides how much runs on its own and what still needs a human approval.

2. Gold Open Source: the safe version in seconds, free

gold.safeguard.sh is free and needs no login. Search any CVE or package to see affected and fixed versions, CVSS, EPSS, CISA KEV status and a recommended safe version, or use the "Am I affected?" checker. The same data is available through a free read API, README status badges and RSS threat feeds. The safeguard-sh/gold-scan-action GitHub Action can fail a build on a known exploited vulnerability, on any known vulnerability, or never, so a version you have just removed cannot quietly come back in next week's merge.

3. Gold Registry: start from components that are already fixed

registry.safeguard.sh provides hardened, drop in replacements for upstream packages and container images. Every Gold artifact is held continuously to zero exploitable critical or high CVEs, zero malware and a completed license check, with SLSA L3 or higher provenance, a Sigstore signature and a public evidence page you can hand to an auditor. An artifact loses its Gold badge if it regresses. Teams can point their own package directory at the registry so hardened equivalents resolve without changing how developers install anything.

When upstream has no fix yet, or will not ship one, Safeguard first works to get the fix accepted upstream. If the issue is severe or upstream will not move, Safeguard builds a fork, fixes the vulnerability there and publishes a hardened Gold build, so you are not stuck on a maintainer's release schedule. You can also request a hardened, compatibility tested build of a specific package. This fork and patch flow is in early access today, so ask which packages it covers before you plan around it.

One honest limit: when the vulnerable code lives in a vendor appliance, a hosted service or an operating system runtime you do not build yourself, the fix still has to come from that vendor or distribution. The steps above apply to the open source dependencies and container images you build and ship, which is where most teams carry the largest and least tracked share of their exposure.

See how Safeguard would help

You can download Safeguard and follow the CLI docs to try these yourself.

The first demo sets up the Safeguard runner on your own machine, approves it with its device code and then scans from the terminal. For a latency-first serving platform, that is how you would scan the inference images and the repositories that build them without sending source or images off the machine, for example running safeguard scan --image against a serving image pinned by digest.

Click through it at your own pace, or open it full screen.

The second demo is a real CLI session: install, open the interactive terminal and ask it to fix a lodash CVE. The assistant reads the finding and the advisory, asks before any command that changes something, edits package.json to the fixed version, runs npm install and npm test with approval, prints a summary and pushes nothing. The same flow applies to the Python side of a serving stack, such as moving a pinned serving engine or one of its dependencies to a fixed version and rerunning your latency and accuracy tests before you ship.

Click through it at your own pace, or open it full screen.

The third demo connects Claude to the Safeguard MCP server with OAuth and shows how to choose which tools Claude can use. When an advisory lands for a serving engine or a CUDA library, an engineer can ask which projects contain the affected component and version and get the answer from their own scan data.

Click through it at your own pace, or open it full screen.

Quick check

Sources

• MLCommons: MLPerf Inference rules (inference_rules.adoc, checked 25 October 2026)
• MLCommons: MLPerf Inference v6.1 results repository and summary.csv
• MLCommons: MLPerf Inference benchmark documentation
• Leviathan, Kalman and Matias: Fast Inference from Transformers via Speculative Decoding (arXiv 2211.17192)
• Chen et al.: Accelerating Large Language Model Decoding with Speculative Sampling (arXiv 2302.01318)
• Cai et al.: Medusa (arXiv 2401.10774)
• Li et al.: EAGLE (arXiv 2401.15077)
• Li et al.: EAGLE-3 (arXiv 2503.01840)
• Dao et al.: FlashAttention (arXiv 2205.14135)
• Dao: FlashAttention-2 (arXiv 2307.08691)
• Shah et al.: FlashAttention-3 (arXiv 2407.08608)
• Dao, Haziza, Massa and Sizov: Flash-Decoding for long-context inference (Stanford CRFM)
• Zhong et al.: DistServe (arXiv 2401.09670)
• Patel et al.: Splitwise (arXiv 2311.18677)
• Kwon et al.: PagedAttention and vLLM (arXiv 2309.06180)
• Shoeybi et al.: Megatron-LM (arXiv 1909.08053)
• Prabhakar et al.: SambaNova SN40L (arXiv 2405.07518)
• NVIDIA: GB200 NVL72
• NVIDIA: H100
• NVIDIA: H200
• NVIDIA: CUDA Programming Guide, CUDA Graphs
• Cerebras: WSE-3 announcement, 13 March 2024
• Cerebras: Introducing Cerebras Inference, 27 August 2024
• Cerebras: Llama 3.1 405B inference, 18 November 2024
• Cerebras: chip page (WSE-3 Turbo)
• Groq: LPU architecture page (LPX rack)
• Groq: licensing agreement announcement, 24 December 2025
• Groq: GroqCloud models documentation (speeds and prices, checked 25 October 2026)
• Abts et al.: A Software-defined Tensor Streaming Multiprocessor for Large-scale Machine Learning, ISCA 2022
• SambaNova: SambaNova Cloud launch, 10 September 2024
• Moonshot AI: Kimi K2 model card
• vLLM: speculative decoding documentation
• Microsoft Azure: Retail Prices API (ND96isr H100 v5 and ND96isr H200 v5, East US 2, checked 25 October 2026)

Last reviewed: 25 October 2026. Hardware specifications are vendor peak figures; vendor speed claims are dated and were not independently verified; cost figures are estimates from the stated assumptions.