LLM Inference Cost by Chip and Model Size, 1B to 1 Trillion Parameters

Serving a language model costs roughly what it costs to stream its weights and its key value cache out of memory once per generated token, divided across however many requests you can batch together. On list prices checked on 25 October 2026, our model puts a 1B model at roughly 0.26 to 0.44 US dollars per million output tokens, a 10B model at roughly 1.4 to 2.3 dollars, a 100B dense model at roughly 4.4 to 7.3 dollars, a 1T dense model at roughly 16 to 29 dollars and a 1T mixture of experts model with 32.6B active parameters at roughly 8 to 14 dollars, with the spread coming from the chip. Those are planning estimates built from stated assumptions, not quotes, and the rest of this post shows every step so you can swap in your own numbers.

The method is simple enough to check with a calculator: weight memory from parameter count and precision, KV cache from the attention layout, chips per replica from HBM capacity, a decode ceiling from memory bandwidth, a prefill cost from FLOPs, and cost per million tokens from the hourly price. We then compare the ceilings with measured MLCommons MLPerf Inference v5.1 results so you can see how far real systems land from the theory. Every chip figure is a vendor peak figure from a vendor page, every price is a public list price with the date we checked it, and every computed number came out of a Python script that we ran for this article. The script is included near the end. It runs on any laptop; it does not touch a GPU.

Key takeaways

• Decode is a memory bandwidth problem first. At batch size 1, a 10B model in FP8 on one H100 SXM tops out near 321 tokens per second, because each token needs about 10.4 GB read from HBM and the H100 SXM moves 3.35 TB/s.
• Batching is the main cost lever. The same weights are read once per step whatever the batch, so tokens per second climbs with batch size until the KV cache, which grows with every sequence, takes over the memory traffic.
• The KV cache, not the weights, often decides how many chips you need. With grouped query attention (GQA), our 100B reference model stores 456 KiB of BF16 KV cache per token, which is 3.8 GB per 8,192 token sequence. Thirty two such sequences need 122 GB, more than the weights in FP8.
• Prefill is cheap per token because it is compute bound. Prefill cost per million input tokens comes out lower than decode cost per million output tokens in every priced row of our tables, by a factor between 1.4 (1T dense on TPU v5p) and 94 (1T MoE on TPU v6e). That gap is why many API price lists charge less for input than for output.
• A 1T mixture of experts model costs about as much memory as a 1T dense model, but far less compute per token. Kimi K2 has about 1.04T total and 32.6B active parameters. It still needs about 1 TB of HBM at FP8, but each token touches about 3% of the weights, so at large batch it decodes many times faster than a dense 1T model on the same chips.
• Quantization from 16 bits to 8 bits is well studied and usually close to lossless; 4 bits needs care. GPTQ reports negligible accuracy loss at 3 to 4 bits on 175B models, while an empirical study of LLaMA3 found "non-negligible degradation" at ultra low bit widths.
• Measured systems land well below the roofline. In MLPerf Inference v5.1, an 8 GPU H200 system with FP8 weights generated 34,812 tokens per second on Llama 2 70B Offline, about 30% of our compute ceiling for that model; an 8 GPU B200 system with FP4 weights generated 102,725 tokens per second, about 20% of its FP4 compute ceiling.
• Chips without a public on-demand price are left out of the cost columns. That covers MI325X, MI355X, Gaudi 3 and Trainium2 on the date we checked. Their throughput ceilings are still shown.

How to read the numbers in this post

There are four kinds of number here and they should not be mixed up.

1. Vendor peak specifications. HBM capacity, HBM bandwidth and peak tensor throughput, copied from NVIDIA, AMD, Intel, Google and AWS pages. NVIDIA lists Hopper and Blackwell tensor throughput with structured sparsity; its Blackwell pages state "Dense is ½ sparse spec shown", and we halve the Hopper sparse figures the same way. Every throughput figure is labelled dense or sparse and with its precision.
2. Public list prices. On-demand hourly prices from the AWS EC2 on-demand price list, the Azure retail prices API and the Google Cloud TPU pricing page, all checked on 25 October 2026. Discounts, reservations, committed use and spot prices change the answer a lot; we show a few where the provider publishes them.
3. Model estimates. Throughput ceilings and costs computed from the formulas below. These are upper bounds or planning estimates, not measurements. They assume perfect scaling across chips in a replica and ignore communication time unless we say otherwise.
4. Measured results. Only from MLCommons MLPerf Inference v5.1, read directly from the submitters' published log summaries in the MLCommons results repository, with the submitter and system named.

We did not run any model on any accelerator for this article. The Python in this post ran on a Mac; the serving commands are shown as they appear in vendor documentation and were not executed.

The chips and their memory systems

For inference, three numbers per chip matter most: how much HBM it has (capacity decides how many chips one copy of the model needs), how fast it reads that HBM (bandwidth decides decode speed), and how many dense low precision operations per second it can do (compute decides prefill speed and the ceiling for very large batches).

Vendor specifications, per chip

Where a vendor gives a figure only for an 8 GPU system, we divide by eight and say so. GiB figures from Google and AWS are converted to decimal GB (1 GiB is 1.074 GB) so every row uses the same unit.

All of these are vendor peak figures. Sustained throughput depends on kernels, batch shapes, interconnect and software versions, and is always lower.

Two details in this table change the cost answer more than people expect. First, the 8-bit column mixes FP8 and INT8, because Google lists INT8 for v5e and v6e and AWS lists INT8 for Inferentia2. Both are 8-bit, but they need different quantization recipes. Second, only Blackwell and MI355X list a dense 4-bit rate. On the others, a 4-bit weight format still saves memory and bandwidth, but the matrix multiply runs at the 8-bit or 16-bit rate after the weights are expanded.

Ridge points: when decode stops being memory bound

The roofline model (Williams, Waterman and Patterson, 2009) says a kernel is memory bound when its arithmetic intensity, in FLOPs per byte of memory traffic, is below the chip's ridge point, which is peak FLOP/s divided by peak bandwidth.

A decode step at batch size B on 8-bit weights does about 2 FLOPs per parameter per sequence and reads each parameter once, so its intensity on the weight traffic alone is about 2B FLOPs per byte. To cross a ridge of 412 on an H200 you would need a batch of about 206 sequences, and the KV cache traffic, which grows with batch, pushes that point further out. That is why nearly every row in the cost tables below is memory bound. The deeper background on HBM, bandwidth and rooflines is in our post on GPU memory and interconnects.

Public prices, checked 25 October 2026

Google also publishes commitment prices per chip-hour: 1.89 USD (one year) and 1.22 USD (three years) for v6e, and 8.40 USD and 5.40 USD for Ironwood. Azure's H200 v5 price ranges from 84.80 USD in westus2 to 131.44 USD in japanwest for the same VM. We found no on-demand price for Trainium2 in the AWS us-east-1, us-east-2 or us-west-2 Linux price lists on the date checked, and no public list price for MI325X, MI355X or Gaudi 3 instances from the providers above, so those chips have no cost column.

For the cost tables we use one price per chip: AWS for H100, H200, B200 and Inferentia2, Azure for MI300X, and Google for the TPUs. The price ranges and why they move are covered in our post on GPU rental prices in 2026.

The four model sizes, made concrete

"A 100B model" is not specific enough to cost. The KV cache depends on the number of layers, the number of key value heads and the head dimension, so we fix one reference architecture per size. The 1B model is Meta's Llama 3.2 1B as published. The 10B, 100B and 1T dense models are our own reference designs built from Llama 3 shapes. The 1T MoE model is Moonshot AI's Kimi K2, as published.

Meta's Llama 3 paper gives the 8B, 70B and 405B shapes (32, 80 and 126 layers; model dimensions 4,096, 8,192 and 16,384; 8 key value heads each). Meta's llama-models repository lists the Llama 3.2 1B shape, and its model card gives 1.23B parameters, which matches our count of 1.24B within rounding. Our parameter count uses the SwiGLU feed forward layer (three matrices), separate input and output embeddings except where Meta ties them, and a 128,256 token vocabulary.

Kimi K2's technical report gives 61 layers, a hidden dimension of 7,168, an expert hidden dimension of 2,048, 384 experts with 8 selected per token plus 1 shared expert, 64 attention heads, multi-head latent attention (MLA) and about 1.04T total and 32.6B activated parameters. The released config file lists a KV latent rank of 512, a RoPE head dimension of 64, and FP8 E4M3 weights with 128 by 128 block scales.

Weight memory at FP16, FP8 and FP4

Weight memory is parameters times bytes per parameter. FP16 and BF16 use 2 bytes, FP8 and INT8 use 1 byte. For 4-bit we use 4.25 bits per parameter: a 4-bit element plus one shared 8-bit scale per block of 32 elements, the layout of the OCP microscaling (MX) formats evaluated by Rouhani and colleagues in 2023. Other 4-bit layouts with smaller blocks carry slightly more overhead.

Two lessons sit in this table. A 1T model does not fit on any single accelerator at any precision listed here; the biggest HBM in the spec table is 288 GB. And a mixture of experts model saves nothing on weight memory: every expert has to be resident somewhere, even though each token uses only a few.

The KV cache, worked from the architecture

During generation the model stores, for every past token and every layer, a key vector and a value vector for each key value head. Grouped query attention (Ainslie and colleagues, 2023) shares each key value head across several query heads, which shrinks this cache by the ratio of query heads to KV heads.

KV bytes per token = 2 (K and V) x layers x KV heads x head dim x bytes per element

1B   : 2 x  16 x  8 x  64 x 2 bytes =    32,768 bytes =    32 KiB
10B  : 2 x  40 x  8 x 128 x 2 bytes =   163,840 bytes =   160 KiB
100B : 2 x 114 x  8 x 128 x 2 bytes =   466,944 bytes =   456 KiB
1T   : 2 x 138 x 16 x 128 x 2 bytes = 1,130,496 bytes = 1,104 KiB

MLA (Kimi K2) stores one compressed latent plus a RoPE key per layer:
1T MoE: 61 x (512 + 64) x 2 bytes = 70,272 bytes = 68.6 KiB

Kimi K2 inherits MLA from the DeepSeek-V3 design, and its KV cache per token is about 16 times smaller than our 1T dense reference. That is an architecture choice, not a chip feature, and it shows how much the attention layout matters to serving cost.

Multiply by context length to get the cache per sequence, and by batch size to get the cache per replica.

At batch 256 and 8,192 tokens, the 10B model's KV cache (344 GB) is 35 times its FP8 weights. This is why serving engines such as vLLM manage the cache in pages (Kwon and colleagues, 2023, report 2 to 4 times higher throughput from PagedAttention at the same latency), and why KV cache quantization is an active research area.

Minimum chips per replica

A replica is one complete copy of the model that can serve requests. We size it as the smallest power of two number of chips whose usable HBM (we assume 90% of the vendor capacity is usable, leaving room for activations, the runtime and fragmentation) holds the FP8 weights plus the BF16 KV cache for at least one 8,192 token sequence. Powers of two match common tensor parallel layouts.

A minimum replica is rarely the cheapest replica. At the minimum, almost all the HBM holds weights, so there is room for only a handful of sequences, and a small batch wastes the bandwidth spent reading the weights. A 1T dense model on 8 H200s, for example, fits with room for just one 8,192 token sequence. In the cost section we let each replica grow to up to 8 times its minimum and pick the size with the lowest cost per token.

The Llama 3 paper gives a real example of the same arithmetic: Meta writes that "Llama 3 405B does not fit in the GPU memory of a single machine with 8 Nvidia H100 GPUs" in BF16, and serves it across 16 GPUs on two machines, with tensor parallelism inside each machine and pipeline parallelism across them.

Decode: a bandwidth bound, and what batching does to it

The formula

Each decode step produces one token for each sequence in the batch. To do it, the chip reads every weight once (shared across the batch) and reads each sequence's KV cache (not shared). If the step is memory bound:

step time  >= (weight bytes + batch x KV bytes per token x average context) / (chips x bandwidth)
tokens/s   <=  batch / step time

If the step is compute bound instead:

step time  >= batch x (2 x active parameters + 4 x layers x context x attention width) / (chips x peak FLOP/s)

We take the larger of the two times. In the tables below the average attended context during decode is 4,096 tokens (for example a 2,048 token prompt and up to 6,144 generated tokens, inside the 8,192 maximum), the KV cache is BF16, the weights are FP8, and the compute peak is the dense 8-bit figure.

Worked example 1: a 10B model on one H100, batch 1

weights read per step : 9.77e9 params x 1 byte            =  9.77 GB
KV read per step      : 1 x 163,840 bytes x 4,096 tokens    =  0.67 GB
total                                                       = 10.44 GB
H100 SXM bandwidth                                          =  3.35 TB/s
ceiling = 3.35e12 / 10.44e9                                 =  about 321 tokens per second

At batch 1, the compute side needs about 20 GFLOPs per token, which the H100's 1,979 dense FP8 TFLOPS could do over 90,000 times per second. Bandwidth is the limit by a factor of several hundred.

Worked example 2: the batching effect

Raising the batch shares the weight read across more sequences. The KV read grows with every sequence you add, so the gain flattens. These rows come from the same script.

tokens/s (100B on 2x H200, FP8 weights, 4,096 average context)

batch   1 |#                                          95
batch   8 |######                                    668
batch  16 |###########                             1,179
batch  32 |##################                      1,910
batch  64 |##########################              2,767  (cache does not fit)
batch 128 |#################################       3,567  (cache does not fit)
batch 256 |#######################################  4,171  (cache does not fit)

Three things happen as the batch grows. Throughput per replica rises, which is what lowers cost per token. Speed per user falls, which is what users feel as slower streaming. And memory runs out, which is why the 64 and above rows need a bigger replica or a smaller cache. This is the central trade in LLM serving: cost per token against tokens per second per user. Any single cost number assumes a point on that curve.

Why the KV cache dominates at long context

In the 100B case at batch 32, each step reads 99.6 GB of weights and 32 x 467 KB x 4,096, which is 61.2 GB, of KV cache. At batch 128 the KV read would be 245 GB, well over twice the weights. Halving the KV cache with FP8 storage or using an attention design with fewer KV heads moves the whole curve up. Making the weights smaller helps much less once the cache dominates, as the precision section below shows.

Prefill: the compute bound side

Prefill processes the prompt. All prompt tokens go through each layer together, so the weights are read once for thousands of tokens and the arithmetic intensity is high. Prefill is compute bound on every chip here for any realistic prompt length.

prefill FLOPs per token ~ 2 x active parameters + 4 x layers x (prompt / 2) x attention width

The attention term uses half the prompt as the average causal span. For a 2,048 token prompt:

To turn FLOPs into time we need a model FLOPs utilization (MFU), the share of peak the run actually achieves. Meta reports 38% to 43% BF16 MFU for Llama 3 405B training. Inference prefill is a different workload, with different kernels and shapes, and published inference MFU figures are scarce, so we use 40% as a stated planning assumption rather than a measurement.

Worked example, 100B on H200 at FP8:

time per million prompt tokens on one chip = 203.1e15 / (1,979e12 x 0.40) = 256.6 s
cost at 7.912 USD per chip-hour            = 256.6 / 3,600 x 7.912      = 0.564 USD
at 60% fleet utilization                   = 0.564 / 0.60               = 0.94 USD per million input tokens

Prefill does not care how many chips are in the replica, to first order: twice the chips finish twice as fast. It does care about the chip's dense 8-bit rate and its price, which is why TPU v6e, with 1,836 INT8 TOPS at 2.70 USD per chip-hour, has the lowest prefill cost in most rows and TPU v5p, whose FP8 rate equals its BF16 rate, has the highest.

Cost per million tokens, chip by chip

The cost model

output cost per M tokens = chips x price per chip-hour / (decode ceiling x derate x utilization x 3,600) x 1e6
input cost per M tokens  = price per chip-hour x prefill FLOPs per M / (8-bit peak x MFU x utilization x 3,600)

The stated assumptions:

• FP8 or INT8 weights, BF16 KV cache, 8,192 token maximum context, 4,096 token average context during decode, 2,048 token prompts.
• Batch up to 256 sequences per replica, limited by the KV cache that fits in 90% of HBM.
• Replica size chosen among powers of two from the minimum up to 8 times the minimum, to minimize cost per output token.
• Derate 0.30: the share of the roofline ceiling a real serving stack reaches. We chose it from the MLPerf comparison below, where measured systems landed at about 20% to 30% of their compute ceilings. Applying it to memory bound rows is a judgment call.
• Utilization 0.60: the share of paid hours spent doing useful work. Real traffic has peaks and troughs; a fleet sized for the peak sits partly idle.
• MFU 0.40 for prefill.

The "ceiling" cost column drops the derate and the utilization, so it is the cost if the hardware ran flat out at its roofline every hour. The "planning" column applies both and is 5.6 times higher. Real deployments usually sit between the two.

1B model

At 1B parameters the weights are tiny and the KV cache at 4,096 tokens of context is almost the whole memory traffic, so every chip lands within a narrow cost band. The cheaper, lower bandwidth chips do as well as the flagships per dollar, because the flagships' extra capacity is not needed. A 1B model is also small enough that the batch cap of 256 is the binding limit; with shorter contexts or a larger batch the costs fall further.

10B model

A 10B model fits on one chip of almost any kind, but the cheapest replica is bigger than one chip. The extra chips hold KV cache for a full batch of 256 long sequences. If your traffic is short prompts and short answers, the one chip replica wins instead. This is the most common sizing mistake for mid sized models: choosing chips by the weights alone.

100B model

Here the bandwidth per dollar ranking starts to show. MI300X on Azure's price and Inferentia2 on AWS's price come out lowest per output token in our model; B200 comes out with the highest throughput per chip but a price per chip-hour that nearly offsets it. Two cautions apply. A 32 chip Inferentia2 or 64 chip TPU v5e replica spreads tensor parallel traffic over many more chips, and our model does not charge for that communication, so these rows flatter small chips. And our model does not account for software maturity, which is often what decides the real derate.

1T dense model

A dense 1T model is expensive on every chip: about 2 TFLOPs per token, 1 TB of FP8 weights, and 1.1 MB of KV cache per token in our reference design. The TPU v5p row is the only compute bound row in the whole set, because v5p's 8-bit rate equals its BF16 rate (459 TFLOPS) and the attention FLOPs of a 138 layer model at 4,096 tokens of context are large. The rows with 128 or 256 small chips are mathematically valid and practically fragile: they depend on a pod scale interconnect, and every extra chip is another device that can fail during a request.

1T mixture of experts model (Kimi K2)

At a batch of 256, the MoE model costs about half what the dense 1T model costs per output token in this model, and its prefill costs about 30 times less, because prefill tracks active parameters. The batch cap of 256 hides most of the MoE advantage on decode, as the next section shows.

Serving a 1T MoE with expert parallelism

Why batch size matters even more for MoE

Each token picks 8 of 384 routed experts in every MoE layer. If routing were uniform and independent (a simplifying assumption; real routers are trained to balance load but are not uniform), the share of experts that at least one token in the batch touches is 1 - (1 - 8/384)^B.

From Kimi K2's published shapes, one routed expert is 3 x 7,168 x 2,048, about 44.0M parameters. The 60 MoE layers hold 384 such experts each, about 1,014.7B routed parameters, of which 8 per layer, 21.1B, are active per token. The remaining active parameters, about 11.5B, are attention, the shared expert, the one dense layer and the embeddings.

At batch 1, a decode step reads only about 33B parameters. At batch 256 it reads nearly all of them. From then on, the weight read per step is fixed, the same as a dense 1T model with FP8 weights, and adding sequences only adds KV traffic (small, thanks to MLA) and compute (small, 65 GFLOPs per token). So the MoE model keeps scaling with batch long after a dense model has run out of KV room.

Sixteen H200s have room for about 1,720 sequences of 8,192 tokens next to 1.04 TB of FP8 weights, because the MLA cache is only 0.58 GB per sequence. That is why operators of large MoE models run very large batches across many chips, often with prefill and decode on separate pools.

The layout

Expert parallel decode for a 384 expert model on 32 GPUs (12 experts per GPU per layer)

          tokens in batch (thousands)
                    |
        [ attention + shared expert ]   data or tensor parallel, replicated
                    |
            router picks 8 of 384
                    |
      all-to-all DISPATCH (token hidden state to the GPUs that own its experts)
      /      |      |      |        |      \
   GPU0   GPU1   GPU2   GPU3  ...  GPU30  GPU31
   e0-11  e12-23 e24-35 e36-47     e360-371 e372-383
      \      |      |      |        |      /
      all-to-all COMBINE (expert outputs back to the token's home GPU)
                    |
               next layer

With 32 GPUs, each holds 12 experts per layer, about 31.7 GB of FP8 routed expert weights across the 60 MoE layers, plus its share of the attention weights and KV cache. With 64 GPUs, each holds 6 experts per layer and about 16 GB of the total weights.

The interconnect bill

Each token's hidden state (7,168 values) goes to 8 experts and comes back in every MoE layer. With FP8 dispatch and BF16 combine, that is about 172 KB per token per layer, or about 10.3 MB per token across the 60 MoE layers. At batch 4,096 on 32 H200s the roofline step time is about 14.4 ms, so each GPU has to move about 92 GB/s of all-to-all traffic to keep up.

Inside an 8 GPU node, NVLink on H200 offers 900 GB/s per GPU, far more than needed. Across nodes the budget is much smaller: AWS lists 3.2 Tbps of network bandwidth for an 8 GPU p6-b200.48xlarge, which is 400 Gb/s or 50 GB/s per GPU. So an expert parallel group that spans nodes can become network bound before it becomes HBM bound. Rack scale NVLink domains such as GB200 NVL72, which Azure describes as 72 GPUs with 130 TB/s of cross sectional NVLink bandwidth, exist largely for this traffic pattern. None of this communication time is in the tables above, so treat the MoE rows as optimistic.

Where quantization helps quality, and where it hurts

Lower precision cuts memory and bandwidth, and on chips with native FP8 or FP4 it raises the compute ceiling too. The question is what it does to output quality. The literature is clear on some points and still open on others.

Now apply that to cost, using the 100B model with a cost optimal replica, BF16 KV and 4,096 tokens of average context:

Going from FP16 to FP8 halves the minimum replica on B200 and adds about 17% throughput per chip. Going from FP8 to FP4 adds only about 9% more, because at this context length the KV cache already makes up most of the bytes read each step. The lesson: at long context, quantizing the KV cache, or choosing an attention design with a smaller cache, is worth more than squeezing the weights further.

Sanity check against MLPerf Inference v5.1

MLCommons publishes every MLPerf submission, including the log summary of each run. We read the Llama 2 70B (99% accuracy target) results from the v5.1 closed division, published 9 September 2025. The submitters' measurement files list the weight data type.

The Offline runs generated about 293 tokens per sample on the H200 system and about 276 on the Lambda B200 system (tokens per second divided by samples per second), so these are short answers with high batch.

Llama 2 70B has 80 layers, 8 KV heads and a model dimension of 8,192, close to our 100B design. Its decode compute ceiling with 8 chips is the dense 8-bit or 4-bit rate times 8, divided by about 2 x 69B FLOPs per token:

8x H200, FP8 : 8 x 1,979e12 / 138e9 = 114,725 tokens/s ceiling ; measured 34,812  = 30.3%
8x B200, FP4 : 8 x 9,000e12 / 138e9 = 521,739 tokens/s ceiling ; measured 102,725 = 19.7%

If we also count KV traffic and assume an average context of 1,024 tokens, which is our assumption and not an MLPerf parameter, the 8x B200 FP4 ceiling becomes memory bound at about 318,000 to 347,000 tokens per second for batches of 1,024 to 2,048, and the measured result is about 30% to 32% of that. Either way, well tuned systems from vendors and their cloud partners reached roughly one fifth to one third of the theoretical ceiling. That is where our 0.30 derate comes from.

You can also turn MLPerf results straight into a cost, using the AWS and Azure list prices above:

These are much lower than our 100B planning numbers (4.8 to 7.3 USD), and the gap is instructive. MLPerf uses a 70B model with short outputs and very large offline batches. Our tables assume a 100B model, 4,096 tokens of average context, a batch cap of 256 and a 0.30 derate. Context length and batch size, not the chip, explain most of the difference. The B200 MLPerf row also uses FP4 weights, while our tables use FP8. If your workload looks like MLPerf (short chats, high concurrency), the MLPerf-derived costs are the better guide. If it looks like ours (long documents, coding agents, retrieval with long context), expect costs closer to the planning columns. For a simpler single chip walk-through of the same idea, see our earlier estimate of GPU cost per million tokens when self-hosting.

The calculator

This is a trimmed version of the script behind every number above. It ran with the system Python 3 on a Mac. Change the chip list, the prices and the assumptions to match your own case.

import math

GB, TB = 1e9, 1e12
CTX_MAX, CTX_AVG, USABLE = 8192, 4096, 0.90
DERATE, UTIL = 0.30, 0.60   # stated assumptions, see text

# chip: HBM GB, bandwidth TB/s, dense 8-bit TFLOPS, USD per chip-hour (None if no public price)
chips = {
    "H100 SXM": (80, 3.35, 1979, 55.04 / 8),
    "H200 SXM": (141, 4.8, 1979, 63.296 / 8),
    "B200":     (180, 8.0, 4500, 113.9328 / 8),
    "MI300X":   (192, 5.3, 2610, 57.6 / 8),
    "TPU v6e":  (32, 1.638, 1836, 2.70),
}

# model: params, active params, layers, KV elements per token (2 x layers x kv_heads x head_dim)
models = {
    "10B":  (9.77e9, 9.77e9, 40, 2 * 40 * 8 * 128),
    "100B": (99.64e9, 99.64e9, 114, 2 * 114 * 8 * 128),
}

def pow2(x):
    return 1 << max(0, math.ceil(math.log2(x)))

def decode_ceiling(params, active, kv_elems, n, bw, peak, batch, w_bytes=1.0, kv_bytes=2.0):
    mem_bytes = params * w_bytes + batch * kv_elems * kv_bytes * CTX_AVG
    t_mem = mem_bytes / (n * bw * TB)
    t_cmp = batch * 2 * active / (n * peak * 1e12)   # attention FLOPs omitted in this short version
    return batch / max(t_mem, t_cmp)

for m, (P, A, L, kv) in models.items():
    for c, (mem, bw, peak, price) in chips.items():
        kv_seq = kv * 2 * CTX_MAX
        n0 = pow2((P + kv_seq) / (USABLE * mem * GB))
        best = None
        n = n0
        while n <= 8 * n0:
            batch = int(min(256, (USABLE * n * mem * GB - P) // kv_seq))
            if batch >= 1:
                tps = decode_ceiling(P, A, kv, n, bw, peak, batch)
                if best is None or n / tps < best[0]:
                    best = (n / tps, n, batch, tps)
            n *= 2
        _, n, batch, tps = best
        cost = n * price / (tps * DERATE * UTIL * 3600) * 1e6
        print(f"{m:5s} {c:9s} chips={n:3d} batch={batch:3d} ceiling={tps:9,.0f} tok/s  planning=${cost:.2f}/M")

Run as printed, it reproduces the 10B and 100B rows of the tables above to within one token per second, because those rows are memory bound and the omitted attention FLOPs do not change them. The full version adds attention FLOPs, the MoE expert touching model, prefill cost and the extra chips. If you change only one thing, change CTX_AVG to your real average context: it moves the answer more than any chip choice.

Common mistakes

• Using sparse FLOPS as if they were dense. NVIDIA's headline Hopper and Blackwell figures are with 2:4 structured sparsity. Dense is half. AMD lists both. Dividing by the wrong one doubles your prefill estimate.
• Comparing FP4 on one chip with FP8 on another. The B200's FP4 rate is twice its FP8 rate. A comparison that runs one chip in FP4 and the other in FP8 is a precision comparison, not a chip comparison. MLPerf measurement files state the weight type; read them.
• Sizing the replica by the weights alone. At 8,192 tokens, the KV cache for a 10B model at batch 256 is 35 times the FP8 weights.
• Treating peak as achieved. Measured MLPerf results from tuned submissions reached about 20% to 30% of the compute ceiling. Your first deployment will probably do worse.
• Ignoring idle time. A fleet that is busy 60% of the time costs 1.67 times more per token than the same fleet at 100%.
• Forgetting the interconnect. Tensor parallel all-reduce and expert parallel all-to-all both cost time that a roofline ignores. A 64 chip replica of small chips looks cheap on paper and may not be in practice.
• Mixing GB and GiB. Google and AWS often quote GiB. 192 GiB is 206 GB. The error is 7%, enough to flip a fit or no-fit decision.
• Assuming MoE saves memory. It saves compute and bandwidth per token. Every expert still has to sit in HBM somewhere.
• Quantizing without checking real outputs. Meta's experience with FP8 on Llama 3 405B showed benchmark scores that looked fine while some answers were corrupted.

Where Safeguard fits, honestly

Nothing in this post is something Safeguard changes. Safeguard does not design, program or speed up accelerators, and it does not reduce GPU cost. The decisions above sit on top of a software stack that is very much our business, though: GPU drivers, the CUDA or ROCm toolkits, NCCL or RCCL, a serving engine, Python dependencies, container base images and the model files themselves. Each of those is a component with a version, a supplier and a vulnerability history, and an inference image is usually rebuilt every time one of them moves.

Safeguard scans repositories, container images and SBOM files (safeguard scan --dir ., --image <name>, --sbom <file>), matches the components against vulnerability databases, ranks findings with reachability analysis, evaluates policies and gates, shows which of your projects contain a given component and version, and can open fix pull requests through autonomous remediation, which ships switched off by default. Our GPU software supply chain guide lists what to put in the SBOM for a GPU serving stack. For loading model files, prefer formats that hold only tensors, such as safetensors, over formats that can execute code when loaded, and pin model files by hash the same way you pin images by digest.

FAQ

How much does it cost to run a 70B or 100B model per million tokens?

It depends mostly on context length, batch size and utilization. Turning measured MLPerf v5.1 Llama 2 70B Offline throughput into cost with list prices gives about 0.31 to 0.58 USD per million output tokens at 100% utilization on 8 GPU systems. Our planning model for a 100B model at 4,096 tokens of average context, a batch cap of 256, a 0.30 derate and 60% utilization gives about 4.4 to 7.3 USD. Your real number is likely between the two.

How many GPUs do I need to serve a 1 trillion parameter model?

At FP8, the weights alone are about 1 TB. That is at least 8 chips with 141 GB or more of HBM (H200, B200, MI300X class), or 16 chips with 80 GB. A practical replica with room for a useful batch is usually 2 to 4 times the minimum. A mixture of experts model with 1T total parameters needs the same memory but far less compute and bandwidth per token.

Is LLM inference memory bound or compute bound?

Decode, the token by token generation phase, is memory bound on every chip in this post unless the batch is in the hundreds or thousands. Prefill, which processes the prompt, is compute bound. That is why output tokens cost more than input tokens.

How do I calculate KV cache size?

Multiply 2 (keys and values) by the number of layers, the number of KV heads, the head dimension and the bytes per element. That gives bytes per token. Multiply by context length and batch size for the total. For a Llama 3 70B style layer stack with 80 layers, 8 KV heads and a head dimension of 128 in BF16, that is 320 KiB per token.

Does FP8 or FP4 quantization reduce quality?

Published work finds 8-bit inference close to lossless with methods that handle outlier activations, such as LLM.int8() and SmoothQuant, and FP8 training matched 16-bit quality in the FP8 formats paper. GPTQ reports negligible loss at 3 to 4 bits on 175B models, but a study of LLaMA3 found non-negligible loss at very low bit widths. Test on your own prompts.

Why are output tokens more expensive than input tokens?

Each output token needs a full pass over the weights and the KV cache, and that pass is limited by memory bandwidth. Input tokens are processed thousands at a time, so the weights are read once for many tokens and the chip runs near its compute limit. In our tables, prefill cost per million tokens is lower than decode cost in every priced row, by a factor between 1.4 and 94.

Which AI chip is cheapest for inference?

There is no single answer. In our model, the chips with the most HBM bandwidth per dollar did best on decode, and the chips with the most dense 8-bit compute per dollar did best on prefill, but the ranking changes with model size, context length, the price you actually pay and how mature the software stack is for your model. Chips without a public on-demand price could not be compared on cost at all.

What does mixture of experts do to serving cost?

It cuts compute per token to the active parameters (32.6B of about 1.04T for Kimi K2) and, at large batch, lets throughput keep growing after a dense model would have run out of room. It does not cut memory, and it adds all-to-all traffic between chips that can become the bottleneck when experts span nodes.

Can I trust MLPerf numbers for my own deployment?

They are the best public measured numbers, with the system, software and precision disclosed. They use fixed models, datasets and accuracy targets, so they tell you what a tuned system can do on that workload. Use them to calibrate your efficiency assumption, not as a promise for a different model or context length.

More in this series

Other posts in our AI chips and costs series:

• AI Accelerator TCO: Buy vs Rent, Capex, Power, Cooling and Utilization
• AI Chips Compared 2026: B200, H200, MI355X, Gaudi 3, TPU, Trainium
• Cheapest LLM Inference: How to Cut Cost per Million Tokens
• Fastest LLM Inference: Lowest Latency and Most Tokens per Second
• How Many GPUs Does a 1 Trillion Parameter Model Need?
• How to Estimate AI Training and Inference Cost: Formulas and a Calculator
• LLM Training Cost by Chip and Model Size: 1B to 1T, Worked

Fix it without waiting for upstream

The cost model above assumes your serving stack keeps running, and that depends on drivers, CUDA or ROCm libraries, serving engines and Python packages that ship security fixes on their own schedules. Rebuilding an inference image to pick up a fix is routine; knowing which images need it, and having a fixed version to move to, is the part teams struggle with.

Whatever the vulnerability, most of the risk sits in two gaps. The first is the time between a fix being published and that fix actually running in your production builds. The second, and worse, is the time before any fix exists at all, while a maintainer works out a patch and a release schedule. Waiting through both gaps is a habit, not a requirement. Safeguard gives teams three ways to close them.

1. Autonomous remediation: pull requests, not tickets

When a fixed version exists, Safeguard's Griffin models open a tested, policy gated pull request that moves the affected dependency to the safe version. You can run it on one repository or in bulk across every project that carries the vulnerable version. Remediation covers eight categories, including dependencies, licenses, security posture and findings. Reachability analysis ranks the vulnerable code paths your application actually calls first, so the first pull requests you merge are the ones that matter most. Autonomous remediation ships switched off, with every category set to Manual, so your team decides how much runs on its own and what still needs a human approval.

2. Gold Open Source: the safe version in seconds, free

gold.safeguard.sh is free and needs no login. Search any CVE or package to see affected and fixed versions, CVSS, EPSS, CISA KEV status and a recommended safe version, or use the "Am I affected?" checker. The same data is available through a free read API, README status badges and RSS threat feeds. The safeguard-sh/gold-scan-action GitHub Action can fail a build on a known exploited vulnerability, on any known vulnerability, or never, so a version you have just removed cannot quietly come back in next week's merge.

3. Gold Registry: start from components that are already fixed

registry.safeguard.sh provides hardened, drop in replacements for upstream packages and container images. Every Gold artifact is held continuously to zero exploitable critical or high CVEs, zero malware and a completed license check, with SLSA L3 or higher provenance, a Sigstore signature and a public evidence page you can hand to an auditor. An artifact loses its Gold badge if it regresses. Teams can point their own package directory at the registry so hardened equivalents resolve without changing how developers install anything.

When upstream has no fix yet, or will not ship one, Safeguard first works to get the fix accepted upstream. If the issue is severe or upstream will not move, Safeguard builds a fork, fixes the vulnerability there and publishes a hardened Gold build, so you are not stuck on a maintainer's release schedule. You can also request a hardened, compatibility tested build of a specific package. This fork and patch flow is in early access today, so ask which packages it covers before you plan around it.

One honest limit: when the vulnerable code lives in a vendor appliance, a hosted service or an operating system runtime you do not build yourself, the fix still has to come from that vendor or distribution. The steps above apply to the open source dependencies and container images you build and ship, which is where most teams carry the largest and least tracked share of their exposure.

See how Safeguard would help

You can download Safeguard and follow the CLI docs to try these yourself.

The first demo sets up the Safeguard runner on your own machine, approves it with its device code and then scans from the terminal. For an inference platform, that is how you would scan the serving images and the repositories that build them without sending source or images off the machine, for example safeguard scan --image against a serving image pinned by digest.

Click through it at your own pace, or open it full screen.

The second demo is a real CLI session: install, open the interactive terminal and ask it to fix a lodash CVE. The assistant reads the finding and the advisory, asks before any command that changes something, edits package.json to the fixed version, runs npm install and npm test with approval, prints a summary and pushes nothing. The same flow applies to the Python side of a serving stack, such as moving a pinned serving engine dependency to a fixed version and rerunning your tests.

Click through it at your own pace, or open it full screen.

The third demo connects Claude to the Safeguard MCP server with OAuth and shows how to choose which tools Claude can use. When a new advisory lands for a driver or serving library, an engineer can then ask which of their projects contain the affected component and version, and get the answer from their own scan data.

Click through it at your own pace, or open it full screen.

Quick check

Sources

• NVIDIA: H100 product page
• NVIDIA: H200 product page
• NVIDIA: HGX platform (HGX B300 and HGX B200 specifications)
• NVIDIA: DGX B200
• NVIDIA: DGX B300
• AMD: Instinct MI300X
• AMD: Instinct MI325X
• AMD: Instinct MI355X
• Intel Gaudi documentation: Gaudi architecture
• Google Cloud: TPU v5e
• Google Cloud: TPU v5p
• Google Cloud: TPU v6e
• Google Cloud: TPU7x (Ironwood)
• Google Cloud: TPU system architecture
• Google Cloud: TPU pricing, checked 25 October 2026
• AWS Neuron documentation: Trainium2
• AWS Neuron documentation: Inferentia2
• AWS: EC2 on-demand pricing, checked 25 October 2026
• AWS: EC2 P5 instances
• AWS: EC2 P6 instances
• AWS: EC2 Inf2 instances
• Microsoft Azure: retail prices API, checked 25 October 2026
• Microsoft Learn: ND GB200 v6 size series
• Microsoft Learn: ND MI300X v5 size series
• MLCommons: MLPerf Inference v5.1 results repository
• MLPerf v5.1, Nebius 8x H200 Llama 2 70B Offline log summary
• MLPerf v5.1, Nebius 8x B200 Llama 2 70B Offline log summary
• MLPerf v5.1, Lambda 8x B200 Llama 2 70B Offline log summary
• MLPerf v5.1, AMD 8x MI300X Llama 2 70B Offline log summary
• MLPerf v5.1, AMD 8x MI325X Llama 2 70B Offline log summary
• MLPerf v5.1, Nebius H200 measurement metadata (weight data type)
• Grattafiori and colleagues: The Llama 3 Herd of Models, arXiv 2407.21783, July 2024
• Meta: llama-models repository, model shapes (sku_list.py)
• Meta: Llama 3.2 model card
• Kimi Team: Kimi K2, Open Agentic Intelligence, arXiv 2507.20534, July 2025
• Moonshot AI: Kimi K2 Instruct config.json
• DeepSeek-AI: DeepSeek-V3 Technical Report, arXiv 2412.19437, December 2024
• Ainslie and colleagues: GQA, arXiv 2305.13245, May 2023
• Kwon and colleagues: PagedAttention and vLLM, arXiv 2309.06180, September 2023
• Dettmers and colleagues: LLM.int8(), arXiv 2208.07339, August 2022
• Xiao and colleagues: SmoothQuant, arXiv 2211.10438, November 2022
• Micikevicius and colleagues: FP8 Formats for Deep Learning, arXiv 2209.05433, September 2022
• Frantar and colleagues: GPTQ, arXiv 2210.17323, October 2022
• Lin and colleagues: AWQ, arXiv 2306.00978, June 2023
• Huang and colleagues: An empirical study of LLaMA3 quantization, arXiv 2404.14047, April 2024
• Liu and colleagues: KIVI, arXiv 2402.02750, February 2024
• Hooper and colleagues: KVQuant, arXiv 2401.18079, January 2024
• Rouhani and colleagues: Microscaling Data Formats for Deep Learning, arXiv 2310.10537, October 2023
• Williams, Waterman and Patterson: Roofline, Communications of the ACM 52(4), April 2009

Last reviewed: 25 October 2026. Prices were checked on that date and change often; vendor specifications are peak figures and some are marked preliminary by their vendors. All costs are estimates built from the stated assumptions.