Cheapest LLM Inference: How to Cut Cost per Million Tokens

The cheapest way to serve a large language model is to keep a memory-bandwidth-efficient accelerator full of concurrent requests, store the weights and the KV cache in as few bits as your quality tests allow, reuse every shared prompt prefix, and pick the smallest model (or a mixture of experts model with few active parameters) that still passes your evaluation. Batching alone moves the cost per million tokens by more than an order of magnitude; quantization, caching and model choice each move it by a further factor of two or more. Chip choice matters, but it matters less than how full you keep the chip.

This guide works through every lever with explicit arithmetic, using vendor specifications and public list prices checked on 25 October 2026. Every number below came out of a python3 calculation from stated inputs, and every cost is an estimate built from assumptions you can change. None of it is a benchmark result.

Key takeaways

• Cost per million tokens is an hourly price divided by delivered throughput. Throughput during decode is limited by memory bandwidth, so the first chip metric to compare is GB/s of memory bandwidth per dollar of hourly price, not peak FLOPs.
• Batching is the largest lever. In our model, a 10B model on one H100 goes from about $5.78 per million output tokens at batch 1 to about $0.25 at batch 32, roughly 23 times cheaper, because the weights are read once per step and shared by every sequence in the batch.
• Quantization pays twice. Fewer bytes per weight means fewer bytes to read per token and fewer GPUs to hold the model. Moving a 100B model from FP16 to FP8 cut our bound cost by about 47 percent; 4-bit weights cut it by roughly another 40 percent. The quality cost must be measured on your task; published results range from negligible to noticeable.
• Prefix caching removes repeated prefill work. For a workload with a 2,000 token shared system prompt, caching cut our modelled per-request cost by about 35 percent. Hosted APIs price cache reads at a small fraction of normal input tokens.
• Mixture of experts is cheap at low batch and less cheap at high batch. A 1T MoE model with 32B active parameters reads far fewer bytes per token than a 1T dense model, but as batch size grows, more experts are touched per step and the advantage narrows.
• Distillation, speculative decoding, spot capacity and autoscaling each offer another factor of 1.5 to 6, with real trade-offs in quality, latency or reliability.
• The serving stack is where these levers live, and it is software with its own advisories. vLLM, SGLang, TensorRT-LLM and llama.cpp have all shipped security fixes in the last two years. Cheap tokens served from an unpatched engine are not cheap.

How to read the numbers in this guide

Three kinds of numbers appear here, and they should not be confused.

1. Vendor peak specifications. Memory capacity, memory bandwidth and peak throughput come from NVIDIA, AMD, Google and AWS product pages and documentation. Each throughput figure is labelled dense or sparse with its precision. These are peaks a chip cannot exceed, not what a real server sustains.
2. List prices. On-demand hourly prices come from the AWS EC2 on-demand price file (publication date 23 October 2026, US East, Linux), the Google Cloud TPU pricing page, and the Azure retail prices API, all checked on 25 October 2026. Prices change; check them again before you rely on them.
3. Modelled costs. These are upper bounds on throughput turned into lower bounds on cost, plus a "planning figure" that assumes you reach 50 percent of the bandwidth bound and keep the hardware 60 percent busy. Those two factors are assumptions, not measurements. Replace them with your own measured numbers as soon as you have them.

Where a chip has no public on-demand price in the sources we checked, we say so and leave it out of the cost tables rather than guess. That applies to AMD Instinct MI300X, MI325X and MI355X in this edition: their specifications are below, but we did not find an on-demand hourly price for them in the AWS price file, the Google Cloud price pages or the Azure retail prices API query we ran. Several smaller clouds do publish prices for them, and the method below works for any price you have.

The cost equation

Every lever in this guide changes one of four terms in a single equation.

                                hourly price of the hardware ($/hr)
cost per 1M tokens  =  ------------------------------------------------------  x 1,000,000
                        tokens per second  x  utilisation  x  3,600 s/hr

tokens per second (decode, one replica)  =  batch size / time per decode step

time per decode step  >=  max(  bytes read per step / memory bandwidth ,
                                FLOPs per step      / peak FLOP rate    )

bytes read per step   ~=  weight bytes actually touched
                          + batch size x average context length x KV bytes per token

Prefill (reading the prompt) and decode (generating output tokens one by one) behave differently.

  request timeline on one replica
  |<---- prefill: whole prompt in one pass ---->|<-- decode: one token per step -->|
  |  compute bound: ~2 x active params FLOPs    |  bandwidth bound: every step     |
  |  per prompt token, large matrix multiplies  |  re-reads the weights and the    |
  |  keep tensor cores busy                     |  KV cache of every sequence      |

During prefill, the model processes hundreds or thousands of prompt tokens at once, so each weight read from memory is used many times and the arithmetic units are the bottleneck. During decode, each step produces one token per sequence, so the GPU reads every weight once and does only a few FLOPs per byte. That is why decode cost tracks memory bandwidth and why batching helps: one read of the weights serves every sequence in the batch. The throughput metrics explainer covers time to first token and inter-token latency in more depth, and the self-hosting cost estimate shows a simpler version of the same calculation.

Assumed model shapes

Cost depends on the KV cache, so we need layer counts and attention shapes. For 10B and 100B we borrow the shapes of Llama 3 8B and 70B from Table 3 of the Llama 3 paper (32 and 80 layers, 8 key/value heads, 128 dimensions per head). For 1B and 1T dense we use stated assumptions. For the 1T mixture of experts model we use the published architecture of Kimi K2 (1T total parameters, 32B activated, 61 layers, 384 experts with 8 selected per token plus 1 shared expert, MLA attention) and, for the KV cache, the MLA dimensions published in the DeepSeek-V3 technical report (a 512 dimension compressed KV vector plus a 64 dimension decoupled key, 61 layers).

KV bytes per token for grouped query attention is 2 (keys and values) x layers x KV heads x head dimension x bytes per element. For MLA it is layers x (compressed dimension + decoupled key dimension) x bytes per element.

Lever 1: choose the chip by memory bandwidth per dollar

Since decode is usually bandwidth bound, the most useful first-pass metric for inference cost is gigabytes per second of memory bandwidth per dollar of hourly price. The table below divides each chip's vendor peak bandwidth by its on-demand price per chip. For 8-GPU instances we divide the instance price by eight.

Notes on the inputs. The B200 bandwidth is NVIDIA's DGX B200 figure of 64 TB/s across eight GPUs. AWS lists 1,432 GB of GPU memory for p6-b200.48xlarge, which is 179 GB per GPU; NVIDIA's DGX B200 page lists 1,440 GB in total. Single-GPU instances such as g6e.xlarge and inf2.xlarge also include host CPUs and memory in the price, which flatters them slightly against a per-GPU slice of an 8-GPU instance, but that is how you actually rent them.

What this metric hides

Bandwidth per dollar is a first filter, not a verdict. Four things can overturn it.

• Capacity. A chip that cannot hold the model forces you to shard across several chips, and every extra chip adds its price. A TPU v5e has 16 GB, which holds a 10B model at 8 bits with little room for KV cache. An H200 holds a 100B model at FP8 on one GPU; an H100 needs two.
• Interconnect. Sharding a model across chips with tensor parallelism adds communication on every layer. Within an NVLink domain this is cheap; across a slower link it can dominate.
• Compute for prefill. Long prompts make prefill a large share of cost, and prefill is compute bound. The chip with the best bandwidth per dollar may not have the best FLOPs per dollar.
• Software. Inferentia2 runs models compiled with the AWS Neuron SDK; TPUs run through XLA based stacks such as JAX or PyTorch/XLA; AMD Instinct runs ROCm. Model support, kernel quality and quantization formats differ by stack, and an unsupported model is infinitely expensive.

The peak throughput figures we use later for prefill, all vendor peaks:

For a fuller side by side of these parts, see the AI accelerator explainer and the HBM, NVLink and interconnect deep dive. For how rental prices move, see GPU rental prices in 2026.

Lever 2: batching, continuous batching and PagedAttention

Batching is the lever with the largest effect and the one most often left on the table.

Why batching is so powerful

At batch size 1, each decode step reads every weight to produce one token. At batch size 32, the same read produces 32 tokens. The only extra bytes are each sequence's own KV cache. Here is the model for a 10B model with FP8 weights and an FP8 KV cache on one H100 at its on-demand price of $6.88 per GPU-hour, with an average live context of 2,048 tokens during decode:

Two things stand out. First, going from batch 1 to batch 32 cuts the bound cost by about 23 times while the time per step rises only from 3.03 ms to 4.27 ms, so each user still sees well over 200 tokens per second in principle. Second, the gains shrink as the KV cache grows: at batch 128, KV reads are already larger than weight reads, and doubling the batch no longer halves the cost.

We also checked that the compute side does not bind first. At batch 128, one step needs 2 x 10B x 128 FLOPs, which at the H100's dense FP8 peak of 1,979 TFLOPS takes about 1.29 ms, well under the 8.11 ms memory time. At much larger batches or longer prompts this changes, which is why the scheduler limit on batch size matters.

Static versus continuous batching

Old serving systems formed a batch, ran it until every sequence finished, then formed the next one. Because outputs have different lengths, short sequences sat idle waiting for long ones. Continuous batching (also called iteration-level scheduling) adds and removes sequences at every decode step, so the batch stays full. vLLM's README lists continuous batching of incoming requests, chunked prefill and prefix caching among its core features; check the documentation of any other engine you evaluate for its equivalent.

PagedAttention: making room for a bigger batch

The batch size you can sustain is limited by KV cache memory. Early systems reserved a contiguous block for each request's maximum length. If a request reserves 4,096 tokens and uses 1,024 on average, 75 percent of that reservation is wasted. The vLLM paper (Kwon and colleagues, SOSP 2023) introduced PagedAttention, which stores the KV cache in fixed-size blocks allocated on demand, like pages in an operating system. Waste is then limited to the unfilled part of each sequence's last block; with 16-token blocks that is on average about 8 tokens, under 1 percent of a 1,024-token sequence. The paper reports that vLLM "improves the throughput of popular LLMs by 2-4x with the same level of latency" compared with the systems it tested. That is the authors' result on their workloads, not a guarantee for yours. Our continuous batching and PagedAttention explainer walks through the mechanism for team servers.

How many sequences fit? With a 10B model at FP8 (10 GB of weights) on an 80 GB H100, and 90 percent of the remaining memory given to the KV cache:

Quantizing the KV cache doubles the batch ceiling, which is why it appears again in the quantization section.

The catch: one engine, many tenants

Batching means many users share one process, one GPU and one set of kernels. That raises the stakes of bugs in the engine. In September 2026 vLLM disclosed an integer overflow in a CUDA kernel that could copy one user's response into another's in the same batch; we covered it in the vLLM September 2026 advisories post. In October 2026 vLLM published eight advisories, most of which let a single request crash or starve an engine shared by every tenant; see vLLM's October advisories. The cheaper you make each token by packing more users into one engine, the more one bad request or one kernel bug can cost you. Keep the engine version current and put request limits in front of it, as described in how to limit request parameters on a self-hosted inference server.

Lever 3: quantization (FP8, FP4, NVFP4, MXFP4, INT4 with AWQ and GPTQ)

Quantization stores weights, and sometimes activations and the KV cache, in fewer bits. For inference cost it does two things at once: it cuts the bytes read per decode step, and it can shrink the number of chips needed to hold the model.

The formats

The bit counts with scales come from simple arithmetic: MXFP4 is (32 x 4 + 8) / 32 = 4.25 bits, and NVFP4 is (16 x 4 + 8) / 16 = 4.5 bits, ignoring the single per-tensor scale. An INT4 scheme with one 16-bit scale per group of 128 weights works out to 4.125 bits. NVIDIA's NVFP4 article states the format cuts memory footprint by about 3.5 times relative to FP16 and 1.8 times relative to FP8, which matches 16 / 4.5 and 8 / 4.5.

Weight memory by model size:

Worked example: a 100B model at three precisions

Batch 32, average live context 2,048 tokens, KV cache at FP8, replica sized to the smallest power of two GPUs that holds weights plus a 4,096-token KV cache with 10 percent headroom:

On H200, FP16 to FP8 cuts the bound cost by about 47 percent, mainly because the model fits on one GPU instead of two. FP8 to 4-bit cuts it by about 40 percent more, because each step reads fewer bytes. Two cautions. Hopper has no FP4 tensor cores, so on H200 the 4-bit row assumes a weight-only kernel that dequantizes on the fly; the bandwidth saving is real but compute runs at higher precision. And at this batch size the cheaper H200 price beats the faster B200 per token in our bound, which flips if you push B200 to a larger batch or use its FP4 compute for prefill heavy work.

What quantization does to quality

The cost side is arithmetic. The quality side must be cited or measured. What the primary sources report:

• LLM.int8() (Dettmers and colleagues, 2022) reported 8-bit inference for models up to 175B parameters "without any performance degradation", using a mixed-precision split that keeps outlier features in 16-bit.
• SmoothQuant (Xiao and colleagues, 2022) reported W8A8 INT8 quantization with "negligible loss in accuracy", up to 1.56 times speedup and 2 times memory reduction.
• FP8 Formats for Deep Learning (Micikevicius and colleagues, 2022) reported FP8 effectively matching 16-bit result quality across the models it tested, and FP8 post-training quantization of language models that resisted INT8.
• GPTQ (Frantar and colleagues, 2022) reported quantizing 175B parameter models to 3 or 4 bits per weight "with negligible accuracy degradation", with end-to-end speedups of about 3.25 times on A100 and 4.5 times on A6000 versus FP16.
• AWQ (Lin and colleagues, 2023) reported that protecting about 1 percent of salient weights, found from activation statistics, greatly reduces 4-bit quantization error, and that its TinyChat runtime gave more than 3 times speedup over a Hugging Face FP16 implementation.
• NVFP4 (NVIDIA technical blog) reported 1 percent or less accuracy degradation versus FP8 on key language modeling tasks for DeepSeek-R1-0528 quantized with post-training quantization.

Read those claims carefully. Each is the authors' own result on their models and benchmarks. Code generation, long-context retrieval, tool calling and non-English text can degrade more than headline averages. The only safe practice is to run your own evaluation set at each precision and accept the cheapest one that passes. Our 4-bit quantization explainer and quantization and self-hosted cost posts describe a simple test plan.

Quantize the KV cache too

The KV cache grows with batch size and context, and at large batches it is read more than the weights. vLLM exposes --kv-cache-dtype with choices that include fp8 and, in recent versions, nvfp4. As shown above, FP8 KV doubles the number of sequences that fit. Its quality impact needs the same testing as weight quantization, and calibrated scales generally behave better than uncalibrated ones; vLLM's quantized KV cache documentation recommends calibration using a dataset.

A supply chain note on quantized models

A quantized checkpoint is a new artifact: someone ran a tool over the original weights and published the result, often under a different account. Check where it came from, prefer formats that do not execute code on load (safetensors rather than pickle based files), and pin the exact revision. Our ML supply chain glossary and base model, adapter and quantized model metadata posts cover provenance for derived models. In vLLM specifically, model loading has been a repeated source of advisories: CVE-2026-22807 allowed code execution at model load in vLLM 0.10.1 to before 0.14.0, CVE-2025-66448 showed the auto_map config field could lead to remote code execution even with trust_remote_code=False, and CVE-2025-62164 covered unsafe tensor deserialization in the prompt embeddings path, fixed in 0.11.0. Swapping in a cheaper checkpoint from an unknown publisher is a cost decision with a security side.

Lever 4: prefix caching and prompt caching

Many production prompts share a long prefix: a system prompt, tool definitions, a policy document, few-shot examples, or the earlier turns of a conversation. Without caching, every request recomputes the KV cache for that prefix during prefill.

How engines do it

vLLM's documentation describes Automatic Prefix Caching as caching "the KV cache of existing queries, so that a new query can directly reuse the KV cache if it shares the same prefix", which lets the new query skip computing the shared part. It is controlled with --enable-prefix-caching and --disable-prefix-caching. SGLang's paper (Zheng and colleagues, 2023) introduced RadixAttention, which keeps cached prefixes in a radix tree so many requests can share branches of it; the authors report up to 6.4 times higher throughput than the systems they compared against on workloads such as agent control, few-shot benchmarks, JSON decoding, retrieval pipelines and multi-turn chat.

Worked example: a self-hosted 10B model

Assume each request has a 2,000-token shared system prompt, 500 unique input tokens and 300 output tokens. Prefill is modelled at 40 percent of the H100's dense FP8 peak and decode uses the batch 32 planning figure from above, both at 60 percent utilisation:

Prefill work falls by 80 percent and the cost per request by about 35 percent. The saving grows as the shared prefix gets longer relative to the unique part, which is exactly the shape of agent workloads with large tool lists.

Hosted API prompt caching

The same idea appears in hosted API pricing. Anthropic's prompt caching documentation prices 5-minute cache writes at 1.25 times the base input token price and cache reads at 0.1 times the base input price for most models, with lower read multipliers listed for some newer models. OpenAI's prompt caching guide says caching is enabled by default for supported models and that reused tokens are billed at a reduced cached-input rate, "discounted up to 95%". Using an illustrative base input price of $3 per million tokens and Anthropic's 1.25x and 0.1x multipliers, a 2,000-token prefix:

The first row is the trap: caching a prefix that is used once costs more than not caching it. Put stable content first and variable content last so the shared prefix is as long as possible.

The security side of caching

A shared prefix cache is shared state across users, so where prompts are sensitive you need isolation between tenants, and the controls for that are themselves attack surface. vLLM's cache_salt request field was the subject of CVE-2026-100647, where unauthenticated requests with very large cache_salt values could stall vLLM before 0.29.0; the fix is to upgrade and cap request body size at your proxy. Treat cache isolation and request size limits as part of the caching design, not an afterthought.

Lever 5: mixture of experts sparsity

A mixture of experts (MoE) model replaces each dense feed-forward block with many expert blocks and a router that picks a few per token. Only the selected experts run, so the active parameters per token are a small fraction of the total. Primary examples:

Why MoE is cheap at low batch

At batch 1, a decode step only reads the experts the router selected, plus the shared parts. For our 1T MoE model with 32B active, that is about 32 GB at FP8 per step instead of 1,000 GB for a 1T dense model. Prefill compute follows active parameters too, roughly 2 x 32B FLOPs per token instead of 2 x 1T.

Why the advantage narrows as batch size grows

Different tokens choose different experts. With a batch of B tokens, each choosing 8 of 384 routed experts, the expected fraction of experts touched in a step is 1 minus (1 minus 8/384) to the power B, assuming independent and uniform routing. Real routers are not uniform, so treat this as a rough guide. To turn it into bytes we split the 1T model into a shared part and a routed part that satisfy both the total (1T) and the active (32B) counts: about 11.4B shared parameters (attention, embeddings, dense layer and shared expert) and about 20.6B routed parameters per token, or about 2.57B per expert across all layers. That split is our assumption, derived from the published totals, not a published breakdown.

On 8 H200 GPUs (38.4 TB/s aggregate, $63.296 per hour for p5en.48xlarge), FP8 weights, MLA KV cache at FP16, average live context 2,048 tokens:

Compare a 1T dense model on the same 8 H200s: at batch 1 the bound is 38 tokens per second and $458 per million tokens; at batch 32 it is 1,203 tokens per second and $14.62. At batch 32 the MoE model is about 51 percent cheaper; at batch 1 it is about 97 percent cheaper. Once nearly every expert is touched each step, the MoE model reads almost as many bytes as a dense model of the same total size, and its remaining advantage comes from the smaller KV cache (MLA) and from lower compute per token.

Two practical consequences follow. MoE models need enough memory to hold every expert even though few run per token: the 1T MoE at FP8 still needs about 1,000 GB, and at batch 256 our model needs 1,074 GB of the 1,128 GB on eight H200s, which leaves almost no headroom. And very large MoE deployments often spread experts across many GPUs (expert parallelism) so each GPU holds and reads a subset, which turns the problem into one of network bandwidth and load balancing. Check whether your serving engine supports expert parallelism, and how it is configured, in the docs for the version you run.

Lever 6: distillation to smaller models

Distillation trains a smaller student model to imitate a larger teacher. Hinton, Vinyals and Dean's 2015 paper "Distilling the Knowledge in a Neural Network" set out the modern approach. DistilBERT (Sanh and colleagues, 2019) is a well cited example: the authors reduced the size of BERT by 40 percent while retaining 97 percent of its language understanding capability and running 60 percent faster.

For cost, the arithmetic is direct. In our bound model at batch 32:

If a distilled 10B model passes your evaluation for a given task, it serves that task for roughly a quarter to a sixth of the cost of a 100B model in this model. The caveats are real. Distillation needs training compute and a good dataset; a student usually loses some general capability; and licence terms on the teacher's outputs may restrict training on them. A common pattern is routing: send easy, high-volume requests to a small distilled model and escalate hard ones to a large model. Our teacher and student distillation explainer covers the method, and small model distillation for security workflows gives a domain example.

Lever 7: speculative decoding as a cost lever

Speculative decoding uses a cheap draft (a small model, extra heads, or n-gram lookup) to propose several tokens, then has the large target model verify them all in one forward pass. Leviathan, Kalman and Matias (2022) showed that with their sampling method the output distribution is unchanged. If each drafted token is accepted with probability alpha, and the draft proposes gamma tokens, the expected number of tokens produced per target pass is (1 minus alpha to the power gamma plus 1) divided by (1 minus alpha). If a draft step costs c times a target step, the expected speedup is that number divided by (gamma x c plus 1).

Published results fall in the same range. Leviathan and colleagues report 2 to 3 times acceleration on T5-XXL with identical outputs. Chen and colleagues (2023) report 2 to 2.5 times decoding speedup on the 70B Chinchilla model. Medusa (Cai and colleagues, 2024) reports over 2.2 times for Medusa-1 and 2.3 to 3.6 times for Medusa-2. EAGLE (Li and colleagues, 2024) reports a 2.7 to 3.5 times latency speedup on LLaMA2-Chat 70B and doubled throughput.

When it saves money and when it does not

Speculative decoding converts spare compute into fewer memory-bound steps. At batch 1, where a GPU is mostly waiting on memory, that spare compute is free, so fewer steps per token means proportionally lower cost per token for a dedicated replica. At large batch sizes the target model's verification pass processes gamma plus 1 tokens for every sequence, which adds compute to a step that is already busier, and rejected draft tokens are wasted work. The gain in tokens per GPU-hour therefore shrinks as batch size grows, and can disappear. Treat speculative decoding primarily as a latency tool that also lowers cost for low-concurrency or latency-bound deployments, and measure it at your real batch size. Our speculative decoding explainer for local inference shows the effect on a single machine. vLLM's README lists n-gram, suffix, EAGLE and DFlash methods, configured with --speculative-config.

Lever 8: spot and preemptible capacity

Cloud providers sell spare capacity at a discount with the right to take it back.

• AWS describes EC2 Spot Instances as up to 90 percent off on-demand prices, and its documentation says an interruption notice is issued two minutes before EC2 stops or terminates a Spot Instance.
• Google Cloud describes Spot VMs as offering discounts of up to 91 percent off on-demand prices for many machine types, GPUs, TPUs and Local SSDs, notes that spot prices can change up to once a day, and says Compute Engine might preempt Spot VMs at any time.
• In the Azure retail prices API on 25 October 2026, the Standard_ND96isr_H100_v5 (8 H100 GPUs) in East US listed $98.32 per hour on demand and $18.169536 per hour as Spot for Linux, an 81.5 percent discount at that moment.

Interruptions cost something: in-flight requests must be retried, and a new replica must load weights and warm up. If a share of paid hours is lost to interruptions and restarts, the effective price rises. Using the Azure figures:

Even with heavy losses the discount survives on paper. The real constraints are elsewhere. Large multi-GPU instances are often the hardest to get as spot capacity. Latency-sensitive interactive traffic cannot simply pause. And a 1T model that takes many minutes to load can lose most of a short spot window to startup. Spot fits batch and offline inference (evaluations, embeddings, document processing), overflow capacity behind an on-demand base, and small models that reload quickly. Note also the price spread between clouds: the same 8 H100 configuration was $55.04 per hour on AWS (p5.48xlarge) and $98.32 on Azure (ND96isr H100 v5) on demand on the date we checked, which is a bigger difference than many of the software levers.

Lever 9: CPU or edge inference for 1B models

A 1B model at 4-bit is about 0.5 to 0.6 GB of weights. That fits easily in system RAM on a server CPU or on a laptop, phone or edge box, and avoids paying for an accelerator at all. llama.cpp's README describes its goal as LLM inference "with minimal setup and state-of-the-art performance on a wide range of hardware", with a plain C/C++ implementation, Apple silicon support through ARM NEON, Accelerate and Metal, and AVX, AVX2, AVX512 and AMX support on x86.

We do not publish a CPU throughput figure here, because it depends heavily on the processor, memory channels, thread count and quantization, and we did not measure one. What we can do is show what a measured number would mean in cost. Using the on-demand price of an AWS c7i.4xlarge (16 vCPU, 32 GiB) of $0.714 per hour, for hypothetical single-stream throughputs:

Compare those with the accelerator bound for the same 1B model at batch 32: $0.072 on an L4, $0.057 on an L40S and $0.023 on Inferentia2. At high, steady volume, a small accelerator kept busy beats a CPU per token. CPU and edge inference wins when volume is low or bursty (an accelerator that sits idle costs the same per hour), when data must stay on the device, or when the device already exists and its marginal cost is close to zero. To get your own number, run llama.cpp's benchmark tool on the target machine, as in the commands below; we did not run them for this post.

# Build llama.cpp and benchmark a 1B GGUF model on this machine (not executed for this post).
git clone https://github.com/ggml-org/llama.cpp
cmake -S llama.cpp -B llama.cpp/build -DCMAKE_BUILD_TYPE=Release
cmake --build llama.cpp/build -j
# Reports prompt processing (pp) and token generation (tg) speeds.
./llama.cpp/build/bin/llama-bench -m ./models/your-1b-model-Q4_K_M.gguf -p 512 -n 128

Check the README for your version: recent releases also ship a combined llama command with llama serve and llama cli subcommands, alongside llama-server. The site has a runtime comparison of Ollama, llama.cpp and vLLM and a post on CPU and GPU offload in llama.cpp.

Lever 10: utilisation and autoscaling

Every number so far assumed the hardware is busy. You pay for an idle GPU by the hour all the same, so cost per token scales with one divided by utilisation:

Going from 20 percent to 60 percent average utilisation cuts the cost per token by three times, with no change to the model or the chip. Ways to get there:

• Autoscale on the right signal. Request rate and GPU utilisation percentage are both misleading for LLM servers. Queue depth, number of running and waiting sequences, KV cache usage and time to first token track saturation more directly; check which of these your engine exports as metrics.
• Account for cold start. Scaling from zero means loading weights, which takes longer for larger models. Keep a warm floor for interactive traffic and scale above it.
• Mix workloads. Fill the troughs of interactive traffic with batch jobs (evaluations, summarisation backlogs, embeddings) at lower priority.
• Share one engine across models carefully. Multi-LoRA serving lets one base model serve many fine-tuned adapters from one replica, which raises utilisation compared with one replica per fine-tune.
• Buy commitment for the base, burst for the peak. Google's TPU price page lists 1-year and 3-year commitment prices well below on-demand (for example $1.89 and $1.22 per chip-hour for Trillium in us-east1 against $2.70 on demand), and AWS's Inferentia2 page lists $0.45 and $0.30 for 1-year and 3-year reserved inf2.xlarge against $0.76 on demand. Commitments only save money if utilisation stays high.

Putting it together: worked costs for 1B, 10B, 100B and 1T

Decode cost by model size

Batch 1 and batch 32, average live context 2,048 tokens, KV cache at 8 bits for the dense models and FP16 MLA for the MoE model, on-demand prices from 25 October 2026. "Planning" applies 50 percent of the bound and 60 percent utilisation.

The multi-GPU rows assume the aggregate bandwidth of all GPUs in the replica is usable, which ignores tensor parallel communication; real figures will be worse. The 1T dense rows are included because the brief asks for them, but nobody should serve a 1T dense model at batch 1.

Prefill cost by model size

Prefill is compute bound, so we price it from dense FP8 peak at an assumed 30 to 50 percent of peak, counting 2 x active parameters FLOPs per prompt token and ignoring attention FLOPs (which add roughly 10 percent at 4,096 tokens of context for the 10B shape, more at longer context):

All figures are dollars per million input tokens at full utilisation. Prefill is much cheaper per token than decode for every size, which is why hosted APIs typically price input tokens below output tokens, but long prompts can still make it the larger share of a request's cost.

A ranked playbook with expected savings

The savings below are either computed in this guide from stated assumptions or quoted from the cited papers. They compound only partly, because several levers act on the same bottleneck.

What to do first at each size

1B. Do not rent a large GPU. Use the smallest accelerator that holds the model with room for a large batch (Inferentia2, L4, TPU v5e), or a CPU or edge device if volume is low or bursty. Quantization matters less for capacity than for fitting on a smaller instance. Utilisation is the lever that matters most because the hardware is cheap per hour but easily idle.

10B. One mid-range accelerator per replica. Turn on continuous batching, FP8 weights and FP8 KV cache, and prefix caching. Add speculative decoding if latency targets are tight and batch sizes are small. Watch utilisation.

100B. Quantize until the model fits on one GPU (FP8 on H200 or B200, 4-bit on H100 class), because removing tensor parallelism saves both the extra GPU and the communication. Cache prefixes aggressively. Ask whether a distilled 10B model, or routing between 10B and 100B, would pass your evaluation.

1T. Serve a mixture of experts model rather than a dense one if one exists that meets your quality bar. Use FP4 formats on hardware that runs them natively, push batch size high so expert reads are amortised, and consider disaggregated prefill and decode on separate pools so each can be sized to its bottleneck. Spot capacity rarely suits interactive 1T serving, but batch jobs can use it.

Serving stacks from their documentation

The levers above are implemented in serving engines. What each project says about itself, from its own README or docs as fetched on 25 October 2026:

A cost-tuned vLLM launch

The command below shows where the levers in this guide become flags. Flag names come from vLLM's engine arguments documentation; we could not run it on this Mac, and you should check every flag against the docs for the version you install.

# Cost-oriented vLLM launch for a 10B-class model on one GPU (not executed for this post).
vllm serve your-org/your-10b-model-fp8 \
  --quantization fp8 \
  --kv-cache-dtype fp8 \
  --enable-prefix-caching \
  --max-model-len 8192 \
  --max-num-seqs 128 \
  --gpu-memory-utilization 0.90 \
  --speculative-config '{"method": "ngram", "num_speculative_tokens": 4, "prompt_lookup_max": 4}'

What each line does for cost: FP8 weights halve bytes per step; the FP8 KV cache doubles the sequences that fit; prefix caching skips repeated prefill; a sensible --max-model-len stops one long context from starving the batch; --max-num-seqs sets the batch ceiling; --gpu-memory-utilization gives the KV cache as much memory as is safe; and n-gram speculation lowers step count for repetitive outputs without a separate draft model. Remove the speculative line if your measured throughput at production batch size gets worse with it.

A small calculator you can run anywhere

This python3 script reproduces the decode bound used in this guide. It needs no GPU. Change the inputs to your model, price and measured efficiency.

# Decode-bound cost per million tokens (python3, standard library only).
def kv_bytes_per_token(layers, kv_heads, head_dim, bytes_per_elem):
    return 2 * layers * kv_heads * head_dim * bytes_per_elem

def decode_cost(params, bytes_per_weight, kv_per_tok, avg_ctx, batch,
                bandwidth_bytes_s, price_per_hr, efficiency=1.0, utilisation=1.0):
    bytes_per_step = params * bytes_per_weight + batch * avg_ctx * kv_per_tok
    step_s = bytes_per_step / bandwidth_bytes_s
    tok_s = batch / step_s * efficiency
    return price_per_hr / (tok_s * utilisation * 3600) * 1e6, tok_s

kv = kv_bytes_per_token(32, 8, 128, 1)        # 10B shape, FP8 KV cache
for b in (1, 8, 32, 64, 128):
    cost, tps = decode_cost(10e9, 1, kv, 2048, b, 3.35e12, 55.04 / 8)
    print(f"batch {b:4d}: {tps:9.0f} tok/s bound, ${cost:.4f} per 1M tokens")

We ran this script with python3 for this post; its batch 32 line prints 7,499 tokens per second and $0.2548 per million tokens, matching the table in the batching section.

Security posture of the serving stack

A guide about cost belongs on a security site because every lever above runs inside software that has had serious advisories. The pattern is consistent across engines: features that improve cost (shared batches, shared caches, multi-node and disaggregated serving, flexible model loading) widen what one request or one network peer can reach.

• vLLM. Beyond the batching, caching and model loading advisories linked above, CVE-2024-9053 was a critical unsafe deserialization flaw in vLLM 0.6.0's RPC server, and CVE-2025-32444 affected vLLM 0.6.5 to before 0.8.5 with the Mooncake integration, which deserialized pickle over open ZeroMQ sockets. As our post on multi-node vLLM network isolation explains, vLLM's documentation says multi-node traffic is insecure by default, and the post shows how to isolate it. The vLLM API key does not protect every endpoint explains a common misconfiguration.
• SGLang. CERT/CC published VU#281278 on 30 July 2026 covering six SGLang vulnerabilities, including unauthenticated remote code execution; see SGLang's six CVEs.
• TensorRT-LLM. NVD published CVE-2026-47472 on 14 July 2026, an unsafe deserialization issue in its IPC channel.
• llama.cpp. Ten flaws were disclosed on 7 August 2026, including CVE-2026-43631, a use-after-free in llama-server.
• Across engines. The ShadowMQ pattern showed the same unsafe messaging code copied between inference engines in 2025.

None of this argues against cheap inference. It argues for treating the engine like any other internet-facing server: pin versions, read its advisories, keep it off untrusted networks, scan the container image, and know which features you have turned on. Is an inference server CVE exploitable in my deployment? gives five questions for triage, and how to scan a GPU inference container image covers the image side.

Common mistakes

1. Comparing chips on peak FLOPs for a decode-heavy workload. Output tokens are mostly bandwidth bound. Compare GB/s per dollar first, then check capacity and software.
2. Quoting the bound as the price. The bandwidth bound is a ceiling on throughput. Real servers reach a fraction of it. Plan with a measured efficiency, and until you have one, use a conservative factor and say so.
3. Cheap tokens that break the latency SLO. Raising batch size lowers cost and raises time per output token and time to first token. A batch that saves 30 percent but pushes the 95th percentile time to first token past your target is not a saving. Set the SLO first, then find the cheapest configuration that meets it.
4. Quantizing without an evaluation set. Paper results are averages on the authors' benchmarks. Code, long context, tool calls and non-English text can degrade more. Test each precision on your own tasks.
5. Forgetting the KV cache. A model that "fits" in memory at batch 1 may leave no room for a useful batch. Size memory for weights plus the KV cache at your target batch and context.
6. Caching prefixes that are not shared, or ordering prompts so nothing is shared. Put stable content first. On APIs, a prefix used once costs more with caching than without.
7. Measuring utilisation as GPU percentage. A GPU can show high utilisation while serving a batch of one. Track tokens per second per GPU, batch occupancy and KV cache usage.
8. Scaling to zero for interactive traffic with a large model. Cold starts include loading tens or hundreds of gigabytes of weights. Keep a warm floor.
9. Running spot capacity without draining. Use the interruption notice to stop admitting new requests and finish or hand off in-flight ones.
10. Treating the serving engine as plumbing. It parses untrusted input from every user and often loads model files from third parties. Patch it on the same schedule as any other exposed service.

FAQ

What is the cheapest way to run LLM inference?

Keep a bandwidth-efficient accelerator busy with large continuous batches, use the smallest model that passes your evaluation, quantize weights and KV cache to the fewest bits your tests allow, and cache shared prompt prefixes. In our model, batching alone cut cost about 23 times for a 10B model on one H100.

How do you calculate cost per million tokens for a self-hosted model?

Divide the hourly price of the hardware by the tokens per second it delivers, divide by 3,600, divide by utilisation, and multiply by one million. For decode, an upper bound on tokens per second is batch size divided by (bytes read per step divided by memory bandwidth).

Which GPU is cheapest for LLM inference?

It depends on model size and software. On 25 October 2026 on-demand prices, Inferentia2, TPU v5e, H200 and TPU v6e led on memory bandwidth per dollar in our table, but capacity, interconnect and framework support can overturn that. A chip that cannot hold your model or run your format is not cheaper.

Does quantization reduce inference cost?

Yes, in two ways: fewer bytes per weight means fewer bytes read per token, and a smaller model may fit on fewer GPUs. For a 100B model on H200, FP16 to FP8 cut our bound cost by about 47 percent and 4-bit cut it by about 40 percent more. Quality impact must be tested on your workload.

Is FP4 good enough for production?

NVIDIA reports 1 percent or less accuracy degradation from FP8 to NVFP4 on key tasks for DeepSeek-R1-0528, and AMD's MI355X runs MXFP4 natively. Those are vendor results on specific models. Run your own evaluation before switching.

How much does batching reduce LLM serving cost?

In our model, a 10B FP8 model on one H100 went from about $5.78 per million tokens at batch 1 to about $0.25 at batch 32 at the bandwidth bound. The vLLM paper reports 2 to 4 times higher throughput than the systems it compared against at the same latency.

Are mixture of experts models cheaper to serve?

Per token, yes, especially at small batch sizes, because each token reads only its selected experts. As batch size grows more experts are touched per step, so the advantage over a dense model of the same total size narrows. MoE models still need memory for every expert.

Does speculative decoding lower cost or only latency?

Both, at low batch sizes, because fewer target steps per token means fewer GPU-seconds per token. At large batch sizes the extra verification compute and rejected drafts can erase the cost benefit. Measure at your production batch size.

Is it cheaper to run a 1B model on a CPU?

At low or bursty volume, often yes, because an idle accelerator still costs money every hour. At high steady volume, a small accelerator kept busy is cheaper per token. Benchmark your CPU with llama.cpp's tool and plug the number into the cost equation.

Should I use spot instances for LLM inference?

For batch jobs, overflow capacity and small fast-loading models, yes; discounts of up to 90 percent (AWS) and 91 percent (Google Cloud) are listed. For interactive serving of large models, interruptions and long reload times usually make an on-demand or committed base the safer choice.

Where Safeguard fits

Safeguard does not design, program or speed up GPUs or other accelerators, and it does not lower your GPU bill. Nothing in the cost model above depends on it. The honest connection is the software stack every one of these levers runs on: the GPU driver and runtime, CUDA or ROCm libraries, a framework build such as PyTorch, a serving engine such as vLLM, SGLang, TensorRT-LLM or llama.cpp, dozens of Python packages, and a container base image. Each is a versioned component, and as the security section shows, the serving engines in particular have a steady stream of advisories.

Safeguard generates and analyses SBOMs in SPDX and CycloneDX, scans repositories, container images and SBOM files (for example safeguard scan --image <name> for a serving image), matches components against vulnerability databases, shows which projects contain a given component and version, applies reachability analysis, and enforces policies and gates. That covers the open source engines, Python packages and container layers in an inference deployment. It does not cover GPU firmware, a host kernel driver installed outside your images, or closed vendor binaries you cannot inventory; those updates still come from the hardware vendor and your OS distribution.

Cost tuning tends to mean frequent engine upgrades to pick up new kernels, quantization formats and caching features, and each upgrade pulls in a new set of Python and system dependencies. Those dependencies, and the serving engine itself, are open source components that need the same inventory and patch discipline as any other production service.

More in this series

Other posts in our AI chips and costs series:

• AI Accelerator TCO: Buy vs Rent, Capex, Power, Cooling and Utilization
• AI Chips Compared 2026: B200, H200, MI355X, Gaudi 3, TPU, Trainium
• Fastest LLM Inference: Lowest Latency and Most Tokens per Second
• How Many GPUs Does a 1 Trillion Parameter Model Need?
• How to Estimate AI Training and Inference Cost: Formulas and a Calculator
• LLM Inference Cost by Chip and Model Size, 1B to 1 Trillion Parameters
• LLM Training Cost by Chip and Model Size: 1B to 1T, Worked

Fix it without waiting for upstream

Whatever the vulnerability, most of the risk sits in two gaps. The first is the time between a fix being published and that fix actually running in your production builds. The second, and worse, is the time before any fix exists at all, while a maintainer works out a patch and a release schedule. Waiting through both gaps is a habit, not a requirement. Safeguard gives teams three ways to close them.

1. Autonomous remediation: pull requests, not tickets

When a fixed version exists, Safeguard's Griffin models open a tested, policy gated pull request that moves the affected dependency to the safe version. You can run it on one repository or in bulk across every project that carries the vulnerable version. Remediation covers eight categories, including dependencies, licenses, security posture and findings. Reachability analysis ranks the vulnerable code paths your application actually calls first, so the first pull requests you merge are the ones that matter most. Autonomous remediation ships switched off, with every category set to Manual, so your team decides how much runs on its own and what still needs a human approval.

2. Gold Open Source: the safe version in seconds, free

gold.safeguard.sh is free and needs no login. Search any CVE or package to see affected and fixed versions, CVSS, EPSS, CISA KEV status and a recommended safe version, or use the "Am I affected?" checker. The same data is available through a free read API, README status badges and RSS threat feeds. The safeguard-sh/gold-scan-action GitHub Action can fail a build on a known exploited vulnerability, on any known vulnerability, or never, so a version you have just removed cannot quietly come back in next week's merge.

3. Gold Registry: start from components that are already fixed

registry.safeguard.sh provides hardened, drop in replacements for upstream packages and container images. Every Gold artifact is held continuously to zero exploitable critical or high CVEs, zero malware and a completed license check, with SLSA L3 or higher provenance, a Sigstore signature and a public evidence page you can hand to an auditor. An artifact loses its Gold badge if it regresses. Teams can point their own package directory at the registry so hardened equivalents resolve without changing how developers install anything.

When upstream has no fix yet, or will not ship one, Safeguard first works to get the fix accepted upstream. If the issue is severe or upstream will not move, Safeguard builds a fork, fixes the vulnerability there and publishes a hardened Gold build, so you are not stuck on a maintainer's release schedule. You can also request a hardened, compatibility tested build of a specific package. This fork and patch flow is in early access today, so ask which packages it covers before you plan around it.

One honest limit: when the vulnerable code lives in a vendor appliance, a hosted service or an operating system runtime you do not build yourself, the fix still has to come from that vendor or distribution. The steps above apply to the open source dependencies and container images you build and ship, which is where most teams carry the largest and least tracked share of their exposure.

See how Safeguard would help

The first demo is a real Safeguard CLI session. It opens the interactive terminal, reads a dependency finding and its advisory, asks before any command that changes something, edits the manifest to a fixed version and runs the install and tests with your approval. The same flow applies to a requirements file in a model serving repository that pins a vulnerable vLLM or other engine version.

Click through it at your own pace, or open it full screen.

The second demo sets up the Safeguard runner on your own machine, approves it with a device code, and scans from the terminal. That suits a GPU build host where you want to scan a serving container image locally before it is pushed.

Click through it at your own pace, or open it full screen.

The third demo connects Claude to the Safeguard MCP server with OAuth and lets you choose which tools Claude can use. It is a quick way to ask which of your projects carry a given serving engine version without leaving a chat.

Click through it at your own pace, or open it full screen.

To try it on your own stack, download Safeguard or read the CLI documentation.

Quick check

Sources

• Kwon and colleagues, "Efficient Memory Management for Large Language Model Serving with PagedAttention" (arXiv 2309.06180, SOSP 2023)
• Zheng and colleagues, "SGLang: Efficient Execution of Structured Language Model Programs" (arXiv 2312.07104)
• Frantar and colleagues, "GPTQ: Accurate Post-Training Quantization for Generative Pre-trained Transformers" (arXiv 2210.17323)
• Lin and colleagues, "AWQ: Activation-aware Weight Quantization for LLM Compression and Acceleration" (arXiv 2306.00978)
• Xiao and colleagues, "SmoothQuant" (arXiv 2211.10438)
• Dettmers and colleagues, "LLM.int8(): 8-bit Matrix Multiplication for Transformers at Scale" (arXiv 2208.07339)
• Micikevicius and colleagues, "FP8 Formats for Deep Learning" (arXiv 2209.05433)
• Open Compute Project, OCP Microscaling Formats (MX) Specification v1.0
• NVIDIA Technical Blog, "Introducing NVFP4 for Efficient and Accurate Low-Precision Inference"
• Leviathan, Kalman and Matias, "Fast Inference from Transformers via Speculative Decoding" (arXiv 2211.17192)
• Chen and colleagues, "Accelerating Large Language Model Decoding with Speculative Sampling" (arXiv 2302.01318)
• Cai and colleagues, "Medusa: Simple LLM Inference Acceleration Framework with Multiple Decoding Heads" (arXiv 2401.10774)
• Li and colleagues, "EAGLE: Speculative Sampling Requires Rethinking Feature Uncertainty" (arXiv 2401.15077)
• Jiang and colleagues, "Mixtral of Experts" (arXiv 2401.04088)
• DeepSeek-AI, "DeepSeek-V3 Technical Report" (arXiv 2412.19437)
• Moonshot AI, Kimi K2 model card and README
• Llama Team, Meta, "The Llama 3 Herd of Models" (arXiv 2407.21783), Table 3
• Hinton, Vinyals and Dean, "Distilling the Knowledge in a Neural Network" (arXiv 1503.02531)
• Sanh and colleagues, "DistilBERT" (arXiv 1910.01108)
• NVIDIA H100 Tensor Core GPU specifications
• NVIDIA H200 Tensor Core GPU specifications
• NVIDIA DGX B200 specifications
• NVIDIA L40S specifications
• NVIDIA L4 specifications
• AMD Instinct MI300X specifications
• AMD Instinct MI355X specifications
• Google Cloud TPU v5e documentation
• Google Cloud TPU v6e (Trillium) documentation
• Google Cloud TPU pricing (checked 25 October 2026)
• AWS Neuron documentation, Inferentia2 architecture
• AWS EC2 P5 instances
• AWS EC2 P6 instances
• AWS EC2 G6e instances
• AWS EC2 Inf2 instances and pricing
• AWS EC2 on-demand price file, US East (N. Virginia), Linux, published 23 October 2026
• Azure Retail Prices API (queried 25 October 2026)
• AWS EC2 Spot Instances
• AWS documentation, Spot Instance interruption notices
• Google Cloud documentation, Spot VMs
• Anthropic documentation, prompt caching
• OpenAI documentation, prompt caching
• vLLM README
• vLLM documentation, Automatic Prefix Caching
• vLLM documentation, engine arguments
• vLLM documentation, quantized KV cache
• SGLang README
• TensorRT-LLM README
• llama.cpp README
• Text Generation Inference README

Last reviewed: 25 October 2026. Cloud prices and serving engine features change often; check the linked price pages and project documentation before relying on a specific figure.