How to Estimate AI Training and Inference Cost: Formulas and a Calculator

To estimate training cost, multiply about 6 FLOPs by the active parameter count and the training tokens, divide by the chip's dense peak FLOP rate times a realistic model FLOPs utilization (MFU) of 35 to 55 percent, and multiply the resulting chip-hours by the hourly price. To estimate inference cost, size the weights and KV cache, bound decode speed by memory bandwidth, and divide the hourly price by tokens actually served.

That is the whole method in two sentences. The rest of this post is the detail that makes those two sentences produce a number you can defend: where the 6 comes from, which spec sheet figure to divide by (and which one to ignore), how much memory a KV cache really takes, why a mixture of experts model is cheap to train but not automatically cheap to serve, and how idle time quietly doubles your cost per token. At the end there is a complete Python calculator, run on this machine, with its output for 1B, 10B, 100B and 1T parameter models on three chips: the NVIDIA H100 SXM, the AMD Instinct MI300X and the Google Cloud TPU v6e (Trillium).

Every chip figure below is a vendor peak figure taken from the vendor's own page or datasheet, and every price is a public list price checked on 25 October 2026. Every cost is an estimate built from stated assumptions. None of it is a benchmark result, and nothing here was measured on real hardware.

Key takeaways

• Training FLOPs are about 6·N·D, where N is the number of parameters used per token and D is the number of training tokens. The rule comes from Kaplan et al. (2020) and checks out against Meta's published Llama 3 405B compute to within about 1 percent.
• Divide by the dense peak, never the sparse one. NVIDIA lists H100 tensor throughput "with sparsity", which is twice the dense rate. Dividing by the sparse figure halves your estimate.
• Published MFU for large training runs sits between about 38 and 55 percent (Llama 3, PaLM, Megatron, MegaScale). Use a range, not a single number, and add a goodput factor for restarts.
• Decode is memory bound. At small batch sizes each generated token requires reading every weight once, so tokens per second per replica cannot exceed memory bandwidth divided by bytes read per step.
• The KV cache is not a rounding error. For an assumed 100B model at an 8,192 token context and batch 32, the KV cache in BF16 is about 94.5 GB, almost as large as the 8-bit weights.
• Mixture of experts cuts training compute by the ratio of active to total parameters, but serving still has to hold every expert in memory, and at moderate batch sizes a decode step touches a large share of them.
• Utilization is a cost multiplier. A replica that is busy half the time costs twice as much per token as the same replica at full load.

The whole method on one page

Before any formula, here is the map. Training and inference share inputs but use them differently: training is compute bound and priced in chip-hours, while serving splits into a compute bound prefill phase and a memory bound decode phase.

                 MODEL                                  CHIP
   N_total, N_active, layers, KV heads,      peak dense FLOP/s (BF16, FP8/INT8)
   head dim, training tokens D               HBM GB, HBM TB/s, watts, $/chip-hour
            |                                             |
            +----------------------+----------------------+
                                   |
         +-------------------------+--------------------------+
         |                                                    |
     TRAINING                                             INFERENCE
  FLOPs = 6 * N_active * D                   memory = weights + KV cache
  chip-s = FLOPs / (peak * MFU)              chips  = memory / usable HBM
  chip-h = chip-s / 3600 / goodput           prefill: 2 * N_active * prompt FLOPs
  $      = chip-h * $/chip-hour                       / (chips * peak * MFU)
  kWh    = chip-h * watts * PUE / 1000       decode:  tokens/s <= batch * BW
                                                      / bytes read per step
                                             $/M tok = $/h * chips * 1e6
                                                      / (tokens/s * util * 3600)

Each box in that diagram gets its own section below, with the source for the formula and at least one worked number.

Step 1: pin down the inputs

Chip inputs, and why the precision and sparsity labels matter

A chip spec sheet lists several throughput figures, one per number format. The one you divide by must match the format the work actually runs in, and it must be the dense figure. The three chips used throughout this post:

All throughput figures are vendor peak figures. Sources: the NVIDIA H100 product page, the AMD Instinct MI300X data sheet, the Google Cloud TPU v6e documentation, the Oracle Cloud price list (queried through Oracle's public cost estimator API for part numbers B98415 and B109485) and the Google Cloud TPU pricing page.

Three details in that table cause most estimation errors, so they deserve a sentence each.

First, NVIDIA's H100 page puts an asterisk on the TF32, BF16, FP16, FP8 and INT8 tensor rows, and the footnote reads "With sparsity." NVIDIA's Hopper architecture whitepaper explains that these are "Effective TFLOPS / TOPS using the Sparsity feature", which exploits fine grained structured sparsity to double throughput. Ordinary dense training and inference do not get that doubling, so the dense BF16 rate is half of 1,979, or 989.5 TFLOPS.

Second, AMD's data sheet prints both columns side by side, dense and "with sparsity", so you can read the dense figure directly.

Third, Google's v6e page lists BF16 and INT8 figures and no sparsity variant, and does not list an FP8 rate. That means an 8-bit inference estimate on v6e is an INT8 estimate, not FP8. The calculator uses INT8 TOPS for v6e's 8-bit column and says so.

On prices: Oracle's list gives a per GPU hour price for its eight GPU bare metal shapes ($10 for H100, $10 for H200, $14 for B200 and $6 for MI300X on 25 October 2026), and Google lists TPU prices per chip-hour. Prices differ by provider, region and commitment. Google's same page lists Trillium at $1.89 per chip-hour on a one year commitment and $1.22 on three years in us-east1, so the on demand figure is the most expensive way to buy the same chip. For a broader view of rental prices and why they move, see GPU rental prices in 2026.

A note on fairness: comparing a $10 H100 hour with a $2.70 v6e hour is not comparing like with like. The chips have different memory sizes, interconnects and software stacks, and the prices come from different clouds with different discount structures. The calculator lets you swap in whatever price you actually pay, and you should.

Model inputs

The founder's question was about four fixed sizes: 1B, 10B, 100B and 1T parameters, with the 1T case covered both as a dense model and as a mixture of experts. For training compute only the parameter count and token count matter. For inference memory you also need the attention shape, because it sets the KV cache size. These shapes are assumptions chosen to look like published models of similar size, not descriptions of any specific product:

The dense token counts follow the Chinchilla rule of about 20 tokens per parameter, explained below. The MoE row borrows its shape from a published model: Moonshot AI's Kimi K2 technical report describes a model with 1.04 trillion total parameters, 32.6 billion activated, 61 layers, 384 experts with 8 active per token, pre-trained on 15.5 trillion tokens (arXiv 2507.20534). Kimi K2 uses multi-head latent attention rather than the grouped query attention assumed in this table, so its real KV cache size differs from the MoE KV figures here; treat them as illustrative.

The 8 KV heads with a head dimension of 128 match the grouped query attention shape Meta published for Llama 3 8B, 70B and 405B (all three use 8 KV heads; Table 3 of the Llama 3 paper).

Training compute: the 6ND rule

Where the 6 comes from

Kaplan et al., "Scaling Laws for Neural Language Models" (arXiv 2001.08361), derive the forward pass cost of a decoder only transformer as about 2N FLOPs per token plus an attention term of 2·n_layer·n_ctx·d_attn, and estimate training compute as C ≈ 6N FLOPs per training token, "C ≈ 6NBS" in their notation for batch size B and steps S.

The intuition is simple. Each parameter takes part in one multiply and one add per token in the forward pass: 2 FLOPs. The backward pass computes gradients with respect to both the activations and the weights, roughly twice the forward work: 4 FLOPs. That gives 6 FLOPs per parameter per token.

Two simplifications hide in that rule, and you should know them:

1. The attention term is left out. Kaplan et al. note that their compute estimate "did not include contributions proportional to n_ctx". For short contexts on large models the attention FLOPs are small next to 6N. For long context training on a small model they are not, and 6ND will underestimate.
2. Activation recomputation is not counted. If you discard activations in the forward pass and recompute them during the backward pass to save memory, the hardware does more than 6ND FLOPs. That extra work shows up as hardware FLOPs utilization, not model FLOPs utilization. More on that in the MFU section.

How many tokens: the Chinchilla rule

Hoffmann et al. ("Training Compute Optimal Large Language Models", arXiv 2203.15556) found that for a compute optimal model, parameters and training tokens should be "scaled equally": double the model, double the tokens. Their 70B Chinchilla model was trained on 1.4 trillion tokens, which is 20 tokens per parameter, and that ratio has become the shorthand.

It is a shorthand for compute optimal training, not a law of what people actually do. Many modern models are trained far past 20 tokens per parameter because a smaller model trained longer is cheaper to serve. Meta trained Llama 3 405B on 15.6 trillion tokens, about 38.5 tokens per parameter. Kimi K2's 15.5 trillion tokens are about 15 per total parameter but about 475 per active parameter. So treat D as an input you set, not a constant. This post uses 20 for the dense sizes because the brief asks for a stated, citable assumption, and you should replace it with your own plan.

Checking the rule against a published run

The best test of a formula is a published number it did not see. The Llama 3 paper reports that the flagship 405B model was pre-trained on 15.6T tokens with a total of 3.8 × 10^25 FLOPs.

>>> 6 * 405e9 * 15.6e12
3.7908e+25

6ND gives 3.79 × 10^25 against a reported 3.8 × 10^25. That agreement is the reason to trust the rule for dense models at this scale.

The four sizes

Under the 20 tokens per parameter rule, compute grows with the square of model size: ten times the parameters means a hundred times the FLOPs. That is why the jump from 100B to 1T dense is from 10^24 to 10^26.

Mixture of experts: count active parameters

In a mixture of experts model each token runs through only a few experts, so the FLOPs per token depend on the active parameter count, not the total. For training compute, use 6·N_active·D. The 1T MoE row trained on 15.5T tokens needs about 2.98 × 10^24 FLOPs, about 40 times less than the 1T dense model on 20T tokens and only about 2.5 times the 100B dense model.

This is a compute estimate only. MoE training adds communication: tokens are sent to whichever device holds their experts and back, an all to all exchange that dense training does not have. That cost shows up as lower MFU, which is the next input.

From FLOPs to chip-hours: MFU and goodput

What MFU means

The PaLM paper (arXiv 2204.02311) popularised model FLOPs utilization as "observed throughput relative to theoretical max throughput": the FLOPs the model mathematically needs (6N per token plus attention), at the tokens per second you actually achieved, divided by the hardware's peak. It deliberately excludes recomputation, so it measures useful work. Hardware FLOPs utilization (HFU) includes the recomputed FLOPs and is therefore higher. PaLM 540B on 6,144 TPU v4 chips reached 46.2 percent MFU and 57.8 percent HFU.

The chip-hours formula is then:

chip_seconds = (6 * N_active * D) / (peak_dense_flops_per_chip * MFU)
chip_hours   = chip_seconds / 3600 / goodput

Published MFU values

Sources: Korthikanti et al., arXiv 2205.05198, MegaScale, arXiv 2402.15627, and the PaLM and Llama 3 papers linked above.

Notice the Llama 3 figures: 430 TFLOPS per GPU at 43 percent implies a peak near 1,000 TFLOPS, which is the dense BF16 figure, not the 1,979 "with sparsity" figure. That is a useful sanity check that published MFU numbers are quoted against dense peaks.

Notice also that MFU fell from 43 to 41 percent when Meta doubled the GPU count, and to 38 percent for long context training. Bigger clusters and longer sequences cost efficiency. These numbers come from teams with large engineering investment in their training stacks. A first run on a new cluster, a new chip or a new framework will usually land below them. That is why the calculator uses a range of 0.35 to 0.55 rather than a single value, and why 0.40 is a reasonable central guess for a well run job.

Goodput: the time when nothing useful happens

MFU measures efficiency while the job is running. It does not count the time the job is not running. The Llama 3 paper reports 466 job interruptions during a 54 day snapshot of 405B pre-training, 47 planned and 419 unexpected, and says the team achieved "higher than 90% effective training time." Each interruption means time spent detecting the failure, restarting and reloading from the last checkpoint, and every step since that checkpoint is computed again.

The calculator divides chip-hours by a goodput factor of 0.9 to account for this. If you have no history on your own cluster, 0.9 is optimistic for a first large run.

Worked example 1: a 100B dense model on H100

Assume a 100B dense model trained on 2T tokens, on H100 SXM at 40 percent MFU, goodput 0.9, at the $10 OCI list price.

FLOPs            = 6 * 100e9 * 2e12              = 1.2e24
effective FLOP/s = 989.5e12 * 0.40               = 3.958e14 per GPU
GPU-seconds      = 1.2e24 / 3.958e14             = 3.03e9
GPU-hours        = 3.03e9 / 3600                 = 842,176
with goodput     = 842,176 / 0.9                 = 935,751
cost             = 935,751 * $10                 = about $9.36 million
on 4,096 GPUs    = 935,751 / 4,096 / 24          = about 9.5 days

If you had divided by the 1,979 TFLOPS sparse figure instead, you would have estimated 467,876 GPU-hours and about $4.7 million: exactly half, and wrong.

Worked example 2: why the 1T dense model is a different kind of project

The same arithmetic for the 1T dense model on 20T tokens gives about 1.2 × 10^26 FLOPs. On 16,384 H100 GPUs at 40 percent MFU and 0.9 goodput that is about 238 days of wall clock time. The 1T MoE model on the same cluster and assumptions finishes in about 5.9 days of compute time (before the extra communication overhead lowers its MFU). This is the plainest illustration of why large models trained today lean towards mixture of experts designs: the training compute gap is a factor of about 40.

Training memory: will it even fit?

Compute tells you how long. Memory tells you the minimum number of chips. Rajbhandari et al.'s ZeRO paper (arXiv 1910.02054) gives the standard accounting for mixed precision training with the Adam optimizer: 2 bytes per parameter for FP16 or BF16 weights, 2 bytes for gradients, and 12 bytes of optimizer state (an FP32 master copy, momentum and variance), "2Ψ + 2Ψ + KΨ = 16Ψ bytes" with K = 12.

These are floors. They assume the model states are sharded perfectly with no duplication (ZeRO stage 3 or fully sharded data parallel) and they leave out activations, which depend on batch size, sequence length and recomputation policy, and which can be as large as the model states. Real jobs need several times these counts, and they use far more chips anyway to finish in reasonable wall clock time. Note that the MoE model needs the same memory for its states as the dense model of the same total size: sparsity saves compute, not storage.

The memory floor is also where per chip HBM capacity shows up in the economics. A 32 GB chip needs six times as many devices as a 192 GB chip just to hold the same states, and every one of those devices has to talk to the others. For how HBM, NVLink and similar fabrics shape these choices, see GPU memory and interconnects explained.

Inference memory: weights plus KV cache

Weights at FP16, FP8 and FP4

Weight memory is the parameter count times bytes per parameter: 2 for FP16 or BF16, 1 for FP8 or INT8, 0.5 for FP4 or INT4. For MoE models, count total parameters, because every expert must be resident even if each token uses few of them.

Real checkpoints carry small extras: quantization scales, embedding tables that some schemes keep at higher precision, and runtime buffers. Budget a margin. The calculator assumes only 90 percent of HBM is usable for weights and KV cache.

The KV cache formula

During generation a transformer stores the key and value vectors for every previous token in every layer so it does not recompute them. The size per token is:

kv_bytes_per_token = 2 (K and V) * layers * kv_heads * head_dim * bytes_per_element
kv_bytes_total     = kv_bytes_per_token * context_tokens * batch

With grouped query attention the relevant head count is the number of KV heads, not query heads, which is why Llama 3's choice of 8 KV heads for every size matters so much for serving.

For the assumed shapes, with a BF16 KV cache:

Two things stand out. For the 10B model the KV cache at this batch and context is more than four times the 8-bit weights. And the KV cache scales linearly with both batch and context, so a service that moves from 8K to 128K context multiplies this column by 16.

Allocation matters too. The vLLM paper (Kwon et al., arXiv 2309.06180) profiled earlier serving systems and found that only 20.4 to 38.2 percent of the memory reserved for the KV cache held actual token states; the rest was lost to reservation for future tokens and fragmentation. PagedAttention allocates the cache in blocks to cut that waste. Your formula gives the bytes the cache needs; a naive allocator can reserve several times more.

Worked example 3: how many chips to serve a 100B model

Serving the assumed 100B model with 8-bit weights, a BF16 KV cache, batch 32 and 8,192 tokens of context needs 100 GB of weights plus 94.5 GB of cache, about 194.5 GB. At 90 percent usable HBM:

• H100 (72 GB usable each): 2 chips hold 144 GB, too little; 4 chips hold 288 GB. 4 chips.
• MI300X (172.8 GB usable each): 2 chips hold 345.6 GB. 2 chips.
• TPU v6e (28.8 GB usable each): 8 chips hold 230.4 GB. 8 chips.

The calculator rounds up to powers of two because tensor parallel groups are usually sized that way. That rounding is itself a cost: H100 needs 4 chips where 3 would hold the bytes.

Decode: a memory bandwidth bound on tokens per second

Why decode is memory bound

Generating one token for one sequence needs about 2·N_active FLOPs, one multiply and one add per active weight. But it also needs every one of those weights read from HBM. With 8-bit weights that is 2 FLOPs per byte of weights read; with 16-bit weights, 1 FLOP per byte. The chips can do far more arithmetic per byte than that. The ratio of peak FLOP/s to bytes per second, the ridge point of the roofline model, tells you how much:

At batch 1 the work sits at 1 or 2 FLOPs per byte, two orders of magnitude below the ridge. The chip spends almost all its time waiting for memory. Batching helps because the weights are read once per step and reused for every sequence in the batch: with 8-bit weights and ignoring the KV cache, decode becomes compute bound only around a batch of ridge divided by 2, roughly 295 sequences on H100, 247 on MI300X and 560 on v6e.

 attainable
 FLOP/s
   ^
   |                        ____________________  peak dense FLOP/s
   |                       /
   |                      /   compute bound region
   |                     /    (prefill, big-batch decode)
   |                    /
   |                   / slope = HBM bandwidth
   |                  /
   |     decode at   /
   |     batch 1 -> x
   |               /
   +--------------+-----------------------------------> FLOPs per byte
                ridge point = peak / bandwidth

Pope et al. ("Efficiently Scaling Transformer Inference", arXiv 2211.05102) work through exactly this trade-off for PaLM on TPU v4: they report 29 ms per token at low batch with int8 weights, and 76 percent MFU when processing input tokens in large batches. Low batch generation is latency and bandwidth limited; large batch prompt processing is compute limited. For the same roofline reasoning applied inside a single kernel, see optimizing a CUDA matrix multiply step by step, and for where tensor core peak figures come from, inside an NVIDIA streaming multiprocessor.

The decode bound

Each decode step reads the weights once and reads every sequence's KV cache:

bytes_per_step = weight_bytes_read + batch * context * kv_bytes_per_token
t_memory       = bytes_per_step / (chips * HBM_bandwidth)
t_compute      = batch * 2 * N_active / (chips * peak_dense_flops)
tokens_per_s   <= batch / max(t_memory, t_compute)

This is an upper bound. It assumes peak bandwidth (achievable bandwidth is lower), perfect scaling across chips (tensor parallelism adds communication), and every sequence at the full context length.

Worked example 4: a 70B model at batch 1

A 70B dense model in FP16 has 140 GB of weights. It does not fit on one 80 GB H100, so put it on two with tensor parallelism. Ignoring the KV cache, each token needs 140 GB read across 6.7 TB/s of combined bandwidth: 20.9 ms per token, an upper bound of about 47.9 tokens per second for a single stream. Nothing about the compute rate enters the answer. Quantizing the weights to FP8 halves the bytes and roughly doubles the bound; a faster chip with the same bandwidth would not change it at all.

Worked example 5: batch size is the lever

Here is the assumed 100B model on MI300X with 8-bit weights and an 8,192 token context, as batch grows (chip count from the memory fit, $6 per chip-hour, 50 percent utilization):

Going from batch 1 to batch 64 cuts the bound cost by a factor of more than 20. The price is latency per user, since each step reads more KV cache, and memory, since the cache grows with the batch. Serving cost is mostly a scheduling problem: how full you can keep each step without breaking your latency target.

Mixture of experts at decode time

For one sequence, an MoE decode step reads only the active parameters. For a batch, different tokens pick different experts, so a step reads the union of the experts chosen. If each of B tokens picks k of E routed experts uniformly at random, the expected fraction of experts touched in a step is:

coverage = 1 - (1 - k/E) ** B

With E = 384 and k = 8 (the Kimi K2 configuration), one token touches 2.1 percent of the experts, a batch of 32 touches 49 percent, and a batch of 256 touches 99.5 percent. Real routing is not uniform, and the calculator treats all parameters as expert parameters, which is a simplification, but the shape is right: at moderate batch sizes an MoE model's decode step reads a large share of its total weights, not just its active ones.

The calculator's sweep for the 1T MoE model on eight MI300X chips (1 TB of 8-bit weights, 8,192 token context, 50 percent utilization):

The single stream bound is fast because each token reads only 32 GB of weights. The cost per token falls slowly with batch because each extra sequence pulls in more experts. MoE serving gets its economics at large batch, and that takes the memory to hold all experts plus a large KV cache. That is why MoE models are often served across many chips with expert parallelism, a setup outside this simple calculator.

Prefill: the compute bound part

Before generating, the model processes the prompt. All prompt tokens go through in parallel, so arithmetic intensity is high and prefill is compute bound:

prefill_seconds = 2 * N_active * prompt_tokens / (chips * peak_dense_flops * MFU_prefill)

The attention term grows with the square of the prompt length and is left out here, so long prompts on small models will take longer than this predicts. With a 2,048 token prompt and an assumed prefill MFU of 50 percent, the calculator gives 0.041 seconds for the 10B model on one H100 (FP8) and 0.157 seconds for the 100B model on two MI300X.

Prefill cost per million input tokens follows from the same formula. For the 10B model on H100 at FP8 and 50 percent MFU, one GPU processes about 49,475 prompt tokens per second (1,979 × 10^12 × 0.5 divided by 2 × 10^10), which works out to about $0.056 per million input tokens at $10 per hour. For the 100B model on two MI300X the figure is about $0.25 per million. Input tokens are much cheaper than output tokens in compute terms, because prefill keeps the chip busy with arithmetic instead of waiting on memory.

Cost per million tokens

The formula:

usd_per_million_tokens = usd_per_chip_hour * chips * 1e6 / (tokens_per_second * utilization * 3600)

Utilization is the share of paid hours in which the replica is actually serving at the batch you assumed. A replica that sits idle overnight, or runs at batch 4 because traffic is thin, has low utilization. Most of the gap between a back of the envelope cost per token and a real one is in this term and in the gap between the upper bound and achieved throughput. For a longer treatment of the self hosting comparison, see GPU cost per million tokens: a self hosting estimate, and for why published per token prices fall over time and what that hides, see is AI getting cheaper?.

Energy: kilowatt-hours and dollars

Energy is chip-hours times power times facility overhead:

kWh = chip_hours * board_watts * PUE / 1000

The board power figures are maximums: NVIDIA lists H100 SXM at up to 700 W (configurable) and AMD lists MI300X at 750 W maximum total board power. Actual draw under training load is usually somewhat lower, and the host CPUs, memory, network and storage add to it. PUE, the ratio of facility power to IT power, is an assumption in the calculator, set to 1.3; use your own facility's figure if you know it. Google does not publish a per chip power figure for TPU v6e, so the calculator prints "n/p" for not published rather than guess.

For the 100B worked example (935,751 H100 GPU-hours), energy is about 852 MWh. The US Energy Information Administration's Electric Power Monthly lists the average US industrial electricity price for August 2026 at 9.64 cents per kWh (preliminary). At that price the electricity costs about $82,000, under 1 percent of the $9.36 million rental estimate. When you rent, power is already inside the hourly price. Energy matters most when you own the hardware, and in total cost of ownership comparisons it competes with depreciation, which is far larger at these hardware prices.

The Python calculator

This is the complete calculator. It uses only the Python standard library and runs on Python 3.9 or later. It ran on this Mac to produce the output below; it does not touch any accelerator, it only does arithmetic.

"""ai_cost_calc.py: back of the envelope training and inference cost for any model and chip.

Every input is an assumption you should replace with your own numbers.
Peak FLOPs are vendor DENSE figures (not the "with sparsity" figures).
"""
from dataclasses import dataclass
from typing import Optional

TERA = 1e12


@dataclass
class Chip:
    name: str
    peak_bf16_dense: float   # FLOP/s, vendor peak, dense
    peak_8bit_dense: float   # FLOP/s or OP/s, vendor peak, dense (FP8 or INT8)
    hbm_gb: float            # GB of HBM per chip
    hbm_tbps: float          # TB/s peak memory bandwidth per chip
    watts: Optional[float]      # vendor max board power, None if not published
    usd_per_hour: float      # public on-demand list price per chip-hour


@dataclass
class Model:
    name: str
    total_params: float      # parameters stored in memory
    active_params: float     # parameters used per token (equals total for dense)
    train_tokens: float      # D
    layers: int
    kv_heads: int
    head_dim: int
    experts: int = 0         # routed experts (0 for dense)
    experts_per_token: int = 0


# ---------- training ----------
def train_flops(m: Model) -> float:
    """About 6 FLOPs per active parameter per token (2 forward, 4 backward)."""
    return 6 * m.active_params * m.train_tokens


def train_chip_hours(m: Model, c: Chip, mfu: float, goodput: float = 0.9) -> float:
    """Chip-hours = FLOPs / (peak dense FLOP/s x MFU), divided by goodput for restarts and idle time."""
    seconds = train_flops(m) / (c.peak_bf16_dense * mfu)
    return seconds / 3600 / goodput


def train_state_bytes(m: Model, bytes_per_param: float = 16) -> float:
    """Mixed precision Adam: 2 (weights) + 2 (grads) + 12 (fp32 master, momentum, variance) bytes per parameter."""
    return m.total_params * bytes_per_param


# ---------- inference ----------
def weight_bytes(m: Model, bytes_per_param: float) -> float:
    return m.total_params * bytes_per_param


def kv_bytes_per_token(m: Model, kv_bytes: float = 2) -> float:
    """K and V, per layer, per KV head, per head dimension."""
    return 2 * m.layers * m.kv_heads * m.head_dim * kv_bytes


def chips_to_fit(total_bytes: float, c: Chip, usable: float = 0.9) -> int:
    """Smallest power of two chip count whose usable HBM holds total_bytes."""
    n = 1
    while n * c.hbm_gb * 1e9 * usable < total_bytes:
        n *= 2
    return n


def weights_read_per_step(m: Model, batch: int, w_bytes: float) -> float:
    """Dense: the whole model once per step. MoE: tokens in a batch route to
    different experts, so a step touches about 1 - (1 - k/E)^batch of the
    experts (uniform routing assumed), never less than the active parameters."""
    if m.experts == 0:
        return m.total_params * w_bytes
    coverage = 1 - (1 - m.experts_per_token / m.experts) ** batch
    return max(m.active_params, m.total_params * coverage) * w_bytes


def decode_tokens_per_s(m: Model, c: Chip, n_chips: int, batch: int, context: int,
                        w_bytes: float, kv_b: float = 2) -> float:
    """Upper bound for one replica: each decode step reads the weights once
    plus every sequence's KV cache, and does 2 x active params FLOPs per token.
    Step time is the larger of memory time and compute time. Assumes perfect
    scaling across n_chips, which real tensor parallelism never reaches."""
    bytes_per_step = weights_read_per_step(m, batch, w_bytes) + batch * context * kv_bytes_per_token(m, kv_b)
    t_mem = bytes_per_step / (n_chips * c.hbm_tbps * TERA)
    t_cmp = batch * 2 * m.active_params / (n_chips * c.peak_8bit_dense)
    return batch / max(t_mem, t_cmp)


def prefill_seconds(m: Model, c: Chip, n_chips: int, prompt_tokens: int, mfu: float = 0.5) -> float:
    """Prefill is compute bound: about 2 x active params FLOPs per prompt token."""
    return 2 * m.active_params * prompt_tokens / (n_chips * c.peak_8bit_dense * mfu)


def usd_per_million(c: Chip, n_chips: int, tokens_per_s: float, utilization: float) -> float:
    return c.usd_per_hour * n_chips / (tokens_per_s * utilization * 3600) * 1e6


def energy_kwh(c: Chip, chip_hours: float, pue: float = 1.3) -> Optional[float]:
    if c.watts is None:
        return None
    return c.watts * chip_hours * pue / 1000


CHIPS = [
    # NVIDIA H100 SXM: 1,979 BF16 and 3,958 FP8 TFLOPS listed "with sparsity", so dense is half.
    Chip("H100 SXM", 989.5 * TERA, 1979 * TERA, 80, 3.35, 700, 10.00),
    # AMD MI300X: dense 1,307.4 BF16 and 2,614.9 FP8 TFLOPS, 192 GB, 5.3 TB/s, 750 W.
    Chip("MI300X", 1307.4 * TERA, 2614.9 * TERA, 192, 5.3, 750, 6.00),
    # Google TPU v6e: 918 BF16 TFLOPS, 1,836 INT8 TOPS, 32 GB, 1,638 GB/s; power not published per chip.
    Chip("TPU v6e", 918 * TERA, 1836 * TERA, 32, 1.638, None, 2.70),
]

MODELS = [
    Model("1B", 1e9, 1e9, 20e9, 16, 8, 64),
    Model("10B", 10e9, 10e9, 200e9, 40, 8, 128),
    Model("100B", 100e9, 100e9, 2e12, 88, 8, 128),
    Model("1T dense", 1e12, 1e12, 20e12, 160, 16, 128),
    Model("1T MoE (32B active)", 1e12, 32e9, 15.5e12, 61, 8, 128, 384, 8),
]


def main() -> None:
    print("TRAINING (6ND, BF16 dense peak, goodput 0.9)")
    print(f"{'model':22}{'FLOPs':>10}  {'chip':9}{'MFU':>5}{'chip-hours':>14}{'cost USD':>16}{'energy MWh':>12}")
    for m in MODELS:
        for c in CHIPS:
            for mfu in (0.35, 0.55):
                h = train_chip_hours(m, c, mfu)
                e = energy_kwh(c, h)
                es = f"{e/1000:,.1f}" if e is not None else "n/p"
                print(f"{m.name:22}{train_flops(m):10.2e}  {c.name:9}{mfu:5.2f}{h:14,.0f}{h*c.usd_per_hour:16,.0f}{es:>12}")
    print()
    print("INFERENCE (8-bit weights, BF16 KV cache, batch 32, 8,192-token context, 50% utilization)")
    print(f"{'model':22}{'chip':9}{'chips':>6}{'weights GB':>11}{'KV GB':>8}{'tok/s bound':>13}{'USD/M tok':>11}{'prefill 2k s':>13}")
    batch, ctx, util = 32, 8192, 0.5
    for m in MODELS:
        for c in CHIPS:
            w = weight_bytes(m, 1)
            kv = batch * ctx * kv_bytes_per_token(m)
            n = chips_to_fit(w + kv, c)
            tps = decode_tokens_per_s(m, c, n, batch, ctx, 1)
            cost = usd_per_million(c, n, tps, util)
            pf = prefill_seconds(m, c, n, 2048)
            print(f"{m.name:22}{c.name:9}{n:6d}{w/1e9:11,.0f}{kv/1e9:8,.1f}{tps:13,.0f}{cost:11.3f}{pf:13.3f}")


if __name__ == "__main__":
    main()

Output

TRAINING (6ND, BF16 dense peak, goodput 0.9)
model                      FLOPs  chip       MFU    chip-hours        cost USD  energy MWh
1B                      1.20e+20  H100 SXM  0.35           107           1,069         0.1
1B                      1.20e+20  H100 SXM  0.55            68             681         0.1
1B                      1.20e+20  MI300X    0.35            81             486         0.1
1B                      1.20e+20  MI300X    0.55            52             309         0.1
1B                      1.20e+20  TPU v6e   0.35           115             311         n/p
1B                      1.20e+20  TPU v6e   0.55            73             198         n/p
10B                     1.20e+22  H100 SXM  0.35        10,694         106,943         9.7
10B                     1.20e+22  H100 SXM  0.55         6,805          68,055         6.2
10B                     1.20e+22  MI300X    0.35         8,094          48,564         7.9
10B                     1.20e+22  MI300X    0.55         5,151          30,904         5.0
10B                     1.20e+22  TPU v6e   0.35        11,527          31,124         n/p
10B                     1.20e+22  TPU v6e   0.55         7,336          19,806         n/p
100B                    1.20e+24  H100 SXM  0.35     1,069,430      10,694,301       973.2
100B                    1.20e+24  H100 SXM  0.55       680,546       6,805,464       619.3
100B                    1.20e+24  MI300X    0.35       809,393       4,856,361       789.2
100B                    1.20e+24  MI300X    0.55       515,069       3,090,412       502.2
100B                    1.20e+24  TPU v6e   0.35     1,152,724       3,112,356         n/p
100B                    1.20e+24  TPU v6e   0.55       733,552       1,980,590         n/p
1T dense                1.20e+26  H100 SXM  0.35   106,943,007   1,069,430,074    97,318.1
1T dense                1.20e+26  H100 SXM  0.55    68,054,641     680,546,411    61,929.7
1T dense                1.20e+26  MI300X    0.35    80,939,350     485,636,098    78,915.9
1T dense                1.20e+26  MI300X    0.55    51,506,859     309,041,153    50,219.2
1T dense                1.20e+26  TPU v6e   0.35   115,272,446     311,235,605         n/p
1T dense                1.20e+26  TPU v6e   0.55    73,355,193     198,059,022         n/p
1T MoE (32B active)     2.98e+24  H100 SXM  0.35     2,652,187      26,521,866     2,413.5
1T MoE (32B active)     2.98e+24  H100 SXM  0.55     1,687,755      16,877,551     1,535.9
1T MoE (32B active)     2.98e+24  MI300X    0.35     2,007,296      12,043,775     1,957.1
1T MoE (32B active)     2.98e+24  MI300X    0.55     1,277,370       7,664,221     1,245.4
1T MoE (32B active)     2.98e+24  TPU v6e   0.35     2,858,757       7,718,643         n/p
1T MoE (32B active)     2.98e+24  TPU v6e   0.55     1,819,209       4,911,864         n/p

INFERENCE (8-bit weights, BF16 KV cache, batch 32, 8,192-token context, 50% utilization)
model                 chip      chips weights GB   KV GB  tok/s bound  USD/M tok prefill 2k s
1B                    H100 SXM      1          1     8.6       11,178      0.497        0.004
1B                    MI300X        1          1     8.6       17,685      0.188        0.003
1B                    TPU v6e       1          1     8.6        5,466      0.274        0.004
10B                   H100 SXM      1         10    42.9        2,025      2.744        0.041
10B                   MI300X        1         10    42.9        3,203      1.041        0.031
10B                   TPU v6e       2         10    42.9        1,980      1.515        0.022
100B                  H100 SXM      4        100    94.5        2,205     10.079        0.103
100B                  MI300X        2        100    94.5        1,744      3.823        0.157
100B                  TPU v6e       8        100    94.5        2,156      5.566        0.056
1T dense              H100 SXM     32      1,000   343.6        2,553     69.631        0.129
1T dense              MI300X        8      1,000   343.6        1,010     26.407        0.392
1T dense              TPU v6e      64      1,000   343.6        2,497     38.450        0.070
1T MoE (32B active)   H100 SXM     16      1,000    65.5        3,087     28.798        0.008
1T MoE (32B active)   MI300X        8      1,000    65.5        2,442     10.921        0.013
1T MoE (32B active)   TPU v6e      64      1,000    65.5        6,037     15.902        0.002

How to read the output

Training. Each model appears on each chip at MFU 0.35 and 0.55, so you get a range rather than a point. A few readings:

• The 1B model costs a few hundred to about a thousand dollars of rented compute at list prices: about 52 to 115 chip-hours. At this scale, setup, data work and evaluation runs will cost more than the training run itself.
• The 100B model lands between about $2 million and $10.7 million depending on chip and MFU, or 515,000 to 1.15 million chip-hours.
• The 1T dense model on 20T tokens lands between about $198 million and $1.07 billion, and 51 million to 115 million chip-hours. These are list price on demand figures; nobody training at that scale pays them, but the chip-hour count is what you would have to secure.
• The 1T MoE model needs about 1.3 million to 2.9 million chip-hours, between $4.9 million and $26.5 million at list prices, because only 32B parameters are active per token. Its real MFU is likely lower than a dense model's because of expert communication, so lean towards the higher end.

The cheapest chip in the training table is not a ranking. It reflects the list prices this post used (on demand Oracle prices for the GPUs, on demand Google prices for the TPU), equal MFU across chips, which is not guaranteed, and nothing about the time to secure capacity, the size of available clusters, or the cost of porting a training stack to a different software ecosystem. Change the prices and MFU to your own and the order can change.

Inference. At batch 32, 8,192 tokens and 50 percent utilization:

• The 1B and 10B models fit on one chip each (the 10B needs two v6e chips because of the 32 GB HBM), and the bound cost ranges from about $0.19 to $2.74 per million output tokens.
• The 100B model needs 4 H100, 2 MI300X or 8 v6e chips; the bound cost is $3.82 to $10.08 per million.
• The 1T dense model needs 32 H100, 8 MI300X or 64 v6e chips, with bound costs of $26 to $70 per million.
• The 1T MoE model at batch 32 costs $10.92 to $28.80 per million under these assumptions, about 40 percent of the dense 1T figure on the same chip, and falls further as batch grows.

Remember these are floors on cost, because tokens per second is an upper bound. Real serving stacks reach some fraction of peak bandwidth, lose time to communication and scheduling, and rarely run every replica at the target batch around the clock.

Adapting it

To use your own numbers:

1. Edit CHIPS with the dense peak, memory and the price you actually pay. For a B200 or MI355X, take the dense figures from the vendor's page; do not use a number that includes sparsity.
2. Edit MODELS with your parameter counts, token budget and attention shape. For an MoE model, set experts and experts_per_token.
3. Change batch, ctx and util in main() to match your traffic.
4. If you have measured MFU or measured tokens per second from your own runs, replace the bound with the measurement. A measurement always beats a bound.

For the hardware comparisons that feed the CHIPS table, the series posts on AI chip architecture, what an AI accelerator is and NVIDIA, AMD, Intel and Apple GPU architectures compared cover where the peak and bandwidth figures come from at the silicon level.

Sanity checking with measured results

A calculator built on vendor peaks and bounds should be checked against measured results before anyone uses it to sign a contract. The neutral public source for measured accelerator performance is MLCommons, whose MLPerf Training and Inference results publish throughput for specific models on specific systems, with the submitter, round and software stack named. This post does not quote MLPerf figures, because the point here is the method, not a leaderboard. Before you trust an estimate, compare its implied throughput with a published MLPerf result for a similar model on the same hardware class, and with your own measurements. If your bound is far above what submitters with tuned stacks achieve, your efficiency assumptions are too generous.

Common mistakes

Dividing by the sparse figure. NVIDIA's H100 tensor core rows are listed with sparsity. Use half. AMD prints both; use the dense column. Using the sparse figure halves your chip-hour estimate, as worked example 1 showed.

Treating peak as achieved. Peak is the ceiling. Published large runs reach 38 to 55 percent MFU, and decode reaches a fraction of peak bandwidth. Any estimate with MFU of 1.0 is not an estimate.

Mixing precisions. Training in BF16 divided by an FP8 peak doubles your apparent speed. On v6e, the 8-bit figure is INT8; there is no listed FP8 rate.

Ignoring the KV cache. For the 10B model at batch 32 and 8K context, a weights only bound gives about 10,720 tokens per second on H100. Including the KV cache gives 2,025. That is a factor of more than five, all in the cache.

Ignoring idle time. Training has restarts and checkpoint reloads (goodput). Serving has troughs in traffic (utilization). The calculator's 0.9 goodput and 0.5 utilization are assumptions; at 25 percent utilization the serving costs double again.

Counting MoE parameters the wrong way. Training FLOPs use active parameters. Memory uses total parameters. Decode bandwidth uses something in between that depends on batch size.

Comparing instance prices with chip prices. Some clouds price an eight GPU instance per hour, others a GPU per hour, others a chip per hour. Normalise to the same unit, and remember an instance price includes CPUs, memory and networking that a per chip comparison hides.

Forgetting everything that is not the chip. Storage for data and checkpoints, network egress, failed experiments, evaluation runs, and engineers' time. The 6ND cost is the cost of the final successful run, not of the project.

Using 20 tokens per parameter as a fact. It is a compute optimal ratio from one study. Production models are often trained on many more tokens per parameter because inference cost dominates over a model's life.

Where Safeguard fits, and where it does not

Safeguard does not design, program or speed up chips, and nothing in this post suggests it lowers GPU, TPU or electricity costs. The connection is the software these estimates quietly assume works. Every number above runs on a stack: kernel drivers, CUDA or ROCm, communication libraries, framework wheels, serving engines, container images and model files pulled from registries. Each of those is a versioned component that can carry known vulnerabilities, and a training run that has to stop for an emergency driver or library upgrade loses goodput, the very factor in the formula.

Safeguard's generally available tooling applies to that software: it generates and analyses SBOMs in SPDX and CycloneDX formats, scans repositories, container images and SBOM files for known vulnerabilities, shows which projects contain a given component and version, and can open fix pull requests through autonomous remediation (off by default). AI asset inventory (AI-BOM discovery) is generally available. For what belongs in a GPU stack SBOM layer by layer, see the GPU software supply chain explained.

FAQ

How do I calculate the FLOPs needed to train an LLM?

Multiply 6 by the number of parameters used per token and by the number of training tokens. For a 10B dense model on 200B tokens that is 1.2 × 10^22 FLOPs. For a mixture of experts model, use the active parameter count. The rule leaves out attention FLOPs that grow with context length, so long context training on small models needs a correction.

What MFU should I assume for a training cost estimate?

Published large scale runs report 38 to 55 percent: 46.2 percent for PaLM 540B, 38 to 43 percent for Llama 3 405B on H100 depending on scale and context, 54.2 percent for a 530B model with selective recomputation, and 55.2 percent for MegaScale's 175B run. Use 35 to 55 percent as a range and expect the low end on a new stack.

Should I use the dense or sparse TFLOPS figure?

Dense. The sparse figure assumes fine grained structured sparsity in the weights, which ordinary training and inference do not use. On NVIDIA's H100 page, the starred tensor rows are with sparsity, so halve them.

How much GPU memory does a model need for inference?

Weights take the parameter count times 2 bytes at FP16, 1 byte at FP8 and half a byte at FP4. Add the KV cache: 2 × layers × KV heads × head dimension × bytes per element, per token, times context times batch. Leave about 10 percent of HBM for runtime buffers.

How big is the KV cache for a long context?

It grows linearly with context. For the assumed 100B shape (88 layers, 8 KV heads, head dimension 128) in BF16 it is about 0.36 MB per token, 2.95 GB per 8,192 token sequence, and about 47 GB per 131,072 token sequence.

What limits tokens per second when generating text?

At small and moderate batch sizes, memory bandwidth. Each step reads the weights and the KV cache, so tokens per second per replica is at most batch size times bandwidth divided by bytes read per step. Only at large batches does compute become the limit.

How do I estimate cost per million tokens?

Divide the hourly price of the chips in one replica by the tokens per second it actually serves times 3,600, multiply by a million, and divide by utilization. The tokens per second figure from a bandwidth bound gives you a floor on cost, not the cost.

Is a mixture of experts model cheaper to run than a dense model?

Cheaper to train per token, by roughly the ratio of active to total parameters. For serving it needs the memory of the full model, and at moderate batches each step touches a large share of the experts, so the saving per token is smaller than the training saving and depends on batch size.

How much electricity does training an LLM use?

Chip-hours times board power times PUE. The 100B example on H100 needs about 935,751 GPU-hours, roughly 852 MWh at 700 W and a PUE of 1.3. At the August 2026 US industrial average of 9.64 cents per kWh that is about $82,000, which is small next to the rental cost.

Can I use this calculator for chips not listed here?

Yes. Add a Chip entry with the vendor's dense peak for the precision you will use, HBM capacity and bandwidth, board power if published, and the price you pay. Take each figure from the vendor's own documentation and check whether it is dense or sparse.

More in this series

Other posts in our AI chips and costs series:

• AI Accelerator TCO: Buy vs Rent, Capex, Power, Cooling and Utilization
• AI Chips Compared 2026: B200, H200, MI355X, Gaudi 3, TPU, Trainium
• Cheapest LLM Inference: How to Cut Cost per Million Tokens
• Fastest LLM Inference: Lowest Latency and Most Tokens per Second
• How Many GPUs Does a 1 Trillion Parameter Model Need?
• LLM Inference Cost by Chip and Model Size, 1B to 1 Trillion Parameters
• LLM Training Cost by Chip and Model Size: 1B to 1T, Worked

The chips in these estimates only produce tokens through drivers, CUDA or ROCm libraries, framework wheels, serving engines and container images, and every one of those is open source or third party code with its own vulnerability history. A cluster that has to pause for an urgent library upgrade loses goodput, so keeping that stack patched is part of keeping the cost estimate true.

Fix it without waiting for upstream

Whatever the vulnerability, most of the risk sits in two gaps. The first is the time between a fix being published and that fix actually running in your production builds. The second, and worse, is the time before any fix exists at all, while a maintainer works out a patch and a release schedule. Waiting through both gaps is a habit, not a requirement. Safeguard gives teams three ways to close them.

1. Autonomous remediation: pull requests, not tickets

When a fixed version exists, Safeguard's Griffin models open a tested, policy gated pull request that moves the affected dependency to the safe version. You can run it on one repository or in bulk across every project that carries the vulnerable version. Remediation categories include dependencies, licenses, security posture and findings, and each one is set separately. Reachability analysis ranks the vulnerable code paths your application actually calls first, so the first pull requests you merge are the ones that matter most. Autonomous remediation ships switched off, with every category set to Manual, so your team decides how much runs on its own and what still needs a human approval.

2. Gold Open Source: the safe version in seconds, free

gold.safeguard.sh is free and needs no login. Search any CVE or package to see affected and fixed versions, CVSS, EPSS, CISA KEV status and a recommended safe version, or use the "Am I affected?" checker. The same data is available through a free read API, README status badges and RSS threat feeds.

3. Gold Registry: start from components that are already fixed

Gold Registry provides hardened, drop in replacements for upstream packages and container images. Every Gold artifact is held continuously to zero exploitable critical or high CVEs and a completed license check, with signed build provenance (signed with Safeguard's key and verifiable with cosign, SLSA Build Level 3 as the design target) and a public evidence page you can hand to an auditor. An artifact loses its Gold badge if it regresses. Teams can point their own package directory at the registry so hardened equivalents resolve without changing how developers install anything.

When upstream has no fix yet, or will not ship one, Safeguard first works to get the fix accepted upstream. If the issue is severe or upstream will not move, Safeguard builds a fork, fixes the vulnerability there and publishes a hardened Gold build, so you are not stuck on a maintainer's release schedule. You can also request a hardened, compatibility tested build of a specific package. Ask which packages it covers before you plan around it.

One honest limit: when the vulnerable code lives in a vendor appliance, a hosted service or an operating system runtime you do not build yourself, the fix still has to come from that vendor or distribution. The steps above apply to the open source dependencies and container images you build and ship, which is where most teams carry the largest and least tracked share of their exposure.

See how Safeguard would help

You can download Safeguard and follow the CLI docs to try these yourself.

The first demo sets up the Safeguard runner on your own machine, approves it with its device code and then scans from the terminal. For an AI training or serving platform, that is how you would scan the repositories and container images that carry your CUDA or ROCm libraries and serving engine, without sending source or images off the machine.

Click through it at your own pace, or open it full screen.

The second demo is a real CLI session: install, open the interactive terminal and ask it to fix a lodash CVE. The assistant reads the finding and the advisory, asks before any command that changes something, edits package.json to the fixed version, runs npm install and npm test with approval, prints a summary and pushes nothing. The same flow applies to the Python side of an inference service, such as moving a pinned framework or serving engine dependency to a fixed version and rerunning your tests.

Click through it at your own pace, or open it full screen.

The third demo connects Claude to the Safeguard MCP server with OAuth and shows how to choose which tools Claude can use. When a new advisory lands for a GPU library, an engineer can then ask which projects contain the affected component and version and get the answer from their own scan data.

Click through it at your own pace, or open it full screen.

Quick check

Sources

• NVIDIA H100 Tensor Core GPU product page and specifications
• NVIDIA H100 Tensor Core GPU Architecture whitepaper (sparsity footnote)
• AMD Instinct MI300X accelerator data sheet
• Google Cloud TPU v6e documentation
• Google Cloud TPU pricing, checked 25 October 2026
• Oracle Cloud Infrastructure compute pricing, checked 25 October 2026
• Kaplan et al., Scaling Laws for Neural Language Models, arXiv 2001.08361
• Hoffmann et al., Training Compute Optimal Large Language Models, arXiv 2203.15556
• Chowdhery et al., PaLM: Scaling Language Modeling with Pathways, arXiv 2204.02311
• Korthikanti et al., Reducing Activation Recomputation in Large Transformer Models, arXiv 2205.05198
• Jiang et al., MegaScale: Scaling Large Language Model Training to More Than 10,000 GPUs, arXiv 2402.15627
• Llama Team, The Llama 3 Herd of Models, arXiv 2407.21783
• Kimi Team, Kimi K2: Open Agentic Intelligence, arXiv 2507.20534
• Rajbhandari et al., ZeRO: Memory Optimizations Toward Training Trillion Parameter Models, arXiv 1910.02054
• Pope et al., Efficiently Scaling Transformer Inference, arXiv 2211.05102
• Kwon et al., Efficient Memory Management for Large Language Model Serving with PagedAttention, arXiv 2309.06180
• US EIA Electric Power Monthly, Table 5.6.A, average price of electricity by sector
• MLCommons MLPerf Inference: Datacenter

Last reviewed: 25 October 2026. Prices and specifications change; check the linked vendor pages before you rely on any figure.