LLM Training Cost by Chip and Model Size: 1B to 1T, Worked

Training a language model costs roughly 6 × parameters × tokens floating point operations, divided by what your chips actually deliver, multiplied by the hourly price. At public list prices checked on 25 October 2026, that works out to a few hundred dollars for a compute-optimal 1B model, tens of thousands for 10B, about $2 million to $9 million for 100B, and hundreds of millions of dollars for a dense 1 trillion parameter model, while a 1T mixture of experts model with 50B active parameters costs about one twentieth of the dense one.

Those ranges are wide on purpose. The list price of a chip-hour varies by a factor of nearly fifteen between a Google TPU v5e and an NVIDIA B300, and the share of peak throughput a real training job sustains varies from the low 30s to the mid 50s in percent across published reports. This post shows every input, every formula and every intermediate number, so you can swap in your own price, your own utilisation and your own token budget and get your own answer. All arithmetic was done in Python on a Mac, and the scripts are included.

It belongs to our series on AI hardware. For the chips themselves, see AI chip architecture, what an AI accelerator is and GPU memory and interconnects. For prices, see GPU rental prices in 2026. For the serving side, see GPU cost per million tokens and whether AI is getting cheaper.

Key takeaways

• Training compute is about 6·N·D FLOPs, where N is the number of parameters that are active for each token and D is the number of training tokens (Kaplan et al., 2020). For a mixture of experts model, N is the active count, not the total.
• Compute-optimal training uses about 20 tokens per parameter, per the Chinchilla paper's own projection table: 20.2 billion tokens for 1B, 205.1 billion for 10B and 21.2 trillion for 1T (Hoffmann et al., 2022). Most production models train on far more tokens than that, because a smaller model that has seen more data is cheaper to serve.
• Model FLOPs utilisation (MFU) in published large runs sits between about 38% and 55%: Llama 3 405B reported 38% to 43% on up to 16,384 H100s, PaLM reported 46.2% on 6,144 TPU v4 chips, and MegaScale reported 55.2% on 12,288 GPUs. This post uses 35% to 45% as the working range.
• On those assumptions, a compute-optimal 100B dense model needs about 750,000 to 960,000 H100-hours, or about 330,000 to 420,000 B200-hours. At AWS on-demand list prices that is $5.15M to $6.62M on H100 and $4.69M to $6.03M on B200.
• A dense 1T model at 21.2 trillion tokens needs about 1.27 × 10^26 FLOPs, which is 79M to 102M H100-hours and roughly $546M to $702M at the H100 on-demand price. The 1T MoE with 50B active parameters on the same tokens needs 20 times less compute.
• Electricity is a small share of a rented run: under 2% of the on-demand bill in our 1T example at 8 cents per kWh. It matters much more when you own the hardware.
• Mixed-precision Adam needs 16 bytes per parameter before activations, so a 1T model carries about 16 TB of model state. Tensor, pipeline and data parallelism with ZeRO sharding are what make that fit on 80 GB to 288 GB devices.
• Published runs cost more than the 6ND arithmetic alone suggests. Meta's Llama 3.1 405B used 30.84M H100-hours, which implies about 34.5% MFU across the whole job, below the 38% to 43% steady-state figure, because restarts, long-context stages and other work all count.

How to read every number in this post

Every estimate here is built from four inputs. Each one has a source or a stated assumption, and each one moves the answer.

Three things these numbers are not. They are not measured benchmark results: the only measurements quoted are the ones published in the papers named. They are not quotes: real contracts at this scale are negotiated, and a large buyer pays less than list. And they are not the full cost of a model: experiments, failed runs, data, evaluation, post-training and staff are all extra, and for frontier labs those extras are large. Every cost below is an estimate built from the stated assumptions.

The cost model, step by step

Step 1: count the FLOPs

A transformer's forward pass costs about 2 FLOPs per parameter per token: one multiply and one add for each weight in each matrix multiply. The backward pass costs about twice the forward pass, because it computes gradients both for the activations and for the weights. Add them and you get about 6 FLOPs per parameter per token. Kaplan and colleagues wrote it down in 2020 as C ≈ 6N floating point operations per training token, with N counting non-embedding parameters.

C  =  6  ×  N_active  ×  D

C         total training compute, FLOPs
N_active  parameters used for each token (all of them for a dense model,
          the routed experts plus shared layers for an MoE)
D         training tokens

The formula leaves out attention's own score computation, which grows with sequence length, and it ignores the extra forward pass you pay if you recompute activations to save memory. That is why MFU, which counts only the 6N "model FLOPs", is lower than hardware FLOPs utilisation, which counts everything the chip actually executed. PaLM reported both for its 540B run: 46.2% MFU and 57.8% hardware FLOPs utilisation.

Step 2: divide by delivered throughput

chip-seconds  =  C  /  (peak_FLOP/s  ×  MFU)
chip-hours    =  chip-seconds / 3600

Peak is the vendor's dense number for the precision you train in. Almost every published large run reports MFU against BF16 peak, so BF16 dense is what we use. FP8 training can raise delivered throughput, but it does not double it in practice, and quoting MFU against an FP8 peak makes the percentage look worse even when the run is faster. Keep the precision and the peak consistent.

Step 3: multiply by the price

cost  =  chip-hours  ×  price_per_chip_hour

Cloud prices are per instance. An 8-GPU instance price divided by 8 gives the per-GPU price; TPU prices are already per chip-hour; AWS lists Trainium2 Capacity Blocks with a per-accelerator figure.

Step 4: wall clock, energy and memory

days    =  C / (peak × MFU × n_chips × effective_time) / 86,400
energy  =  chip-hours × kW_per_chip × server_overhead × PUE
memory  =  16 bytes × N_total (model state)  +  activations

Memory uses the total parameter count, not the active one, because an MoE model still has to store every expert's weights, gradients and optimizer state somewhere.

The calculator

This is the script behind the tables. It runs on any machine with Python 3; it does not touch a GPU.

# Training cost estimator: 6ND FLOPs, MFU range, chip-hours, cost, energy.
# Peak = vendor dense BF16 peak in FLOP/s. Prices in USD per chip-hour.

CHIPS = {
    # name: (dense BF16 peak FLOP/s, on-demand $/chip-h, 3-year commit $/chip-h or None)
    "H100 SXM":        (989.5e12, 6.88,  5.395),
    "B200 (HGX)":      (2.25e15,  14.242, None),
    "MI300X":          (1.3e15,   6.00,  2.634),
    "TPU v6e":         (918e12,   2.70,  1.22),
    "TPU7x Ironwood":  (2.307e15, 12.00, 5.40),
    "Trainium2":       (667e12,   2.235, None),   # Capacity Block price
}

def training_flops(params_active, tokens):
    return 6 * params_active * tokens

def chip_hours(flops, peak, mfu):
    return flops / (peak * mfu) / 3600

def wall_clock_days(flops, peak, mfu, n_chips, effective=0.9):
    return flops / (peak * mfu * n_chips * effective) / 86400

def energy_mwh(hours, kw_per_chip, pue=1.2):
    return hours * kw_per_chip * pue / 1000

if __name__ == "__main__":
    n, d = 100e9, 2e12                     # 100B dense, Chinchilla tokens
    c = training_flops(n, d)
    print(f"FLOPs: {c:.3e}")
    for name, (peak, od, commit) in CHIPS.items():
        lo, hi = chip_hours(c, peak, 0.45), chip_hours(c, peak, 0.35)
        line = f"{name:15s} {lo:>12,.0f} to {hi:>12,.0f} chip-h  on-demand ${lo*od:,.0f} to ${hi*od:,.0f}"
        if commit:
            line += f"  3-yr ${lo*commit:,.0f} to ${hi*commit:,.0f}"
        print(line)
    days = wall_clock_days(c, 989.5e12, 0.45, 1024), wall_clock_days(c, 989.5e12, 0.35, 1024)
    print(f"1,024 H100s: {days[0]:.1f} to {days[1]:.1f} days")
    print(f"H100 energy at 0.91 kW per GPU, PUE 1.2: {energy_mwh(chip_hours(c, 989.5e12, 0.45), 0.91):,.0f} MWh")

Its output on our machine:

FLOPs: 1.200e+24
H100 SXM             748,601 to      962,487 chip-h  on-demand $5,150,375 to $6,621,911  3-yr $4,038,703 to $5,192,618
B200 (HGX)           329,218 to      423,280 chip-h  on-demand $4,688,724 to $6,028,360
MI300X               569,801 to      732,601 chip-h  on-demand $3,418,803 to $4,395,604  3-yr $1,500,855 to $1,929,670
TPU v6e              806,907 to    1,037,452 chip-h  on-demand $2,178,649 to $2,801,120  3-yr $984,427 to $1,265,691
TPU7x Ironwood       321,084 to      412,822 chip-h  on-demand $3,853,008 to $4,953,867  3-yr $1,733,853 to $2,229,240
Trainium2          1,110,556 to    1,427,857 chip-h  on-demand $2,482,092 to $3,191,262
1,024 H100s: 33.8 to 43.5 days
H100 energy at 0.91 kW per GPU, PUE 1.2: 817 MWh

The tables later in this post were generated by a longer version of the same script with all twelve chips and unrounded prices, so the last digit can differ by a few dollars from this short version.

Input 1: chip specifications from the vendors

Every figure below is a vendor peak, not a measurement. NVIDIA, AMD and AWS publish sparse figures alongside dense ones; sparse figures assume 2:4 structured sparsity, which dense LLM pretraining does not use, so only the dense column feeds the cost model. Where a vendor lists only a sparse number, the dense number is half of it, and NVIDIA's pages say so in their footnotes.

A few notes on reading that table.

• Per-GPU Blackwell figures are derived. NVIDIA's HGX page lists 8-GPU totals with a footnote that dense is half of the sparse figure shown. Dividing 36 PFLOPS sparse BF16 by 8 GPUs and then by 2 gives 2.25 PFLOPS dense per B200 or B300. The same page lists the B300 and B200 with identical BF16 and FP8 totals; the B300's advantage on that page is FP4 inference throughput and memory, neither of which helps a BF16 training run much.
• Google does not label TPU figures as dense or sparse. We treat them as dense peak. Google lists 8,960 chips per v5p pod, 256 per v6e pod and 9,216 per TPU7x pod.
• Intel's Gaudi 3 figure is stated as 1.8 PFLOPS of FP8 and BF16 compute in its architecture documentation, without a dense or sparse label. We use it as dense, which flatters Gaudi 3 if it is not.
• Power figures differ in scope. H100, H200 and AMD figures are per board. NVIDIA's DGX B200 figure is for a whole 8-GPU system including CPUs and networking. Google and AWS do not publish per-chip power on the pages we used, so we do not estimate energy for TPUs or Trainium2.

Training speed depends mostly on the BF16 column and on how well the interconnect lets you keep the chips busy. Memory capacity decides how many chips you need just to hold the model state, and therefore how much parallelism, and therefore how hard it is to keep MFU up. Our AI chip architecture explainer and inside an NVIDIA streaming multiprocessor cover why peak and delivered throughput diverge.

Input 2: prices, checked on 25 October 2026

All prices are public list prices in US dollars, Linux, checked on 25 October 2026. The 3-year column is the published commitment price where one exists: Azure reserved VM instance prices for H100, H200 and MI300X, and Google committed use prices for TPUs. Azure lists a reservation as a total for the term, so we divide by 26,280 hours (3 × 8,760) and by 8 GPUs.

What the table says, and what it does not.

• The same chip has very different prices. An H100 is $6.88 per GPU-hour on AWS on-demand and $12.29 on Azure pay-as-you-go in East US. Azure's own price feed shows the 8-GPU H100 VM at about $18.17 per hour as spot capacity, about 82% below pay-as-you-go, which is only usable by a job that can survive being interrupted. Our rental price survey covers specialist clouds, which list H100s from about $2.69 to $4.50.
• Commitments roughly halve the list. Google's 3-year commitment prices for every TPU generation are 55% below on-demand. Azure's 3-year H100 reservation is 56% below its own pay-as-you-go, and its 1-year reservation is 36% below. Against AWS's lower on-demand H100 price, the Azure 3-year figure is only 22% lower, which is why the discount you can claim depends on which list you start from.
• Trainium2 has no on-demand price on AWS's Linux on-demand list for the regions we checked. The $2.235 per chip-hour is the Capacity Block rate, which is a prepaid reservation for a fixed window.
• MI355X, MI325X, Gaudi 3 and GB200 have no price in our tables. We did not find a public on-demand list price for them from a major cloud that we could tie to a per-chip figure with confidence on the check date. Azure lists an ND GB200 v6 VM at $108.16 per hour in East US, but we have not confirmed the per-VM GPU count from Azure's documentation, so we leave it out rather than guess.

For the per-chip comparison, the useful single number is dollars per delivered unit of compute. At 40% MFU, here is the cost to deliver 10^21 model FLOPs (one zettaFLOP), which is roughly what a 2.9B model needs at 20 tokens per parameter:

This is the table people want to turn into a ranking, and it does not support one on its own. It assumes the same MFU on every chip, and MFU is exactly what differs: a framework that has had years of tuning on one architecture will reach a higher fraction of peak there than on a newer one. A smaller chip such as the TPU v5e needs many more devices for the same job, which costs more communication and lowers MFU at scale; with only 16 GB per chip it cannot practically hold a large model's training state without very wide sharding. The cheapest compute per FLOP is not cheapest per finished model if it takes three extra weeks to get running.

Input 3: utilisation, from published training reports

MFU is the ratio of the model FLOPs a job completes per second to the hardware's theoretical peak. The term comes from the PaLM paper, and it is now the standard way large training reports describe efficiency. These are the published figures we anchor on.

Meta's own explanation for the drop from 43% to 41% is instructive: at 16,384 GPUs the data parallel degree doubled to 128, so each data parallel group got a smaller batch to keep the global batch at 16M tokens, and smaller per-group batches use the hardware less well. Scale costs efficiency, which is why the MFU you plan for should fall as the cluster grows.

MLCommons MLPerf Training results report time to train a fixed benchmark rather than MFU for a full pretraining run, so we do not derive utilisation from them here. We use 35% to 45% for every chip and model size. That is deliberately conservative against MegaScale's 55.2% and generous for small clusters running untuned code, where 20% to 30% is common in practice. The range does not account for the second effect that matters as much as MFU: time lost to failures. Meta reports 466 job interruptions during a 54-day snapshot of Llama 3 405B pretraining, about 78% of the unexpected ones attributed to confirmed or suspected hardware issues, and an effective training time above 90%. We apply 90% effective time to wall-clock estimates. The chip-hour and cost tables do not include it, so for a large run add roughly 10% to them.

A mixture of experts model is harder to run at high MFU, because tokens are routed across devices through all-to-all communication and the expert matrices are smaller. DeepSeek-V3's technical report does not quote an MFU, but it does give enough to compute the delivered model FLOP rate: 37B activated parameters, 14.8T pretraining tokens and 2,664K H800 GPU-hours for pretraining. By 6·N·D that is about 3.29 × 10^24 FLOPs, or about 343 TFLOPs of model compute per GPU per second, against the 380 to 430 Meta reports for a dense model on H100s. DeepSeek also trained in FP8 for much of the model, so the comparison is approximate. We keep 35% to 45% for the MoE case but flag it as the optimistic side.

Input 4: how many tokens

The Chinchilla paper fitted a scaling law to more than 400 training runs and concluded that, for a fixed compute budget, parameters and tokens should grow equally: double the model, double the data. Its Table 3 projects the compute-optimal token count for several sizes. We use those numbers directly where the table has them and 20 tokens per parameter where it does not.

Why an over-trained case at all? Because almost nobody ships a compute-optimal model any more. Chinchilla optimises for the cheapest training run that reaches a given loss. If you will serve the model to millions of users, inference dominates the lifetime cost, and a smaller model trained on more data is cheaper to serve at the same quality. Meta trained Llama 3 8B and 70B on 15T+ tokens: about 1,875 tokens per parameter for 8B and 214 for 70B. The 405B flagship saw 15.6T tokens, about 38.5 per parameter. A fixed 15T budget for the three smaller sizes reflects that practice. For 1T, 15T would be below Chinchilla, so the over-trained case doubles the Chinchilla figure to 40T instead.

For the MoE model we assume 1T total parameters with 50B active per token, about 5% of the total. DeepSeek-V3's published ratio is 37B active out of 671B total, about 5.5%. We are not describing any specific model, only using a published ratio as a reasonable shape.

Total training compute for every case

Our 6·N·D results for 1B, 10B and 1T match the FLOPs column of Chinchilla's Table 3 (1.21 × 10^20, 1.23 × 10^22 and 1.27 × 10^26), which is a useful check that the formula and the token counts are consistent. Llama 3 405B on 15.6T tokens gives 3.79 × 10^25, matching the 3.8 × 10^25 FLOPs Meta reports.

Each step up in model size at Chinchilla tokens multiplies compute by about 100, because both N and D grow tenfold. That single fact explains most of the cost tables below: going from 10B to 100B does not cost ten times more, it costs a hundred times more, unless you hold the token count fixed.

1B parameters: worked estimate

A compute-optimal 1B model is a small job. At 1.21 × 10^20 FLOPs, an H100 at 40% MFU delivers that in about 85 hours of one GPU, which is a single 8-GPU server for about half a day.

At that size, the price per chip-hour hardly matters: the whole run costs less than an engineer's day. What matters is setup. A small model will not reach 35% MFU on a large chip without care, because its matrices are small relative to what a B200 or Ironwood chip needs to stay busy. A batch of short sequences on a model with a hidden size of 2,048 does not saturate a device built for hidden sizes of 8,192 and up, so expect the lower end of the MFU range or below it on the biggest chips.

The over-trained 1B case is a different budget entirely. Fifteen trillion tokens is 743 times more data than Chinchilla's optimum, and compute scales with it:

A few hundred thousand dollars to train a 1B model on 15T tokens at list prices, and a fifth to a half of that at committed TPU or MI300X prices. Wall clock on 256 H100s is about 10 to 13 days; on 256 B200s, about 4.5 to 5.7 days.

Memory for 1B. Mixed-precision Adam needs 16 bytes per parameter: 2 for BF16 weights, 2 for BF16 gradients and 12 for the FP32 master weights plus Adam's two FP32 moment estimates, exactly as the ZeRO paper breaks it down. For a 1B model built as 20 layers with a hidden size of 2,048 (12 × 20 × 2,048² is about 1.01B parameters), that is about 16 GB of model state. Activations for one 4,096-token sequence, with selective activation recomputation, are about 5.7 GB. That fits on every chip in our table except the TPU v5e's 16 GB, with plain data parallelism and no sharding at all.

10B parameters: worked estimate

At 205.1 billion tokens a 10B model needs 1.23 × 10^22 FLOPs, about 100 times the 1B run.

That is tens of thousands of dollars on demand, or a single 8-GPU H100 server for about 40 to 51 days. On 64 H100s at 90% effective time it is 5.6 to 7.1 days; on 64 B200s, 2.4 to 3.1 days.

The over-trained case on 15T tokens:

Low single-digit millions of dollars at list price, and about 25 to 33 days on 1,024 H100s.

Memory for 10B. At 16 bytes per parameter, a 10B model (50 layers, hidden size 4,096) carries about 161 GB of model state. That no longer fits on one 80 GB H100, so plain data parallelism, which keeps a full copy on every device, fails. Two standard fixes:

• ZeRO stage 3 or fully sharded data parallel (FSDP) splits weights, gradients and optimizer states across the data parallel group. On 8 H100s, model state drops to about 20 GB per GPU, plus about 28.5 GB of activations per 4,096-token sequence: about 49 GB, which fits.
• ZeRO stage 1 shards only the optimizer states. Per device memory is 4 bytes per parameter for weights and gradients plus 12 divided by the data parallel degree. On 64 H100s that is about 42 GB of state plus 28.5 GB of activations, about 71 GB: it fits, with little headroom.

On a 256 GB MI325X or a 288 GB MI355X, the 10B model's roughly 190 GB of state and activations fits on one device with plain data parallelism. On 192 GB parts such as the MI300X or an Ironwood chip it sits right at the limit, so a little sharding is still needed. Fewer sharding decisions is a real advantage for small teams, separate from any throughput number.

100B parameters: worked estimate

A 100B dense model at 2T tokens needs 1.2 × 10^24 FLOPs. This is the first size where the choice of chip, price and commitment shows up as millions of dollars.

On demand, the estimates range from about $2.2M on TPU v6e to about $8.7M on TPU v5p. With 3-year commitments, about $1.0M to $5.2M. Wall clock on 1,024 H100s is 34 to 44 days, and on 1,024 B200s about 15 to 19 days.

Over-trained on 15T tokens:

Tens of millions of dollars, and on 8,192 H100s about 32 to 41 days of wall clock.

Memory for 100B. At 80 layers and a hidden size of 10,240 (about 100.7B parameters), model state is about 1.6 TB, and activations for one 4,096-token sequence across all layers are about 114 GB with selective recomputation, or about 651 GB without it. No single device holds that. A typical layout:

• Tensor parallelism of 8 splits every weight matrix across the 8 GPUs of one server, which share NVLink or Infinity Fabric. It also divides activation memory by 8 when combined with sequence parallelism.
• Pipeline parallelism of 4 or 8 puts a contiguous block of layers on each group of 8 GPUs.
• Data parallelism with ZeRO stage 1 fills out the rest of the cluster.

With TP 8, PP 8 and DP 16 on 1,024 H100s, model state is about 7.5 GB per GPU and activations about 14.3 GB, about 22 GB in total. With PP 4 and DP 32, about 28 GB. Both fit comfortably, which leaves room for larger micro-batches or longer sequences. On MI355X, PP 2 with DP 64 also fits at about 41 GB of 288 GB.

1T parameters, dense: worked estimate

A dense 1T model at Chinchilla's 21.2T tokens needs 1.27 × 10^26 FLOPs. The Chinchilla authors themselves noted that a 1T model is unlikely to be the optimal choice unless you have a budget of about 10^26 FLOPs, more than 250 times what Gopher used.

Hundreds of millions of dollars at list price, and $428M to $550M even at the 3-year committed rate we have for the most widely available GPU, the H100. The TPU v5e row is arithmetic only: with 16 GB per chip, a 1T dense training run would have to shard its 16 TB of state across thousands of chips before activations, which is not how anyone would build it.

Over-trained on 40T tokens:

At this point the rental model itself starts to break. A run on 16,384 H100s at Chinchilla tokens would take about 224 to 288 days of wall clock, between seven and ten months. On 65,536 H100s it drops to 56 to 72 days, but 65,536 GPUs at 0.91 kW each with a PUE of 1.2 is a site drawing about 72 MW, and nobody rents that on demand. Anyone training a dense 1T model owns or has reserved the capacity for years, which is why the 3-year column, or the purchase economics in a total cost of ownership model, is the realistic one at this size.

Memory for 1T dense. With 128 layers, a hidden size of 25,600 and 200 attention heads of size 128 (about 1.007T parameters), model state is about 16.1 TB: 2 TB of BF16 weights, 2 TB of gradients and 12 TB of FP32 master weights and Adam states. Just holding that, at 90% usable memory, takes at least 223 H100s, 99 B200-class GPUs, 93 Ironwood chips or 62 MI355X GPUs. Real layouts use far more devices than that minimum, for throughput.

Activations assume one 4,096-token sequence per micro-batch, sequence parallelism alongside tensor parallelism, and selective recomputation unless stated. With one-forward-one-backward pipeline scheduling, the first stage holds activations for as many micro-batches as there are stages, so the activation memory per device works out to roughly all layers' worth divided by the tensor parallel degree, as Korthikanti and colleagues show. Full recomputation keeps only each layer's input and replays the forward pass in the backward pass, which costs roughly one extra forward pass, about a third more compute, and lowers hardware efficiency relative to MFU. Which layout a team actually picks depends on the interconnect: tensor parallelism of 16 only makes sense where 16 devices share a fast scale-up domain, such as a TPU slice or an NVL72 rack, not across two 8-GPU servers.

1T parameters, mixture of experts: worked estimate

Our MoE model has 1T total parameters and 50B active per token. Compute follows the active count, so at 21.2T tokens it needs 6 × 50B × 21.2T = 6.36 × 10^24 FLOPs, exactly one twentieth of the dense 1T run.

Over-trained on 40T tokens:

Tens of millions of dollars instead of hundreds, and 45 to 58 days on 4,096 H100s for the Chinchilla-token case. That gap is the reason most of the largest published open models are mixtures of experts.

Three caveats keep this from being a free lunch.

1. Memory follows the total, not the active count. All 16 TB of model state still has to live somewhere. Expert parallelism spreads the experts across devices: if 990B of the 1T parameters sit in experts and you use 64-way expert parallelism, each device holds about 15.5B expert parameters, which is about 248 GB of state at 16 bytes each. Adding ZeRO-1 across a data parallel group of 8 cuts that to about 85 GB. At 128-way expert parallelism, it is about 124 GB before ZeRO and 43 GB with it.
2. Utilisation is usually lower. Routing tokens to experts on other devices needs all-to-all communication twice per MoE layer, and load imbalance leaves some experts idle. DeepSeek-V3's report devotes a long section to overlapping that communication with compute, which tells you how much it costs if you do not.
3. The comparison is not quality-neutral. A 1T MoE with 50B active parameters is not the same model as a 1T dense one. Whether it reaches the same quality at a twentieth of the compute is an empirical question, and this post does not answer it.

Wall-clock time on a stated cluster

Chip-hours tell you the bill; wall clock tells you when you get the model. These estimates divide the compute by the cluster's delivered throughput and by 90% effective training time.

Two cautions. First, wall clock does not shrink linearly with cluster size forever: Meta's own MFU fell from 43% to 41% when it doubled from 8,192 to 16,384 GPUs, and at some point adding devices adds more communication than compute. Second, the per-chip peak ratios carry straight through. A B200's dense BF16 peak is about 2.27 times an H100's, an Ironwood chip's about 2.33 times, and an MI355X's about 2.53 times, so on paper they finish proportionally sooner. Whether they do in practice depends on the software stack reaching the same MFU on each, which is not something a spec sheet can tell you.

As a check on the method, the Llama 3 405B run at 3.79 × 10^25 FLOPs on 16,384 H100s at Meta's reported 41% MFU works out to about 66 days of pure compute, or about 73 days at 90% effective time. Meta does not publish an exact wall-clock duration for pretraining, but it does describe a 54-day snapshot period within it, which is consistent with a run of a few months.

Energy in MWh and the power bill

Energy only matters directly to whoever pays the power bill, and in a rental that is the provider, who prices it in. It still shapes where large runs can happen at all. These estimates use chip-hours from the tables, a vendor power figure, a server overhead factor of 1.3 for figures that cover only the accelerator board, and a PUE of 1.2. The overhead and PUE are our assumptions, not vendor numbers. For B200 we use NVIDIA's DGX B200 system figure of about 14.3 kW divided by 8, which already includes CPUs, memory and networking, so no further server overhead is added.

At $0.08 per kWh, the 100B Chinchilla run's electricity costs about $52,000 to $84,000 across these chips, and the dense 1T run about $5.5M to $8.9M. At $0.05 to $0.15 per kWh the H100 1T figure moves between about $5.6M and $16.7M at the high end of energy use. Set against the on-demand compute bill, power at 8 cents is around 1.3% of the H100 1T total. That ratio is why energy barely moves a rental decision and dominates a build decision: an owner pays the power directly, every hour, for the life of the hardware.

The energy per run is closer across chips than the chip-hours are. A newer chip draws more power per device and finishes sooner, so the products land in a similar band. Treat any claim that one chip is "greener" for training with the same care as a throughput claim: it depends on the MFU each one reaches, and on what the overhead and PUE really are in the building that houses it.

For scale, Meta's model card states 39.3M cumulative H100 GPU-hours for the Llama 3.1 family, with power counted as peak capacity per device, 700 W, adjusted for power usage efficiency. The 405B model alone used 30.84M GPU-hours. At 700 W that is about 21,588 MWh at the board level before any overhead.

Memory for training, in detail

Memory is the constraint that decides how many chips a job needs before you even think about speed. There are four kinds.

Per-parameter model state, mixed-precision Adam (ZeRO paper, section 3.1)

  BF16 weights            2 bytes   ─┐
  BF16 gradients          2 bytes    ├─ 4 bytes, needed by forward/backward
  FP32 master weights     4 bytes   ─┐
  FP32 Adam momentum      4 bytes    ├─ 12 bytes, needed only by the optimizer step
  FP32 Adam variance      4 bytes   ─┘
                         ─────────
                         16 bytes per parameter

Activations (Korthikanti et al. 2022, equation 1), per layer, 16-bit:
  s · b · h · (34 + 5·a·s/h) bytes
  s = sequence length, b = micro-batch size, h = hidden size, a = attention heads
  selective recomputation drops the 5·a·s/h term; full recomputation keeps ~2·s·b·h

Plus: temporary buffers, communication buckets, fragmentation (often 10% or more)

The activation term surprises people. For our 1T architecture at a 4,096-token sequence, 5·a·s/h is 5 × 200 × 4,096 / 25,600 = 160, which is more than four times the 34 in front of it. The attention scores, the softmax output and the dropout masks grow with the square of the sequence length, and they are what selective recomputation, or a fused attention kernel that never stores the scores, eliminates. Without it, one 4,096-token sequence through a 1T model needs about 2.6 TB of activations; with it, about 456 GB, before any tensor parallel split.

How each kind of parallelism splits memory

                    weights   grads   optimizer   activations
Data parallel (DP)    copy     copy     copy        split by batch
ZeRO-1                copy     copy     ÷ DP        split by batch
ZeRO-2                copy     ÷ DP     ÷ DP        split by batch
ZeRO-3 / FSDP         ÷ DP     ÷ DP     ÷ DP        split by batch
Tensor parallel (TP)  ÷ TP     ÷ TP     ÷ TP        ÷ TP (with sequence parallel)
Pipeline (PP)         ÷ PP     ÷ PP     ÷ PP        ~ all layers' worth on stage 1
Expert parallel (EP)  experts ÷ EP for MoE layers; dense layers as above

The ZeRO paper gives the numbers for each stage. With 64-way data parallelism, model state for a 7.5B model falls from 120 GB per device with standard data parallelism to about 31.4 GB with optimizer sharding. With all three stages, it is 1.9 GB. The paper's Table 1 even includes a 1T model: 16,000 GB of model state per device without partitioning, and 15.6 GB per device with weights, gradients and optimizer states all partitioned across 1,024 devices. More generally, optimizer-state sharding brings 16 bytes per parameter down towards 4 as the data parallel degree grows, and full sharding brings it down to 16 divided by the degree.

Each of these has a communication price. ZeRO-3 and FSDP gather each layer's weights before using them, which adds traffic proportional to the model size on every step. Tensor parallelism exchanges activations inside every layer, so it needs the fastest links you have, which is why it stays inside an NVLink, Infinity Fabric or TPU interconnect domain. Pipeline parallelism sends only activations between stages but leaves devices idle at the start and end of each batch, the "pipeline bubble", which you shrink by using many micro-batches. The Megatron-LM paper by Narayanan and colleagues is the standard reference for combining all three.

A 1T dense layout on 16,384 H100s (TP 8, PP 16, DP 128)

  one model replica = 16 pipeline stages × 8 GPUs = 128 GPUs
  ┌──────── server 1 (NVLink) ────────┐   ┌──────── server 2 ────────┐        ┌─── server 16 ───┐
  │ GPU0 … GPU7 : layers 1-8, TP=8    │ → │ layers 9-16, TP=8        │ → … → │ layers 121-128  │
  └───────────────────────────────────┘   └──────────────────────────┘        └─────────────────┘
        ▲ activations flow forward, gradients flow back, over the scale-out network

  128 such replicas, each on its own slice of data
  ZeRO-1 shards the 12 bytes/param optimizer state across the 128 replicas
  per GPU: 32.2 GB state + 3.4 GB activations (full recompute) = 35.6 GB of 80 GB

Sanity checks against published training runs

An estimate method is only worth trusting if it reproduces published numbers. Two families of open models publish enough to check.

Llama 3.1

Meta's model card gives GPU-hours on H100-80GB for each model, and the paper gives parameters, tokens and MFU.

The token count for 8B and 70B is listed as "15T+", so the implied MFU for those two is a lower bound: more tokens would push it higher. Meta's reported 38% to 43% steady-state MFU for 405B, applied to 3.79 × 10^25 FLOPs, gives 24.7M to 28.0M GPU-hours, or about 80% to 91% of the 30.84M reported. The remainder is the overhead every real run carries: restarts after interruptions, the long-context stage at lower MFU, and evaluation along the way. The low implied MFU for 8B fits a common pattern: small models struggle to keep a large cluster busy.

This is the most important lesson in the post. If your estimate uses steady-state MFU, add 10% to 25% for everything else, and more if your team has not run at this scale before. Meta did not publish what it paid, and the dollar columns above are what those GPU-hours would cost at today's public list prices, not what Meta spent.

DeepSeek-V3

DeepSeek's report gives a cost table: 2,664K H800 GPU-hours for pretraining, 119K for context extension and 5K for post-training, a total of 2,788K, which it prices at $5.576M assuming $2 per H800 GPU-hour. The report says this covers only the official training run and excludes prior research and ablation experiments on architectures, algorithms and data. It also says pretraining took 180K GPU-hours per trillion tokens, which is 3.7 days per trillion tokens on its 2,048-GPU cluster.

Our MoE method reproduces the shape. At 37B active parameters and 14.8T tokens, 6·N·D is 3.29 × 10^24 FLOPs, about half our 1T MoE Chinchilla case. Dividing by 2,664K GPU-hours gives about 343 TFLOPs of model compute per GPU per second. Two things follow. The $2 per hour assumption is well below the H100 on-demand list prices on major clouds today; the same 2,788K GPU-hours at AWS's $6.88 H100 rate would be about $19.2M. And the exclusion of research and ablations is the same kind of gap as in the Llama check: the headline training run is not the cost of getting there.

What these estimates leave out

• Experiments before the final run. Scaling law sweeps, architecture ablations and data mixture tests. For a frontier model these can rival or exceed the final run; DeepSeek explicitly excludes them from its figure.
• Failed or restarted runs. Loss spikes that force a rollback cost the compute since the last checkpoint.
• Post-training. Supervised fine-tuning, preference tuning and reinforcement learning. DeepSeek reports only 5K GPU-hours for post-training of V3, but that varies enormously between projects.
• Data. Licensing, collection, filtering, deduplication and tokenisation, which need CPU clusters and storage, not accelerators.
• Storage and networking. Checkpoints of a 1T model with optimizer state are about 16 TB each. Writing them often enough to survive interruptions needs a fast parallel file system, and egress fees if you move them between clouds.
• People. The team that keeps a 16,384-GPU job running through 466 interruptions in 54 days is not free.
• Inference. Usually the larger lifetime cost of a model in production. That is covered in our cost per million tokens post.

Common mistakes

1. Using total parameters for an MoE's compute. Compute follows active parameters. A 1T MoE with 50B active costs about the same per token as a 50B dense model, not a 1T one. Memory, on the other hand, follows the total.
2. Using sparse peak as the denominator. The headline figures on several vendor pages assume 2:4 structured sparsity. Dense training does not use it. Halve the sparse figure, or use the dense column directly.
3. Mixing precisions. Quoting MFU against BF16 peak and then multiplying by an FP8 peak double counts the gain. Pick one precision and use its peak consistently.
4. Assuming 100% or even 60% MFU. The best published large-scale figure we found is 55.2%. Most large runs report 38% to 46%, and small or untuned jobs do worse.
5. Forgetting the whole-job overhead. The Llama 3.1 405B check shows steady-state MFU accounts for about 80% to 91% of the GPU-hours Meta reported. Plan for the rest.
6. Comparing prices across different billing models. On-demand, spot, Capacity Blocks and 3-year reservations are different products. A Capacity Block or a reservation price is not an on-demand price, and spot capacity can disappear mid-run.
7. Ignoring memory until the job fails. At 16 bytes per parameter, a 10B model does not fit on one 80 GB GPU with plain data parallelism. Work out the layout before you book the cluster.
8. Treating the cheapest compute per FLOP as the cheapest model. If one platform needs weeks of porting and tuning to reach the MFU you assumed, the engineering time and the delay are part of the cost.
9. Scaling wall clock linearly. Doubling the cluster rarely halves the time, as Meta's 43% to 41% MFU drop shows.
10. Quoting a single number. Every input here has a range. Report a range, and show the inputs, so the reader can rerun it.

FAQ

How much does it cost to train a 1B parameter model?

At Chinchilla-optimal data, about 20 billion tokens, roughly $220 to $880 at the on-demand list prices checked on 25 October 2026, depending on the chip, or about 76 to 97 H100-hours. On 15 trillion tokens, which is closer to how small models are trained today, about $160,000 to $655,000 at list price.

How much does it cost to train a 100B parameter model?

At 2 trillion tokens, about 1.2 × 10^24 FLOPs, the on-demand estimates range from about $2.2M to $8.7M across the chips we priced, or about $1.0M to $5.2M at 3-year commitment prices. On 15 trillion tokens, about $16M to $65M at list price. These figures cover the final run's compute only.

How much would it cost to train a 1 trillion parameter model?

A dense 1T model on 21.2 trillion tokens needs about 1.27 × 10^26 FLOPs: roughly $230M to $924M at on-demand list prices across the chips we priced, or $104M to $550M at 3-year commitment prices. A 1T mixture of experts model with 50B active parameters on the same data needs a twentieth of the compute, roughly $11.5M to $46M on demand.

What is the 6ND formula for training FLOPs?

Training compute is about 6 × N × D FLOPs, where N is the number of parameters used per token and D is the number of training tokens. The forward pass costs about 2 FLOPs per parameter per token and the backward pass about 4. Kaplan and colleagues set it out in their 2020 scaling laws paper.

What is a good MFU for LLM training?

Published large runs report 38% to 46% (Llama 3 405B at 38% to 43%, PaLM 540B at 46.2%), and MegaScale reported 55.2% on 12,288 GPUs. Planning at 35% to 45% is a reasonable range for a well-tuned job; small or new setups often land lower.

How many tokens should I train on?

For the cheapest training run at a given quality, about 20 tokens per parameter, per the Chinchilla paper. If you will serve the model heavily, training a smaller model on many more tokens usually lowers the total cost, which is why models like Llama 3 8B saw about 1,875 tokens per parameter.

How much GPU memory does training need per parameter?

About 16 bytes per parameter for mixed-precision Adam, according to the ZeRO paper: 2 for weights, 2 for gradients and 12 for FP32 master weights and the two Adam moments. Activations come on top and depend on sequence length, batch size and recomputation.

Is it cheaper to train on TPUs, Trainium or AMD GPUs than on NVIDIA?

On list price per peak FLOP, several alternatives are cheaper per chip-hour in our tables, especially with commitments. Whether a finished model is cheaper depends on the MFU your software stack reaches on each, the porting work, and availability at the scale you need. The tables assume the same MFU everywhere, which is the part you have to test.

Does FP8 training halve the cost?

No. FP8 doubles the peak on chips that support it, but not every operation runs in FP8, and published FP8 runs do not report double the delivered throughput of BF16 ones. Treat any saving as something to measure on your own model.

How much electricity does training a large model use?

A compute-optimal 100B model uses roughly 650 to 1,050 MWh in our estimates, and a dense 1T model at Chinchilla tokens roughly 69,000 to 111,000 MWh, including assumed server overhead and a PUE of 1.2. At 8 cents per kWh that is under 2% of the on-demand compute bill for the 1T case.

Where Safeguard fits, and where it does not

Safeguard does not design, program or speed up chips, and it does not lower what you pay per GPU-hour. Nothing in this post depends on Safeguard, and every estimate above stands on its own.

The connection is the software stack that every one of these training runs sits on. A training cluster runs a driver, a CUDA or ROCm or Neuron toolkit, communication libraries, a framework, a training library and a container image, and every one of those is a versioned open source or vendor component with its own security advisories. Our GPU software supply chain post walks through those layers. A month-long run on thousands of accelerators is expensive to interrupt, which makes it tempting to pin everything and never patch; knowing exactly which components and versions are in the image is what lets you make that trade-off on purpose.

What Safeguard does there, as generally available features: it generates and analyses SBOMs in SPDX and CycloneDX, scans repositories, container images and SBOM files for known vulnerable dependencies, shows which projects contain a given component and version, ranks findings with reachability analysis, and can open fix pull requests through autonomous remediation, which ships switched off. AI asset inventory (AI-BOM discovery) is in early access. What Safeguard does not do: it does not scan GPU firmware or host kernel drivers that live outside the images and SBOMs you give it, and it does not make the training run faster or cheaper.

A training image pulls framework wheels, communication libraries and serving dependencies, and each rebuild for a new run pulls them again. The same supply chain hygiene that applies to any service applies to the image you are about to run on a cluster costing thousands of dollars an hour.

More in this series

Other posts in our AI chips and costs series:

• AI Accelerator TCO: Buy vs Rent, Capex, Power, Cooling and Utilization
• AI Chips Compared 2026: B200, H200, MI355X, Gaudi 3, TPU, Trainium
• Cheapest LLM Inference: How to Cut Cost per Million Tokens
• Fastest LLM Inference: Lowest Latency and Most Tokens per Second
• How Many GPUs Does a 1 Trillion Parameter Model Need?
• How to Estimate AI Training and Inference Cost: Formulas and a Calculator
• LLM Inference Cost by Chip and Model Size, 1B to 1 Trillion Parameters

Fix it without waiting for upstream

Whatever the vulnerability, most of the risk sits in two gaps. The first is the time between a fix being published and that fix actually running in your production builds. The second, and worse, is the time before any fix exists at all, while a maintainer works out a patch and a release schedule. Waiting through both gaps is a habit, not a requirement. Safeguard gives teams three ways to close them.

1. Autonomous remediation: pull requests, not tickets

When a fixed version exists, Safeguard's Griffin models open a tested, policy gated pull request that moves the affected dependency to the safe version. You can run it on one repository or in bulk across every project that carries the vulnerable version. Remediation covers eight categories, including dependencies, licenses, security posture and findings. Reachability analysis ranks the vulnerable code paths your application actually calls first, so the first pull requests you merge are the ones that matter most. Autonomous remediation ships switched off, with every category set to Manual, so your team decides how much runs on its own and what still needs a human approval.

2. Gold Open Source: the safe version in seconds, free

gold.safeguard.sh is free and needs no login. Search any CVE or package to see affected and fixed versions, CVSS, EPSS, CISA KEV status and a recommended safe version, or use the "Am I affected?" checker. The same data is available through a free read API, README status badges and RSS threat feeds. The safeguard-sh/gold-scan-action GitHub Action can fail a build on a known exploited vulnerability, on any known vulnerability, or never, so a version you have just removed cannot quietly come back in next week's merge.

3. Gold Registry: start from components that are already fixed

registry.safeguard.sh provides hardened, drop in replacements for upstream packages and container images. Every Gold artifact is held continuously to zero exploitable critical or high CVEs, zero malware and a completed license check, with SLSA L3 or higher provenance, a Sigstore signature and a public evidence page you can hand to an auditor. An artifact loses its Gold badge if it regresses. Teams can point their own package directory at the registry so hardened equivalents resolve without changing how developers install anything.

When upstream has no fix yet, or will not ship one, Safeguard first works to get the fix accepted upstream. If the issue is severe or upstream will not move, Safeguard builds a fork, fixes the vulnerability there and publishes a hardened Gold build, so you are not stuck on a maintainer's release schedule. You can also request a hardened, compatibility tested build of a specific package. This fork and patch flow is in early access today, so ask which packages it covers before you plan around it.

One honest limit: when the vulnerable code lives in a vendor appliance, a hosted service or an operating system runtime you do not build yourself, the fix still has to come from that vendor or distribution. The steps above apply to the open source dependencies and container images you build and ship, which is where most teams carry the largest and least tracked share of their exposure.

See how Safeguard would help

You can download Safeguard and follow the CLI docs to try these yourself.

The first demo sets up the Safeguard runner on your own machine, approves it with its device code and then scans from the terminal. For a training team, that is how you would scan the container image a run will use, and the repository that builds it, without sending source or images off the machine, for example with safeguard scan --image against the training image pinned by digest.

Click through it at your own pace, or open it full screen.

The second demo is a real CLI session: install, open the interactive terminal and ask it to fix a lodash CVE. The assistant reads the finding and the advisory, asks before any command that changes something, edits package.json to the fixed version, runs npm install and npm test with approval, prints a summary and pushes nothing. The same flow applies to the Python side of a training stack, such as moving a pinned framework or data loading dependency to a fixed version and rerunning your tests before the next run starts.

Click through it at your own pace, or open it full screen.

The third demo connects Claude to the Safeguard MCP server with OAuth and shows how to choose which tools Claude can use. When a new advisory lands for a library in your training image, an engineer can ask which projects contain the affected component and version, and get the answer from their own scan data.

Click through it at your own pace, or open it full screen.

Quick check

Sources

Papers and reports:

• Kaplan et al., Scaling Laws for Neural Language Models (arXiv 2001.08361, January 2020)
• Hoffmann et al., Training Compute-Optimal Large Language Models (Chinchilla, arXiv 2203.15556, March 2022)
• Chowdhery et al., PaLM: Scaling Language Modeling with Pathways (arXiv 2204.02311, April 2022)
• Jiang et al., MegaScale: Scaling Large Language Model Training to More Than 10,000 GPUs (arXiv 2402.15627, February 2024)
• Llama Team, Meta, The Llama 3 Herd of Models (arXiv 2407.21783, July 2024)
• Meta, Llama 3.1 model card (training GPU-hours and power)
• DeepSeek-AI, DeepSeek-V3 Technical Report (arXiv 2412.19437, December 2024)
• Rajbhandari et al., ZeRO: Memory Optimizations Toward Training Trillion Parameter Models (arXiv 1910.02054, October 2019)
• Korthikanti et al., Reducing Activation Recomputation in Large Transformer Models (arXiv 2205.05198, May 2022)
• Narayanan et al., Efficient Large-Scale Language Model Training on GPU Clusters Using Megatron-LM (arXiv 2104.04473, April 2021)

Chip specifications:

• NVIDIA H100 product page and specifications
• NVIDIA H200 product page and specifications
• NVIDIA HGX platform specifications (HGX B200 and HGX B300)
• NVIDIA GB200 NVL72 specifications
• NVIDIA DGX B200 specifications
• AMD Instinct MI300X specifications
• AMD Instinct MI325X specifications
• AMD Instinct MI355X specifications
• Intel Gaudi architecture documentation (Gaudi 3)
• Google Cloud TPU v5e documentation
• Google Cloud TPU v5p documentation
• Google Cloud TPU v6e (Trillium) documentation
• Google Cloud TPU7x (Ironwood) documentation
• AWS Neuron documentation: Trainium2 architecture

Prices, all checked 25 October 2026:

• AWS EC2 On-Demand pricing (p5, p5en, p6-b200, p6-b300; US East, read from the page's published Linux price data for N. Virginia, Ohio and Oregon)
• AWS EC2 Capacity Blocks for ML pricing (trn2.48xlarge, p5.48xlarge)
• Azure Retail Prices API, queried directly (ND96isr H100 v5, ND96isr H200 v5, ND96isr MI300X v5, ND128isr GB200 v6; pay-as-you-go, spot and reservations)
• Google Cloud TPU pricing (on-demand, DWS and committed use)

Last reviewed: 25 October 2026. Prices change often; recheck the linked price pages before relying on any figure here.