How Many GPUs Does a 1 Trillion Parameter Model Need?

Just to hold the training state of a dense 1 trillion parameter model, with mixed precision and the Adam optimizer, you need about 16 TB of accelerator memory before you store a single activation. That is at least 87 GB200 class GPUs, 200 H100s or 78 TPU Ironwood chips. But memory sets only the floor. The real number comes from time: about 20 trillion training tokens is roughly 1.2 × 10^26 FLOPs, which works out to around 15,000 GB200 GPUs or 38,000 H100s for a 90 day run at published utilisation levels. A 1 trillion parameter mixture of experts (MoE) model with 32 billion active parameters needs about 40 times less training compute, so a few hundred to a few thousand GPUs will do. For serving, the dense model needs about 5 to 24 accelerators per replica just to hold its weights and cache, depending on the chip and the precision. The MoE model needs a similar number of GPUs to hold its weights, but each replica produces far more tokens for the money.

This post works through the whole 1 trillion parameter case from first principles: what has to fit in memory, how the work is split across GPUs, how fast the links between them must be, how big the cluster gets, how many megawatts it draws, and what it costs at public cloud prices. Every computed number below came from a python3 calculation, and the scripts are in the post so you can rerun them with your own assumptions. Chip specifications come from vendor pages, measured efficiency figures come from peer reviewed or primary technical reports, and prices come from public price pages checked on 25 October 2026. All costs are estimates built from stated assumptions, not quotes. The code in this post is plain Python arithmetic and ran on a Mac; it does not touch a GPU.

Key takeaways

• Memory floor for dense training is about 16 TB. Mixed precision Adam keeps about 16 bytes per parameter (2 for FP16 weights, 2 for gradients, 12 for FP32 master weights, momentum and variance), as derived in the ZeRO paper. One GB200 NVL72 rack has 13.4 TB of HBM in total, so a dense 1T training job does not fit in one rack even before activations.
• Time, not memory, sets the cluster size. At 6 × N × D FLOPs with N = 1 trillion and D = 20 trillion tokens, training is 1.2 × 10^26 FLOPs. At 45% model FLOPs utilisation (MFU), that is about 30 million GB200 GPU hours at BF16, or about 15,000 GPUs for 90 days.
• MoE changes the bill, not the memory. A 1.04 trillion parameter MoE model with 32 billion active parameters (the published Kimi K2 shape) trained on 15.5 trillion tokens needs about 3.0 × 10^24 FLOPs, about 40 times less than the dense case. It still needs about 16 TB for its training state.
• Parallelism follows the network. Tensor parallelism stays inside an NVLink or ICI domain because it talks hundreds of times per step. Pipeline parallelism crosses nodes because it sends little. Data parallelism crosses the whole cluster but can overlap with compute. Expert parallelism wants an all to all fabric, which is why 72 GPU NVLink domains matter so much for MoE.
• Serving a dense 1T model is memory bandwidth bound. Each decode step must read about 1 TB of FP8 weights. On a GB200 NVL72 with 576 TB/s of aggregate HBM bandwidth, the floor is about 2.9 ms per step at batch 32 with a 32K context, which caps one replica at about 11,000 tokens per second.
• Serving a 1T MoE model is much cheaper per token. At large batch every expert is read once per step anyway, but the compute per token is 32 billion parameters, not 1 trillion, so one NVL72 replica can reach a bandwidth bound of about 220,000 tokens per second at batch 512 with an 8K context.
• Cost ranges are wide on purpose. At AWS Capacity Block rates, dense 1T training lands between about $128 million and $700 million on NVIDIA parts depending on chip, precision and MFU. The MoE equivalent lands between about $5 million and $26 million. These are estimates, not quotes.
• Power is tens of megawatts. A 15,000 GPU Blackwell class cluster at about 1.8 kW per GPU (system level) and a PUE of 1.2 draws about 33 MW at the facility.

The two models we are sizing

"1 trillion parameters" describes two very different machines, so we size both.

Model A, dense 1T. Every parameter takes part in every token. We use the configuration that the Megatron-LM team published for their 1 trillion parameter GPT model in 2021: 128 transformer layers, hidden size 25,600 and 160 attention heads. A quick check of 12 × L × h², the usual parameter count for a transformer block stack, gives 1,006.6 billion parameters, matching the paper's 1,008 billion. For inference we also need key value (KV) cache sizes, and the 2021 configuration uses full multi head attention, which makes the KV cache very large. A modern dense model would use grouped query attention. We therefore assume 16 KV heads of dimension 160 for serving, and we show the multi head case only to make the point about why nobody serves it that way.

Model B, MoE 1T. Each token is routed to a few experts, an idea that reached the trillion scale in Google's Switch Transformers paper, which described a 1.6 trillion parameter sparse model (Switch-C, 1,571 billion parameters in its table). Because each token touches only a few experts, most parameters sit idle for any given token. We use the published shape of Kimi K2: 1.04 trillion total parameters, 32.6 billion activated per token, 61 layers (one dense, 60 MoE), 384 experts with 8 active per token plus 1 shared expert, 64 attention heads, hidden size 7,168, and Multi head Latent Attention (MLA). It was pre trained on 15.5 trillion tokens. For its KV cache we assume MLA latent sizes equal to DeepSeek-V3's published ones (a 512 dimension compressed KV vector and a 64 dimension decoupled rotary key per layer), because the K2 report text we read compares K2 to DeepSeek-V3 but does not restate those two numbers.

The last row is the heart of the matter. A dense 1T model spends about 31 times more arithmetic per token than a 32B active MoE model, in both training and inference. The weights you must keep in memory are about the same.

The hardware we are sizing on

All figures below are vendor peak figures. We mark them dense (no structured sparsity) or sparse. Where a vendor prints a sparse figure and states that dense is half, we compute the dense figure and say so.

Three notes on that table. First, NVIDIA's GB200 NVL72 page says that tensor core figures are "in sparse" and that dense is one half of the sparse figure; the DGX B200 page says the same for its FP8 line. Second, AMD lists dense and structured sparsity figures separately, so we use AMD's dense numbers directly. Third, Google's TPU documentation does not describe its figures as sparse or dense; it lists one peak per precision per chip, which we treat as dense.

Pod and rack sizes matter as much as chips:

• GB200 NVL72: 72 Blackwell GPUs and 36 Grace CPUs in one liquid cooled rack, one 72 GPU NVLink domain, 13.4 TB of HBM3E and 576 TB/s of HBM bandwidth in total.
• HGX or DGX B200: 8 GPUs per server, 1,440 GB of HBM, about 14.3 kW maximum system power, and eight ConnectX-7 ports at up to 400 Gb/s for scale out.
• TPU v5p pod: 8,960 chips on a 3D torus.
• TPU Ironwood pod: 9,216 chips.

                 Scale up domain (fast, small)           Scale out network (slower, huge)
  HGX B200     [ 8 GPUs | NVLink 5 | 1.44 TB HBM ]  ==>  8 x 400 Gb/s NICs per server
  GB200 NVL72  [ 72 GPUs | NVLink 5 | 13.4 TB HBM ] ==>  InfiniBand or Ethernet between racks
  TPU v5p      [ up to 8,960 chips on ICI 3D torus ] ==> data center network, 50 Gb/s per chip
  TPU Ironwood [ up to 9,216 chips on ICI ]          ==> data center network, 100 Gb/s per chip

The single most important number for parallelism planning is the ratio between the scale up link and the scale out link. On a DGX B200, each GPU has 1.8 TB/s of NVLink bandwidth in total, which we read as about 900 GB/s in each direction, and one 400 Gb/s network port, which is 50 GB/s. That is an 18 to 1 ratio. Everything that talks often must stay on the fast side of that line.

Part 1: training memory

Model states: the 16 bytes per parameter rule

The ZeRO paper (Rajbhandari and others, 2020) gives the standard accounting for mixed precision training with Adam. For a model with Ψ parameters you keep 2Ψ bytes of FP16 weights, 2Ψ bytes of FP16 gradients, and KΨ bytes of optimizer state, where K = 12 for mixed precision Adam (an FP32 copy of the weights plus FP32 momentum and variance). The total is 16Ψ bytes. For Ψ = 1 trillion that is 16 TB.

FP8 training does not change this much. DeepSeek-V3's FP8 framework keeps master weights and weight gradients in FP32 and stores its optimizer states in BF16 rather than FP32, so its state is somewhat below 16 bytes per parameter; most of the FP8 savings show up in activations, communication and compute speed. We keep 16 bytes per parameter as a conservative planning number for both models. For the MoE model it is 16.6 TB, because it has 1.04 trillion parameters, all of which must be trained even though each token touches only a few experts.

Here is the minimum number of devices just to hold 16 TB, first at 100% of HBM and then at a more realistic 90%:

# Model state memory floor for 1T parameters (16 bytes per parameter, ZeRO accounting)
import math
GB = 1e9
N = 1e12
states = 16 * N
devices = {"H100": 80, "H200": 141, "B200": 180, "GB200": 186, "MI355X": 288,
           "TPU v5p": 95 * 1.073741824, "TPU Ironwood": 192 * 1.073741824}
for name, hbm in devices.items():
    print(name, math.ceil(states / (hbm * GB)), math.ceil(states / (0.9 * hbm * GB)))
print("NVL72 total HBM, TB:", 72 * 186 / 1000)   # 13.392, less than 16

The punchline is the last line. One GB200 NVL72 rack holds 13.4 TB of HBM, and 16 TB is more than that. A dense 1T model's full training state cannot live in one rack. You need at least two racks or you must offload optimizer state to CPU memory. The NVL72 rack does carry 17 TB of LPDDR5X attached to its Grace CPUs, which is one reason CPU offload is less painful on Grace systems than on x86 servers with PCIe links, but offload still costs time.

Activations: the term people forget

The second consumer of memory is activations, which are kept from the forward pass for the backward pass. Korthikanti and others (2022) derived the activation memory for one transformer layer as s·b·h·(34 + 5·a·s/h) bytes in 16 bit precision, where s is sequence length, b is micro batch size, h is hidden size and a is the number of attention heads.

For Model A with s = 2,048, b = 1, h = 25,600 and a = 160, the bracket is 34 + 64 = 98, and one layer needs about 5.14 GB of activations. With 128 layers, a single sequence would need over 650 GB with no tricks. Three tricks bring this down:

1. Tensor parallelism with sequence parallelism. Splitting the layer across t GPUs divides the stored activations by t.
2. Selective recomputation. Recomputing the attention softmax and dropout, rather than storing them, removes the 5·a·s/h term. The paper shows the per layer memory becomes 34·s·b·h/t.
3. Pipeline parallelism. Each GPU holds only its stage's layers, but with the 1F1B schedule the first stage keeps activations for up to p micro batches in flight, which the paper shows adds up to the same total as storing all L layers for one micro batch.

With t = 8, selective recomputation gives about 0.22 GB per layer, so the first pipeline stage holds about 28.5 GB of activations. Full recomputation would cut that to 2·s·b·h per layer, about 13.4 GB in total, at the price of a third more compute.

# Activation memory per layer and per first pipeline stage (Korthikanti et al. 2022)
L, h, a, s, b, t = 128, 25600, 160, 2048, 1, 8
sbh = s * b * h
full = sbh * (34 + 5 * a * s / h)          # bytes, no parallelism, no recompute
selective = sbh * 34 / t                   # TP + sequence parallel + selective recompute
print(full / 1e9, selective / 1e9, selective * L / 1e9)   # 5.14, 0.223, 28.5 GB

Putting the memory together: the Megatron layout

The Megatron-LM paper trained its 1T model on 3,072 A100 GPUs with tensor parallelism 8, pipeline parallelism 64 and data parallelism 6 (8 × 64 × 6 = 3,072). With 512 way model parallelism, each GPU holds 16 TB ÷ 512 = 31.25 GB of model state. Add about 28.5 GB of activations on the first stage, plus buffers, and the job fits in an 80 GB A100. The paper reports 163 teraFLOP/s per GPU, 52% of the A100's peak, and 502 petaFLOP/s in aggregate.

On today's parts the same idea gives a lot more headroom. With 180 to 192 GB per device you can halve the pipeline depth, use larger micro batches or longer sequences, or shard optimizer state across data parallel ranks with ZeRO stage 1. ZeRO stage 1 alone takes optimizer state from 12Ψ to 12Ψ ÷ Nd bytes per GPU, and the paper notes the total model state falls toward 4Ψ when the data parallel degree is large.

Part 2: training compute and cluster size

The FLOP count

Kaplan and others (2020) give the estimate C ≈ 6·N floating point operations per training token, for N non embedding parameters: 2N for the forward pass and 4N for the backward pass. For a whole run, C ≈ 6·N·D. Hoffmann and others (2022), the Chinchilla paper, found that compute optimal training scales tokens in proportion to parameters, and their headline model used 70 billion parameters on 1.4 trillion tokens, which is 20 tokens per parameter. Their Table 3 lists 21.2 trillion tokens and 1.27 × 10^26 FLOPs for a 1 trillion parameter model.

We use 20 tokens per parameter as a stated assumption for Model A:

• Model A: C = 6 × 10^12 × 2 × 10^13 = 1.2 × 10^26 FLOPs.
• Model B: C = 6 × 32 × 10^9 × 15.5 × 10^12 = 2.98 × 10^24 FLOPs, using the active parameter count and K2's published token count.

The ratio is about 40. Note that Model B used about 15 tokens per total parameter but about 484 tokens per active parameter. MoE models are trained far past the dense compute optimal point measured on active parameters, because active parameters set the compute bill while total parameters set the capacity.

Two caveats. 6ND ignores attention FLOPs, which grow with sequence length, so it slightly undercounts. And it counts model FLOPs only; activation recomputation adds hardware FLOPs that MFU deliberately does not credit.

What MFU should you assume?

Model FLOPs utilisation is the fraction of peak that turns into useful model FLOPs. Published figures for large dense training runs:

So for a dense model we use 38%, 45% and 55% as low, middle and high. For MoE models there are fewer published MFU figures. We can derive an implied one from DeepSeek-V3, which reports 2.664 million H800 GPU hours for pre training on 14.8 trillion tokens with 37 billion active parameters. 6 × 37 × 10^9 × 14.8 × 10^12 = 3.29 × 10^24 FLOPs, divided by 2.664 million hours, is about 343 TFLOP/s per GPU. If we assume the H800 has the same dense tensor peak as the H100 SXM (we could not find a current H800 spec sheet on NVIDIA's site, so this is an assumption), that is about 35% of the dense BF16 peak and about 17% of the dense FP8 peak, even though DeepSeek trained in FP8. MoE all to all traffic and load imbalance cost real efficiency. We therefore use 25% and 35% of the dense BF16 peak for Model B, and we show the FP8 case for comparison while noting it is optimistic.

GPU hours and cost

The prices below are AWS EC2 Capacity Blocks for ML effective hourly rates per accelerator in US East (N. Virginia) or the listed region, and Google Cloud TPU on demand prices per chip hour, all checked on 25 October 2026. Capacity Block rates are reservations for a fixed window, not on demand prices; no public price was available for the MI355X on the pages we checked, so we show its hours without a cost.

# Training GPU hours and cost: hours = C / (dense_peak * MFU * 3600)
C_dense = 6 * 1e12 * 20e12          # 1.2e26
C_moe   = 6 * 32e9 * 15.5e12        # 2.976e24
chips = {  # name: (dense BF16 TFLOPS, dense FP8 TFLOPS, $ per device hour or None)
    "H100 SXM": (989.5, 1979, 5.970),
    "H200":     (989.5, 1979, 7.895),
    "B200":     (None, 4500, 14.208),
    "GB200":    (2500, 5000, 10.582),
    "MI355X":   (2500, 5000, None),
    "TPU v5p":  (459, 459, 4.20),
    "Ironwood": (2307, 4614, 12.00),
}
for name, (bf16, fp8, price) in chips.items():
    for prec, peak in (("BF16", bf16), ("FP8", fp8)):
        if peak is None:
            continue
        for mfu in (0.38, 0.45, 0.55):
            hours = C_dense / (peak * 1e12 * mfu * 3600)
            cost = f"${hours * price / 1e6:,.0f}M" if price else "no public price"
            print(f"{name:9} {prec} MFU {mfu:.2f}: {hours / 1e6:6.1f}M device hours, {cost}")

Selected results for Model A, the dense 1T on 20 trillion tokens:

Google's 3 year commitment prices are 45% of on demand for both TPU generations ($5.40 ÷ $12.00 for Ironwood, $1.89 ÷ $4.20 for v5p), which moves Ironwood BF16 at 45% MFU from $385 million to about $173 million. Long reservations for GPUs work similarly. Do not compare an on demand TPU number with a reserved GPU number or the reverse.

And for Model B, the MoE 1T on 15.5 trillion tokens:

A sanity check against a real report: DeepSeek-V3, a 671B total, 37B active MoE, reports 2.788 million H800 GPU hours for its full training. Our H100 BF16 row at 25% MFU says 3.34 million hours for a model with slightly fewer active parameters and slightly more tokens. Same order of magnitude, which is all a planning model can promise.

Two more caveats belong next to these tables. First, they cover one clean pre training run. Real projects also pay for ablations, failed runs, restarts, long context extension, supervised fine tuning and reinforcement learning, and none of that is here. Second, they assume you can get the capacity at all, and 15,000 GPUs is not a self service order.

Cluster size for a target wall clock

GPU hours become a cluster size once you pick a schedule. We assume 90 days and 90% goodput, the share of wall clock time doing useful steps rather than restarting from checkpoints. That goodput is an assumption; the Llama 3 paper reports interruptions during a 54 day period of 405B pre training and attributes about 78% of unexpected interruptions to confirmed or suspected hardware issues, so 90% is not automatic at this scale.

wall_hours = 90 * 24 * 0.9      # 1,944 useful hours
for name, hours in [("H100 BF16 45%", 74.86e6), ("GB200 BF16 45%", 29.63e6),
                    ("GB200 FP8 45%", 14.81e6), ("Ironwood BF16 45%", 32.11e6),
                    ("TPU v5p BF16 45%", 161.38e6)]:
    n = hours / wall_hours
    print(f"{name}: {n:,.0f} devices, {n / 72:.0f} NVL72 sized groups")

For Model B on GB200 at BF16 and 30% MFU, the same 90 days need only about 570 GPUs, eight NVL72 racks. Squeezing it into 30 days takes about 1,700. On H100 at 30% MFU it is about 1,430 GPUs for 90 days. That is why MoE has made trillion parameter models something many more labs can train.

The Megatron paper offers a cross check. It estimated 84 days to train its 1T model on 450 billion tokens with 3,072 A100s at 163 TFLOP/s per GPU, using a FLOP formula that also counts attention and activation recomputation. Plain 6ND on the same inputs gives about 63 days, so the paper's own estimate is about a third higher than 6ND would suggest, which matches the extra work its formula counts.

Part 3: parallelism layouts for training

There are five ways to split the work, and a trillion parameter run uses most of them at once.

Layout 1: dense 1T on 8 GPU HGX servers

This is the Megatron and Llama 3 recipe. Put TP = 8 inside each server, where NVLink connects all eight GPUs. Stack pipeline stages across servers. Replicate the whole pipeline with data parallelism.

 DP replica 1                                DP replica 2 ... DP replica d
 +------------------- pipeline (PP = 16 to 64 stages) -------------------+
 | server 1: TP8 | server 2: TP8 | server 3: TP8 | ... | server p: TP8   |
 |  layers 1-8   |  layers 9-16  |  layers 17-24 |     |  last layers   |
 +-----------------------------------------------------------------------+
     NVLink inside each box          InfiniBand or Ethernet between boxes

Llama 3 405B used TP 8, PP 16 and DP 64 on 8,192 GPUs, and TP 8, CP 16, PP 16, DP 8 for its long context stage on 16,384 GPUs. A dense 1T model on B200 servers would look similar with more pipeline stages or more memory per stage. The weak point is that TP cannot grow past 8 without crossing the slow network.

Layout 2: dense 1T on GB200 NVL72 racks

With 72 GPUs in one NVLink domain, TP can stay at 8 while pipeline stages that share a rack talk over NVLink rather than the network. A natural layout puts TP 8 × PP 9 inside one rack, so a 72 layer slice runs entirely on NVLink, and then extends the pipeline across two racks for a model whose state needs more than 13.4 TB. Data parallelism then spans racks over InfiniBand or Ethernet.

 Rack A (72 GPUs, one NVLink domain)        Rack B (72 GPUs)
 [TP8|TP8|TP8|TP8|TP8|TP8|TP8|TP8|TP8]  ->  [TP8|TP8|TP8|TP8|TP8|TP8|TP8|TP8|TP8]
   PP stages 1..9 over NVLink           IB    PP stages 10..18 over NVLink
 x about 106 such rack pairs for the GB200 FP8 cluster above, joined by data parallelism

The extra benefit is not raw speed but freedom: you can choose TP 16 or larger for layers that need it, or use a 72 way expert parallel group for MoE, without paying the network penalty.

Layout 3: TPU pods

A TPU v5p pod connects 8,960 chips on a 3D torus with 1,200 GB/s of bidirectional ICI bandwidth per chip, and Google's documentation says that slices of 4 × 4 × 4 chips or larger have full torus wraparound links. On TPUs the usual approach is to describe a device mesh and let the XLA compiler insert the collectives. A dense 1T model on Ironwood might use a mesh such as data × fully sharded data × tensor, with tensor parallelism mapped to one torus axis and FSDP to the others. Because ICI spans the whole pod, the hard boundary moves from "8 GPUs" or "72 GPUs" to "one pod". Our Ironwood estimate of about 16,500 chips for 90 days is more than one pod of 9,216, so the job would span two pods joined by the data center network, with only data parallel traffic crossing it.

The PaLM 540B paper is the reference for this pattern: it trained across two TPU v4 pods, 6,144 chips in total, using data parallelism between pods, and reported 46.2% MFU.

Layout 4: MoE 1T

DeepSeek-V3, the most detailed public MoE training report, used 16 way pipeline parallelism, 64 way expert parallelism spanning 8 nodes, and ZeRO stage 1 data parallelism on a 2,048 H800 cluster. It also reports that on its cluster NVLink offered 160 GB/s and InfiniBand 50 GB/s, so it restricted each token to at most four nodes to keep all to all traffic bounded. On an NVL72 system, a 64 or 72 way expert parallel group fits inside one NVLink domain, so the all to all never touches the network. With 384 experts and EP 64, each GPU hosts 6 experts per MoE layer; with EP 96, it hosts 4.

Part 4: interconnect bandwidth, worked

Here are four worked cases that show why each kind of parallelism lands where it does.

Case 1: data parallel gradient all reduce, dense 1T

In the Megatron layout, each GPU holds 1T ÷ 512 = 1.95 billion parameters, so 3.9 GB of BF16 gradients. A ring all reduce sends and receives about twice the buffer size per GPU, so about 7.8 GB per step. At 50 GB/s (one 400 Gb/s port per GPU), that is 0.16 seconds.

How long is a step? The paper's global batch was 3,072 sequences of 2,048 tokens, 6.29 million tokens. At 6ND that is 3.77 × 10^19 FLOPs per step, and at 3,072 GPUs × 163 TFLOP/s, a step takes about 75 seconds. So the gradient all reduce is 0.2% of the step and easy to overlap with the backward pass. Data parallelism can cross the slow network.

Case 2: pipeline boundary traffic

At each stage boundary, one micro batch sends s·b·h activations forward and the same size of gradients back: 2,048 × 1 × 25,600 × 2 bytes = 105 MB. With 512 micro batches per pipeline per step (3,072 ÷ 6 data parallel replicas), each stage moves about 107 GB per step, about 1.4 GB/s averaged over a 75 second step. Again, the network is fine. The paper itself reports an effective bisection bandwidth of 892 GB/s for pipeline parallel traffic and 13 TB/s for data parallel traffic across the whole 3,072 GPU job.

Case 3: tensor parallel all reduce during decode, dense 1T

Inference changes the picture. During decode, every layer needs two all reduces of the activations (after attention and after the MLP). With hidden size 25,600, BF16 activations and batch 32, each message is 1.64 MB, and there are 256 of them per token step. A ring all reduce over 8 GPUs moves 2 × 7/8 of the message per GPU, so each GPU sends about 734 MB per step.

At batch 1 the message is only 51 KB and bandwidth stops mattering; latency does. 256 sequential collectives per token, each costing a few microseconds at best, set a floor on per token latency that no amount of HBM bandwidth removes. That is the honest case for big NVLink domains in inference: they keep the many small collectives off the network.

Case 4: MoE all to all

For Model B, each token at each MoE layer is dispatched to 8 experts and the results are combined back. Assume FP8 dispatch (7,168 bytes per expert copy), which DeepSeek-V3 describes for its own MoE training, and BF16 combine (14,336 bytes per copy), which is our assumption. That is about 172 KB per token per MoE layer, or 10.3 MB per token across 60 MoE layers, in each of the forward and backward passes.

A GB200 GPU training Model B at 30% of its 2.5 PFLOPS dense BF16 peak processes about 3,900 tokens per second (2.5 × 10^15 × 0.3 ÷ (6 × 32 × 10^9)). If every expert lived on another GPU, that would need about 81 GB/s of all to all traffic per GPU. That is 1.6 times a 400 Gb/s port and under a tenth of NVLink 5's one direction bandwidth. On H100 at the same MFU the figure is about 32 GB/s per GPU, which fits a 400 Gb/s port on paper, but real all to all traffic is bursty and uneven. DeepSeek-V3's node limited routing and overlap scheduling exist because of this pressure.

d, k, moe_layers = 7168, 8, 60
per_token = (d * k * 1 + d * k * 2) * moe_layers          # FP8 dispatch + BF16 combine, bytes
for name, peak in (("GB200", 2500e12), ("H100", 989.5e12)):
    tokens_per_s = peak * 0.30 / (6 * 32e9)
    print(name, round(tokens_per_s), "tok/s,", tokens_per_s * per_token * 2 / 1e9, "GB/s all to all")

Part 5: serving a 1T model

Weights at each precision

Block scaled 4 bit formats store a scale factor per block of values, and the overhead depends on block size and scale width; the last row shows the arithmetic for one 8 bit scale per 16 values as an example, not a vendor specification.

The KV cache

The KV cache stores the keys and values of every token in every active sequence, for every layer. For grouped query attention the size per token is 2 × layers × kv_heads × head_dim × bytes. Pope and others (2022) showed on PaLM 540B that multi query attention, where many query heads share one key value head, cuts this memory enough to support up to 32 times longer contexts; grouped query attention and MLA are later points on the same trade off.

This table explains most of what is surprising about serving. For a dense model with ordinary GQA, the KV cache at 32 users with 32K context is as large as the FP8 weights. With full multi head attention it would be thirteen times larger, which is why the original 2021 configuration is a training artefact, not a serving design. MLA shrinks the cache by roughly 19 times compared with our GQA assumption, which is a large part of why MoE models with MLA are cheap to serve at long context.

Minimum devices per replica

Total memory is weights plus KV cache. Assuming 90% of HBM is usable (the rest goes to activations, communication buffers and fragmentation):

In practice you round up to the parallelism you can use: 16 H100s or H200s (two HGX servers), 8 or 16 B200s, one NVL72 rack, 8 MI355X, or a TPU slice. An 8 × H200 server has 1,128 GB of HBM, which holds 1 TB of FP8 dense weights with almost nothing left for KV cache, so a dense 1T model on H200 realistically means two servers joined by the network. An 8 × MI355X server has 2,304 GB, which holds the dense FP8 case in one box. An 8 × B200 server holds the MoE FP8 case comfortably.

Decode: the memory bandwidth bound

In the decode phase, each step generates one token per sequence in the batch. The GPUs must read every weight they use and every sequence's KV cache from HBM. If they can read at most B_total bytes per second, a step cannot be shorter than (weight bytes + KV bytes) ÷ B_total. This gives an upper bound on throughput; real systems reach some fraction of it.

def decode_bound(weight_bytes, kv_bytes, batch, hbm_bw):
    step = (weight_bytes + kv_bytes) / hbm_bw      # seconds per decode step, lower bound
    return step, 1 / step, batch / step             # step, tokens/s per user, tokens/s per replica

kv_dense = 2 * 128 * 16 * 160 * 2                   # 1.31 MB per token, BF16
step, per_user, replica = decode_bound(1e12, kv_dense * 32768 * 32 * 0.5, 32, 576e12)
print(step * 1e3, per_user, replica)                # 2.93 ms, 341 tok/s, 10,925 tok/s

Every row needs far fewer FLOPs than the hardware offers, which confirms that decode is bandwidth bound in all of them. The NVL72 rows look so much faster than the 8 and 16 GPU rows because a replica on NVL72 spreads the same weights across 72 GPUs' worth of HBM bandwidth. That is a real effect, not a trick, but it has a cost: you are paying for 72 GPUs per replica.

Remember that these are ceilings. They ignore the collectives from Part 4, kernel launch overheads, scheduling, and the fact that a batch is never perfectly full. Treat them as the number you can never beat, not the number you will see.

The MoE subtlety: how many experts does a batch touch?

At batch 1, a MoE layer reads only 8 of 384 routed experts, so the step reads about 32 GB of active weights rather than 1 TB. As the batch grows, different tokens choose different experts. If routing were uniform and independent, the fraction of routed experts touched in a layer would be 1 - (1 - 8/384)^B:

So at production batch sizes a MoE model reads nearly all its weights every step, just like a dense model. What it saves is arithmetic: each token needs 6.4 × 10^10 FLOPs instead of 2 × 10^12. That is why it can run a batch of 512 at about the same step time as a dense model at batch 32, and why its throughput per replica is about 20 times higher in our table. Real routers are not uniform, so this is a model, not a measurement.

Prefill: the compute bound

Prefill processes the whole prompt in one pass, and it is compute bound. At about 2 × N_active FLOPs per token and an assumed 50% MFU:

These ignore attention's quadratic term, which matters at long context.

Disaggregated prefill and decode

Because prefill is compute bound and decode is bandwidth bound, running both on the same GPUs means each phase interferes with the other. Splitwise (Patel and others, 2024) and DistServe (Zhong and others, 2024) separate them onto different machines and move the KV cache between them. Splitwise reports up to 1.4 times higher throughput at 20% lower cost, or 2.35 times more throughput under the same power and cost budgets, in its own evaluation. DeepSeek-V3 describes a production version: prefill units of 4 nodes and 32 GPUs, with TP 4 plus sequence parallelism for attention and 32 way expert parallelism for MoE; decode units of 40 nodes and 320 GPUs, with TP 4 for attention combined with DP 80, and EP 320 for MoE, where each GPU hosts one expert.

           prompt                         KV cache transfer            generated tokens
 client --------> [ PREFILL POOL ] ===============================> [ DECODE POOL ] ------> client
                   compute bound                                        bandwidth bound
                   large TP or EP                                       huge batch, wide EP
                   e.g. 32 GPUs per unit                                e.g. 320 GPUs per unit

The KV cache transfer is the price of disaggregation. For Model A at 1.31 MB per token, an 8K prompt carries 10.7 GB of KV cache; at 50 GB/s that is about 0.21 s, longer than the 0.09 s prefill itself on NVL72. For Model B with MLA, the same prompt carries 0.58 GB, about 12 ms at 50 GB/s. This is one more reason MLA and MoE pair so well with disaggregated serving.

Cost per million tokens

Cost per million output tokens = replica price per hour ÷ million tokens per hour actually produced. We assume a realised throughput of 50% of the decode ceiling and 60% average utilisation, so 30% of the ceiling. Both are stated assumptions, not measurements.

These are output token costs at Capacity Block rates, without prefill, without the KV transfer, without staff, storage or networking, and without any margin. The point is the ratio, not the figure: at our assumptions, serving the MoE 1T costs roughly 2 to 20 times less per output token than serving the dense 1T, depending on which configurations you compare.

For a broader method that covers smaller models, see our self hosting cost per million tokens estimate, and for why the per token price keeps falling, see is AI getting cheaper.

Fleet size for a given demand

Suppose a service must produce 10 billion output tokens a day, which is about 116,000 tokens per second on average. Using the realised throughputs above:

Peak load is higher than average load, so a real fleet adds headroom on top of this.

Part 6: power

NVIDIA's GB200 NVL72 page, as we read it, does not state a rack power figure, so we use the DGX B200's published maximum system power of about 14.3 kW for 8 GPUs, about 1.79 kW per GPU including CPUs, NICs, fans and storage, as a proxy for a Blackwell class GPU at system level. This is a proxy and an assumption: an NVL72 rack is liquid cooled and has a different host design. For H100 we use the 700 W GPU TDP from NVIDIA's page and multiply by 1.5 for host, network and storage (an assumption). We assume a power usage effectiveness (PUE) of 1.2 for the facility.

The energy for the GB200 dense training run is about 29.6 million GPU hours × 1.79 kW × 1.2, about 64 GWh. For scale, that is about 2.7 days of the full output of a 1 GW power station, spread over three months in one campus.

gpus = 29.63e6 / (90 * 24 * 0.9)                 # about 15,242
it_mw = gpus * 14.3 / 8 / 1000                    # DGX B200 system power per GPU as a proxy
print(round(gpus), it_mw, it_mw * 1.2)            # 15242, 27.3 MW, 32.7 MW
print(29.63e6 * 1.7875 * 1.2 / 1e6, "GWh")        # about 63.6 GWh

The Llama 3 paper adds a warning that averages hide: when thousands of GPUs pause and resume together, for example around checkpoints or collective communication, power draw across the data center can swing by tens of megawatts in an instant, which the authors describe as stretching the limits of the power grid. Planning for megawatts is not only about the average.

Part 7: putting it together

Here is the short answer to the title question, in one table. All of these depend on the assumptions stated above, and they are estimates rather than measurements.

The trade offs, stated neutrally:

• 8 GPU HGX servers are the most widely available unit and the easiest to rent, but they cap tensor and expert parallel groups at 8 GPUs before traffic hits the network.
• NVL72 racks remove that cap and help most with MoE expert parallelism and with large tensor parallel serving. You pay for 72 GPUs at a time, need liquid cooling, and each failure domain is larger.
• TPU pods offer the largest single interconnect domain, thousands of chips on ICI, and a compiler driven sharding model. You commit to the JAX or PyTorch XLA stack and to one cloud.
• High memory GPUs such as the MI355X fit the dense 1T FP8 serving case in one 8 GPU server, which simplifies serving. The software stack is ROCm rather than CUDA, and no public cloud price was available on the pages we checked.

How to do this estimate for your own model

1. Write down N_total and N_active. For a dense model they are equal.
2. Memory floor for training: 16 bytes × N_total, then divide by usable HBM per device.
3. Compute for training: 6 × N_active × D. State your D.
4. Device hours: compute ÷ (dense peak at your training precision × MFU × 3,600). Use a published MFU range, never one number.
5. Cluster size: device hours ÷ (days × 24 × goodput).
6. Power: devices × system kW per device × PUE.
7. Serving memory: weight bytes at the serving precision + KV bytes per token × context × batch.
8. Decode ceiling: (weights read + KV read) ÷ total HBM bandwidth per replica.
9. Prefill time: 2 × N_active × prompt tokens ÷ (peak × MFU).
10. Cost per million tokens: replica hourly price ÷ realised million tokens per hour, with your utilisation stated.

Prices change quickly; see our running notes on GPU rental prices and why they move. For the hardware underneath these numbers, our posts on GPU memory and interconnects, NVIDIA GPU architecture from Tesla to Blackwell and AI chip architecture go deeper.

Common mistakes

• Using sparse FLOPs as if they were dense. NVIDIA's headline tensor figures for Hopper and Blackwell are with 2:4 structured sparsity. Training and most inference run dense, so divide by two. Doubling your estimate of available compute halves your cost estimate, wrongly.
• Counting total parameters for MoE compute. Compute follows active parameters. Memory follows total parameters. Mixing them up gives an answer 30 times off in one direction or the other.
• Forgetting optimizer state. Weights alone are 2 TB at BF16 for a 1T model. The full training state is 16 TB.
• Ignoring the KV cache. For a dense model with GQA at long context, the cache can match or exceed the weights. Size it for your real context and batch, not for a demo prompt.
• Assuming the vendor peak is achievable. Published MFU for large training runs sits around 38% to 55%, and MoE runs are lower. Decode reaches some fraction of the bandwidth ceiling, never all of it.
• Assuming one rack holds a dense 1T training job. One NVL72 has 13.4 TB of HBM; the training state needs 16 TB.
• Treating a reservation price as on demand, or the reverse. Capacity Blocks, on demand, spot and multi year commitments differ by a factor of two or more. Label which one you used.
• Ignoring goodput. A 90 day plan with 100% goodput is a plan that slips.
• Placing tensor or expert parallel groups across the slow network. The worked numbers above show a tensor parallel decode step slowing by about five times when its all reduce moves from NVLink to a 400 Gb/s port.
• Forgetting that every layer of software is also part of the plan. Driver, CUDA or ROCm, NCCL or RCCL, the framework and the serving engine all have versions that change with security updates. A cluster you cannot patch quickly is a cluster that sits idle when you do patch it.

Where Safeguard fits

Safeguard does not design, program or speed up GPUs, TPUs or interconnects, and nothing in its product lowers the cost per token or per training hour in this post. The honest connection is the software that runs on all of this hardware. A training or serving image typically carries a GPU driver user space, a CUDA or ROCm toolkit, cuDNN or MIOpen, NCCL or RCCL, a container toolkit, a framework such as PyTorch or JAX, a serving engine, and the model files themselves. Each is a versioned component, and each gets security advisories.

Safeguard's generally available tooling applies to that software layer: SBOM generation and analysis in SPDX and CycloneDX, scanning of repositories, container images and SBOM files, vulnerability matching enriched with EPSS and CISA KEV data, reachability analysis to rank the findings your code actually calls, and autonomous remediation that opens tested fix pull requests, switched off by default. For the model files, prefer formats that do not execute code on load, such as safetensors, over pickle based checkpoints. Safeguard's AI-BOM discovery, which inventories AI assets, and its scanning of model artifact formats such as pickle, safetensors, GGUF and ONNX, are in early access. Safeguard does not scan GPU firmware, NVSwitch firmware or host kernel drivers installed outside your images; those updates still come from the hardware vendor and your operating system distribution. Our post on the GPU software supply chain lists what belongs in the SBOM of a GPU node.

FAQ

How many GPUs do you need to train a 1 trillion parameter model?

For a dense model trained on about 20 trillion tokens in 90 days, our estimate is about 15,000 GB200 class GPUs or about 38,500 H100s at 45% MFU, using BF16. FP8 roughly halves those counts if you can hold the same MFU. For a mixture of experts model with 32 billion active parameters on 15.5 trillion tokens, a few hundred to about 1,500 GPUs is enough for the same 90 days. The minimum just to hold the training state is about 87 GB200 class GPUs or 200 H100s.

How much memory does a 1 trillion parameter model need?

For inference, the weights alone are 2 TB at BF16, 1 TB at FP8 and 0.5 TB at FP4, plus the KV cache, which for a dense model at 32K context and 32 concurrent users can add another 0.7 to 1.4 TB. For training with mixed precision Adam, the model state is about 16 TB, plus activations.

Can a 1 trillion parameter model run on a single 8 GPU server?

A 1T MoE model at FP8 fits on one 8 × B200 server (1,440 GB) with room for a useful KV cache. A dense 1T model at FP8 fits on one 8 × MI355X server (2,304 GB) and, at FP4 with a modest cache, on one 8 × B200 server. It does not fit on one 8 × H100 server (640 GB), and on one 8 × H200 server (1,128 GB) it leaves almost no room for the cache.

What is the difference between dense and MoE for a 1 trillion parameter model?

A dense model uses all 1 trillion parameters for every token. A mixture of experts model routes each token to a few experts, so only a fraction of the parameters (32 billion out of 1.04 trillion in Kimi K2) do work per token. MoE cuts compute per token by about 30 times but still needs memory for all parameters, and it adds all to all communication between GPUs.

How much does it cost to train a 1 trillion parameter model?

Using 6ND with 20 trillion tokens, published MFU ranges and AWS Capacity Block rates checked on 25 October 2026, a dense 1T run lands at roughly $128 million to $700 million on NVIDIA parts for one clean pre training run. A 1T MoE with 32 billion active parameters lands at about $5 million to $26 million. These are estimates; real projects also pay for experiments, failures and post training.

Why does NVL72 matter for trillion parameter models?

It puts 72 GPUs in one NVLink domain with 130 TB/s of total NVLink bandwidth, so tensor parallel and expert parallel traffic, which happens many times per step, stays off the slower scale out network. Our worked example shows MoE all to all traffic of about 81 GB/s per GPU during training, more than a 400 Gb/s port carries but well within NVLink.

Is decode compute bound or memory bound for a 1T model?

Memory bandwidth bound in every case we worked. A dense 1T FP8 decode step at batch 32 needs about 22 PFLOPS of compute against 360 PFLOPS available on an NVL72, while the step time is set by reading 1.7 TB from HBM. Prefill, which processes the prompt, is compute bound.

How much power does a trillion parameter training cluster use?

About 33 MW at the facility for a 15,000 GPU Blackwell class cluster, using a proxy of 1.79 kW per GPU at system level and a PUE of 1.2. An H100 cluster for the same job draws about 49 MW under our assumptions. A MoE training cluster for the same 90 days needs a few megawatts or less.

What is disaggregated prefill and decode?

It runs the prompt processing phase and the token generation phase on separate pools of GPUs, each with its own parallelism, and moves the KV cache between them. Splitwise and DistServe proposed it in research, and DeepSeek-V3 describes using separate prefill and decode units in deployment.

Do TPUs need fewer chips than GPUs for a 1T model?

It depends on the chip generation, not the category. An Ironwood chip has 192 GiB of HBM and 2,307 TFLOPS of BF16, close to a GB200 GPU, so the counts are similar. A TPU v5p has 95 GiB and 459 TFLOPS, so it needs about five times more chips for the same training job. TPU pods connect up to 8,960 (v5p) or 9,216 (Ironwood) chips on one ICI network.

More in this series

Other posts in our AI chips and costs series:

• AI Accelerator TCO: Buy vs Rent, Capex, Power, Cooling and Utilization
• AI Chips Compared 2026: B200, H200, MI355X, Gaudi 3, TPU, Trainium
• Cheapest LLM Inference: How to Cut Cost per Million Tokens
• Fastest LLM Inference: Lowest Latency and Most Tokens per Second
• How to Estimate AI Training and Inference Cost: Formulas and a Calculator
• LLM Inference Cost by Chip and Model Size, 1B to 1 Trillion Parameters
• LLM Training Cost by Chip and Model Size: 1B to 1T, Worked

Fix it without waiting for upstream

A trillion parameter cluster runs on thousands of copies of the same software image, and every one of them carries a driver user space, a CUDA or ROCm toolkit, communication libraries and a Python stack full of open source dependencies. When one of those components gets a security fix, the speed at which you can roll it out across the fleet decides how long the cluster runs exposed or sits idle.

Whatever the vulnerability, most of the risk sits in two gaps. The first is the time between a fix being published and that fix actually running in your production builds. The second, and worse, is the time before any fix exists at all, while a maintainer works out a patch and a release schedule. Waiting through both gaps is a habit, not a requirement. Safeguard gives teams three ways to close them.

1. Autonomous remediation: pull requests, not tickets

When a fixed version exists, Safeguard's Griffin models open a tested, policy gated pull request that moves the affected dependency to the safe version. You can run it on one repository or in bulk across every project that carries the vulnerable version. Remediation covers eight categories, including dependencies, licenses, security posture and findings. Reachability analysis ranks the vulnerable code paths your application actually calls first, so the first pull requests you merge are the ones that matter most. Autonomous remediation ships switched off, with every category set to Manual, so your team decides how much runs on its own and what still needs a human approval.

2. Gold Open Source: the safe version in seconds, free

gold.safeguard.sh is free and needs no login. Search any CVE or package to see affected and fixed versions, CVSS, EPSS, CISA KEV status and a recommended safe version, or use the "Am I affected?" checker. The same data is available through a free read API, README status badges and RSS threat feeds. The safeguard-sh/gold-scan-action GitHub Action can fail a build on a known exploited vulnerability, on any known vulnerability, or never, so a version you have just removed cannot quietly come back in next week's merge.

3. Gold Registry: start from components that are already fixed

registry.safeguard.sh provides hardened, drop in replacements for upstream packages and container images. Every Gold artifact is held continuously to zero exploitable critical or high CVEs, zero malware and a completed license check, with SLSA L3 or higher provenance, a Sigstore signature and a public evidence page you can hand to an auditor. An artifact loses its Gold badge if it regresses. Teams can point their own package directory at the registry so hardened equivalents resolve without changing how developers install anything.

When upstream has no fix yet, or will not ship one, Safeguard first works to get the fix accepted upstream. If the issue is severe or upstream will not move, Safeguard builds a fork, fixes the vulnerability there and publishes a hardened Gold build, so you are not stuck on a maintainer's release schedule. You can also request a hardened, compatibility tested build of a specific package. This fork and patch flow is in early access today, so ask which packages it covers before you plan around it.

One honest limit: when the vulnerable code lives in a vendor appliance, a hosted service or an operating system runtime you do not build yourself, the fix still has to come from that vendor or distribution. The steps above apply to the open source dependencies and container images you build and ship, which is where most teams carry the largest and least tracked share of their exposure.

See how Safeguard would help

The first demo is a real Safeguard CLI session. You install the CLI, open the interactive terminal and ask it to fix a lodash CVE in a repository; it reads the finding and the advisory, asks before any command that changes something, edits package.json, runs the install and tests with your approval and pushes nothing. The same flow applies to the Python dependencies of a training or serving repository, and safeguard scan --image takes a CUDA or ROCm based image.

Click through it at your own pace, or open it full screen.

The second demo sets up the Safeguard runner on your own machine, approves it with a device code and then scans from the terminal. Teams that train large models often keep images, checkpoints and training code on build hosts they control; the runner lets those scans happen there.

Click through it at your own pace, or open it full screen.

The third demo connects Claude to the Safeguard MCP server with OAuth and shows how you choose which Safeguard tools Claude can use. Once connected, you can ask in plain language which of your projects or images carry a given NCCL, CUDA toolkit or serving engine version before you roll an update across a fleet.

Click through it at your own pace, or open it full screen.

To try it on your own images, download the Safeguard CLI and follow the CLI documentation.

Quick check

Sources

Prices and vendor pages checked on 25 October 2026.

• NVIDIA: GB200 NVL72 product page and specifications
• NVIDIA: GB300 NVL72 product page and specifications
• NVIDIA: DGX B200 specifications
• NVIDIA: H100 product page and specifications
• NVIDIA: H200 product page and specifications
• AMD: Instinct MI355X specifications
• Google Cloud: TPU v5p documentation
• Google Cloud: TPU7x (Ironwood) documentation
• Google Cloud: Cloud TPU pricing
• AWS: Amazon EC2 Capacity Blocks for ML pricing
• Kaplan et al.: Scaling Laws for Neural Language Models, 2020
• Hoffmann et al.: Training Compute-Optimal Large Language Models (Chinchilla), 2022
• Rajbhandari et al.: ZeRO: Memory Optimizations Toward Training Trillion Parameter Models, SC20
• Narayanan et al.: Efficient Large-Scale Language Model Training on GPU Clusters Using Megatron-LM, SC21
• Korthikanti et al.: Reducing Activation Recomputation in Large Transformer Models, 2022
• Chowdhery et al.: PaLM: Scaling Language Modeling with Pathways, 2022
• Jiang et al.: MegaScale: Scaling Large Language Model Training to More Than 10,000 GPUs, NSDI 2024
• Llama Team, Meta: The Llama 3 Herd of Models, 2024
• Fedus, Zoph and Shazeer: Switch Transformers, JMLR 2022
• DeepSeek-AI: DeepSeek-V3 Technical Report, 2024
• Kimi Team: Kimi K2: Open Agentic Intelligence, technical report
• Pope et al.: Efficiently Scaling Transformer Inference, MLSys 2023
• Patel et al.: Splitwise: Efficient Generative LLM Inference Using Phase Splitting, ISCA 2024
• Zhong et al.: DistServe: Disaggregating Prefill and Decoding for Goodput-optimized LLM Serving, OSDI 2024

Last reviewed: 25 October 2026. Vendor specifications, cloud prices and capacity offerings change often, and every cost in this post is an estimate built from the assumptions stated next to it. Check the linked pages before you size a cluster or a budget on these numbers.