AI Chips Compared 2026: B200, H200, MI355X, Gaudi 3, TPU, Trainium

No single AI chip wins on every metric in 2026. On vendor peak figures, NVIDIA's B300 and AMD's MI355X lead per chip on low precision throughput and memory (288 GB of HBM3E at 8 TB/s each), Google's Ironwood TPU and NVIDIA's NVL72 racks lead on scale-up domain size, and AWS Trainium2 and Google's TPUs have the lowest public hourly prices per chip. Which one is "fastest" depends on whether you are training, processing prompts (prefill) or generating tokens (decode), because each of those stresses a different metric.

This post puts the current data center accelerators from NVIDIA, AMD, Intel, Google and AWS side by side, one table per metric family: peak throughput at each precision, dense and sparse; memory capacity, bandwidth and type; scale-up and scale-out interconnect; power and performance per watt; form factor and rack scale; software stack; and availability with public prices. It then shows which metric matters for training, prefill and decode with worked numbers, and closes with measured MLPerf results, named by round and submitter.

Every specification comes from the vendor's own documentation, every measured result from MLCommons, and every price from a public list or reservation price page that we checked on 25 October 2026. All links are in the Sources section. Every derived number (ratios, per-chip splits, cost estimates) was computed with Python. Where a vendor has not published a figure, the table says so rather than guessing. The code in this post is plain Python that we ran on a Mac; we did not run anything on the accelerators themselves.

Key takeaways

• Peak FLOPS are not comparable until you normalise them. NVIDIA publishes most Tensor Core figures "with sparsity", which is twice the dense rate. AMD publishes both. Google, Intel and AWS mostly publish dense figures. Every table below states dense or sparse and the precision.
• Per chip, on vendor peak figures, the 2025 and 2026 parts roughly double the 2023 parts on dense BF16 and FP8. The NVIDIA B200 and B300 list 2.25 PFLOPS dense BF16 per GPU in an HGX system, the AMD MI355X 2.5 PFLOPS, Google's Ironwood 2.3 PFLOPS, against about 0.99 PFLOPS for the H100 and H200.
• FP4 is where the newest chips separate. The B300 lists 13.5 PFLOPS dense FP4 per GPU (15 PFLOPS per GPU in the NVL72 rack), the MI355X 10.1 PFLOPS of MXFP4, the B200 9 PFLOPS. Older chips have no FP4 path at all.
• Memory bandwidth decides decode speed. Generating one token at batch size 1 reads every weight once, so the 8 TB/s parts (B200, B300, MI355X) and Ironwood at about 7.4 TB/s set the upper bound for single-stream decode, well ahead of the 3.35 TB/s H100.
• Scale-up domain size matters for very large models. An HGX or AMD 8-GPU node has an 8-chip fast domain; the GB200 and GB300 NVL72 racks have 72; a Trainium2 UltraServer has 64; TPU pods connect 256 (v5e, v6e), 8,960 (v5p) or 9,216 (Ironwood) chips over inter-chip links.
• Measured results narrow the gaps that peak figures suggest. In MLPerf Inference v6.0, Llama 2 70B Offline, the top 8-GPU MI355X result (AMD) and the top 8-GPU B200 result (HPE) were within about half a percent of each other per accelerator, both using FP4 weights.
• Prices are public for cloud only, and only for some chips. AWS, Azure and Google Cloud publish hourly prices; we found no public purchase price list for any of these accelerators.

How to read an AI chip comparison without being misled

Before the tables, five rules that apply to every number below.

Rule 1: peak is not achieved. A vendor's peak figure is the number of multiply and add operations the matrix units can retire per second at the maximum clock, with every unit busy. Real training jobs reach a fraction of it. Meta's Llama 3 paper reports an overall BF16 model FLOPs utilisation (MFU) of 38 to 43 percent on its H100 training clusters of 8,192 and 16,384 GPUs. That is a well tuned, very large job; smaller teams often see less.

Rule 2: dense and sparse figures differ by a factor of two. NVIDIA's and AMD's matrix units support 2:4 structured sparsity: if two of every four weights in a block are zero, the hardware skips them and doubles its effective rate. Very few production language models are trained or served in that format. A sparse figure is useful only if your model is actually 2:4 sparse, so this post leads with dense numbers.

Rule 3: the precision has to match. FP8 is roughly twice BF16 on most of these chips; FP4 is roughly twice FP8 on the chips that support it. Comparing one chip's FP4 number with another's FP8 number makes a 2x difference out of nothing. MLPerf results also record the weight data type, which matters for the same reason.

Rule 4: per chip, per node and per rack are different units. NVIDIA quotes the GB200 and GB300 at rack scale (72 GPUs), DGX and HGX at node scale (8 GPUs), and H100 and H200 per GPU. AWS quotes Trainium2 per chip, per 16-chip instance and per 64-chip UltraServer. Google quotes TPUs per chip and per pod. We have divided where the vendor's own totals allow it, and say so.

Rule 5: a chip is only as good as its software path. A model that runs well on CUDA may need porting work on ROCm, the Intel Gaudi software suite, XLA for TPUs or the AWS Neuron SDK. That cost is real, does not appear in any spec table, and is covered in its own section below.

The chips in this comparison

We left out chips without public specification pages at this level of detail. Where a vendor publishes a later variant (for example the H100 NVL or H200 NVL PCIe cards), we mention it only where it changes a conclusion.

Metric family 1: peak throughput by precision, dense and sparse

This is the table most comparisons stop at. All figures are vendor peak figures in TFLOPS (or TOPS for INT8). "Dense" means without structured sparsity. Where the vendor publishes only a sparse figure and states that dense is half, we show the dense value as half and mark it.

NVIDIA, per GPU

NVIDIA marks H100 and H200 Tensor Core figures "with sparsity"; the HGX table states that FP8, INT8, BF16 and TF32 figures are sparse with dense equal to half, and gives FP4 as "sparse | dense". The B200 and B300 columns are the HGX 8-GPU totals divided by eight, which is our arithmetic, not an NVIDIA per-GPU figure. Two things stand out. First, the B300 trades FP64 and INT8 throughput for FP4: its HGX table lists 10 TFLOPS of FP64 and 3 POPS of sparse INT8 per 8 GPUs, against 296 TFLOPS and 72 POPS for the HGX B200. Second, the B300's dense FP4 rate is 3x its dense FP8 rate rather than the usual 2x, which NVIDIA's Blackwell Ultra technical blog describes as 15 PFLOPS of dense NVFP4 per GPU (the 13.5 PFLOPS here comes from the HGX B300 table; the NVL72 rack works out to 15, as shown below).

NVIDIA rack scale: GB200 NVL72 and GB300 NVL72

Divided by 72, the GB200 NVL72 gives 10 PFLOPS dense FP4, 5 PFLOPS dense FP8 and 2.5 PFLOPS dense BF16 per GPU, slightly above the HGX B200's 9, 4.5 and 2.25. The GB300 NVL72 gives 15 PFLOPS dense FP4 per GPU, matching the Blackwell Ultra blog. The same GPU silicon is listed at different rates in different systems because the systems run it at different power and clock limits, which is one more reason to compare like with like.

AMD Instinct, per GPU

AMD publishes both dense and sparse figures on its product pages, so no halving is needed. The MI325X has the same compute as the MI300X; it changes the memory (256 GB of HBM3E at 6 TB/s instead of 192 GB of HBM3 at 5.3 TB/s) and raises board power from 750 W to 1,000 W. The MI355X moves to CDNA 4 on TSMC 3 nm compute dies, roughly doubles dense BF16 and FP8, adds MXFP6 and MXFP4 at 10.1 PFLOPS, and like the B300 gives up FP64 matrix throughput (78.6 TFLOPS, down from 163.4) to spend silicon on low precision.

Intel Gaudi 3

These are Intel's own figures from the Gaudi 3 white paper (July 2025 revision). Intel does not publish a structured sparsity figure, so these are dense. Note two quirks: FP8 and BF16 run at the same matrix rate on Gaudi 3, so moving to FP8 halves memory and traffic but does not double compute, and FP16 runs at well under a third of the BF16 rate, so BF16 is the format to use on this chip.

Google TPUs, per chip

Google publishes TPU figures as peak compute per chip without a sparsity variant, so these are dense. On v5p and v6e the FP8 figure equals the BF16 figure; Ironwood is the first TPU in this list where FP8 is twice BF16. The Ironwood documentation describes two TensorCores and four SparseCores per chip. SparseCores are separate units for embedding lookups, which matter for recommendation models; they are not structured sparsity in the NVIDIA and AMD sense.

AWS Trainium and Inferentia, per chip

Trainium3 doubles Trainium2's FP8 rate and adds MXFP4 at the same rate as MXFP8, while BF16 stays essentially flat (667 to 671). AWS states these per chip in the Neuron documentation, alongside per-instance totals.

The dense comparison in one picture

Dense BF16, the precision most training runs still use for their main matrix multiplies, per chip, vendor peak. Each block is 100 TFLOPS.

Dense BF16 TFLOPS per chip (vendor peak, rounded to the nearest 100 for the bar)
TPU v5e            197  ##
Inferentia2        190  ##
TPU v5p            459  #####
Trainium2          667  #######
Trainium3          671  #######
TPU v6e            918  #########
H100 / H200        990  ##########
MI300X / MI325X  1,307  #############
Gaudi 3          1,678  #################
B200 / B300      2,250  ######################   (HGX, per GPU)
TPU7x Ironwood   2,307  #######################
MI355X           2,500  #########################
GB200 in NVL72   2,500  #########################  (rack total / 72)

And dense FP8, which most inference and an increasing share of training now uses:

Dense FP8 TFLOPS per chip (vendor peak)
TPU v5p            459  #####
TPU v6e            918  #########
Trainium2        1,299  #############
Gaudi 3          1,678  #################
H100 / H200      1,979  ####################
Trainium3        2,517  #########################
MI300X / MI325X  2,615  ##########################
B200 / B300      4,500  #############################################
TPU7x Ironwood   4,614  ##############################################
MI355X           5,000  ##################################################
GB200 in NVL72   5,000  ##################################################

The picture is a ranking of peak arithmetic and nothing more. A chip with a lower peak but more memory, a bigger scale-up domain, a lower price or a better software fit for your model can deliver more useful work per dollar. The remaining metric families are where that shows up.

Metric family 2: memory capacity, bandwidth and type

For large language models, memory is at least as important as arithmetic. Capacity decides how many chips you need to hold the weights and the KV cache. Bandwidth decides how fast decode runs. The ratio of compute to bandwidth decides which workloads are compute bound.

Watch the units. Google and AWS publish some capacities in GiB (2 to the 30th bytes) and NVIDIA and AMD in GB (10 to the 9th). 192 GiB is about 206 GB, so Ironwood holds slightly more per chip than a 192 GB MI300X. Where the vendor page did not name the HBM generation, we wrote "HBM" rather than guess. One small inconsistency is worth knowing about: Microsoft's documentation for its ND GB200 v6 virtual machines lists each Blackwell GPU with 192 GiB, while NVIDIA's rack table works out to 186 GB per GPU. The difference is likely between physical capacity and usable capacity, but neither page says so.

Two derived ratios that predict behaviour

The first ratio is the ridge point of the roofline model (Williams, Waterman and Patterson, 2009): peak FLOPS divided by memory bandwidth, in FLOPs per byte. A kernel whose arithmetic intensity is below the ridge point is memory bound on that chip; above it, compute bound. The second is capacity per unit of compute, in GB per dense BF16 TFLOPS, which hints at how much memory each unit of arithmetic has to work with.

Read it like this. Decoding one token for one user does about 2 FLOPs per weight while reading each weight once, so at FP8 weights its arithmetic intensity is about 2 FLOPs per byte, and at BF16 about 1. Every chip in the table has a ridge point in the hundreds, so single-stream decode is deeply memory bound everywhere, and bandwidth sets its speed. Batching many users together reuses each weight read across the batch, and the intensity rises roughly in proportion to the batch size until it crosses the ridge point. A chip with a high ridge point and little memory, such as the TPU v6e (560 FLOP per byte, 32 GB), needs large batches, and therefore memory for a large KV cache, before its compute is fully used. A chip with a low ridge point, such as the H200 or the TPU v5p, reaches its compute ceiling at smaller batches. Neither is better in general; they suit different serving patterns.

Metric family 3: scale-up and scale-out interconnect

Models that do not fit on one chip are split across many, and then the links between chips matter as much as the chips. There are two tiers. Scale-up links connect a small group of chips with very high bandwidth and low latency, usually inside one node or rack, and carry tensor parallel and expert parallel traffic. Scale-out networks connect those groups across a data center, usually over InfiniBand or Ethernet with RDMA, and carry data parallel and pipeline parallel traffic.

The architectural difference is easiest to see in a picture. NVIDIA and AMD build switched or fully connected islands of 8 (or 72), then connect islands with a network. Google builds a torus where every chip links directly to its neighbours, with no switch, and extends it to thousands of chips.

Switched island (HGX, NVL72)              Torus (TPU pod, 2D slice shown)

  GPU0  GPU1  GPU2  GPU3                    T == T == T == T ==+
    \     \   /     /                       |    |    |    |   |
     [ NVSwitch / NVLink switch trays ]     T == T == T == T ==+
    /     /   \     \                       |    |    |    |   |
  GPU4  GPU5  GPU6  GPU7                    T == T == T == T ==+
                                            |    |    |    |   |
  any GPU to any GPU: one switch hop        T == T == T == T ==+
  at full NVLink rate                       +====+====+====+  (wraps around)
                                            each chip talks to its neighbours
                                            directly; far chips take several hops

Fully connected mesh (MI300X / MI355X 8-GPU, Gaudi 3 8-OAM)
  every OAM has a direct link (or a group of 3 Ethernet ports on Gaudi 3)
  to every other OAM; no switch chip, but bandwidth to one peer is a fraction
  of the chip's total link bandwidth

The trade-offs follow from the shapes. A switch gives every GPU full bandwidth to any other GPU in the domain, which suits all-to-all traffic such as mixture of experts routing, but switch chips cost power and money and the domain ends at the edge of the switch. A torus has no switches and scales to thousands of chips, but traffic between distant chips crosses several hops, so the compiler and the parallelism layout have to keep heavy traffic between neighbours. A fully connected mesh is cheap and simple for 8 chips but does not grow beyond that without an external network, and any single pair of chips gets only one link's worth of bandwidth (for example 128 GB/s on MI300X, or 3 x 200 Gb/s, which is 75 GB/s per direction, between two Gaudi 3 cards).

A worked number: why scale-up bandwidth matters for tensor parallelism

Under tensor parallelism, every transformer layer ends with an all-reduce of the activations across the tensor parallel group. Take a model with hidden size 8,192, a micro-batch of 16 sequences of 4,096 tokens, and BF16 activations. One layer's activation tensor is 16 x 4,096 x 8,192 x 2 bytes, about 1.07 GB. A ring all-reduce moves about 2 x (n - 1) / n of that per GPU, so with 8 GPUs about 1.88 GB per GPU per all-reduce. With two all-reduces per layer in the forward pass (after attention and after the MLP), that is about 3.76 GB per layer per GPU. At NVLink 4's 900 GB/s (which is bidirectional, so about 450 GB/s each way) that is roughly 8 ms per layer; at NVLink 5's 1.8 TB/s, roughly 4 ms. Over 80 layers the communication alone is 0.67 seconds or 0.33 seconds per forward pass, unless it overlaps with compute. This is why tensor parallelism is normally kept inside the fast domain, and why a larger fast domain (72 GPUs on NVL72, 64 on a Trainium2 UltraServer) lets larger models use tensor or expert parallelism without touching the slower scale-out network. These numbers are illustrative; real frameworks overlap communication with compute and use different collective algorithms.

Metric family 4: power and performance per watt

Power is where vendor documentation is least comparable. NVIDIA and AMD publish per-module power for some parts; NVIDIA publishes only system power for its Blackwell DGX systems on the pages we used; Google publishes efficiency ratios and pod-level power rather than per-chip power; AWS does not publish Trainium power on the pages we used.

The module-level and system-level numbers cannot be compared directly. A DGX system's 14.3 kW includes two host CPUs, eight NICs, two DPUs, NVSwitch chips, fans and storage, so dividing by eight (1,787 W per GPU slot) overstates the GPU's own draw by a large margin. The H100's 2.83 TFLOPS per watt is a module figure and the DGX B200's 2.52 is a whole-system figure; on a like-for-like basis the B200 is more efficient than the H100, but the documents we used do not let us say by exactly how much. The Ironwood pod figure divides Google's 42.5 exaflops (FP8, 9,216 chips times 4,614 TFLOPS, which our arithmetic confirms) by "nearly 10 MW" and includes everything in the pod; treat it as an order of magnitude, not a measurement.

Two practical points matter more than the per-watt ranking. First, the newest parts are liquid cooled at rack scale: Google describes the 9,216-chip Ironwood pod as liquid cooled, the GB300 NVL72 is "fully liquid-cooled", and the MI355X page lists both passive and active cooling options. A facility designed for air-cooled 700 W modules cannot simply take 1,400 W modules. Second, MLPerf has a power measurement track, and that is the only place where measured performance per watt under the same rules is published; most submissions do not include power.

Metric family 5: form factor and rack scale

What you actually buy or rent is a system, not a chip. The building blocks differ more than the chips do.

Two derived observations. A GB300 NVL72 rack holds about 20 TB of HBM in a single NVLink domain, enough for the FP8 weights of a dense 1 trillion parameter model (about 1 TB) many times over, with most of the remainder available for KV cache. A Trainium2 UltraServer holds 6 TiB in one NeuronLink domain. An 8-GPU HGX B200 node holds 1.44 TB; an 8-GPU MI355X node 2.3 TB. Those capacities determine how many nodes one model replica needs, which in turn decides whether the model's tensor and expert parallel traffic stays on the fast links.

Metric family 6: software stack

The software stack is the metric with no number, and often the one that decides the project. Each vendor ships its own compiler, kernel libraries, collective communication library and framework integration, and each has a different ecosystem around it.

In practice, three questions settle the software metric for a given team. Does your model's architecture (attention variant, mixture of experts routing, quantisation format) have a tuned kernel on that stack today? Does your serving framework support that chip as a first-class target, including the low-precision format you plan to use? And how much engineering time can you spend porting and tuning? The answers change month to month, so check the vendor's current release notes for your model rather than relying on a general reputation.

Every one of these stacks is also a large set of versioned software components: drivers, runtimes, compilers, math and collective libraries, container toolkits and framework builds. Each ships security fixes on its own schedule. We come back to that in the Safeguard section.

Metric family 7: availability and public prices

Read the price column with care. AWS Capacity Blocks are reservations for a fixed future window, priced by AWS "based on trends in supply and demand" and next due for an update in January 2027 according to the page; they are not on-demand prices. Azure's figures come from its public retail prices API for pay-as-you-go Linux VMs. Google's are its published Cloud TPU list prices per chip-hour. Prices vary by region, by commitment and over time, and negotiated prices are usually lower. The site's GPU rental prices post explains why they move.

Price per unit of peak compute and per unit of bandwidth

Dividing the hourly price by the chip's peak gives a crude but useful first filter: what a peak dense BF16 PFLOP-hour and a TB/s-hour of memory bandwidth cost on each offer. These are estimates built from the stated list prices and vendor peak figures only.

On this crude measure the TPU v6e and Trainium2 offers are the cheapest per peak BF16 PFLOP-hour and Trainium2 the cheapest per unit of bandwidth. Do not stop here. Peak per dollar ignores achieved utilisation, which differs by chip and by software maturity for your model; it ignores memory capacity (a 32 GB v6e needs many more chips than a 288 GB MI355X to hold a large model, and those extra chips add communication); and it ignores the cost of porting. It is a starting filter, not an answer.

Which metric matters for training, prefill and decode

Large language model work splits into three phases with different bottlenecks. The research paper that set out this split most clearly for inference is Pope and colleagues' "Efficiently Scaling Transformer Inference" (2022); the training side follows from the scaling law literature.

Training: compute times utilisation

The standard estimate for training compute is about 6 x N x D floating point operations, where N is the parameter count and D the number of training tokens (Kaplan and colleagues, 2020). The Chinchilla paper (Hoffmann and colleagues, 2022) found that compute-optimal training uses roughly 20 tokens per parameter. Take a dense 70 billion parameter model trained on 1.4 trillion tokens (20 tokens per parameter): 6 x 70e9 x 1.4e12 is 5.88 x 10 to the 23rd FLOPs.

Divide by each chip's dense BF16 peak times an assumed MFU of 35 to 45 percent (a range around the 38 to 43 percent the Llama 3 paper reports for H100s), and multiply by the list price. The table is an estimate built entirely from those stated assumptions; it is not a measurement.

The big caveat is the assumption that every chip reaches the same MFU. That is unlikely. MFU depends on the software stack's maturity for your model, on memory capacity (small-memory chips need more model parallelism and so more communication), and on the interconnect. The TPU v6e looks cheap here, but with 32 GB per chip a 70B training run needs aggressive sharding across many chips, which tends to lower MFU. The H200 shows the same hours as the H100 because its peak compute is the same; its extra memory helps by allowing larger micro-batches or less parallelism, which can raise MFU, but the formula cannot see that. The series' other posts work through training cost by model size in more detail.

Prefill: compute bound

Processing a prompt of P tokens through a model of N parameters takes about 2 x N x P FLOPs for the weight matrix multiplies (attention adds more for long prompts). For a 70B model and a 4,096-token prompt that is 5.73 x 10 to the 14th FLOPs. All tokens in the prompt are processed together, so the arithmetic intensity is high and prefill is compute bound. The ideal time on one chip at dense FP8 peak, and at an assumed 50 percent utilisation, would be:

A 70B model at FP8 needs about 70 GB for weights alone, so on the H100 (80 GB), Gaudi 3 (128 GB) and TPU v6e (32 GB) it would in practice span several chips; the table shows the compute cost per chip-equivalent, which is what matters for cost per prompt token. Time to first token for a user is then roughly this compute time divided by the number of chips working on the prompt, plus communication and queueing.

Decode: bandwidth bound

At batch size 1, generating each token reads every weight once. The upper bound on tokens per second for one sequence is memory bandwidth divided by the bytes of weights read per token. For a 70B model with FP8 weights (70 GB) and an 8B model with BF16 weights (16 GB):

These are ceilings that ignore the KV cache, kernel overheads and communication; real systems reach a fraction of them. When a model is split across k chips with tensor parallelism, each chip reads 1/k of the weights per token, so the ceiling rises roughly k-fold until communication dominates. For example, eight H100s holding a 70B FP8 model have 26.8 TB/s in total, a ceiling of about 383 tokens per second per sequence.

Batching is what makes decode economical. With B sequences in a batch, each weight read serves B tokens, so throughput rises almost linearly with B until either compute or memory capacity runs out. Memory runs out first in most cases because of the KV cache. Take an assumed 70B architecture with 80 layers, grouped query attention with 8 key-value heads of dimension 128, and a BF16 KV cache. Each token needs 80 x 8 x 128 x 2 (key and value) x 2 bytes, which is 327,680 bytes, or about 0.33 MB. At a context of 4,096 tokens and a batch of 64 sequences that is about 85.9 GB, more than the 70 GB of FP8 weights. Weights plus cache come to about 156 GB: too much for one H200 (141 GB), comfortable on one B200 (180 GB) or MI355X (288 GB). An FP8 KV cache halves the cache to about 43 GB. This arithmetic is why memory capacity, not just bandwidth, has become a headline metric for inference chips, and why the 256 GB and 288 GB parts are marketed for serving. The self-hosting cost per million tokens post walks through the same logic for cost.

Measured performance: MLPerf highlights

Vendor peaks tell you what a chip could do. MLPerf, run by the MLCommons consortium, is the industry's peer-reviewed benchmark where submitters run fixed models to a fixed accuracy target under fixed rules, and the results are published with the full system description and code. The inference methodology is described in the MLPerf Inference Benchmark paper (Reddi and colleagues, 2020). Everything in this section comes from MLCommons' own announcements and results files.

Training rounds

We have not restated individual time-to-train results here. MLPerf Training scores are computed from several runs with an official scoring method, and copying one number out of context loses the system size and the scoring. Use the MLCommons results table for the round and benchmark you care about, and compare systems of the same size. Two patterns from the announcements are still worth noting. Training v5.0 was the first round with AMD submitting MI300X and MI325X results itself and the first with Google's Trillium TPU. And by v6.0 the two newest benchmarks are both mixture of experts models, which put the all-to-all bandwidth of the scale-up domain under test, not only the matrix units.

Inference: Llama 2 70B across chips

Llama 2 70B is the most widely submitted MLPerf Inference benchmark (MLCommons reported 24 submitters for it in v5.1), which makes it the most useful single benchmark for comparing chips. The table below lists, for each accelerator type, the highest Offline throughput in the closed division, datacenter suite, "available" category, at the 99 percent accuracy target, from the v6.0 (1 April 2026) and v5.1 (9 September 2025) results files. MLPerf reports whole-system throughput; the per-accelerator column is our division of that system result by the number of accelerators, which is not an official MLPerf metric.

What the measured numbers say, with the caveats attached:

• The B200 and the MI355X were close. In v6.0, AMD's 8 x MI355X system reached 99.5 percent of HPE's 8 x B200 system's Offline throughput, both with FP4 weights. The B300 system from Cisco was about 11 percent above the B200 system. Peak FP4 ratios would have predicted a bigger B300 lead (13.5 against 9 PFLOPS dense); the measured gap is smaller because decode throughput also depends on bandwidth, which is 8 TB/s on all three.
• The 2025 generation is about 3x the 2023 generation on this test, but part of that is FP4. Per accelerator, the v6.0 B200 result is 2.94 times the v5.1 H200 SXM result. The H200 ran FP8 weights and has no FP4 path, so the comparison mixes hardware and number format. That is a fair comparison of what each chip can do on this benchmark, but not of silicon alone.
• System scale changes per-chip results. The GB300 NVL72 result is above the 8-GPU B300 result per accelerator, and the GB200 NVL72 result slightly below the 8-GPU B200 result. Rack-scale systems are built for very large models and long contexts, where their 72-GPU NVLink domain matters more than on a 70B model that fits in a few GPUs.
• Software moves results between rounds. The same MI325X hardware appears in both rounds within a few percent, from different submitters. Submitter choices of serving stack, batch size and kernel tuning matter.
• Several chips in this post do not appear. The v5.1 and v6.0 inference results files list no TPU, Trainium, Inferentia or Gaudi accelerators in the closed datacenter division; Google's submissions in both rounds used NVIDIA B200, H200 and GB200 systems. Absence from MLPerf is not evidence that a chip is slow, but it means there is no like-for-like measured comparison, and you should ask the vendor for results on your own model.

From MLPerf throughput to an estimated cost per million tokens

Combining an MLPerf result with a public price gives a rough serving cost. This is an estimate built from stated assumptions: the Offline throughput above per accelerator, the AWS Capacity Block price per GPU-hour, 100 percent utilisation and no overheads. Real services run at lower utilisation, with latency targets that lower throughput, so real costs will be higher.

On these assumptions the Blackwell parts are about 27 to 39 percent cheaper per token than Hopper on Llama 2 70B, despite costing 1.8 to 2.7 times as much per hour, because each GPU delivers about 2.9 to 3.7 times the throughput with FP4 weights. Notice also that on these prices the H200 is slightly more expensive per token than the H100: its price premium is larger than its measured throughput gain on this model. That can flip for larger models or longer contexts, where the H200's extra memory allows bigger batches. MI355X, TPU and Trainium cannot be placed in this table without mixing data sources, because the first has no public AWS price and the others have no MLPerf result here. The cost per token trends post covers why per-token prices fall over time and the caveats in comparing them.

Putting it together: a decision table

No row in this table names a winner. Each names the metric that dominates and the chips that are strong on that metric, with the trade-off.

Common mistakes

• Comparing sparse with dense. A sparse NVIDIA figure next to a dense TPU figure exaggerates the gap by 2x. Halve the NVIDIA "with sparsity" number unless your model really is 2:4 sparse.
• Comparing different precisions. FP4 against FP8 is another hidden 2x (3x on the B300). Compare at the precision you will actually run, and validate model quality at that precision.
• Comparing a rack figure with a chip figure. GB200 and GB300 NVL72 numbers are for 72 GPUs. Divide before you compare.
• Treating peak as achieved. Plan with an MFU range, such as the 38 to 43 percent the Llama 3 paper reports, not with 100 percent.
• Ignoring memory capacity. A chip that needs twice as many units to hold the model pays twice the communication and often a lower MFU.
• Ignoring the KV cache. At long contexts and large batches the cache can exceed the weights, as the 70B example shows.
• Mixing GB and GiB. 192 GiB is about 206 GB.
• Reading MLPerf per-accelerator numbers as official. MLPerf reports per system. Per-chip division is reasonable for like-sized systems, but say you did it.
• Taking reservation prices for on-demand prices. AWS Capacity Blocks, Google commitments and Azure pay as you go are different products with different prices.
• Forgetting the software bill. Porting kernels, retuning parallelism and requalifying a model are real costs that no spec sheet shows.
• Forgetting the software supply chain. Every chip's driver, runtime, compiler, collective library and serving framework is a versioned dependency with its own security advisories.

Where Safeguard fits

Safeguard does not design, program or speed up AI accelerators, and nothing in its product lowers GPU, TPU or Trainium cost or makes a model train or serve faster. The honest connection is the software that runs on all of this hardware. Whichever chip you choose, your training and serving images carry a driver user space, a runtime (CUDA, ROCm, the Intel Gaudi software suite, XLA libraries or the Neuron SDK), math and collective libraries (cuDNN and NCCL, MIOpen and RCCL, and their equivalents), a container toolkit, a framework build such as PyTorch or JAX, and a serving stack. Each is a versioned component, and each gets security advisories on its own schedule. Model files are part of the same chain: loading weights from a format that can execute code on load is a known risk, which is why formats such as safetensors exist.

Safeguard's generally available tooling applies to that software layer: SBOM generation and analysis in SPDX and CycloneDX, scanning of repositories, container images and SBOM files, vulnerability matching enriched with EPSS and CISA KEV data, reachability analysis to rank the findings your code actually calls, and autonomous remediation that opens tested fix pull requests, switched off by default. In practice that means you can scan a CUDA, ROCm or Neuron based image, see which of your images carry a given NCCL, container toolkit or serving framework version, and get a pull request that moves a pinned version forward when a fix exists. AI-BOM discovery, an inventory of AI assets across code and cloud, is in early access. Safeguard does not scan accelerator firmware or host kernel drivers installed outside your images; those updates come from the hardware vendor, your cloud provider and your operating system distribution. For the full list of GPU stack components worth tracking, see the GPU software supply chain post.

FAQ

Which AI chip is the fastest in 2026?

On vendor peak per chip, the AMD MI355X (2.5 PFLOPS dense BF16, 10.1 PFLOPS MXFP4) and NVIDIA's Blackwell parts (2.25 to 2.5 PFLOPS dense BF16, 9 to 15 PFLOPS dense FP4 depending on the system) lead, with Google's Ironwood close behind at 2.3 PFLOPS BF16 and 4.6 PFLOPS FP8. On measured MLPerf Inference v6.0 Llama 2 70B throughput, the GB300, B300, B200 and MI355X systems were at the top. "Fastest" still depends on the workload and the system size.

What is the difference between dense and sparse TFLOPS?

Sparse figures assume 2:4 structured sparsity, where two of every four weights are zero and the hardware skips them, doubling the effective rate. Dense figures assume no such zeros. Most language models run dense, so dense figures are the ones to compare. NVIDIA publishes most Tensor Core figures as sparse; halve them for dense.

How does the B200 compare with the H200?

Per GPU on vendor figures, the B200 in an HGX system has 2.27 times the H200's dense BF16 and FP8 throughput (2,250 against 989.5 TFLOPS BF16), adds FP4, has 180 GB of HBM3E at 8 TB/s against 141 GB at 4.8 TB/s, and doubles NVLink bandwidth to 1.8 TB/s. On MLPerf Llama 2 70B Offline, the top 8-GPU B200 result was 2.94 times the top 8-GPU H200 SXM result per GPU, partly because it used FP4 weights.

How does the AMD MI355X compare with the NVIDIA B200?

On AMD's and NVIDIA's own figures, the MI355X has slightly more dense BF16 and FP8 throughput (2.5 and 5.0 PFLOPS against 2.25 and 4.5), 288 GB of HBM3E against 180 GB, the same 8 TB/s bandwidth, and a higher module power of 1,400 W. In MLPerf Inference v6.0, AMD's 8 x MI355X system reached 99.5 percent of HPE's 8 x B200 system on Llama 2 70B Offline. The main practical difference is the software stack: ROCm against CUDA.

Are Google TPUs faster than NVIDIA GPUs?

Per chip, Ironwood's 4,614 TFLOPS of FP8 and 192 GiB of HBM at about 7.37 TB/s are in the same class as the B200. TPUs differ mainly in system design: thousands of chips connected directly in a torus, programmed through XLA with JAX or PyTorch/XLA, and available only as a Google Cloud service. There are no TPU results in the MLPerf Inference v5.1 or v6.0 closed datacenter files, so there is no like-for-like measured comparison there.

Is AWS Trainium cheaper than NVIDIA GPUs?

On public list prices, yes per chip-hour: an AWS Capacity Block for 16 Trainium2 chips costs $35.7608 per hour, $2.235 per chip, against $5.970 per H100. Per peak dense BF16 PFLOP-hour that is $3.35 against $6.03. Whether it is cheaper per useful token or per training run depends on the MFU you achieve with the Neuron SDK on your model and on the porting work.

Why does memory bandwidth matter so much for LLM inference?

During decode, every weight is read once per generated token while doing only about two FLOPs per weight. That is far below any modern chip's ridge point, so bandwidth, not compute, sets the speed. A 70B model with FP8 weights can generate at most about 48 tokens per second per sequence on one H100's 3.35 TB/s, and about 114 on one B200's 8 TB/s.

What does MFU mean and what is a realistic value?

Model FLOPs utilisation is the share of the chip's peak FLOPS spent on the model's useful arithmetic. Meta's Llama 3 paper reports 38 to 43 percent BF16 MFU on very large H100 clusters. Smaller or less tuned jobs commonly reach less. Use a range when you estimate.

Is Intel Gaudi 3 still worth considering?

On Intel's figures, Gaudi 3 has 1,678 TFLOPS of dense BF16 (more than the H100's 989.5), 128 GB of HBM2e at 3.7 TB/s and 24 integrated 200 GbE ports, so it needs no separate scale-out NICs. Its FP8 rate equals its BF16 rate, so it gains less from FP8 than other chips. It does not appear in the MLPerf Inference v5.1 or v6.0 closed datacenter results. Fit depends on price and on support for your model in the Intel Gaudi software suite.

Where do these numbers come from?

Every specification comes from the vendor's own product pages, white papers or documentation, every measured result from MLCommons' MLPerf announcements and results files, and every price from AWS, Azure or Google Cloud public price pages checked on 25 October 2026. Derived figures were computed in Python and are labelled.

Picking a chip settles the hardware, but every accelerator in this post runs a deep stack of vendor and open source software, and that stack is where most security fixes land. The practices below apply to the components you build into your own training and serving images; firmware and host drivers outside those images still come from the vendor.

More in this series

Other posts in our AI chips and costs series:

• AI Accelerator TCO: Buy vs Rent, Capex, Power, Cooling and Utilization
• Cheapest LLM Inference: How to Cut Cost per Million Tokens
• Fastest LLM Inference: Lowest Latency and Most Tokens per Second
• How Many GPUs Does a 1 Trillion Parameter Model Need?
• How to Estimate AI Training and Inference Cost: Formulas and a Calculator
• LLM Inference Cost by Chip and Model Size, 1B to 1 Trillion Parameters
• LLM Training Cost by Chip and Model Size: 1B to 1T, Worked

Fix it without waiting for upstream

Whatever the vulnerability, most of the risk sits in two gaps. The first is the time between a fix being published and that fix actually running in your production builds. The second, and worse, is the time before any fix exists at all, while a maintainer works out a patch and a release schedule. Waiting through both gaps is a habit, not a requirement. Safeguard gives teams three ways to close them.

1. Autonomous remediation: pull requests, not tickets

When a fixed version exists, Safeguard's Griffin models open a tested, policy gated pull request that moves the affected dependency to the safe version. You can run it on one repository or in bulk across every project that carries the vulnerable version. Remediation covers eight categories, including dependencies, licenses, security posture and findings. Reachability analysis ranks the vulnerable code paths your application actually calls first, so the first pull requests you merge are the ones that matter most. Autonomous remediation ships switched off, with every category set to Manual, so your team decides how much runs on its own and what still needs a human approval.

2. Gold Open Source: the safe version in seconds, free

gold.safeguard.sh is free and needs no login. Search any CVE or package to see affected and fixed versions, CVSS, EPSS, CISA KEV status and a recommended safe version, or use the "Am I affected?" checker. The same data is available through a free read API, README status badges and RSS threat feeds. The safeguard-sh/gold-scan-action GitHub Action can fail a build on a known exploited vulnerability, on any known vulnerability, or never, so a version you have just removed cannot quietly come back in next week's merge.

3. Gold Registry: start from components that are already fixed

registry.safeguard.sh provides hardened, drop in replacements for upstream packages and container images. Every Gold artifact is held continuously to zero exploitable critical or high CVEs, zero malware and a completed license check, with SLSA L3 or higher provenance, a Sigstore signature and a public evidence page you can hand to an auditor. An artifact loses its Gold badge if it regresses. Teams can point their own package directory at the registry so hardened equivalents resolve without changing how developers install anything.

When upstream has no fix yet, or will not ship one, Safeguard first works to get the fix accepted upstream. If the issue is severe or upstream will not move, Safeguard builds a fork, fixes the vulnerability there and publishes a hardened Gold build, so you are not stuck on a maintainer's release schedule. You can also request a hardened, compatibility tested build of a specific package. This fork and patch flow is in early access today, so ask which packages it covers before you plan around it.

One honest limit: when the vulnerable code lives in a vendor appliance, a hosted service or an operating system runtime you do not build yourself, the fix still has to come from that vendor or distribution. The steps above apply to the open source dependencies and container images you build and ship, which is where most teams carry the largest and least tracked share of their exposure.

See how Safeguard would help

The first demo is a real Safeguard CLI session. You install the CLI, open the interactive terminal and ask it to fix a lodash CVE in a repository; it reads the finding and the advisory, asks before any command that changes something, edits package.json, runs the install and tests with your approval and pushes nothing. The same flow applies to the Python and container dependencies of a training or serving repository, whichever accelerator it targets, and safeguard scan --image takes a CUDA, ROCm or Neuron based image.

Click through it at your own pace, or open it full screen.

The second demo sets up the Safeguard runner on your own machine, approves it with a device code and then scans from the terminal. Teams that build accelerator images on their own build hosts, close to the hardware, can run the scans there rather than shipping images elsewhere.

Click through it at your own pace, or open it full screen.

The third demo connects Claude to the Safeguard MCP server with OAuth and shows how you choose which Safeguard tools Claude can use. Once connected, you can ask in plain language which of your projects or images carry a given NCCL, RCCL, container toolkit or serving framework version.

Click through it at your own pace, or open it full screen.

To try it on your own images, download the Safeguard CLI and follow the CLI documentation. For background on accelerators in general, see AI chip architecture, what an AI accelerator is and how it works, GPU memory and interconnects and NVIDIA, AMD, Intel and Apple GPU architectures compared.

Quick check

Sources

• NVIDIA: H100 product page and specifications
• NVIDIA Technical Blog: NVIDIA Hopper Architecture In-Depth, 22 March 2022 (H100 SXM5 HBM3)
• NVIDIA: H200 product page and specifications
• NVIDIA: HGX platform (HGX B300 and HGX B200 specifications)
• NVIDIA: DGX B200
• NVIDIA: DGX B300
• NVIDIA: GB200 NVL72
• NVIDIA: GB300 NVL72
• NVIDIA Technical Blog: Inside NVIDIA Blackwell Ultra, 22 August 2025
• AMD: Instinct MI300X specifications
• AMD: Instinct MI325X specifications
• AMD: Instinct MI355X specifications
• Intel: Gaudi 3 AI accelerator white paper, July 2025 revision
• Google Cloud: TPU v5e documentation
• Google Cloud: TPU v5p documentation
• Google Cloud: TPU v6e (Trillium) documentation
• Google Cloud: TPU7x (Ironwood) documentation
• Google Cloud: TPU system architecture
• Google: Ironwood, the first Google TPU for the age of inference, 9 April 2025
• Google Cloud: Cloud TPU pricing (checked 25 October 2026)
• AWS Neuron documentation: Trainium2 architecture
• AWS Neuron documentation: Trainium3 architecture
• AWS Neuron documentation: Inferentia2 architecture
• AWS Neuron documentation: Amazon EC2 Trn2 architecture
• AWS: EC2 Capacity Blocks for ML pricing (checked 25 October 2026)
• Microsoft Azure: Retail Prices API (queried 25 October 2026)
• Microsoft Learn: ND GB200 v6 series
• MLCommons: MLPerf Training v5.0 results, 4 June 2025
• MLCommons: MLPerf Training v5.1 results, 12 November 2025
• MLCommons: MLPerf Training v6.0 results, 16 June 2026
• MLCommons: MLPerf Inference v5.0 results, 2 April 2025
• MLCommons: MLPerf Inference v5.1 results, 9 September 2025
• MLCommons: MLPerf Inference v6.0 results, 1 April 2026
• MLCommons: Inference v6.0 results repository and summary file
• MLCommons: Inference v5.1 results repository and summary file
• Reddi and colleagues: MLPerf Inference Benchmark, 2020
• Kaplan and colleagues: Scaling Laws for Neural Language Models, 2020
• Hoffmann and colleagues: Training Compute-Optimal Large Language Models, 2022
• Llama Team, Meta: The Llama 3 Herd of Models, 2024
• Pope and colleagues: Efficiently Scaling Transformer Inference, 2022
• Williams, Waterman and Patterson: Roofline, Communications of the ACM 52(4), April 2009

Last reviewed: 25 October 2026. Vendor specifications change between product revisions, some figures here are marked preliminary by their vendors, and cloud prices change often; check the linked pages before you size or price a system on them.