GPU utilization explained: allocation, kernel, MFU and the 60% figure

"GPU utilization" can mean three different measurements: how much of the time you paid for was spent running your code, how much of that time the GPU was running work at all, and how much of the chip's peak arithmetic your model actually used. A 60% result is common on the third and the first, and rare on the second.

This matters because GPUs are rented by the hour and the hourly price is high. This article defines each measure, lists the published figures with their sources, shows why 60% or lower is common, and works out what low utilization does to your cost per useful hour.

Key numbers at a glance

• Model FLOPs utilization (MFU) for PaLM 540B: 46.2% on 6,144 TPU v4 chips, with 45.7% when self-attention is excluded. Hardware FLOPs utilization for the same run: 57.8%. Source: the PaLM paper, arXiv 2204.02311.
• MFU for Megatron-Turing NLG 530B: 30.2% on 2,240 A100 GPUs. Gopher 280B: 32.5% on 4,096 TPU v3 chips. GPT-3 175B: 21.3% on V100 accelerators. Same PaLM paper table.
• Llama 3 405B pre-training on 16,000 H100 GPUs: BF16 MFU of 41% (400 TFLOPs per GPU) in one configuration and 38% (380 TFLOPs per GPU) with very long sequences. Source: the Llama 3 paper, arXiv 2407.21783.
• Llama 3 405B, 54 day snapshot: 466 job interruptions, of which 47 were planned and 419 unexpected, and GPU issues were 58.7% of unexpected ones. The paper also says effective training time was higher than 90%.
• DeepSeek-v3 training: around 20% to 30% MFU, which Modal describes as unofficial, with no official number.
• Raw matrix multiplications: 70% to 80% MFU, according to Modal's guide.
• Modal's guide: the majority of organizations achieve less than 70% GPU allocation utilization at peak demand, citing a 2024 industry report whose underlying data is not reproduced in the guide. The former Banana serverless platform ran at an aggregate utilization of around 20%, per the same guide. Modal says its users can reach in excess of 90% in aggregate, which is a vendor claim.
• dstack single-GPU benchmark (22 July 2025): GPU utilization about 95% to 100% on both a VM and bare metal while fine-tuning a small model.
• CoreWeave 2025 revenue from committed take-or-pay contracts: over 98% (annual report), which means a customer pays for the capacity whether or not it is used.

Why there are three numbers

Imagine renting a delivery truck for a month. You can ask three questions. How many hours was the truck out on the road, as opposed to parked in the yard? While on the road, how often was the engine actually turning? And when it was carrying loads, how full was it compared to its capacity? Each question has a different answer and a different cause.

Modal's guide on GPU utilization makes the same three way split for GPUs. The definitions below follow that guide, with a plain-language reading added.

1. Allocation utilization: did the GPU you paid for run your code?

Modal defines it as GPU-seconds running application code divided by GPU-seconds paid for. If you rent a GPU for 100 hours and your application runs for 60 of them, allocation utilization is 60%.

The losses here are about scheduling and demand. A GPU sits waiting for a job to be submitted. A cluster is reserved for the peak and idle at night. A model is loaded into memory for hours between requests. A team forgot to shut down a notebook. This is the number your invoice cares about.

Modal's guide says the majority of organizations achieve less than 70% allocation utilization at peak demand, citing a 2024 industry survey. The guide also says the Banana serverless platform, which Modal's authors describe as a former platform, ran at an aggregate of around 20%. Modal says its own users can exceed 90% in aggregate. That is a claim from a vendor selling a way to raise utilization, so read it as a goal, not a typical result.

2. Kernel utilization: while allocated, was the GPU busy?

A kernel is a small program that runs on the GPU. Modal defines GPU kernel utilization as GPU-seconds running kernels divided by GPU-seconds paid for. This is close to what the common monitoring tool shows as "GPU utilization" in a simple view, though you should check exactly how your tool defines it.

Losses here are about the GPU waiting on something else: data not yet loaded from disk, a slow data pipeline, the processor not preparing the next batch quickly enough, network transfers, or weights being downloaded. Modal says 90% or more is achievable on GPUs, where on a CPU such a figure would often trigger an alert, and it warns that this metric does not care whether the code running on the GPU is using the hardware's real capacity.

NVIDIA's own tools show why. The nvidia-smi documentation defines its GPU utilization field as the "percent of time over the past sample period during which one or more kernels was executing on the GPU", with a sample period between 1 second and 1/6 second depending on the product. One small kernel running counts the same as the whole chip working flat out. The same page defines memory utilization as the percent of time device memory was being read or written, which again says nothing about how much of the memory bandwidth was used.

NVIDIA's Data Center GPU Manager (DCGM) offers finer profiling metrics that sit between kernel utilization and MFU. Its documentation defines SM activity as "the fraction of time at least one warp was active on a multiprocessor, averaged over all multiprocessors" (a streaming multiprocessor, or SM, is one of the many compute blocks on the chip, and a warp is a group of threads it runs together). DCGM says a value of 0.8 or greater is "necessary, but not sufficient, for effective use of the GPU", and that a value below 0.5 likely indicates ineffective use. It also reports tensor activity, the fraction of cycles the tensor cores that do matrix maths were active, and DRAM activity, where it notes that about 0.8 is the practical maximum. These are the numbers to look at when a dashboard says 100% but the job feels slow.

That warning is the important one. A GPU can be marked as 100% busy while running a tiny, inefficient piece of work that uses a small fraction of its arithmetic units. The dstack benchmark, for example, reports GPU utilization of about 95% to 100% for a small-model fine-tune on both a VM and bare metal. The post does not say that this equals high efficiency of the arithmetic, and a small model on a big GPU is unlikely to fill it.

3. MFU: how much of the chip's peak did the model use?

MFU stands for model FLOPs utilization. A FLOP is one arithmetic operation on a decimal number, and FLOP/s is how many per second. A chip has a theoretical maximum rate. The PaLM paper defines MFU as the observed throughput, in tokens per second, divided by the theoretical maximum throughput of the system at peak FLOPs, counting only the operations the model needs for the forward and backward passes, not extra work done to save memory. That is the paper's definition, and it argues that MFU allows fairer comparisons across systems than hardware FLOPs utilization, which depends on implementation choices.

In plain terms, MFU asks how much of the chip's advertised arithmetic went into useful model work. It is the hardest of the three to raise and the hardest to measure.

What published MFU numbers look like

The PaLM paper's table is a good reference point because it covers several well known training runs:

• PaLM 540B: 46.2% on 6,144 TPU v4 chips.
• Gopher 280B: 32.5% on 4,096 TPU v3 chips.
• Megatron-Turing NLG 530B: 30.2% on 2,240 A100 GPUs.
• GPT-3 175B: 21.3% on V100 accelerators.

The Llama 3 paper reports 41% BF16 MFU for its 405 billion parameter model on 16,000 H100 GPUs in its main configuration, and 38% for the same GPU count with sequences of 131,072 tokens. Modal cites the Llama 3 figure as 38% to 41% and says DeepSeek-v3 training ran around 20% to 30%, without an official number. These are very large, expert-run, heavily tuned training jobs.

Notice what this says. Teams at the top of the field, with custom software and engineers dedicated to the problem, get 40% to 46% of peak on training. So "60% or lower" is not a failure. For MFU it is the normal range, and a figure above 50% would be exceptional. Modal says matrix multiplication alone, a single well suited operation, reaches 70% to 80%, which gives a sense of the practical ceiling.

For inference, Modal says MFU may be closer to the matrix multiplication figure, but it knows of no published large scale results. None of the other sources cited here gives one either.

So why is 60% or lower so common?

The number 60% is not an official benchmark or threshold in any of the sources cited here. It is a convenient round figure, and which of the three measures it describes changes the story.

• If 60% is allocation utilization, it means 40% of the paid time is idle. That is typical, given Modal's report that most organizations are below 70% at peak. The causes are demand patterns and scheduling.
• If 60% is kernel utilization, it is low by Modal's standard, since 90% or more is described as achievable. The cause is usually something starving the GPU, such as data loading.
• If 60% is MFU, it is higher than any of the published large training figures cited here. A well run training job at 40% MFU is doing well.

Several causes pile up at each level, and most are mundane.

Idle time between jobs. Teams reserve capacity for peaks. Research cycles have gaps between experiments. Inference traffic is lumpy, and few services have a flat load.

Waiting for data. The GPU finishes a step and waits for the next batch to be read, decoded and transferred. Slow storage or a slow input pipeline leaves the chip idle.

Communication. In large training, GPUs must exchange results with each other after each step. While they wait for that exchange, they are not computing. The Llama 3 configuration with longer sequences shows a lower MFU, 38% against 41%, which is one example of how layout choices change efficiency.

Memory limits. Many operations are limited by how fast data moves in and out of GPU memory, not by arithmetic speed. This is especially true for generating text, which the companion explainer on LLM throughput covers.

Failures and restarts. The Llama 3 paper reports 466 job interruptions in a 54 day snapshot, 419 of them unexpected, with GPU problems the largest category at 58.7% of the unexpected ones. The paper says effective training time stayed higher than 90%, but each interruption costs restarts and lost progress, and a larger cluster fails more often.

Software efficiency. A model not written to use the GPU's features well can leave much of the chip unused. Modal's guide says high utilization on the first two measures does not imply high MFU.

What low utilization costs

Take a GPU billed at $3.99 per hour, which is the Lambda 8-GPU H100 SXM list price per GPU in the IntuitionLabs survey of 10 October 2026 and on Lambda's pricing page on 19 October 2026. The figures below are simple division on that list price, not a quote.

• At 90% allocation utilization, the effective cost per busy hour is $3.99 divided by 0.9, or about $4.43.
• At 60%, it is $3.99 divided by 0.6, or $6.65.
• At 40%, it is about $9.98.
• At 20%, the level Modal attributes to Banana, it is about $19.95.

Now add MFU. If your training job runs at 41% MFU, the chip is doing useful model arithmetic at 41% of its peak. To buy the equivalent of one hour of fully used peak arithmetic you pay for about 2.4 hours, which at $3.99 is about $9.73. If that GPU is also only allocated 60% of the time, the number rises again. The three losses multiply. These figures are calculated from the numbers above.

This is why buyers who think about cost per result, such as dollars per million tokens or per finished training run, tend to focus on utilization more than on hourly price. A 20% cheaper rate is less valuable than raising allocation utilization from 40% to 60%.

Why it matters to a provider and a customer differently

On take-or-pay contracts, the customer pays for capacity whether it uses it or not. CoreWeave's annual report says committed contracts were over 98% of its 2025 revenue. So the customer carries the utilization risk on a committed cluster. That gives a customer strong reason to measure allocation utilization and plan capacity carefully, and to ask about options such as reselling or resizing.

For on-demand rentals, you pay only while the instance runs, but you can still pay for idle time if you forget to stop it.

How to raise each measure

For allocation utilization: schedule jobs to fill gaps, shut down idle instances, use shared queues for teams, mix steady and interruptible work, and consider pay-per-use serverless options for lumpy inference. The companion explainer on GPU rental prices covers spot and committed options.

For kernel utilization: profile your pipeline, load data in parallel with compute, keep data on fast storage near the GPUs, and avoid host side work between steps.

For kernel and SM level efficiency: watch DCGM's SM activity and tensor activity alongside the simple utilization figure, using NVIDIA's 0.8 guide for SM activity as a floor rather than a goal.

For MFU: use well tuned open frameworks and kernels, choose batch sizes and parallelism layouts that suit the hardware, and measure on your own model. Remember published training values of 30% to 46% before setting a target.

What is firm and what is uncertain

Firm: the definitions in the PaLM paper, Modal's guide and NVIDIA's nvidia-smi and DCGM documentation, the MFU figures in the PaLM and Llama 3 papers, the interruption statistics in the Llama 3 paper, and CoreWeave's contract mix.

Uncertain: how common 60% is on each measure across the industry, because no public census exists in the sources cited here. The 2024 survey Modal cites is known here only through Modal's summary. The DeepSeek-v3 MFU is described by Modal as unofficial. Inference MFU has no published large scale figure in these sources. The cost illustrations rely on a single list price and on simple division.

What to watch next

• Published MFU numbers for large inference deployments.
• Whether new chips and software raise typical training MFU above the 40% to 46% range.
• Providers publishing utilization or idle capacity data for customers.
• More detailed failure statistics from large clusters.

For related reading, see LLM throughput metrics explained, GPU rental prices in 2026, bare metal against virtualized GPU cloud and the AI chip market in 2026.

For the open source code running on those GPUs, Safeguard offers SBOM and dependency scanning and a free scan at gold.safeguard.sh/scan.

Quick check

Sources

• PaLM: Scaling Language Modeling with Pathways, arXiv 2204.02311 (HTML version)
• PaLM abstract page, arXiv
• The Llama 3 Herd of Models, arXiv 2407.21783 (HTML version)
• GPU utilization guide, Modal
• Benchmarking AMD GPUs: bare-metal and VMs, dstack, 22 July 2025
• CoreWeave Form 10-K for fiscal year 2025, SEC EDGAR
• nvidia-smi documentation, NVIDIA, read 19 October 2026
• DCGM feature overview, profiling metrics, NVIDIA, read 19 October 2026
• Lambda pricing, read 19 October 2026
• H100 rental prices compared, IntuitionLabs, first-party prices checked 10 October 2026

Modal sells GPU infrastructure, and its figures are its own claims where marked. Cost figures are simple division on a published list price.

Last reviewed: 19 October 2026.