The cost of serving tokens from your own GPU is a division. Dollars per hour on top, tokens per hour underneath. The top number is published. The bottom number depends on your model, your batch size, your prompt lengths and how busy the GPU is, so no price page can give it to you. This post gives the formula, real hourly prices from 18 October 2026, and a table of assumed throughputs. It is an estimate, labelled as one.
Published hourly prices
| GPU | Provider and product | Price per GPU hour |
|---|---|---|
| NVIDIA H100 SXM 80 GB | Lambda, on demand, 8 GPU instance | $3.99 |
| NVIDIA H100 | Together AI, GPU clusters, on demand | $3.99 |
| NVIDIA H100 | Together AI, dedicated inference, on demand | $5.49 |
| NVIDIA B200 SXM6 180 GB | Lambda, on demand, 8 GPU instance | $6.69 |
| NVIDIA B200 | Together AI, GPU clusters, on demand | $8.19 |
| NVIDIA B200 | Together AI, dedicated inference, on demand | $8.99 |
The Lambda page lists 8 GPU rows first and does not label them, so we read them as the 8 GPU configuration. Lambda's page does not list H200. Together shows reserved prices lower than on demand, for example H100 at $3.19 for reserved 181 days or more. We did not fetch purchase prices for owned hardware, so owning is not covered here.
The formula
cost per million tokens = price per GPU hour / (tokens per second x utilization x 3,600 / 1,000,000)
Tokens per second means output tokens a GPU generates in total across all requests, not per user. Utilization is the share of the hour the GPU is doing useful work.
Estimate table
These throughputs are assumptions chosen to show the range, not benchmarks of any model or GPU. Use your own measurements. Values are dollars per million output tokens.
| GPU price per hour | 300 tok/s, 30% busy | 300 tok/s, 90% busy | 1,000 tok/s, 60% busy | 3,000 tok/s, 30% busy | 3,000 tok/s, 90% busy |
|---|---|---|---|---|---|
| H100 at $3.99 | $12.32 | $4.10 | $1.85 | $1.23 | $0.41 |
| H100 at $5.49 | $16.94 | $5.65 | $2.54 | $1.69 | $0.56 |
| B200 at $6.69 | $20.65 | $6.88 | $3.10 | $2.07 | $0.69 |
| B200 at $8.99 | $27.75 | $9.25 | $4.16 | $2.77 | $0.92 |
How to read it
The same GPU can cost $0.41 or $12.32 per million tokens depending on assumptions. That is a factor of 30. A low-utilization GPU is the usual way self hosting turns out dearer than an API. A well fed GPU at high throughput can land below hosted per token prices, such as the $0.60 output price Together lists for gpt-oss-120B, but you only learn that by measuring.
Also add what the formula leaves out: staff time, monitoring, idle nights and weekends, model upgrades, redundancy and the egress and storage bill.
Run it
price_per_hour = 3.99 # from the provider page
tokens_per_second = 1000 # measure this on your model
utilization = 0.6
tokens_per_hour = tokens_per_second * utilization * 3600
print(round(price_per_hour / (tokens_per_hour / 1e6), 2)) # dollars per MTok
For a direct comparison with hosted APIs, see open weight models versus API cost. Safeguard lets you bring your own model, including local or on-prem ones.
Quick check
What single number most changes cost per million tokens?
Utilization together with throughput. The hourly price is fixed, so the tokens you actually produce per hour decide the unit cost.
Sources
- Lambda pricing, fetched 18 October 2026
- Together AI pricing, fetched 18 October 2026
