Quantization runs a model at lower numerical precision, so its weights take fewer bits and use less GPU memory. That can let a model fit on smaller or fewer GPUs. The tradeoff is precision, which can reduce output accuracy, so test on your own tasks before you rely on it.
The core idea
The vLLM documentation puts it in one sentence: quantization trades off model precision for a smaller memory footprint. Storing each weight in fewer bits means the model needs less GPU memory, and the same page says that lets larger models fit on a wider range of devices.
The documentation lists many methods, including several weight formats and options for quantizing the key value cache, and says hardware support varies by method, so check the compatibility table before choosing. It does not quantify accuracy loss, because that depends on the model and method.
Why this belongs in a cost conversation
Serving a model yourself shifts the question from per-token rates to hardware and utilization. Memory is often the first constraint, since the weights must fit before anything else. A smaller footprint can mean fewer GPUs for the same model, or room for more concurrent requests on the same GPU. This is general reasoning, not a measured claim, and the savings you see depend on your model, hardware and traffic.
Hosted providers make these choices behind the API and do not tell you the details. This post does not make claims about what any hosted provider does internally.
What can go wrong
| Risk | What to check |
|---|---|
| Accuracy loss on your task | Run your evaluation set on both versions |
| Uneven loss across tasks | Test long context, code and non-English text separately |
| Hardware mismatch | Confirm the method is supported on your GPU |
| Unverified weights | Quantized files are often community conversions, so verify provenance |
| Changed behavior | Re-run safety and refusal tests, not only quality tests |
A test plan
- Build a fixed evaluation set from real tasks, with expected outcomes.
- Run the full precision model and record scores, latency and memory.
- Run the quantized version on the same set.
- Compare scores per task type, not only the average.
- Compare memory and throughput under realistic load.
- Decide using your own acceptance threshold, written in advance.
Keep results with the model version and method so you can reproduce them.
The supply chain angle
Quantized models are often distributed as separate artifacts from the original. Treat them like any dependency. Pin a revision, verify the source, scan the container that serves them, and store the hash. We cover related checks in how to scan a GPU inference container image and Hugging Face offline mode for production inference.
If you are weighing self-hosting against a hosted API, read what goes into a self-hosted versus hosted cost comparison.
Where Safeguard fits
For the serving stack itself, Safeguard's SBOM and dependency scanning can inventory the packages in an inference container, and the free scan at gold.safeguard.sh/scan is a way to look at one of your repositories. Neither evaluates model accuracy.
Worked example: will it fit?
The sizes are rough and illustrative. A model with eight billion weights stored at 16 bits per weight needs about 16 gigabytes for weights alone, because eight billion times two bytes is sixteen billion bytes. At 8 bits it needs about 8 gigabytes, and at 4 bits about 4 gigabytes. These figures cover weights only. The key value cache, activations and the serving software need memory on top, and the cache grows with context length and concurrent requests.
| Precision | Approximate weight memory | Effect |
|---|---|---|
| 16 bit | 16 GB | Baseline |
| 8 bit | 8 GB | Half the weight memory |
| 4 bit | 4 GB | A quarter, with more accuracy risk |
So a GPU with limited memory might not hold the 16 bit version at all, but could hold the 8 bit one with room for the cache. That is the practical meaning of fitting on smaller hardware. It does not tell you the quality is acceptable, which is why the test plan matters.
Acceptance criteria to write before testing
- A minimum score on your evaluation set for each task type.
- A maximum allowed drop versus the full precision model.
- A latency target at your expected concurrency.
- A pass on safety and refusal checks you already run.
If any criterion fails, try a different method or a larger model rather than lowering the bar afterward.
Keeping the setup repeatable
Record the model name, revision, quantization method, serving software version and hardware in a file next to the deployment. When you update any of them, rerun the evaluation. Without this, quality drifts without anyone noticing.
Choosing a method
The vLLM page lists many options and recommends one compressor tool as a starting point, while noting that hardware support varies. A reasonable order is to confirm your GPU supports the method, try the least aggressive precision first, run your evaluation, and only then try lower precision if memory still does not fit. Keep each result in a table with the method, memory use, throughput and scores, so the decision is easy to defend later.
Operating notes
Keep a record of what you serve. The model name and revision, the quantization method, the serving software version, the GPU type and the container image digest should all live in version control. If a user reports a quality regression, you can then tell whether something changed. Rerun your evaluation after any of those items changes, even a minor serving update, because numerical kernels can differ between versions.
Plan for rollback. Keep the previous image and weights available so you can revert quickly. And plan for patching: serving software has security advisories like any dependency, and a quantized deployment needs the same update discipline as the rest of your stack.
What this means for you
If you plan to serve a model yourself, quantization is one of the first options you will meet, because memory is often the limiting factor. It can let a model fit where it otherwise would not. But it changes the model, so treat it as a change that needs testing, not a free setting.
Common mistakes
The common errors are trusting a leaderboard instead of your own evaluation, testing only the average and missing a drop on one task type, downloading a converted file without checking where it came from, and forgetting to rerun the evaluation after a serving software update. Some teams also forget that the memory for the key value cache grows with context length and concurrency, so a model that fits on paper may not fit under load. Test on your tasks, at your concurrency, with a pinned and verified artifact.
Try it yourself
Click through it at your own pace, or open it full screen. It scans a container image for known issues. A self-hosted model runs in an image like this, so look at how the scan lists packages and findings, and picture it applied to your serving container.
Quick check
Does quantization change what the model can do?
It can reduce accuracy because precision is lower. The amount depends on the model and method, so test on your tasks.
Does quantization always save money?
Not automatically. It reduces memory use, which may allow cheaper hardware or more concurrency, but savings depend on your setup.
Are quantized model files safe to download?
Treat them like any dependency. Verify the source, pin a revision, and scan what you run.
Sources
Last reviewed: 19 October 2026.
