Quantization stores a model's weights using fewer bits, for example 4 or 8 instead of 16. The model needs less memory and can run faster, usually with a small loss in accuracy. How small depends on the method, the model and your task, so test it.
The basic idea
A model's weights are numbers. Storing each as a 16 bit value takes twice the memory of 8 bits and four times that of 4 bits. Quantization maps the original values onto a smaller set of levels, with scaling factors to keep them close.
Two broad types exist. Post-training quantization converts a finished model. Quantization aware training bakes the lower precision into training.
What GPTQ reports
The GPTQ paper describes a one-shot weight quantization method that reduces weights to 3 or 4 bits each with little accuracy loss compared with the uncompressed model. The authors report end-to-end speedups of about 3.25 times on NVIDIA A100 GPUs and about 4.5 times on A6000 GPUs against FP16 inference. They say it can quantize a 175 billion parameter GPT model in roughly four GPU hours and run generative inference on a single GPU, where it would otherwise need several.
These are the authors' numbers on their setup and hardware. Treat them as evidence the approach works rather than a promise for your deployment.
Quantization during training
QLoRA uses a related idea for fine-tuning: it keeps the base model frozen in 4-bit form and trains small adapters on top. The authors report that this allows a 65B model to be tuned on a single 48 GB GPU. See QLoRA explained.
Trade-offs
| Choice | Memory | Speed | Risk |
|---|---|---|---|
| 16 bit | Highest | Baseline | None from precision |
| 8 bit | About half | Often faster | Small quality drift |
| 4 bit | About a quarter | Often faster | More drift on hard tasks |
| 2 to 3 bit | Lowest | Varies | Noticeable loss likely |
The memory column follows from arithmetic on weight storage only. Real usage also includes activations and the key value cache.
How to test it
- Run your eval set on the full precision model. Save scores.
- Quantize and rerun the same set.
- Compare by category. Rare cases and long inputs tend to degrade first.
- Measure latency and memory on the real hardware you will deploy to.
- Re-check safety behaviour, see fine-tuning and safety.
Supply chain angle
Quantized models are often shared as converted files from third parties. Check who produced the file, compare hashes where provided, and prefer formats that cannot execute code on load. Treat model files like any downloaded dependency.
Quick check
Does 4-bit mean four times faster?
No. Smaller weights reduce memory traffic, but actual speedups depend on hardware and software. The GPTQ paper reports about 3.25 to 4.5 times over FP16 in its tests.
Can I fine-tune a quantized model?
Yes, with methods like QLoRA that train adapters on a frozen quantized base.
Sources
- GPTQ: Accurate Post-Training Quantization for Generative Pre-trained Transformers, fetched 19 October 2026.
- QLoRA: Efficient Finetuning of Quantized LLMs, fetched 19 October 2026.
Last reviewed: 19 October 2026.
