Start from the memory you can spare, subtract room for the KV cache and the system, and pick the highest bit quant whose file fits in what is left. Then test it on your own prompts. Going lower than Q4 class quants is where quality loss tends to show, so verify before you commit.
The method in four steps
- Find your budget. On a discrete GPU, that is free VRAM. On a machine with shared memory, it is total memory minus what the system and your other apps need.
- Reserve cache and buffers. Work out the KV cache for the context you actually use. See how the cache grows.
- Compute file sizes. Use bits per weight to estimate each option.
- Pick the biggest that fits, then test it.
The Hugging Face GGUF page lists the effective bits per weight for the K-quants: Q6_K at 6.5625, Q5_K at 5.5, Q4_K at 4.5, Q3_K at 3.4375 and Q2_K at 2.625. It also lists IQ4_XS at 4.25 and IQ3_S at 3.44, which use an importance matrix. The page marks the older Q4_0 and Q8_0 types as legacy methods that are not used widely.
Decision table for a 13B model
Weight memory is parameters in billions times bits per weight divided by 8. These are estimates, computed from the published numbers.
- Q6_K at 6.5625 bits: about 10.7 GB
- Q5_K at 5.5 bits: about 8.9 GB
- Q4_K at 4.5 bits: about 7.3 GB
- Q3_K at 3.4375 bits: about 5.6 GB
- Q2_K at 2.625 bits: about 4.3 GB
If you have 12 GB free and need a modest context, Q5_K at about 8.9 GB leaves around 3 GB for cache and buffers. With 8 GB free, Q4_K is roughly at the limit, and a smaller model might serve you better than a lower quant of the big one.
Quality is task dependent
The quantization papers support the general idea that moderate quantization can keep accuracy. The GPTQ paper reports 3 or 4 bit weights with negligible accuracy degradation relative to the uncompressed baseline on the models it tested, and reasonable accuracy at 2 bit or ternary levels. The LLM.int8 paper reports halving inference memory at 8 bit without degradation on the models it studied.
Those are results on specific models and benchmarks, using methods that differ from GGUF's K-quants. Hugging Face's own guide says that 4 bit quantization can in practice often lead to different results compared to 8 bit or bfloat16, and that it is up to the user to try it out.
So the table gives you candidates, and your task decides. Code generation, structured output and non English text are all places to look for early degradation.
Does a lower quant run faster?
Not automatically. The Hugging Face guide notes that in its bitsandbytes example inference was a bit slower with quantized weights, because of the dequantization work. Local runtimes with optimized kernels behave differently, and memory bandwidth often dominates. Measure tokens per second yourself on your hardware rather than assuming.
A quick test routine
- Pick 20 to 30 real prompts from your work.
- Run them on two adjacent quants, say Q4_K and Q5_K, with the same settings.
- Compare the answers side by side for errors that matter to you.
- Keep the smaller one if you cannot tell them apart.
The eval guide shows how to keep this repeatable. For the format itself, see what GGUF and quantization are.
Quick check
Is Q4_K the same as 4 bit?
Not exactly. Q4_K stores about 4.5 bits per weight once block scales are counted, per the Hugging Face GGUF page.
Should I always take the lowest quant that fits?
No. Take the highest that fits with room for cache. Lower quants save memory but are where quality loss is most likely to show.
Sources
- GGUF, Hugging Face Hub docs
- GPTQ: Accurate Post-Training Quantization for Generative Pre-trained Transformers (arXiv)
- LLM.int8(): 8-bit Matrix Multiplication for Transformers at Scale (arXiv)
- Optimizing LLMs for Speed and Memory, Hugging Face Transformers docs
Last reviewed: 19 October 2026.
