GGUF is a binary file format for storing language models for local inference, holding both the weights and metadata. Quantization stores those weights in fewer bits, so the model uses less memory and runs faster on ordinary hardware, usually at some cost in output quality that you should measure.
What GGUF is
The Hugging Face documentation describes GGUF as a binary format optimised for quick loading and saving of models, designed for use with GGML and other executors. It was developed by the author of llama.cpp. Unlike tensor only formats such as safetensors, GGUF encodes both the tensors and a standardised set of metadata.
In practice: one file, you point your local runner at it, and it loads.
What llama.cpp does with it
The llama.cpp README describes the project as LLM inference in C/C++, aimed at running models locally and in the cloud. It supports CPU use with no dependencies, optimised paths for Apple silicon, CUDA for NVIDIA, HIP for AMD, Vulkan and others, and hybrid CPU plus GPU inference for models larger than video memory. It lists 1.5 bit through 8 bit integer quantization to reduce memory use and speed up inference, and can launch an OpenAI compatible API server.
What quantization means
A model's weights are numbers, normally stored in 16 bits each. Quantization stores them using fewer bits, grouped in blocks with a scale value. Fewer bits means a smaller file and less memory traffic.
The Hugging Face table lists the types. A few examples, with the bits per weight it states:
- Q8_0: 8 bit, a legacy method per the table.
- Q6_K: 6 bit, 6.5625 bits per weight.
- Q5_K: 5 bit, 5.5 bits per weight.
- Q4_K: 4 bit, 4.5 bits per weight.
- Q2_K: 2 bit, 2.625 bits per weight.
The "K" types use super blocks with per block scales, which is why the bits per weight come out a little above the headline number.
What you give up
Lower bit widths reduce quality. How much depends on the model and the task, and it varies too much for a general number to be useful. A summarisation task may tolerate Q4 well, while exact code generation may not. Test on your own prompts.
A practical selection method
- Find your memory budget: the model file plus room for the context cache.
- Pick the largest quantization that fits comfortably.
- Run twenty of your real tasks on it, and on one step higher if it fits.
- Compare outputs. If the lower one is good enough, keep it.
A bigger model at lower precision often beats a smaller model at higher precision, but test both for your use. That is a common rule of thumb, not a guarantee.
Check the source of the file
A GGUF file is a download from a stranger, in most cases. Prefer the original publisher's repository, check the publisher name carefully, and note that model files and the tools that load them have had vulnerabilities. See embedding model supply chain risks.
Quick check
Is GGUF the same as safetensors?
No. Safetensors stores tensors only. GGUF stores tensors plus metadata, and targets GGML based runtimes.
Can I fine-tune a quantized GGUF file?
Training is normally done on the original weights, then the result is converted. See QLoRA explained.
Sources
Last reviewed: 19 October 2026.
