llama.cpp supports CPU plus GPU hybrid inference, which it describes as partially accelerating models larger than the total VRAM. You choose how many layers go to the GPU, and the rest run on the CPU. It lets a big model run on a small card, at a speed cost.
What the project says
The llama.cpp README states its main goal as enabling LLM inference with minimal setup and state of the art performance. It lists backends including Metal for Apple silicon, CUDA for NVIDIA GPUs, Vulkan, and CPU support with AVX, AVX2, AVX512 and AMX on x86. It also lists CPU plus GPU hybrid inference to partially accelerate models larger than the total VRAM capacity.
The layer setting
The server README documents a GPU layers option. Its value can be an exact number, auto or all, and the default is auto. A transformer model is a stack of layers. Each layer you place on the GPU runs faster, and each layer left on the CPU runs slower.
llama-server -m model.gguf -ngl 24
The number is yours to tune. Confirm option spelling in your build's help output.
Finding a setting that fits
- Start with auto and read the startup log to see what was placed on the GPU.
- If you run out of memory, lower the number of GPU layers or the context size.
- If you have memory to spare, raise it until just before failure, leaving some margin.
- Measure tokens per second at each step with your real prompt length.
Remember that the KV cache also needs memory. Offloading layers frees weight memory but the cache still has to live somewhere. See how the KV cache grows.
What it costs
I do not have a published number to give you for the slowdown, and it varies a lot with the hardware. The general shape is clear from how the system works. Layers on the CPU read their weights from system memory, which is slower than GPU memory, so generation speed drops as more layers move off the GPU. Prompt processing and token generation can be affected differently, so test both with a realistic workload.
Hugging Face's cache documentation describes a similar trade for the KV cache: offloading to the CPU saves GPU memory at some throughput cost.
When to choose something else
Hybrid offload is a good way to try a larger model or to run occasional jobs. For daily interactive use, compare it against:
- a smaller model at a higher quant, see choosing a quant level
- a lower quant of the larger model
- a hosted model for tasks that do not touch sensitive data, see the local or hosted decision guide
Quick check
Does offloading change the model's answers?
It moves where computation runs. It should not change what the model is, though small numeric differences between backends are possible.
Is more GPU layers always better?
More layers on the GPU is faster, until you run out of memory for the cache and buffers. Leave margin.
Sources
Last reviewed: 19 October 2026.
