Install the NVIDIA driver, add NVIDIA's package repository, install the container toolkit packages, then run the toolkit command that configures the Docker runtime and restart Docker. Each step changes the host, so pin versions, review the configuration diff, and record what you installed.
The steps, in order
Based on NVIDIA's install guide:
- Install the GPU driver, preferably through your distribution's package manager, as the guide recommends.
- Add NVIDIA's production repository for your package manager. Experimental packages are optional, and a production host does not need them.
- Install the four packages: nvidia-container-toolkit, nvidia-container-toolkit-base, libnvidia-container-tools and libnvidia-container1, all pinned to the same version.
- Configure Docker:
sudo nvidia-ctk runtime configure --runtime=docker
sudo systemctl restart docker
The first command updates /etc/docker/daemon.json. The restart makes Docker read it.
What to check at each step
| Step | What to review |
|---|---|
| Repository | Use the vendor documented repository for your package manager |
| Packages | Pin all four to one version so they cannot drift apart |
| Configure | Look at the change before you apply it |
| Restart | Plan it, because it restarts the container runtime |
The guide notes that the nvidia-ctk config command prints its changes to standard output unless you pass the in place or output options. It advises reviewing previews before production updates, especially when you use a specific config file. That preview step is a cheap control, so use it.
Variants worth knowing
Other container engines are supported: containerd, CRI-O and Podman. For Podman, NVIDIA recommends the Container Device Interface for device access. Rootless Docker needs extra steps, including setting the no cgroups option in the toolkit configuration, so do not copy the rootful commands blindly.
The guide also lists a known cgroup issue: on systems using the systemd cgroup driver, containers can lose access to the GPUs after a daemon reload. That is a reliability point, but it explains why teams are tempted to loosen settings to make things work. Fix the cause rather than adding privileged mode.
Habits that keep the host tidy
- Keep the toolkit current. A serious flaw was fixed in v1.16.2, covered in our CVE-2024-0132 check.
- Do not run training containers as root by default. See should ML training containers run as non-root.
- Pin images by digest, as described in pin CUDA and ML base images by digest.
- Scan the image before it reaches the GPU host.
Try it yourself
Click through it at your own pace, or open it full screen. Watch how packages inside an image are listed and checked.
Safeguard can scan container images and dependencies, and its free scan lives at gold.safeguard.sh/scan.
Quick check
Do I need the toolkit just to run a CPU container?
No. The toolkit is for exposing NVIDIA GPUs to containers. Hosts without GPUs do not need it.
Is it fine to copy the rootful commands for rootless Docker?
No. The guide lists different commands and an extra configuration option for rootless setups.
Sources
Last reviewed: 19 October 2026.
