GPU Software Supply Chain: Drivers, CUDA, cuDNN, NCCL and Your SBOM

The GPU software supply chain is every piece of software between your model code and the silicon: the kernel driver and its firmware, the user mode CUDA driver, the CUDA toolkit and runtime, libraries such as cuBLAS, cuDNN and NCCL, the framework wheels that bundle them, the container images, and the container toolkit that wires GPUs into those containers. Each layer has its own version, its own release cadence and its own security bulletins, so each one belongs in your SBOM, split between the host and the image.

Most teams track the Python packages in an AI service and stop there. The layers underneath are treated as "the GPU stuff", installed once by whoever built the node image and then left alone. That is a mistake with a paper trail: NVIDIA's container toolkit alone has had six security bulletins between September 2024 and June 2026, and the September 2026 GPU display driver bulletin lists 114 CVE identifiers. This post walks the stack layer by layer, explains the compatibility rules that make upgrades feel risky, shows the real advisories, and then lays out exactly what to record in an SBOM and how to pin and patch each layer.

It is part of our series on how processors work. For the hardware side, see our explainers on AI chip architecture, what an AI accelerator is and how a GPU is made. A note on method: every version number, date and score below comes from a vendor document or a standards body, linked in the Sources section. The Python scripts in this post were run on a Mac with sample input; the shell commands that talk to a GPU follow the vendor documentation but were not executed, because the machine this was written on has no NVIDIA GPU.

Key takeaways

• The NVIDIA stack has at least eight separately versioned layers: GPU firmware (VBIOS and GSP), the kernel mode driver, the user mode driver (libcuda.so), the CUDA runtime and toolkit, math and deep learning libraries (cuBLAS, cuDNN, NCCL, TensorRT), framework wheels, container images, and the container toolkit plus orchestration add ons. AMD's ROCm and Intel's oneAPI stacks have the same shape.
• The driver lives on the host. The CUDA runtime and libraries usually live in the image. The container toolkit joins the two at container start. That split decides which SBOM a component belongs in and who patches it.
• CUDA has three compatibility modes: backward (newer driver, older toolkit, always fine), minor version (older driver within the same major family, with limits such as no PTX JIT), and forward (older driver across major families, through the cuda-compat package on data center GPUs only).
• Compatible is not the same as patched. A driver can satisfy CUDA's minimum version and still sit below the fixed version in the latest security bulletin.
• Since 1 October 2026 NVIDIA also publishes its security bulletins on GitHub in Markdown, CSAF and CVE record formats, which makes machine matching against your inventory practical.
• An image SBOM does not see the host driver, and a host package list does not see the CUDA libraries inside wheels. You need both, linked, plus the firmware versions that only nvidia-smi reports.
• Pin images by digest, because NVIDIA's CUDA image tags are refreshed for CVE fixes and deleted on a published schedule.

What is the GPU software supply chain?

Think of it as two stacks that meet at a single interface. The host stack is owned by whoever runs the machine: firmware, kernel driver, user mode driver, container runtime and the NVIDIA Container Toolkit. The workload stack is owned by whoever builds the image: CUDA runtime, math libraries, cuDNN, NCCL, the framework, the serving engine and the application. The meeting point is the CUDA driver API, the user mode driver that the runtime calls into.

 WORKLOAD STACK (inside the container image, owned by the image builder)
 +-------------------------------------------------------------------+
 | Your model code, serving app, tokenizers, web framework           |
 | Serving engine: Triton Inference Server, TensorRT-LLM, others      |
 | Framework: PyTorch, JAX, TensorFlow (Python wheels)               |
 | DL libraries: cuDNN, NCCL, TensorRT, cuSPARSELt, NVSHMEM           |
 | CUDA math libraries: cuBLAS, cuFFT, cuRAND, cuSOLVER, cuSPARSE     |
 | CUDA runtime (libcudart), NVRTC, nvJitLink                          |
 +------------------------------ CUDA driver API --------------------+
 HOST STACK (on the node, owned by the platform team)
 +-------------------------------------------------------------------+
 | User mode driver: libcuda.so, PTX JIT compiler, NVVM               |
 | NVIDIA Container Toolkit: runtime, hook, libnvidia-container, CDI  |
 | Container runtime (containerd, CRI-O, Docker) and Kubernetes add ons|
 | Kernel mode driver: nvidia.ko and related modules                  |
 | GPU firmware: VBIOS, GSP firmware, InfoROM                          |
 +-------------------------------------------------------------------+
 HARDWARE: GPU, NVLink or PCIe, host CPU

Every box in that diagram has its own version and release stream, and several have their own security bulletins. The questions are the ones you ask for npm or Maven: what is running, where it came from, what is vulnerable, and how to reach a fixed version without breaking the workload. The GPU twist is stricter coupling between layers, and components that ordinary inventory tools do not see.

The layers, one by one

GPU firmware: VBIOS, GSP and InfoROM

The lowest software layer is firmware on the board itself. nvidia-smi -q reports a VBIOS version, an InfoROM image version and, on GPUs that use it, a GSP firmware version. NVIDIA's nvidia-smi documentation describes the InfoROM as "a small, persistent store of configuration and state data for the GPU", and notes that the InfoROM image version, like the VBIOS version, uniquely identifies what is flashed on the board.

GSP firmware is the one that surprises people. NVIDIA's open kernel module README states that the kernel modules "must be used with GSP firmware and user-space NVIDIA GPU driver components from a corresponding" driver release. In other words, GSP firmware ships with the driver and is versioned with it, while the VBIOS is flashed on the board and changes only when someone updates it deliberately. For an SBOM that means two different firmware entries with two different owners: GSP firmware moves with every driver upgrade, VBIOS updates come from the board or server vendor.

The kernel mode driver

The kernel mode driver is the nvidia kernel module and its companions. NVIDIA publishes an open source version of these modules on GitHub. At the time of writing the README describes release 615.78.08, states that the modules can be used "on any Turing or later GPU", and that they can be built for x86_64 or aarch64. Individual files are MIT licensed unless noted otherwise. NVIDIA also notes in the same README that it does not expect to provide per commit history, with "only be one git commit per driver release".

Open source does not mean invulnerable: the September 2026 GPU display driver bulletin includes several CVEs in "the open-source kernel module". What the open module gives you is source you can read and rebuild, and a version that maps one to one onto a driver release.

How the driver is installed matters as much as which driver. NVIDIA ships it as a .run installer and as distribution packages, and a runfile install does not register with dpkg or rpm. An inventory that only asks the package manager will report no NVIDIA driver on a machine that has one, so ask the driver directly through nvidia-smi or /proc/driver/nvidia/version.

The user mode driver: libcuda.so and friends

NVIDIA's CUDA Compatibility guide is explicit: "The driver package includes both the user mode CUDA driver ( libcuda.so ) and kernel mode components necessary to run the application." The user mode side also includes the PTX JIT compiler (libnvidia-ptxjitcompiler.so), NVVM for JIT link time optimisation, and other shared objects.

This is the layer containers borrow from the host. The CUDA container images contain the CUDA runtime, but not libcuda.so; that comes from the host's driver installation and is made available inside the container when it starts. nvidia-smi reports the version this layer supports as "CUDA Version", which the tool's documentation defines as "The CUDA 'User Mode Driver' version". That number is the newest CUDA version the installed driver supports, not the version of any toolkit installed on the machine. Confusing the two is one of the most common inventory errors we see.

The CUDA toolkit and runtime

The CUDA toolkit is a bundle of separately versioned components. The release notes for CUDA 13.4 Update 2 list them in a table, and the version numbers are not all the same:

Source: NVIDIA CUDA Toolkit 13.4 Update 2 release notes, Table 1.

That table is the first SBOM lesson of this post. "CUDA 13.4" is not a component version. If an advisory says "cuBLAS before 13.8", you need the cuBLAS version, and it is not 13.4.

The second lesson is in the same release notes: "The NVIDIA driver is no longer bundled with the CUDA Toolkit, on Windows starting with CUDA 13.1, and on Linux starting with CUDA 13.4." NVIDIA's compatibility guide repeats it: before CUDA 13.4 every toolkit shipped with a driver package "for convenience", and starting with 13.4 "the CUDA driver must now be obtained separately". Install scripts written for older toolkits that assumed the toolkit installer would also update the driver will now silently leave the driver where it was.

The toolkit also contains developer tools that rarely reach production but often reach CI. The January 2026 CUDA Toolkit bulletin fixed four CVEs in Nsight Systems and Nsight Visual Studio Edition, all marked as affecting "All versions prior to CUDA Toolkit 13.1". If your build images use the devel flavour of the CUDA image, those tools are in them.

Math and deep learning libraries: cuBLAS, cuDNN, NCCL, TensorRT

Above the runtime sit the libraries that frameworks actually call.

• cuBLAS and the other math libraries ship inside the toolkit, with their own versions as shown above.
• cuDNN ships separately. Its support matrix lists, for cuDNN 9.27.0, one build for CUDA 13.x (toolkits 13.0 to 13.4.1) and one for CUDA 12.x (12.0 to 12.9). For the CUDA 13.x build the matrix lists a Linux driver of 615.71.09 or later; for the CUDA 12.x build it lists 525.60.13 or later on Linux and 527.41 or later on Windows. NVIDIA's compatibility guide adds that "each cuDNN version requires a certain version of cuBLAS", so cuDNN and cuBLAS have to be inventoried as a pair.
• NCCL, the collective communication library for multi GPU and multi node training, has its own release stream. The NCCL release notes, last updated 23 September 2026, cover releases up to 2.32.3.
• TensorRT and TensorRT-LLM are NVIDIA's inference compiler and LLM runtime. Both have had their own security bulletins in 2026.

None of these libraries has to come from the toolkit installer. They can arrive as distribution packages from NVIDIA's apt or dnf repositories, as layers in the CUDA images, or as Python wheels from PyPI. Often they arrive by two of those routes at once.

Framework wheels: the hidden second copy

Modern frameworks pull CUDA libraries in as ordinary Python dependencies. PyPI's metadata for PyTorch 2.14.1 (uploaded 30 September 2026) lists these requirements on Linux:

cuda-toolkit[cublas,cudart,cufft,cufile,cupti,curand,cusolver,cusparse,nvjitlink,nvrtc,nvtx]==13.0.3
cuda-bindings<14,>=13.0.3
nvidia-cudnn-cu13==9.24.0.43
nvidia-cusparselt-cu13==0.8.1
nvidia-nccl-cu13==2.30.7
nvidia-nvshmem-cu13==3.4.5
triton~=3.8.0

So pip install torch on Linux installs a CUDA 13.0.3 runtime, cuDNN 9.24.0.43 and NCCL 2.30.7 into site-packages, whatever CUDA toolkit the system has. Build that on a CUDA image that already contains cuDNN and the final image holds two cuDNN versions, one from the base layer and one from the wheel. An OS package scan reports the first, a Python dependency scan reports the second, and the process may load either.

The wheel names carry the major CUDA version (-cu12, -cu13). That suffix is part of the package name, so nvidia-cudnn-cu12 and nvidia-cudnn-cu13 are different packages to a resolver and to a vulnerability matcher. When you search your inventory for "cuDNN", search for both.

Container images

NVIDIA publishes CUDA images on Docker Hub, built from Dockerfiles that are open source under the 3-clause BSD licence. The Docker Hub page describes three flavours:

At the time of writing the page lists CUDA 13.4.3 tags such as 13.4.3-cudnn-runtime-ubuntu26.04. Two policies on that page matter for supply chain work. The first: "Supported tags are updated to the latest CUDA, cuDNN, and TensorRT versions. These tags are also periodically updated to fix CVE vulnerabilities." A tag is therefore a moving pointer; the same tag can resolve to different content on different days. The second is the CUDA Container Support Policy: tags "will be deleted Six Months after the last supported 'Tesla Recommended Driver' has gone end-of-life OR a newer update release has been made for the same CUDA version." If your Dockerfile says FROM nvidia/cuda: followed by an old tag, the build will eventually fail, and a rebuild in a hurry during an incident is the worst time to discover that.

The page also notes that the images "may include components licensed under open-source licenses such as GPL" and that NVIDIA archives the sources. That is useful for licence compliance: the OS layer of a CUDA image is an ordinary Linux distribution layer and should be scanned like one.

The NVIDIA Container Toolkit

A container cannot use a GPU by default. The NVIDIA Container Toolkit is the host software that gives it one. Its architecture overview lists four main packages:

├─ nvidia-container-toolkit (version)
│  ├─ libnvidia-container-tools (>= version)
│  └─ nvidia-container-toolkit-base (version)
│
├─ libnvidia-container-tools (version)
│  └─ libnvidia-container1 (>= version)
└─ libnvidia-container1 (version)

The components inside them are the NVIDIA Container Runtime (nvidia-container-runtime), which wraps runc and injects a prestart hook into the container spec; the NVIDIA Container Runtime Hook, which implements that runc prestart interface; and libnvidia-container with its CLI nvidia-container-cli, which does the actual work of configuring the container. The older nvidia-docker2 and standalone nvidia-container-runtime packages are described as deprecated, with their functionality merged into nvidia-container-toolkit.

The second mechanism is the Container Device Interface (CDI). From toolkit v1.12.0 the toolkit can generate CDI specifications, which NVIDIA describes as "an open specification for container runtimes that abstracts what access to a device, such as an NVIDIA GPU, means". From v1.18.0 a systemd service, nvidia-cdi-refresh, generates and refreshes the spec at /var/run/cdi/nvidia.yaml automatically. The documentation lists two situations it does not handle: the removal of NVIDIA GPU drivers and the reconfiguration of MIG devices. After either, you have to regenerate the spec by hand.

Which mode you run matters for security, as the advisory history below shows: some toolkit CVEs affected only the default hook based mode and not CDI, and at least one affected CDI mode too.

At the time of writing, the toolkit's GitHub releases show v1.20.1, published 19 September 2026, as the latest release, after v1.20.0 on 13 August 2026 and v1.19.1 on 21 May 2026.

Orchestration add ons and serving stacks

In Kubernetes the toolkit is usually installed and configured by the NVIDIA GPU Operator. Its documentation describes it as automating "the management of all NVIDIA software components needed to provision GPU", including the NVIDIA drivers, the Kubernetes device plugin for GPUs, the NVIDIA Container Toolkit, automatic node labelling and DCGM based monitoring. Each of those is a separately versioned component, and NVIDIA's bulletins name them individually: the July 2025 container toolkit bulletin listed fixed versions for the Container Toolkit, the GPU Operator, the Kubernetes Device Plugin and the MIG Manager, and a separate July 2026 bulletin covered the DCGM Exporter.

At the top of the stack sit model serving engines. NVIDIA's bulletin repository shows Triton Inference Server as the product with the most bulletins in 2026 so far (eight), with TensorRT and TensorRT-LLM together on five. These are application dependencies in the ordinary sense: they ship in images, they have versions, and their fixes arrive as new image tags or wheels. Our site covers several of these individually, for example the Triton DALI backend issue and a TensorRT advisory.

The AMD and Intel equivalents

The same layering exists on other vendors' GPUs, with different names.

For ROCm the compatibility matrix is explicit that the stack "depends on a coordinated stack of compatible firmware, driver, and user space components". The ROCm 10.1.0 matrix lists component versions such as AMD SMI 27.1.0, hipBLAS 3.7.0, hipBLASLt 1.4.1, MIOpen 3.6.1, rocBLAS 5.7.0 and RCCL 2.30.7, along with a list of supported amdgpu driver versions and supported kernels per distribution. AMD publishes its own security bulletins, including a February 2026 graphics driver bulletin listing 15 CVEs and a May 2026 bulletin (AMD-SB-6031) for the AMD Device Metrics Exporter in the ROCm ecosystem.

On the Intel side, the Compute Runtime README describes it as "an open source project providing compute API support (Level Zero, OpenCL) for Intel graphics hardware", lists OpenCL 3.0 and Level Zero 1.18 support across platforms from DG1 and Alchemist to Battlemage, Lunar Lake and Panther Lake, and states that it is distributed under the MIT License. Intel's oneAPI system requirement pages refused our scripted requests, so this post does not quote oneAPI toolkit versions.

The rest of this post uses NVIDIA for the worked examples because its documentation and bulletins are the most detailed, but every rule about splitting host and image inventories applies to ROCm and oneAPI unchanged.

Who owns each layer, and who patches it

Before getting into compatibility, it helps to be clear about ownership. Most GPU patching failures we hear about are not technical. A fix was available, but the team that owned the image assumed the platform team would handle it, or the reverse.

The table also shows where an SBOM has to come from. Anything in the top half is invisible to an image scan. Anything in the bottom half is invisible to a host package query. You need both, and you need a way to join them, which we come to below.

The compatibility rules, explained

GPU upgrades feel risky because the layers are coupled. The coupling is documented, though, and once you know the three rules you can plan upgrades without guessing.

Rule 1: backward compatibility, always

NVIDIA's guide puts it simply: "Backwards compatibility ensures that a newer NVIDIA driver can be used with an older CUDA Toolkit. This is implicit and most simple way of doing upgrades." An application built with CUDA 12.2 keeps working on a 615 driver. This is why upgrading the host driver first is almost always safe for the workloads already running on it, and why the driver is the layer you should keep closest to current.

Rule 2: minor version compatibility, within a major family

From CUDA 11 onward, an application built with any toolkit in a major family can run on an older driver from the same family, as long as the driver meets a minimum. The ranges, from the CUDA 13.4 Update 2 release notes:

The release notes also give the driver branch that corresponds to each 13.x release: R580 for CUDA 13.0, R590 for 13.1, R595 for 13.2, R610 for 13.3 and R615 for 13.4. They add that "CUDA 13.4 new features and newly enabled platforms require an R615 or later driver."

Minor version compatibility has three caveats, and the compatibility guide lists them:

1. Limited feature set. If a feature spans both toolkit and driver, the application can get cudaErrorCallRequiresNewerDriver on an older driver.
2. No PTX JIT. "Applications that compile device code to PTX will not work on older drivers." If your build ships only PTX for a GPU architecture, or relies on JIT compilation at runtime, you need a newer driver.
3. You must build for the target architecture, for example nvcc -arch=sm_xx, so that ready made machine code (SASS) is embedded in the binary.

Before CUDA 11 there was no minor version compatibility: every 10.x minor release had its own minimum driver (410.48 for CUDA 10.0, 418.39 for 10.1, 440.33 for 10.2 on Linux). If you still run anything built with CUDA 10, the driver coupling is tighter than you might expect.

Rule 3: forward compatibility, across major families, on data center GPUs

If you want to run, say, a CUDA 13.4 application on a node that must stay on a 570 driver, minor version compatibility will not help, because 570 is below the CUDA 13.x minimum of 580. NVIDIA's answer is the forward compatibility package, introduced in CUDA 10.0. The guide is clear that it "is applicable only for systems with" NVIDIA data center GPUs, select NGC Server Ready SKUs of RTX cards, and Jetson boards. It does not apply to ordinary consumer GPUs.

The package is named after the highest toolkit it supports, cuda-compat- followed by the major and minor version, and it is built from a newer driver's user mode libraries. NVIDIA's example: cuda-compat-13-4 is built from driver 615 libraries and installs files such as libcuda.so.615.31, libnvidia-ptxjitcompiler.so.615.31 and libnvidia-nvvm.so.615.31 into /usr/local/cuda-13.4/compat/. The package "only provides the libraries, and does not configure the system to find such libraries", so you point LD_LIBRARY_PATH at that directory. The support matrix shows cuda-compat-13-4 as compatible with drivers from the 535 (CUDA 12.2) branch through the 610 (CUDA 13.3) branch.

From a supply chain angle, the compat package is interesting for one reason: it is a copy of user mode driver libraries at a different version from the host driver. If it is installed inside an image, your image now carries driver code. If an NVIDIA driver bulletin lists a user mode layer CVE (the October 2024 bulletin, for example, listed several "in the user mode layer" on Windows), a Linux compat package built from an affected branch is something you would need to check. Record it in the SBOM as its own component with its own version.

Worked examples

To make the rules concrete, here is a small Python function that encodes the tables above. It was run on a Mac against the versions shown; it does not query a GPU.

# Minimum driver branch for CUDA minor version compatibility, from the
# CUDA Toolkit 13.4 Update 2 release notes, Table 3.
MINOR_COMPAT = {11: (450, 525), 12: (525, 580), 13: (580, None)}
# Driver branch for each toolkit release: 13.x from Table 2, 12.x and 11.8
# from the Linux x86_64 column of Table 4 of the same release notes.
BRANCH = {(11, 8): 520, (12, 0): 525, (12, 1): 530, (12, 2): 535, (12, 3): 545,
          (12, 4): 550, (12, 5): 555, (12, 6): 560, (12, 8): 570, (12, 9): 575,
          (13, 0): 580, (13, 1): 590, (13, 2): 595, (13, 3): 610, (13, 4): 615}

def check(driver: str, cuda: str, uses_ptx_jit: bool = False) -> str:
    drv_major = int(driver.split(".")[0])
    major, minor = (int(x) for x in cuda.split(".")[:2])
    lo, hi = MINOR_COMPAT[major]
    if (major, minor) in BRANCH and drv_major >= BRANCH[(major, minor)]:
        return "OK: driver branch is at or above the toolkit's own branch"
    if hi is not None and drv_major >= hi:
        return "OK: backward compatibility (newer driver, older toolkit)"
    if drv_major >= lo:
        if uses_ptx_jit:
            return "FAIL: minor version compatibility does not cover PTX JIT; upgrade driver or use cuda-compat"
        return "OK with limits: minor version compatibility (build SASS with -arch=sm_XX)"
    return "FAIL: driver too old for this major release; upgrade driver or use the forward compatibility package (data center GPUs only)"

Output for six combinations:

driver 560.35.03   CUDA 12.8  ptx_jit=False -> OK with limits: minor version compatibility (build SASS with -arch=sm_XX)
driver 570.124.06  CUDA 12.8  ptx_jit=False -> OK: driver branch is at or above the toolkit's own branch
driver 570.124.06  CUDA 13.4  ptx_jit=False -> FAIL: driver too old for this major release; upgrade driver or use the forward compatibility package (data center GPUs only)
driver 580.65.06   CUDA 13.4  ptx_jit=False -> OK with limits: minor version compatibility (build SASS with -arch=sm_XX)
driver 580.65.06   CUDA 13.4  ptx_jit=True  -> FAIL: minor version compatibility does not cover PTX JIT; upgrade driver or use cuda-compat
driver 615.71.09   CUDA 13.4  ptx_jit=False -> OK: driver branch is at or above the toolkit's own branch

The fourth line matches NVIDIA's own example in the compatibility guide, where a Tesla T4 on driver 580.65.06 runs a deviceQuery built with CUDA 13.4 and reports "CUDA Driver Version / Runtime Version 13.0 / 13.4". The third line is the case that bites teams who upgrade a framework wheel to a new CUDA major version without touching the node image: the application fails at initialisation, and the guide shows the error as cudaGetDeviceCount returned 3 followed by "initialization error".

Libraries add their own constraints on top. The cuDNN 9.27.0 support matrix lists a Linux driver of 615.71.09 or later for its CUDA 13.x build, which is stricter than the general CUDA 13.x minimum of 580. When a library's support matrix and the general CUDA rule disagree, follow the library's matrix.

Compatible is not the same as patched

The compatibility rules tell you what will run. They say nothing about what is safe. That needs a second check against the security bulletins, and the two answers are often different.

NVIDIA's GPU display driver bulletin for September 2026 (initial release 30 September 2026, revised 13 October 2026 to correct one CVSS score) lists fixed Linux versions per driver branch. For the data center ("Tesla") products on Linux they are:

Now run the same driver versions from the compatibility example through a patch check. Again, this was run on a Mac with fixed inputs:

FIXED_SEP_2026 = {580: "580.178.04", 595: "595.91.07", 610: "610.57.04", 615: "615.71.09"}

def vt(v): return tuple(int(x) for x in v.split("."))

def patched(driver: str) -> str:
    branch = vt(driver)[0]
    fixed = FIXED_SEP_2026.get(branch)
    if fixed is None:
        return f"{driver}: branch R{branch} not listed in this bulletin; move to a listed branch"
    return f"{driver}: {'patched' if vt(driver) >= vt(fixed) else 'NOT patched'} (fixed in {fixed})"

580.65.06: NOT patched (fixed in 580.178.04)
580.178.04: patched (fixed in 580.178.04)
595.71.05: NOT patched (fixed in 595.91.07)
615.71.09: patched (fixed in 615.71.09)
570.124.06: branch R570 not listed in this bulletin; move to a listed branch

Driver 580.65.06 is perfectly compatible with CUDA 13.4 applications under minor version compatibility, and it is also below the fixed version for its branch. Driver 570.124.06 runs every CUDA 12.x workload and is not a branch the September 2026 bulletin lists a fix for at all. Two practical rules fall out of this:

• Compare versions as tuples, never as strings. As text, "580.65.06" sorts after "580.178.04", because the character 6 comes after 1. A string comparison would call the vulnerable driver newer than the fixed one.
• Track the branch, not just the version. NVIDIA fixes vulnerabilities per supported branch. If your fleet sits on a branch that has dropped out of the bulletin tables, there may be no fixed version on it, and the fix is a branch move with its own compatibility check.

Real advisories in the GPU stack

The container toolkit: six bulletins in under two years

The NVIDIA Container Toolkit sits on the boundary between a container and its host, which is exactly where you do not want a vulnerability. Its bulletin history, from NVIDIA's own security bulletins:

A few details from those bulletins are worth reading closely.

CVE-2024-0132 is the one most people remember. NVIDIA describes it as a time of check, time of use (TOCTOU) vulnerability "when used with default configuration where a specifically crafted container image may gain access to the host file system", and states that it "does not impact use cases where CDI is used". NVIDIA scored it 9.0. NIST's National Vulnerability Database scored the same CVE 8.3, with a higher attack complexity and no privileges required. NVD's published date is 26 September 2024, one day after NVIDIA's bulletin. Score disagreements like this are normal; record which score your policy uses. Our CVE-2024-0132 explainer covers it in more depth.

CVE-2025-23359, 139 days later, is another TOCTOU issue "when used with default configuration" that could give a crafted image access to the host file system. It affected versions up to 1.17.3, so a team that patched to 1.16.2 in September 2024 and stopped there was exposed again. Patching to "the version in the last advisory" is not a policy. Patching to current, on a schedule, is.

CVE-2025-23266 broke the comfortable rule that CDI mode was the safe mode. NVIDIA describes it as a vulnerability "in some hooks used to initialize the container", and the affected versions note says "CDI mode only for versions prior to 1.17.5". The same bulletin lists fixed versions for the Kubernetes device plugin (only when its device list strategy is cdi-annotations or cdi-cri) and for the MIG manager (only when CDI is enabled). The bulletin was revised on 12 August 2025 to update affected products and mitigation steps, which is a reminder to re-read bulletins after their first release.

CVE-2024-0136 and CVE-2024-0137 were present "only when the NVIDIA Container Toolkit is configured in a nondefault way". Your configuration is part of your exposure, which is an argument for recording the toolkit mode (legacy hook or CDI, and any non default options) next to the version.

None of the bulletins we read list these issues as exploited in the wild, and CVE-2024-0132's NVD record carries CISA's SSVC assessment of "exploitation: none" as of its date. That can change; check CISA's Known Exploited Vulnerabilities catalogue rather than this post.

The worked lesson: take a host that has nvidia-container-toolkit 1.17.3 installed. Against the table, it is affected by CVE-2025-23359 (fixed in 1.17.4), by CVE-2025-23266 and CVE-2025-23267 if it uses CDI mode (fixed in 1.17.8), and by CVE-2026-24260 (fixed in 1.19.1). Moving to the current release, v1.20.1 at the time of writing, clears all of them in one step. Moving to "the fix for the CVE in today's ticket" would leave some of them open.

The display driver: big quarterly bulletins

NVIDIA's GPU display driver bulletins cover the kernel mode driver, the user mode driver and the vGPU software, for Windows and Linux, across every supported branch. They are large. The September 2026 bulletin contains 114 distinct CVE identifiers, and its affected versions table runs to dozens of rows, one per product line, platform and branch: GeForce, RTX and Quadro, data center (Tesla), vGPU guest drivers and the Virtual GPU Manager. Several of the CVEs are in "the open-source kernel module" (for example in DMA mapping and the DMA-BUF import path) and many are in "the kernel mode layer".

For a platform team the operational detail that matters is the per branch fixed version, which is why the patch check above keys on the branch. For vGPU deployments there are two components to patch: the guest driver inside each VM and the Virtual GPU Manager on the hypervisor, with different version numbers. The September 2026 bulletin lists, for example, vGPU 19.6 with guest driver 580.178.04 and Virtual GPU Manager 580.178.05 on Linux hypervisors.

The CUDA toolkit, frameworks and serving stacks

CUDA toolkit bulletins tend to cover developer tools that ship with the toolkit: cuobjdump, nvdisasm, Nsight Systems and similar. The January 2026 bulletin fixed four CVEs in Nsight components, all affecting "All versions prior to CUDA Toolkit 13.1". These matter on developer machines and in devel build images rather than in production runtime images, which is one more reason to keep build and runtime images separate. An earlier example on our site is the cuobjdump issue in the CUDA toolkit.

Higher up, the bulletin volume is in the ML software. Here is how NVIDIA's public bulletin repository on GitHub breaks down by product and year. The counts are bulletins, not CVEs, and NVIDIA's README notes that coverage of older years is still expanding, so treat the early years as a floor:

We counted those from the repository's directory listing on 25 October 2026 by reading the title of every bulletin file. The trend is clear even with the caveat: the share of bulletins aimed at AI frameworks, training tools and serving engines has grown sharply, and those are exactly the components that live in images and change most often.

We did not find a bulletin titled for NCCL or cuDNN in the repository. That is not evidence that they have no vulnerabilities; it means that, as of this writing, NVIDIA has not published a standalone bulletin for them there. They still belong in the SBOM, because a future advisory has to be matched against something.

Where to get the advisories in machine readable form

NVIDIA's product security page says that "Starting October 1, 2026, NVIDIA PSIRT will only publish security bulletins on GitHub in Markdown, CSAF, and CVE formats", with the product security website continuing to carry them "in parallel". The repository is organised by year and bulletin ID; each bulletin folder contains a Markdown file, a CSAF JSON file, CVE record files and SHA-256 checksums for each. For CVE-2024-0132, the folder 2024/5582 holds 5582.md, 5582.json, CVE-2024-0132.json and the matching .sha256 files.

That is a real improvement for automation. CSAF documents state products, versions and remediation in a structured form that a matcher can consume, instead of a human copying fixed versions out of an HTML table. AMD publishes bulletins on its product security page, with IDs such as AMD-SB-6031. Subscribe to both vendors' notifications, and wire the CSAF feed into whatever system holds your inventory.

Why GPU components hide from your usual tools

If you run an ordinary SBOM generator against a GPU host and a GPU image, you will get a lot of output and still miss important parts of the stack. These are the blind spots and why they exist.

Runfile driver installs. The .run installer puts files on disk without registering a package. A package manager query on that host returns nothing for the driver.

Host libraries injected at start. libcuda.so is not in the image; the container toolkit makes the host's copy available at start. An image SBOM cannot list it, and should not, but your risk picture must join the two, because a vulnerable host driver affects every GPU container on that host.

Libraries inside wheels. nvidia-cudnn-cu13, nvidia-nccl-cu13 and the cuda-toolkit wheel extras put shared objects under site-packages. OS level scanners do not see them; Python dependency scanners see the package names but may not connect them to CUDA advisories that name "cuDNN" rather than the wheel.

Duplicate copies. A CUDA base image plus a framework wheel can mean two cuDNN and two NCCL versions in one image. List both, with file paths where your format allows.

Statically linked CUDA code. CUDA libraries can be linked statically (the cuDNN support matrix even has a column for static linking support). A statically linked binary carries library code with no separate file to find. The only reliable record is the build: capture the toolkit and library versions used at link time in the SBOM for the binary.

Firmware. No package manager knows the VBIOS or GSP firmware version. Only the driver tools report them.

Configuration. Whether the toolkit runs in legacy hook mode or CDI mode, and with which options, decided exposure to several of the bulletins above. That is configuration, not a package, and SBOM formats are not designed to hold it. Record it as a property next to the toolkit component.

What to put in your SBOM

Start from the 2026 minimum elements

On 29 July 2026 CISA, the NSA, the FBI and international partners published the 2026 Minimum Elements for a Software Bill of Materials, which "updates and replaces" the 2021 NTIA minimum elements. Its Appendix A lists the data fields: SBOM Author, SBOM Author Signature, SBOM Timestamp, SBOM Tool Name and Tool Version, SBOM Version, SBOM Data Format Name and Version, SBOM Generation Context, and for each component the Component Name, Version, Producer, Identifiers, License, Hash Algorithm and Hash Value, plus Component Dependency Relationships. The guidance also notes that some software types, naming artificial intelligence as one, "may require additional elements". Our post on the 2026 minimum elements update goes field by field; here is how they map onto a GPU stack.

Use the right component types

CycloneDX already has the vocabulary for this. The CycloneDX 1.7 schema defines component types including device-driver ("A special type of software that operates or controls a particular type of device"), firmware ("A special type of software that provides low-level control over a device's hardware"), platform, operating-system, container, library, application and machine-learning-model. Use them:

SPDX 3.0 can express the same inventory with its own element types and relationships; pick the format your consumers ask for. CycloneDX released version 1.7.2 of its specification on 17 September 2026, and SPDX's latest final release is 3.0.1, with 3.1 at release candidate stage on GitHub.

Use package URLs that match how the component was installed

The purl specification has types for every route GPU software takes: deb and rpm for distribution and NVIDIA repository packages, pypi for wheels, conda for conda packages, oci and docker for images, and generic for things with no package manager such as a runfile driver. Two practical points:

• For packages installed from NVIDIA's own apt or dnf repository, add the repository as a repository_url qualifier, so a matcher does not confuse NVIDIA's package with a distribution rebuild of the same name.
• For wheels, the CUDA major version is part of the package name (pkg:pypi/nvidia-cudnn-cu13@9.24.0.43). Matching rules that look for "cudnn" need to include those names.

One SBOM per layer, linked

The cleanest structure we have found is three SBOMs with explicit links between them:

 Host SBOM (generation context: operations, one per node image or node)
   device-driver  nvidia-driver 580.65.06        <- from nvidia-smi
   firmware       nvidia-gsp-firmware 580.65.06
   firmware       nvidia-vbios <version>
   application    nvidia-container-toolkit 1.17.3-1  + property: mode=cdi|legacy
   application    gpu-operator, device-plugin, dcgm-exporter (cluster level)
          ^
          | "runs on" link, recorded by the scheduler or by your deploy metadata
          |
 Image SBOM (generation context: build, one per image digest)
   container      registry/app@sha256:...
   library        cuda-cudart 13.4.92, cublas 13.8.0.4      <- OS packages
   library        libcudnn9 (from base image)               <- OS packages
   library        nvidia-cudnn-cu13 9.24.0.43              <- wheel
   library        nvidia-nccl-cu13 2.30.7                   <- wheel
   application    tritonserver or your serving app
          ^
          | "contains" link
          |
 Application SBOM (from the source repository, per build)
   library        torch 2.14.1, transformers, fastapi, ...
   machine-learning-model  weights and their provenance (optional, see ML-BOM)

The host SBOM changes when the node image or driver changes. The image SBOM changes with every build and is keyed by image digest. The application SBOM comes from the repository. When a driver bulletin lands, you search host SBOMs. When a cuDNN or Triton advisory lands, you search image SBOMs. When you need to know which running services are affected by a host driver issue, you follow the "runs on" link from the image to the host, which you can usually derive from your scheduler's records.

A host SBOM fragment, generated

There is no standard tool that turns nvidia-smi output into an SBOM, so here is a short script that does it for the driver and firmware and adds the GPU related packages from dpkg-query. We ran it on a Mac against the sample input shown; the driver version in the sample is the one NVIDIA uses in its own compatibility guide, the VBIOS string is a placeholder, and the toolkit package version is deliberately an old, vulnerable one for the worked example. On a real host, feed it the output of nvidia-smi -q and dpkg-query -W -f='${Package} ${Version}\n'.

#!/usr/bin/env python3
"""Build a small CycloneDX 1.6 JSON SBOM for the GPU layer of a Linux host."""
import json, re, sys, uuid, datetime

GPU_PKG = re.compile(r"^(nvidia-|libnvidia-|cuda-|libcudnn|libnccl|tensorrt|libnvinfer|datacenter-gpu-manager)")

def smi_fields(text):
    want = {"Driver Version": "driver", "CUDA Version": "cuda_umd",
            "VBIOS Version": "vbios", "GSP Firmware Version": "gsp"}
    out = {}
    for line in text.splitlines():
        if ":" in line:
            k, v = (s.strip() for s in line.split(":", 1))
            if k in want and want[k] not in out:
                out[want[k]] = v
    return out

def build(smi_text, pkg_text):
    f = smi_fields(smi_text)
    comps = [{"type": "device-driver", "bom-ref": "nvidia-driver", "name": "nvidia-driver",
              "version": f.get("driver", "unknown"), "supplier": {"name": "NVIDIA"},
              "properties": [{"name": "cuda:umd-version", "value": f.get("cuda_umd", "unknown")}]}]
    for key, name in (("vbios", "nvidia-vbios"), ("gsp", "nvidia-gsp-firmware")):
        if key in f:
            comps.append({"type": "firmware", "bom-ref": name, "name": name,
                          "version": f[key], "supplier": {"name": "NVIDIA"}})
    for line in pkg_text.splitlines():
        parts = line.split()
        if len(parts) == 2 and GPU_PKG.match(parts[0]):
            n, v = parts
            comps.append({"type": "library" if n.startswith("lib") else "application",
                          "bom-ref": f"deb:{n}", "name": n, "version": v,
                          "purl": f"pkg:deb/ubuntu/{n}@{v}?arch=amd64"})
    return {"bomFormat": "CycloneDX", "specVersion": "1.6",
            "serialNumber": f"urn:uuid:{uuid.uuid4()}", "version": 1,
            "metadata": {"timestamp": datetime.datetime.now(datetime.timezone.utc).isoformat(timespec="seconds"),
                         "lifecycles": [{"phase": "operations"}],
                         "tools": {"components": [{"type": "application", "name": "gpu_host_sbom.py", "version": "0.1"}]}},
            "components": comps}

if __name__ == "__main__":
    smi, pkgs = (open(p).read() for p in sys.argv[1:3])
    print(json.dumps(build(smi, pkgs), indent=2))

With the sample input it produced eight components: one device-driver (580.65.06), two firmware entries (VBIOS and GSP), the four container toolkit packages at 1.17.3-1, and a cuda-compat-13-4 package at 615.31-1ubuntu1. The openssl line in the sample package list was correctly ignored, since the OS packages belong in your general host SBOM. Before you use it, change the purl namespace and add a repository_url qualifier to reflect where your packages actually came from, and add hashes; the script leaves both out to stay short.

Step by step: inventory, pin and patch a GPU stack

The commands in this section follow NVIDIA's documentation. We did not run the ones that need a GPU. Run them on one node first.

Step 1: inventory the host

# Driver, user mode CUDA version, VBIOS and GSP firmware
nvidia-smi -q | grep -E "Driver Version|CUDA Version|VBIOS Version|GSP Firmware Version|Inforom Version"

# Kernel module version as the kernel sees it
cat /proc/driver/nvidia/version
modinfo nvidia | grep -E "^version|^license"

# Packaged GPU components (Debian and Ubuntu)
dpkg-query -W -f='${Package} ${Version}\n' | grep -E "^(nvidia-|libnvidia-|cuda-|libcudnn|libnccl)"

# Packaged GPU components (RHEL family)
rpm -qa --qf '%{NAME} %{VERSION}-%{RELEASE}\n' | grep -E "^(nvidia-|libnvidia-|cuda-|libcudnn|libnccl)"

# Container toolkit version and mode
nvidia-ctk --version
nvidia-ctk cdi list            # lists CDI devices if CDI specs are present
ls -l /var/run/cdi/ /etc/cdi/ 2>/dev/null

# Forward compatibility packages, if any
ls -d /usr/local/cuda-*/compat 2>/dev/null

If nvidia-smi reports a driver but dpkg-query or rpm shows no driver package, the driver came from a runfile or a driver container. Record that, because the upgrade procedure differs.

In Kubernetes, also record the GPU Operator chart version and the image tags (by digest) of the device plugin, the toolkit container and the DCGM exporter. These are cluster level components and belong in the cluster's SBOM.

Step 2: inventory the image

# What the image says about itself
docker run --rm --gpus all IMAGE@sha256:DIGEST bash -lc '
  ls /usr/local/cuda*/version.json 2>/dev/null && cat /usr/local/cuda*/version.json | head -50
  ldconfig -p | grep -E "libcudart|libcublas|libcudnn|libnccl" 
  python3 -m pip list 2>/dev/null | grep -i -E "^(torch|nvidia-|cuda-|triton|tensorrt)"
'

# What the framework actually loads at runtime
docker run --rm --gpus all IMAGE@sha256:DIGEST python3 -c '
import torch
print("torch", torch.__version__)
print("built for CUDA", torch.version.cuda)
print("cuDNN", torch.backends.cudnn.version())
print("NCCL", torch.cuda.nccl.version())
'

The second block is the most useful one. It reports the cuDNN and NCCL versions the framework actually uses, which resolves the "two copies" question from earlier. If those versions differ from the OS packages in the image, the OS copies are dead weight that still shows up in scans; consider removing them or moving to a slimmer base.

For the full image SBOM, generate it from the image itself in CI, keyed by digest. With the Safeguard CLI that is safeguard scan --image <name> (or point it at an SBOM you already produce with safeguard scan --sbom <file>), with results written as JSON under ./output.

Step 3: pin every layer

Pinning is what turns an inventory into something you can reproduce and patch on purpose.

# Pin the base image by digest, not by tag. NVIDIA refreshes supported
# tags for CVE fixes and deletes old tags on a published schedule.
FROM nvidia/cuda:13.4.3-cudnn-runtime-ubuntu26.04@sha256:<digest-you-verified>

# Pin Python dependencies, including the CUDA wheels, with hashes
COPY requirements.lock .
RUN python3 -m pip install --require-hashes -r requirements.lock

# requirements.lock (excerpt, generated with hashes by your lock tool)
torch==2.14.1 --hash=sha256:...
nvidia-cudnn-cu13==9.24.0.43 --hash=sha256:...
nvidia-nccl-cu13==2.30.7 --hash=sha256:...

On the host, NVIDIA's install guide shows how to pin the container toolkit packages to one version with apt:

export NVIDIA_CONTAINER_TOOLKIT_VERSION=1.20.1-1
sudo apt-get install -y \
  nvidia-container-toolkit=${NVIDIA_CONTAINER_TOOLKIT_VERSION} \
  nvidia-container-toolkit-base=${NVIDIA_CONTAINER_TOOLKIT_VERSION} \
  libnvidia-container-tools=${NVIDIA_CONTAINER_TOOLKIT_VERSION} \
  libnvidia-container1=${NVIDIA_CONTAINER_TOOLKIT_VERSION}

All four packages must move together; the dependency tree requires matching versions. For the driver, pin the branch in your node image build and record the exact version in the node image's SBOM.

Pinning by digest has one cost: you no longer get NVIDIA's CVE refreshes of the tag automatically. That is the right trade, because you want to choose when content changes, but it means you need a job that checks for a newer digest behind your tag, rebuilds, tests and opens a change. That is the same pattern as dependency update automation in any ecosystem.

Step 4: patch in the right order

Because of backward compatibility, the safe order for a full stack update is bottom up:

1. Driver and firmware first. Upgrade the host driver to the fixed version on your branch (or move branch). Drain the node, upgrade, reboot or reload the modules, then confirm with nvidia-smi -q. Existing workloads keep running on the newer driver.
2. Container toolkit and cluster add ons next. Upgrade all four toolkit packages together, then rerun nvidia-ctk runtime configure if your runtime config changed, and restart the container runtime as the install guide shows (for example sudo systemctl restart containerd). If you use CDI, regenerate the spec after any driver removal or MIG change, because nvidia-cdi-refresh does not handle those.
3. Images last. Rebuild images on a new base digest and updated wheels, run your GPU tests, and redeploy.

Going top down (new framework wheels first) is the path that produces the CUDA 13.4 on a 570 driver failure from the worked example.

Step 5: verify and keep watching

After each change, regenerate the affected SBOM and diff it against the previous one. Check the result against the latest bulletins: NVIDIA's GitHub CSAF files, AMD's product security bulletins, and your distribution's advisories for the OS layers. Then schedule the next check. Bulletins arrive monthly for the ML software and roughly quarterly for the display driver, so a monthly review of the GPU layers is a reasonable minimum.

Common mistakes

• Reading nvidia-smi's "CUDA Version" as the installed toolkit. It is the user mode driver's supported CUDA version. The toolkit and the libraries in your images can be older or, with compatibility rules, newer.
• Recording "CUDA 13.4" as a component. It is a family. cuBLAS, cuFFT, cuSOLVER and the runtime each have their own version, and advisories use those.
• Trusting the package database on a runfile host. A runfile driver is invisible to dpkg and rpm. Ask the driver.
• Building images from tags. CUDA image tags are refreshed and deleted. Pin by digest and rebuild on purpose.
• Patching to the fix in today's ticket. The container toolkit history shows repeated issues in the same area months apart. Patch to current.
• Assuming CDI mode is immune. It avoided CVE-2024-0132 but not CVE-2025-23266.
• Comparing driver versions as strings. "580.65.06" is lower than "580.178.04". Compare numerically, part by part.
• Upgrading wheels before the driver. A CUDA 13 wheel on a 570 driver fails at initialisation. Go bottom up.
• Forgetting the compat package. cuda-compat puts driver user mode libraries at a different version on the system, often inside images. Inventory it as its own component.
• Leaving vGPU half patched. Guest drivers and the Virtual GPU Manager are separate components with separate fixed versions.
• Dropping firmware from the inventory. VBIOS and GSP versions only come from driver tools. Without them, a firmware related advisory cannot be matched.

FAQ

What is the difference between the NVIDIA driver and the CUDA toolkit?

The driver is host software: kernel modules plus the user mode CUDA driver, libcuda.so, and GSP firmware. The toolkit is developer and runtime software: the compiler, the CUDA runtime library, math libraries and tools. Applications ship the runtime and libraries they need; the driver must be installed on whatever machine runs them. Since CUDA 13.4 on Linux, the toolkit no longer bundles a driver at all.

Which CUDA version does my driver support?

nvidia-smi prints it as "CUDA Version", described in NVIDIA's documentation as the user mode driver version. A driver supports that CUDA version and, through backward compatibility, everything older. Under minor version compatibility it can also run newer toolkits from the same major family with some limits, for example CUDA 13.4 applications on a 580 driver if they do not need PTX JIT.

Can I run a newer CUDA toolkit on an older driver?

Within a major family, yes, if the driver meets the family minimum (580 for CUDA 13.x, 525 for 12.x, 450 for 11.x) and you build for the target architecture without relying on PTX JIT. Across major families, only on data center GPUs, select NGC Server Ready RTX SKUs and Jetson, using the cuda-compat forward compatibility package.

Does the NVIDIA driver belong in my container image SBOM?

No. The user mode driver is supplied by the host at container start, so it belongs in the host SBOM. Your image SBOM should list the CUDA runtime, libraries, wheels and applications in the image. Link the two so you can see which running services depend on a host with a given driver.

How do I find which cuDNN and NCCL versions my PyTorch image really uses?

Ask the framework: torch.backends.cudnn.version() and torch.cuda.nccl.version() report what it loaded. Compare that with the OS packages from dpkg-query and the wheels from pip list. Framework wheels pin their own copies (PyTorch 2.14.1 pins nvidia-cudnn-cu13==9.24.0.43 and nvidia-nccl-cu13==2.30.7), so an image can carry two versions of each.

Was CVE-2024-0132 fixed by switching to CDI?

NVIDIA's bulletin says CVE-2024-0132 "does not impact use cases where CDI is used", and the fix was toolkit 1.16.2 and GPU Operator 24.6.2. But CVE-2025-23266 later affected CDI mode in versions before 1.17.5, and all modes up to 1.17.7. The durable fix is to run the current toolkit release, whichever mode you use.

Where do I get NVIDIA security bulletins in machine readable form?

From NVIDIA's product security repository on GitHub. Since 1 October 2026 NVIDIA publishes bulletins there in Markdown, CSAF and CVE record formats, with SHA-256 checksums, in parallel with its product security website. AMD lists its bulletins on its product security page.

How often should we patch GPU drivers?

At least as often as the display driver bulletins arrive, which in the bulletin repository has been three or four times a year since 2022, plus out of band for critical issues. Container toolkit and serving stack bulletins arrive more often. A monthly review of the host and image GPU layers against new bulletins is a practical baseline.

Do ROCm and oneAPI need the same treatment?

Yes. ROCm's compatibility matrix describes a coordinated stack of firmware, kernel driver and user space components, and lists versions for each, such as RCCL 2.30.7 and MIOpen 3.6.1 in ROCm 10.1.0. AMD publishes security bulletins for its graphics drivers and ROCm components. Intel's Compute Runtime for Level Zero and OpenCL is open source and versioned. Split host and image inventories the same way.

Where Safeguard fits, and where it does not

Safeguard does not design, program or tune GPUs, and it does not make GPU code faster. It works on the software side of this stack, the same way it works on any other software supply chain.

What applies here, with each capability's maturity:

• SBOM generation and analysis (GA). Safeguard generates and analyses SBOMs in SPDX and CycloneDX, from source repositories, container images and uploaded SBOM files. That covers the image layer and the application layer above. For the host layer, you can generate an SBOM such as the one from the script above and upload it, so driver, firmware and toolkit entries sit next to everything else.
• Software composition analysis across containers (GA). Safeguard's SCA inventories transitive dependencies across language ecosystems, including PyPI, and container layers. That is how the duplicate cuDNN and NCCL copies in a framework image show up as two components rather than one.
• Vulnerability management (GA). Findings are matched against vulnerability databases and vendor bulletins, enriched with EPSS and CISA KEV, and you can ask the platform which projects contain a given component and version. CSAF and VEX upload are also GA, so a vendor's CSAF advisory or your own VEX statement about a non default toolkit mode can sit alongside the findings.
• Reachability analysis (GA) ranks vulnerable code paths your application actually calls. It is most useful in the application and framework layers; it does not reason about kernel drivers.
• Policies and gates (GA) can block a build or release when a project carries a component below a version you set, or one with a known exploited vulnerability, for example an outdated framework wheel in an inference image.
• Autonomous remediation (GA, off by default) opens tested pull requests that bump dependencies in repositories, such as a pinned framework wheel in a Python lock file. It ships with every category set to Manual.
• AI-BOM discovery is Early Access (EA). It inventories AI assets and produces an ML-BOM. If you want the models served by this stack in the same inventory, ask about scope before you plan around it.

Where Safeguard does not help: it does not install or upgrade GPU drivers, flash VBIOS, manage Kubernetes node pools or reboot hosts. It does not scan firmware images. The fix for a driver CVE still comes from NVIDIA, AMD or Intel and has to be rolled out by your platform team. What Safeguard adds is the inventory and matching that tell you where that fix is needed, and the automation for the parts of the stack that live in your repositories and images.

A GPU fleet is rebuilt and redeployed constantly, and every rebuild pulls CUDA images, framework wheels, serving engines and their own open source dependencies again. The driver fix has to come from the GPU vendor, but most of what sits in your images and repositories is open source software you build and ship yourself, and that is where these steps apply.

More in this series

Other posts in our GPUs, CPUs, QPUs and lithography series:

• AMD GPU Architecture Explained: GCN, RDNA, CDNA, Infinity Fabric and HIP
• CPU vs GPU vs QPU Explained: How Each Works and Why They Coexist
• CUDA Programming Model Explained: Grids, Blocks, Warps and Memory
• GPU Memory and Interconnects: HBM, GDDR, NVLink, PCIe, CXL
• CUDA vs HIP vs SYCL vs OpenCL vs Metal vs Vulkan vs Triton Compared
• How a QPU Works: Qubit Chips, Fridges, Control Pulses and Readout
• How CPUs Work: Pipelines, Out-of-Order, Caches, Branches, SIMD
• How EUV Lithography Machines Work: Tin Plasma, Mirrors and High-NA
• How GPUs Work: Graphics Pipeline, Shader Cores, Ray Tracing and Compute
• Inside an NVIDIA SM: Warp Schedulers, Registers, Tensor Cores
• Intel GPU Architecture: Xe Cores, XMX, Ponte Vecchio and oneAPI SYCL
• NVIDIA GPU Architecture History: From Tesla and Fermi to Blackwell
• NVIDIA vs AMD vs Intel vs Apple GPUs: Architectures Compared
• Optimizing a CUDA Matrix Multiply Step by Step, From Naive to Tensor Cores
• Photolithography Explained: Contact Printing to DUV Immersion and Overlay
• Tile-Based GPUs Explained: Apple, Mali, Adreno, PowerVR and TBDR

Fix it without waiting for upstream

Whatever the vulnerability, most of the risk sits in two gaps. The first is the time between a fix being published and that fix actually running in your production builds. The second, and worse, is the time before any fix exists at all, while a maintainer works out a patch and a release schedule. Waiting through both gaps is a habit, not a requirement. Safeguard gives teams three ways to close them.

1. Autonomous remediation: pull requests, not tickets

When a fixed version exists, Safeguard's Griffin models open a tested, policy gated pull request that moves the affected dependency to the safe version. You can run it on one repository or in bulk across every project that carries the vulnerable version. Remediation covers eight categories, including dependencies, licenses, security posture and findings. Reachability analysis ranks the vulnerable code paths your application actually calls first, so the first pull requests you merge are the ones that matter most. Autonomous remediation ships switched off, with every category set to Manual, so your team decides how much runs on its own and what still needs a human approval.

2. Gold Open Source: the safe version in seconds, free

gold.safeguard.sh is free and needs no login. Search any CVE or package to see affected and fixed versions, CVSS, EPSS, CISA KEV status and a recommended safe version, or use the "Am I affected?" checker. The same data is available through a free read API, README status badges and RSS threat feeds. The safeguard-sh/gold-scan-action GitHub Action can fail a build on a known exploited vulnerability, on any known vulnerability, or never, so a version you have just removed cannot quietly come back in next week's merge.

3. Gold Registry: start from components that are already fixed

registry.safeguard.sh provides hardened, drop in replacements for upstream packages and container images. Every Gold artifact is held continuously to zero exploitable critical or high CVEs, zero malware and a completed license check, with SLSA L3 or higher provenance, a Sigstore signature and a public evidence page you can hand to an auditor. An artifact loses its Gold badge if it regresses. Teams can point their own package directory at the registry so hardened equivalents resolve without changing how developers install anything.

When upstream has no fix yet, or will not ship one, Safeguard first works to get the fix accepted upstream. If the issue is severe or upstream will not move, Safeguard builds a fork, fixes the vulnerability there and publishes a hardened Gold build, so you are not stuck on a maintainer's release schedule. You can also request a hardened, compatibility tested build of a specific package. This fork and patch flow is in early access today, so ask which packages it covers before you plan around it.

One honest limit: when the vulnerable code lives in a vendor appliance, a hosted service or an operating system runtime you do not build yourself, the fix still has to come from that vendor or distribution. The steps above apply to the open source dependencies and container images you build and ship, which is where most teams carry the largest and least tracked share of their exposure.

See how Safeguard would help

You can download Safeguard and follow the CLI docs to try these yourself.

The first demo sets up the Safeguard runner on your own machine, approves it with its device code and then scans from the terminal. For a GPU platform, that is how you would scan CUDA based images and the repositories that build them without sending source or images off the machine, for example safeguard scan --image against an inference image pinned by digest, or safeguard scan --sbom against a host SBOM like the one generated above.

Click through it at your own pace, or open it full screen.

The second demo is a real CLI session: install, open the interactive terminal and ask it to fix a lodash CVE. The assistant reads the finding and the advisory, asks before any command that changes something, edits package.json to the fixed version, runs npm install and npm test with approval, prints a summary and pushes nothing. The same flow applies to the Python side of a GPU service, such as moving a pinned framework wheel or serving engine dependency to a fixed version and rerunning your tests.

Click through it at your own pace, or open it full screen.

The third demo connects Claude to the Safeguard MCP server with OAuth and shows how to choose which tools Claude can use. When a new NVIDIA bulletin lands, an engineer can then ask which of their projects contain an affected component and version, and get the answer from their own scan data rather than from memory.

Click through it at your own pace, or open it full screen.

Quick check

Sources

• NVIDIA: CUDA Compatibility guide (overview)
• NVIDIA: Why CUDA Compatibility
• NVIDIA: Minor Version Compatibility
• NVIDIA: Forward Compatibility
• NVIDIA: CUDA Toolkit 13.4 Update 2 release notes
• NVIDIA: cuDNN support matrix
• NVIDIA: NCCL release notes
• NVIDIA: nvidia-smi documentation
• NVIDIA: open GPU kernel modules (README and licence)
• NVIDIA: Container Toolkit architecture overview
• NVIDIA: Container Toolkit installation guide
• NVIDIA: Container Toolkit CDI support
• NVIDIA: Container Toolkit releases on GitHub
• NVIDIA: GPU Operator overview
• NVIDIA: CUDA images on Docker Hub (flavours, tag policy, licences)
• NVIDIA: Product Security page
• NVIDIA: product security bulletins repository on GitHub
• NVIDIA bulletin 5582: Container Toolkit, September 2024 (CVE-2024-0132, CVE-2024-0133)
• NVIDIA bulletin 5585: Container Toolkit, November 2024 (CVE-2024-0134)
• NVIDIA bulletin 5599: Container Toolkit, January 2025 (CVE-2024-0135 to CVE-2024-0137)
• NVIDIA bulletin 5616: Container Toolkit, February 2025 (CVE-2025-23359)
• NVIDIA bulletin 5659: Container Toolkit, July 2025 (CVE-2025-23266, CVE-2025-23267)
• NVIDIA bulletin 5850: Container Toolkit, June 2026 (CVE-2026-24260)
• NVIDIA bulletin 5861: GPU Display Driver, September 2026
• NVIDIA bulletin 5586: GPU Display Driver, October 2024
• NVIDIA bulletin 5755: CUDA Toolkit, January 2026
• NIST NVD: CVE-2024-0132
• NIST NVD: CVE-2025-23266
• AMD: ROCm compatibility matrix
• AMD: Product Security bulletins
• Intel: Compute Runtime for oneAPI Level Zero and OpenCL (README)
• oneAPI Level Zero loader
• UXL Foundation: oneDNN, oneMath and oneCCL
• PyPI: torch 2.14.1 metadata
• PyPI: nvidia-cudnn-cu12 and nvidia-nccl-cu12
• CISA: 2026 Minimum Elements for a Software Bill of Materials
• CISA: 2026 Minimum Elements PDF
• CycloneDX: specification overview
• CycloneDX: 1.7 JSON schema (component types)
• CycloneDX: specification releases
• SPDX: specification releases
• Package URL specification: package types

Last reviewed: 25 October 2026.