How to Build GPU as a Service Yourself: A DIY Guide

To build GPU as a service yourself, put GPU servers on a supported Linux release, install the vendor driver (NVIDIA's driver or AMD's ROCm stack), add a GPU aware container runtime, and run Kubernetes with the vendor's device plugin or GPU Operator so tenants request GPUs like CPU and memory. Then add sharing (MIG or time slicing), per tenant quotas, usage metering from GPU metrics, and a patch routine for drivers, the container toolkit and model serving images.

Every command below comes from NVIDIA, AMD, Kubernetes, Kueue or vLLM documentation fetched on 23 October 2026. Versions move quickly in this stack, so treat every version string as an example and copy the current one from the linked page.

Key takeaways

• GPU as a service is five layers: hardware, host driver, container runtime integration, Kubernetes scheduling, and the workloads on top. Each has its own release cycle and security bulletins.
• NVIDIA recommends installing its driver through the distribution package manager; a compute-only install skips desktop components a server does not need.
• In Kubernetes, the device plugin advertises GPUs as whole integer resources such as nvidia.com/gpu or amd.com/gpu, and a quota uses the requests. prefix, for example requests.nvidia.com/gpu: 4.
• MIG splits supported GPUs (Ampere generation and newer) into as many as seven instances with separate memory paths and fault isolation. Time slicing shares any GPU but gives no memory or fault isolation between tenants. That difference should drive your tenancy design.
• The container toolkit is part of your attack surface. NVIDIA's July 2025 bulletin rated CVE-2025-23266 at 9.0 and fixed it in Container Toolkit 1.17.8 and GPU Operator 25.3.2. Track toolkit and operator versions as closely as the driver.
• Bill on allocation (GPU hours requested and held), not only on utilization. NVIDIA's own docs note that DCGM Exporter cannot tie metrics to containers when time slicing is enabled.
• Model servers have their own exposure. vLLM's security page says its --api-key flag protects only some endpoint prefixes and says not to rely on it alone.

What does GPU as a service actually mean?

GPU as a service means people or teams get GPU capacity on demand without owning the hardware. A DIY version usually serves an internal audience such as data scientists, training jobs or inference endpoints; selling spare capacity to outside customers raises the security bar a lot. Whatever the audience, the service promises a way to ask for a GPU (or a slice of one), fair access, separation of one user's code and data from another's, and accounting of who used what. DIY means you assemble those promises from open source and vendor components, which gives you control over cost and data location, and also every patch and every driver mismatch.

What do you need: the five layers

Batch clusters sometimes use a traditional HPC scheduler instead of Kubernetes. This guide uses Kubernetes because its device plugin and quota model are documented end to end by the Kubernetes project and both GPU vendors.

How do you choose hardware for a DIY GPU service?

This guide does not rank GPU models or vendors. The useful questions are about features and fit, answered from the vendor's own documentation.

Hardware selection criteria

The MIG User Guide says MIG is supported starting with the NVIDIA Ampere generation (compute capability 8.0 or higher) and that a GPU can be partitioned into up to seven instances. Its supported GPU table lists data center and workstation parts, not consumer gaming cards, so a homelab built from gaming GPUs is limited to whole card allocation, time slicing or MPS. For AMD, check ROCm's compatibility matrix for your exact GPU and kernel before you buy.

Power, cooling and the building

GPU servers draw far more power per rack unit than typical web servers and turn almost all of it into heat. Plan rack power, cooling and airflow with the server vendor's specifications, and have a licensed electrician design and install any new circuits. For the cost side, see our estimate of GPU cost per million tokens when self hosting.

Drivers, CUDA and ROCm: what goes on the host?

NVIDIA driver basics

NVIDIA's Driver Installation Guide covers branch 615 at the time of writing, including Ubuntu 22.04, 24.04 and 26.04. Points that shape a service design:

• Use the package manager. NVIDIA recommends distribution packages over the generic .run installer.
• Open kernel modules. NVIDIA says its kernel modules (nvidia.ko, nvidia-uvm.ko and others) are open source, dual licensed MIT and GPLv2. The Ubuntu install target is nvidia-open.
• Kernel headers must match the running kernel at install time and whenever the driver is rebuilt.
• Compute-only installs. Since the 560 branch you can skip desktop components; on Ubuntu that is libnvidia-compute plus nvidia-dkms-open. Fewer packages mean fewer things to patch.
• Version locking. From branch 590, Ubuntu package names drop the branch number, and you select a branch with a pinning package such as nvidia-driver-pinning-<branch>, installed before the driver.

Where CUDA fits

The driver is what the kernel needs; the CUDA Toolkit is what developers need to build CUDA code. Most tenants bring CUDA inside their container image, so hosts usually need only the driver. NVIDIA's CUDA guide documents apt install cuda-toolkit for hosts that do need it, such as build machines.

AMD and ROCm basics

AMD's ROCm quick start for Ubuntu 24.04 installs a repository package named amdgpu-install from repo.radeon.com, then the amdgpu-dkms kernel driver, then a reboot, then the rocm meta package. Two details matter for a shared service:

• GPU device permissions. AMD's install page says GPU access is controlled by membership in the render and video groups by default. It also documents a udev rule alternative that sets mode 0666 on /dev/kfd and the DRM render nodes, which grants GPU access to all users on the host. On a host shared between people, group membership is the tighter choice.
• Fast moving versions. At the time of writing, AMD's quick start page used the 7.2.4 installer and also said it had moved to a newer install page, which describes ROCm 10.1.0. Always copy the installer URL from AMD's current page.

How do containers get access to GPUs?

A container does not see a GPU by default. Something has to mount the device nodes and inject user space driver libraries that match the host kernel module. For NVIDIA GPUs, that is the NVIDIA Container Toolkit, installed from NVIDIA's signed repository. Its nvidia-ctk runtime configure command edits the container engine configuration:

• For Docker it modifies /etc/docker/daemon.json.
• For containerd it creates a drop in file at /etc/containerd/conf.d/99-nvidia.toml and updates the imports setting in /etc/containerd/config.toml.
• For CRI-O it creates /etc/crio/conf.d/99-nvidia.toml.
• For Podman, NVIDIA recommends the Container Device Interface (CDI) instead.

CDI is an open specification for describing what access to a device means, so any CDI aware runtime can use it. NVIDIA added CDI spec generation in toolkit v1.12.0. From v1.18.0, a systemd service called nvidia-cdi-refresh regenerates the spec at /var/run/cdi/nvidia.yaml automatically, and nvidia-ctk cdi list shows the devices it defines. CDI also matters for security history: NVIDIA's description of CVE-2024-0132 said that flaw did not affect setups using CDI.

NVIDIA also notes a known issue: on systems using systemd cgroup drivers, containers can lose access to requested GPUs when systemctl daemon-reload runs. Keep that in mind when you automate host configuration.

How does Kubernetes schedule GPUs?

Device plugins

Kubernetes has a device plugin framework, stable since v1.26, that lets a vendor plugin register with the kubelet over gRPC and advertise devices as an extended resource such as nvidia.com/gpu. The scheduler then places pods on nodes with free devices.

The Kubernetes "Schedule GPUs" page says GPUs go in the limits section; if you also set requests they must be equal, and you cannot set a request without a limit. The device plugin page adds that extended resources are integers, cannot be overcommitted, and cannot be shared between containers. Every GPU sharing method below works around those rules, either by presenting slices of a GPU as separate devices (MIG) or by advertising one GPU as several replicas (time slicing).

The NVIDIA GPU Operator

The NVIDIA GPU Operator automates what you would otherwise install by hand on every node. Its getting started page (patch release v26.7.2 at the time of writing) says a default install deploys the GPU driver, Container Toolkit, Device Plugin, DCGM Exporter and MIG Manager as pods on every GPU worker node, plus Node Feature Discovery. It can also use a driver and toolkit you pre-install, via driver.enabled=false and toolkit.enabled=false:

Either way, the operator namespace runs privileged components. Its docs tell clusters that use Pod Security Admission to label the gpu-operator namespace with pod-security.kubernetes.io/enforce=privileged. Treat that namespace as part of the host: no tenant should ever get rights in it. Our guide to Kubernetes Pod Security Standards explains why privileged namespaces need tight RBAC.

AMD device plugin

AMD's Kubernetes device plugin runs as a DaemonSet on every node with an AMD GPU and advertises amd.com/gpu. Its README says that if you enable the experimental device health checks, the pods must be allowed to run privileged to access /dev/kfd. AMD also publishes a GPU Operator that uses the plugin.

Dynamic Resource Allocation

Kubernetes Dynamic Resource Allocation (DRA), stable since v1.35, lets drivers and administrators define device classes that workloads claim. The docs list advantages over device plugins: CEL based device filtering, sharing a device through a shared claim, and per-claim configuration. ResourceQuota supports DRA too. Check whether your vendor's DRA driver supports your hardware; most vendor docs still walk through the device plugin path.

MIG, MPS or time slicing: how should tenants share a GPU?

A full data center GPU is often more than one inference service needs. Sharing raises utilization but changes the isolation story.

Comparison table

What MIG gives you

The MIG User Guide explains that each instance gets separate paths through the memory system: crossbar ports, L2 cache banks, memory controllers and DRAM address busses are assigned to one instance. NVIDIA says that for multi-tenant providers, MIG ensures one client cannot impact the work or scheduling of other clients. Operational facts to know first:

• Enabling MIG mode alone is not enough; you must create GPU instances and compute instances before CUDA workloads can run.
• MIG devices you create do not survive a reboot. NVIDIA points to its MIG Partition Editor (mig-parted) to recreate the layout at startup; in Kubernetes, MIG Manager does this for you.
• On Hopper and newer GPUs, MIG mode itself is not persistent across reboots and enabling it no longer needs a GPU reset. On Ampere GPUs, MIG mode is stored on the GPU and persists until you disable it.
• Profiles are named by slices and memory, such as 1g.5gb, 3g.20gb and 7g.40gb on a 40 GB A100 in NVIDIA's example, or 1g.10gb and 3g.40gb on an 80 GB H100.
• NVIDIA notes that NVML and nvidia-smi do not attribute utilization metrics to MIG devices on Ampere GPUs, which affects metering.

What time slicing and MPS do not give you

NVIDIA's time slicing page says that, unlike MIG, there is no memory or fault isolation between replicas. The device plugin README adds that workloads on a time-sliced GPU share its memory and one fault domain, so if one crashes, they all do. MPS partitions memory and compute per workload, but NVIDIA marks it experimental in the device plugin, unsupported on MIG devices, and without error isolation. Time slicing and MPS are mutually exclusive and apply to all GPUs on a node.

Time slicing and MPS are for tenants who already trust each other.

Scheduling, quotas and fair sharing

Per namespace GPU quotas

Kubernetes ResourceQuota is the first control. For extended resources, the docs say only quota items with the requests. prefix are allowed, because overcommit is not possible. The Kubernetes example limits a namespace to four GPUs with requests.nvidia.com/gpu: 4. Give each tenant its own namespace and quota, and pair that with RBAC so tenants cannot edit their own quota. Our walkthrough on setting up Kubernetes RBAC covers the role design.

Queues, priorities and fair sharing with Kueue

ResourceQuota rejects pods that exceed a limit; it does not queue them, and users usually want training jobs to wait for capacity rather than fail. Kueue, a Kubernetes SIGs project, decides when a job should wait, be admitted or be preempted. Its overview lists StrictFIFO and BestEffortFIFO queueing, fair sharing and cohorts between tenants, preemption, topology aware scheduling and quota for DRA devices, with integrations for batch Jobs, Kubeflow training jobs, RayJob, JobSet and plain pods.

A sensible pattern is: ResourceQuota as the hard ceiling per tenant, Kueue as the queue and fairness layer, and node labels or taints to steer tenants to the right GPU types.

Steering pods to the right hardware

The Kubernetes GPU page recommends labeling nodes by accelerator type and using node selectors. Node Feature Discovery can apply those labels automatically, and the GPU Operator's GPU Feature Discovery adds labels such as nvidia.com/gpu.product. Time slicing appends -SHARED to the product label by default, so a node selector can keep a workload off shared GPUs entirely.

Metering and billing

Allocation. Kubernetes knows which pod held which GPU resource and for how long. That is the fairest billing basis, because an idle reserved GPU still blocks everyone else. Bill GPU hours by resource type: whole GPU, each MIG profile, shared replica.

Utilization. NVIDIA's DCGM Exporter, deployed by the GPU Operator by default, exposes GPU metrics at /metrics on port 9400, with an option (off by default) to map metrics to pods. Use it for capacity planning and spotting idle reservations. Two documented limits keep it from being the only billing input: it does not associate metrics to containers under time slicing, and on Ampere GPUs NVML does not attribute utilization to MIG devices.

Model serving stacks

Most DIY GPU services end up hosting inference. vLLM's Kubernetes guide shows a plain Deployment with a PersistentVolumeClaim for the model cache and an optional Secret for gated models, and lists other deployment routes including Helm, KServe, KubeRay, LeaderWorkerSet and llm-d. This guide does not rank them.

vLLM's security page is worth reading in full before you expose an endpoint:

• Inter-node traffic is insecure by default. vLLM says multi-node communication (PyTorch distributed, KV cache transfer, parallelism traffic) must be protected by an isolated network, because it has no authorization or encryption.
• The API key is partial. --api-key protects only endpoints under /v1, /v2, /inference and /cohere; vLLM says not to rely on it alone.
• Use a reverse proxy or Kubernetes Gateway that allowlists the endpoints you intend to expose and adds authentication, rate limiting and logging.
• Never enable development mode in production. vLLM warns VLLM_SERVER_DEV_MODE=1 exposes endpoints including arbitrary RPC execution.

The sample manifest uses the vllm/vllm-openai:latest tag and --trust-remote-code. On a shared service, pin a version or digest so every node runs the image you scanned, and allow remote model code only for models you have reviewed. We cover the broader picture in ML model serving infrastructure security and recent issues in our vLLM September 2026 advisories post.

Tenant isolation risks you have to design for

A GPU service is a multi-tenant compute platform with an unusually privileged driver stack.

Containers share a kernel. The Kubernetes multi-tenancy docs say containers offer a weaker isolation boundary than virtual machines and recommend sandboxing, such as virtual machines, for untrusted code. GPU passthrough or vGPU is the GPU equivalent.

Node isolation helps but does not finish the job. Dedicating nodes to a tenant with taints and tolerations limits what an escaped container reaches, but Kubernetes notes the kubelet and API server are still shared. It also notes node isolation is easier to bill, per node.

The GPU container stack has had serious bugs. Two examples from NVIDIA's own records:

The toolkit runs with host privileges at container start, so a bug there turns a malicious image into a host problem, which is exactly the threat when tenants bring their own images.

Network exposure. Inference servers and distributed training traffic often assume a trusted network. Use NetworkPolicies to keep tenant namespaces apart and to keep multi-node traffic on its own segment; see our Kubernetes network policies deep dive and the related post on multi-tenant Kubernetes controller isolation.

Step by step: a small GPU cluster

This walkthrough builds a small NVIDIA based service on Ubuntu 24.04 nodes in an existing Kubernetes cluster that uses containerd. It pre-installs the driver and toolkit on hosts, then lets the GPU Operator handle the device plugin, metrics and MIG Manager. Commands are copied from the vendor pages listed in Sources. Run them on a test node first.

Step 1: Prepare each GPU node

Check the distribution and kernel, then install the matching headers, as NVIDIA's pre-installation actions describe:

hostnamectl
uname -r
sudo apt install linux-headers-$(uname -r)

Enable NVIDIA's network repository with the cuda-keyring package, replacing <distribution> with the value NVIDIA's Ubuntu page gives for your release:

wget https://developer.download.nvidia.com/compute/cuda/repos/<distribution>/x86_64/cuda-keyring_1.1-1_all.deb
sudo dpkg -i cuda-keyring_1.1-1_all.deb
sudo apt update

Step 2: Pin a driver branch, then install a compute-only driver

Pinning first keeps an unattended upgrade from jumping branches under running jobs:

sudo apt install nvidia-driver-pinning-<branch>
sudo apt -V install libnvidia-compute nvidia-dkms-open
sudo reboot

If you need the full driver with desktop components, NVIDIA's Ubuntu page uses sudo apt install nvidia-open instead. After the reboot, confirm the loaded version and that the persistence daemon is running:

cat /proc/driver/nvidia/version
sudo systemctl restart nvidia-persistenced
nvidia-smi

Step 3: Install and configure the NVIDIA Container Toolkit

Add NVIDIA's signed repository and install a specific toolkit version. The docs used 1.20.1-1 at the time of writing:

sudo apt-get update && sudo apt-get install -y --no-install-recommends \
   ca-certificates \
   curl \
   gnupg2

curl -fsSL https://nvidia.github.io/libnvidia-container/gpgkey | sudo gpg --dearmor -o /usr/share/keyrings/nvidia-container-toolkit-keyring.gpg \
  && curl -s -L https://nvidia.github.io/libnvidia-container/stable/deb/nvidia-container-toolkit.list | \
    sed 's#deb https://#deb [signed-by=/usr/share/keyrings/nvidia-container-toolkit-keyring.gpg] https://#g' | \
    sudo tee /etc/apt/sources.list.d/nvidia-container-toolkit.list

sudo apt-get update

export NVIDIA_CONTAINER_TOOLKIT_VERSION=1.20.1-1
  sudo apt-get install -y \
      nvidia-container-toolkit=${NVIDIA_CONTAINER_TOOLKIT_VERSION} \
      nvidia-container-toolkit-base=${NVIDIA_CONTAINER_TOOLKIT_VERSION} \
      libnvidia-container-tools=${NVIDIA_CONTAINER_TOOLKIT_VERSION} \
      libnvidia-container1=${NVIDIA_CONTAINER_TOOLKIT_VERSION}

Do not enable the experimental repository on a production service. Configure containerd for Kubernetes and restart it:

sudo nvidia-ctk runtime configure --runtime=containerd
sudo systemctl restart containerd

If the node also has Docker for local testing, NVIDIA's sample workload check is:

sudo nvidia-ctk runtime configure --runtime=docker
sudo systemctl restart docker
sudo docker run --rm --runtime=nvidia --gpus all ubuntu nvidia-smi

Check the CDI devices the refresh service generated:

nvidia-ctk cdi list

Step 4: Install the GPU Operator

From a machine with kubectl and helm, create the namespace and label it privileged if you use Pod Security Admission:

kubectl create ns gpu-operator
kubectl label --overwrite ns gpu-operator pod-security.kubernetes.io/enforce=privileged

Check whether Node Feature Discovery already runs, because the operator deploys its own by default:

kubectl get nodes -o json | jq '.items[].metadata.labels | keys | any(startswith("feature.node.kubernetes.io"))'

NVIDIA publishes the chart as an OCI artifact and documents a Cosign signature check before install:

cosign verify \
    --key https://api.ngc.nvidia.com/v2/catalog/containers/public-key \
    --insecure-ignore-tlog \
    nvcr.io/nvidia/cloud-native-charts/gpu-operator:v26.7.2

Note that NVIDIA's documented command skips the transparency log check. The key check still confirms the chart was signed with NVIDIA's published key.

Because the driver and toolkit are already on the hosts, install with both disabled. For this scenario NVIDIA's docs say to set the default runtime to nvidia first, following the toolkit's configuration page:

helm repo add nvidia https://helm.ngc.nvidia.com/nvidia \
    && helm repo update

helm install --wait --generate-name \
     -n gpu-operator --create-namespace \
     nvidia/gpu-operator \
     --version=v26.7.2 \
     --set driver.enabled=false \
     --set toolkit.enabled=false

Verify the pods, the ClusterPolicy and the advertised GPUs:

kubectl get pods -n gpu-operator
kubectl get clusterpolicy
kubectl get nodes -o json \
    | jq '.items[] | select(.status.allocatable["nvidia.com/gpu"] != null) | {name: .metadata.name, gpus: .status.allocatable["nvidia.com/gpu"]}'

Step 5: Run a test workload

Create cuda-vectoradd.yaml from NVIDIA's example:

apiVersion: v1
kind: Pod
metadata:
  name: cuda-vectoradd
spec:
  restartPolicy: OnFailure
  containers:
  - name: cuda-vectoradd
    image: "nvcr.io/nvidia/k8s/cuda-sample:vectoradd-cuda12.5.0-ubuntu22.04"
    resources:
      limits:
        nvidia.com/gpu: 1

kubectl apply -f cuda-vectoradd.yaml
kubectl logs pod/cuda-vectoradd
kubectl delete -f cuda-vectoradd.yaml

The log should end with Test PASSED.

Step 6: Create a tenant with a GPU quota

Using the Kubernetes ResourceQuota example, give a tenant namespace a ceiling of four GPUs:

kubectl create namespace myspace
cat <<EOF > compute-resources.yaml
apiVersion: v1
kind: ResourceQuota
metadata:
  name: compute-resources
spec:
  hard:
    requests.cpu: "1"
    requests.memory: "1Gi"
    limits.cpu: "2"
    limits.memory: "2Gi"
    requests.nvidia.com/gpu: 4
EOF
kubectl create -f ./compute-resources.yaml --namespace=myspace

Size the CPU and memory values for real GPU jobs; the numbers above are the Kubernetes docs example. Then bind the tenant's group to a namespaced role, and add Kueue if you want jobs to wait in a queue rather than fail on quota.

Step 7: Partition GPUs with MIG

On a MIG capable node, set the strategy and apply a profile through a node label. MIG Manager reconfigures the GPUs and updates nvidia.com/mig.config.state:

kubectl patch clusterpolicies.nvidia.com/cluster-policy \
    --type='json' \
    -p='[{"op":"replace", "path":"/spec/mig/strategy", "value":"single"}]'

kubectl label nodes <node-name> nvidia.com/mig.config=all-1g.10gb --overwrite

kubectl get node <node-name> -o=jsonpath='{.metadata.labels}' | jq .

MIG Manager stops GPU clients on the node while it applies the change, so use a maintenance window. On a host without Kubernetes, the MIG User Guide does the same by hand:

sudo nvidia-smi -i 0 -mig 1
nvidia-smi mig -lgip
sudo nvidia-smi mig -cgi 9,3g.20gb -C
nvidia-smi -L

Step 8 (alternative): Time slicing for trusted, low priority pools

For a development pool of older GPUs, NVIDIA's sample ConfigMap advertises four replicas per GPU:

apiVersion: v1
kind: ConfigMap
metadata:
  name: time-slicing-config-all
data:
  any: |-
    version: v1
    flags:
      migStrategy: none
    sharing:
      timeSlicing:
        renameByDefault: false
        failRequestsGreaterThanOne: false
        resources:
          - name: nvidia.com/gpu
            replicas: 4

kubectl create -n gpu-operator -f time-slicing-config-all.yaml
kubectl patch clusterpolicies.nvidia.com/cluster-policy \
    -n gpu-operator --type merge \
    -p '{"spec": {"devicePlugin": {"config": {"name": "time-slicing-config-all", "default": "any"}}}}'
kubectl get events -n gpu-operator --sort-by='.lastTimestamp'

To limit it to certain nodes, NVIDIA documents per-node configs selected with a label such as kubectl label node <node-name> nvidia.com/device-plugin.config=tesla-t4. Consider renameByDefault: true so shared capacity is advertised as nvidia.com/gpu.shared and nobody lands on it by accident.

Step 9: The AMD path

For AMD Instinct nodes, AMD's quick start for Ubuntu 24.04 (installer 7.2.4 at the time of fetch) runs:

wget https://repo.radeon.com/amdgpu-install/7.2.4/ubuntu/noble/amdgpu-install_7.2.4.70204-1_all.deb
sudo apt install ./amdgpu-install_7.2.4.70204-1_all.deb
sudo apt update
sudo apt install "linux-headers-$(uname -r)" "linux-modules-extra-$(uname -r)"
sudo apt install amdgpu-dkms
sudo reboot
sudo usermod -a -G render,video $LOGNAME
sudo apt install rocm

Verify with rocminfo and amd-smi version, which AMD's current install page documents. Then deploy AMD's device plugin and request amd.com/gpu in pod limits:

kubectl create -f https://raw.githubusercontent.com/ROCm/k8s-device-plugin/master/k8s-ds-amdgpu-dp.yaml

For production, download that manifest, review it, and pin it to a tagged release rather than applying from the master branch.

Step 10: Put a model server behind a gateway

Deploy your serving stack from its own docs, with a pinned image, an internal Service, NetworkPolicies limiting who can reach it, and a reverse proxy or Gateway in front that allowlists the endpoints you intend to expose. Keep multi-node serving traffic on an isolated network as vLLM's security page requires.

How do you keep drivers and the GPU stack patched?

1. Subscribe to the right feeds. NVIDIA's product security page says that from 1 October 2026, NVIDIA PSIRT publishes bulletins on GitHub in Markdown, CSAF and CVE formats, in the nvidia/product-security repository, alongside the website. CSAF is machine readable, so you can feed it into your own tracking. Follow AMD's ROCm release notes the same way.
2. Keep a version inventory per node: driver (cat /proc/driver/nvidia/version), toolkit package, GPU Operator chart, device plugin, MIG Manager and serving image digests. The July 2025 bulletin touched four of those.
3. Patch drivers by draining nodes. On Ubuntu, NVIDIA's upgrade path is apt dist-upgrade under your pinning package. Cordon and drain, upgrade kernel and headers together, reboot, recreate MIG layouts if you manage them by hand, and rerun the vectoradd test before you uncordon.
4. Rebuild tenant images when base CUDA or ROCm images, frameworks or model servers get fixes; see how to scan Docker images for vulnerabilities.
5. Verify what you deploy. Extend NVIDIA's Cosign chart check to admission policy that admits only signed images; see enforcing Cosign signatures in Kubernetes admission and the Helm chart supply chain blueprint.

Where Safeguard fits, honestly

Safeguard does not install GPU drivers, partition GPUs, schedule jobs, meter usage or isolate tenants, and it does not inspect host kernel modules or GPU firmware. Everything in the walkthrough works without it. The NVIDIA driver and toolkit packages on your hosts are tracked through your package manager and NVIDIA's bulletins.

The overlap is the software you build and ship on top: the container images your tenants run, the serving images you publish, and the application dependencies inside them. Safeguard scans repositories, container images and SBOM files (for example safeguard scan --image <name> from the CLI, with results written as JSON under ./output), matches components against vulnerability databases, uses reachability analysis to rank findings, and shows which of your projects contain a given component and version. When an advisory lands for a framework or model server, that shows which tenant facing images carry it. Policies and gates can block promotion of a known vulnerable image, and autonomous remediation, off by default, can open fix pull requests.

Common mistakes

• Treating time slicing as isolation. NVIDIA says it gives no memory or fault isolation. Use MIG, whole GPUs, separate nodes or virtual machines for tenants who do not trust each other.
• Letting drivers float. Without a pinning package, an unattended upgrade can move a node to a new branch mid job. Pin, then upgrade on purpose.
• Forgetting the toolkit and operator in patch plans. The July 2025 bulletin needed new versions of the toolkit, GPU Operator, device plugin and MIG Manager. Teams that tracked only the driver missed three of them.
• Giving tenants rights in the GPU Operator namespace. It runs privileged pods. Keep it admin only.
• Using the 0666 udev rule on shared AMD hosts. It gives every local user GPU access. Prefer render and video group membership.
• Billing on utilization alone. DCGM metrics cannot be attributed to containers under time slicing, and MIG attribution has limits on some GPUs. Bill on allocated GPU hours.
• Exposing a model server with only an API key. vLLM says its key protects only some endpoints. Put a gateway in front with an explicit allowlist.
• Running latest images and --trust-remote-code everywhere. Pin image versions or digests, and allow remote model code only for reviewed models.
• Assuming MIG survives a reboot. MIG devices do not, and on Hopper and newer the mode itself does not. Let MIG Manager or mig-parted recreate the layout.

FAQ

Can I build my own GPU cloud?

Yes. You need a supported Linux distribution, the vendor driver, a GPU aware container runtime, Kubernetes with the vendor device plugin or GPU Operator, ResourceQuota and a queue such as Kueue, DCGM Exporter for metrics, and a gateway in front of model servers. The hard parts are operations: patching, tenant isolation, metering and support.

Do I need the GPU Operator, or can I just use the device plugin?

You can run the NVIDIA device plugin alone if you install and patch the driver and toolkit on every host yourself. The operator adds automation, GPU Feature Discovery labels, metrics and MIG Manager. Many teams use it with driver.enabled=false so the OS team keeps owning the driver.

What is the difference between MIG and time slicing?

MIG partitions a supported GPU into as many as seven hardware instances with separate memory paths and fault isolation. Time slicing advertises one GPU as several replicas that take turns, with no memory or fault isolation. MIG suits multi-tenant inference; time slicing suits trusted development pools and older GPUs.

Which GPUs support MIG?

NVIDIA's MIG User Guide says MIG is supported starting with the Ampere generation, meaning compute capability 8.0 or higher, and lists specific data center and workstation GPUs in its supported GPU table. Check that table for your exact model.

How do I limit how many GPUs a team can use in Kubernetes?

Create a ResourceQuota in the team's namespace with an item such as requests.nvidia.com/gpu: 4. Kubernetes only allows the requests. prefix for extended resources. For queueing and fair sharing on top of hard limits, add Kueue.

Is it safe to run untrusted customer containers on shared GPUs?

Not on shared nodes with time slicing. Kubernetes recommends sandboxing such as virtual machines for untrusted code, and container toolkit flaws like CVE-2025-23266 show how a malicious image can reach the host. Use dedicated nodes or virtual machines with GPU passthrough, keep the toolkit patched, and isolate the network.

Does Kubernetes support AMD GPUs?

Yes. AMD's device plugin advertises amd.com/gpu once ROCm and the amdgpu-dkms driver are on the hosts, and AMD also publishes a GPU Operator.

How often should I update GPU drivers?

Whenever a security bulletin affects your branch, and on a planned schedule otherwise. Watch NVIDIA's nvidia/product-security GitHub repository and AMD's release notes, drain nodes before upgrading, and test on one node before rolling out.

Fix it without waiting for upstream

A GPU service is mostly container images: CUDA and ROCm base images, Python frameworks, model servers and the tenants' own application code, all of which pull open source components that receive their own CVEs. The host driver and kernel modules still come from NVIDIA or AMD on their schedule, but the images you build and ship on top are where the steps below apply.

Whatever the vulnerability, most of the risk sits in two gaps. The first is the time between a fix being published and that fix actually running in your production builds. The second, and worse, is the time before any fix exists at all, while a maintainer works out a patch and a release schedule. Waiting through both gaps is a habit, not a requirement. Safeguard gives teams three ways to close them.

1. Autonomous remediation: pull requests, not tickets

When a fixed version exists, Safeguard's Griffin models open a tested, policy gated pull request that moves the affected dependency to the safe version. You can run it on one repository or in bulk across every project that carries the vulnerable version. Remediation covers eight categories, including dependencies, licenses, security posture and findings. Reachability analysis ranks the vulnerable code paths your application actually calls first, so the first pull requests you merge are the ones that matter most. Autonomous remediation ships switched off, with every category set to Manual, so your team decides how much runs on its own and what still needs a human approval.

2. Gold Open Source: the safe version in seconds, free

gold.safeguard.sh is free and needs no login. Search any CVE or package to see affected and fixed versions, CVSS, EPSS, CISA KEV status and a recommended safe version, or use the "Am I affected?" checker. The same data is available through a free read API, README status badges and RSS threat feeds. The safeguard-sh/gold-scan-action GitHub Action can fail a build on a known exploited vulnerability, on any known vulnerability, or never, so a version you have just removed cannot quietly come back in next week's merge.

3. Gold Registry: start from components that are already fixed

registry.safeguard.sh provides hardened, drop in replacements for upstream packages and container images. Every Gold artifact is held continuously to zero exploitable critical or high CVEs, zero malware and a completed license check, with SLSA L3 or higher provenance, a Sigstore signature and a public evidence page you can hand to an auditor. An artifact loses its Gold badge if it regresses. Teams can point their own package directory at the registry so hardened equivalents resolve without changing how developers install anything.

When upstream has no fix yet, or will not ship one, Safeguard first works to get the fix accepted upstream. If the issue is severe or upstream will not move, Safeguard builds a fork, fixes the vulnerability there and publishes a hardened Gold build, so you are not stuck on a maintainer's release schedule. You can also request a hardened, compatibility tested build of a specific package. This fork and patch flow is in early access today, so ask which packages it covers before you plan around it.

One honest limit: when the vulnerable code lives in a vendor appliance, a hosted service or an operating system runtime you do not build yourself, the fix still has to come from that vendor or distribution. The steps above apply to the open source dependencies and container images you build and ship, which is where most teams carry the largest and least tracked share of their exposure.

See how Safeguard would help

None of these demos set up GPUs, drivers or Kubernetes. They show the software supply chain side: finding and fixing vulnerable components in the code and images that run on your GPU service. To try the CLI yourself, download it from https://safeguard.sh/download and follow the setup guide at https://docs.safeguard.sh/docs/cli .

The first demo is a real CLI session: you ask safeguard to fix the lodash CVE in a repo, and it reads the finding, asks before each command, edits package.json to lodash 4.17.21, runs npm install and npm test with your approval, and pushes nothing. The same flow applies to a dependency inside an inference API image you are about to publish to tenants.

Click through it at your own pace, or open it full screen.

The second demo sets up the Safeguard runner on your own machine, approves it with its device code, and then scans from the terminal. For a DIY GPU service, that is how you would scan serving images and model server repositories inside your own network; our post on how the local runner and offline mode work has the details.

Click through it at your own pace, or open it full screen.

The third demo shows the desktop app sending work to a computer running the CLI, with each command approved before it runs, so a person stays in charge of build machines for GPU images.

Click through it at your own pace, or open it full screen.

Quick check

Sources

• NVIDIA Driver Installation Guide
• NVIDIA Driver Installation Guide: Ubuntu
• NVIDIA Driver Installation Guide: Pre-installation Actions
• NVIDIA Driver Installation Guide: Post-installation Actions
• NVIDIA Driver Installation Guide: Kernel Modules
• NVIDIA Driver Installation Guide: Compute-only and Desktop Installation
• NVIDIA Driver Installation Guide: Version locking
• NVIDIA CUDA Installation Guide for Linux
• NVIDIA Container Toolkit: Installing the NVIDIA Container Toolkit
• NVIDIA Container Toolkit: Support for Container Device Interface
• NVIDIA Container Toolkit: Running a Sample Workload
• NVIDIA GPU Operator: Installing the NVIDIA GPU Operator
• NVIDIA GPU Operator: Platform Support
• NVIDIA GPU Operator: Time-Slicing GPUs in Kubernetes
• NVIDIA GPU Operator: GPU Operator with MIG
• NVIDIA MIG User Guide
• NVIDIA MIG User Guide: Introduction
• NVIDIA MIG User Guide: Supported GPUs
• NVIDIA MIG User Guide: Concepts
• NVIDIA MIG User Guide: Deployment Considerations
• NVIDIA MIG User Guide: Getting Started with MIG
• NVIDIA k8s-device-plugin README
• NVIDIA DCGM Exporter documentation
• NVIDIA Product Security
• NVIDIA product-security repository on GitHub
• NVIDIA Security Bulletin: NVIDIA Container Toolkit, July 2025 (bulletin 5659)
• NVD: CVE-2024-0132
• NVD: CVE-2025-23266
• AMD ROCm: Quick start installation guide
• AMD ROCm: Install AMD ROCm
• AMD GPU Device Plugin for Kubernetes README
• Kubernetes: Device Plugins
• Kubernetes: Schedule GPUs
• Kubernetes: Resource Quotas
• Kubernetes: Dynamic Resource Allocation
• Kubernetes: Multi-tenancy
• Kueue: Overview
• vLLM: Using Kubernetes
• vLLM: Security

Last reviewed: 23 October 2026. Driver branches, toolkit, operator and ROCm versions change often; copy the current version strings from the linked vendor pages before you install.