Build a 3-Node Homelab Kubernetes Cluster With GPU Passthrough

To build a three node homelab Kubernetes cluster with GPU passthrough, install Proxmox VE on three machines that support an IOMMU, bind the GPU to the vfio-pci driver, and hand it to a q35, UEFI virtual machine. Then run one Kubernetes server VM per host (k3s with embedded etcd or kubeadm with stacked etcd), install the NVIDIA driver and Container Toolkit in the GPU VM, and expose the card through the NVIDIA device plugin or GPU Operator.

The rest of this guide is the detail that decides whether it keeps working after the first upgrade. Every command and version below was checked against the project's own documentation in October 2026.

Key takeaways

• Passthrough lives or dies on the IOMMU group. The GPU and its audio function must sit in a group of their own, or with only their root port.
• Intel and AMD both enable the IOMMU by default on kernel 6.8 or newer, according to the Proxmox VE admin guide. You still have to enable VT-d or AMD-Vi in firmware.
• A VM with a passed through GPU cannot live migrate. Plan Kubernetes, not the hypervisor, to provide availability for GPU work.
• Three servers is the smallest useful HA layout: they survive one failure, and four still survive only one.
• On k3s, install the NVIDIA driver and Container Toolkit first; k3s detects the runtime, and pods need it as default or runtimeClassName: nvidia.
• Time slicing shares one GPU with no memory or fault isolation.
• Upgrade one minor version at a time, servers first, after taking an etcd snapshot you know how to restore.
• Electricity is the recurring cost: at the July 2026 US average of 18.31 cents per kWh, 100 W of continuous draw costs about 160 US dollars a year.

If you have not built the physical side yet, start with how to build a tiny data center at home. If you want to rent the GPU to other people afterwards, read how to build GPU as a service yourself. This post sits between those two.

The architecture in one picture

The layout this guide builds is deliberately boring:

Each Kubernetes VM is a full server that also runs workloads. You can add GPUs to the other nodes later without changing the design.

Why VMs instead of bare metal? You can snapshot a node before a risky upgrade and rebuild its OS without touching the hypervisor. If you never need that, bare metal k3s on three machines is simpler, and the Kubernetes half of this guide applies unchanged.

Hardware plan: what passthrough actually needs

CPU and chipset

The Proxmox VE admin guide states the requirement plainly: the hardware must support IOMMU interrupt remapping, in both the CPU and the motherboard. On Intel that means VT-d. On AMD it means AMD-Vi. On arm64 the equivalent is the SMMU.

The Proxmox wiki says passthrough is not possible without interrupt remapping. A module option, allow_unsafe_interrupts=1, forces it through, but the admin guide warns it can make the system unstable; treat a board that needs it as the wrong board.

The admin guide also notes that server grade hardware often has better passthrough support than consumer hardware, though many modern consumer systems work. The practical advice is to check your board vendor's Linux notes before you buy, not after.

IOMMU groups and ACS

The Linux kernel VFIO documentation explains why groups exist: a multi function card, or a PCIe bridge without Access Control Services (ACS), can let devices talk without the IOMMU seeing the traffic, so devices that cannot be isolated share a group, and VFIO treats the group as the unit of ownership. In the documentation's words, the group "is the minimum granularity that must be used to ensure secure user access."

What that means for you:

• The GPU's graphics function and its HDMI audio function will normally share a group. That is fine; you pass both.
• The Proxmox admin guide says it is also fine for the GPU to share a group with its own root port or PCIe bridge.
• If the GPU shares a group with your NVMe controller or network card, you cannot pass the GPU without also handing those to the VM.

The Proxmox wiki lists three fixes: enable ACS in firmware, move the card to another slot, and as a last resort the pcie_acs_override=downstream kernel option, which the wiki says "is not without risks." The override pretends devices are isolated when the hardware has not promised it; never use it on a cluster that runs other people's workloads.

GPU

For CUDA work you need no display output; the Proxmox admin guide says a monitor or remote desktop is only needed for graphics. Check these before you buy:

• UEFI capable ROM. The admin guide recommends OVMF (UEFI firmware for the VM) for best compatibility, and says the GPU needs a UEFI capable ROM for that. Most modern cards have one.
• VRAM size and the 64 bit MMIO window. The Proxmox wiki documents a "BAR 3: can't reserve" error when passing a large VRAM card to an OVMF VM. By default OVMF reserves 32 GB of address space for the 64 bit PCI MMIO window, which is not enough for cards with 24 GB of VRAM or more. The fixes are to use the host CPU type, which changes how OVMF sizes the window, or to set a larger window with a firmware argument. Both are covered in the walkthrough.
• Reset and Resizable BAR quirks. The wiki notes a reset bug on some AMD Navi cards, and recommends turning Resizable BAR off on the host for some cards because VMs do not support it.

Memory, disks and network

Published floors: kubeadm wants 2 GB of RAM and 2 CPUs for control plane machines, k3s lists 2 cores and 2 GB for a server, and Longhorn's best practices page recommends 3 nodes with 4 vCPUs and 4 GiB each. Add your workloads on top; 6 to 8 vCPUs and 16 to 32 GB of RAM per Kubernetes VM is comfortable.

For disks, the k3s requirements page says etcd is write intensive, and Longhorn warns that spinning disk latency can make volumes unstable. Use NVMe for VM disks and Longhorn data.

Passthrough has one memory consequence people miss. The device does DMA straight into guest memory, so the hypervisor has to keep that memory mapped for the device. Plan as if the GPU VM's RAM is fully committed on the host; do not count on ballooning to give it back.

One 2.5 or 10 Gb link per node with VLANs is enough; Longhorn replication traffic saturates it first.

Hypervisor setup: Proxmox VE with VT-d or AMD-Vi and vfio-pci

Do these steps on the node that holds the GPU. The other two nodes only need Proxmox installed and joined to the same Proxmox cluster if you want one management view.

Step 1: enable the IOMMU in firmware

Enter firmware setup and enable the option the board calls VT-d, IOMMU or AMD-Vi. The admin guide notes the exact name varies by board. While you are there, enable SR-IOV and ACS if they are offered, and disable Resizable BAR for the slot you are passing through.

Step 2: kernel command line

The Proxmox admin guide says that with AMD CPUs the IOMMU is enabled by default, and with recent kernels (6.8 or newer) the same is true for Intel. On older kernels, Intel needs intel_iommu=on. The guide also documents iommu=pt, which puts devices the host keeps for itself in passthrough mode and can improve performance.

How you edit the command line depends on which boot loader your Proxmox install uses.

# Which boot loader is in use?
proxmox-boot-tool status

# GRUB: edit the default command line, then regenerate
sudo nano /etc/default/grub
# GRUB_CMDLINE_LINUX_DEFAULT="quiet iommu=pt"
# (add intel_iommu=on only on kernels older than 6.8 with an Intel CPU)
sudo update-grub

# systemd-boot: the command line is a single line in /etc/kernel/cmdline
sudo nano /etc/kernel/cmdline
sudo proxmox-boot-tool refresh

Reboot, then confirm.

dmesg | grep -e DMAR -e IOMMU -e AMD-Vi
dmesg | grep 'remapping'

You want a line saying the IOMMU or DMAR is enabled, and either AMD-Vi: Interrupt remapping enabled or DMAR-IR: Enabled IRQ remapping in x2apic mode, which are the strings the Proxmox wiki lists.

Step 3: load the VFIO modules

The admin guide lists three modules to load at boot: vfio, vfio_iommu_type1 and vfio_pci. Put them in a file under /etc/modules-load.d/, refresh the initramfs and check.

printf "vfio\nvfio_iommu_type1\nvfio_pci\n" | sudo tee /etc/modules-load.d/vfio.conf
sudo update-initramfs -u -k all
# after the next reboot:
lsmod | grep vfio

Step 4: check the IOMMU groups

pvesh get /nodes/$(hostname)/hardware/pci --pci-class-blacklist ""

Find the GPU (vendor 0x10de for NVIDIA) and read its iommugroup column. Then confirm nothing else important shares that number. If it does, go back to the slot and ACS options above before reaching for the override.

Step 5: bind the GPU to vfio-pci

The admin guide says Proxmox tries to make the device unavailable to the host automatically. If that does not work, it gives two methods: pass the vendor and device IDs to vfio-pci, or blacklist the host driver. You can do both, and the guide adds a softdep line for cases where the GPU driver loads before vfio-pci.

# Find the vendor:device IDs for every function of the card
lspci -nn | grep -i nvidia
# 01:00.0 VGA compatible controller [0300]: NVIDIA Corporation ... [10de:XXXX]
# 01:00.1 Audio device [0403]: NVIDIA Corporation ... [10de:YYYY]

# Claim both functions for vfio-pci (replace with your IDs)
echo "options vfio-pci ids=10de:XXXX,10de:YYYY" | sudo tee /etc/modprobe.d/vfio.conf

# Keep host GPU drivers off the card (the wiki lists these for NVIDIA)
echo "blacklist nouveau" | sudo tee -a /etc/modprobe.d/blacklist.conf
echo "blacklist nvidia*" | sudo tee -a /etc/modprobe.d/blacklist.conf

sudo update-initramfs -u -k all
sudo reboot

After the reboot, the admin guide's check is:

lspci -nnk -s 01:00
# Kernel driver in use: vfio-pci

If it still shows nouveau or nvidia, add the soft dependency the guide describes, for example softdep nouveau pre: vfio-pci, refresh the initramfs and reboot again.

One warning from the Proxmox wiki is worth repeating: blacklisting a driver applies to every device that uses it. If you pass through an Intel discrete GPU while the host uses an Intel iGPU, blacklisting i915 takes the iGPU away from the host too. Prefer the ids= method when the host needs another card from the same vendor.

Step 6: create the GPU VM

The admin guide says the best compatibility comes from the q35 machine type, OVMF instead of SeaBIOS, and PCIe instead of PCI. PCIe passthrough is only available on q35. For compute only use, you do not need x-vga.

# Create VM 101 with q35, OVMF, host CPU type, 8 cores, 32 GB RAM
qm create 101 --name k8s-1 --machine q35 --bios ovmf --cpu host \
  --cores 8 --memory 32768 --balloon 0 \
  --efidisk0 local-lvm:1,efitype=4m,pre-enrolled-keys=0 \
  --scsihw virtio-scsi-single --scsi0 local-lvm:120,iothread=1,discard=on \
  --net0 virtio,bridge=vmbr0,tag=20 \
  --ostype l26

# Pass both GPU functions as one PCIe device (01:00 means all functions)
qm set 101 -hostpci0 01:00,pcie=1

The 01:00 shorthand is documented in the admin guide as equivalent to ticking All Functions in the web interface. The tag=20 on the network device puts the VM on VLAN 20; the admin guide documents tag as a VLAN ID from 1 to 4094.

Two more settings from the Proxmox documentation:

• CPU type. The admin guide explains that the host CPU type gives the VM exactly the host's CPU flags, at the cost of live migration between different CPUs. The GPU VM cannot migrate anyway, so host costs nothing here, and the wiki notes it also helps OVMF size the MMIO window for large GPUs.
• Large VRAM cards. If the guest logs BAR ... can't reserve, the wiki gives a firmware argument that raises the 64 bit MMIO window, for example to 64 GB:

qm set 101 --args '-fw_cfg name=opt/ovmf/X-PciMmio64Mb,string=65536'

If you plan to manage the card from several nodes later, Proxmox also supports cluster wide resource mappings, so a VM refers to a named mapping with mapping= instead of a raw PCI address. With one GPU in one node, the raw address is fine.

Step 7: what passthrough gives up

The Proxmox wiki is explicit: a device passed to a VM is no longer available to the host, and VMs with passed through devices cannot be migrated. Put plainly, your GPU VM is pinned to node 1. If node 1 is down, the GPU is down, and Kubernetes should reschedule CPU workloads elsewhere while GPU pods wait. Design for that rather than fighting it.

Kubernetes choice: k3s or kubeadm, and the three server HA layout

Which distribution

This guide gives k3s commands in full and kubeadm equivalents where they differ.

As of 23 October 2026, the Kubernetes project maintains release branches for 1.37, 1.36 and 1.35. The 1.37.1 patch release came out on 15 September 2026, and 1.37 reaches end of life on 28 October 2027 according to the project's releases page. Pick a supported minor and stay within the support window.

Why three servers, and why not four

The k3s documentation explains the arithmetic. An HA embedded etcd cluster must have an odd number of servers. For n servers, quorum is (n/2)+1, rounded down in the division. So:

Two servers tolerate no failures, and four tolerate the same single failure as three. The kubeadm HA documentation also recommends an odd number of control plane nodes.

In a three node homelab, every node is a server. k3s labels them control-plane,etcd and, by default, still schedules workloads on them. That is what you want with only three machines.

Prepare each Kubernetes VM

Use the same Linux distribution and version on all three VMs. The GPU Operator documentation notes that if the operator manages drivers, all GPU nodes must run the same OS version. Then on each VM:

# Unique hostname and product_uuid per node (kubeadm requires both)
hostnamectl set-hostname k8s-1          # k8s-2, k8s-3 on the others
sudo cat /sys/class/dmi/id/product_uuid

# Patch the OS before Kubernetes goes on
sudo apt update && sudo apt full-upgrade -y

If you cloned the VMs from one template, regenerate the machine ID and check the product UUIDs differ. kubeadm's install page lists a unique hostname, MAC address and product_uuid for every node as a requirement.

Open the ports the k3s requirements page lists between the three VMs: TCP 6443 for the API server from all nodes, TCP 2379 and 2380 between servers for embedded etcd, UDP 8472 for Flannel VXLAN (or UDP 51820 if you choose the WireGuard backend), and TCP 10250 between nodes if you run the metrics server.

Install k3s with embedded etcd

Write a config file first so every server gets identical settings. The k3s HA page warns that some flags must match on all servers, including the cluster and service CIDRs, the disabled components and secrets encryption.

# /etc/rancher/k3s/config.yaml on all three servers
token: "REPLACE-WITH-A-LONG-RANDOM-VALUE"
tls-san:
  - 10.0.20.10          # a fixed registration address if you use one
disable:
  - servicelb           # we will run MetalLB instead
flannel-backend: wireguard-native
secrets-encryption: true
protect-kernel-defaults: true
write-kubeconfig-mode: "0600"

The k3s networking page says ServiceLB must be disabled with --disable=servicelb on all servers if you want a different load balancer such as MetalLB. The basic network options page says VXLAN is Flannel's default backend and wireguard-native enables encryption between nodes. The last two options come from the k3s CIS hardening guide. protect-kernel-defaults makes the kubelet refuse to start if the kernel settings it expects are not in place; the hardening guide lists the sysctl file to create first.

Install the first server with cluster-init, then join the others. The k3s docs pipe the installer to sh; download it and read it first if you prefer, and pin the version.

# k8s-1
curl -sfL https://get.k3s.io -o k3s-install.sh
less k3s-install.sh
sudo INSTALL_K3S_VERSION="v1.36.5+k3s1" sh k3s-install.sh server --cluster-init

# k8s-2 and k8s-3
sudo INSTALL_K3S_VERSION="v1.36.5+k3s1" sh k3s-install.sh server \
  --server https://k8s-1.lab.internal:6443

# check from any server
sudo k3s kubectl get nodes -o wide

The version string above is an example of the k3s naming pattern; check the k3s releases page and pick the current patch of a supported minor. Pinning matters because the install script's default channel moves.

The kubeadm equivalent

With kubeadm you need a stable API endpoint before you start. The kubeadm HA guide asks for a load balancer that forwards TCP 6443 to all control plane nodes, then:

# first control plane node
sudo kubeadm init --control-plane-endpoint "k8s-api.lab.internal:6443" --upload-certs

# the other two, using the join command kubeadm prints
sudo kubeadm join k8s-api.lab.internal:6443 --token ... \
  --discovery-token-ca-cert-hash sha256:... \
  --control-plane --certificate-key ...

The guide notes the uploaded certificates are deleted after two hours. In a homelab, the load balancer is often a virtual IP managed by a small daemon on the control plane VMs, or a separate tiny VM. Whatever you choose, test it with the first node before adding the others, as the guide recommends.

Installing the NVIDIA driver and Container Toolkit, then the device plugin or GPU Operator

There are two working patterns. Choose one per cluster.

On k3s, pattern A is the path the k3s documentation describes, so it is the one this guide uses.

Step 1: driver in the GPU VM

Inside k8s-1, confirm the guest sees the card, then install the driver with the distribution's package manager, which is what NVIDIA's Container Toolkit guide recommends.

lspci -nn | grep -i nvidia
# install a server driver branch from your distribution's packages, e.g. on Ubuntu:
sudo apt install -y nvidia-headless-<BRANCH>-server nvidia-utils-<BRANCH>-server
sudo reboot
nvidia-smi

Replace the branch with one your distribution ships and your CUDA images support; the k3s documentation's own example uses this nvidia-headless-NNN-server package naming. nvidia-smi should list the card. If it does not, go back to the host and confirm vfio-pci owns it and the VM uses q35 and OVMF.

Step 2: NVIDIA Container Toolkit

The Container Toolkit install guide gives an apt repository with a signed keyring and pins every package to one version. At the time of writing it shows version 1.20.1-1.

sudo apt-get update && sudo apt-get install -y --no-install-recommends ca-certificates curl gnupg2

curl -fsSL https://nvidia.github.io/libnvidia-container/gpgkey | \
  sudo gpg --dearmor -o /usr/share/keyrings/nvidia-container-toolkit-keyring.gpg
curl -s -L https://nvidia.github.io/libnvidia-container/stable/deb/nvidia-container-toolkit.list | \
  sed 's#deb https://#deb [signed-by=/usr/share/keyrings/nvidia-container-toolkit-keyring.gpg] https://#g' | \
  sudo tee /etc/apt/sources.list.d/nvidia-container-toolkit.list

sudo apt-get update
export NVIDIA_CONTAINER_TOOLKIT_VERSION=1.20.1-1
sudo apt-get install -y \
  nvidia-container-toolkit=${NVIDIA_CONTAINER_TOOLKIT_VERSION} \
  nvidia-container-toolkit-base=${NVIDIA_CONTAINER_TOOLKIT_VERSION} \
  libnvidia-container-tools=${NVIDIA_CONTAINER_TOOLKIT_VERSION} \
  libnvidia-container1=${NVIDIA_CONTAINER_TOOLKIT_VERSION}

On a kubeadm node with a stock containerd, the guide's next step is sudo nvidia-ctk runtime configure --runtime=containerd, which writes a drop in at /etc/containerd/conf.d/99-nvidia.toml and adds the import to config.toml. On k3s you skip that: k3s manages its own containerd configuration.

Step 3: let k3s find the runtime

The k3s advanced options page says that when the NVIDIA runtime executables are present, k3s adds NVIDIA runtimes to its containerd configuration automatically after a restart, and ships RuntimeClass definitions for them.

sudo systemctl restart k3s
sudo grep nvidia /var/lib/rancher/k3s/agent/etc/containerd/config.toml
kubectl get runtimeclass

You now have a choice the k3s docs spell out. Either every GPU pod sets runtimeClassName: nvidia, or you make NVIDIA the default runtime on the GPU node with the default-runtime option. On a node whose only special job is the GPU, setting it as default avoids a whole class of "the pod ran but saw no GPU" mistakes:

# add to /etc/rancher/k3s/config.yaml on k8s-1 only
default-runtime: nvidia

After the change, run one ordinary CPU pod on that node as well as a GPU pod, to confirm both still start.

Step 4a: the standalone device plugin

The device plugin framework, documented on kubernetes.io, lets a vendor plugin register hardware with the kubelet so the scheduler sees it as an extended resource. NVIDIA's plugin advertises nvidia.com/gpu. Its README shows a Helm install; the version in its example at the time of writing is 0.17.1.

helm repo add nvdp https://nvidia.github.io/k8s-device-plugin
helm repo update
helm upgrade -i nvdp nvdp/nvidia-device-plugin \
  --namespace nvidia-device-plugin --create-namespace \
  --version 0.17.1

The README lists its prerequisites as an NVIDIA driver of version 384.81 or later and the Container Toolkit 1.7.0 or later, both of which you already have.

Step 4b: or the GPU Operator with your pre-installed stack

The GPU Operator adds GPU Feature Discovery labels, the DCGM exporter for metrics, validation pods and a ClusterPolicy you manage declaratively. Its getting started page, for operator v26.7.2, says that if the cluster enforces Pod Security Admission, label the operator namespace as privileged before installing. It also documents verifying the chart's signature with cosign before you install it as an OCI artifact.

kubectl create ns gpu-operator
kubectl label --overwrite ns gpu-operator pod-security.kubernetes.io/enforce=privileged

cosign verify \
  --key https://api.ngc.nvidia.com/v2/catalog/containers/public-key \
  --insecure-ignore-tlog \
  nvcr.io/nvidia/cloud-native-charts/gpu-operator:v26.7.2

helm install --wait gpu-operator -n gpu-operator \
  oci://nvcr.io/nvidia/cloud-native-charts/gpu-operator \
  --version=v26.7.2 \
  --set driver.enabled=false \
  --set toolkit.enabled=false

The two --set flags are the documented scenario for nodes where the driver and toolkit are already installed. The operator's documentation adds that if Node Feature Discovery is already running, you must disable the operator's own copy.

Verify the way the docs do:

kubectl get pods -n gpu-operator
kubectl get clusterpolicy
kubectl get nodes -o json | \
  jq '.items[] | select(.status.allocatable["nvidia.com/gpu"] != null) | {name: .metadata.name, gpus: .status.allocatable["nvidia.com/gpu"]}'

The last command should show k8s-1 with "gpus": "1".

Scheduling GPU workloads, time slicing and a sample CUDA job

How GPU requests work

The Kubernetes GPU scheduling page sets three rules. GPUs are requested in limits. If you also set requests, the two must be equal. You cannot set a GPU request without a limit. Kubernetes uses the limit as the request when only the limit is given.

Run the CUDA vector add test

This is the sample from NVIDIA's getting started page. If you did not set NVIDIA as the default runtime on k3s, add runtimeClassName: nvidia under spec.

# cuda-vectoradd.yaml
apiVersion: v1
kind: Pod
metadata:
  name: cuda-vectoradd
spec:
  restartPolicy: OnFailure
  # runtimeClassName: nvidia
  containers:
  - name: cuda-vectoradd
    image: "nvcr.io/nvidia/k8s/cuda-sample:vectoradd-cuda12.5.0-ubuntu22.04"
    resources:
      limits:
        nvidia.com/gpu: 1

kubectl apply -f cuda-vectoradd.yaml
kubectl logs pod/cuda-vectoradd
# [Vector addition of 50000 elements]
# ...
# Test PASSED
kubectl delete -f cuda-vectoradd.yaml

"Test PASSED" means the whole chain works: firmware IOMMU, vfio-pci on the host, the guest driver, the toolkit, the runtime, the device plugin and the scheduler.

Keep CPU pods off the GPU node, or not

With only three nodes, you probably want CPU work on the GPU node too. If you do not, taint it and let GPU pods tolerate the taint:

kubectl taint nodes k8s-1 nvidia.com/gpu=present:NoSchedule

    tolerations:
    - key: nvidia.com/gpu
      operator: Exists
      effect: NoSchedule

The Kubernetes GPU page also describes labelling nodes by accelerator type and using node selectors when you have more than one kind of card. GPU Feature Discovery, which the operator deploys, adds those labels for you.

Time slicing: sharing one card between pods

A homelab GPU usually sits idle between experiments. Time slicing lets several pods share it. NVIDIA's GPU sharing documentation describes the mechanism: you define a number of replicas per GPU, and each replica can be handed to a different pod. It is clear about the cost: unlike Multi-Instance GPU (MIG), "there is no memory or fault-isolation between replicas."

With the GPU Operator, the documented steps are a ConfigMap, a patch to the ClusterPolicy and, optionally, node labels.

# time-slicing-config-all.yaml
apiVersion: v1
kind: ConfigMap
metadata:
  name: time-slicing-config-all
data:
  any: |-
    version: v1
    flags:
      migStrategy: none
    sharing:
      timeSlicing:
        resources:
        - name: nvidia.com/gpu
          replicas: 4

kubectl create -n gpu-operator -f time-slicing-config-all.yaml
kubectl patch clusterpolicies.nvidia.com/cluster-policy \
  -n gpu-operator --type merge \
  -p '{"spec": {"devicePlugin": {"config": {"name": "time-slicing-config-all", "default": "any"}}}}'

After the device plugin restarts, a node with one GPU advertises four. Three details from NVIDIA's page matter in practice:

• More replicas do not mean more compute. If two pods request one replica each and a third requests two, all three get an equal share of GPU time. Setting failRequestsGreaterThanOne: true makes a request for more than one replica fail instead of silently misleading you.
• The product label changes. With the default renameByDefault: false, the resource stays nvidia.com/gpu and the node's product label gains a -SHARED suffix, which you can use in a node selector to keep exclusive jobs off shared cards.
• Metrics lose per container attribution, and the operator does not watch the ConfigMap for changes.

The device plugin README adds that time slicing and MPS are mutually exclusive.

Per node configs are selected with the nvidia.com/device-plugin.config label, so an exclusive training node can coexist with a shared one.

Storage and networking

Storage: Longhorn or NFS

Longhorn's installation requirements, from its current documentation, include Kubernetes 1.34 or newer, open-iscsi with the iscsid daemon running on every node, an NFSv4 client if you want ReadWriteMany volumes or backups to NFS, mount propagation enabled, and the common tools bash, curl, findmnt, grep, awk, blkid and lsblk. Its best practices page recommends a dedicated disk for Longhorn data.

# on every node
sudo apt install -y open-iscsi nfs-common
sudo systemctl enable --now iscsid

# then install Longhorn with Helm
helm repo add longhorn https://charts.longhorn.io
helm repo update
helm install longhorn longhorn/longhorn --namespace longhorn-system --create-namespace

Longhorn's own docs have a k3s specific page; read it before installing on k3s. With three nodes, keep the default of three replicas per volume so a node can fail without data loss, and remember that a replica rebuild after a reboot copies data across the network.

For NFS, the Kubernetes SIGs nfs-subdir-external-provisioner creates a subdirectory per volume on an existing NFS export.

A 20 GB model on a three replica Longhorn volume uses 60 GB of raw disk; for read mostly weights, NFS or a local path volume on the GPU node is often the better trade.

Networking: CNI, VLANs and MetalLB

CNI. k3s ships Flannel: VXLAN works anywhere, wireguard-native encrypts node traffic, and host-gw needs one layer 2 segment. k3s adds its own network policy controller. With kubeadm, pick a CNI that enforces NetworkPolicy if anyone else will run workloads here.

VLANs. A simple, defensible split for three nodes:

On Proxmox, make the bridge VLAN aware and set tag= on each VM network device, as shown in the VM creation step. Keep etcd ports 2379 and 2380 and the kubelet port 10250 off the home network entirely.

MetalLB. Without a cloud, a Service of type LoadBalancer stays pending forever. MetalLB fills that gap. Its installation page, at version 0.16.1, gives a native manifest. If kube-proxy runs in IPVS mode, the page says you must set strictARP: true first.

kubectl apply -f https://raw.githubusercontent.com/metallb/metallb/v0.16.1/config/manifests/metallb-native.yaml

Then give it a pool and advertise it on layer 2, which needs no router configuration. The configuration page documents both resources.

apiVersion: metallb.io/v1beta1
kind: IPAddressPool
metadata:
  name: homelab-pool
  namespace: metallb-system
spec:
  addresses:
  - 10.0.30.240-10.0.30.250
---
apiVersion: metallb.io/v1beta1
kind: L2Advertisement
metadata:
  name: homelab-l2
  namespace: metallb-system
spec:
  ipAddressPools:
  - homelab-pool

Exclude that range from your router's DHCP scope. MetalLB also supports BGP mode if your router speaks BGP.

Hardening: RBAC, Pod Security Admission, node patching and image provenance

The NSA and CISA Kubernetes Hardening Guidance (the 1.2 revision is dated August 2022) centres on scanning containers and pods for vulnerabilities and misconfigurations, running with the least privilege possible, and separating networks. The Kubernetes project's own summary of the guide highlights non-root containers, immutable container filesystems and scanning images at build and deploy time.

RBAC

The Kubernetes RBAC good practices page gives the rules that matter most for a small cluster:

• Assign permissions at the namespace level with RoleBindings, not ClusterRoleBindings, where you can.
• Avoid wildcard permissions, especially on all resources.
• Do not use cluster-admin except where it is specifically needed, and avoid adding users to system:masters, whose access cannot be revoked by removing bindings.

Keep the k3s admin kubeconfig (/etc/rancher/k3s/k3s.yaml) to yourself, and give anyone else a namespace and a Role:

apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata:
  name: gpu-jobs
  namespace: ml
rules:
- apiGroups: ["", "batch"]
  resources: ["pods", "pods/log", "jobs", "configmaps", "persistentvolumeclaims"]
  verbs: ["get", "list", "watch", "create", "delete"]

Pair it with a ResourceQuota on requests.nvidia.com/gpu so one namespace cannot take every replica.

Pod Security Admission

Pod Security Admission has been stable since Kubernetes 1.25. It enforces the three Pod Security Standards (privileged, baseline and restricted) in three modes: enforce rejects violating pods, audit records them in the audit log, and warn shows a warning to the user.

The k3s CIS hardening guide gives a cluster wide configuration that defaults every namespace to restricted and exempts kube-system, passed with kube-apiserver-arg: admission-control-config-file=/var/lib/rancher/k3s/server/psa.yaml. Then relax only the namespaces that need it. GPU system components legitimately need privileges, which is why NVIDIA's docs label the gpu-operator namespace privileged. Longhorn also needs to run as root. Your own workloads do not.

kubectl label ns ml pod-security.kubernetes.io/enforce=restricted \
  pod-security.kubernetes.io/warn=restricted

A CUDA container runs fine as non-root with no added capabilities. If a GPU image only works as root, treat that as a defect in the image, not a reason to loosen the namespace.

Node and hypervisor patching

Patch three layers on a schedule, not when you remember:

A guest to host escape in KVM or QEMU turns a compromised pod into a compromised node; our Proxmox VE patch guide for PwnEscape walks through checking and patching that layer. The CIS Kubernetes Benchmark, available free from CIS for non commercial use, covers the Kubernetes layer; the k3s hardening guide maps its controls to k3s flags.

Image provenance

Everything in this cluster arrives as a container image or a Helm chart: k3s system images, the GPU Operator, the device plugin, Longhorn, MetalLB, your CUDA images and whatever notebook image you pull next week. Three habits help:

• Verify signatures where the publisher provides them. Kubernetes documents verifying its own signed release artifacts with cosign, and NVIDIA documents a cosign check for the GPU Operator chart, as shown above.
• Pin by version, and preferably by digest. Every install command in this guide pins a version. A floating tag can change under you.
• Know what is inside the images. CUDA base images carry a full Linux userland. Keep an SBOM for the images you run and check it when advisories land.

If you learned Kubernetes on a laptop first, the same supply chain thinking applies to the tools themselves; see how to install kubectl and minikube safely.

Backups and upgrades

etcd snapshots

On k3s, scheduled snapshots are on by default for embedded etcd. The etcd snapshot page documents the defaults: every 12 hours (cron 0 */12 * * *), keeping 5 snapshots per server. On demand snapshots have no retention and must be pruned by hand.

# take one before any change
sudo k3s etcd-snapshot save --name pre-upgrade
sudo k3s etcd-snapshot ls

Copy snapshots off the cluster. k3s can upload them to S3 compatible storage; the docs note that with three etcd nodes each cycle uploads three objects, so to keep two cycles you set the S3 retention to about six.

Restoring on k3s uses --cluster-reset with a restore path on one server, after stopping k3s on all of them. The docs say you can restore with the same or a newer k3s version that respects the version skew policy, but never with an older version you could not have upgraded from.

# on all servers
sudo systemctl stop k3s
# on k8s-1 only
sudo k3s server --cluster-reset \
  --cluster-reset-restore-path=/var/lib/rancher/k3s/server/db/snapshots/pre-upgrade-...
sudo systemctl start k3s
# on k8s-2 and k8s-3: follow the docs to remove the old etcd data directory, then start k3s

The k3s documentation has the exact steps for the other servers; follow them rather than improvising, because rejoining with stale etcd data is how a restore turns into a split cluster.

On kubeadm, the Kubernetes etcd page shows etcdctl snapshot save and recommends etcdutl to check the snapshot, because etcdctl snapshot status was deprecated in etcd 3.5.

ETCDCTL_API=3 etcdctl --endpoints https://127.0.0.1:2379 \
  --cacert /etc/kubernetes/pki/etcd/ca.crt \
  --cert /etc/kubernetes/pki/etcd/server.crt \
  --key /etc/kubernetes/pki/etcd/server.key \
  snapshot save /var/backups/etcd-$(date +%F).db
etcdutl --write-out=table snapshot status /var/backups/etcd-$(date +%F).db

An etcd snapshot restores cluster objects, not volume data; back up Longhorn volumes and Proxmox VMs separately, and test a restore at least once.

Rolling Kubernetes upgrades

Both distributions follow the same rule: no skipping minor versions. The kubeadm upgrade page says you can go from 1.36.x to 1.37.x, or from 1.37.x to a newer 1.37 patch, and that skipping minor versions is unsupported. The k3s manual upgrade page says the version skew policy applies and that you upgrade servers first, one at a time, then agents.

k3s, per server:

sudo k3s etcd-snapshot save --name pre-1.37
kubectl drain k8s-3 --ignore-daemonsets --delete-emptydir-data
# on k8s-3
sudo INSTALL_K3S_VERSION="v1.37.1+k3s1" sh k3s-install.sh server
kubectl uncordon k8s-3
# wait for Ready and for Longhorn volumes to be healthy, then k8s-2, then k8s-1

kubeadm, per the upgrade page: unhold and upgrade the kubeadm package, run kubeadm upgrade plan and kubeadm upgrade apply v1.37.x on the first control plane node, kubeadm upgrade node on the others, then drain each node, upgrade kubelet and kubectl, restart the kubelet and uncordon.

Do the GPU node last, and rerun the vector add test after it returns. Driver and toolkit upgrades are a separate change on a separate day, so that if something breaks you know which change did it.

Power, noise and cost math

The formula

Annual energy in kWh = average watts × 24 × 365 ÷ 1000. Annual cost = kWh × price per kWh.

The US Energy Information Administration's Electric Power Monthly gives an average residential price of 18.31 cents per kWh for July 2026, up from 17.45 cents in July 2025. Use your own bill's rate if you have it; prices vary widely by state and season.

Worked example 1: three idle nodes

Say each node draws 40 W at idle with its Kubernetes VM running and nothing heavy scheduled.

• Average draw: 3 × 40 W = 120 W
• Energy: 120 × 24 × 365 ÷ 1000 = 1,051.2 kWh per year
• Cost at 18.31 cents: 1,051.2 × 0.1831 = about 192.47 US dollars per year

That is about 16 dollars a month before the GPU does anything. Measure your own idle draw with a plug in power meter; it is often higher than the parts list suggests because of disks, fans and network gear.

Worked example 2: adding the GPU

Add a GPU that idles at 20 W and draws 300 W under load, busy 25 percent of the time.

• GPU average: (0.25 × 300 W) + (0.75 × 20 W) = 75 W + 15 W = 90 W
• Cluster average: 120 W + 90 W = 210 W
• Energy: 210 × 24 × 365 ÷ 1000 = 1,839.6 kWh per year
• Cost at 18.31 cents: 1,839.6 × 0.1831 = about 336.83 US dollars per year

Run the GPU flat out around the clock instead and its share alone is 300 × 8,760 ÷ 1000 = 2,628 kWh, about 481 dollars a year at the same price. Duty cycle matters more than the number on the box.

Worked example 3: break even against renting

If a rented GPU of similar class costs R dollars per hour, and your hardware cost H dollars and lasts Y years, then your hourly cost at U hours of use per year is roughly (H ÷ Y + annual electricity) ÷ U. With H = 1,500, Y = 3, electricity from example 2 of 337, and U = 2,190 hours (25 percent duty), that is (500 + 337) ÷ 2,190 = about 0.38 dollars per hour. Compare that to R. Halve your usage and the hourly cost nearly doubles, because the hardware cost does not shrink. These numbers are illustrative inputs, not market prices; plug in your own.

Noise and heat

Every watt becomes heat in the room: a 210 W cluster is a small heater running all year. For noise, prefer larger, slower fans and quiet idle fan curves; data centre rack servers are usually far louder than desktop class machines at the same load. Never power off a second server to save money: with three servers, losing two loses quorum.

Common mistakes

• Passing the GPU without its audio function. Pass 01:00 (all functions), not 01:00.0, or the group is not fully owned by VFIO and the VM refuses to start.
• Choosing i440fx or SeaBIOS by habit. PCIe passthrough is only available on q35, and OVMF gives the best compatibility for modern cards.
• Forgetting the MMIO window on a large VRAM card. "BAR can't reserve" in the guest log is the symptom; cpu: host or the documented firmware argument is the fix.
• Running two Kubernetes servers. Two servers tolerate zero failures. Use one or three.
• Leaving ServiceLB on and installing MetalLB. Disable ServiceLB on every k3s server first.
• Running nvidia-ctk runtime configure on k3s. k3s manages its own containerd config and detects the runtime itself.
• Pods that run but see no GPU on k3s. Either set runtimeClassName: nvidia or make NVIDIA the node's default runtime.
• Requesting two time sliced replicas and expecting double speed. Each pod gets an equal share regardless of how many replicas it asked for.
• Skipping a minor version in an upgrade, or upgrading the GPU stack and Kubernetes on the same day.
• Never testing a restore. The first restore should not be the one you need.

Where Safeguard fits, honestly

Most of this guide is outside what Safeguard does. Safeguard does not configure hypervisors, manage devices, check firmware or IOMMU settings, install GPU drivers or operate Kubernetes clusters. If your VM will not start because of an IOMMU group, the steps above are what fix it.

The honest link is the software you run on top. A cluster like this one pulls dozens of container images and Helm charts, and your own projects pull language dependencies into the images you build. Safeguard's CLI can scan a repository, a container image or an SBOM file (safeguard scan --dir ., --image <name>, --sbom <file>), match components against vulnerability databases, and the platform shows which projects contain a given component and version, so when an advisory lands for a library inside a CUDA base image or a notebook image, you can find where you use it. Policies and gates can stop a known vulnerable image from being built in CI. Safeguard does not enforce anything inside the cluster itself; admission control remains Pod Security Admission and whatever controllers you choose.

FAQ

Can you do GPU passthrough to a Kubernetes node on Proxmox?

Yes. Bind the GPU to vfio-pci on the Proxmox host, pass it to a q35, OVMF VM with hostpciN and pcie=1, then run the Kubernetes node inside that VM with the NVIDIA driver, Container Toolkit and device plugin. The VM cannot live migrate while it holds the card.

How do I check IOMMU groups in Proxmox?

Run pvesh get /nodes/NODENAME/hardware/pci --pci-class-blacklist "" and read the iommugroup column. The GPU should share its group only with its own functions, such as HDMI audio, or its root port.

Is k3s or kubeadm better for a homelab?

k3s has fewer moving parts: one binary with a runtime, CNI and load balancer included, embedded etcd for HA and an official CIS hardening guide. kubeadm is closer to how most production clusters are built and is better if learning upstream tooling is the goal. Both are conformant Kubernetes.

Why do I need three control plane nodes?

etcd needs a majority to accept writes. Three servers have a quorum of two, so they survive one failure. Two servers survive none, and four servers still survive only one.

Should I use the GPU Operator or just the device plugin?

The device plugin alone is enough to schedule nvidia.com/gpu. The GPU Operator adds feature discovery labels, DCGM metrics, validation and declarative configuration such as time slicing. On k3s with a pre-installed driver and toolkit, install the operator with driver.enabled=false and toolkit.enabled=false.

How do I share one GPU between multiple pods?

Enable time slicing in the device plugin configuration and set the number of replicas per GPU. Each replica can go to a different pod. There is no memory or fault isolation between replicas, so one pod running out of GPU memory can affect the others.

Can I use Longhorn on a 3 node homelab?

Yes; three nodes is Longhorn's minimum recommended layout, with 4 vCPUs and 4 GiB of RAM per node and SSD or NVMe disks. Install open-iscsi and an NFSv4 client on each node first.

How do I back up a k3s cluster?

k3s takes etcd snapshots every 12 hours by default and keeps 5 per server. Take an on demand snapshot with k3s etcd-snapshot save before changes, copy snapshots off the cluster, and back up volume data separately with Longhorn backups or your storage system.

Fix it without waiting for upstream

Passthrough, drivers and etcd are fixed by their own projects, and the steps above cover those. But every image this cluster runs, from the GPU Operator to your own model API, ships open source components whose fixes you do control, and a rebuilt node pulls all of them again, so these habits still matter for a homelab.

Whatever the vulnerability, most of the risk sits in two gaps. The first is the time between a fix being published and that fix actually running in your production builds. The second, and worse, is the time before any fix exists at all, while a maintainer works out a patch and a release schedule. Waiting through both gaps is a habit, not a requirement. Safeguard gives teams three ways to close them.

1. Autonomous remediation: pull requests, not tickets

When a fixed version exists, Safeguard's Griffin models open a tested, policy gated pull request that moves the affected dependency to the safe version. You can run it on one repository or in bulk across every project that carries the vulnerable version. Remediation covers eight categories, including dependencies, licenses, security posture and findings. Reachability analysis ranks the vulnerable code paths your application actually calls first, so the first pull requests you merge are the ones that matter most. Autonomous remediation ships switched off, with every category set to Manual, so your team decides how much runs on its own and what still needs a human approval.

2. Gold Open Source: the safe version in seconds, free

gold.safeguard.sh is free and needs no login. Search any CVE or package to see affected and fixed versions, CVSS, EPSS, CISA KEV status and a recommended safe version, or use the "Am I affected?" checker. The same data is available through a free read API, README status badges and RSS threat feeds. The safeguard-sh/gold-scan-action GitHub Action can fail a build on a known exploited vulnerability, on any known vulnerability, or never, so a version you have just removed cannot quietly come back in next week's merge.

3. Gold Registry: start from components that are already fixed

registry.safeguard.sh provides hardened, drop in replacements for upstream packages and container images. Every Gold artifact is held continuously to zero exploitable critical or high CVEs, zero malware and a completed license check, with SLSA L3 or higher provenance, a Sigstore signature and a public evidence page you can hand to an auditor. An artifact loses its Gold badge if it regresses. Teams can point their own package directory at the registry so hardened equivalents resolve without changing how developers install anything.

When upstream has no fix yet, or will not ship one, Safeguard first works to get the fix accepted upstream. If the issue is severe or upstream will not move, Safeguard builds a fork, fixes the vulnerability there and publishes a hardened Gold build, so you are not stuck on a maintainer's release schedule. You can also request a hardened, compatibility tested build of a specific package. This fork and patch flow is in early access today, so ask which packages it covers before you plan around it.

One honest limit: when the vulnerable code lives in a vendor appliance, a hosted service or an operating system runtime you do not build yourself, the fix still has to come from that vendor or distribution. The steps above apply to the open source dependencies and container images you build and ship, which is where most teams carry the largest and least tracked share of their exposure.

See how Safeguard would help

None of these demos configure a hypervisor or a GPU. They show how Safeguard handles the software you run on the cluster.

The first demo is a real Safeguard CLI session: asked to fix a lodash CVE, the assistant reads the finding and advisory, asks before each command, edits package.json, runs npm install and npm test with your approval, and pushes nothing. The same flow applies to the services you build into images for this cluster, such as a model API or a notebook extension, when one of their dependencies has a fixed version available.

Click through it at your own pace, or open it full screen.

The second demo sets up the local runner on your own machine, approves it with a device code, then scans from the terminal. In a homelab, that means scanning repositories and images from your own hardware, for example on the workstation where you build the CUDA images you push to the cluster.

Click through it at your own pace, or open it full screen.

The third demo connects Claude to the Safeguard MCP server with OAuth and chooses which tools Claude can use. Once your projects and image SBOMs are in Safeguard, the same connection lets you ask which of them contain a given component and version when an advisory names a library in your GPU images.

Click through it at your own pace, or open it full screen.

To try the CLI, download Safeguard and follow the CLI documentation.

Quick check

Sources

• Proxmox VE Administration Guide: QEMU/KVM virtual machines, PCI(e) passthrough
• Proxmox VE wiki: PCI Passthrough
• Linux kernel documentation: VFIO, Virtual Function I/O
• Kubernetes: Creating Highly Available Clusters with kubeadm
• Kubernetes: Installing kubeadm
• Kubernetes: Upgrading kubeadm clusters
• Kubernetes: Operating etcd clusters for Kubernetes
• Kubernetes: Releases
• Kubernetes: Device Plugins
• Kubernetes: Schedule GPUs
• Kubernetes: Pod Security Admission
• Kubernetes: Role Based Access Control Good Practices
• Kubernetes: Verify Signed Kubernetes Artifacts
• Kubernetes blog: A Closer Look at NSA/CISA Kubernetes Hardening Guidance
• K3s: High Availability Embedded etcd
• K3s: Requirements
• K3s: Advanced Options (NVIDIA Container Runtime)
• K3s: Networking Services
• K3s: Basic Network Options
• K3s: etcd-snapshot
• K3s: Manual Upgrades
• K3s: CIS Hardening Guide
• NVIDIA: Installing the NVIDIA Container Toolkit
• NVIDIA: GPU Operator Getting Started
• NVIDIA: Time-Slicing GPUs in Kubernetes
• NVIDIA k8s-device-plugin on GitHub
• Longhorn: Install
• Longhorn: Best Practices
• MetalLB: Installation
• MetalLB: Configuration
• Kubernetes SIGs: nfs-subdir-external-provisioner
• CIS Kubernetes Benchmarks
• US EIA Electric Power Monthly, Table 5.6.A

Last reviewed: 23 October 2026. Kubernetes, k3s, NVIDIA and Proxmox release versions change often; check each project's documentation for the current version before running any command above.