Yes, by default. Docker's guidance is that if a service can run without privileges, you should use USER to switch to a non-root user. Training jobs rarely need root, and they often run untrusted code from notebooks and dependencies, so limiting what that code can do is a cheap, effective step.
Why it matters for ML jobs
A training container usually pulls many packages, loads model files, and runs code a researcher just wrote. If any of it turns out to be hostile, root inside the container gives the attacker more room: writing to system paths, installing tools, and reaching more of the mounted volumes. Non-root does not make a container safe by itself, but it removes easy options.
Set the user in the Dockerfile
Docker's build best practices say to create the user and group first and to assign explicit numeric IDs, because automatically assigned IDs can change across rebuilds. They also advise against sudo.
FROM python:3.12-slim
RUN groupadd -g 10001 trainer && useradd -u 10001 -g trainer -m trainer
WORKDIR /home/trainer/app
COPY --chown=trainer:trainer . .
USER trainer
The explicit ID 10001 matters when you mount volumes, since file ownership on the host is numeric.
Tighten the Kubernetes pod
If jobs run on Kubernetes, the security context adds controls. Two are documented in the Kubernetes tasks page:
- allowPrivilegeEscalation controls whether a process can gain more privileges than its parent, for example through a setuid binary. The page says it defaults to true when not defined, so set it to false.
- readOnlyRootFilesystem mounts the container's root file system as read only.
securityContext:
allowPrivilegeEscalation: false
readOnlyRootFilesystem: true
The page also notes that allowPrivilegeEscalation false cannot be combined with a privileged container or one with the CAP_SYS_ADMIN capability. That is a useful test: if your job needs either, ask why.
What breaks, and how to fix it
| Symptom | Likely cause | Fix |
|---|---|---|
| Permission denied writing a cache | Cache path owned by root | Point the cache at a writable volume or the user's home |
| Cannot write to model directory | Volume ownership | Match the numeric user ID on the volume |
| Read only file system error | Library writes to temp paths | Mount a small writable temp volume |
| GPU not visible | Runtime or device setup | Check the toolkit setup, not user privileges |
A read only root file system will reveal every place your code writes. Treat that as a free inventory.
A rollout plan
- Pick one training image and add the USER line.
- Run a short job and fix write paths.
- Add the pod security settings in a test namespace.
- Make non-root the template default, with a documented exception process.
Pair this with pinning base images by digest.
Safeguard scans container images and dependencies, and policies can gate a build that violates your rules. Try the free scan at gold.safeguard.sh/scan.
Quick check
Does non-root stop all container escapes?
No. It reduces what code can do inside the container. Keep the runtime and kernel updated as well.
Do GPUs require root?
Not as a rule. GPU access depends on the toolkit and device setup, so troubleshoot that rather than granting root.
Sources
Last reviewed: 19 October 2026.
