AI accelerators are specialized processors — GPUs, TPUs, and NPUs — built to run the massively parallel math behind model training and inference, and they carry a security surface that most application teams never account for: shared memory that is not always cleared between tenants, sprawling driver stacks pulled in as dependencies, and firmware you neither wrote nor can easily inspect. The performance story is well told. The security story is not, and it matters more every time a workload moves onto shared or rented accelerator hardware.
So, what are AI accelerators in practical terms? They are chips optimized for the linear algebra that dominates machine learning — dense matrix multiplication, mostly — trading the general-purpose flexibility of a CPU for enormous parallel throughput. NVIDIA and AMD GPUs, Google's TPUs, and the NPUs baked into modern phones and laptops all fall under the term. Wherever a model runs at scale, an accelerator is doing the heavy lifting.
What are AI accelerators, and why the security model differs
A CPU runs a handful of powerful cores. An AI accelerator runs thousands of simpler ones, plus tiers of specialized memory designed for throughput rather than isolation. That architectural choice is the root of the security difference. Isolation between workloads — the thing your operating system and hypervisor work hard to guarantee on a CPU — is weaker and younger on accelerators, because the hardware was designed for a world where one trusted user owned the whole device.
That assumption broke the moment accelerators became multi-tenant. Cloud GPU instances, shared inference clusters, and rented training capacity all put untrusted workloads on the same physical silicon. The security model has been catching up ever since, and the gaps are real.
GPU memory leakage: the LeftoverLocals lesson
The clearest example of the risk is LeftoverLocals, tracked as CVE-2023-4969 and disclosed by Trail of Bits in January 2024. The flaw was simple and damning: affected GPUs did not clear local memory between kernel executions, so a malicious kernel could read the "leftovers" a previous workload left behind.
For AI workloads that meant an attacker could recover another process's model data — the researchers demonstrated reconstructing an LLM's responses on an AMD Radeon RX 7900 XT running llama.cpp, pulling as much as 181MB per query, enough to rebuild the output with high accuracy. The demonstration took roughly ten lines of OpenCL. Affected vendors included Apple, AMD, Qualcomm, and Imagination; NVIDIA and Arm confirmed their devices were not impacted.
The lesson generalizes past this one CVE: on shared accelerator hardware, uninitialized memory is a cross-tenant data exposure risk, and you cannot assume the hardware zeroes anything for you. If you run sensitive inference on shared GPUs, that is a threat you have to reason about explicitly.
The driver and runtime stack is a dependency you own
An AI accelerator is useless without its software stack — CUDA and cuDNN for NVIDIA, ROCm for AMD, the vendor's runtime libraries, plus the framework layers (PyTorch, TensorFlow, JAX) that sit on top. Every one of those is code running with high privilege close to the hardware, and every one is a dependency in your supply chain even though you did not write a line of it.
These stacks are large, native, and update on the vendor's cadence, not yours. A vulnerability in a GPU driver or runtime library is not a niche concern — it is privileged code with direct hardware access. Treat the accelerator software stack the way you treat any other dependency: inventory the exact versions in your container images, track advisories against them, and rebuild when the vendor patches. A software composition analysis pass over your ML container images will surface the framework and library versions you are actually shipping, which is often different from what your Dockerfile claims after a base-image update.
Firmware and supply chain trust
Below the driver sits firmware — the accelerator's own microcode and management controllers — which is almost entirely opaque to the workloads running above it. You are trusting the vendor's build and update pipeline, and you generally cannot audit what runs there. That is an acceptable trust boundary for most teams, but it is a trust boundary, and it should be a conscious one.
The practical controls are procedural: source accelerators and firmware updates only through vendor channels, verify update signatures where the platform supports it, and keep firmware current, because firmware-level fixes for hardware flaws (including memory-clearing behavior) are a common remediation path. For confidential AI workloads, look at the confidential-computing features some accelerators now ship — memory encryption and attestation that let you verify what code is running before you hand it a model or data.
Practical defenses for AI workloads on accelerators
You will not redesign the silicon, but you control the layers around it:
- Do not share accelerators across trust boundaries for sensitive workloads. If cross-tenant memory leakage is in your threat model, dedicated hardware or strict scheduling isolation is the answer, not hoping the driver clears memory.
- Pin and scan the full stack. Lock CUDA/ROCm, cuDNN, and framework versions in your images, scan them continuously, and patch on vendor advisories rather than on a fixed calendar.
- Zero your own buffers. Where your code allocates accelerator memory directly, clear it before and after handling sensitive data instead of assuming the runtime does.
- Attest before you trust. On platforms that support confidential computing, use attestation to confirm the firmware and runtime state before loading proprietary models or regulated data.
- Watch the runtime. Anomalous kernel launches, unexpected memory allocation patterns, and unauthorized processes touching the device are all observable and worth alerting on.
None of these are exotic. They are the same supply-chain and isolation disciplines you already apply elsewhere, pointed at a layer most teams have historically ignored. The Academy has deeper material on threat-modeling ML infrastructure if you want to formalize this into a review process.
FAQ
What are AI accelerators in one sentence?
They are chips — GPUs, TPUs, and NPUs — specialized for the parallel matrix math that dominates machine learning, delivering far higher throughput on model training and inference than a general-purpose CPU can.
Are NVIDIA GPUs vulnerable to LeftoverLocals?
No. During the coordinated disclosure of CVE-2023-4969, NVIDIA confirmed its devices were not impacted, as did Arm. The affected vendors were Apple, AMD, Qualcomm, and Imagination.
Do I need to worry about accelerator security if I use a managed AI service?
Less directly — the provider owns the hardware, firmware, and isolation. But you still own your model and data handling, and you should confirm the provider's tenant isolation guarantees and whether they offer confidential-computing options for sensitive workloads.
How do I know which accelerator libraries are in my container?
Scan the built image, not the Dockerfile. Software composition analysis inspects the actual installed CUDA, ROCm, cuDNN, and framework versions, which frequently drift from what the Dockerfile specifies once base images update underneath you.