The most useful concerns about AI, from a security standpoint, are concrete and near-term: prompt injection, data leakage, poisoned model supply chains, and decisions no one can explain. Once you set aside the abstract debates, these concerns about AI are the ones that show up in real systems this year, and each has a mitigation you can actually implement.
AI systems fail differently from traditional software. A SQL query either runs or errors; a language model produces plausible output even when it is completely wrong or has been manipulated. That gap between "looks fine" and "is fine" is where most of the risk lives.
Prompt injection is the new injection
Prompt injection is the AI-era cousin of SQL injection: untrusted input reaches the model and changes what it does. It comes in two forms. Direct injection is a user typing "ignore your previous instructions" into a chat box. Indirect injection is subtler and more dangerous: malicious instructions hidden in a document, web page, or email that your AI later reads and treats as commands.
Imagine a support agent that summarizes incoming tickets and can call internal tools. If an attacker plants text in a ticket that reads like an instruction to exfiltrate data through one of those tools, the model may follow it. The concerning part is that there is no clean separation between "data" and "instructions" in a prompt the way there is between data and SQL keywords in a parameterized query.
The defenses are architectural rather than a single filter. Treat all model output as untrusted. Never give a model direct access to a dangerous capability without a human or a deterministic policy check in between. Constrain tool permissions tightly, and log every tool call so you can audit what the model actually did.
Data leakage flows in two directions
The first direction is training and prompt data leaking out. If you send sensitive customer data to a third-party model, you need to know whether it is retained, whether it trains future models, and where it is processed. For regulated data this is a residency and contractual question, not just a technical one.
The second direction is the model leaking data it should not. Retrieval-augmented systems that pull from internal documents can surface information the current user is not authorized to see, because the retrieval step often bypasses the access controls the original application enforced. The fix is to apply authorization at retrieval time, filtering the document set to what the requesting user can access before it ever reaches the model.
The model supply chain is a real supply chain
Teams have spent years learning to scan open-source software dependencies for vulnerabilities. Models are now dependencies too, and the same instincts apply. A model pulled from a public hub can carry risk in several forms: malicious code embedded in an unsafe serialization format, a backdoor trained into the weights, or simply an unknown provenance you cannot vouch for to an auditor.
Older model formats that rely on Python pickle serialization can execute arbitrary code on load, which means downloading and running a model file is closer to running an untrusted executable than opening a data file. Prefer safe serialization formats, pin models by hash, and record model provenance in your inventory the same way you record software components. The discipline that software composition analysis brought to packages, described in our SCA overview, maps cleanly onto model artifacts.
Accountability and the black box
When an AI system denies a loan, flags a transaction, or triages a security alert, someone will eventually ask why. If the answer is "the model decided," that is a problem for compliance, for customer trust, and for debugging. This concern is less about a breach and more about operating a system you cannot fully reason about.
Practical accountability does not require perfect explainability. It requires logging inputs and outputs, versioning the model and its prompts, and being able to reproduce a decision after the fact. When you can replay what the system saw and produced, you can investigate a bad outcome instead of shrugging at it.
Autonomy amplifies every other risk
The shift from AI that answers questions to AI that takes actions changes the stakes. An agent that can open pull requests, run commands, or move money turns a prompt injection from an embarrassing wrong answer into a real-world action. The mitigations compound accordingly: least-privilege tool access, human approval gates for consequential actions, rate limits, and hard boundaries the agent cannot cross regardless of what it is told.
The governing principle is the same one that has always worked in security. Assume the component can be manipulated, and design so that manipulation cannot cause disproportionate harm. An AI agent should be sandboxed and permission-scoped exactly like any other automated actor in your system.
Turning concern into practice
The teams handling AI risk well are not the ones with the most sophisticated fears. They are the ones treating AI systems as software that needs the same rigor as everything else: input validation, least privilege, dependency provenance, audit logging, and a way to reproduce what happened. If you want a structured path through these controls, our academy covers securing AI-enabled applications.
FAQ
What is the biggest security concern with AI right now?
Prompt injection, especially the indirect kind where malicious instructions hide in content the model later reads. It is hard to fully prevent because models do not cleanly separate instructions from data, so the defense is architectural: treat model output as untrusted and gate any dangerous capability behind a check.
Can AI models contain malware?
Yes. Model files in older serialization formats can execute arbitrary code when loaded, and weights can carry trained-in backdoors. Prefer safe serialization formats, verify provenance, pin models by hash, and scan model artifacts the way you scan software dependencies.
How do I stop an AI system from leaking sensitive data?
Control both directions. For data you send out, verify retention and residency terms with your provider. For data the model might surface, apply authorization at retrieval time so users only ever see documents they are permitted to access.
Do AI agents need special security controls?
Yes. Because agents take actions rather than just producing text, give them least-privilege tool access, require human approval for consequential operations, rate-limit them, and sandbox them. Treat an agent like any other automated actor with restricted permissions.