Security in AI systems builds on the same foundations as traditional application security — input validation, least privilege, dependency hygiene — but adds attack surfaces that didn't exist in conventional software: prompts that double as both data and instructions, training pipelines with their own supply chain, and models whose behavior can't be fully predicted from source code review alone. Teams that assume their existing AppSec program already covers AI systems are usually missing at least one of these categories entirely.
What's genuinely new about security in AI systems versus traditional software?
What's genuinely new is that the model's input and its instructions often flow through the same channel, which traditional software rarely allows. In a conventional web application, user input is data — it gets validated, sanitized, and treated as untrusted, but it's structurally separate from the application's own logic. In a large language model application, a user's prompt and the system's instructions both arrive as natural-language text, which means a sufficiently crafted input can cause the model to treat malicious instructions embedded in user content as if they came from the system itself. This is the mechanism behind prompt injection, and it doesn't have a clean equivalent in traditional AppSec the way SQL injection or XSS do — there's no simple parameterization fix, because the model's entire function is interpreting flexible natural language.
How does the AI supply chain differ from a traditional software supply chain?
The AI supply chain adds layers that traditional software composition analysis doesn't cover: training data provenance, model weights sourced from a third-party hub, and fine-tuning datasets that may themselves be poisoned or subtly biased in ways that don't show up until the model is deployed and behaving unexpectedly in production. A traditional SCA tool can tell you a Python package has a known CVE; nothing comparable yet gives you the same confidence about a downloaded model checkpoint's training provenance. Organizations pulling pretrained models from public hubs are, in effect, running third-party code they can't fully audit — the model's behavior is the artifact, and reviewing billions of weights isn't equivalent to reviewing source code.
Do existing frameworks like the OWASP Top 10 apply to AI systems?
Existing frameworks apply as a foundation, but they've needed a dedicated extension because AI systems introduce risk categories the original OWASP Top 10 never anticipated. The OWASP Top 10 for Large Language Model Applications, first published in 2023 and updated since, covers prompt injection, insecure output handling, training data poisoning, and excessive agency — the risk that an AI system with tool access takes an action beyond what a user actually intended or authorized. That last category matters more every year as AI systems move from answering questions to executing actions — sending emails, modifying records, calling internal APIs — because a manipulated model with real permissions can cause real damage, not just generate a wrong answer.
Does this change how vulnerability scanning and remediation should work?
It changes what needs scanning, more than how scanning fundamentally works. Code that calls an LLM API, handles its output, and grants it tool access is still ordinary application code that static and dynamic analysis can and should cover — insecure output handling, for instance, is really just insufficient output validation wearing a new name. What's added on top is model-specific testing: red-teaming prompts for injection resistance, checking whether the application blindly trusts model output as safe to execute or display, and auditing what permissions an AI agent actually holds versus what it strictly needs. Safeguard's SAST/DAST scanning covers the application code wrapping an AI integration — the part traditional tooling was always built to catch — while model-specific red-teaming remains a distinct, additional practice most teams still need to build out separately.
Who should own AI security on a team that doesn't have a dedicated AI security function yet?
Ownership should sit with whoever already owns application security, extended to cover the new categories, rather than spun off as a completely separate function too early. The underlying skills — threat modeling, understanding attacker incentives, secure design review — transfer directly; what's needed is additional context on prompt injection patterns and model-specific failure modes, which is learnable without a from-scratch security hire. Teams building this out benefit from structured material specifically on the LLM-era OWASP categories rather than assuming general security experience automatically covers it.
FAQ
Is prompt injection the same as jailbreaking?
They're related but distinct. Jailbreaking typically refers to getting a model to bypass its own content policies and produce output it was trained to refuse. Prompt injection is broader — it's getting a model to follow attacker-supplied instructions embedded in otherwise normal input, which can happen even without any content-policy violation at all.
Can traditional input validation stop prompt injection?
It helps but doesn't fully solve it, because the "malicious" content in a prompt injection attack is often grammatically and semantically identical to legitimate user input — there's no reliable syntactic signature to filter on the way there is for something like a SQL injection payload.
Does using a well-known model provider eliminate AI supply chain risk?
It reduces some risk but doesn't eliminate it. Provider-side model updates, fine-tuning on your own data, and the permissions you grant the model within your application all remain your responsibility regardless of how reputable the underlying model provider is.
Do AI systems need a separate SBOM equivalent?
A growing number of practitioners argue yes — sometimes called an "AI bill of materials" — covering model version, training data sources, and fine-tuning history, though tooling and standards here are considerably less mature than software SBOMs.