On November 18, 2024, the OWASP GenAI Security Project published version 2.0 of the OWASP Top 10 for LLM Applications, and the list quietly redrew the boundary of what an application security engineer is expected to know. It is also the clearest job description yet for the AI security engineer — a role that barely existed two years ago. LLM01, ranked first, is prompt injection — and OWASP is explicit that it is not just a new entry in an old category: it cannot be fixed by parameterized queries or input escaping the way SQL injection can, because large language models do not structurally separate instructions from data the way a database driver separates a query from its arguments. The other nine items on the list — sensitive information disclosure, supply chain vulnerabilities, data and model poisoning, improper output handling, excessive agency, system prompt leakage, vector and embedding weaknesses, misinformation, and unbounded consumption — describe a risk surface that SAST, DAST, and SCA tools were never built to see. At the same time, CycloneDX shipped a dedicated ML-BOM profile and SPDX added a parallel AI/dataset profile, giving the industry its first real, citable machine-readable formats for AI asset inventories. Together these developments mark the moment "AI security" stopped being a subtopic of AppSec and started requiring its own skill set. This piece maps what that skill set actually contains.
What does the OWASP LLM Top 10 say traditional AppSec engineers are missing?
The OWASP Top 10 for LLM Applications (v2.0, November 18, 2024) is built around failure modes that don't map to CWE categories an AppSec engineer would recognize from a decade of web application work. LLM04, data and model poisoning, describes an attacker corrupting a training or fine-tuning dataset so the resulting model behaves maliciously under specific triggers — there is no equivalent "input validation" fix, because the vulnerability is baked into the weights before the application code ever runs. LLM06, excessive agency, covers agents that are granted more tool access, autonomy, or permissions than a task requires, so a successful prompt injection can cascade into real actions — sending emails, executing code, modifying records — rather than just leaking data. LLM07, system prompt leakage, and LLM10, unbounded consumption (resource exhaustion via unbounded generation or recursive agent loops), round out a list where roughly half the entries describe behaviors specific to probabilistic, natural-language systems. None of these map cleanly onto a Bandit rule ID or a CVE database lookup, which is exactly OWASP's point.
Why do ML-BOM and AI-SBOM formats need to exist separately from software SBOMs?
A conventional software SBOM in CycloneDX or SPDX format lists packages, versions, and licenses — it has no field for a model's parameter count, quantization, training dataset lineage, or whether its weights are signed. CycloneDX addressed this with a dedicated ML-BOM profile for machine-learning bill-of-materials data, part of the same OWASP-stewarded, Ecma-424-standardized specification that reached version 1.7 in October 2025, up from 1.6 in April 2024 — the spec has been iterating roughly twice a year specifically to keep pace with AI asset types. SPDX 3.0 shipped a parallel AI and dataset profile covering comparable ground. The reason these can't just be bolted onto an existing software SBOM field is structural: a model asset needs provenance for the data that shaped its behavior, not just the code that loads it, and a fine-tuned model's lineage — base model, adapter, dataset, hyperparameters — is a dependency graph a package-lock.json has no vocabulary for. An engineer who can query a software SBOM for a vulnerable transitive dependency but can't produce an ML-BOM entry for a Hugging Face model with unverified provenance is only covering half the inventory.
What new technical risks does model deployment introduce that AppSec tooling doesn't catch?
Model weight files are executable in ways source code scanners don't check for. PyTorch's default serialization for .pt files relies on Python's pickle module, and pickle deserialization can execute arbitrary code at load time — a risk class (CWE-502) that traditional SAST tools flag in application code but that most pipelines never extend to third-party model artifacts pulled from a registry. Beyond the file format itself, a model's weights can carry statistical backdoors — a specific input pattern ("trigger") that produces attacker-chosen output, or a class-level bias that misclassifies inputs matching a trigger — findings that are inherently probabilistic and require statistical testing against the weights themselves, not a line-by-line code review. Neither risk class shows up in a dependency-CVE feed, because the vulnerability isn't in a package version number; it's in the trained parameters. This is the core argument for why "AI security" needs distinct tooling: a scanner built to parse source code and match known-vulnerable version strings has no mechanism for inspecting a 7-billion-parameter weight tensor for a trojan trigger.
What does prompt-injection and guardrail testing look like as a discipline?
Testing an LLM application for prompt-injection resistance is a fundamentally different exercise than a DAST crawl, because the attack surface is natural language rather than structured HTTP parameters, and there is no fixed grammar of "malicious payloads" to fuzz against — attackers can phrase the same intent an effectively unlimited number of ways. A defensive testing harness for this discipline runs benign test cases against a model's guardrails to confirm that jailbreak attempts, system-prompt extraction attempts, and instruction-override attempts are actually caught, rather than assuming a filter works because it was configured. This looks more like breach-and-attack simulation than classic penetration testing: you're not proving a payload achieves code execution, you're proving a detection or refusal behavior holds under adversarial-but-safe conditions, with human sign-off gating anything higher-impact. An AppSec engineer trained to think in terms of injection points and sanitization boundaries has to relearn the exercise as one of behavioral testing against a probabilistic system that can be coaxed rather than exploited in the traditional sense.
Why are regulators accelerating the timeline for this skill set to exist?
Two frameworks are converting "AI risk is hard to define" into "AI risk documentation is a compliance obligation with a deadline." NIST released the AI Risk Management Framework 1.0 in January 2023, giving organizations a structured vocabulary — govern, map, measure, manage — for AI risk that auditors and procurement teams have since started asking about directly. The EU AI Act entered into force on August 1, 2024, with obligations phasing in through 2026 and 2027 depending on risk tier and system category, and it requires documentation of training data, risk assessments, and human oversight for higher-risk AI systems — obligations that fall on whoever can produce a model's provenance and lineage records on demand. Neither framework can be satisfied by a team that only knows how to triage CVEs in a requirements.txt; both require someone who can answer "where did this model's training data come from, and can you prove it" — which is an AI-BOM question, not an SCA question.
How Safeguard helps
Safeguard's AI-BOM discovery inventories models, prompts, vector stores, agents, and MCP servers across code, cloud, and connected SaaS, then emits the asset data in CycloneDX ML-BOM extension format and the SPDX 3.0 AI profile — the two AI-SBOM standards this piece names above — so a governance question about training-data lineage or weight provenance has a queryable answer instead of a manual audit. On top of that inventory, Safeguard's model-security scanning checks weight files for pickle-based executable payloads and runs statistical tests for trojan triggers and class-level backdoors, flagging findings with a likelihood score rather than a binary pass/fail. For the prompt-injection and jailbreak side of the discipline, Safeguard's Red Team engine includes a defensive-only AI/LLM guardrail validation harness that probes guardrail resistance with benign test cases under a signed rules-of-engagement, gated by human approval for any higher-impact step — giving a security team evidence that a guardrail holds without ever running a weaponized payload against production. Together, the two capabilities cover the AI-BOM and prompt-injection-testing pillars the AI security engineer role is being defined around, on top of the reachability and SCA coverage a traditional AppSec program already expects.