Shipping a large language model into production is not the same as calling an API. You have introduced a component that reasons over untrusted input, may hold privileges, can be manipulated through natural language, and depends on a supply chain of models, prompts, and data that most application security programs have never inventoried. The OWASP Top 10 for LLM Applications exists precisely because these risks recur across teams, and it is the most useful scaffold available for a pre-launch review. This checklist walks that scaffold and turns each category into concrete questions to answer before you go live.
Treat every unchecked box as a finding, not a formality.
How to use this list. Run it as a gate before launch and again after any material change — a new model version, a new tool, a new data source, a new prompt. Assign an owner to each section rather than treating it as one person's homework, because the supply-chain rows belong to a different team than the output-handling rows. And wire the deterministic rows (the ones a scanner can verify) into continuous integration so they stay checked between reviews instead of drifting the moment attention moves elsewhere. A checklist that is walked once and filed is a snapshot of a system that no longer exists.
LLM01 — Prompt injection
- Trusted instructions and untrusted data are structurally separated in the prompt.
- The model's privileges are scoped to the minimum the feature requires, so a successful injection has a small blast radius.
- Any content the model ingests (documents, web pages, tool output, retrieved passages) is treated as potentially adversarial.
- Consequential or irreversible actions require explicit human confirmation.
LLM02 — Sensitive information disclosure
- Secrets, credentials, and internal system details are not placed in prompts or system messages the model could regurgitate.
- Personally identifiable information is minimized in context and redacted where possible.
- Output is checked so the model cannot leak other users' data or its own configuration.
LLM03 — Supply chain
- Every model, base weight, and adapter is inventoried with a known, verified source.
- Model files use safe serialization formats; untrusted pickle-based weights are not loaded without sandboxing.
- Third-party plugins, MCP servers, and tools are pinned, reviewed, and monitored for changes.
- All application dependencies pass automated composition analysis.
LLM04 — Data and model poisoning
- Training and fine-tuning data sources are vetted and access-controlled.
- Retrieval indexes and knowledge bases are protected against injection of poisoned documents.
- Provenance is recorded for datasets so a tainted source is traceable.
LLM05 — Improper output handling
- Model output is never passed unescaped into HTML, a shell, a query, or
eval. - Output rendered in a browser is encoded to prevent cross-site scripting.
- Output used to drive actions is validated against an allowlist of permitted operations.
LLM06 — Excessive agency
- The model has the minimum tools, permissions, and autonomy the task needs — nothing "just in case."
- Write, delete, and spend capabilities are gated behind human approval or hard limits.
- Tool scopes are explicit and deny-by-default.
LLM07 — System prompt leakage
- The system prompt contains no secrets whose disclosure would be harmful.
- Security does not depend on the system prompt staying hidden; assume it can be extracted.
LLM08 — Vector and embedding weaknesses
- The embedding/vector store enforces access control so one tenant cannot retrieve another's data.
- Documents entering the index are validated and stripped of hidden instructions.
LLM09 — Misinformation
- User-facing output includes appropriate uncertainty and does not present hallucinations as fact in high-stakes contexts.
- Critical outputs (legal, medical, financial, security) are grounded and human-reviewed.
LLM10 — Unbounded consumption
- Rate limits and token/cost budgets cap runaway usage and denial-of-wallet attacks.
- Recursive or agentic loops have hard iteration limits.
Cross-cutting: observability
- Every high-privilege model action is logged with full context.
- Anomalous sequences (ingest untrusted content → attempt sensitive action) trigger alerts.
- You can answer "what did the model do, when, and why" after an incident.
Cross-cutting: governance and change
- There is a named owner for the security of each LLM feature.
- Changes to models, prompts, tools, and data go through a review process, not a silent config edit.
- Adversarial tests (a red-team corpus) run against the system on a schedule and after each change, not just once at launch.
- An incident playbook exists specifically for "the model did something it should not have."
How Safeguard helps
Several rows on this checklist are code and supply-chain problems, and that is where Safeguard does the heavy lifting. The Griffin AI detection engine targets the improper-output-handling patterns behind LLM05 — model output reaching a shell, a query, or unescaped markup — and flags them before they ship. Software composition analysis (SCA) covers the LLM03 supply-chain rows, inventorying dependencies and catching known-vulnerable or hallucinated packages, while the Safeguard MCP server gives the tools you connect to your model the least-privilege, auditable scopes that keep LLM06 excessive agency in check. When a scan turns up a fixable issue, auto-fix remediation proposes the patch so a checklist finding becomes a merged fix rather than a backlog item.
Run this list before launch, wire the code-level rows into CI so they stay checked, and your LLM feature goes live as a considered system rather than a hopeful one.
Ready to automate the code and supply-chain rows? Create a free account, read the documentation, or review pricing plans to find the right fit.