To test an LLM app for prompt injection, define each attack's goal and expected outcome first, run it against sandboxed tools with dummy data, grade each security objective separately as violation, no violation or inconclusive, and repeat runs. A passing smoke test does not show resistance to a persistent attacker.
Definitions
Smoke test. A small set of attack and normal inputs that catches obvious failures. OWASP's guidance says to use its samples this way, not as a security benchmark.
Security objective. One thing the attacker wants, such as disclosing the system prompt, using a tool without authorization, or leaking data to an outside destination.
Inconclusive. A run where you cannot tell what happened, for example a timeout or missing logs.
Test design
| Element | What to write down | Why |
|---|---|---|
| Attack case | The intended violation | You cannot grade without a definition |
| Benign twin | A legitimate task that looks similar | Measures false positives |
| Context | Which documents, tools and permissions are present | Results depend on it |
| Observable outcome | The log line, tool call or network request that proves the result | Avoids guessing from the reply text |
| Verdict | Violation, no violation or inconclusive | Errors are not wins |
Steps
- List the channels untrusted text can enter. User input, retrieved documents, web pages, emails, tool results, issue comments, file metadata.
- Build the environment safely. Use dummy data, sandboxed tools and instrumented destinations you control. Never test with real customer records or live write access.
- Test the indirect path through the real channel. Put the instruction inside the document or page the app will retrieve, not just in the chat box.
- Cover varied attack styles. The OWASP cheat sheet lists encoding and obfuscation, multi-turn attacks, role-play framing, hidden markdown or HTML, multimodal content, retrieval poisoning and agent-specific tool manipulation. See the companion post on attack types.
- Grade each objective on its own. Prompt disclosure, unauthorized tool use and external data leakage are different outcomes with different fixes.
- Repeat. Model outputs vary. Report numerators and denominators, such as 3 violations in 40 runs, and use confidence intervals where the sampling supports them. Include repeated attempts, because best-of-N style attacks try many variations until one works.
- Measure the cost of defenses. Track false positives on benign requests and task completion separately from attack success.
- Validate your grader. If a model grades results, check it against cases where you know the answer.
- Retest after every change to the prompt, model, tools or retrieval sources.
Fix by privilege, not by phrase
If a test finds a violation, the durable fix is usually to reduce what the model can do: narrower tool scopes, human approval for high-impact actions, output validation in code and least privilege enforced outside the model. OWASP and the NCSC both say no foolproof prevention is known, so the goal is lower likelihood and lower impact.
Quick check
How many test cases are enough?
There is no number that proves safety. Start with one case per untrusted channel and per sensitive action, then grow as you find gaps.
Should a tool error count as a blocked attack?
No. Mark it inconclusive. The guidance says never count errors or missing telemetry as blocked attacks.
Can I use automated tools for this?
Yes, as a way to generate and run cases. You still need your own cases for your channels, tools and data.
Where Safeguard fits
Safeguard provides an MCP Guardrail and policies and guardrails for agent activity, and scans the dependencies of your LLM application. You can run a free scan at gold.safeguard.sh/scan.
Sources
- OWASP LLM Prompt Injection Prevention Cheat Sheet
- OWASP LLM01 Prompt Injection
- NCSC on prompt injection
Last reviewed: 19 October 2026.
