How to test your LLM app for prompt injection (OWASP LLM01)

To test an LLM app for prompt injection, first map everything the model can read and every action it can take. Then check, for each path, whether untrusted text can change behavior, and whether least privilege and human approval limit the damage. OWASP says no fully reliable prevention exists, so test impact, not just refusal.

What does OWASP mean by prompt injection?

OWASP's LLM01 entry says prompt injection happens when inputs alter an LLM's behavior or output in unintended ways, and that the inputs can be imperceptible to humans as long as the model parses them. It separates two main types.

Direct injection is when a user's own input changes the model's behavior, on purpose or by accident. Indirect injection is when the model ingests outside content, such as a web page or file, containing instructions that change what it does. OWASP also flags multimodal risk, where instructions hide in an image that accompanies text.

OWASP adds that retrieval augmented generation and fine tuning do not fully prevent the problem, and jailbreaking is a related form that tries to make a model ignore its safety rules.

This post is a test plan. It deliberately includes no attack strings.

Step 1: draw the trust map

Write down, for one feature at a time:

• Every input source: user messages, uploaded files, web pages, email, tickets, database fields, tool results.
• Every capability: tools, APIs, file access, code execution, data it can return.
• Who owns each source, and whether an outsider can write to it.

The riskiest combination is an untrusted source that reaches a model with a powerful capability. OWASP notes that severity depends on the business context and the model's level of agency.

Step 2: test each path with the same questions

For every untrusted input and capability pair, ask:

1. Can this content change the model's stated role or task?
2. Can it cause a tool call the user did not request?
3. Can it expose the system prompt, other users' data or infrastructure details? OWASP lists these as potential impacts.
4. Does anything outside the model, in plain code, check that call before it runs?

Use your own test content. Plant harmless marker instructions in a document, a web page and a ticket in a staging environment, and see whether the app changes behavior or touches something it should not. Record results per path.

Step 3: check the controls OWASP lists

• Constrained behavior: the system prompt defines the role and limits.
• Validated output formats: deterministic code checks the shape of answers.
• Input and output filtering: semantic and string checks for disallowed content.
• Least privilege: the app has its own tokens, privileged functions run in code and not through the model, and access is limited to what is needed.
• Human approval: high risk or privileged actions need a person.
• Segregated external content: untrusted content is clearly separated and labeled.
• Adversarial testing: OWASP recommends regular penetration tests and breach simulations, treating the model as an untrusted user.

Treating the model as an untrusted user is the useful stance. Anything you would not let a stranger do, the model should not be able to do on its own.

Step 4: decide pass or fail on impact

A model that sometimes follows an injected instruction is expected today. Pass means that when it does, the blast radius is small: it cannot call a destructive tool without approval, cannot reach other tenants' data and cannot send data outward unchecked.

Where Safeguard fits

If you connect agents to Safeguard over its MCP server at mcp.safeguard.sh, tool calls are unlimited on every plan, and tenant admins can switch off client types. You can run Safeguard with Full AI, Zero AI or your own model. For your own app, the controls above remain your responsibility.

See the guardrail idea at /demos/mcp-guardrail.

What to do

1. Draw the trust map for each LLM feature.
2. Mark every path where outsiders can write to a source the model reads.
3. Move privileged actions out of the model and into code with checks.
4. Add human approval to anything destructive or external.
5. Run the staging tests above on every release that changes prompts, tools or data sources.
6. Log tool calls so you can see unusual ones.

Quick check

Sources

• OWASP GenAI: LLM01 Prompt Injection, fetched 18 October 2026

Last reviewed: 18 October 2026.