Microsoft's current rules of engagement for security testing list testing AI systems by trying to bypass their restrictions, including system prompts, as an allowed activity inside your own tenant. AWS's page lists Amazon Bedrock AgentCore among services that can be tested without prior approval. Providers now expect customers to test AI features, and enterprise buyers are starting to ask whether you did.
If your product has a chat assistant, a summarizer or an agent that calls tools, the usual web test plan misses most of the risk. This post gives you a test list built on the OWASP Top 10 for LLM Applications 2025.
What's affected
Any feature where a model reads user or third-party content and produces text or actions: support chatbots, document Q&A, email assistants, coding agents, and agents with tool or API access.
OWASP defines prompt injection (LLM01:2025, CWE-1427 is the matching weakness class) as input that changes the model's behavior in ways the developer did not intend, and says no method is known to fully prevent it. It separates direct injection, from the user's own prompt, from indirect injection, from web pages or files the model reads. Excessive Agency (LLM06) covers damage from too much functionality, permission or autonomy given to tools.
How to check if you're exposed
Test these cases against a staging copy with seeded fake data in two tenants.
- Direct injection. Ask the assistant to ignore its instructions, reveal its system prompt, and act in a different role. Record what it discloses. (LLM01, LLM07 System Prompt Leakage.)
- Indirect injection. Put instructions inside content the assistant will read: a PDF, a support ticket, a web page, a calendar invite. Ask a harmless question and see whether the hidden text changes the answer or triggers a tool call.
- Cross-user data. In tenant A, ask for tenant B's documents by name. If you use retrieval, check whether the vector store filters by tenant before the model sees results. (LLM02 Sensitive Information Disclosure, LLM08 Vector and Embedding Weaknesses.)
- Tool permissions. List every tool the model can call. For each, ask: could a low-privilege user make the model use it on something they cannot access directly? Check whether the tool runs as the user or as a shared service account.
- Tool abuse through injection. Combine cases 2 and 4. OWASP's example is an email assistant that can also send mail, where one malicious incoming message makes it forward the inbox to an attacker.
- Output handling. Ask the model to return HTML, a markdown image link to an external URL, or a SQL fragment. Check whether your front end or database layer treats the output as trusted. (LLM05 Improper Output Handling.)
- Consumption. Send a modest set of long prompts and check for per-user limits. Do not flood: cloud policies prohibit denial of service. (LLM10 Unbounded Consumption.)
What to do now
- Cut tool functionality to the minimum. OWASP prefers granular tools over open-ended ones such as a generic shell or URL fetcher.
- Run tools as the user. Use the caller's identity and a minimal OAuth scope, not a broad service account.
- Enforce authorization downstream. OWASP says to implement authorization in downstream systems rather than rely on the model to decide whether an action is allowed.
- Require approval for high-impact actions such as sending, deleting, paying and sharing.
- Separate untrusted content. Label external content as data, and validate expected output formats in code.
- Log tool calls with user, arguments and result, and rate-limit risky actions.
- Put the cases above in your scope sheet and ask the vendor which they have run on LLM features before.
How to avoid this trap
- Treat the model as an untrusted user at every trust boundary. OWASP recommends exactly this framing for penetration tests.
- Do not ship a tool whose permissions exceed those of the least privileged user who can talk to the assistant.
- Keep secrets and API tokens in application code, not in the prompt.
Why it matters
A model cannot reliably tell instructions from data, so any content it reads can steer it. The fix is not a cleverer prompt. It is limiting what a steered model can do, and checking every action outside the model.
A scope sheet for an ordinary web test will not mention any of this. Add a line for AI features and make the vendor name the cases.
Try it yourself
List every tool your assistant can call and write next to each one the identity it runs as. Any tool running as a shared account with broader rights than the user is a finding waiting to happen.
Click through it at your own pace, or open it full screen.
Quick check
What is the difference between direct and indirect prompt injection?
Direct injection comes from the user's own prompt. Indirect injection comes from external content the model reads, such as web pages or files.
Where should authorization for a tool action be enforced?
In the downstream system, not in the model's decision, according to OWASP's LLM06 guidance.
