Red team an internal chatbot by treating the model as an untrusted user and testing what it can reach: its tools, its data sources and the content it reads. Scope the test in writing, run it against a staging copy, record every result, and fix access first, wording second.
Why is the chatbot's access the real target?
OWASP lists prompt injection as the first risk for language model applications and says plainly that it is unclear whether fool-proof prevention exists. The UK NCSC goes further: current models do not enforce a boundary between instructions and data inside a prompt, so the residual risk has to be managed by design.
That changes the question. You are not testing whether the bot can be talked into saying something rude. You are testing what an attacker gets if they control the words the bot reads. If the bot can search the wiki, open tickets and send email, then "whatever the worst case would be of giving an attacker direct access to those tools" is the impact, in the NCSC's phrasing.
How do you scope the exercise?
The OWASP GenAI Red Teaming Guide frames the work in four areas: the model, how it is implemented, the supporting infrastructure, and runtime behavior. For an internal bot, most findings sit in implementation and infrastructure.
Write down before you start:
- The environment: a staging copy with test data, never production records.
- The tools the bot can call and the permission each one runs with.
- The data sources it can retrieve from, and who can edit them.
- Stop conditions, and the person who can pause the test.
- Who receives findings, and the deadline for a response.
What belongs on the checklist?
Tool abuse. List every tool. For each, ask what happens if the bot calls it with arguments an outsider chose. Confirm that destructive or outward actions need a human approval step handled in code, not in the prompt.
Data scope. Check whether the bot answers from documents the asking user could not open directly. Retrieval often runs with a service account that sees more than the person does.
Indirect input. Anything the bot reads is an input: wiki pages, tickets, uploaded files, web pages, email. OWASP's own scenarios include a poisoned document in a retrieval store changing the answers. Test whether low-trust content can steer behavior.
Output handling. Check whether responses are rendered as links, images or markup in a way that could send conversation content elsewhere. Validate output formats with deterministic code.
Sensitive disclosure. Ask whether the bot reveals its system prompt, internal hostnames or other users' data.
Logging. Confirm that you can reconstruct which user asked what and which tools ran. If you cannot, you cannot investigate an incident.
Failure behavior. See what happens with very long input, odd encodings and repeated requests. Look for crashes and cost spikes as well as leaks.
Do not rely on a deny list of phrases like "ignore previous instructions". The NCSC notes there are endless rephrasings, so a filter helps a little and proves nothing.
What to do
- Write the scope document and get an owner to sign it.
- Inventory tools, data sources and the identity each one runs as.
- Reduce privileges first: give the bot the permissions of the least trusted party whose content it reads.
- Run the checklist above on staging and log each attempt with the result.
- Rank findings by what an attacker could do, not by how clever the prompt was.
- Fix, retest, and repeat on a schedule and after any change to tools or data sources.
Quick check
Is a system prompt that says "never reveal secrets" enough?
No. Prompt instructions can be bypassed. Keep secrets out of the model's reach and enforce limits in code.
How often should we re-test?
After any change to the tools, data sources or model, and on a regular schedule. OWASP recommends ongoing adversarial testing rather than a single pass.
Sources
- OWASP LLM01: Prompt Injection, fetched 18 October 2026
- NCSC: Prompt injection is not SQL injection, fetched 18 October 2026
- OWASP GenAI Red Teaming Guide, fetched 18 October 2026
Last reviewed: 18 October 2026.
