To red team an internal chatbot, get written approval, map what the bot can read and do, then test it against the OWASP LLM risks in a staging copy with real permissions but fake data. Record every prompt and response, rate what you find, and fix the access behind the bot, not only its wording.
What are you actually testing?
A chatbot is rarely just a model. It is a model plus a system prompt, a retrieval index, tools, and the permissions of whoever or whatever it runs as. The OWASP GenAI Red Teaming Guide, published in January 2025, frames the work in four areas: model evaluation, implementation testing, infrastructure assessment and runtime behavior analysis. That is a useful split for an internal bot.
Most internal incidents will not be clever model tricks. They will be a bot that can read the HR share because it was easier to give it broad access.
How do you set the scope?
Write down four things before anyone types a test prompt.
- The bot's purpose and its intended users.
- Every data source and tool it can reach, with the identity it uses.
- Who approved the test, and which environment is in bounds.
- What counts as a stop condition, such as real personal data appearing in output.
Use a staging copy. Seed it with fake records that look real, including a few planted markers you can search for later. If a marker appears in an answer to the wrong user, you have a finding.
Which risks do you test first?
The OWASP Top 10 for LLM applications (2025 edition) is a practical checklist. For an internal bot, these entries tend to matter most:
- LLM01 Prompt Injection: can text in a user message, a document or a web page change what the bot does?
- LLM02 Sensitive Information Disclosure: can a user get answers drawn from data they should not see?
- LLM05 Improper Output Handling: does anything downstream trust the bot's output, such as a script, a ticket system or a browser?
- LLM06 Excessive Agency: what can the bot do on its own, and who confirms it?
- LLM07 System Prompt Leakage: does the system prompt hold secrets that should live elsewhere?
- LLM08 Vector and Embedding Weaknesses: can one team's documents surface in another team's search?
Test by role. Log in as a junior employee, a contractor and an admin, and ask the same questions. Differences in answers show where access control lives, or does not.
How do you run the tests without making a mess?
Keep it boring and repeatable. Build a small test set per risk, run it on every change to the prompt, model or tools, and store results with the version tested. A tester should be able to rerun last month's set in an hour.
Do not publish working attack strings in tickets or chat channels. Describe the behavior and the impact, and keep the exact inputs in the restricted test log.
Include indirect paths. Put a harmless instruction in a wiki page the bot indexes and see whether it is followed. Then check whether the bot tells the user it saw an instruction.
How do you rate and fix what you find?
Rate by what an attacker gains, not by how odd the prompt was. Reading another team's salary file is serious even if the trick was trivial. A rude answer is not.
Fixes that last are mostly outside the model:
- Give the bot the calling user's permissions, not a shared service account.
- Remove tools it does not need and require confirmation for actions that change data.
- Filter retrieval results by the user's access before they reach the model.
- Treat output as untrusted input to the next system.
- Move secrets out of prompts.
Prompt wording changes help, but plan on them being bypassed.
What to do
- Get written approval and name an owner for the bot.
- List its data sources, tools and identities.
- Build a staging copy with fake data and planted markers.
- Test each OWASP risk above as at least three roles.
- Log prompts, outputs and versions in a restricted location.
- Fix access and tool scope first, then prompts.
- Retest on every model, prompt or tool change and schedule a full review twice a year.
Quick check
Do we need a specialist tool to start?
No. A spreadsheet of test cases, a staging bot and a few roles will find the most serious problems. Tools help once you need volume and regression runs.
Is red teaming the same as a penetration test?
They overlap. A penetration test of the hosting and APIs is part of it, but red teaming also covers model behavior, retrieval and what the bot does when given hostile content.
Sources
- OWASP GenAI Red Teaming Guide
- OWASP Top 10 for LLM Applications 2025
- OWASP Top 10 for Large Language Model Applications project page
Last reviewed: 18 October 2026.
