The most common mistake in red teaming an internal chatbot is testing the model and ignoring the system around it: the documents it retrieves, the tools it can call and the people whose data it can reach. Scope the whole application, test it repeatedly, and fix permissions before prompts.
This post is defensive. It describes mistakes and fixes and includes no attack strings.
What does red teaming mean here?
NIST's generative AI profile (AI 600-1) defines AI red-teaming as a structured testing exercise used to probe an AI system to find flaws and vulnerabilities, often in a controlled environment and in collaboration with system developers. OWASP's GenAI Red Teaming Guide frames the work in four areas: model evaluation, implementation testing, infrastructure assessment and runtime behavior analysis. An internal chatbot touches all four.
Which mistakes come up most?
1. Testing only the model. NIST warns that jailbreak and prompt engineering tests may not systematically assess validity or reliability risks. A model that refuses rude requests can still leak a confidential file through a retrieval path. Test the application end to end.
2. Ignoring connected data and tools. OWASP lists impacts of prompt injection that depend on what the model can reach: exposure of sensitive information, unauthorised access to functions available to the model and commands run in connected systems. Severity depends on the business context and how much agency the model has. List every data source and tool first, then test each.
3. Skipping indirect input. OWASP separates direct injection, where a user types the input, from indirect injection, where the model reads external content such as web pages or files that contain instructions. An internal chatbot that summarises tickets, wikis or email is exposed to indirect input from anyone who can write to those sources. Include them in scope.
4. Treating access control as the model's job. OWASP recommends least privilege, with the application holding its own API tokens and handling privileged functions in code instead of through the model, and human approval for high risk actions. If the bot can read everything the service account can read, a good prompt filter will not rescue you. Check that retrieval honours the asking user's permissions.
5. A one-off exercise. OWASP says no foolproof prevention is known and recommends regular adversarial testing, treating the model as an untrusted user. Model versions, prompts, documents and tools change. Rerun the same test set after each change and keep the results.
6. No scope, rules or authority. Testing against live internal data without agreement creates its own incident. Write down what is in scope, which accounts and data may be used, who is told when and how to stop.
7. Findings with no owner. A report that lists issues but not fixes, owners and dates gets filed. NIST's profile treats red-teaming as one input to governance decisions and documentation. Track each finding to closure.
What should you check besides prompts?
Output handling: does the application treat the model's text as untrusted before it renders or executes it? OWASP recommends defining expected output formats and validating them with deterministic code.
Logging: can you see which documents were retrieved and which tools were called for a given answer? Without that, an incident cannot be reconstructed.
Rate and cost limits: can one user cause large spend or load?
Third-party parts: models, plugins and packages the chatbot depends on are a supply chain. Keep an inventory.
What to do
- Draw the system: model, prompts, retrieval sources, tools, identities and logs.
- Write the scope and rules of engagement, and get sign off.
- Test direct input, indirect input from every writable source, and permission boundaries.
- Verify least privilege for the bot's credentials and approvals for risky actions.
- Keep a regression set and rerun it after every model, prompt or data change.
- Assign every finding an owner and a due date, and retest after the fix.
Quick check
Is a list of jailbreak prompts a red team?
No. It tests one layer. NIST notes such tests may not assess reliability systematically, and OWASP's guide covers four areas including implementation and infrastructure.
Who should hold the permissions, the model or the app?
The application. OWASP recommends the app holds its own tokens and handles privileged functions in code, with human approval for high risk actions.
Sources
Last reviewed: 18 October 2026.
