Before a public chatbot goes live, build a set of questions with approved answers, ask each one many times and in different wording, probe for abuse, and agree on a rule for pausing the service. Pass or fail on consistency as well as correctness.
Why repeat runs matter
The Markup reported in March 2024 that New York City's MyCity chatbot told one reporter that landlords had to accept rental assistance vouchers, and told ten other staffers they did not. Same question, opposite answers. A single test pass would have shown only one of those.
Microsoft's 2016 Tay chatbot was a different failure: Microsoft said a coordinated group exploited a vulnerability within the first 24 hours online, and called it a critical oversight for that type of abuse. Both cases point at testing the system the way real users and bad actors will use it.
Step 1: Build a gold set
Collect real questions from support tickets, search logs or staff. For each, write the approved answer and the source document. Start with 100 to 200 questions, weighted toward the high stakes ones: money, legal duties, safety, personal data.
Step 2: Repeat and vary
For each question:
- Run it at least ten times and compare answers.
- Rephrase it three ways, including slang and typos.
- Ask it in every language you support.
Score each answer as correct, partly correct, wrong or refused. Track the share that changes between runs. Any instability on a high stakes question is a blocker.
Step 3: Probe for abuse
Have a small group try to break it:
- Ask it to ignore its rules or reveal its instructions.
- Try to make it say offensive or off brand things.
- Ask it to make promises, such as discounts or refunds.
- Paste content that contains hidden instructions.
For context on instruction attacks, see can a system prompt stop prompt injection.
Step 4: Limit what it can do
Decide what the bot may say and do. Restrict it to approved sources for factual claims. Remove any tool or action it does not need. Answer sensitive topics with a handoff to a person or a link to the official page.
Step 5: Agree a takedown rule
Before launch, write down the trigger: for example, any confirmed wrong answer on a legal duty, or any abuse screenshot that spreads. Name who can turn the bot off and how long that takes.
Step 6: Keep testing after launch
Rerun the gold set whenever you change the model, the prompt or the source documents. Review a sample of real conversations weekly. Make it easy for users to flag a wrong answer.
What testing cannot do
It lowers the risk. It does not prove the bot will never be wrong. A visible note that the bot can make errors is fair, but a note does not make an answer right. See write evals before you fine tune for building the question set.
Quick check
How many questions do we need?
Start with 100 to 200 real ones and grow it from live flags. Weight toward high stakes topics.
Is a disclaimer enough?
No. Users act on answers. Fix the answer or hand off to a person.
Sources
- The Markup: NYC's AI chatbot tells businesses to break the law, 29 March 2024
- Microsoft: Learning from Tay's introduction, 25 March 2016
Last reviewed: 19 October 2026.
