A good two week AIDR trial uses your own AI traffic, a written list of scenarios, success criteria agreed beforehand, and a scoring sheet. Week one covers setup and observation. Week two covers constructed attack scenarios, response tests and review. At the end, you should know what it saw, what it missed and what it cost in effort.
Before day one: set the rules
Write these down and share them with the vendor.
- Scope. Which two or three AI systems are in the trial. Choose one chat feature, one agent that calls tools and one internal tool if you have them.
- Data. What the tool may see. Prompts can hold personal or confidential data, so decide masking, retention and access first.
- Success criteria. Specific, measurable and agreed before you see results.
- Owners. One person from security, one from the team that owns each system, and one person who can pause a feature.
Keep a risk list for choosing scenarios. MITRE ATLAS catalogues adversary tactics against AI systems, and OWASP publishes lists for LLM applications and for agentic applications. Pick five risks that apply to your systems.
Week one: setup and baseline
Days 1 and 2: connect and check. Connect the tool to the chosen systems. Confirm that events appear, with the fields you need. Record how long setup took and what engineering time it used. Setup effort is a real cost.
Days 3 to 5: observe normal. Let it run on ordinary traffic. Do not tune yet. Record what it flags and let analysts label a sample of alerts as real, harmless or unclear. This gives a baseline false positive picture.
Day 5: inventory check. Ask what the tool believes exists: models, tools, agents. Compare it with your own inventory. Differences in either direction are useful.
Week two: scenarios and response
Run constructed scenarios in a test copy or with agreed test accounts. Do not use real customer data, and do not publish the scenarios. Keep them defensive: you are checking whether you can see and stop misuse, not building an attack kit.
Day 8: instruction in content. Place a harmless planted instruction in a document or ticket the system reads, and see whether the tool notices that the model followed text from content.
Day 9: data in output. Put a fake secret, clearly marked as a test value, in a source the system can read. See whether it appears in output and whether the tool catches it.
Day 10: tool misuse. Check an agent calling a tool outside its normal pattern, or with a wider scope than usual. OWASP names excessive functionality, permissions and autonomy as root causes of agent harm, so scope violations are the natural test.
Day 11: response. Test the actions: pause an agent, revoke a credential, route output to review. Time each one and check the rollback.
Day 12: noise and cost. Review alert volume per day and the analyst time spent. Check whether the traffic from tests changed the billing basis you were quoted.
Days 13 and 14: score and decide. Fill the scoring sheet, then hold a review with all owners.
What should the scoring sheet contain?
Use 0 to 3 for each row.
- Setup effort against estimate.
- Inventory accuracy.
- Share of scenarios detected, and how soon.
- Alert precision on the labelled sample.
- Quality of the explanation for each alert.
- Response actions that worked, with timing and rollback.
- Privacy handling and retention fit.
- Analyst time per day.
- Export of data and exit terms.
What is a fair pass mark?
Decide it before the trial. For example, "detect four of five scenarios, explain each alert in plain words, contain within the target we set, and need less than an agreed amount of analyst time per day". We do not give numbers here because they depend on your team and risk.
What should make you stop?
Three red flags: the vendor will not let you use your own traffic, the tool cannot explain an alert from its own evidence, or the trial requires data sharing that your policy forbids.
What to do
- Write scope, data rules, success criteria and owners before day one.
- Choose five risks from the public catalogues and one scenario for each.
- Keep week one for baseline, with no tuning.
- Run constructed scenarios in a test copy with test data only.
- Time every response action and its rollback.
- Score with the sheet and compare to the pass mark you set earlier.
Safeguard focuses on the application and software supply chain side, including MCP Guardrail for MCP servers, and does not replace a dedicated AI detection tool.
Quick check
Can we trial AIDR on production traffic?
Yes for observation, if privacy rules allow it. Run attack style scenarios only in a test copy with test data, never against real customers.
What if the tool detects everything in the trial?
Check that the scenarios were not shared in advance and that the false positive count is reasonable. Perfect scores in a short trial are a reason to look harder.
Sources
- MITRE ATLAS
- OWASP LLM Top 10, Excessive Agency
- OWASP Top 10 for Agentic Applications for 2026
- NIST AI Risk Management Framework
Last reviewed: 18 October 2026.
