A quarterly review should cover five things: which tools and accounts are in use, what data went in, what errors were caught, what permissions assistants hold, and what changed in vendor terms or your policy. Keep it to one hour and end with named actions.
Why review at all
Use of assistants spreads and changes quietly. A pilot for summaries can become routine use on sensitive code. Terms change. Models change and behaviour shifts. A regular look catches drift before it becomes an incident.
The checklist
1. Tools and accounts
- List assistants in use, including ones nobody approved. Ask people, since logs rarely show it all.
- Check that work use is on company accounts.
- Remove tools nobody uses.
2. Data
- Sample five recent uses and check the inputs against your data tiers.
- Look for secrets, personal data or open vulnerability details.
- Confirm retention settings still match your rule.
3. Accuracy
- Count errors caught in review this quarter: wrong versions, invented packages, bad fixes.
- Note which task types produced most of them.
- Decide whether any task should move to a stricter process.
4. Permissions
- List every tool, token and integration an assistant can use.
- Remove anything not needed. Prefer read-only.
- Check approvals are still required for actions that change state.
5. Changes
- Re-read vendor terms for changes to retention or training use.
- Note model updates and rerun your test cases.
- Update the policy page and tell the team what changed.
Metrics worth tracking
| Metric | Why |
|---|---|
| Errors caught per hundred uses | Shows whether verification works |
| Share of uses on approved accounts | Shows policy reach |
| Time saved on summaries, self reported | Shows value, loosely |
| Incidents or near misses | Shows where rules fail |
Avoid counting prompts as a success measure. Volume says nothing about quality.
Outputs of the review
Write down three things: what you will keep, what you will tighten, and who does what by when. Share a short summary with leadership, plain and honest, including mistakes. Reports that only show good news stop being read.
Related practice
For first thirty day rollouts see AI coding assistant 30 day pilot plan for a security team. For checking claims against sources, see checklist, verify LLM security findings before acting.
Where Safeguard fits
Safeguard provides asset discovery, agent identity and MCP Guardrail for governing agent tool use, plus compliance evidence automation to keep the review's evidence in one place. Free scan: https://gold.safeguard.sh/scan.
Quick check
How long should the review take?
About an hour. If it takes longer, narrow the scope and sample more.
Who should run it?
The team lead or a security owner, with input from people who use the tools daily.
Last reviewed: 19 October 2026.
