To judge AI alert triage, track what it got wrong as carefully as what it saved. The useful measures are missed true positives, agreement with senior analysts, how often analysts edit its output, and time to escalate real incidents. Minutes saved alone can rise while safety falls.
Why speed misleads
If the assistant closes more alerts faster, the dashboard looks great. But a model that dismisses everything is also fast. Speed has to be paired with a measure of correctness, taken from a source the model did not influence.
Build a reference set
Pull a sample of closed alerts across your main types. Have a senior analyst label each one with the right outcome and priority, without seeing the model's answer. Keep the set stable so you can rerun it after changes. Refresh it as your alerts change.
This is the same approach as measuring tool quality in a pilot: ground truth first, then compare. A two week pilot plan shows how to run it.
The measures to track
Missed true positives. Real incidents the model rated low or closed. This is the number that matters most. Review each one.
False escalations. Benign alerts it raised. These cost analyst time and feed alert fatigue.
Agreement with the reference. The share of alerts where model and senior analyst reached the same outcome, reported by alert type, not as one overall figure.
Edit rate. How often an analyst changes the model's summary or priority. A high rate means the output is not trusted or not right.
Time to escalate real incidents. Measure from alert creation to a human seeing a true positive. It should not get worse.
Evidence completeness. Did the summary mention the key facts an analyst needed? Score a sample.
Report by segment
An average hides problems. Break results down by alert source, severity and asset type. A model might do well on phishing and poorly on cloud identity alerts, and that tells you where to allow autonomy and where to require review. See how much autonomy an AI SOC agent should have.
Watch for drift
Log formats change, attackers change and models update. Rerun the reference set on a schedule and after any model or prompt change. Record the date and result.
Avoid vanity numbers
Be careful with figures like alerts handled per day or analyst hours saved when they come without error data. If a vendor or internal team quotes a percentage improvement, ask what reference set it was measured on and what the miss rate was. Do not use a number you cannot trace.
Quick check
What is the single most important metric?
Missed true positives, reviewed one by one. A miss on a real incident outweighs many saved minutes.
How big should the reference set be?
Big enough to include several examples of each important alert type, including rare serious ones. Start with what an analyst can label carefully and grow it.
Last reviewed: 19 October 2026.
