How to calibrate risk estimates: a 20 minute team exercise

To calibrate risk estimators, ask them for 90 percent confidence ranges on ten questions whose answers you know, then count how many ranges contained the true value. If fewer than about nine did, the team is overconfident and should widen its ranges. Repeat after feedback to see improvement.

Why bother

Risk numbers often come from expert judgment, since hard data is thin. NIST SP 800-30 notes that expert judgment plays a more visible role in semi-quantitative work and that rigor drops when subjective calls are hidden or uncertainty is significant. Calibration does not remove judgment. It trains it, and shows how far to trust it.

The exercise

Allow 20 minutes and four to eight people.

1. Prepare ten factual questions. Choose things with a number as the answer that nobody will know exactly, such as the length of a river, a year something was founded, or the number of employees at a famous company. Check each answer from a reliable source before the session.
2. Set the rule. For each question, write a low and a high value so that you are 90 percent sure the true answer is between them.
3. Write independently. No talking until everyone has finished.
4. Reveal the answers. Each person counts how many of ten ranges contained the true value.
5. Discuss. Most people hit far fewer than nine, often five or six. That is overconfidence, and it is normal.

What the result means

• Nine or ten hits: well calibrated, or ranges were wide. Check that the ranges were not uselessly wide.
• Six to eight hits: mild overconfidence. Widen ranges a bit.
• Five or fewer: strong overconfidence. Ranges should be much wider, and your point estimates deserve less trust.

This is a teaching tool, not a scientific test, since ten questions is a small sample.

Apply it to risk estimates

Before the next risk workshop:

1. Remind people of their own miss rate from the exercise.
2. Ask for ranges first, then a most likely value inside them.
3. Ask each estimator to name one reason the true value could be outside their range, then adjust.
4. Compare ranges across people. A large disagreement is information. Discuss the evidence each person used.

A useful trick

Instead of asking for a single number, ask "would you take a bet that the answer is inside this range, paid at nine to one if you are right and a loss if you are wrong?" People who hesitate should widen the range.

Keep a track record

When incidents or near misses happen, compare them against earlier estimates. Over time you learn which estimators and which kinds of risk run optimistic. Record the comparison in your register notes.

Pair with shared evidence

Give estimators the same data before they estimate: incident history, log summaries, test results and recent changes. Calibration helps people use evidence well. It does not replace it.

Quick check

Sources

• NIST SP 800-30 Rev. 1, Guide for Conducting Risk Assessments

Last reviewed: 19 October 2026.