What is AI bias and how do you test for it?

AI bias is a systematic skew in what an AI system produces, one that leads to unfair or harmful outcomes for some people or groups. It can come from the data, the model design or the way people use the system. Testing means comparing outcomes across groups, not only measuring overall accuracy.

Where does bias come from?

NIST Special Publication 1270, Towards a Standard for Identifying and Managing Bias in Artificial Intelligence, says biases are endemic across technology processes and can lead to harmful impacts regardless of intent. It organizes the sources into three groups: statistical and computational, human, and systemic. That framing is useful because a fix in one group does not fix the others.

Statistical and computational bias comes from the data and the algorithm. Training data may under represent a group, labels may be noisy, or a proxy variable such as postal code may stand in for something it should not.

Human bias comes from the people who collect, label, build and use the system. A labeler's assumptions become the ground truth. A reviewer who over trusts an automated score stops thinking.

Systemic bias comes from institutions and history. If past decisions were unequal, a model trained to copy them will copy the inequality.

Is bias the same as being wrong?

No. A model can be highly accurate overall and still work far worse for one group, because the overall figure averages over the majority. Fairness questions live in the differences between groups.

What does testing look like?

Start by naming the decision and the people it affects. Then pick measures that fit the harm.

For a screening tool, compare selection rates across groups. For a classifier, compare false positive and false negative rates across groups, since the cost of each error differs. For a generative system, review outputs on matched prompts that differ only in a name, a dialect or a described background.

There is no single correct fairness metric. Several common definitions cannot all hold at once in the same system, so you have to choose what matters for your use and document why.

Constructed example: the resume screen

A hiring team trains a model on ten years of past hiring decisions to rank resumes. It scores well against those decisions. A review that compares selection rates by group shows women with career gaps rank lower. The past decisions penalized gaps. The model learned that pattern and amplified it.

The team removes the gap feature, checks for proxies, adds a human review of borderline cases and publishes the monitoring plan. The overall accuracy figure barely changes, which is exactly why it was never a useful alarm.

Why does this belong in security work?

Bias is a trust and risk problem alongside the safety ones. Poisoned or manipulated training data can introduce skew on purpose, which NIST's adversarial machine learning taxonomy covers as a family of attacks on AI systems. Regulators and customers also ask how automated decisions are governed. A documented test and monitoring process answers that.

How to check or apply this

1. Write down the decision the system supports, who is affected, and what a harmful error looks like.
2. Review the training data for coverage, label quality and proxy variables.
3. Measure outcomes by group using metrics you chose in advance, and record the reasoning.
4. Test with matched inputs that vary one attribute at a time.
5. Keep a human review path for contested or borderline decisions.
6. Monitor after launch. Data drifts, and a fair model can become unfair.
7. Keep versioned records of data, models and test results so you can explain a decision later.

Quick check

Sources

• NIST SP 1270, Towards a Standard for Identifying and Managing Bias in Artificial Intelligence
• NIST AI 100-2 E2025, Adversarial Machine Learning
• NIST AI Risk Management Framework

Last reviewed: 18 October 2026.