Fairness, bias, and decisions about people

Home Learn Fairness, bias, and decisions about people

In short

An AI system can work well on average and still serve some people poorly. Fairness means looking at who benefits, who faces errors, and whether someone can question a result. Accountability means a person or organization can explain and change the process.

Why it matters

A tool that sorts requests, translates instructions, or suggests opportunities can affect how quickly someone gets help. If the examples used to design it are narrow, unfamiliar language or circumstances may be handled worse. A pleasant answer to most people doesn’t reveal whose needs were missed.

How it works

First define the decision the system influences. Is it offering information, prioritizing a request, or making a recommendation about a person? Identify the groups and situations where an error would have different consequences. Use representative inputs and review outcomes in the context where the tool will be used. NIST’s AI Risk Management Framework calls for evaluating bias and fairness in the mapped use case. Its Measure playbook discusses performance across groups and intersecting groups, along with qualitative feedback from people affected.

An average accuracy figure can hide uneven service. Look at examples that were wrongly denied, delayed, or sent to the wrong path, and compare the consequences. Involve people who understand the work and the people receiving the result. Ask what information a reviewer sees, who can correct a mistake, and whether a person can reach a human route. A single metric cannot decide every question of fairness; the choice of task, categories, and threshold is itself a human decision.

Keep a record of the intended use, the examples reviewed, the limits you found, and the person who owns the process. If the tool is updated, check whether an improvement for one common case makes another case worse. UNESCO’s 2021 AI ethics recommendation connects fairness, transparency, and human oversight. That broad principle becomes practical when the affected person has an understandable path to challenge an outcome.

Example

Illustrative example: this scenario is invented.

Nora, 42, reviews a draft assistant that sorts questions for a public wellness fair. It recognizes short requests for parking, but a longer message about sensory access ends up in a general-information pile. Nora adds a clear access category and a human route for uncertain messages. She reviews whether the category works for varied wording rather than declaring success from the common requests alone. The measure is whether people reach appropriate help, not whether the sorting looks tidy.

She asks a volunteer to read several differently worded access questions and check whether the proposed route makes sense. Any unclear category goes to a person.

What it doesn’t mean

Fairness isn’t a claim that every output must be identical. A balanced dataset alone doesn’t fix a poorly chosen goal or an inaccessible appeal path. A human reviewer also needs enough context and authority to correct an outcome. Accountability requires a real owner who can change the workflow after someone points out a problem.

Try it

Write four made-up questions for a public community fair, including one with unusual wording and one about access needs. Ask any assistant to sort them into help categories. Look for a category that was forced without enough evidence. Write a human-review path for that case and decide what feedback would improve the categories.

Keep going

Sources

Share this page