AI Bias and Fairness

AI is already being used to make or support decisions about people — in hiring, credit, healthcare, and criminal justice. These uses raise serious questions about fairness.

How AI bias happens

AI systems learn from historical data. If that history reflects discrimination — fewer women in senior roles, lower rates of loan approval for certain communities — the AI may learn to replicate those patterns. It is not that the algorithm is deliberately discriminatory; it has just learned what "success" looks like from a biased historical record.

Real examples

Amazon built a hiring AI in the 2010s that downgraded resumes containing the word "women" (as in "women in technology club"). It had learned from a decade of hiring decisions in a male-dominated industry. Amazon scrapped the tool when the bias was discovered.

COMPAS, a tool used in US courts to assess recidivism risk, was found to have significantly different false positive rates for Black defendants compared to white defendants — wrongly flagging Black defendants as high risk at roughly twice the rate.

Why bias is hard to fix

Defining fairness mathematically is harder than it sounds. There are multiple mathematical definitions of fairness, and it is often mathematically impossible to satisfy more than one at the same time. Decisions about which definition of fairness to use are ultimately value judgements, not technical ones.

What this means for NZ

New Zealand government agencies have committed through the Algorithm Charter to consider impacts on Maori and other groups when deploying algorithms. But the gap between commitment and rigorous implementation remains significant.

AI bias is not a technical problem with a technical fix. It reflects whose data was collected, whose interests shaped the system, and whose definition of "success" was encoded into the training objective.