How AI is Evaluated

Model evaluation sounds like a technical topic — but the questions it raises are fundamentally about fairness, accountability, and what we mean by "working".

What does it mean for a model to work?

The obvious answer is "it gets the right answer". But right for whom, in what circumstances, and compared to what baseline? A model that is 95% accurate overall might be only 70% accurate for a specific demographic group — which could be far worse than the 85% accuracy it replaced for that group.

Benchmark performance vs. real-world performance

AI systems are often evaluated on standard benchmarks — test datasets designed to measure capability. A model might perform brilliantly on the benchmark and poorly in real-world use, because real-world data is messier, more diverse, and different from the test data in ways that matter.

The problem of distributional shift

AI models perform well when the data they encounter in deployment looks similar to the data they were trained on. When the real world changes — a new disease variant, a market shift, a cultural event — model performance can degrade unexpectedly. This is called distributional shift and it is a significant practical challenge.

Who evaluates the evaluators?

Most AI systems are evaluated by the companies that build them, using criteria those companies choose. Independent evaluation and external auditing of AI systems is still rare, though it is being increasingly discussed in policy circles. The EU AI Act requires third-party conformity assessment for high-risk AI applications.

Asking "how was this AI evaluated?" and "by whom?" is one of the most important questions you can ask about any AI system being applied to real-world decisions.