Evaluating AI Systems for Safety

How do you know if an AI system is safe? The question sounds simple. The answer is not. Safety is not a single property — it is a family of properties that include accuracy, fairness, robustness, reliability, alignment with intended use, and absence of harmful outputs. Measuring each requires different methodology, and measuring all of them together on a real system is a genuinely hard scientific and engineering problem. This lesson is about evaluation: how safety properties are defined, measured, and — critically — where measurement breaks down.

Benchmarks: The Standard Measure and Its Limits

The dominant paradigm for evaluating AI capability is the benchmark: a curated dataset of inputs with known correct outputs, used to produce a summary accuracy score. Benchmarks like ImageNet (image classification), SQuAD (reading comprehension), and MMLU (multidomain knowledge) have driven enormous progress by providing a common measuring stick that allows researchers worldwide to compare systems. But benchmarks have well-known failure modes as safety evaluation tools. Benchmark saturation: when multiple systems reach near-human or superhuman performance on a benchmark, the benchmark no longer distinguishes between them. MMLU was considered a demanding test of language model knowledge; by 2024, frontier models were approaching 90% accuracy on it, well above expert-human performance on many categories. Benchmark contamination: large language models are trained on enormous text corpora scraped from the internet. If benchmark questions and answers appear in that training data, the model may have memorized answers rather than reasoning its way to them. Evaluating a model on data it may have trained on produces inflated and untrustworthy scores. Goodhart's Law in benchmarks: when a benchmark becomes a target, optimizing for it displaces the actual goal. Models can be fine-tuned to perform well on specific benchmarks without improving on the underlying capability the benchmark was designed to measure. High benchmark scores can coexist with striking failures on closely related tasks.

Fairness Metrics and Their Tensions

A safety evaluation that only measures aggregate accuracy will miss disparate impacts. A loan classifier that is 90% accurate overall may be 95% accurate for one demographic group and 75% for another. Fairness metrics attempt to make these disparities visible. Demographic parity requires that a positive outcome be predicted at the same rate across groups. If 30% of applicants from Group A are approved and only 18% from Group B, demographic parity is violated regardless of whether the groups have different underlying credit risk. Equalized odds requires that the true positive rate and the false positive rate be equal across groups. A recidivism classifier satisfies equalized odds if it correctly identifies defendants who will reoffend at the same rate for all racial groups, and incorrectly flags defendants who will not reoffend at the same rate for all groups. Calibration requires that predicted probabilities match actual outcomes: a model that says 70% probability of default should be right 70% of the time, equally across groups. The Impossibility Theorem of fairness (Chouldechova 2017, Kleinberg et al. 2016) proves that demographic parity, equalized odds, and calibration cannot all be satisfied simultaneously when base rates differ between groups. Every fairness metric embodies a value judgment about which inequalities are acceptable. Choosing a metric is a moral decision, not only a technical one.

Evaluating Large Language Models for Safety

Large language models present special evaluation challenges. Their outputs are open-ended natural language rather than fixed-label classifications, which means standard accuracy metrics barely apply. Evaluation must cover a vastly larger behavioral space. Harmfulness evaluation attempts to measure whether a model produces harmful outputs — misinformation, instructions for dangerous activities, hate speech, manipulation. Evaluators construct test sets of prompts designed to elicit harmful responses and measure refusal rates and harm severity. The challenge: the space of possible harmful outputs is essentially unbounded, and a model can fail on prompts not in the test set. Honesty and calibration evaluation tests whether a model knows what it does not know — whether it expresses uncertainty appropriately, admits ignorance, and does not confidently confabulate false information. TruthfulQA is a benchmark specifically targeting questions where language models frequently produce plausible-sounding but false answers. Alignment evaluation attempts to measure whether a model behaves in accordance with human values — helpfulness, harmlessness, honesty — across diverse situations. Constitutional AI and RLHF produce models evaluated partly by human raters who score outputs on these dimensions. The reliability of human ratings as a gold standard is itself contested: different raters disagree, raters can be influenced by fluency, and what is helpful or harmful is context-dependent.