Algorithmic Bias, Rigorously

The word 'bias' does a lot of work in everyday conversation — it describes prejudice, unfairness, one-sidedness, a tilted perspective. In machine learning the term carries a precise technical meaning that is related to but distinct from all of these. Before you can reason about algorithmic bias, fix it, or audit a system for it, you need to know exactly what it is and what it is not. This lesson establishes that foundation with precision.

Two Meanings of Bias in Machine Learning

In the statistical and machine learning literature, 'bias' first appeared as a mathematical property of estimators. An estimator is biased if its expected value differs from the true value it is trying to estimate. In the bias-variance tradeoff of supervised learning, bias refers to systematic error introduced by a model's assumptions — a linear model applied to a curved relationship will be biased because the model family cannot represent the truth. Algorithmic bias in the fairness sense is different. It refers to a systematic disparity in a model's outputs — errors, predictions, or decisions — that correlates with membership in a socially meaningful group. The group membership is typically defined by attributes like race, gender, age, disability status, national origin, or socioeconomic class. The disparity is not random noise; it is patterned and predictable based on group membership. For this module, when we say 'algorithmic bias' without qualification, we always mean the second sense: differential error rates or systematically unequal outcomes for socially defined groups.

Consider a hiring algorithm that screens resumes. One possible normative standard: the algorithm should accept candidates at the same rate across demographic groups. Another: the algorithm should accept candidates at rates proportional to their performance in the job, regardless of whether that performance differs across groups. A third: the algorithm should make no predictions that would not be made if the protected attribute (race, gender) were legally permissible to consider. These are three different standards. A system can satisfy one while violating another. Bias analysis is always relative to a specified standard. This is not relativism or an excuse to ignore fairness. It is a recognition that fairness is a multi-dimensional concept and that specifying which dimension you are measuring is a precondition for meaningful measurement. Later lessons in this module will examine the formal mathematical versions of these standards — demographic parity, equalized odds, predictive parity, and more — and explore why they cannot all be simultaneously satisfied in general.

Distinguishing Algorithmic Bias from Related Concepts

Algorithmic bias is often conflated with four related concepts, each of which is distinct. Random error: all statistical models make errors. If those errors are distributed uniformly across groups — the model is equally likely to be wrong for any individual regardless of group membership — that is random error, not bias. Bias requires the errors to correlate with group membership. Technical error: a bug in code, a miscalibrated sensor, a corrupted dataset. These cause wrong outputs but are not systematically related to social group membership. They can coexist with bias but are not the same thing. Human prejudice: individual humans hold biased beliefs and make biased decisions. Algorithmic bias is related to human prejudice because the training data and design choices for ML systems are created by humans. But a system can exhibit algorithmic bias even if no individual involved in creating it was consciously prejudiced — the bias can enter through historical data that reflects past discrimination, through proxy variables that correlate with protected attributes, or through design choices that were neutral in intent but unequal in effect. Disparate impact vs. disparate treatment: legal concepts from employment law that are relevant but not identical to algorithmic bias. Disparate treatment is intentional discrimination based on a protected attribute. Disparate impact is an employment practice that, though facially neutral, falls more heavily on a protected group. Algorithmic systems can produce disparate impact without disparate treatment, and assessing them requires different tools.

The Role of Protected Attributes and Proxies

A protected attribute is a characteristic of a person that should not be the basis for certain decisions — race, sex, religion, national origin, disability status, and age (in many legal contexts). Many jurisdictions prohibit using protected attributes directly in decisions like hiring, lending, housing, and criminal sentencing. Removing the protected attribute from a model's input features does not solve the problem. This is one of the most important technical insights in the fairness literature. Many other features — zip code, name, school attended, language style in free-text fields — are correlated with protected attributes and act as proxies. A model trained on these features will implicitly use the protected attribute through its proxies, often with more statistical power than if the protected attribute had been included directly. In the United States, zip code correlates strongly with race due to historical residential segregation. A lending model that includes zip code but not race is still, in effect, considering race. The word for this — using a neutral variable to achieve the effect of a prohibited variable — is laundering. Algorithmic systems can launder protected attributes through proxies even when designed by well-intentioned engineers who genuinely believe they have removed sensitive information.