AI and Big Decisions in Society
Getting a job. Receiving bail or being held before trial. Being accepted to a school. Getting approved for a loan. Receiving a healthcare diagnosis. These are decisions that can shape the direction of a person's entire life. Increasingly, AI systems play a role in all of them. When AI influences consequential decisions at scale, the stakes for getting it right — and the costs of getting it wrong — become enormous.
AI in Hiring
Millions of job applications pass through AI screening systems before a human ever looks at them. These systems scan resumes for keywords, rank candidates, and sometimes conduct automated video interviews where AI analyzes facial expressions, word choice, and tone of voice to rate the candidate. The appeal is efficiency: a company receiving ten thousand applications cannot have human recruiters read every one carefully. AI can narrow the pool quickly. The concern is accuracy and fairness. In 2018, Amazon scrapped an AI recruiting tool it had developed after discovering it was systematically downgrading resumes that included the word women (as in women's chess club) because it had been trained on decades of historical hiring data in which men dominated the workforce. The AI had learned to replicate past patterns — patterns that reflected historical gender bias, not actual job performance.
When a resume screening system uses rigid keyword matching, it may reject strong candidates who described their experience differently, who come from different educational backgrounds, or whose resumes were written in a slightly different format. People who are already underrepresented may be filtered out at scale before a human ever has the chance to recognize their potential.
AI in Criminal Justice
In several U.S. states, courts use algorithmic risk assessment tools when making decisions about bail, sentencing, and parole. These systems analyze factors associated with a defendant's history and produce a risk score indicating how likely the person is to commit another offense. ProPublica's 2016 investigation into one widely used tool called COMPAS found that it was significantly more likely to label Black defendants as high-risk when they actually went on to commit no new offenses — a type of error called a false positive — while being more likely to label white defendants as low-risk when they went on to commit new offenses. The tool was not equally wrong in both directions. Defenders of these tools argue they are more consistent than human judges, who are also subject to bias and bad days. Critics argue that automating bias makes it more systematic and harder to challenge, and that affected individuals have a right to understand and contest the basis of decisions that affect their freedom.
AI in Healthcare
AI shows genuine promise in healthcare. Deep learning systems can identify cancerous cells in medical images with accuracy comparable to or exceeding specialist physicians in some studies. AI models have accelerated drug discovery dramatically. Predictive models can flag patients at risk of deterioration before obvious symptoms appear. But healthcare AI also faces serious equity challenges. Medical training data has historically overrepresented white patients from wealthy countries — meaning models may perform less well for patients from underrepresented groups. One study found that a widely used algorithm that predicted which patients needed extra care was significantly less likely to identify Black patients as needing care, because it used healthcare costs as a proxy for health needs — and Black patients had historically been given less care even when equally sick, resulting in lower costs.
Many AI systems use proxy variables — measurable inputs that stand in for the thing they actually want to predict. Using healthcare cost as a proxy for health need assumes that sicker people cost more. But if some groups historically received less care when they were sick, they cost less — and the proxy produces biased outputs. Identifying and questioning proxy variables is a critical skill for evaluating AI fairness.
The Right to Explanation and Human Oversight
Across all these high-stakes contexts, two principles appear consistently in AI governance discussions: the right to explanation and the requirement for human oversight. The right to explanation means that when an AI system makes or influences a significant decision about a person, that person should be able to find out that AI was used and understand the basis on which the decision was made. This is recognized as a right under GDPR in Europe. Human oversight means that a trained human being reviews AI-influenced decisions — especially when they can be appealed or when errors have serious consequences. AI should augment human judgment in high-stakes settings, not replace it without recourse.
Match each AI-in-society concept to its correct description.
Terms
Definitions
Drag terms onto their definitions, or click a term then click a definition to match.
Amazon's hiring AI downgraded resumes containing the word 'women' because it was trained on historical hiring data. What does this demonstrate?
Why is using healthcare cost as a proxy for health need potentially biased?
Scrutinize a High-Stakes Decision
- Step 1: Choose one high-stakes decision domain: hiring, criminal justice, healthcare, education admissions, or loan approvals.
- Step 2: Describe how an AI system is currently being used or could plausibly be used in that decision.
- Step 3: Identify one proxy variable the AI might use, and explain one way that proxy could introduce bias against a specific group.
- Step 4: Describe one harm that could result from a false positive error and one harm from a false negative error in your chosen domain.
- Step 5: Write two requirements you would insist on if this AI were deployed at your school, in your city, or in a court in your community. For each requirement, explain the specific harm it is meant to prevent.