AI and Due Process: When Algorithms Decide Your Legal Rights
In 2016, an American named Eric Loomis was sentenced to six years in prison in Wisconsin. The sentencing judge relied in part on a score produced by an algorithm called COMPAS — a proprietary risk-assessment tool that predicted Loomis was at high risk of reoffending. Loomis asked to see how the algorithm reached that score. The company that built COMPAS refused, calling it a trade secret. Loomis argued this violated his constitutional right to due process. The Wisconsin Supreme Court disagreed. His case landed squarely at the intersection of artificial intelligence and the Fifth and Fourteenth Amendments.
What You'll Learn
By the end of this lesson you will be able to: • Define procedural and substantive due process under the Fifth and Fourteenth Amendments • Explain how AI risk-assessment tools like COMPAS are used in the criminal justice system • Analyze the constitutional challenges raised when opaque algorithms influence legal decisions • Evaluate reform proposals for making algorithmic decision-making consistent with due process rights
Due Process: The Constitutional Guarantee
The Fifth Amendment (1791) states that no person shall 'be deprived of life, liberty, or property, without due process of law.' The Fourteenth Amendment (1868) extends this protection against state governments, not just the federal government. Courts have interpreted due process in two ways: **Procedural due process** guarantees fair procedures before the government takes something from you — notice of the action, a hearing, the right to present evidence, a neutral decision-maker, and a reasoned explanation for the outcome. If any of these steps is missing, the process is unconstitutional regardless of whether the underlying decision was correct. **Substantive due process** limits what the government can do even if it follows all the right procedures. Certain fundamental rights — liberty, privacy, family autonomy — cannot be infringed at all without an extremely compelling reason, no matter how careful the procedure was.
A core element of procedural due process is that a defendant must be able to understand and challenge the reasoning behind a decision affecting their liberty. Courts have long held that sentencing based on false information — or information the defendant cannot see, test, or rebut — violates due process. The COMPAS dispute is about whether that principle extends to algorithmic scores whose internal logic is a trade secret.
How AI Has Entered the Justice System
Risk-assessment algorithms are now used at multiple stages of the criminal justice process in the United States: **Pretrial detention:** Tools like the Public Safety Assessment (PSA) score defendants on flight risk and danger to determine whether they are released before trial. Nine states used PSA in 2023. **Sentencing:** COMPAS, developed by Equivant (formerly Northpointe), scores defendants on recidivism risk on a scale of 1–10. Wisconsin, California, and other states have used it at sentencing. **Parole decisions:** Algorithmic risk scores influence parole board decisions in at least 17 states as of 2022. **Predictive policing:** PredPol (now Geolitica) used crime history data to predict where crimes would occur and direct patrols. Los Angeles, Chicago, and Santa Cruz all used it before banning it amid bias concerns. Santa Cruz became the first U.S. city to ban predictive policing entirely in 2020. A 2016 ProPublica investigation of COMPAS found that Black defendants were nearly twice as likely as white defendants to be falsely flagged as high risk of future crime — a disparity the company disputed but could not fully explain because the algorithm's details were proprietary.
The Constitutional Problem
The core due process problem with opaque AI in court is this: if an algorithm influences your sentence, you have the right to challenge the factors it weighed — but if those factors are a trade secret, challenge is impossible. **Loomis v. Wisconsin (2016):** The Wisconsin Supreme Court held that COMPAS did not violate due process because the score was only one factor among many, and Loomis had access to the general methodology even if not the source code. Critics noted that judges often defer heavily to algorithmic scores in practice, making the 'one factor among many' reasoning fragile. **State v. Flores (New Jersey, 2020):** A court found that the state's failure to disclose the full methodology of a risk-assessment tool used at sentencing violated due process — the first time a U.S. court took that position. **Formal challenges in federal courts** have mostly failed on standing or harmless-error grounds, but legal scholars argue the issue is unresolved at the federal constitutional level. No U.S. Supreme Court case has directly ruled on AI risk-assessment tools and due process as of 2026.
Match each due process concept to its correct definition.
Terms
Definitions
Drag terms onto their definitions, or click a term then click a definition to match.
Eric Loomis argued that using a COMPAS score at sentencing violated procedural due process. Which of the following best explains the core of his argument?
Reform Proposals and Where Law Stands Now
Scholars, legislators, and civil rights organizations have proposed several approaches to reconcile AI with due process: **Mandatory disclosure laws:** Several states have passed or proposed laws requiring that any algorithm used in sentencing be fully disclosed to defendants and their attorneys. Illinois passed the Pretrial Fairness Act (2021), which limits algorithmic tools at bail hearings. **Algorithmic impact assessments:** Proposed federal legislation such as the Algorithmic Accountability Act would require companies to audit tools for bias and accuracy before deploying them in high-stakes settings. **The right to explanation:** The EU's General Data Protection Regulation (GDPR, effective 2018) gives individuals a right to a meaningful explanation of automated decisions that significantly affect them — a model some U.S. advocates propose adopting domestically. **Banning risk scores at sentencing:** New York City's Local Law 49 (2017) requires audits of automated decision systems used by city agencies; some advocates argue no risk score should influence liberty deprivation without the ability to fully examine and challenge the underlying model. The fundamental tension remains unsettled: AI tools can make decisions more consistent and arguably less subjectively biased than individual judges — but consistency without transparency may just be bias at scale, hidden from scrutiny.
A state court uses an algorithm to recommend parole decisions. The algorithm is publicly documented and defendants receive a printed report explaining their score. Which due process concern does this design BEST address?
Draft a Model Algorithmic Due Process Policy
You are advising a state legislature that wants to allow AI risk-assessment tools at sentencing — but only in a way that is constitutional. 1. Review the four reform proposals covered in this lesson: mandatory disclosure, algorithmic impact assessments, the right to explanation, and banning risk scores entirely. 2. Draft a one-page policy (three to five paragraphs) that addresses all four of these questions: a. When may a judge consider an AI risk score? (At sentencing only? At bail hearings? Both?) b. What must be disclosed to the defendant? (Source code? Training data? Just the methodology?) c. What right does the defendant have to challenge the score in court? d. Who audits the algorithm for accuracy and bias, and how often? 3. For each policy choice you make, cite at least one legal principle from this lesson (procedural due process, substantive due process, the right to confrontation, etc.) to justify it. 4. Deliverable: a formatted policy draft titled 'Model Algorithmic Sentencing Transparency Act' with a brief title and numbered sections.
Want to keep learning?
Sign up for free to access the full curriculum — all subjects, all ages.
Start Learning Free