Interpretability: The Understanding Gap

A frontier language model has hundreds of billions of parameters — numerical values distributed across thousands of matrices, processed through dozens of layers of computation. When it produces an output, the computation that generated that output has involved billions of floating-point multiplications and additions. No human can trace that computation in detail. We can observe the input and the output; the middle is a black box. This is the interpretability problem: we build and deploy powerful AI systems whose internal computations we do not understand. The problem is not merely academic. Deployed AI systems make consequential decisions — in healthcare, criminal justice, credit, hiring, and military applications — and the inability to explain why a decision was made creates accountability, safety, and trust problems that have no easy technical resolution.

What Interpretability Means and Why It Matters

Interpretability research aims to answer questions of the form: why did this model produce this output? Which parts of the input were most influential? What internal representations does the model form? What is the model 'doing' computationally when it solves a task? These questions have different answers depending on the level of analysis. At the coarsest level, input attribution methods try to identify which input tokens or pixels most influenced the output — a saliency map highlights which words in a sentence most affected a sentiment classification. At a finer level, probing classifiers test whether specific model layers contain linearly decodable information about specific concepts (does layer 12 of a language model represent syntactic structure?). At the finest level, mechanistic interpretability attempts to reverse-engineer specific circuits — subgraphs of the neural network that implement specific algorithms. Why does this matter practically? Consider a hospital deploying a model that reads patient notes and flags high readmission risk. The model achieves 85% accuracy — substantially better than the clinical baseline. The hospital wants to deploy it. A clinician asks: for this specific patient, why is the model predicting high risk? If the model is a black box, the answer is 'because the model says so.' That is not an acceptable clinical justification. The clinician cannot verify whether the model is tracking genuine medical predictors or spurious signals. They cannot explain the decision to the patient. They cannot override the model with medical knowledge if they cannot see its reasoning.

Current interpretability methods fall into two broad categories: post-hoc and intrinsic. Post-hoc interpretability applies to a trained model after the fact. The model is treated as a fixed function, and tools analyze its behavior. LIME (Local Interpretable Model-agnostic Explanations) fits a simple interpretable model (a linear function) to a model's behavior in a local neighborhood around a specific input, using that linear approximation as an explanation. SHAP (SHapley Additive exPlanations) uses game-theoretic concepts to assign each input feature a contribution score for a specific prediction. These methods have significant limitations. They describe behavior in a local neighborhood, not the model's actual computation. Two very different internal mechanisms can produce the same local explanation. Adversarial examples — inputs slightly modified to cause misclassification — reveal that post-hoc explanations can be stable even as the model's output flips. If the explanation does not change when the output changes, the explanation is not capturing the actual decision mechanism. Intrinsic interpretability trains models that are interpretable by design — decision trees, rule lists, linear models. These are interpretable because their structure is human-readable. The limitation is a capability-interpretability tradeoff: for most complex tasks, interpretable models are substantially less capable than deep neural networks. Choosing full interpretability means accepting a lower performance ceiling. Mechanistic interpretability is a newer research direction, pioneered partly by Anthropic's interpretability team and researchers like Chris Olah. It attempts to understand what neural networks compute at the circuit level — identifying specific neurons, attention heads, and their interactions that implement specific algorithms. Early results are striking: circuits have been found that implement induction (detecting repeated patterns), indirect object identification in language, and modular arithmetic. But these circuits have been found in small models or specific small components of large models; scaling mechanistic understanding to an entire frontier model remains an open problem.

The Stakes and Open State of the Problem

The interpretability gap is not narrowing as fast as capability is advancing. Models are becoming more powerful faster than interpretability tools can explain them. This creates a widening accountability deficit: as AI systems take on more consequential roles, our ability to audit their reasoning decreases relative to the complexity of that reasoning. From a safety perspective, interpretability is critical for detecting misalignment — the concern that a model has learned objectives or values that differ from what its designers intended. Without the ability to examine what a model is optimizing for internally, detecting misalignment requires purely behavioral testing, which is subject to exactly the distribution shift and brittleness problems discussed in previous lessons. A model might behave correctly under all tested conditions while pursuing a subtly different goal that only manifests on out-of-distribution inputs or in novel situations. The EU AI Act and similar regulatory frameworks require explanation of automated decisions in high-stakes contexts. The legal concept of 'right to explanation' exists in the GDPR. These legal requirements create practical pressure: you cannot deploy a black-box model for certain decisions if you are legally required to explain them. Current post-hoc interpretability methods do not reliably produce explanations that meet legal standards of accuracy. Mechanistic interpretability remains one of the most important open problems in the field — not because it would necessarily be exploited by bad actors if solved, but because without it, the field cannot verify that its most powerful systems are doing what they are supposed to be doing.