Transparency and Explainability

When a judge sentences you to prison, you have the right to know the reasons. When a bank denies your mortgage application, the Equal Credit Opportunity Act requires that you receive an explanation. These are not arbitrary bureaucratic requirements — they are mechanisms of accountability. If you know the reason for a decision, you can contest it, identify errors, and appeal. Now consider that consequential decisions — loan approvals, parole recommendations, medical diagnoses, job screenings — are increasingly made or influenced by machine-learning models whose internal logic is not easily accessible to anyone, including their creators. This lesson is about why that opacity exists, what researchers are doing to reduce it, and what is at stake.

Why Are Many AI Systems Opaque?

The short answer is that the opacity of modern neural networks is a side effect of their power. A large neural network may have billions of parameters — numerical weights distributed across many layers of computation. When the network processes an input, it computes a sequence of transformations through all of those layers. The final output — a classification, a score, a generated text — is the result of billions of multiplicative interactions. There is no single rule that summarizes what happened. The computation is distributed across the entire network in a way that resists reduction to a simple explanation. This is fundamentally different from a decision tree or a linear regression model, where you can read off exactly which features contributed to a decision and by how much. Deep neural networks trade interpretability for representational power — and for many tasks, the representational power wins handily. Two terms are often used interchangeably but are worth distinguishing: Transparency refers to the ability to inspect the model itself — to see its architecture, weights, and computational process. Explainability (also called interpretability) refers to the ability to generate a human-comprehensible account of why a model produced a particular output for a particular input. A model can be fully transparent (its weights are public) but still not explainable (no one can understand those billions of weights). Conversely, a model can be explainable via post-hoc analysis even if it is not directly interpretable.

Researchers have developed a range of techniques to make AI decisions more explainable, even when the underlying model is opaque. SHAP (SHapley Additive exPlanations): Based on game theory's Shapley values, SHAP attributes each feature's contribution to a particular prediction. If a model denies a loan application, SHAP can estimate how much each feature (income, credit score, ZIP code) contributed to that denial. The attribution is mathematically principled but is a post-hoc explanation — it explains the model's behavior without necessarily revealing the model's true internal reasoning. LIME (Local Interpretable Model-agnostic Explanations): LIME fits a simple, interpretable model (like a linear regression) locally around a specific input to approximate what the complex model is doing in that neighborhood. It gives a local explanation — valid near that data point — without claiming to explain the whole model. Attention visualization: In transformer models, attention weights indicate which parts of the input the model 'attended to' when producing its output. These are often visualized as heatmaps. However, research has shown that attention weights do not necessarily correspond to causal importance — a model can attend to a word without that attendance being the reason for its decision. Concept-based explanations: Rather than attributing to raw features, these methods identify high-level human concepts (e.g., 'zebra stripe pattern') that the model uses and explain predictions in those terms.

The Case for a Right to an Explanation

The EU's General Data Protection Regulation (GDPR), adopted in 2018, includes provisions that have been interpreted as creating a limited right to an explanation for automated decisions. Article 22 allows individuals to request human review of decisions made solely by automated processing when those decisions 'significantly affect' them. The scope and enforceability of this right have been actively debated by legal scholars, but the underlying intuition is clear: consequential automated decisions should be contestable, and contesting a decision requires understanding its basis. Opponents of strong explainability requirements raise several objections: Accuracy vs. interpretability trade-off: More interpretable models (linear models, decision trees) are often less accurate than opaque models (deep neural networks). Mandating interpretability might mean mandating worse predictions — and in medical diagnosis, a worse prediction may cause more harm than the opacity. Fidelity problems: Post-hoc explanations approximate the model's behavior but do not perfectly reproduce its reasoning. An explanation that is comprehensible but inaccurate may be worse than no explanation — it provides false confidence. Propriety and gaming: If the explanation reveals the features that lead to approval, applicants will optimize for those features rather than the underlying qualities the features are meant to measure.