The Limits of Generative AI

Everything in this module so far has been about what generative AI can do, and it genuinely is remarkable. But a complete picture requires honesty about what these systems cannot do, where they fail, and why trusting their output uncritically can cause real harm. This is not a reason to fear or avoid these tools — it is a reason to use them wisely. The most capable users of generative AI are not the ones who trust it most; they are the ones who understand its limitations best.

Hallucination: Confident and Wrong

The most important limitation to understand is hallucination. A language model can produce text that is fluent, detailed, confidently stated, and completely false. This is called hallucination. Hallucination happens because the model does not know what it does not know. When asked a question it cannot answer reliably, it does not say 'I am not sure' and stop. Instead, it continues the next-token prediction process and generates text that sounds like a reasonable answer — even if the facts underneath are invented. Examples are easy to find. Ask a model to cite sources for a claim and it will often produce plausible-sounding but entirely fictitious academic papers — complete with realistic author names, journal titles, and publication years, none of which exist. Ask it about a person's biography and it may confidently include events that never happened. This is not lying in the human sense — the model has no intent. It is a statistical system producing statistically plausible text, and sometimes plausible is not the same as true.

The second major limitation is the absence of true understanding. This is a subtle but important distinction. A model can explain quantum entanglement in beautifully clear prose, pass a medical licensing exam, and discuss the philosophical implications of consciousness — but it does not understand any of it the way a physicist, doctor, or philosopher does. It has learned the statistical structure of how humans write about these topics. It can generate text that is correct and useful — but not because it has the underlying insight a human expert has. This matters in practice because it means the model cannot reliably detect when it is wrong. A human expert who makes an error often has a nagging sense of uncertainty. A language model has no such sense — it generates incorrect text with the same fluency and confidence as correct text. The stylistic signals you use to evaluate human writing (Does this sound confident? Is the reasoning step-by-step?) do not reliably distinguish correct from incorrect model output.

Bias and the Training Data Problem

A generative model is a statistical mirror of its training data. If that data contains biases — and all large datasets do — the model will reflect them. These biases are not always obvious. Representation bias: If the training data over-represents some perspectives and under-represents others (for example, predominantly English-language text from Western sources), the model's 'default' outputs will reflect those dominant perspectives. It may perform worse on topics, languages, and cultural contexts that were less represented in training. Historical bias: Training data is a snapshot of human writing up to a certain point. It encodes the assumptions, stereotypes, and blind spots present in that writing. A model trained before widespread awareness of a social issue will reflect older, less nuanced views on that issue. Amplification: Models can amplify biases that were subtle in the training data, because they learn to produce the most statistically likely output — which can exaggerate patterns rather than averaging them out. None of these can be completely eliminated. Researchers work to reduce their impact through careful data curation and fine-tuning, but no current model is bias-free.