How Large Language Models Predict Words: Tokens, Patterns, and You

When you type a message to an AI chatbot, it does not look up the answer in a database. It does not know facts the way you know your own name. Instead it does something stranger and more interesting: it predicts, one chunk of text at a time, what word most likely comes next given everything that came before. This is called next-token prediction, and it is the engine inside every large language model (LLM) you have ever used — ChatGPT, Claude, Gemini, Llama. Understanding how it works does not just explain AI. It reveals deep truths about the structure of human language itself, which is why linguists pay close attention to what LLMs can and cannot do.

By the end of this lesson, you will be able to: • Define token and explain how LLMs split text into tokens before processing it • Describe how training on large text corpora enables next-token prediction • Explain what self-attention means in a transformer model and why context determines meaning • Analyze what LLMs reveal about morphology, syntax, and semantics from a linguistics perspective

LLMs do not process text letter by letter or word by word. They split text into tokens — chunks that are roughly parts of words, whole short words, or occasionally single characters. Real tokenization examples from GPT-4 (verifiable at platform.openai.com/tokenizer): • 'The' → one token • 'unbelievable' → three tokens: 'un', 'believ', 'able' • 'ChatGPT' → three tokens: 'Chat', 'G', 'PT' • '2025' → one token • A single emoji can be 3 to 4 tokens Why does this matter? The model sees these pieces, not whole words. It learns that 'un-' often signals negation — not because anyone explained that rule, but because 'un-' appears in front of negative concepts millions of times in training data. The prefix's meaning emerges from statistical pattern alone. Most LLMs use Byte Pair Encoding (BPE), which merges the most frequent character sequences into single tokens. GPT-4 operates with roughly 100,000 distinct tokens in its vocabulary.

An LLM is trained by feeding it enormous amounts of text — the Common Crawl web corpus alone is over 270 billion words — and having it do one repeated task: predict the next token. Given the tokens 'The', 'cat', 'sat', 'on', 'the', predict what comes next. The model guesses. If it is wrong, the error signal flows backward through the network (backpropagation) and adjusts billions of numerical weights very slightly. Repeat this 10 trillion times and the model learns: • Grammar (verbs follow subjects) • Collocations ('fast food,' not 'quick food') • World knowledge ('the capital of France is' → 'Paris') • Conversational patterns ('How are you?' → typical responses) All of this emerges from statistics over human-written text — no rules were hand-coded. GPT-4 has an estimated 1.8 trillion parameters — numerical weights that collectively encode these patterns. The scale makes human intuition about 'how big' nearly impossible to grasp.

Before transformers — the architecture introduced in Google's landmark 2017 paper 'Attention Is All You Need' — language models processed text sequentially, losing earlier context. Transformers changed this with self-attention. Self-attention lets the model weigh the relevance of every previous token when predicting the next one. This solves a classic linguistics problem: lexical ambiguity. Consider the word 'bank': • 'She sat by the river bank and read her book.' → bank means riverbank • 'She went to the bank to deposit her paycheck.' → bank means financial institution The attention mechanism looks back at 'river' or 'deposit' and adjusts the representation of 'bank' accordingly. In linguistics this is called contextual disambiguation — and humans do it effortlessly without thinking about it. Attention also resolves anaphora (pronoun reference): in 'The trophy did not fit in the suitcase because it was too big,' attention helps determine whether 'it' refers to the trophy or the suitcase. The presence of 'too big' in the context of fitting signals the trophy — the suitcase would need to be 'too small.'

LLMs have become unexpected tools for theoretical linguists. Here is what the models reveal: **Morphology:** LLMs correctly handle irregular plurals (mice, not mouses), irregular past tenses (went, not goed), and derivational suffixes (-ness, -tion, -ly) purely from frequency patterns — without being given a single grammatical rule. This supports usage-based linguistics, which argues that language is learned from statistical exposure rather than innate rule systems. **Syntax:** LLMs pass many grammaticality judgments previously thought to require deep grammatical knowledge — correctly flagging 'The children seems tired' as ungrammatical. However they struggle with deeply nested structures like 'The horse raced past the barn fell,' a garden-path sentence where humans also stumble at first. **Semantics:** LLMs learn word meanings from context — distributional semantics. Linguist J.R. Firth wrote in 1957: 'You shall know a word by the company it keeps.' This is essentially the training objective of every LLM. Words appearing in similar contexts receive similar internal representations. **Limits:** LLMs struggle with grounding (connecting language to physical reality), pragmatics (understanding speaker intent beyond literal meaning), and consistent factual reasoning across long documents.