Retrieval-Augmented Generation

A language model's knowledge is frozen at its training cutoff. GPT-4's weights encode patterns from text collected before early 2023. Claude's weights encode patterns from text collected before its own cutoff. No matter how capable the model is, it cannot tell you what happened last week, what your company's internal policy document says, or what the current price of a stock is — because none of that was in its training data. Retrieval-augmented generation, universally abbreviated RAG, is the architectural pattern that fills this gap.

The idea is elegantly simple: instead of expecting the model to remember relevant information from training, fetch that information from an external source at query time and inject it directly into the prompt. The model then generates its response by reasoning over the retrieved content, not just its frozen weights. This makes the model's effective knowledge updatable, domain-specific, and far more accurate on specialized or current topics — all without retraining a single parameter.

The RAG Pipeline Step by Step

A RAG pipeline has two major phases: ingestion and retrieval. During ingestion, a corpus of documents — company wikis, research papers, product manuals, historical records — is processed offline. Each document is split into smaller chunks, typically 200-500 tokens each, so that retrieved content is focused rather than sprawling. Each chunk is then converted into a dense vector embedding using an embedding model. An embedding is a list of hundreds or thousands of floating-point numbers that captures the semantic meaning of the chunk. Semantically similar chunks cluster near each other in this high-dimensional space. The embeddings are stored in a vector database alongside the original chunk text. During retrieval, when a user sends a query, the same embedding model converts the query into a vector. The vector database performs a nearest-neighbor search — finding the stored chunks whose embeddings are closest to the query's embedding in vector space. The top-k most similar chunks are retrieved and inserted into the prompt before the model generates a response. The model now has both the user's question and the most relevant context passages, and it synthesizes an answer from both.

Embeddings: Meaning as a Point in Space

The concept of an embedding deserves a closer look. An embedding model — such as OpenAI's text-embedding-3-large or Cohere's embed-english-v3.0 — is a neural network trained to produce vectors such that semantically similar text ends up near each other in the vector space. The sentence 'the dog chased the ball' will be embedded near 'a puppy ran after a toy' even though no individual word matches. 'The stock market fell sharply' will be far from both. This property — semantic proximity reflected in geometric proximity — is what makes RAG work. Keyword search would require the query and the document to share exact words. Embedding-based search finds conceptually related content even when the vocabulary differs entirely. For knowledge-base question answering, this is decisive: users ask questions in their own words; documents are written by authors using different vocabulary.

When RAG Is and Is Not the Right Tool

RAG is exceptionally well suited to question answering over a known corpus: customer support over a product manual, legal research over a case database, medical information retrieval over clinical guidelines. It is the right tool when the answer exists verbatim or near-verbatim in stored documents, and when freshness matters — the corpus can be updated without retraining the model. RAG is a poor fit when the required knowledge is genuinely absent from the corpus — then the model may confidently synthesize an answer from irrelevant retrieved chunks, a failure called hallucination from bad retrieval. RAG also struggles when questions require complex multi-hop reasoning across many documents simultaneously, since the retrieved chunks may not include all the necessary links. And RAG adds meaningful latency: embedding the query, searching the vector store, and injecting results all take time before the model even begins generating.