The Memory System

Every language model starts each forward pass with no memory of anything outside its current context window. For a simple chatbot, this is manageable — the entire conversation fits in the window. For an agent executing a 50-step research task over hours or days, it is a crippling limitation: the task accumulates far more state than any context window can hold, and the agent must continue working reliably across multiple sessions. The memory system is the architectural component that makes continuity possible.

Four Types of Agent Memory

Researchers and practitioners have converged on a taxonomy of four memory types, each implemented differently and serving a different purpose. In-context memory is the simplest: it is the content of the current context window itself — the system prompt, the conversation so far, tool call results, and any injected text. It is fast (no retrieval needed), perfectly accurate (exactly what was written), and ephemeral (gone when the context window is cleared). It is limited by the context window size, which is why all other memory types exist. External memory stores information outside the model in a retrievable database. Two main forms are used: a key-value or document store for exact lookup (retrieve a specific record by ID), and a vector database for semantic retrieval (retrieve records whose meaning is similar to a query string). External memory is the solution when the total relevant knowledge exceeds what fits in context. The agent uses a retrieval tool to fetch relevant chunks into the context window as needed. In-weights memory is the parametric knowledge baked into the model's weights during training — everything the model 'knows' about history, science, programming, and the world without looking anything up. It is always available and requires no retrieval, but it is static (frozen at training time), may be incorrect, and is not updatable at inference time. In-cache memory (also called KV cache) is a systems-level optimization: the intermediate computation states from previously processed tokens are cached so that long repeated prefixes (like a fixed system prompt) do not need to be recomputed on every call. It reduces latency and cost but does not add new information — it simply avoids redundant computation.

Memory Management Strategies

Long-running agents need explicit strategies for managing what stays in context and what gets offloaded. The three most common strategies are summarization, sliding window, and selective retrieval. Summarization compresses earlier parts of the conversation or task history into a shorter summary before they are dropped from context — the agent retains gist without token-by-token detail. The sliding window approach drops the oldest messages from context once the window is full, keeping only the most recent N tokens. This is simple but loses information about early task state. Selective retrieval uses a retrieval tool to fetch only the specific past context relevant to the current step, rather than maintaining a linear history at all — more complex to implement but scales indefinitely. Real agents often combine strategies: a sliding window for recent conversation, a summary stored to external memory, and a vector retrieval tool for long-term knowledge. The choice depends on the task's memory access patterns — does the agent need to refer back to the beginning frequently, or only to the most recent state?