Memory for AI Agents

How AI agents remember beyond the context window: working memory, long-term stores, episodic vs semantic memory, and the pitfalls.

Ask an agent about a decision it made yesterday and, by default, it has no idea. Close the session and everything it knew evaporates. Memory is the layer that fixes this — and building it well is harder than it looks.

Why the context window isn't enough

An LLM's context window holds everything the model can see for a given call: system prompt, tool definitions, conversation history, retrieved documents. It's large in modern models, but it's still finite, and every token in it costs money and attention. More importantly, each API call is stateless — the model itself remembers nothing between requests. Whatever isn't in the prompt this time doesn't exist.

That's fine for a single-turn chatbot. For an agent that runs for hours or acts consistently for the same user over weeks, it's a hard limit. Two problems show up in practice:

  • Loss within a session. A long-running agent task can generate more transcript than fits in the window, forcing something to be dropped or summarized.
  • Loss across sessions. Without an external store, the agent starts from zero every time — re-asking questions the user already answered, repeating mistakes it already made and fixed.

Memory is the general term for the systems that address both: keeping the right information available now, and carrying the right information forward.

Short-term (working) memory

Short-term memory is whatever lives in the current context window: the running transcript, a scratchpad the agent writes intermediate reasoning to, tool outputs from this task. It's fast — no retrieval step, no external call — but it's bounded by window size and it disappears when the session ends unless something explicitly saves it.

Common working-memory patterns:

  • The transcript itself. Simplest form — just the message history so far.
  • A scratchpad. A dedicated section (in-context or a temp file) where the agent jots plans, intermediate results, or a todo list, separate from the conversational flow.
  • Structured state. In frameworks like LangGraph, working memory is often an explicit state object threaded through each step, not just raw text.

Working memory answers what's relevant to finishing this task right now. It's cheap and precise, but it's not durable.

Long-term memory

Long-term memory is an external store — a database, vector index, key-value store, or plain files — that persists after the session ends. The agent explicitly writes to it and retrieves from it; nothing lands there automatically. Tools like Mem0, Letta, and Zep, or a plain vector store such as Redis, pgvector, or MongoDB Atlas Vector Search, are common building blocks, but the pattern matters more than the product: write selectively, retrieve relevantly, expire deliberately.

Episodic vs. semantic memory

Long-term memory is usually split into two kinds, borrowing the distinction from cognitive science:

  • Episodic memory — what happened. Specific events, tied to a time and context: on July 12, the user asked to switch their billing plan and the agent completed it. It's a log of experience.
  • Semantic memory — durable facts, independent of when they were learned. The user is on the annual plan. This customer prefers terse responses. Semantic memory is usually distilled from episodic memory — you extract the stable fact out of the one-off event.

A third category shows up often enough to mention: procedural memory — learned rules or routines for how to do something, like an updated system prompt or a corrected tool-use pattern. It's less mature in most current tooling than episodic and semantic memory, but it follows the same write/retrieve shape.

The distinction matters because it drives what you store and how you retrieve it. You don't want a semantic fact (user's timezone is CST) cluttered with every episode that ever mentioned it, and you don't want an episodic log flattened into a single fact when the specifics matter (which support ticket, which version of the API).

Getting memory into context: retrieval

Long-term memory only helps if it makes it back into the prompt at the right moment. That's a retrieval problem — the same one covered in RAG vs. tools vs. long context: embed the query, search the memory store for relevant records, inject the top matches into context before the model call.

A typical read looks like:

1. User/task arrives → agent forms a retrieval query
   (e.g., "user's plan preferences," "past errors with this API")
2. Query the long-term store (vector search, keyword filter, or both)
3. Rank and select top-k matches, filtered by recency/relevance
4. Inject selected memories into the prompt, clearly scoped
   (e.g., under a "Known context" section, not mixed into instructions)
5. Model reasons/acts with both working memory and retrieved memory

Retrieval for memory is mechanically the same as RAG over documents — same embeddings, same vector search — but the corpus is the agent's own past, not a static knowledge base.

Deciding what to write

The harder question is the write side: what earns a permanent record, and when. Left unconstrained, agents either write nothing (and gain nothing from long-term memory) or write everything (and drown retrieval in noise). Practical approaches:

  • Explicit triggers. Write only on specific signals: task completion, an explicit user correction (no, I prefer email over Slack), or an error the agent had to work around.
  • End-of-task consolidation. Instead of writing continuously, have the agent (or a separate summarization step) review a completed session and extract the handful of facts worth keeping — turning episodic detail into semantic summary.
  • User-visible memory. Many production agents show users what got saved and let them edit or delete it. This does double duty: it catches bad writes early and it addresses the privacy problem below.

A minimal stored record often looks like:

{
  "id": "mem_8f21",
  "type": "semantic",
  "subject": "user:4471",
  "content": "Prefers responses under 200 words; flags anything longer.",
  "source_episode": "session_2026-07-14T18:02Z",
  "confidence": 0.9,
  "created_at": "2026-07-14T18:03:11Z",
  "last_used_at": "2026-08-02T09:41:00Z"
}

Fields like source_episode, confidence, and last_used_at aren't decoration — they're what let you debug a bad retrieval later or prune memories nobody's touched in months.

Compaction and summarization as memory hygiene

When a session runs long enough that the transcript threatens to overflow the window, something has to give. This is context engineering territory: compaction (dropping or truncating older turns) and summarization (collapsing older turns into a condensed summary that's kept in-context) are the standard techniques.

Treat this as memory hygiene, not just a context-length fix:

  • A summary that stays in the working context is short-term memory in condensed form.
  • If any part of that summary is worth keeping past this session, write it to long-term memory explicitly — don't assume compaction alone gives you persistence.
  • Summarize before you're forced to; summarizing under pressure (context nearly full) tends to produce worse cuts than summarizing on a deliberate schedule.

Pitfalls

Memory systems fail in a few predictable ways:

  • Staleness. A stored fact (user is on the free plan) goes out of date and the agent keeps acting on it. Memories need either an expiry, a confidence score that decays, or a way to be overwritten when new information contradicts them.
  • Unbounded growth. Without pruning, a long-term store accumulates redundant or superseded entries, which slows retrieval and dilutes relevance. Deduplicate on write; periodically retire low-value or unused memories.
  • Irrelevant retrieval. Pulling the top-k nearest memories by embedding similarity doesn't guarantee they're useful for the current task — semantically similar isn't the same as currently relevant. Filter by recency, source, or explicit tags in addition to similarity.
  • Privacy. Long-term memory about a specific person is personal data, full stop. It needs the same handling as any other stored PII: scoping per user, a way to view and delete what's stored, and clear boundaries on what gets written in the first place (don't silently log sensitive details just in case).

None of these are solved by a clever prompt — they're store design and lifecycle decisions, made up front.

Where memory fits in the bigger picture

Memory isn't a bolt-on feature; it's one of the core systems an agent needs alongside tool use and retrieval, which is why it's a load-bearing piece of how to build your first AI agent. Start with working memory and a simple scratchpad, add long-term storage only once you have a concrete case for persistence across sessions, and be deliberate about the write path — that's usually where memory systems succeed or fail.

← Back to Articles