What Is AI Agent Memory?

A large language model has no memory. Every request is answered from scratch, using only the tokens you put in front of it. The illusion of continuity in a chat interface comes from replaying the transcript on every turn — and that illusion breaks the moment the transcript outgrows the context window.

For a chatbot, that's an annoyance. For an agent that runs for weeks across hundreds of sessions, it's the central engineering problem. AI agent memory is the layer that solves it: a store that outlives any single request, that the agent writes to and reads from deliberately, and that stays accurate as it grows.

This guide covers what such a system has to do, the vocabulary used to describe it, and the design decisions that separate a memory layer from a pile of embeddings.

Context is not memory

The two get conflated constantly, and the distinction matters.

Context is the working set for the task at hand. It is what the model can currently see. It is bounded by the context window, it is expensive per token, and it is discarded when the session ends. Context is short-lived by design.

Memory is durable. It survives the session, it is far larger than any context window, and it is selectively retrieved — you pull the handful of items relevant to the current turn rather than replaying everything.

The relationship between them is the whole game: memory is the store, context is the window onto it, and retrieval is the function that decides what crosses from one to the other. A memory system that dumps its entire contents into context has not solved anything — it has just moved the context-window problem one layer down.

This is why serious memory layers treat the two as separate stores with different lifecycles. In phorvec, the context system holds the working set for an ongoing task, with its own branching and decay rules, while the memory store holds items that have earned durability. An explicit operation promotes something from one to the other — you lock in an insight from the current session before the working context decays away.

The four things a memory system must do

Most discussions of agent memory stop at "store embeddings, search them." That's one stage out of four. A memory store that only ever accumulates degrades into an unusable pile within weeks of real traffic.

1. Write

Something has to decide what is worth remembering. Every conversation turn? Only conclusions? Facts extracted by a smaller model at ingest time?

This is a genuine trade-off, not a solved problem. Storing raw turns verbatim is fast, cheap and preserves exact recall — if the user said "my flight is at 6:40am," those exact words are retrievable. Pre-digesting each session into structured facts costs compute at ingest but gives the model a cleaner cross-session summary to reason over later.

Both approaches are defensible, and they compose: you can store the verbatim turns and layer extracted facts on top, retrieving from both. phorvec supports both ingest strategies and publishes benchmark results for each so the trade-off is measurable rather than a matter of taste.

2. Retrieve

Given the current turn, find the relevant memories. This is where most systems are weakest, because pure semantic similarity has a well-known failure mode: it is excellent at concepts and unreliable at specifics. Ask for UserServiceImpl and a vector index cheerfully returns every class that is vaguely service-shaped.

The fix is not to abandon vectors — it's to combine them with a keyword index that handles exact terms, and fuse the two rankings. We cover the mechanics of this in Hybrid Search for Agent Memory.

3. Consolidate

Real memory stores accumulate near-duplicates. The same fact gets written five times in slightly different words across five sessions. Low-value chatter piles up around a handful of important decisions. Related memories sit in the store with no connection between them.

Consolidation is the maintenance pass that fixes this: merging near-duplicates, compressing clusters of related low-importance items into representative summaries, and building links between memories that are semantically related but never explicitly connected. Without it, retrieval quality degrades as the store grows — the opposite of what you want.

4. Forget

The most under-built stage, and the one that separates a memory system from a log file.

Not everything deserves to be remembered forever. Human memory handles this with decay: importance fades unless something is reinforced. Agent memory systems model the same behaviour explicitly, usually with a half-life — an item's importance halves every N hours unless it's pinned or re-retrieved.

Decay needs an escape hatch, though. Some things must never fade: a user's stated allergy, a hard architectural constraint, a security policy. So decay is paired with pinning, and items flow through lifecycle zones — active, archived, then queued for deletion — with configurable grace periods at each step. Memory Consolidation and Decay covers this in detail.

Types of memory, and why the taxonomy is useful

Borrowed from cognitive science, the standard breakdown is:

The taxonomy earns its keep because each type wants different handling. Episodic memories need timestamps and temporal ordering — questions like "what did we decide before the migration?" are unanswerable without them. Semantic facts need supersession: when a fact changes, the old version has to be invalidated rather than left to compete with the new one in search results. Procedural memory is best stored as named, searchable artifacts you retrieve by intent rather than by similarity to a conversation.

A system that treats all four as undifferentiated text in one vector index will answer the easy questions and fall over on the rest.

Where knowledge graphs fit

Vector search finds things that are similar. It cannot answer questions about how things are connected.

"Which services depend on the auth module?" is not a similarity question. Neither is "who made this decision, and what did it supersede?" These need typed relationships between entities — a graph.

The productive pattern is not graph instead of vectors, it's graph alongside vectors: extract entities and relationships from stored content into a property graph, then let queries use both. That makes questions like "find everything related to authentication that's also semantically close to this design note" answerable in one hop — the graph narrows the candidate set, the vector index ranks within it. phorvec maintains a typed property graph next to the vector index for exactly this, with relationship types like DEPENDS_ON, SUPERSEDES and CONTRADICTS.

Contradiction is a first-class problem

Once multiple sessions — or multiple agents — write to the same store, they will eventually disagree with each other.

Session one records "we're using Postgres." Session four records "we moved to DynamoDB." Both are in the store. Both are retrievable. Nothing about a vector index prefers the newer one, and an agent that retrieves both will confidently produce nonsense.

There are two complementary answers. Supersession handles the case where you know a fact has changed: record the new version as replacing the old one, so the store holds an ordered chain rather than two equally-current claims — with the superseded state still readable for audit. Conflict detection handles the case where you don't know: scan the store for statements that semantically oppose each other, classify them by severity, and surface them for review rather than silently serving both.

This matters most in multi-agent setups, where two agents can reach contradictory conclusions without ever reading each other's work. phorvec's conflict_check runs this scan across a team's shared pool, and there's a worked walkthrough of the two-agent case in the docs.

Where the memory layer should live

An architectural choice with real consequences.

A hosted memory service is the easy on-ramp — no infrastructure, managed scaling. The cost is that every memory your agents form leaves your environment, and your agent's recall now depends on someone else's uptime.

A local-first memory layer runs in your process or on your machine. Retrieval is a local index lookup rather than a network round-trip, memory works offline and in air-gapped environments, and the data never leaves. The cost is that you own the storage.

For agents that accumulate memory about proprietary codebases, internal architecture or customer data, the calculus usually favours local. That's the bet phorvec makes: each agent's memory is a single portable file on your disk, holding its vector index, keyword index, knowledge graph and audit log together. The case for one file per agent covers the trade-offs.

A checklist for evaluating a memory system

If you're choosing or building one, these are the questions that actually separate them:

• Retrieval: does it handle exact identifiers and phrases, or only semantic similarity?
• Temporal reasoning: can it answer "what did we decide, and when?" — or only "what is similar to this?"
• Consolidation: is there a maintenance pass, or does quality degrade as the store grows?
• Forgetting: can memories decay, and can important ones be exempted?
• Contradiction: what happens when two stored facts disagree?
• Portability: can you export everything and move it, or is your memory locked to a vendor?
• Locality: where does the data live, and does recall need a network?

Getting started

phorvec is a memory layer built around the lifecycle described above — write, retrieve, consolidate, forget — running locally, with one portable file per agent.

pip install phorvec

That installs the engine in-process for use from Python. For editor and agent integrations it also ships as a signed binary speaking the Model Context Protocol, which any MCP client can spawn. The Getting Started guide covers both paths, and the Core Features reference documents each subsystem in detail.