Hybrid Search for Agent Memory

Vector search is the default answer for agent memory retrieval, and for good reason: it finds content that means the same thing as your query even when it shares no words with it. Ask about "authentication failures" and you'll surface a note about "login errors" that keyword matching would miss entirely.

But run a memory layer in production for a while and a pattern emerges. Semantic search is excellent at concepts and unreliable at specifics.

Where pure vector search breaks

Embeddings compress meaning into a fixed-length vector. That compression is lossy, and what it loses first is exactly what engineers search for.

Identifiers. Search for UserServiceImpl and the embedding encodes something like "a user-related service class." Every other service class in the codebase is nearby in vector space. The thing you actually named ranks somewhere in the middle of a pile of near-neighbours.

Exact phrases. A stored decision reads "we chose Postgres over DynamoDB for the billing service." Query "billing database choice" and semantic search does fine. Query the exact sentence and it does no better — the vector has no notion of literal match, so an item that quotes your query word-for-word is not preferred over one that merely paraphrases it.

Rare terms. Error codes, version strings, ticket numbers, function names, config keys. These carry enormous signal precisely because they're rare, and embeddings systematically under-weight them — a token seen a handful of times in training contributes little to the vector's direction.

This isn't a flaw in any particular embedding model. It's what dimensionality reduction does. The fix is to keep a second index that's good at exactly what vectors are bad at.

BM25: the other half

BM25 is a keyword ranking function that has been the backbone of text search for decades. It scores a document against a query using term frequency, adjusted by two corrections that matter here:

• Inverse document frequency — rare terms count for more. UserServiceImpl appearing in a memory is far more informative than the appearing in it, and BM25 weights accordingly. This is precisely the signal embeddings lose.
• Length normalisation — a long document isn't allowed to win just by containing more words.

BM25 has the inverse weakness of vector search: it cannot connect "authentication failure" to "login error" because they share no terms. Alone, it's brittle. Alongside a vector index, it's the exact complement.

Fusing two rankings

So you run both searches. Now you have two ranked lists and one problem: their scores aren't comparable. Cosine similarity lives in roughly 0–1 with a distribution that depends on the embedding model. BM25 scores are unbounded and depend on corpus statistics. Adding or averaging them is meaningless, and normalising them requires assumptions about distributions that vary per query.

Reciprocal Rank Fusion sidesteps the problem by throwing away the scores and keeping only the ranks:

RRF_score(item) = Σ  1 / (k + rank_in_list)
                 lists

An item ranked 1st contributes 1/(k+1), ranked 2nd contributes 1/(k+2), and so on, summed across every list it appears in. The constant k damps the influence of the very top positions so a single list can't dominate.

Two properties make this the right tool:

1. Scale-free. Only ordering matters, so incommensurable scoring systems combine cleanly.
2. Agreement wins. An item that ranks moderately well on both signals beats an item that ranks first on one and is absent from the other. That's the behaviour you want — corroboration across independent retrieval methods is genuine evidence of relevance.

That second property is the real payoff. A memory that is both semantically on-topic and contains your exact identifier is almost certainly the one you wanted, and RRF surfaces it without any hand-tuned weighting.

How phorvec implements this

phorvec combines three retrieval signals in a single hybrid_search call:

Both indexes live in the same .avdb file as the memory they index, so a hybrid query is two local index lookups and a fusion step — no network hop, no second service to keep in sync with the first.

Tuning with alpha

The alpha parameter controls the blend between vector and keyword scores before fusion:

• alpha = 1.0 — pure vector search
• alpha = 0.0 — pure BM25 keyword search
• alpha = 0.5 — equal weighting (the default)

Because it's a query-time parameter, you can match it to the retrieval task rather than committing globally. Exact identifier lookups benefit from lower alpha; open-ended conceptual queries benefit from higher alpha. In practice the default is right most of the time, and the ability to drop alpha for a "find me this exact symbol" query is what keeps the system usable for code-heavy workloads.

Seeing why something ranked

Retrieval quality problems are miserable to debug when ranking is a black box. Passing debug: true to hybrid_search returns a per-result breakdown: the raw vector score, the raw BM25 score, and the fused RRF rank.

That turns "why did this irrelevant memory come back third?" into an answerable question. Usually the answer is visible immediately — the item scored well on BM25 because it repeats a common term, and poorly on vectors, and no amount of alpha tuning will fix a memory that was written badly in the first place.

Retrieval is necessary, not sufficient

Hybrid search meaningfully improves what comes back from a memory store. It does not, on its own, make a memory system good.

The other half is the state of the store you're searching. An index full of near-duplicates returns five versions of the same fact and crowds out everything else. An index with no notion of time can't answer "what did we decide most recently." An index holding two contradictory statements will happily return both.

Those are consolidation, temporal metadata and conflict-detection problems, not ranking problems — and they're covered in Memory Consolidation and Decay.

Try it

pip install phorvec

The Core Features reference documents hybrid_search, the alpha parameter and the debug output in full, and Accuracy & Memory Modes covers how the retrieval stack is evaluated.