Accuracy & Memory Modes

The short version

Out of the box, phorvec answers 79.7% of long-conversation questions correctly on LongMemEval-s, versus 67.7% for naive retrieval-augmented generation (RAG) — a +12-point lead, concentrated exactly where a memory engine should win: multi-session synthesis and temporal reasoning.

A higher-accuracy distilled mode pushes that to 85.3% (+17.7 over RAG) in exchange for a heavier ingest step.

Benchmark: LongMemEval-s

LongMemEval is the standard benchmark for long-term conversational memory. The -s split asks 500 questions against haystacks of ~48 prior chat sessions (~115k tokens) each.

Methodology. 300 questions (seed 0), Claude Sonnet 4.6 as both the answer model and the LLM judge at temperature 0, identical top-20 retrieval budget for every system. Same harness, same questions, same judge — only the memory layer changes.

The pattern is the whole point: phorvec is at parity with RAG on the easy single-session questions and pulls +18 to +20 points ahead on the questions that require pulling facts across many sessions or reasoning about when something was said. Naive RAG retrieves chunks; phorvec retrieves memory.

Memory modes: default vs distilled

phorvec supports two ingest strategies. Both store your conversation turns; they differ in whether a small language model also pre-digests each session into structured facts at ingest time.

Default (recommended for most workloads)

Conversation turns are stored directly as they arrive and retrieved with hybrid vector search. No external model is involved at ingest, so ingest is fast, cheap, and has no extra dependencies.

• Score: 79.7% overall (+12.0 over RAG)
• Ingest cost: near-instant; CPU-only embeddings
• Use it when: you want the simplest, fastest path that still decisively beats RAG — which is almost always.

Distilled (high-accuracy, synthesis-heavy workloads)

In addition to the raw turns, a small local model (e.g. Ollama qwen14b) extracts a handful of structured facts per session — preferences, counts, decisions — and stores them alongside the verbatim turns. At retrieval, those distilled facts give the answer model a cleaner cross-session summary to reason over.

• Score: 85.3% overall (+17.7 over RAG, +5.6 over default)
• Ingest cost: runs a local LLM over every session — materially slower and heavier at ingest
• Use it when: your workload leans on multi-session synthesis, temporal reasoning, or knowledge-updates, and you can afford the ingest-time compute.

Side by side

Distilled wins overall and is strongest on multi-session (+13 over default) and knowledge-update (+13). The one row where default edges ahead — single-session preference — sits within run-to-run noise (just 17 questions in that category).

Distilled already contains everything default does. This is the part that's easy to miss: distilled doesn't replace your conversation with summaries — it stores the verbatim turns (exactly what default stores) and adds the distilled facts on top, then retrieves from both. So distilled is a superset of default, not an alternative to it. It has default's word-for-word recall plus cross-session synthesis. That's why there's nothing to "route" — you don't pick default for preferences, distilled for multi-session, because distilled's memory already includes default's. The only thing you trade for the extra accuracy is ingest-time compute (the local model that does the distilling). If you can afford that, distilled is strictly the more capable memory.

Choosing a mode

Rule of thumb: start with default — it already beats naive RAG by +12 points with none of the extra cost. Reach for distilled when your traffic is dominated by cross-session synthesis questions and you can afford the ingest-time compute.

How to run each mode

Default (out of the box)

Default is phorvec's standard behaviour — there's no special setup:

• Ingest: context_store each conversation turn. Use storage_mode: "verbatim" to capture turns word-for-word (that's what the benchmark uses).
• Retrieve: context_retrieve with the user's query; top_k: 20 matches the benchmark.

That's the +12pp-over-RAG configuration — nothing else required.

Distilled (add a session-distillation step at ingest)

Distilled layers an ingest-time pattern on top of default. For each session, alongside storing the raw turns:

1. Send the session transcript to a local instruction model and ask it to extract a short list of durable, user-relevant facts (preferences, counts, decisions).
2. Store both the raw turns and each extracted fact via context_store (tag the facts so they're distinguishable). Keep raw-turn storage on — the verbatim turns are what preserve specific recall.
3. Query exactly as in default; raw turns and facts are retrieved together.

Model we tested: Ollama qwen14b (a 14B local model) for fact extraction — chosen as a quality/cost balance that runs on a single GPU. It is the floor we validated, not a ceiling: any stronger instruction model is welcome and will likely do better — a larger Qwen, Llama 3.x 70B, or a hosted frontier model if API cost is acceptable. Distillation quality scales with the extractor. (Answer generation and the LLM judge in every benchmark used Claude Sonnet 4.6.)

Distilled runs your chosen extractor over every session at ingest — a one-time cost per session, not per query. Budget for it accordingly.