What memory systems actually do. A reader-matched head-to-head on LongMemEval-S.

Abstract

Memory benchmarks for LLM agents are dominated by single-vendor pipelines: each system is evaluated end-to-end against its own published numbers, with vendor-controlled generators, judges, and extraction pipelines. Side-by-side numbers from different papers are not directly comparable, and when they appear in pitch decks they hide as much as they reveal.

We run Atlaso against mem0 on LongMemEval-S (n = 500) under a matched reader — Qwen 3.5-9B produces the final answer for both arms — and three independent judges (Anthropic Haiku 4.5 strict, GPT-4o strict, GPT-5 permissive). Reader-matched, Atlaso leads mem0 by +6.6 percentage points on strict-judge accuracy (McNemar p ≈ 0.007) and by +6.3 to +7.4 F1 points across all three judges; no judge flips the direction. Under the same protocol, Letta ties Atlaso exactly, and Cognee and LangMem trail far behind.

A 2×2 retriever-reader swap with MemPalace separates retrieval recall (where MemPalace wins with R@5 = 96.2%) from end-to-end QA accuracy (where the Atlaso substrate edges ahead — directionally, not statistically significantly — while sending 3.4× fewer context tokens at 3.6× lower cost per query).

We also publish the limits of this study: mem0's published 93.4% does not reproduce under matched conditions (we observe 44.2% with their own judge prompt, a 49.2pp gap), but we did not reproduce mem0's full published pipeline, so the gap is best read as "methodology + pipeline," not "methodology alone." On the adversarial LoCoMo subset Atlaso loses to mem0 by 11.5 percentage points, and we discuss why.

Correction (August 2026). An earlier version of this page presented a four-judge head-to-head table as a shared-reader result. It was not: the Atlaso arm in that table used a GPT-5 reader while mem0 used Qwen 3.5-9B — a mismatch we found by re-auditing our own run artifacts (the _returned_model field in the published JSONLs). The headline comparison on this page is now fully reader-matched; the GPT-5-reader configuration is reported separately and labeled as such (§3.2). The uncorrected version remains visible in the git history. We publish this correction for the same reason we publish our losses: the method is the argument.

1. Why this study

The memory-system landscape has matured fast over the last year. mem0, LangMem, Letta, Cognee, Honcho, MemPalace and a handful of others all publish benchmark numbers on LongMemEval, LoCoMo and adjacent fixtures. Yet the published numbers are largely incomparable: each system evaluates itself with its own judge, its own reader, its own fixture preprocessing, and its own definition of what counts as a correct answer.

This is not bad faith — it is the natural shape of an early field. But it produces a specific failure mode: a pitch deck slide that lines up "mem0 93.4% vs Atlaso X%" can be off by 50 percentage points purely because the two numbers were generated under different rules.

We wrote this study to do one thing: hold every variable except the memory system fixed, then measure.

1.1 What we hold fixed

1. Shared fixture — LongMemEval-S, n = 500 questions, identical question IDs across all arms (verified by hashing the sorted question-ID list of every run).
2. Matched reader — Qwen 3.5-9B for the final answer step, in every arm of the headline comparison: Atlaso, mem0, Letta, Cognee, and LangMem all feed their retrieved context to the same reader with the same prompt template. A separate GPT-5-reader configuration of Atlaso exists in the artifacts and is reported in §3.2, clearly labeled — it is not part of the head-to-head.
3. Shared judges — three independent LLM judges score the same answers: Anthropic Haiku 4.5 (strict, both raw accuracy and calibrated-abstention F1), OpenAI GPT-4o (strict), and OpenAI GPT-5 (permissive). mem0's own verbatim judge prompt (fetched from their public benchmarks repo) is additionally used for the reproduction analysis in §4.
4. Label-blind normalizer — before each judge sees an answer, we strip system-identifying fingerprints (e.g. <CONFIRMED> polarity headers, "Atlaso says:" banners, signature abstention phrases) so judges cannot identify the arm from the answer text.

2. Method

2.1 Fixture

LongMemEval-S is a published benchmark of 500 long-context conversational-memory questions spanning five categories: user-said-facts, updated-facts, preferences, multi-chat synthesis, and time-based reasoning. We use the exact JSONL fixture and gold answers from the LongMemEval repository, pinned by commit SHA, with no preprocessing.

2.2 Arms

Each arm consists of (i) an ingestion step that reads the conversation history and writes to the system's memory store, and (ii) a retrieval step that produces a context for the reader. mem0 default uses gpt-4o-mini as its extraction LLM at ingestion time, per its OSS default configuration. Atlaso writes deposits with zero LLM calls at ingestion.

2.3 Judges

Each headline-arm answer is scored by the three shared judges independently; the mem0 verbatim judge prompt — reproduced from the memory-benchmarks repository on GitHub (fetched 2026-05-06), with no edits beyond formatting — is additionally applied where §4 uses it. Per-question judge outputs are stored in the run JSONLs alongside the system answer, retrieved contexts, and gold answer, so any third party can re-judge with a different model.

A note on pre-registration. The v1.8 protocol — under which the canonical n = 500 numbers were generated — was iteratively converged across prereg-option3-v1.1 through prereg-option3-v1.8 during pilot runs. Every version is a public git tag; the full v1.1 → v1.8 diff history, pilot run JSONLs, and the locked v1.8 specification are all in the repository. We do not claim the canonical run was preregistered before any data was collected; we publish the entire version trail so the protocol-tuning surface is auditable rather than hidden.

3. Main result: reader-matched, three judges, same direction

With Qwen 3.5-9B producing the final answer for both arms, Atlaso beats mem0 on the same 500 questions: 53.6% vs 47.0% on strict-judge accuracy (+6.6pp; McNemar on the 141 discordant pairs: χ² = 7.26, p ≈ 0.007), and by +6.3 to +7.4 points of calibrated-abstention F1 across all three judges. No judge flips the direction.

Table 1: Atlaso vs mem0-default on LongMemEval-S (n = 500), matched Qwen 3.5-9B reader in both arms, three independent judges.

The multi-judge structure is the point. A single judge can be biased toward one system's answer style — for example a permissive judge might over-credit hedged answers, while a strict judge might over-penalize abstention. By scoring the same 500 answers under three independent prompts, we get a sensitivity band rather than a single number. Atlaso leads in every band.

3.1 The wider field, same protocol

Four more systems ran under the identical fixture, reader, and judges. We report all of them, including the tie:

Table 1b: the wider field on the same protocol. \LangMem v0.0.30 ships no extraction LLM, so its arm is asymmetric by design — we report it for completeness, not as a like-for-like defeat. Honcho is excluded entirely: its run failed on infrastructure (queue timeouts on all 500 questions), which is not a memory-quality measurement.*

Letta tying Atlaso is worth stating plainly: on this fixture and protocol, Letta's memory quality matches ours. The differences between the two products are elsewhere — Letta's memory lives inside its own agent runtime, while Atlaso's travels across the tools you already use — but on raw benchmark quality we report a tie because that is what we measured.

3.2 The strong-reader configuration (not a head-to-head)

The artifacts also contain an Atlaso configuration run with a GPT-5 reader: 61.8% (Haiku strict), 69.2% (GPT-4o strict), 57.8% (GPT-5 permissive), 56.4% (mem0's verbatim judge). These cells show what the substrate does under a frontier reader, and we use the last one in §4 and the 2×2 swap in §5 (where both cells share that same GPT-5 reader). They are not comparable against the Qwen-reader mem0 numbers above, and an earlier version of this page erred by presenting them as if they were — see the correction note at the top.

4. Why mem0's published 93.4 does not reproduce

mem0 publishes a headline number of 93.4% accuracy on LongMemEval-S in their memory-benchmarks repository. That number anchors most external comparisons to mem0. We tried to reproduce it.

Running mem0 with its default OSS pipeline (gpt-4o-mini extractor, default top-k, default storage configuration) on the same n = 500 fixture, and scoring with mem0's own verbatim judge prompt, we observe 44.2% — a 49.2 percentage point gap from the published number, under mem0's own scoring rule.

We are careful with how we describe this gap. We did not reproduce mem0's full published pipeline (their managed-platform configuration, top-k settings, and proprietary post-processing). So the 49.2pp gap is best read as "methodology + pipeline," not "methodology alone." What we can say definitively:

• Under mem0's own published judge prompt, mem0's OSS default pipeline scores 44.2%, not 93.4%.
• Under that same judge, Atlaso's GPT-5-reader configuration (§3.2) scores 56.4% — with no extraction LLM and no proprietary platform configuration. (The reader-matched Atlaso arm was not scored under this judge; the reader-matched result is §3's +6.6pp.)
• The honest comparison floor is somewhere between mem0's 44.2% and their published 93.4%; future work that reproduces mem0's full managed-platform pipeline will narrow this band.

5. Retrieval recall is not QA accuracy

MemPalace publishes an R@5 (top-5 retrieval recall) of 96.6% on LongMemEval, a striking number that is sometimes lined up against end-to-end QA numbers. These are different axes. R@5 measures whether the right session is in the top-5 retrieved chunks; QA accuracy measures whether the final reader produces a correct answer. A system can dominate one axis and lose the other.

We ran a clean 2 × 2 retriever-reader swap to test this. Both cells share the same GPT-5 reader and the same LongMemEval-S fixture; the retrieval source is the only variable, so this comparison is internally matched.

Table 2: 2×2 retriever-reader swap on LongMemEval-S, shared GPT-5 reader in both cells. R@5 is computed on the 479 questions with a gold session label; end-to-end QA is on the full n = 500. MemPalace wins R@5 by +31pp. Atlaso is directionally ahead on end-to-end QA (+3.2pp — not statistically significant, McNemar p ≈ 0.17) while sending 3.4× fewer context tokens at 3.6× lower cost per query.

The category-level breakdown is more striking. MemPalace returns whole sessions (dense embeddings on session-level documents) while Atlaso returns specific high-relevance turns (BM25 + scope filtering on deposit-level units). On preference questions — where the right answer is one short user statement within a long session — coarse session retrieval drops the signal: accuracy falls from 56.7% to 16.7%, a −40 percentage point gap.

The interpretation: high R@5 is not free QA accuracy. It is a useful component, but the substrate the reader runs against — what is stored, in what shape, and how is it filtered — is where end-to-end QA is won or lost.

6. Cost

The structural fact comes first, because it is the one that survives any pricing change: Atlaso makes zero LLM calls when it writes a memory. mem0's default OSS pipeline calls gpt-4o-mini on ingestion; in our instrumented runs that extraction step is 90–99% of mem0's total per-query cost depending on how long the ingested histories are.

Table 3: Measured per-query cost on LongMemEval-S, matched Qwen 3.5-9B reader in both arms (all costs traced from instrumented runs at pinned OpenRouter prices; mem0 extraction cost is its measured gpt-4o-mini spend). The gap between the two fixtures is driven by history length: the full fixture's much longer histories make mem0's extraction step dominate.

Two disclosures that must travel with any cost claim. First, Atlaso reads more: because we index raw deposits rather than LLM-compressed facts, our read-time context is roughly 6× larger than mem0's in these runs (≈3,000 vs ≈500 prompt tokens per query on the n = 211 slice). Our cost advantage is write-side. Second, and consequently, the per-query sign can flip at frontier reader prices: if you deploy an expensive reader, the read-context premium can outweigh the extraction saving. Cost comparisons between memory systems are meaningless unless the reader and its price are named — which is also why we name ours.

We deliberately do not publish a $/year savings extrapolation. Early drafts of this work projected dollar savings at 100K daily-active-users; on review the extrapolation multiplied a per-question extractor cost by a per-message rate, which is a unit error. The defensible cost claims are the measured per-query numbers above plus the structural ingestion-time difference; any TCO calculation downstream of that depends on assumptions about message volume, retention horizon, reader choice, and storage tier that vary by deployment.

7. Where we lose: LoCoMo

LoCoMo is an adversarial conversational-memory benchmark designed with planted distractors and temporal-reasoning traps. It is a different distribution from LongMemEval. We ran the same shared-reader protocol on a 200-question LoCoMo subset. Atlaso loses.

Table 4: LoCoMo adversarial subset (n = 200, Haiku 4.5 strict judge). Atlaso loses to mem0-default by 11.5pp.

The honest interpretation is that Atlaso's contradiction-aware retrieval is currently tuned for conversational-memory questions where the answer either exists clearly in the field or does not. On LoCoMo's temporal- and distractor-heavy questions, the system over-abstains, or returns the planted distractor with high confidence. We publish this loss because hiding it would make the rest of the study less trustworthy, not more — and because where a system fails is more diagnostic than where it succeeds.

We do not yet have matched-condition LoCoMo numbers for the other systems in the broader landscape, so we do not draw conclusions about the LoCoMo leaderboard. The fixed comparison here is Atlaso vs mem0, same fixture, same judge.

8. Limitations

We have tried to be conservative about what this study establishes. Specifically:

1. Mem0's full pipeline was not reproduced. The 49.2pp gap between 93.4% and 44.2% conflates judge methodology with pipeline configuration. Until mem0's full managed-platform pipeline is reproduced at or near 93.4% on the same fixture, the gap should be read as "methodology + pipeline," not isolated to either one.
2. The mem0-verbatim judge was not run on every arm. It scored mem0's arm (the 44.2% in §4) and Atlaso's GPT-5-reader configuration (§3.2), but not the reader-matched Atlaso, Letta, Cognee, or LangMem arms. The reader-matched head-to-head therefore rests on three judges, not four. Honcho is excluded from §3.1 because its run failed on infrastructure (all 500 questions timed out in its ingestion queue) — an availability observation, not a quality measurement. LangMem's arm is asymmetric by design (no extraction LLM in v0.0.30).
3. Pre-registration is a version trail, not a clean prior commit. The v1.8 protocol was iteratively converged through pilot runs (v1.1 → v1.8 are all public tags). We do not claim the canonical run was preregistered before any data was collected; we publish the full version trail so the protocol-tuning surface is auditable rather than hidden.
4. Cost is per-query, not TCO — and reader-dependent. The per-query numbers are measured at Qwen 3.5-9B prices; §6 states the read-context asymmetry under which the sign can flip at frontier reader prices. Deployment cost depends on assumptions we do not make.
5. Single subset of LoCoMo. The LoCoMo result is on n = 200, Haiku 4.5 strict, one judge. The other judges were not run on LoCoMo at this scope.
6. A correction was required. The original version of this page presented the §3.2 GPT-5-reader cells as the shared-reader head-to-head. We found the error in our own artifacts, corrected the page, and left the history public.

9. Reproducibility

The entire study is reproducible from the open-source repository. The arm scripts, fixtures, judge prompts (including mem0's verbatim prompt as fetched), run JSONLs with raw per-question scores and contexts, the bootstrap CI script, and the protocol version history are all in the public repo. There is no proprietary platform call in the Atlaso path; mem0 is run via its public OSS configuration with the default extractor.

• Code: github.com/atlaso-labs
• Fixture: LongMemEval-S (commit SHA pinned in the run config)
• Judges: Anthropic Haiku 4.5, OpenAI GPT-4o, OpenAI GPT-5, mem0 verbatim (prompts in research/option3/judge.py)
• Per-question outputs: every system answer, retrieved context, judge rationale and gold answer is in the run JSONLs under research/option3/runs/, including the _returned_model field per row — the field that surfaced our own reader-mismatch correction
• Protocol version: prereg-option3-v1.8

The practical takeaway: hold the reader, the fixture, and the judge fixed before you compare memory systems. When you do, the gap between what systems claim and what they deliver becomes visible — and the direction of the answer stops depending on whose judge you used.