The implementation of memory system evolves as the models and harnesses evolve. Within a span of 6 months, the industry moved from vector embeddings to markdown in file trees, and then from LLM-wiki like graphs to now more sophisticated event-log, artifact graphs. These are responses to the changing complexity of information and demands from the people, and teams using them. But the ways we evaluate memory systems have not evolved at the same pace, at least in the industry zeitgeist.

Just within last three months there have been 31 published papers on memory evaluation methodologies, but the industry still maintains their leaderboards on LoCoMo, LongMemEval, and BEAM. These are still relevant for stress testing certain memory properties in some conversational agent use cases, but the agent-native work is substantially different today both in quantity and complexity.

The memory systems today are also far more capable than the capabilities tested in these benchmarks, and would benefit from more sophisticated and realistic stress testing.

This article provides a breakdown of the 3 popular benchmarks, describes the agent-native work common now, and explores different motivations to design new evaluation methodology. At the end we provide an appendix of new benchmark approaches, measuring different aspects of memory applied in various agent-native contexts.

0. Expectations from a memory system

Before choosing a benchmark, what are the expectations from a memory system, and things that are currently called Memory. Almost all the expectations from a memory system as of 2026 is a form of question answering, especially for a question asked from history, but they are not the same question answering problem.

The different contexts for memory systems (overlapping categories)

Chat Conversations. In a typical user assistant interface, limited to only the text content within chat.

Coding Agents. More work oriented. A user assistant interface here is the frontend interface for projects, changes, experiments, design, and artifacts that operate in a larger workspace and its tooling.

Personal Computing. Memory distributed across a person's activity. Includes use of agents, photos, media, files, browser history, social platforms.

Company Brain. Shared context across different team functions, tooling and processes. Memory here is more adherence, provenance and keeping the company state current.

Knowledge Base. Growing body of knowledge across a subject, papers, claims, experiments, and canonical organization knowledge. Memory here is linking artifacts, more Wiki shaped.

Agent System. Scoped to agent architectures. Maintenance of context, reusable procedures, especially needed in long horizon agents, multi agent systems.

Most real world use cases have some overlap with each other depending on the context and interface design. However, the way memory is evaluated is broadly two categories with their own subcategorization.

Retrieval Shaped

1. Retrieval of a fact from a conversation. Where did I grow up? What food do I like? What did I say about this thing X, last month?
2. Finding exact artifact "What have we established from our research on topic X so far, cite all generated documents and downloaded papers."
3. Reconstruction of work state Where were we with this? What was tried, what failed, what was decided, and what is still blocked? This is not necessarily one fact sitting in one message. The answer may have to bring several objects together into one coherent picture.
4. Contradiction Resolution, Forgetting, Reasoning An old plan may still be historically relevant without being the plan we should follow now. Memory has to keep changes, corrections, rejected ideas, and superseded states from collapsing into one pile and reconstruct them
5. Object-specific continuation A certain module, an idea, a concept, a small specific thread spread across weeks/months needs to be reconstructed in a larger work space. Broadly reconstruction of work state, but reoriented around a specific object.

Beyond Retrieval

1. Preference, Recommendation and Personalization. The user profile changes over time, and the system has to use the latest preference, its history and the reasons behind it in a new response or recommendation.

2. Maintenance, Learning and Memory Lifespan. Over time the memory itself has to be revised, consolidated, compressed or forgotten, without losing the things that will matter in later work.

3. Needs Before Asked. The first request does not always contain the actual need; it can be hidden in the workspace, earlier activity or a future intention that has to be carried forward.

4. Memory-Conditioned Action. Finding the old context is not enough if it does not change the next tool call, workflow step or action inside the environment.

5. Scope, Provenance and Safe Write. Not everything seen by an agent should become durable memory, and not every stored memory should be trusted, shared or used in the same way.

Hence the claim, that there is no useful answer to “which memory benchmark should I run?” The answer depends first on what the memory is responsible for, what sources can establish the current state, and what failure would matter in deployment.

1. Popular Memory Benchmarks — Honestly, Sequential Conversational QA

LoCoMo, LongMemEval, BEAM.

Why these 3 ?

Because these are the most popular ones, against which the claims are made, and they recur across the public evaluation pages of memory companies even though their adoption and protocols are not identical, and honestly the wrong evaluation for what the majority of companies are designing their product around.

From my market analysis of a set of 115 memory / company brain products.

So we open up these benchmarks beyond their papers and code, to understand what one item is, what a memory system sees, and how the output is scored.

---

1.1 LoCoMo

Feb 2024 · arXiv:2402.17753 · ACL 2024 · data

TLDR:

• Synthetic Data. World and Task constructed first. History generated later.
• Task is 10 QA at the end of each history.
• Scoring is word overlap. 1-2 word answers.
• ~16k tokens

How is this constructed?

• LoCoMo creates synthetic long-term relationships between pairs of fictional people.
• Each person begins with a profile and a private timeline of dated, sometimes causally connected life events— jobs, relationships, interests, travel, personal changes, and other experiences.
• Some events are connected, so that one event causes or explains another. The paper calls this a temporal event graph.
• LLM agents then role-play the pair across many sessions, gradually revealing those events through conversation; human annotators repair inconsistencies and verify the dialogue against the graph.

Objects

• World: two fictional people, each with a persona and a dated event graph. This is the hidden construction used to make the dialogue coherent.
• History: the finished multi-session conversation between that pair. A session is one dated exchange of messages; the history is all of those sessions in order.
• QA item : that history, one later question, and a reference answer:

$$I_{j,k}=(H_j,q_{j,k},a^\star_{j,k}).$$

The system uses the history $H_j$ to answer the question $q_{j,k}$. The evaluator holds the reference answer $a^\star_{j,k}$ and scores the response. The same history can be used again for a different question.

In locomo10:

• There are 10 histories: ten fictional pairs.
• Each contains 19–32 dated sessions across 167–293 days.
• Each completed history is reused for multiple QA items (Median 197 per history)

Scoring

The public LoCoMo number is its question-answering score.

• The paper: “F1 score for exact matches,” after normalizing the prediction and reference; it asks models to reproduce dialogue wording where possible.

• The released scorer: defines that normalization. For ordinary questions, it:

  • lowercases the answer and reference;
  • removes punctuation and the words a, an, the, and and;
  • stems words, so painted and painting both become paint;
  • ignores word order and counts only shared normalized word stems.

If $M$ is the number of shared normalized word stems, the score is:

$$P=\frac{M}{\text{normalized words generated}}, \qquad R=\frac{M}{\text{normalized words in the reference}}, \qquad F1=\frac{2PR}{P+R}.$$

This is word overlap, not semantic equivalence. If the reference is “2022” and the system says “She painted it in 2022,” then only one of five normalized words matches: $P=1/5$, $R=1$, and $F1\approx0.33$. Answering only “2022” scores $1.0$.

▸ Relative-time resolution from one utterance — data Caroline and Melanie talk across 19 sessions. In their first session, dated May 2023:

Melanie: By the way, take a look at this. Caroline: Thanks, Melanie! Is this your own painting? Melanie: Yeah, I painted that lake sunrise last year! It's special to me.

The later question is “When did Melanie paint a sunrise?” The gold answer is 2022. That string never appears in the dialogue: the system must find the relevant turn and resolve “last year” against its timestamp.

The reasoning is real; the score only sees the final answer words.

Multi-hop. One item asks what activities Melanie partakes in. The reference is “pottery, camping, painting, swimming.” The scorer checks the comma-separated parts separately. Add “skiing” as a fifth part and the answer can still receive full credit.

Adversarial. One item asks what Caroline realised after a charity race; the dialogue says it was Melanie. The right answer is not mentioned. This is a binary right-or-wrong check, not F1.

The benchmark is also small by 2026 context standards. The released locomo10 histories average roughly 16,600 tokens. That entire conversation now fits comfortably inside frontier context windows. Honcho argues that this makes LoCoMo a weak stress test for a dedicated memory architecture, while Mem0’s own 2026 overview calls its context modest and notes that it does not explicitly test knowledge updates. These are memory-company blogs, not neutral research, but the underlying point about scale is real.

The long history is also not usually used as one long connected history. Memora reports that 94% of LoCoMo questions require evidence from no more than two sessions. RealMem contrasts its project-state setting with LoCoMo’s casual human–human conversations and questions asked only after the history ends. Pith makes the narrower practitioner criticism: LoCoMo can test retrieval while missing stale beliefs, unresolved contradictions, and whether retrieved information should still be trusted.

So LoCoMo is useful for conversational assistants and companions that need to remember people, events, dates, and relationships from dialogue. But a high LoCoMo score is weak evidence for maintaining a changing user, project, coding workspace, or company. MemoryArena makes that transfer gap visible: systems with near-saturated LoCoMo performance still struggle when remembered information must guide later actions.

---

1.2 LongMemEval

Oct 2024 · arXiv:2410.10813 · data

TLDR:

• 500 histories, not 500 users
• 94% history is synthetic and filler hay in the stack, assembled from different datasets
• Sessions shared across history, no unique footprint for each history.
• Task constructed first, history generated later.
• One needle in a haystack shaped question asked at the end of each history.
• roughly the same amount of conversation spread across different timestamp ranges.
• 2 to 5 word answers.

How is this constructed: LongMemEval is explicitly designed as a conversational needle-in-a-haystack benchmark. Each of its 500 questions is paired with a configurable chat history: about 115,000 tokens and 50 sessions in LongMemEval-S, or 500 sessions and roughly 1.5 million tokens in LongMemEval-M. Its construction runs backwards from the test:

• An ontology of 164 user attributes is used to generate background paragraphs, from which candidate questions and answers are created and then rewritten by human experts;
• Each answer is decomposed into one to six evidence statements;
• Those statements are embedded indirectly in LLM-simulated, human-edited evidence sessions—to convey that a user bought a car, for example, the simulated user might ask about insurance;
• For each question, those evidence sessions are shuffled into unrelated filler: 25% ShareGPT, 25% UltraChat, and 50% simulated sessions about other non-conflicting attributes. Timestamps are then assigned to make the resulting sequence plausible. (paper, §3.2 and Appendix A.2)

The task. The system ingests the completed sequence of timestamped sessions. Only after the history ends does it receive one question and produce one answer. The questions isolate five abilities—information extraction, multi-session reasoning, knowledge updates, temporal reasoning, and abstention—including 30 false-premise questions where the correct answer is that the requested fact was never mentioned.

What the timestamps show. The histories are not uniformly distributed, but they are also not realistically distributed. LongMemEval-S contains both dozens of sessions packed into one calendar date and roughly the same number of sessions spread across several months.

• Most of the histories are short in calendar time: 349 of 500 histories span no more than fourteen days.
• 94 histories place all their sessions on one date
• 242 place their first and last sessions ten calendar days apart.
• There is a real longer tail—42 histories exceed ninety days—but the amount of conversation barely changes. Histories placed on one date contain a median 47 sessions; histories longer than ninety days contain a median 48.

These are haystack histories, not users.

• The release contains no persistent user identifier.
• Its 500 items contain 23,867 session slots but only 19,195 unique session IDs. Of those IDs, 3,932 appear in more than one item, accounting for 8,594 slots—36% of all session placements.
• Every item contains at least one such reused session; in the median item, 36% of its sessions also appear in another item.
• The median item contains 2 annotated answer-bearing sessions among 48 sessions; also reported by the authors of Memora.
• Approximately 96% of its session slots form the haystack. As described above, that haystack deliberately mixes ShareGPT, UltraChat, and simulated conversations about unrelated, non-conflicting attributes. The examples below show what this construction produces in practice: one sparse fact hidden among unrelated sessions, and one explicit fact updated later.

▸ Example 1 — single fact in a 53-session haystack — data One evaluation item presents ~53 separate chat sessions (each ~12 turns) as a single user's history, on unrelated topics—workout plans, task-management apps, expense tracking, meal prep. The needle is dropped once, mid-conversation, inside a session about staying organized:

user: ...By the way, do you have any tips on creating a routine for my new job? assistant: [long list of routine tips] user: I graduated with a degree in Business Administration, which has definitely helped me in my new role...

After all 53 sessions, the probe names the exact attribute to retrieve: "What degree did I graduate with?" The gold answer is Business Administration. The task is to recover one clause from unrelated productivity conversations.

▸ Example 2 — a knowledge update across two days — data This is an explicit update case. On April 23, inside a session about endurance training, the user mentions a result:

user: ...I recently set a personal best time in a charity 5K run with a time of 27:12. Do you have any tips...

The next day, April 24, in a different session, the number changes:

user: ...I'm hoping to beat my personal best time of 25:50 this time around.

Later, the benchmark asks: "What was my personal best time in the charity 5K run?" The gold answer is 25:50 — the later value. This is what knowledge update means operationally in this item: the history contains two values, but the response must resolve to the value treated as current.

Scoring

The official evaluator asks GPT-4o whether the final response is correct:

$$z_i=J_{\tau_i}(q_i,a_i^\star,\hat a_i)\in\{0,1\}.$$

This is semantic rather than exact-string matching as in LoCoMo. Better, but still different from what today's systems expect.

The archive is large, but the answer dependency is usually shallow. The authors of Memora report that 85% of LongMemEval questions require evidence from no more than two sessions. Under their audit, LongMemEval has an average consolidation demand of 1.9 sessions, a maximum of six, and knowledge updates involving at most two mutations. The 115K tokens therefore create a retrieval pressure, but most items do not require the system to integrate a person’s history across dozens of sessions.

The question answer pair is still retrieval, or inference style shaped, the final target is usually still an entity, number, date, current value, or short phrase.

The table below is stats on what the answers are like. Usually 1-2 word answers.

LongMemEval is a meaningful step beyond LoCoMo. It tests a much larger user–assistant archive, explicit updates, temporal reasoning, information supplied by both user and assistant, preferences, and false premises. The questions can require real search and reasoning. (paper)

But the raw shape is specific:

• 500 histories built for 500 eventual questions, not 500 users;
• usually one or two answer-bearing sessions inside roughly 48;
• mostly unrelated, non-conflicting material around that evidence;
• roughly the same amount of conversation spread across different timestamp ranges;
• one supplied question and one compact answer at the end.

So a high LongMemEval score is good evidence that a pipeline can answer explicit questions from a very large timestamped chat archive.

It is much weaker evidence that one memory has naturally accumulated and changed with a person over months, or that it will remain useful for future questions nobody designed the history to answer.

Later conversational benchmarks make similar criticisms from different directions: RealMem moves questions into ongoing project dialogue, while PERMA models preferences as something that emerges and changes through related events.

---

1.3 BEAM

Oct 2025 · arXiv:2510.27246 · ICLR 2026 · data

TLDR:

• 10M tokens history
• 20 questions asked from each history
• No declared time stamp. Time span is narrational

BEAM fixes two real limitations in the earlier benchmarks. It does not reach scale by filling a history with unrelated conversations, and it does not limit the test to a few forms of retrieval.

Each history begins with a generated user profile, relationships, an explicit timeline, and 15–20 evolving narratives. These are expanded into a long user–assistant conversation covering one of nineteen domains, including coding, mathematics, health, finance, relationships, education, and personal life. The resulting conversations range from 128K to ten million tokens.

Every conversation receives the same twenty question types: two each for facts, updates, cross-session reasoning, preferences, summarization, time, abstention, instructions, event ordering, and contradictions.

The exact final questions are generated after the conversation. But the plans are deliberately checked and expanded so that every history contains material for all ten abilities. This gives BEAM much better coverage than earlier benchmarks. It also means that the headline score is performance on a deliberately balanced test—not an estimate of how often these problems occur in a real product. (paper, §§2.2–2.3)

One released coding-themed history asks:

Question: When does my first sprint end?
Rubric: March 29.

The coding topic makes the history more realistic and technically varied. But the fact becomes true because it was stated in the conversation. BEAM is not checking a repository, sprint board, test run, or deployment state.

Scoring.

BEAM breaks the expected answer into small required claims, or “nuggets.” Each nugget receives no, partial, or full credit:

$$[ s(q)=\frac{1}{K}\sum_{k=1}^{K}J(\hat y,g_k), \qquad J(\hat y,g_k)\in\left\{0,\frac12,1\right\}. ]$$

This is a meaningful improvement over LoCoMo’s word overlap and LongMemEval’s single binary decision. Event ordering uses a separate sequence score. Since every ability contributes the same number of questions, all ten abilities receive equal weight.

BEAM does not report a comparable “months of use” span. Its scale is tokens and messages inside a completed conversation. The tiers also use different histories and sample sizes; the 10M tier joins ten interlocking plans. So the score drop from 1M to 10M cannot be attributed to length alone.

BEAM is strong evidence for a product whose main memory really is an extremely long conversation.

The transfer is less direct for coding systems, personal computers, or company memory, where the current state may live in files, repositories, tests, databases, tickets, or other systems. Hindsight compares BEAM’s ten million tokens with company documentation or the output of a software project. The size may be comparable. BEAM does not establish that the underlying memory problem is comparable.

Section 1 has now done the close reading. The next question is what complete experiment sits behind a number—and which parts of that experiment must still match before the number can be used as evidence for another product.

2. Why these benchmarks don't measure today's Agent-Native work

TLDR:

• Agent Native work is more complex and generates higher quantity and complexity of information.
• All historical data is causal, related and introduces more complexity and challenges for a memory system.
• A memory system has to perform coherence throughout the evolving scope of work.

The agent native work looks different today. A person working with agents now writes a request and then waits. The agent may then spend a long time, reading files, searching, running commands, editing code, reading and opening pages, computer use, spawning subagents, producing artifacts, and downloading artifacts.

The work leaves several overlapping kinds of evidence:

• session traces preserve requests, attempts, reasoning, observations, corrections, and claims;
• executable state preserves what code, configuration, tests, commits, and processes actually existed or ran;
• durable artifacts preserve documents, papers, datasets, reports, images, exports, and intermediate outputs that may outlive the session.

And each user has their own way to go about it, with their own preferences, using concurrent sessions, across their choice of harnesses beyond the standard ones, or some might make their own harnesses. Some sessions may start and end within an hour, while other sessions may go for days.

The models keep evolving and their capabilities change, the tools also evolve. And each user may have their own way of scoping their work and projects. Some may scope and separate each of their repos, not needing or requiring anything from any other repo's work. Others may have different repos, which are part of the same general work thread.

For example solopreneurs, as an evolving category will have to have their agents reuse the insights (the how and why) from the sessions of past projects into different projects, or may use the downloaded research from an unrelated project in a different context, all within the context of their company. Research to Backend to Marketing.

If we call this combined set a work-state, then it becomes clear that one real demand of a memory system is to work in an evolving work-state environment, and keep up with it.

This is the first major difference from the previous section. There are three important things here.

1. The state is not just a conversation in a chat history but an evolving work state, operating in a real computer with durable artifacts.
2. The task is not a sequential end-of-history QA but multiple queries asked in that evolving work state at different time $t$.
3. The noise generated is not unrelated filler for needle in a haystack, the noise is retrospective information in a casually evolving world. The answer depends on independently originated sources, none of which is sufficient alone.

We will unpack these 3 below,

2.1 Beyond a conversational chat interface

There is a clean formal way to describe part of this difference, and AMA-Bench is the strongest reference we had found earlier. The important move is that the remembered record comes from an agent interacting with an environment.

The paper models agent–environment interaction as a partially observable Markov decision process:

$$\mathcal{M} = (\mathcal{S}, \mathcal{A}, \mathcal{O}, P, r).$$

In plain language:

• the environment has a state $s_t$;
• the agent does not see all of it directly;
• it takes an action $a_t$;
• it receives an observation $o_t$;
• the action may move the environment into another state. The accumulated trajectory is:

$$h_t = (x, a_1, o_1, \ldots, a_t, o_t),$$

where $x$ is the original task instruction.

The memory system builds state from that trajectory and later retrieves context for a query:

$$m_t = \operatorname{Build}(h_t), \qquad c_t = \operatorname{Retrieve}(m_t, q_t).$$

The POMDP language matters because the agent does not directly see the complete state of the environment. It sees observations produced by its actions. A command reveals one thing; a database query reveals another; a failed test changes what the agent should believe about the patch it just wrote.

I would not impose a literal POMDP on every personal-computing system ( I personally would ). The useful point is simpler: agent work is partially observed, actions change the process, and later evidence depends on what happened earlier.

This becomes useful in personal computing and company memory context. A person or organization produces evidence across conversations, repositories, files, tickets, browser sessions, databases, documents, and other agents. Those sources were not jointly written to answer one future question. They have different purposes, time boundaries, permissions, and authority.

AMA gives us:

one task instruction
→ action–observation trajectory
→ later query

Personal computing and company-brain memory introduce a larger continuity problem:

one person (or several)
→ several harnesses, and sessions
→ several overlapping work objects
→ traces, code, files and services changing over time
→ queries arising throughout that process

The session receiving a request is the immediate access context. It is not necessarily the full provenance boundary of the answer.

For example one real query shape from our work, lightly combined for public use, was:

We are preparing the research report on our judge-calibration study. Reconstruct how the baseline and scoring design changed from the original experiments through the paper review, rubric-engine rewrite, schema migration, and adversarial audit. Determine whether the apparent reliability improvement reflects a real change in judged behaviour or a change in which score field was being compared; which thresholds came from published research and which were local choices; which results and figures remain defensible; and what must be recomputed before publication. Cite the papers, sessions, commits, code paths, and raw run artifacts. Separate what the team believed at each stage from what the evidence supports now, and end with the smallest experiment that would resolve the remaining uncertainty.

It sounds like a paper-editing question. But no single paper draft contains the answer.

The system has to combine research papers and reading notes, old agent sessions, Git history, rubric specifications, schema changes, analysis scripts, raw judge outputs, tables, retractions, and the current report. Those objects were produced by different user-agent interactions in different harnesses and different states of the research.

This particular thread spans about five weeks, across fourteen Claude Code and Codex sessions, from the first baseline experiments to the later paper review, implementation changes, numerical corrections, and adversarial audit. The task is to retrieve the right objects amongst a larger ocean of related work, and then determine how they relate: what the team believed at each stage, which number was computed from which field, which interpretation later became invalid, which source now has authority, and what still needs to be run. And then finally compose a defensible answer in natural language space with citations.

This is a standard expectation from a modern memory system as of mid 2026.

A good response therefore does not retell the archive. It reconstructs enough of the operative state for useful continuation.

The insight here is:

1. reconstruct history where history is required
2. inspect the current world where current authority lives
3. compose the answer resolving the contradictions

2.2 Questions at different times in the same evolving workspace

The second major difference is that questions keep emerging at different times in the same evolving work-state. They can be asked while the work is still changing. The intuition is that a memory system lives inside the work-state, over months of work, and ideally forever, so it must be measured for that.

Take one simple question:

What is the current release plan?

At time $t_1$, the supported answer might be:

tests have passed
release candidate is ready
deployment is scheduled

At $t_2$, after a failed deployment:

deployment is blocked
migration failed
rollback is in progress
the previous release plan is no longer current

At $t_3$, after a corrective patch:

rollback is complete
the new patch is under review
deployment has been rescheduled
one monitoring issue remains unresolved

The question string did not change, only the work-state did and so the answer for the same query changes depending on when the question is asked.

This gives us a different kind of evaluation.

• Did the answer change when the relevant world changed?
• Did it remain stable when the relevant world did not?
• Did it avoid using evidence that appeared only later? (relevant when making a benchmark)

This is not merely an explicit knowledge update placed inside one completed fixture. The memory process is being probed at several checkpoints while the larger process continues.

2.3 Real lived work-state has no noise and can't be generated looking backward

The third difference is the shape of the noise.

LongMemEval deliberately surrounds answer evidence with unrelated, non-conflicting filler. This creates controlled search pressure without destabilizing the intended answer.

Real work contains related residue:

• work on the same module, creating same keyword set
• research corpus that has same bag of words
• an abandoned thread
• several near-identical drafts.
• an old plan that still sounds plausible but is definitely wrong
• a summary written before a later correction
• an exploratory paper whose ideas reappear in another project.

This material is not automatically irrelevant. It was produced by the same evolving process. It may explain why the present has its current shape.

The memory system may need to preserve distinctions such as:

this happened
this was attempted
this was believed at the time
this later failed
this remains current
this should no longer guide action

And the future question is not known while the record is being created. Naturally accumulated work is produced before anybody knows which detail will matter later.

A failed test is recorded before anyone knows that three weeks later someone will ask why the release slipped. A paper is downloaded and processed and experimented with before anyone knows that one paragraph will cause a design reversal a few weeks later. A temporary workaround may later become either irrelevant clutter or the missing explanation for the current architecture.

The memory has to preserve useful identities, relations, ordering, and uncertainty under unknown future demands.

To ask “What is the current release plan?” at $t_1$, $t_2$, and $t_3$, we first need a defensible account of what evidence existed at each point, what the candidate was allowed to see, what remained unavailable, and what the judge could use without giving the candidate hindsight.

Taking this argument further, the shape of the noise is contextual to the user, and the company. It depends on the specific personalized way of working with agents as we discussed above. They can be using the same tools in a substantially different ways. Different decision making processes and shared responsibility are specific to each entity, and will generate noise for a memory evaluation. So how to design the right evaluations ?

We have to construct the bounded world in which that question can be answered and judged.

That is one of the problems we are trying to make concrete in Replay Lab.

3. Designing memory benchmarks for agent-native work

Popular benchmarks usually begin with questions and construct histories around them.

Replay Lab reverses that order:

1. preserve the world
2. reconstruct its history
3. sample real tasks
4. admit only defensible tasks
5. evaluate whether memory returns useful working state

We have a real work archive. We have a real set of questions emerging from that archive, either from the native harness queries where the work is done, or a custom memory solution. We will know the exact time and session in which the question was asked. Whether it existed from a memory system already implemented or whether the queries are extracted from work harnesses itself, after data cleaning, we end up with real queries.

Then we define the boundary at each state. How the work-state evolved over time, measure in bounded slices. Why ?

Suppose the final archive contains an abandoned plan from April, a correction written in May, and a clean summary produced in June. If we give all three to a system and ask what was happening in April, the system has hindsight. It may produce an excellent summary of how the project ended while failing the historical reconstruction task completely.

We define the memory problem, not as a retrieval problem, but rather a safe and useful reconstruction of the work-state at time t.

The reference-time model begins with a limitation that is easy to forget while building a benchmark: the complete lived state was never recorded.

Let $X_t$ stand for what was actually going on at time $t$: the user’s active intent, abandoned work, files, agent context, decisions, confusion, social context, and facts that were never written down. It also encompasses the shared understanding of the work between user and the agent, which may not be true in a new session, as the transformer will update it's biases differently from the user, and gradually the same words will start meaning different things to the transformer if left unchecked.

Tip: Maintain a glossary of your work in your memory system beyond the object and category names in code.

Now.

We do not possess $X_t$. This state is hidden and is only partially observed through the evidence present on the machine.

What we have is a surviving evidence surface $E_{\le t}$: files, Git history, traces, path dates, messages, manifests, external references, and later human annotations when those annotations are explicitly part of benchmark construction.

The candidate receives a bounded state reconstructed from that evidence:

$$S_t=\operatorname{State}(E_{\le t},t).$$

In plain language: $X_t$ is what really happened, $E_{\le t}$ is the partial record that survived, and $S_t$ is what the environment can responsibly expose at that historical moment.

A candidate carries its own memory through the replay:

$$m_{c,t}=\operatorname{update}_{M_c}(m_{c,t-1},S_t), \qquad o_{c,t}=M_c(q_t,S_t,m_{c,t}).$$

When an query $q_t$ arrives, the candidate answers from the current bounded state and its accumulated private memory.

A judge receives the output together with a private packet $Z_t$, bounded to the same reference time. That packet may contain admitted evidence, expected claims, tempting stale material, and the rubric. It never enters candidate memory and cannot write back into it.

Scoring

In agent native work, a response to a query from a memory system has to be a long form natural language answer, that may or may not contain exact file artifacts. It is necessary for it have to citations.

But reconstruction (or composition) is either a non verifiable task, or partially verifiable.

Because only a part of it can be deterministic (ex. the artifacts, sessions IDs, etc.). But the composition of those artifacts as a reconstruction is a prose with transformer reasoning in between, and the measurement for that has to be a subjective non deterministic measurement done by an LLM-judge.

At most we can design rubrics with per criteria references, so we can arrive at a better grading reference for the LLM-judge. There is a whole set of research here in how to design these rubrics, and multiple ways to normalize them into a scalar score across different types of task. Ideally, we mix a hybrid of deterministically verifiable rubrics and non deterministic.

However, a long-form reconstruction is not wholly subjective, and it is not wholly deterministic.

For example. Some claims can be checked exactly: did the cited commit exist at the reference time, which field did the scorer read, were the two reported numbers calculated over the same cells, and does the corrected arithmetic produce the stated result? Other claims require judgment: did the answer distinguish a real experimental finding from a schema-migration artifact, trace why the team’s belief changed, separate paper-derived methods from local choices, and propose an experiment that actually resolves the remaining uncertainty?

The evaluator therefore needs more than a model asked whether the answer “looks good.” It needs the task’s admitted evidence, one criterion per claim, and the right verifier for each criterion. This is the same basic move made by Interactive Evaluation Requires a Design Science: specify which artifacts count as evidence and how they become judgments. PReMISE makes the related point that a rubric is a measurement specification; changing it changes the evaluator.

Using the publication question from the previous section, one defensible answer at the 23 April 2026 reference time would be:

We should not publish the claim that the hybrid rubric improved cross-judge agreement by 21 percentage points. The original comparison placed the v1 score of 0.456, calculated over 57 cells, beside a v2-hybrid score of 0.667, calculated over nine selected cells. It also compared the judges’ raw overall_verdict field. On the nine cells shared by both rubrics, the authoritative engine verdict agrees in five cases under v1 and five under v2-hybrid: (5/9=0.556) for both. The supported improvement is therefore zero, not 21 points. [WHERE_WE_ARE_20260421.html; original sweep 3891c0a7, correction 4688ddb]

The apparent finding came from a migration in what “the verdict” meant. Under v1, overall_verdict passed directly through as the score. Under v2-hybrid, the engine computed its own label from criterion scores using a weighted equation and ignored overall_verdict, even though the old schema still required the judge to emit it. Six of nine cells contain a disagreement between that ignored field and the engine label. The count is real; the interpretation is not. It cannot support the earlier claim about judge “gestalt.” The pilot also used a v1-shaped prompt that failed to show the judge v2-hybrid’s criterion guidance, so those nine cells are not calibration-grade evidence for the new rubric. [WHERE_WE_ARE_20260421.html, measurement-honesty correction 4688ddb]

Some of the new design is still well grounded. Autorubric supplies the per-criterion architecture and normalized weighted aggregation. Yeadon et al. motivate separating highly referenceable criteria from essay-like judgment. Zhang motivates retaining holistic judgment where atomic checks miss global support. Feuer et al. supply the factor-collapse diagnostic. But the local cutoff (|\rho|<0.85), the pass and partial thresholds, the penalty weight, and the three-by-three rubric shape are our calibration choices, not values established by those papers. The (\kappa) gate is adapted from the Landis–Koch “moderate agreement” convention. [RUBRIC_ENGINE_SPEC.html, value-provenance table at 6659d867]

What remains defensible is the engineering result: the rubric can be represented as typed criteria, aggregated deterministically, and audited criterion by criterion. What is not yet defensible is an empirical claim that the current rubric is more reliable, or that the old experiment ranks memory conditions. The current engine had no fresh calibration run at this reference time. The environment audit also found that 32 of 38 zero-condition cells read the supposedly hidden slice and that roughly 70% of the probes could collapse into retrieval, so another judge-only pass cannot repair the broader architecture comparison. The smallest experiment for the rubric claim is a locked, same-cell A/B using the current prompt and rubric, the same answer artifacts and judges, and both per-criterion agreement and engine-level scores. Any memory-architecture claim must wait for enforced evidence boundaries and a reconstruction-only probe set. [N1_FRAMING_IMPLICATIONS_20260420.md at 4272f24; adversarial audit at b17e4c41]

That answer moves across four kinds of evidence: papers that motivated the design, session history that explains what the team believed, Git commits and code that establish what the scorer actually did, and raw result tables that establish the corrected number. The answer does not have to use the same sentences. It has to preserve the same evidence-backed distinctions.

A compact rubric for this task could look like:

Each criterion can receive full, partial, or no credit. The result is a scalar score, but the breakdown still shows what the system reconstructed correctly and what it missed. How to construct and calibrate these rubrics is a separate problem that I will return to later.

The rubric is only the last step. The larger method begins with a real, evolving archive rather than a set of future questions. It preserves the evidence available at each reference time, samples questions that arose from the work itself, prevents the candidate from using later evidence, and then measures whether its memory reconstructed enough of the state for useful continuation.

This gives us a reproducible experiment. The same state, reference time, question, evidence boundary, and judge packet can be replayed against another memory system. It can scale across more questions, time slices, environments, and candidates. It can also be customized around the memory obligation: a coding team may care about branches, tests, and abandoned implementations; a research lab about evidence, claim lineage, and unresolved experiments; a company brain about authority, permissions, and changes that propagate across shared work.

This is not another universal benchmark. It is one way to make the experiment resemble the environment in which the memory system will be used. The resulting score can then make a precise claim: how well this system satisfied this kind of memory obligation, in this environment, under these evidence boundaries.

4. Conclusion

A memory system can score highly on LoCoMo, LongMemEval, or BEAM and still fail when a user asks it to continue real work. The contradiction is less about the capabilities of memory system and more about what is being observed and what is the measuring instrument.

Evaluating implemented memory system against a small sample, or more dangerously the wrong sample of data, will make it harder to work with multiple agents as the information density, complexity and quantity increases.

There is another design question to be had of the infrastructure to run these stateful evaluations because it turns out choosing any standard sandbox environment (docker, firecracker, kata) is not good enough for historical reconstruction, and requires special tweaks and tuning, the right answer is using qemu-microvm, and we will explore why in a future article.

There are details of rubric based evaluations, which need elaboration, because an LLM-judge is powerful but comes with its own quirks.

The positioning we take at Replay Lab is that each individual and each organization should have their own custom evaluation methodology so their memory system actually works for the details that they care about, whether it's provenance, time saved, cost saved, or proactiveness. All these measures are entirely unique to each entity (person, or a team, or an organization). Also the conversation of benchmarking extends to an environment because the status quo is now self evolving agents and harnesses that need a fresher and better signals as the harnesses, agents and tools evolve and get better.

Our work with Syke (https://github.com/saxenauts/syke) is a self evolving memory harness, and our position is that each harness should evolve to adapt to the shape of information of the entity that depends on it.

Beyond a static evaluation, we will later extend the idea of dynamically moving evaluations that keep evolving autonomously.

For now, we are working on evaluating some popular memory systems in our methodology, and will release a leaderboard soon. Meanwhile you can check out the appendix for a curated list of memory benchmarks that are useful and valuable in their given contexts.

Appendix

The benchmarks below are grouped by the memory obligation they make observable, not by publication date. Many span more than one category; each appears once under its primary task.

Conversation and personal-history memory

Workspaces, projects, artifacts, and distributed sources

Agent trajectories and consequential use

Currentness, forgetting, and memory lifespan

Prospective and self-evolving memory

Shared-memory governance and security

Real-trace and environment neighbours

These are part of the research corpus because they expose data and environment shapes that memory benchmarks can inherit; they are not all memory benchmarks.

Measurement and benchmark-design references