Beyond Retrieval Bench

Abstract

What is Beyond Retrieval Bench?

As context lengths of LLMs continue to grow, existing benchmarks remain retrieval-centric — testing whether models can locate and reproduce surface-level facts. We argue that the most valuable form of memory is functional memory: the ability to abstract from observations into underlying rules, compose distributed evidence, and revise beliefs when new information contradicts prior observations. We introduce Beyond Retrieval Bench, a procedurally generated benchmark spanning three complementary capabilities — Memory Retention, Memory Composition, and Memory Revision — across four symbolic domains with deterministic answer verification. Empirical evaluation reveals that even state-of-the-art long-context LLMs struggle with multi-stage composition and especially rule revision, highlighting a critical gap between current capabilities and true executable memory.

Task Types

Domains

∞

Token Contexts

∞

Reasoning Hops

Low

Leakage Risk

Motivation

Why Functional Memory Matters

Human memory is inherently imprecise and highly compressed — we remember the law, not every measurement. Current retrieval benchmarks miss this entirely.

🔬

The Scientist Analogy

A scientist does not memorize every data point. They remember the law that best explains the observations — and update it when new evidence arrives.

💻

The Programmer Analogy

A programmer does not recall every line of a codebase. They internalize the design invariant that governs how changes should be made.

🤖

The Assistant Analogy

A user does not expect an assistant to recite past examples verbatim, but to infer a preference or policy and apply it correctly in new situations.

Benchmark Design

Three Complementary Capabilities

Each task tests a distinct aspect of functional memory, from single-rule retention to compositional integration to belief revision.

Memory Retention

Preserve inferred rules across long distractors

The model observes examples of a hidden operator f early in the context, then must retain and apply the rule to a absolute new input after a long distractor passage.

[Instruction prefix] [Examples of operator f] f(4, 7) → 29 f(10, 2) → 26 ... [Long distractor · · ·] [Query: f(1, 1) = ?]

Memory Composition

Integrate distributed rule fragments

Multiple operators are scattered across the context. The final query requires composing them: y = f_K(f_K−1(⋯f₁(x)⋯)).

[Examples of f₁] [Distractor · · ·] [Examples of f₂] [Distractor · · ·] [Examples of fₖ] [Distractor · · ·] [Query: fₖ(⋯f₁(x)⋯) = ?]

Memory Revision

Update beliefs when rules change

The model first sees examples of f^old, then later examples of f^new. The query must be answered with the updated rule — not the original one.

[Initial examples of f_old] f_old(x) → ... [Distractor · · ·] [Updated examples of f_new] f_new(x) → ... [Distractor · · ·] [Query: current f(x*) = ?]

For all three tasks, the queries involve entirely novel problems that do not explicitly appear in the provided text. Therefore, the model cannot derive the answers through simple text matching. Instead, it must infer the correct operators and apply them to these new instances.

Domains

Four Symbolic Domains

Procedurally generated tasks span diverse rule types, preventing surface-level pattern matching and reducing data leakage risk.

🔢

Numerical Patterns

Arithmetic combinations, modular ops, digit-level transformations, piecewise expressions.

f(x₁, x₂) = 2·x₁ + 3·x₂

🔤

String Transformations

Reversal, rotation, prefix/suffix extraction, concatenation, delimiter wrapping.

f(x) = concat(reverse(x), first(x))

⚙️

Set Operations

Union, intersection, difference, symmetric diff, ordered selection, element relabeling.

f(A,B) = sorted((A∪B) \ (A∩B))

📁

File System Agent

Agent trajectory tasks: file moves, renames, and structured folder transformations.

move "draft" files → review/

Ablation Controls

Three Experimental Controls

Carefully designed controls isolate individual factors and enable principled ablation analysis.

Full Condition

All

Standard condition with examples followed by a long distractor and then the query.

[prefix], [examples], [distractor], [question]

Control A

Zero-Example

No demonstrations provided. Measures whether models exploit artifacts or prior knowledge.

[prefix], [distractor], [question]

Control B

Zero-Distance

Examples placed immediately before the query. Isolates memory-distance degradation.

[prefix], [examples], [question]

Comparison

How We Compare to Existing Benchmarks

Beyond Retrieval Bench uniquely combines long-range compositional reasoning, rule learning, and continual update — none of which are primary focuses in prior work.

Benchmark	Exact Text Match	Chain Reasoning	Max Hops	Max Length	NLU Tasks	Leakage Risk	Rule Learning	Update / Revision
NIAH	Yes	No	—	128K	No	Exists	No	No
RULER	Yes	No	—	128K	No	Exists	No	No
MRCR	Partial	No	—	≥1M	Partial	Exists	No	No
BABILong	Partial	Yes	≤320	≥10M	Yes	Exists	No	Partial
Michelangelo	Partial	Yes	≤20	≥1M	Yes	Low	No	Partial
LongBench v2	No	Partial	≤10	2M	Yes	Exists	Partial	No
NoliMa	No	Yes	≤4	128K	Partial	Exists	No	No
Ours (BRB)	Partial	Yes	∞(Theoretically)	∞(Theoretically)	Yes	Low ✦	Yes ✦	Yes ✦

✦ Primary design focus. Low leakage risk because all contexts are procedurally generated fresh for each evaluation.

Evaluation Methodology

Robust Accuracy via Hybrid Judgment

Each case is scored by four independent signals; a case is considered correct when at least three of the four signals agree.

Substring Match

Check whether the correct answer appears as a substring within the model response.

Regex Word-Boundary

Apply word-boundary regex matching on the full response to avoid false positives.

Post-Marker Regex

Extract the answer after the 【Answer】 marker and apply word-boundary matching.

LLM Judge

LLM-thinking independently judges whether the model response is semantically correct.

Voting rule: A response is marked correct (1) when ≥ 3 of the 4 signals return positive. Otherwise it is marked incorrect (0).

Experiment Setup

Scale & Conditions

📏

Context Lengths

Evaluated at 7 controlled lengths:

32K 64K 128K 200K 400K 700K 1M ∞

🔗

Composition Depth (K)

Memory Composition evaluated at K operators:

5 10 20 40 60 ∞

🔄

Revision Steps (edit_K)

Memory Revision evaluated at edit steps:

1 3 5 10 15 ∞

Evaluated Models

Model	Total Params	Active Params	Context Window	Architecture	Type	Access Date
DeepSeek-chat (v4-flash-preview)	284B	13B	1M	Transformer-style	Open-source	20260424
DeepSeek-reasoner (v4-flash-preview-thinking)	284B	13B	1M	Transformer-style	Open-source	20260424
DeepSeek-v4-pro (v4-pro-preview-thinking)	1.6T	49B	1M	Transformer-style	Open-source	20260424
MiniMax-M2.7	229B	10B	200K	Transformer-style	Open-source	20260318
Kimi-K2.6	1T	32B	256K	Transformer-style	Open-source	20260420
Kimi-K3	2.8T	~50B to 104B	1M	Hybird DeltaNet-style	Open-source	20260716
Qwen-3.5-plus	397B	17B	1M	Hybird DeltaNet-style	Open-source	20260216
Doubao-seed-1.6-flash	-	-	256K	-	Proprietary	20250611
Doubao-seed-2.0-pro	-	-	256K	-	Proprietary	20260214

Results

Experimental Results

Performance across three task types reveals distinct capability gaps in state-of-the-art long-context LLMs.

Memory Retention Results

Accuracy vs. Context length

Memory Composition Results

Accuracy vs. Context length

Memory Revision Results

Accuracy vs. Context length

Memory Retention Accuracy Drop

Accuracy drop vs. Context length

Memory Composition Accuracy Drop

Accuracy drop vs. Context length

Memory Revision Accuracy Drop

Accuracy drop vs. Context length

Pretrain Model Support

Pretrain-Compatible Evaluation

A dedicated version of Beyond Retrieval Bench is designed for evaluating base / pretrain models that have not been instruction-tuned. Instead of generating free-form answers, models are scored via perplexity (PPL) / log-probability over carefully designed option sets — eliminating any dependence on instruction following.

Evaluation protocol: For each question, the model is presented with a fixed candidate set. The candidate with the lowest perplexity (highest log-probability) is selected as the model's answer. Each distractor is crafted to target a specific failure mode, making results both diagnostic and interpretable.

🔁

Memory Retention

Tests whether the model retains and applies a rule encountered far earlier in the context.

✓ correct retrieval_similar random_wrong

retrieval_similar: plausible answer derived by surface-level token matching rather than rule application.

🔗

Memory Composition

Tests whether the model can compose two or more rules to produce a correct answer.

✓ correct surface_similar wrong_rule random_wrong

surface_similar: textually close but wrong; wrong_rule: applies one rule correctly but fails on composition.

🔄

Memory Revision

Tests whether the model updates a rule when a later rule overrides it.

✓ correct retrieval_similar old_rule random_wrong

retrieval_similar: based on surface matching; old_rule: applies the superseded rule, revealing failure to update.

Contributions

Key Contributions

Formalize functional memory as a long-context capability distinct from retrieval-based memory.
Introduce a benchmark evaluating rule retention, composition, and revision under controlled long-context conditions.
Design procedurally generated tasks across multiple symbolic domains with deterministic answer verification.
Evaluate contemporary long-context LLMs and show systematic failures in abstraction, composition, and especially updating.
Provide analysis dimensions including context length, rule complexity, distractor distance, and update interference.
Demonstrate that long context alone does not guarantee functional memory: attending to distant tokens is distinct from forming, preserving, composing, and revising rule representations.

Limitations & Future Work

Limitations

⚠️

Synthetic Data

Tasks are procedurally generated from symbolic templates, not drawn from natural-language documents. Performance on synthetic rules may not directly transfer to real-world scenarios.

🌍

Domain Coverage

Current domains cover numerical, string, set, and file-system tasks. Real-world functional memory (e.g., scientific hypotheses, social preferences) remains outside scope.

🚀

Future Directions

Results suggest new directions for model architectures and training paradigms explicitly targeting rule compression, long-range composition, and continual belief revision.

Authors

👤

Zhou Ying

Tsinghua University & Deepseek AI

👤

Wu Yuhan

Tsinghua University

Acknowledgements

Hu Shiqiang · Huang Yanwen · Chen Qinyu · Dai Damai · Chang Jiang · Mo Shulan · Jia Xiaoxuan