← all documents · raw markdown · 5 KB

RSHL Continual-Memory Benchmark — Method

Harness: benchmarks/rshl_memory_bench.mjs · Schema: rshl-memory-bench/1.0 · Drafted: 2026-08-17

---

1. The claim under test

RSHL's architectural claim is not that it is fast, and not that it retrieves well.

Those are already measured — see reports/RSHL-BENCH-COMPARISON-2026-07-02.md, which

covers encode throughput, cosine throughput, scan bandwidth and recall@1 under bundle

superposition at 4096 and 16384 dimensions.

The claim is narrower and has never been measured here:

> A memory that keeps learning, and stays accountable while it does.

Both halves matter, and the published field currently splits between them:

| System | Learns continuously | Per-fact provenance |

|---|---|---|

| Titans / MIRAS (Google, Jan 2025) | yes — memory updates at inference | no — memory is distributed weights |

| MemOS (Jul 2025), Eywa (May 2026) | no — store over a frozen model | yes — evidence separated from belief |

| RSHL | yes — Hebbian reinforcement, gated consolidation, decay, sleep | yes — cell-level source, timestamp, strength |

That intersection is the thing to score, because it is the thing nobody else is

positioned to score. Every metric below exists to test one half or the other.

---

2. Metrics, and the question each one answers

| Metric | Question a skeptic would ask | Why this metric |

|---|---|---|

| recall1_before / recall5_before | Can it get a stored fact back at all? | Floor. Everything else is meaningless without it. |

| retention | Does learning 600 new things damage the first 120? | Catastrophic forgetting — the standard continual-learning metric, so this number is directly comparable to the literature. |

| updateWinRate | When a fact *changes*, does the new value outrank the old? | A frozen model scores 0 by construction — it cannot learn the change at all. This is the single clearest separation from RAG. |

| provenanceAccuracy | Can it name the source and date of what it just said? | A distributed-weight memory scores 0 by construction — there is no per-fact record to attach either to. This is the separation from Titans. |

| gateRecall / falseRefusals | Does it refuse junk, and *only* junk? | Continual learning without a gate is a machine for accumulating garbage. Both halves are needed: refusing everything scores 1.0 and is useless. |

| reinforcementDelta | Does retrieval actually strengthen a cell? | Tests that Hebbian learning is real and not decorative. |

| latency_p50_ms / p95 | Is any of this affordable? | An accountable memory nobody can wait for is not a product. |

---

3. Protocol

1. Seed N facts (default 120), unique subject → unique value, source=bench-<run>-e1.

2. Probe each fact by natural-language question. Record recall@1, recall@5, provenance, latency.

3. Interfere: ingest N noise cells (default 600), unrelated, same region.

4. Re-probe the original facts. retention = recall1_after / recall1_before.

5. Update a subset (default 30) with new values, source=bench-<run>-e2, later timestamps.

Re-probe: does the new value outrank the old? → updateWinRate.

6. Gate: submit 60 deliberately refusable items (empty, too-short, exact duplicates)

and 30 clean ones. → gateRecall, falseRefusals.

7. Reinforce: measure strength on 10 cells, retrieve each 5 times, measure again.

Data is generated from a fixed seed, so two runs on one machine are comparable and a

regression is a regression rather than a different draw.

---

4. What this does NOT show

Stated here rather than discovered by a reviewer:

Synthetic facts. Templated sentences, not natural language from a real corpus.

Retrieval on templates is easier than retrieval on prose.

One machine, one run. No variance across seeds, no confidence intervals.

A single number from a single run is an anecdote with decimals.

No external baseline. It scores RSHL against itself over time. It does not run

the same protocol against Eywa, MemOS or a vector-DB-plus-LLM stack, so it cannot

yet support any claim of the form "better than X".

updateWinRate is ranking-based. It asks whether the new value outranks the

old, not whether the old was erased. Those are different properties and only the

first is measured.

It writes to the live lattice. Every cell is tagged bench-<runId>-* in region

benchmark and must be cleaned up afterwards. Numbers taken from a lattice still

holding a previous run's cells are not clean numbers.

---

5. What would make this publishable

In rough order of how much each buys:

1. A baseline. Same protocol, same facts, against a vector DB + frozen LLM.

updateWinRate and provenanceAccuracy should separate sharply. Without a

baseline the numbers describe RSHL; with one they make an argument.

2. Variance. Five seeds, report mean and spread.

3. Natural language. Swap templated facts for a real corpus slice.

4. A hostile reviewer's pass over §4 — the caveat list is the credibility, and

the fastest way to lose an argument is to have someone else find the caveat first.

---

6. Results

|---|---|---|---|---|---|---|---|

node benchmarks/rshl_memory_bench.mjs              # plan only, writes nothing
node benchmarks/rshl_memory_bench.mjs --run --save # measure, write report to reports/