# RSHL Continual-Memory Benchmark — Method

**Harness:** `benchmarks/rshl_memory_bench.mjs` · **Schema:** `rshl-memory-bench/1.0` · **Drafted:** 2026-08-17

---

## 1. The claim under test

RSHL's architectural claim is not that it is fast, and not that it retrieves well.
Those are already measured — see `reports/RSHL-BENCH-COMPARISON-2026-07-02.md`, which
covers encode throughput, cosine throughput, scan bandwidth and recall@1 under bundle
superposition at 4096 and 16384 dimensions.

The claim is narrower and has never been measured here:

> **A memory that keeps learning, and stays accountable while it does.**

Both halves matter, and the published field currently splits between them:

| System | Learns continuously | Per-fact provenance |
|---|---|---|
| Titans / MIRAS (Google, Jan 2025) | **yes** — memory updates at inference | no — memory is distributed weights |
| MemOS (Jul 2025), Eywa (May 2026) | no — store over a frozen model | **yes** — evidence separated from belief |
| RSHL | yes — Hebbian reinforcement, gated consolidation, decay, sleep | yes — cell-level source, timestamp, strength |

That intersection is the thing to score, because it is the thing nobody else is
positioned to score. Every metric below exists to test one half or the other.

---

## 2. Metrics, and the question each one answers

| Metric | Question a skeptic would ask | Why this metric |
|---|---|---|
| `recall1_before` / `recall5_before` | Can it get a stored fact back at all? | Floor. Everything else is meaningless without it. |
| `retention` | Does learning 600 new things damage the first 120? | **Catastrophic forgetting** — the standard continual-learning metric, so this number is directly comparable to the literature. |
| `updateWinRate` | When a fact *changes*, does the new value outrank the old? | A frozen model scores **0 by construction** — it cannot learn the change at all. This is the single clearest separation from RAG. |
| `provenanceAccuracy` | Can it name the source and date of what it just said? | A distributed-weight memory scores **0 by construction** — there is no per-fact record to attach either to. This is the separation from Titans. |
| `gateRecall` / `falseRefusals` | Does it refuse junk, and *only* junk? | Continual learning without a gate is a machine for accumulating garbage. Both halves are needed: refusing everything scores 1.0 and is useless. |
| `reinforcementDelta` | Does retrieval actually strengthen a cell? | Tests that Hebbian learning is real and not decorative. |
| `latency_p50_ms` / `p95` | Is any of this affordable? | An accountable memory nobody can wait for is not a product. |

---

## 3. Protocol

1. **Seed** N facts (default 120), unique subject → unique value, `source=bench-<run>-e1`.
2. **Probe** each fact by natural-language question. Record recall@1, recall@5, provenance, latency.
3. **Interfere**: ingest N noise cells (default 600), unrelated, same region.
4. **Re-probe** the original facts. `retention = recall1_after / recall1_before`.
5. **Update** a subset (default 30) with new values, `source=bench-<run>-e2`, later timestamps.
   Re-probe: does the new value outrank the old? → `updateWinRate`.
6. **Gate**: submit 60 deliberately refusable items (empty, too-short, exact duplicates)
   and 30 clean ones. → `gateRecall`, `falseRefusals`.
7. **Reinforce**: measure `strength` on 10 cells, retrieve each 5 times, measure again.

Data is generated from a fixed seed, so two runs on one machine are comparable and a
regression is a regression rather than a different draw.

---

## 4. What this does NOT show

Stated here rather than discovered by a reviewer:

- **Synthetic facts.** Templated sentences, not natural language from a real corpus.
  Retrieval on templates is easier than retrieval on prose.
- **One machine, one run.** No variance across seeds, no confidence intervals.
  A single number from a single run is an anecdote with decimals.
- **No external baseline.** It scores RSHL against itself over time. It does not run
  the same protocol against Eywa, MemOS or a vector-DB-plus-LLM stack, so it cannot
  yet support any claim of the form "better than X".
- **`updateWinRate` is ranking-based.** It asks whether the new value outranks the
  old, not whether the old was erased. Those are different properties and only the
  first is measured.
- **It writes to the live lattice.** Every cell is tagged `bench-<runId>-*` in region
  `benchmark` and must be cleaned up afterwards. Numbers taken from a lattice still
  holding a previous run's cells are not clean numbers.

---

## 5. What would make this publishable

In rough order of how much each buys:

1. **A baseline.** Same protocol, same facts, against a vector DB + frozen LLM.
   `updateWinRate` and `provenanceAccuracy` should separate sharply. Without a
   baseline the numbers describe RSHL; with one they make an argument.
2. **Variance.** Five seeds, report mean and spread.
3. **Natural language.** Swap templated facts for a real corpus slice.
4. **A hostile reviewer's pass** over §4 — the caveat list is the credibility, and
   the fastest way to lose an argument is to have someone else find the caveat first.

---

## 6. Results

| Run | Date | recall@1 | retention | updateWinRate | provenance | gateRecall | p50 |
|---|---|---|---|---|---|---|---|
| _(pending first run)_ | | | | | | | |

```
node benchmarks/rshl_memory_bench.mjs              # plan only, writes nothing
node benchmarks/rshl_memory_bench.mjs --run --save # measure, write report to reports/
```