# KAI Memory Engine — Scale-Out Design (Resonance-Addressed, Tiered, Self-Tuning) **Owner:** Ryan (nastermodx) · **Drafted:** 2026-07-02 · **Status:** design / goal (not yet built) **Companion evidence:** `reports/RSHL-BENCHMARK-AND-NEXT-PHASE-2026-07-02.md` (the math) and `benchmarks/rshl_scale_sweep.mjs` (run it to see the wall). > KAI is a **memory engine** — storage and recall, not a chatbot. This doc is about scaling that memory engine to millions of cells while keeping recall fast, on whatever hardware it runs on. --- ## 1. The problem (measured, not guessed) Today the live engine answers queries by **full parallel scan** — it compares your query against *every* cell. Measured on John (`rshl_scale_sweep.mjs`, DIM 16384): - per-cell scan cost is a dead-flat **~1.25 µs/cell** → query latency is **O(N)** (linear in cell count). - real-time crossover ≈ **12.8 K cells** (single-thread; the Rust parallel scan pushes it higher, same shape). - at **250 K cells** a query is already **331 ms** (3 queries/sec) while using < 4 GB of 42 GB. - RAM could **store ~2.57 M cells**, but full scan can only **query ~12.8 K in real time** → a **~201× gap** between what fits and what's usable. **So the memory is nearly free; the *scan* is the wall.** Everything below is about not scanning what we don't need. --- ## 2. The architecture (the idea, named) **A resonance-addressed, hardware-tiered, self-tuning memory engine.** ### 2a. Resonance-addressed symbol / section index Cluster the cells into **sections**. Each section is represented by ONE summary vector — a **symbol**. A query doesn't scan all cells; it **resonates** against the symbols first (a few thousand comparisons), and the strongest-resonating symbols point to the sections that actually hold the answer. Then it only scans cells inside those sections. `5M → ~few-thousand symbols + ~few-thousand cells`. This is an inverted-file / centroid index (IVF), driven by RSHL resonance instead of plain distance. *(Already half-built: `KMeansIndex` "sub-500 µs routing" and the `HNSW` graph in `universe.rs`.)* ### 2b. Prefetch-on-read (streaming) As KAI **parses the input**, each concept it reads fires its symbol, the symbol resonates to that concept's section, and that section is pulled into the fast tier **just in time** to be searched. We don't load everything — we **stream in exactly the sections the input is asking about, in the order it asks**. By the time he's "reading" the word, its region is already hot. ### 2c. Cross-stream tiering (GPU / RAM / SSD as one path) - **GPU cores** run the resonance math — one query vs thousands of cells in parallel (the cosine/phasor kernel). *(Bones exist: `gpu_kernel.rs`, `wgpu`.)* - **RAM** holds the hot working set — the symbol index + currently-warm sections. - **SSD (memory-mapped)** holds the cold bulk of the 5M cells, streamed in only for sections resonance points to. *(`memmap2` already a dependency.)* - **Adaptive spill:** start in VRAM + RAM; when the working set overflows, spill to SSD — trade a little speed to stay **stable**, never fall over. ### 2d. Save with connections (structure over duplication) As KAI grows, don't store everything densely and redundantly — lean on the **edges** (associations between cells). Store the relationship, derive the rest. Graph edges (HNSW / associative links) compress meaning: more held in less space. ### 2e. Self-tuning controller ("algorithms that build algorithms") A small controller that, given **the hardware in front of it + current cell count + the recall target**, *picks and tunes the strategy*: full-scan when small, resonance-router + N probes when big, GPU-batch under heavy query load, SSD-tier when it outgrows RAM. Not magic — it **selects and tunes over known strategies** from measured tradeoffs. *(Seed already built: the bench auto-detects the machine and auto-picks the fastest kernel + auto-scales the workload.)* ### 2f. Self-sufficiency KAI **auto-decides everything knowable** — detects specs, benchmarks himself, researches facts, chooses the index — and does **not** bounce those questions back to the user. He only checks in on things that are the user's to choose (preferences, goals). *Facts: he finds them himself. Wants: he confirms.* --- ## 3. The one law the design must obey Moving data between tiers costs time — **RAM ≈ nanoseconds, PCIe-to-GPU ≈ microseconds, SSD ≈ ~100 µs (~1000× RAM).** So the entire game is **minimizing crossings**: keep hot data where the compute is, batch GPU work so one transfer serves thousands of comparisons, touch the SSD rarely and in big sequential gulps. Every "adaptive/stable" decision is really a decision about *where to put the data so you cross the least.* Respect this and it flies. ### Known tradeoff knobs - **Probes vs recall:** the router is *approximate* — if the right cell sits in a section whose symbol didn't resonate, you miss it. Check the top-N sections (more probes) to raise recall at the cost of speed. `probe_sweep.rs` already exists to find the smallest N that keeps recall ≥ 95%. - **Superposition cap:** bundling many cells into ONE working vector craters past ~32 concepts (measured). So a vast context is held as a **list of retrieved cells**, not one blended symbol. Vast retrieval: yes. Vast blending: capped. --- ## 4. What already exists in the codebase (don't reinvent) - `KMeansIndex` (`sparse_vec.rs`) — the symbol/section router ("sub-500 µs"), on `Universe.kmeans_index`, built by `rebuild_index()`. - `HNSW` (`hnsw_rs`) — the branching graph, on `Universe.hnsw`, rebuilt in `rebuild_index()`. - `cosine_avx512` (VPOPCNTDQ) — live SIMD in the similarity path (`sparse_vec.rs`), scalar fallback. - `gpu_kernel.rs` + `wgpu` — GPU compute bones. - `memmap2` — dependency in place for disk-backing. - `probe_sweep.rs`, `hnsw_bench.rs` — tuning harnesses already in-tree. - **The gap:** the engine **deliberately skips building the index at boot** (`src/core/engine.rs:433`: `// universe.rebuild_index(0.0); // Skipped ...`) and the mask_pool/KMeans build is **capped at ~200 K cells** (`universe.rs:879`). So live queries fall back to full scan. --- ## 5. The plan (phased, benchtest-before-wire) ### M1 — Activate the existing ANN index at scale (PRIMARY, low-risk) 1. **Prove the O(N) wall** (done): `rshl_scale_sweep.mjs` → measured ~1.25 µs/cell, ~201× gap. 2. **Measure the fix** on the real engine: `cargo run --release --bin probe_sweep` and `--bin hnsw_bench` at 10 K / 50 K / 100 K / 250 K cells → recall@K + latency vs full scan; find the crossover where the index wins. 3. **Wire activation:** replace the disabled `engine.rs:433` with an **auto-`rebuild_index` at a cell-count threshold** (the measured crossover); keep instant-boot full scan for small lattices. Flag-gated `KAI_LATTICE_INDEX`, reversible. 4. **Lift the 200 K mask_pool cap** (`universe.rs:879`) once RAM headroom is confirmed. 5. **Verify:** recall@K within noise of full scan (correctness gate); query p95 stays under target as cells grow to 100 K+. ### M2 — Resonance router + prefetch + tiering + self-tuning (the full vision) 1. **Resonance routing:** drive the section router by RSHL resonance (phasor coherence), not plain distance (ties into fixing `phase_angle()` to carry real semantic phase — ALGORITHM-INVENTORY #1). 2. **Prefetch-on-read:** as input is parsed, fire symbols per concept and stream the pointed-to sections into RAM/VRAM just-in-time. 3. **Cross-stream tiering:** GPU resonance kernel (`gpu_kernel.rs`/`wgpu`) + RAM hot set + SSD memmap cold bulk; adaptive spill; minimize crossings. 4. **Self-tuning controller:** generalize the bench's auto-kernel/auto-scale seed into a strategy selector (full-scan | probes | HNSW | GPU-batch | disk-tier) keyed on hardware + cell count + recall target. 5. **Connections-over-duplication:** use graph edges to compress as the lattice grows. **Done when:** a 1M+-cell lattice, mostly on SSD with a hot RAM/VRAM working set, answers queries in a few ms at recall within noise of full scan, and the engine picks its own strategy per machine without being told. --- ## 6. Constraints (repo hard rules — do not violate) - **Surgical, reversible, flag-gated edits only.** Never whole-file-rewrite `oracle.html` / `kaiverse.js`. Confirm before changing core query/encode behavior. - **Don't break the browser KAIVERSE** (the current visual-overhaul focus) — this memory work is a separate track. - **Benchtest before wiring** every step; verify against the **real Windows files** (mounted reads can be stale/truncated); show diffs as evidence, not timestamps. - **Keep the Codex updated** and bump `Cargo.toml` version in sync when the Codex version bumps. - Never touch the running engine or the persisted brain (`kai-state.json`) during benchmarking — throwaway in-memory only.