Owner: Ryan (nastermodx) · Drafted: 2026-07-02 · Status: design / goal (not yet built)
Companion evidence: reports/RSHL-BENCHMARK-AND-NEXT-PHASE-2026-07-02.md (the math) and benchmarks/rshl_scale_sweep.mjs (run it to see the wall).
> KAI is a memory engine — storage and recall, not a chatbot. This doc is about scaling that memory engine to millions of cells while keeping recall fast, on whatever hardware it runs on.
---
Today the live engine answers queries by full parallel scan — it compares your query against *every* cell. Measured on John (rshl_scale_sweep.mjs, DIM 16384):
**So the memory is nearly free; the *scan* is the wall.** Everything below is about not scanning what we don't need.
---
A resonance-addressed, hardware-tiered, self-tuning memory engine.
Cluster the cells into sections. Each section is represented by ONE summary vector — a symbol. A query doesn't scan all cells; it resonates against the symbols first (a few thousand comparisons), and the strongest-resonating symbols point to the sections that actually hold the answer. Then it only scans cells inside those sections. 5M → ~few-thousand symbols + ~few-thousand cells. This is an inverted-file / centroid index (IVF), driven by RSHL resonance instead of plain distance. *(Already half-built: KMeansIndex "sub-500 µs routing" and the HNSW graph in universe.rs.)*
As KAI parses the input, each concept it reads fires its symbol, the symbol resonates to that concept's section, and that section is pulled into the fast tier just in time to be searched. We don't load everything — we stream in exactly the sections the input is asking about, in the order it asks. By the time he's "reading" the word, its region is already hot.
gpu_kernel.rs, wgpu.)*memmap2 already a dependency.)*As KAI grows, don't store everything densely and redundantly — lean on the edges (associations between cells). Store the relationship, derive the rest. Graph edges (HNSW / associative links) compress meaning: more held in less space.
A small controller that, given the hardware in front of it + current cell count + the recall target, *picks and tunes the strategy*: full-scan when small, resonance-router + N probes when big, GPU-batch under heavy query load, SSD-tier when it outgrows RAM. Not magic — it selects and tunes over known strategies from measured tradeoffs. *(Seed already built: the bench auto-detects the machine and auto-picks the fastest kernel + auto-scales the workload.)*
KAI auto-decides everything knowable — detects specs, benchmarks himself, researches facts, chooses the index — and does not bounce those questions back to the user. He only checks in on things that are the user's to choose (preferences, goals). *Facts: he finds them himself. Wants: he confirms.*
---
Moving data between tiers costs time — RAM ≈ nanoseconds, PCIe-to-GPU ≈ microseconds, SSD ≈ ~100 µs (~1000× RAM). So the entire game is minimizing crossings: keep hot data where the compute is, batch GPU work so one transfer serves thousands of comparisons, touch the SSD rarely and in big sequential gulps. Every "adaptive/stable" decision is really a decision about *where to put the data so you cross the least.* Respect this and it flies.
probe_sweep.rs already exists to find the smallest N that keeps recall ≥ 95%.---
KMeansIndex (sparse_vec.rs) — the symbol/section router ("sub-500 µs"), on Universe.kmeans_index, built by rebuild_index().HNSW (hnsw_rs) — the branching graph, on Universe.hnsw, rebuilt in rebuild_index().cosine_avx512 (VPOPCNTDQ) — live SIMD in the similarity path (sparse_vec.rs), scalar fallback.gpu_kernel.rs + wgpu — GPU compute bones.memmap2 — dependency in place for disk-backing.probe_sweep.rs, hnsw_bench.rs — tuning harnesses already in-tree.src/core/engine.rs:433: // universe.rebuild_index(0.0); // Skipped ...) and the mask_pool/KMeans build is capped at ~200 K cells (universe.rs:879). So live queries fall back to full scan.---
1. Prove the O(N) wall (done): rshl_scale_sweep.mjs → measured ~1.25 µs/cell, ~201× gap.
2. Measure the fix on the real engine: cargo run --release --bin probe_sweep and --bin hnsw_bench at 10 K / 50 K / 100 K / 250 K cells → recall@K + latency vs full scan; find the crossover where the index wins.
3. Wire activation: replace the disabled engine.rs:433 with an auto-rebuild_index at a cell-count threshold (the measured crossover); keep instant-boot full scan for small lattices. Flag-gated KAI_LATTICE_INDEX, reversible.
4. Lift the 200 K mask_pool cap (universe.rs:879) once RAM headroom is confirmed.
5. Verify: recall@K within noise of full scan (correctness gate); query p95 stays under target as cells grow to 100 K+.
1. Resonance routing: drive the section router by RSHL resonance (phasor coherence), not plain distance (ties into fixing phase_angle() to carry real semantic phase — ALGORITHM-INVENTORY #1).
2. Prefetch-on-read: as input is parsed, fire symbols per concept and stream the pointed-to sections into RAM/VRAM just-in-time.
3. Cross-stream tiering: GPU resonance kernel (gpu_kernel.rs/wgpu) + RAM hot set + SSD memmap cold bulk; adaptive spill; minimize crossings.
4. Self-tuning controller: generalize the bench's auto-kernel/auto-scale seed into a strategy selector (full-scan | probes | HNSW | GPU-batch | disk-tier) keyed on hardware + cell count + recall target.
5. Connections-over-duplication: use graph edges to compress as the lattice grows.
Done when: a 1M+-cell lattice, mostly on SSD with a hot RAM/VRAM working set, answers queries in a few ms at recall within noise of full scan, and the engine picks its own strategy per machine without being told.
---
oracle.html / kaiverse.js. Confirm before changing core query/encode behavior.Cargo.toml version in sync when the Codex version bumps.kai-state.json) during benchmarking — throwaway in-memory only.