Status: DESIGN / READINESS ONLY — no training code written, no engine touched.
Date: 2026-07-04 · Author: engineering pass (read-only investigation)
Rev 2 (2026-07-04): added two owner refinements — §6 CLOUD COMPUTE (pretrain heavy in
the cloud, adapt light at home) and §7 CONTINUAL / LIFELONG LEARNING (the "open part" —
replay-based overnight consolidation of the LLM brain module, wired into KAI's *existing* sleep/
hippocampus loop). §8 revises the architecture, phases, risks and open decisions accordingly.
Scope of this doc: map the *real* current code, then specify exactly what it would take
to make KAI's RSHL lattice *trainable like an LLM* (next-token prediction with backprop),
so the owner can green-light a build. This is grounded in files that exist today — every
claim below cites a real path.
> One-line honest summary: KAI's RSHL core is a *fixed hash-based feature + associative
> memory*, not a gradient-trainable network. You cannot "backprop the lattice" as-is. But you
> already own every other piece needed to train a small LLM (a BPE tokenizer, a Candle
> transformer implementation, 100M+ tokens of corpus, and CUDA-capable Candle). The realistic
> path is: **add a small, genuinely backprop-trained language model, and keep RSHL as the
> feature/memory/retrieval substrate around it** — not to make the 16384-dim ternary lattice
> itself differentiable.
---
*fixed, non-invertible hash function* (SparseVec::encode, src/core/sparse_vec.rs:308).
It has zero learnable parameters and you can't decode tokens back out of it, so it can't
be the *output* layer of a language model.
next-token *head* on top of frozen RSHL features (proves the training loop cheaply, honors
"keep the RSHL core"). Then grow into (c) — a small trainable transformer LM with RSHL
used as a kNN memory (retrieval-augmented). Do not pursue (b) (making the lattice ops
differentiable) as the primary path — high research risk, low payoff on this hardware.
Candle (src/cognition/bitnet_llama.rs) and a generation loop with logits + sampling**
(src/cognition/language_warehouse.rs:474-598) — but it is **inference-only, loaded from a
frozen GGUF. There is no training loop, no loss, no optimizer, no autodiff anywhere** in
the current "training" path. Today's "training" (overnight_pipeline.py,
distill_from_bitnet.py) is *memory ingestion* (binding prompt→answer pairs into the
lattice), not gradient descent.
transformer from scratch (~5–30M params, char- or small-BPE, context 256–512) on his corpus
in hours-to-days. It cannot train a 2B model from scratch, and shouldn't try. Scope =
*small LM, RSHL-augmented, trained on his own data.*
---
The RSHL core is a Vector Symbolic Architecture / hyperdimensional computing substrate:
SparseVec — a 16384-dim ternary vector, values in {−1, 0, +1}, stored sparse (nz: Vec<u16> indices + vals: Vec<i8>), default ~4% density (~655 active dims;
SPARSITY = 0.04, DIM = 16384, src/core/sparse_vec.rs:18-22).
SparseVec::encode (sparse_vec.rs:308) is a fixed hash — it sumsweighted hash contributions from char trigrams, normalized words (weight ×3, proper nouns ×6),
word-pairs (some order info), char bigrams, and char 4-grams, then keeps the top-|magnitude|
~655 dims as ±1. No learned weights. Deterministic. Many-to-one (not invertible).
bind = element-wise ternary multiply, self-inverse (sparse_vec.rs:757, 784); bundle/superpose_sparse = add + re-sparsify (:576, :729); cosine = retrieval metric
(:500); cleanup via nearest-stored-vector search across the Universe lattice.
nearest-neighbor by cosine. The one genuinely adaptive component is ResponseMlp
(src/cognition/corpus_trainer.rs:250-370): a single-hidden-layer sparse net updated by
Hebbian reward rules (train_step, hebbian_update), not backprop.
A next-token LLM needs, at minimum: (1) a differentiable map from context → a distribution over
a fixed vocabulary, and (2) gradients that flow back to update parameters to minimize
cross-entropy. RSHL fails both natively:
encode is a hash; there's nothing to differentiate.hash is lossy and collides by design. So RSHL can be an *input/context* representation or a
*memory*, never the softmax *output* head.
bind/bundle/threshold-sparsify are hard/argmax operations;no gradient path.
Conclusion: "make RSHL trainable like an LLM" = *attach a real trainable next-token model to
the system and let RSHL feed / augment it.* The RSHL core stays intact (per project hard rules);
we add a parallel, trainable, Python/Candle-side model.
(a) Trainable next-token HEAD on frozen RSHL features.
RSHL = frozen feature extractor + memory. A small trainable head does softmax next-token
prediction: token_{t+1} = Head( RSHL_features(context ≤ t) ).
LM head"; overfit-a-batch sanity check is trivial.
ceiling. Good as a proof of the training loop and as a memory/bias signal, not as a
fluent generator on its own.
(b) Differentiable relaxation of the lattice ops.
Replace ternary/threshold ops with soft (real-valued, temperature-softmax) analogues so gradients
flow through a "learned encode."
neural net, losing the sparsity/energy properties that make RSHL interesting, and it's the
worst fit for a 6 GB laptop. Not recommended as the primary path.
(c) Hybrid: small trainable transformer LM + RSHL retrieval/memory (RECOMMENDED).
Train a *small real language model* (its own learned token embeddings + a few transformer/MLP
blocks) on his corpus. Use RSHL as an external kNN memory (kNN-LM / RAG style): at each step,
retrieve the nearest stored continuations from the lattice and interpolate their next-token
evidence with the transformer's softmax.
reuses what you already have (the Candle transformer scaffold + language_warehouse
retrieval); RSHL stays central as memory + feature + retrieval, satisfying "keep the RSHL core,
not just softmax."
Recommendation: build (a) first as Phase 0 (loop proof), then evolve into (c). Skip (b).
his corpus (jsonl) ──► clean/dedup ──► tokenizer (Llama-3 BPE, EXISTS: models/tokenizer.json)
│ │
│ token id stream ──► shards (.bin of u16/u32 ids)
▼ │
RSHL encode (FIXED) ▼
context → SparseVec ───────────────► Trainable model
(feature/memory) ┌──────────────────────────────────────────┐
│ Phase 0 (a): Head(RSHL_feats) → softmax │
│ Phase 1+ (c): small Transformer LM │
│ + kNN-LM interpolation w/ RSHL │
└──────────────────────────────────────────┘
│
next-token logits ──► cross-entropy loss vs true next id
│
AdamW backward/step (PyTorch or Candle autodiff)
│
checkpoints (.safetensors) ──► eval (perplexity /
│ next-token top-1/top-5)
▼
plug back into KAI: LM logits + RSHL kNN evidence bias the existing
generate() sampler in language_warehouse.rs (the plumbing already exists)
---
Each item is mapped to real files.
data/harvest.jsonl — 1,388,700 lines / 301 MB, schema `{"text", "region", "source","strength"}` (general knowledge statements, wikipedia-swarm etc.). ~60–80M BPE tokens.
data/training_corpus/bulk_conversations.jsonl — 103 MB, schema {"text"} (reasoning/conversation prose). ~20–25M tokens.
data/training_corpus/corpus_discord_YYYYMMDD.jsonl — his own KAI/Discord data, schema {"input", "reply", "state", "timestamp", ...} — real prompt→reply pairs (ideal for
instruction/response tuning). Tens of MB across days. ~5–10M tokens.
data/bulk.jsonl (7.5 MB, 7,406 lines), data/kai-transcript.jsonl (579 lines), data/kai-texts.bin (768 KB), data/dictionary.json (24 MB), data/epistemic-rejections.jsonl.
Plenty for a small LM; his-own-data-only is enough for a persona/style model, borderline for
from-scratch general fluency (mix in harvest/bulk for that).
data_prep script: source jsonl → strip noise (reuse the noise filter idea in corpus_trainer.rs:147 is_noisy) → dedup → normalize → write a single clean text/id stream →
shard to .bin. Nothing like this exists on the *gradient-training* side yet.
data/transcripts.db is 0 bytes (empty) — despite the name, the real conversational corpus lives in data/training_corpus/*.jsonl, not a SQLite DB. Don't build the
pipeline around transcripts.db.
models/tokenizer.json (1.9 MB, Llama-3 BPE, vocab 128,256) and models/BitNet/tokenizer.json. The Candle transformer config already assumes this vocab
(bitnet_llama.rs:41 vocab_size: 128256). Loadable in Python via HF tokenizers, in Rust via
the tokenizers crate.
6 GB). Recommend either char-level (vocab ~100) or a small custom BPE (4k–8k) trained on
his corpus for the small-LM phases; keep the 128k Llama tokenizer only if/when you want the LM
to interoperate with BitNet tokens. So: tokenizer *exists*; a *small* vocab is a 1-line
tokenizers training step to build.
src/cognition/bitnet_llama.rs — a complete LLaMA-family forward pass in Candle (RoPE, GQA repeat_kv, causal mask, RMSNorm, KV-cache; config hidden 2560 / 30 layers /
vocab 128256). This is a *reference implementation* to shrink, not a thing to train at
2B scale.
src/cognition/language_warehouse.rs — sparse ternary word embeddings extracted from BitNetCargo.toml:64-66, 90 — candle-core/nn/transformers 0.10.2 already deps, with an llm-cuda feature (candle-*/cuda). Candle supports autodiff (Var, .backward(),
candle_nn::AdamW), so a training loop can live in Rust.
RSHLHead — e.g. Linear(16384 → hidden) → GELU → Linear(hidden → vocab), or a2–4 layer MLP. ~1–20M params depending on vocab. Input = mean/któ-pooled RSHL features of
the context window.
TinySLM — learned token embedding (dim 256–512) + 2–6 transformer blocks +tied output head. ~5–30M params. Plus a kNN-LM interpolation module that queries the RSHL
Universe for nearest continuations.
tokenizers, datasets, mature AMP/checkpointing, easy overfit-a-batch). **Port the winning small model to
Candle** for in-engine integration later (Candle is already wired and CUDA-capable, so the
trained .safetensors loads natively next to bitnet_llama.rs).
overnight_pipeline.py / distill_from_bitnet.pydo memory ingestion / distillation-by-recall (bind prompt→answer into the lattice); grep
confirms no torch, no autograd, no loss.backward, no optimizer in the Python
pipeline. ResponseMlp::train_step is Hebbian, not SGD.
(context_ids, target_id) → forward → **cross- entropy** → AdamW → grad-clip → cosine LR w/ warmup → checkpoint .safetensors every N steps →
resume. Add gradient accumulation + fp16/bf16 AMP to fit 6 GB. ~150 lines PyTorch.
benchmarks/rshl_vs_llm.mjs, rshl_fullbench.mjs, rshl_scale_sweep.mjs measure energy / latency / cost per token, and explicitly state
*"RSHL RETRIEVES, LLM GENERATES … not capability equivalence"* (rshl_vs_llm.mjs:27-29).
They do not measure language quality (perplexity / next-token accuracy).
lm_eval harness computing perplexity and next-token top-1/top-5 accuracy on a held-out split. Wire it to sit *beside* rshl_vs_llm so you can report *quality* next to
the existing *cost* numbers (quality-per-joule becomes a real comparison for the first time).
laptop CPU (already the RSHL substrate).
accum) trains at ~thousands of tokens/sec; one epoch over ~10–50M tokens in hours, a decent
small model in 1–3 days of wall time. LoRA-fine-tuning a small (<1B) model also fits.
full fine-tune of any multi-B model; 128k-vocab softmax on long context at batch. Don't attempt.
a separate offline process (same discipline as distill_from_bitnet.py) — never in the
live serving engine, never sets KAI_NATIVE_BRAIN.
language_warehouse.rs:474-598 runs forward → logits → apply bias → sample(temp, top-p, repeat-penalty) → next_token, and it
*already* biases logits from lattice evidence (:584 logits_vec[token_id] += bias). A trained
head/LM drops into exactly this point.
.safetensors; either (a) use its logits as the bias source, or (c) replace/interpolate the forward logits with the small LM's + RSHL kNN evidence. Because
the sampler plumbing exists, plug-back is small once a checkpoint exists.
Checklist headline: *Tokenizer ✔, corpus ✔ (~100M tokens), transformer scaffold ✔ (Candle,
inference-only), CUDA ✔, integration seam ✔. Must build: the gradient-training half — data-prep/
shard script, a small trainable model def, a next-token training loop with autodiff, and a
perplexity/accuracy eval harness. None of these exist today.*
---
Goal: a real backprop next-token model runs end-to-end on his data, overfits a single batch.
corpus_discord_*.jsonl; RSHLHead (Linear(16384→512)→GELU→Linear(512→vocab)); PyTorch train loop; cross-entropy.
the loop is wrong — fix before scaling. Then confirm held-out perplexity beats a unigram
baseline.
Goal: a small transformer that generates his-style text with falling perplexity.
corpus_discord + a mixed slice of harvest/bulk; 8k BPE; TinySLM (emb 384, 4 blocks, ctx 512, ~15–25M params); AMP + grad-accum + checkpoint/
resume; lm_eval (perplexity + top-1/5).
the existing rshl_vs_llm harness extended with quality.
Goal: kNN-LM — interpolate the small LM's softmax with next-token evidence retrieved from the
RSHL Universe.
and blends their token evidence with LM logits (p = λ·p_LM + (1−λ)·p_RSHL); tune λ.
Goal: KAI's live language actually improves.
language_warehouse.rs sampler seam; optionally port TinySLM to Candle so it loads natively beside bitnet_llama.rs; keep it flag-gated + offline-trained.
---
1. Architecture: confirm (a)→(c) staged (recommended) vs insisting on (b) differentiable
lattice (not recommended — research risk, poor 6 GB fit).
2. Tokenizer granularity: char-level (simplest POC) vs small custom BPE 4k–8k
(recommended for Phase 1) vs reuse Llama-3 128k (only if BitNet-token interop matters).
3. Data policy: his-own-data-only (persona/style, ~5–10M tokens — coherent "Kai voice",
weaker general fluency) vs mix in harvest/bulk (~100M tokens — more fluent, less purely
"him"). Recommended: his-data-heavy + a harvest/bulk minority for grammar.
4. Framework/where it lives: PyTorch offline for R&D then Candle port (recommended),
vs train directly in Candle/Rust from the start (slower iteration, but no port and Candle is
already wired with llm-cuda).
5. RSHL's role: frozen feature only (a) vs feature + kNN memory (c, recommended) — i.e.
how central RSHL stays in the trained system.
---
small LM (≤~30M params), RSHL-augmented.
it ingests memory.
kaiverse.js, oracle.html, or the RSHL core ops. Training is an offline, additive track, consistent with distill_from_bitnet.py's
safety model.
---
The 4050-only framing above (§2.6) was deliberately conservative. The owner has cloud —
a VPS / Google Cloud / rented cloud GPU or TPU — so the *hardware ceiling moves*. This does
not change the honest architecture (RSHL is still not backprop-trainable; §1 stands); it
changes where the one big compute cost is paid.
There are two very different compute jobs, and they want two different machines:
scratch is the expensive part: many passes over billions of tokens. This is embarrassingly
well-suited to a rented A100/L4/TPU for a day or two, then you never pay it again.
keeping KAI current with *his* lived experience is cheap: small LoRA/fine-tune deltas over
thousands (not billions) of tokens. This is exactly what a 6 GB laptop is fine at, and it's
what the continual-learning loop in §7 needs.
> Rule of thumb: pretrain is a capex you rent once; consolidation is an opex you run at home.
> Bring the cloud-trained .safetensors home, and from then on the 4050 only ever does small
> bounded updates + inference. Phase 0 (the proof-of-concept, §3) still runs entirely local —
> you don't rent anything until the loop is proven.
~$0.6/hr spot) or an L4/L40S (cheaper, slower), you can pretrain a ~30M–125M-param**
transformer from scratch on a Chinchilla-ish token budget (roughly 20 tokens/param → ~0.6B–2.5B
tokens) in hours, not weeks. For calibration, Karpathy's llm.c reproduces GPT-2 124M
(10B tokens) on an 8×A100 node in a few hours for tens of dollars; a single-GPU, smaller-
token run of a 30–125M model is the same order of magnitude.
~$15–60 raw compute; call it ~$50–200 all-in with experimentation.
cost-effectively if you use JAX/Flax; more toolchain friction than PyTorch-on-A100.
no payoff for KAI's purpose). The BitNet 2B stays a *frozen teacher*, never a from-scratch target.
(on-demand for a clean run, spot/interruptible if you checkpoint often). A100 80 GB removes the
6 GB VRAM straitjacket entirely for a ≤125M model — big batches, longer context, fast epochs.
Google Cloud is the choice if you specifically want TPU or want the VPS + storage in one place.
tokenizers/datasets). Output .safetensors. Keep the model definition deliberately
Candle-portable (plain LLaMA-ish blocks) so it later loads next to bitnet_llama.rs in-engine.
data/harvest.jsonl + bulk_conversations.jsonl for grammar/world-structure, corpus_discord_*.jsonl for "Kai voice"). Upload the sharded .bin
once; train; download weights.
.safetensors base checkpoint (+ tokenizer) copied to C:\KAI\models\.From here on, everything is local (§7).
---
This is the core of the owner's idea: KAI's small LLM brain module should *keep learning over
time* — "an open part for real-time learning… consolidation over time like how humans and living
creatures do." That is precisely Complementary Learning Systems (CLS) theory, and — the finding
that matters — KAI already implements the CLS memory half in real Rust today. The LLM piece
plugs into machinery that already exists; we are not inventing the biology, we are adding a cortex.
The telemetry line *"Hippocampus CA3/CA1: Short-Term Patterns | Queued for Sleep Replay | Graduated
to Universe"* is not decoration — it is a live loop. Verified this pass:
src/cognition/hippocampus.rs. A CA3 autoassociative pattern bank (CA3_CAPACITY = 256, hippocampus.rs:44), a CA1 recent-pattern buffer
(CA1_BUFFER = 12, :47), DG-style pattern separation (separate(), :288) and CA3
pattern completion (complete(), :229). Each stored trace carries strength,
survival_count, emotional_charge, region, source (HippocampalPattern, :65-91).
This is one-shot learning: a single Ryan turn is bound immediately —
self.engine.hippocampus.store(&input, …) at main.rs:4741, alongside episodic.store.
hippocampus.rs:367 consolidate_into_universe(). Three biological gates: Gate 1 strength threshold (0.55 neutral / 0.45 emotional, :395);
Gate 2 novelty (skip if Universe already knows it, boosted cosine > 0.65 → *reinforce* instead
of re-store, :419); Gate 3 survival ≥ 2 cycles unless emotional fast-track ≥ 0.60 (:405).
Promoted traces are removed from hippocampus and written into the Universe (universe.store,
:443) — literally *"Graduated to Universe."*
main.rs:890-929. Every 50 ticks (~4 min): hippocampus.decay() then consolidate_into_universe(&mut universe, coherence). A spiral-coherence gate (tau_r < 0.35
suppresses consolidation, hippocampus.rs:372) mirrors stress-impaired memory transfer. Real
stats are persisted to data/hippocampus_status.json (main.rs:914) — that's the dashboard readout.
src/cognition/sleep.rs. Clock-gated to ~3:05 AM (should_sleep(), :147), it runs NREM (scan episodic for high salience×vividness) → SWS
(consolidate top events, global synaptic downscale ×0.92, prune weak cells) → REM
(recombine top events into novel associations) → wake report (run_cycle(), :177-260).
overnight_pipeline.py. Confirmed **memory ingestion, NOTgradient training:** it scrapes/synthesizes study material, uses a teacher LLM (Groq/OpenRouter/
Gemini or the offline BitNet teacher) to quiz/grade KAI, distills BitNet→lattice via
bulk-ingest, and refreshes the ANN index. It even **stops cleanly at ~4:30 AM
(PIPELINE_STOP_HOUR, :106) specifically so the engine-side sleep/consolidation can run
without new material flooding in.** No torch, no loss.backward, no optimizer anywhere.
Bottom line of the audit: KAI already has the *hippocampus (fast) + Universe (slow) + sleep-
replay (transfer)* CLS triad — but the "slow store" is the RSHL Universe lattice, which (per §1)
is associative, not a trainable cortex. **The continual-learning upgrade = add the LLM brain module
as a second slow store, and let the SAME replay machinery consolidate into it.**
Per-token online SGD on the live model — "train on every message as it arrives" — is the obvious
idea and it is wrong: it causes catastrophic forgetting (each new gradient overwrites prior
knowledge), it is unstable (one weird input can wreck the model), and it can't be rolled back. Real
brains don't do this either — they *encode fast* (hippocampus, now) and *consolidate slow* (cortex,
during sleep). We copy that.
Add one new offline stage to the nightly window that already exists (after ingest+weave,
before/around the engine's 3 AM sleep). It never touches the live serving path; it produces new
weights that are hot-swapped in atomically.
1. What gets replayed. The *day's lived experience*, sourced from the CLS machinery that's
already recording it: the traces that graduated to Universe that day (the high-value,
survival-tested memories from consolidate_into_universe), the high-salience episodic events
the sleep NREM scan already selects (sleep.rs top events), and the Ryan prompt→reply pairs
from corpus_discord_*.jsonl. These are exactly the memories biology replays.
2. How a training batch is formed. Turn each replayed item into next-token training text:
prompt→reply pairs become instruction/response examples; graduated Universe facts become short
completion targets. Crucially, mix in a reservoir sample of OLD consolidated data (a fixed
random slice of prior corpus/weights-era data) so each batch is *new experience + old memory* —
this is the single most important anti-forgetting lever. Target a small, bounded batch
(e.g. a few hundred–few thousand examples/night), not "everything ever."
3. LoRA, not full fine-tune (recommended). Run a small LoRA update on the brain module
(low-rank adapters on the attention/MLP projections), not a full-weight fine-tune. Why LoRA
for the nightly cycle: (a) it fits the 4050 trivially (only adapter params train); (b) it is
inherently forgetting-resistant — the frozen base can't drift; (c) adapters are **tiny,
versioned, and hot-swappable/rollback-able** — if a night's consolidation degrades eval, discard
that adapter. Reserve a full fine-tune for rare, larger "deep consolidation" passes if ever needed.
4. How forgetting is prevented (layered): (i) replay old + new in every batch [primary];
(ii) small learning rate + few steps — nudge, don't overwrite; (iii) LoRA caps how far
weights can move; (iv) optional EWC / regularization anchoring important weights toward their
pretrained values; (v) eval-gated commit (§7.4) — a consolidation that regresses held-out
perplexity is rejected, not shipped.
5. How the updated weights hot-swap into the live loop. The generation seam already exists —
language_warehouse.rs:474-598 runs forward → logits → bias → sample → next_token, and already
biases logits from lattice evidence (:584). The nightly stage writes a new adapter/checkpoint;
at the next safe idle boundary the engine atomically loads the new weights (swap the adapter
the brain module loads at that seam), flag-gated and reversible. No live-path SGD, ever.
Fast and slow, exactly like awake-encoding + sleep-consolidation in humans:
KAI can recall it within the same conversation (that's hippocampus.store at main.rs:4741
durable ones into the LLM brain module's weights, so over weeks KAI's *generation* — not just its
retrieval — reflects lived experience. The "open part" is this nightly LoRA-on-replay stage: an
always-available channel for real-time-encoded experience to become consolidated skill over time.
---
to start": 125M/GPT-2-small class is the smallest size that generates genuinely coherent text,
cheap to cloud-pretrain (§6, <~$200), and small enough to LoRA-adapt + run on the 4050). A ~30M
variant remains the ultra-cheap fallback for Phase 0.
episodic + associative memory and the kNN-retrieval layer around the brain module (§1, Phase 2
of the original plan). RSHL is *how experience is captured and selected for replay*; the brain
module is *what consolidates it into fluent generation.*
sleep/hippocampus scheduler — the bridge that turns fast memories into slow weights.
RSHLHead or a tiny LM, PyTorch loop on the 4050, overfit one batch to ~0 loss as the
green-light gate. No cloud, no consolidation yet. *Verify:* loss→0 on a batch; beats unigram.
the corpus, rent 1× A100 (§6.3), pretrain a 50–125M model in PyTorch, download .safetensors.
*Verify:* held-out perplexity curve down; coherent samples; base loads locally on the 4050.
*Effort:* low-medium. *Cost:* ~$50–200. Pretrain is paid once.
the nightly replay→batch→LoRA stage of §7.3, sourcing replay from graduated-Universe + sleep
NREM top events + Discord pairs, mixing in a reservoir of old data; small LR; write a versioned
adapter. Slot it into the existing ~3–4:30 AM window. *Verify:* after N nights, held-out perplexity
is flat-or-down (no forgetting) and new-experience recall improves; a bad night's adapter is
rejected by the eval gate. *Effort:* medium. Runs on the 4050.
the language_warehouse.rs seam (flag-gated, reversible); a small eval-over-time dashboard
(perplexity + next-token top-1/5 tracked per consolidation cycle). *Verify:* owner A/Bs KAI with
the consolidated brain on/off; the time-series shows learning accrue without drift. *Effort:* medium.
batch; LoRA (frozen base); small LR/few steps; optional EWC; eval-gated commit.
drift). *Mitigate:* bounded batch size + step count; per-night versioned adapters with rollback;
periodic "reset to pretrained base + best adapter"; cap how many adapters stack before a
consolidate-the-adapters pass.
*Mitigate:* a frozen held-out set for comparability + a rotating fresh set for realism;
track top-1/5 accuracy alongside perplexity; keep qualitative sample logs; the owner remains the
human verifier (project rule).
distill_from_bitnet.py; never set KAI_NATIVE_BRAIN from the trainer; hot-swap only at idle,
flag-gated, reversible.
1. Brain-module size to start: ~30M (ultra-cheap, weaker fluency) vs ~50–125M
(recommended — coherent, still cloud-cheap and 4050-adaptable) vs ~350M (stretch, low-hundreds $).
2. Consolidation update type: LoRA nightly (recommended — cheap, safe, rollback-able) vs
periodic full fine-tune (heavier, more capacity, more forgetting risk) vs a hybrid
(nightly LoRA + rare full "deep consolidation").
3. Cloud provider/instance for Phase-1 pretrain: RunPod/Lambda/Vast.ai 1× A100 80 GB
(recommended, PyTorch) vs Google Cloud TPU (JAX, cheaper-at-scale, more friction) vs
GCP A100 VM (one place for VPS + storage + GPU).
4. How often consolidation runs: nightly (recommended — matches biology + the existing
3 AM sleep window) vs on-idle (whenever the resource governor says there's headroom) vs
manual (owner triggers a consolidation pass on demand).
5. Replay source weighting: how much to weight *Ryan's own pairs* (persona) vs *graduated-
Universe facts* (knowledge) vs *reservoir of old data* (stability) in each night's batch.
---
src/core/sparse_vec.rs (DIM/SPARSITY :18-22, encode :308, cosine :500, bundle :576, bind :757, unbind :784).
src/cognition/corpus_trainer.rs:250-370.src/cognition/bitnet_llama.rs (config :36-45).src/cognition/language_warehouse.rs:474-598 (sampler:909-942).
models/tokenizer.json (Llama-3 BPE, vocab 128256), models/BitNet/tokenizer.json.data/harvest.jsonl (1.39M lines/301MB), data/training_corpus/bulk_conversations.jsonl (103MB), data/training_corpus/corpus_discord_*.jsonl (his pairs), data/bulk.jsonl,
data/kai-texts.bin; data/transcripts.db = empty (0 bytes).
benchmarks/rshl_vs_llm.mjs (framing :27-29), rshl_fullbench.mjs, rshl_scale_sweep.mjs.
Cargo.toml:64-66 (candle 0.10.2), :90 (llm-cuda); GPU = RTX 4050 Laptop 6 GB.overnight_pipeline.py (teacher/quiz/distill + PIPELINE_STOP_HOUR :106 stops for engine-side consolidation), distill_from_bitnet.py.
src/cognition/hippocampus.rs (CA3_CAPACITY :44, CA1_BUFFER :47, complete :229, separate :288, consolidate_into_universe
:367 with gates :395/:405/:419 and universe.store graduation :443); one-shot bind on input at
main.rs:4741; driver loop main.rs:890-929 (every 50 ticks, coherence-gated, telemetry →
data/hippocampus_status.json :914); sleep/replay scheduler src/cognition/sleep.rs
(should_sleep :147 ~3:05 AM, run_cycle NREM→SWS→REM :177-260, downscale ×0.92 :64).