← all documents · raw markdown · 13 KB

KAI Capability Audit and Next Work

Date: 2026-09-09. Scope: the live Oracle OS route, memory retention, retrieval,

learning feedback, conversation continuity, and language composition.

Operating Constraint

Keep KAI running. No shutdown, restart, forced termination, deployment over the

running executable, or live pruning is authorized in this session. A later stop

must begin with a successful clean flush. All Rust repairs below are SOURCE

changes, pending a later owner-authorized deployment. Tests use a separate build

directory and small temporary universes, not the live brain files.

Current Evidence

The earlier Claude notes and audits were read, then checked against current code

and live endpoints. Historical diagnoses are not assumed to remain current.

Initial process: kai.exe PID 13784, started 2026-09-09 09:07:04 local time.
Live telemetry at 15:32 UTC: 51,823 cells and 1,902,673 synapses. Growth alone

does not measure better retention, reasoning, or answer quality.

[internal endpoint] reports cognition_live=false; predictor, dreams, amygdala, and

theory_of_mind=false on this headless path. These are runtime status flags,

not proof that no cognition of any kind occurs.

Native model reports mounted, but native_decode_count=0, vocab_entries=0,

BitNet/dense experts not mounted. A mounted flag alone does not prove a working

language generator; these counters are specific to their instrumented paths.

word_calculus::Operator::carry() still returns constants. Some lattice and

synapse weights change at runtime; the calculus carries are not learned yet.

The calculus-to-lattice bridge is now called in Oracle. The older claim that

it has no callers is no longer correct.

Raw retrieval evidence: reports/kai-path-audit-2026-09-09T15-32-00-347Z.json.

Live Language Baseline

Six distinct Oracle probes were sent as CodexProbe, without teaching answers or

applying rewards. Manual review: **0/6 correct, five gap responses and one wrong

answer**. This is a small selected diagnostic set, not an intelligence score or

an LLM comparison. The vowel question was repeated during response-parser

diagnosis; repeats are not counted as additional cases.

| Probe | Actual behavior | Expected capability |

|---|---|---|

| Which letters are vowels? | "The letter s is a consonant." | List a, e, i, o, u. Earlier source fix is still absent from the live binary. |

| Explain the difference between a river and a lake. | "That's at an edge I haven't fully mapped." | Retrieve both concepts and contrast them. |

| Mira handed a key to Theo. Who received the key? | No solid answer in memory. | Extract Theo as recipient from the supplied sentence. |

| Same sentence; who gave the key? | No solid answer in memory. | Extract Mira as agent; distinguish argument roles. |

| All zargs are blue. Neri is a zarg. What color is Neri? | No solid answer in memory. | Apply a supplied rule to a new instance. |

| What is 17 multiplied by 23? | No solid answer in memory. | Route arithmetic to an exact calculation and return 391. |

These outputs did not contain word salad, but refusing to answer these cases

does not establish better understanding. Grammar and factual correctness must

be scored separately.

Speech probes can modify the session and trigger input-calculus learning. Later

retrieval returned the probe questions themselves as word-calculus memories,

including repeated copies of the arithmetic question. Correct arithmetic text

also existed among lower-ranked hits. This demonstrates that storing and

finding related words does not ensure that the right evidence reaches speech.

Repairs Prepared

Retention

Seven new failure tests were run before implementation; all seven failed.

1. Successive deltas renamed old changes into backups that normal loading never

replayed. A pending delta now forces a full checkpoint before another delta.

This deliberately trades some extra full-save I/O for a loadable history.

2. A first save with few dirty cells could write only a delta, without a base.

Delta saves now require an existing base snapshot.

3. HashSet iteration could write appended cells out of order; the loader counted

out-of-order appends as applied even when it discarded them. Writes are

ordered; legacy deltas are parsed and validated in full, then ordered before

application. Missing/truncated entries fail instead of partially applying.

4. Failed saves could clear dirty flags. Flags now clear only after reported

success, so failed cell/metadata/synapse writes remain pending for retry.

5. Delta writes and synapse/metadata writes could fail while returning success.

Errors now propagate into SaveResult. Files are synced before rename.

6. Streamed saves ignored rename failure and removed the pending delta anyway.

A failed cell replacement now returns failure and preserves the delta.

7. Confidence rewards, confidence reductions, and cell retirement did not mark

cells dirty. These mutations now participate in incremental persistence.

8. The boot guard only recognized uncompressed metadata, so compressed/partial

snapshots could appear to be an empty first boot after a load failure. It now

recognizes both repository-root and direct-data layouts and partial state.

9. Failed compact loads could fall back to stale legacy JSON. Legacy fallback

now occurs only when compact state is absent. Both boot defects were

reproduced in additional failing tests before the source fixes.

_(implementation details omitted)_

successive saves, legacy append order, failure injection, boot guard detection,

incomplete delta rejection, and save/reload of reward/correction/retirement.

This does not prove past live learning was lost, nor recover historical deltas.

Retrieval and Continuity

QueryHit conversion now preserves stored date, author, channel, message, and

keywords. Several query functions previously replaced these with zero/empty

values. All those constructors use the shared conversion now.

Oracle's initial full scan applies the artifact demotion already used by

ordinary indexed retrieval, before truncation can evict useful evidence.

Full/indexed/associative entry paths exclude retired cells and known raw echo

sources, and check cellular ownership. Oracle passes its request user_id to

the initial full scan, multi-hop fallback, and training query. This is not a

complete audit of session isolation or downstream decoder ownership.

Oracle/public turn kinds are normalized to the user/kai roles the voice code

consumes, preserving other speakers as other speakers. Previously existing

repetition and follow-up checks could not recognize Oracle's own KAI turns.

_(implementation details omitted)_

[internal module]. Three focused retrieval tests pass. The wider core

run also exposed an old test that simulated a clock jump as ordinary idleness;

its fixture now tests continuous sweeps, with a separate clock-jump guard test.

The production LTD clock behavior was not changed.

Work Order and Acceptance Checks

|---|---|---|---|

| 1 | Durable checkpoints | Repairs above are queued; cells, synapses, metadata, and text cache are still separate files, without one snapshot-generation transaction. Concurrent/stale snapshot ordering and parent-text cache consistency remain unverified. | Isolated crash/failure injection at every replacement boundary; recover one consistent generation or explicitly refuse it. Then save/reload a copied real snapshot and compare cells, provenance, vectors, and weights. |

| 2 | Outcome credit | KEEP_WORKS is opt-in. Its global latest-label trace can target the wrong conversation, is reusable, and does not get recorded by the new rshl_lattice_reply path. Malformed outcome bodies default to positive feedback. | Oracle returns a turn ID plus actual contributing cell IDs. Exactly one validated outcome updates that turn, never another user/turn. Test missing, duplicate, delayed, and malformed feedback. Do not merely enable the flag. |

| 3 | Evidence selection | Confidence strongly multiplies surface relevance; full scan, indexed retrieval, reranker, and associative expansion use different score meanings. Reranker permutes scores across evidence to keep gate thresholds stable. | Separate relevance, source reliability, and answer confidence in telemetry. Evaluate paraphrases, distractors, outdated facts, negation, and unknown answers on unseen cases before changing thresholds. |

| 4 | Questions versus facts | Input-calculus commits can retrieve questions and duplicate fragments as answer evidence. Current speaker/system detection also treats any name containing 'oracle' as system, including the default Ryan@OracleOS name. | Store utterance intent and provenance; preserve useful declarative premises but prevent an unanswered question from becoming its own answer. Test default owner messages and repeated probes. |

| 5 | Working memory | Six recent global session turns and role-normalization are not enough for reliable references, instructions, or separate conversations. Oracle responses currently serialize a roughly 14 MB session on every turn. | Separate per-conversation context with entity/referent state. Pass pronoun, changed-instruction, topic-return, and multi-user tests; return bounded turn responses without breaking consumers. |

| 6 | Meaning and reasoning | Recipient/agent and new-rule tests fail. Existing claimstore/agent modules contain useful parsing, contradiction, and tool logic, but module existence is not end-to-end Oracle behavior. | Wire existing components through a typed proposition/goal path. Distinguish subject/relation/object, negation, tense, and scope. Pass fresh-name role reversals and two-step supplied-rule tests; use exact tools for arithmetic. |

| 7 | Learnable word calculus | Carries are fixed. Punctuation segmentation is present, but it does not establish contextual word roles or complete glyph semantics. | Persist bounded learned adjustments trained from verified examples/outcomes. Test minimal pairs across punctuation, apostrophes, numbers, negation, and roles; show improvement on withheld combinations with existing skills retained. |

| 8 | Grounded composition | Existing knowledge may fail to reach speech; narrow hardcoded answer helpers can pass memorized drills without generalization. | Plan assertions from selected evidence, generate coherent text, then verify each assertion. Score correctness, relevance, grammar, and unsupported claims separately on fresh prompts. |

| 9 | Active cognition | Current headless status reports major modules inactive. Turning on every module could also activate untested decay/storage behavior. | Share tested orchestration with Oracle incrementally. Each module reports actual invocation/output and demonstrates benefit in an ablation test before becoming part of the live learning loop. |

| 10 | Capability measurement | No controlled LLM comparison was run. The existing continual-memory benchmark writes into the live lattice. | Run controlled teaching/interference/update/reload tests on an isolated brain with held-out questions. Compare the same tasks against a specified LLM configuration, with resource and teacher assistance recorded. |

Repeating This Audit

Verified locally after the relevant repairs: 199 selected tests passed.

| Suite | Passed |

|---|---:|

| Persistence, including fault injection and boot guards | 15 |

| Shared core, including retrieval and synaptic retention | 128 |

| Oracle routing and conversation roles | 5 |

| Reranking | 12 |

| Native voice | 39 |

git diff --check passed for the four changed Rust modules. The audit script

passed node --check; its default plan made no requests, and its read-only

live run completed all six retrieval cases and wrote the evidence report above.

Tests do not demonstrate live post-deployment improvement or LLM-level behavior.

Release verification also passed: `cargo build --release --bin kai --target-dir

[local path]` produced v9.10.1003 in the alternate build directory.

That executable was not launched or copied over the live one. Final health check:

PID 13784 still running and [internal endpoint] reports alive. Applying the build remains

pending a later owner-authorized clean flush and restart.

node benchmarks/kai_path_audit.mjs prints the plan without requests.

node benchmarks/kai_path_audit.mjs --run --save reads live status/retrieval and

saves a compact report. It does not call ingestion, reward, save, pruning,

shutdown, or restart endpoints. --replies additionally sends Oracle turns;

those can be learned by the running engine. Do not treat previously probed

questions as withheld evaluation data.

Offline verification uses `cargo test --lib <filter> --target-dir [local path]

--jobs 2 -- --test-threads=1. Filters include persistence::tests, core::`,

bridge::oracle_server::, cognition::rerank::, and cognition::voice::tests.

The next substantive implementation is turn-scoped Oracle outcome credit and

question/fact separation, followed by contextual propositions and composition.

Nothing here establishes LLM-level capability or a timetable for reaching it.