← all documents · raw markdown · 8 KB

The root cause of the language problem

*2026-08-30. Measured against the live engine, not reasoned from the code. Every number below can be

re-run with the commands at the bottom.*

---

THE MEASUREMENT THAT ENDS THE GUESSING

Asked KAI *"what is your name?"*. The retrieval returned this, and the fourth column is the one that

matters — it is the raw relevance, recovered by dividing the final score back out by the confidence

multiplier the scorer applies:

| rank | final score | confidence | raw relevance | cell |
|---:|---:|---:|---:|---|
| 1 | 2.628 | 4.00 | 0.8086 | "Your internal code has a special label called a metadata identifier…" |
| 2 | 2.595 | 4.20 | 0.7699 | "My name is KAI." |
| 3 | 2.543 | 4.00 | 0.7824 | "The developer discovered a hidden feature…" |
| 4 | 1.440 | 2.50 | 0.7201 | [ENGRAM] User felt neutral about 'what is your name?'… |
| 5 | 0.616 | 0.41 | 0.8297 | "Claude: what is your name?" *(quarantined echo)* |
| 6 | 0.272 | 5.00 | 0.0708 | "The concept of a convex set is fundamental…" |

Read the raw relevance column. Every plausible cell scores between 0.72 and 0.83. A span of

0.11. The lattice cannot tell them apart.

By the encoder's own measure, **"My name is KAI." is LESS relevant to "what is your name?" than a

lecture about metadata identifiers.** 0.7699 against 0.8086. And the single most relevant thing in

the entire lattice — 0.8297, the highest number in the column — is the quarantined echo of the

question itself.

That is the disease. Everything else is a symptom.

---

THE CAUSAL CHAIN

1. The encoder has no compositional semantics

SparseVec::encode (sparse_vec.rs:308) is layered hashing of surface form:

Layer 1 character trigrams weighted 1x

(plus) per-character positional hashes

Layer 2 normalized word hashing weighted 3x

Layer 3 word bigrams

Layers 4-5 character bigrams and 4-grams

There is no learned representation and no composition. Two English sentences that share function

words and a topic land in nearly the same region of the space. For a five-word question, overlap with

almost any lattice sentence is high and undifferentiated — which is exactly the 0.72–0.83 band above.

This is precisely the hole the letter/word calculus was designed to fill. Meaning built UP from

letters that fuse, words that carry roles, and boundaries that carry weight is *compositional*. Bag-

of-hashes is not. The calibration already recorded the consequence: **L4, word-role-in-context,

scores 0%** — role_of(word) takes a word and no context, so "name" in *what is your name* is the

same token as "name" in *the name of the function*, and the geometry has no way to separate them.

2. Because relevance is flat, CONFIDENCE becomes the real ranker

universe.rs:2094:

let boosted = if raw > 0.15 {

let s = if cell.claim.confidence >= 2.9 { 0.85 } else { 0.5 };

raw * (s + 0.6 * cell.claim.confidence.min(5.0))

} else { raw };

The multiplier runs from 0.50 (confidence 0) to 3.85 (confidence 5) — a 7.7x span.

Measured relevance spread across genuine candidates: 1.15x (0.72 → 0.83).

Confidence is roughly seven times more powerful than relevance in deciding what KAI says.

Ranking is therefore not "what answers this question" but "what is the most-trusted cell that is

vaguely on topic". Rank 5 above proves it: the most relevant item in the lattice was pushed to fifth

place by a low confidence score.

3. Confidence measures curation, not aptness

Confidence is how sure the system is that a claim is TRUE and worth keeping. It says nothing about

whether the claim ANSWERS THIS QUESTION. In the live lattice:

ingested lecture material 4.0 - 5.0

his own identity facts 4.2

engram records of failures 2.5

a genuine live utterance 0.41

So bulk ingested teaching material outranks his own identity by construction, on every question,

forever.

4. The reply is the winning cell's text, verbatim

The generative path cannot rescue a bad pick, because the generative path does not compose — it

renders. word_calculus::render() re-punctuates a word list that cosine already chose, and

calculus_vsa, the bridge from the calculus into the geometry, has zero callers. So whatever

retrieval puts at rank 1 is what comes out of his mouth.

5. Every failure is stored, and becomes the best match for its own question

Rank 4 above: [ENGRAM] User felt neutral about 'what is your name?'… — a record of him failing that

exact question, now sitting in the lattice as a high-relevance match for it. Every failed answer

makes the next attempt at the same question worse.

---

WHY EVERY SYMPTOM FOLLOWS FROM THIS

| symptom seen tonight | explained by |
|---|---|
| byte-identical answers to *"are you tired?"* and *"what did I ask a moment ago?"* | flat relevance -> same top-confidence cells win regardless of question (1 + 2) |
| *"what is your name?"* answered with Normalizing Flows | high-confidence ingest material, topically adjacent, beats the identity cell (2 + 3) |
| his own name sitting at rank 2 and never spoken | a 0.04 relevance difference decided it, against a 7.7x confidence lever (1 + 2) |
| replaying Leo's transcript as his own memory | conversation cells carry high confidence; nothing in ranking knows whose words they are (3 + 4) |
| the 40-second essay on the Earth's 23.5-degree tilt for 17 x 23 | surface-form hashing matched the numeral; no semantics to know it was arithmetic (1) |
| a stored tutoring correction emitted verbatim, scaffolding included | rank-1 cell text is the reply (4) |
| answers getting worse for repeated questions | ENGRAM feedback loop (5) |

---

WHAT THIS MEANS FOR THE FIX

The fixes I shipped tonight (846–853) were all downstream of this. They were worth doing — the

Oracle path had no voice at all, the commit gate was judging a constant, the multi-hop expansion

hung on a greeting — but none of them could have solved this, because none of them touch how meaning

is measured.

Ranked by how much of the problem each removes:

1. Compositional encoding — the word/letter calculus doing the work it was built for.

Addresses cause 1, which every other cause depends on. This is the owner's design and it is the

only item on this list that raises the ceiling rather than rearranging what is under it.

The specific keystone is L4: role-in-context, already named in the Codex as *"the single

largest gap in the language stack and the direct expression of Ryan's thesis"*. Role induction

now supplies a learned role distribution from 39,690 cells; nothing yet selects from it using a

word's actual neighbours.

2. Separate "true" from "apt" in ranking. Confidence should not be a 7.7x multiplier on

relevance. A cell being trustworthy and a cell answering the question are two different

judgements and the scorer currently multiplies them into one number. Cheap to change, large

effect, and testable immediately against the table at the top of this document.

3. Stop storing failures where they answer their own question. The ENGRAM loop is measurable and

self-reinforcing.

4. Wire calculus_vsa. Until the calculus can put a vector into the geometry it cannot choose a

word, and item 1 has no path to the mouth.

Item 2 is the fastest honest win and item 1 is the actual answer. Doing 2 first would make him

noticeably better within a day; doing only 2 would leave the ceiling exactly where it is.

---

RE-RUN EVERY NUMBER HERE

# the table at the top — raw relevance recovered from score and confidence

POST [internal service][internal endpoint] {"query":"what is your name?","user_id":"x"}

# for each hit: multiplier = (conf >= 2.9 ? 0.85 : 0.5) + 0.6 * min(conf, 5.0)

# raw_relevance = score / multiplier

# the scorer itself

sed -n '2079,2100p' [internal module]

# the encoder — confirm there is no semantic layer

sed -n '308,360p' [internal module]

# the calculus scores, including L4 at 0%

cargo run --release --bin kai_language_calibration