Status: PLANNING ONLY — no code was changed producing this document (read-only investigation).
Date: 2026-07-01
Scope: Five workstreams (A–E) to calibrate Leo/Oracle conversation: voice-input timing, stability, transcript registration, new-joiner onboarding, and vitals/emotion-aware context.
Codex/Cargo: NOT bumped (no code change). Bump only when the implementing edits land.
> Grounding note. Every "current code" claim below cites the real file + function + line range read via the Read/Grep tools against the Windows files (the WSL mount was cross-checked with byte-accurate grep -n). Line numbers are current as of the reads on 2026-07-01; re-confirm exact offsets at edit time because these files (leo.mjs 6735 lines, gemini-live-bridge.mjs 2493 lines) move.
> The fragile path. "Voice-out" = Gemini Live native-audio (Charon) → onAudioChunk → the 20ms playout frame loop in leo.mjs. It is dialed in and was made worse once by a "paced producer" change. Every proposal below is INPUT-side or PROMPT-side and must not touch onAudioChunk, the audio frame loop, nsUpdateCamera-style timing, or the turn-complete/interrupt handlers that drive playback. Isolation notes are called out per workstream.
---
The fragmented-voice bug (Workstream A) is a premature-flush bug, not a "missing aggregation" bug. An aggregation layer already exists: gemini-live-bridge.mjs forwards every tiny STT partial via onInputTranscript (bridge lines 1203–1216, deliberately un-filtered), and leo.mjs accumulates them into bridge._inputTranscript and flushes on a timer (bridge.onInputTranscript, leo.mjs 3681–3703; flushInputTranscript, leo.mjs 3443–3679). The problem is the flush trigger at leo.mjs 3690–3692:
const hasSentenceEnd = /[.!?…]\s*$/.test(acc) || /[,;:]\s+[A-Z]/.test(acc);
const longEnough = acc.length >= 48 || (acc.split(/\s+/).length >= 7);
if ((hasSentenceEnd || longEnough) && acc.length >= 10) { flushInputTranscript(); return; }
bridge._inputTranscriptTimer = setTimeout(flushInputTranscript, 1400);
This flushes the moment the buffer reaches 7 words OR 48 chars OR any sentence-ish punctuation — so one spoken thought gets chopped mid-utterance. The Discord evidence maps exactly onto these triggers:
"at locations of like where people are" → 7 words → immediate flush (hits longEnough word-count)."at or" → short residual, flushed by the 1400ms timer."things are at?" → ends with ? → immediate flush (hits hasSentenceEnd).Each fragment is then treated as a complete turn (recorded via recordHumanVoiceEvent, posted as who [Voice]: …, and eligible to trigger a reply through the response watchdog at leo.mjs 3554–3566). The fix is to endpoint on silence, not on punctuation/length, and to make the buffer per-speaker. This is the highest-leverage, lowest-risk change in the whole plan and it lives entirely on the input side.
Suggested phase order: A → C → B → D → E (rationale in the Phasing section). A first because it is the owner's #1 symptom and is isolated; C next because clean per-speaker transcript identity is a prerequisite that D and E both lean on.
---
STT ingress. Human speech reaches Leo through Gemini Live's inputTranscription, handled in gemini-live-bridge.mjs _handleServerMessage (the serverContent branch), lines 1202–1216:
const inputTranscription = serverContent?.inputTranscription || serverContent?.input_transcription;
if (inputTranscription?.text) {
const txt = inputTranscription.text;
// FORWARD ALL non-empty chunks. Gemini streams the USER's words in tiny pieces...
if (txt && /[a-z0-9]/i.test(txt)) { this.onInputTranscript?.(txt); }
}
A prior fix deliberately removed a per-chunk length >= 8 filter here (comment at 1205–1212) because it was dropping the small building-block deltas so the user's line never accumulated. Correct call — but it pushed all endpointing responsibility downstream to leo.mjs, which then endpoints wrongly.
Accumulation + flush (the aggregation layer that already exists). leo.mjs:
bridge.onInputTranscript (3681–3703): appends each delta to the single buffer bridge._inputTranscript, resets a debounce timer, and flushes early on the punctuation/length triggers quoted in the TL;DR.flushInputTranscript (3443–3679): trims the buffer, drops <3-char noise (3451–3455), records the human voice event (3458–3464), runs self-echo suppression (3467–3501), narration-command intercept (3509–3514), AI/fleet-speech suppression (3525–3547), then posts ${who} [Voice]: ${spokenText} to the resolved transcript channel (3549–3552) and arms the response watchdog (3554–3566).Why one utterance fragments. Two compounding causes:
1. The early-flush triggers (3690–3692) fire on natural sub-phrase boundaries (7 words, a trailing ?, a comma+capital), so a single continuous utterance is emitted as 2–4 turns.
2. The buffer is global, not per-speaker (bridge._inputTranscript is one string). If two people speak close together, or Gemini attributes partials to whoever bridge._currentSpeaker currently is, fragments can also cross-contaminate speakers.
Introduce a proper endpointing/aggregation layer on the input side only, replacing the eager flush with silence-based finalization:
1. Silence-timeout endpointing. Replace the "flush on 7 words / 48 chars / punctuation" logic in bridge.onInputTranscript (leo.mjs 3690–3696) with a rolling inactivity timer as the *primary* endpoint: each incoming delta resets a timer of LEO_UTTERANCE_SILENCE_MS (propose default 700–900ms, env-tunable). The utterance is only flushed when no new STT delta has arrived for that window. Keep a max-utterance safety cap (e.g. flush anyway at ~12s or ~400 chars) so a monologue still lands. Sentence-ending punctuation should *lengthen confidence* but no longer force an immediate flush on its own.
2. Per-speaker buffering. Change bridge._inputTranscript (single string) into a per-speaker map keyed by bridge._currentSpeakerId (the id already resolved at flush time, leo.mjs 3523). Each speaker gets its own buffer + its own silence timer, so overlapping speakers don't merge and a speaker change forces a flush of the previous speaker's buffer. This is the natural home for the "buffer partial STT fragments per speaker" requirement.
3. Finalization signal, if available. If Gemini Live emits a segment-final marker on inputTranscription (needs a one-session log capture to confirm — see Open Questions), prefer that as the endpoint and use the silence timer only as a fallback. Do this in the bridge (around 1203–1216) by passing an isFinal flag through onInputTranscript(txt, isFinal); leo.mjs flushes immediately on isFinal, else runs the silence timer.
Landing zones: leo.mjs 3681–3703 (rewrite the flush-decision block), plus a small buffer-structure change touching every bridge._inputTranscript reference (2360, 3444–3447, 3682–3685, 5408, 5446). Optional bridge signature tweak at gemini-live-bridge.mjs 1202–1216.
Risk: LOW–MEDIUM. This is purely input-text timing; it never touches onAudioChunk, the frame loop, onTurnComplete (1250–1273), or onInterrupted (1276–1283). The one real risk is added latency: a 700–900ms silence window delays when Leo "sees" the finished utterance, which could make him feel slower to respond. Mitigations: (a) make the window env-tunable and iterate with the owner; (b) keep the existing response watchdog (3554–3566) as the safety net but re-point it so it counts from utterance *finalization*, not from each fragment (today it can be re-armed by every fragment, which is part of the erratic timing). Do not shorten the window so far that it re-introduces fragmentation — this is an eyeball-iterate knob. Keep everything behind LEO_UTTERANCE_SILENCE_MS / a LEO_UTTERANCE_ENDPOINTING=1 flag so it is instantly revertible.
---
Four nested recovery layers exist (bottom-up):
1. Launcher chain, dependency-gated at the top. Start-KAI.ps1 (stages 122–212) gates each stage on the previous being *ready*: Ollama → KAI engine kai.exe --oracle (waits up to EngineTimeoutSec≈300s for port :3334, Start-KAI.ps1 155–164, hard-throws if it never answers) → kai_supervisor.py (RAM watchdog, only after engine confirmed) → overnight pipeline → run-oracle-discord.ps1 -NoStartKai. run-oracle-discord.ps1 (19–44, 302–342, 598) sets the sleep-list and calls run-ecosystem.ps1, which just runs node ecosystem-manager.mjs.
2. In-tree supervisor: ecosystem-manager.mjs. Owns fleet startup order but by timer-stagger only, not readiness gating: Dashboard, then Oracle (523–524), KAI after 2s (526–530), then Analyst/Researcher/Kai-Coder at +1200ms stagger (536–547), then Leo/Gemini/Claudey/X/Groq ("Leo first = voice anchor", 537, 548–561). scriptForProcess() maps names to scripts (106–113): Oracle→oracle-gateway.mjs, Leo→bots/leo.mjs, X/Claudey/Groq→native-bot.mjs, KAI→bots/kai.mjs, Dashboard→command-center-server.mjs. Respawn on child.on('close') (468–506): 5s re-spawn, with a wedge guard (recordExitAndCheckWedged, 80–86) that suspends auto-respawn after 3 exits in 60s. IPC restart paths (RESTART_BOT, SLEEP/WAKE, PHOENIX_PROTOCOL, RESTART_ALL) at 361–462; a file-queue fallback (state/restart_requests.json, polled every 5s, 581–598) fed by process-supervisor.mjs queueRestartToFile().
3. Per-process guards. Leo's unhandledRejection / uncaughtException are log-and-stay-alive (leo.mjs 6639–6654) with a transient-net whitelist (6636–6637). Manager has the same (17–37). ipc.mjs handles EADDRINUSE by probing the port holder's /health and exiting 0 on self-duplicate/collision (151–200).
4. Outermost watchdog: phoenix-watchdog.ps1 (Task Scheduler, every 5 min). 3-pulse check — engine :3334, manager PID + heartbeat <45s, dashboard :3001 (38–65). Crash-loop aware: ≥15 "exited with code" lines ⇒ restart just the engine (throttled once/8min, 105–129); manager-down ⇒ storm-guarded full relaunch honoring authorized_stop.json (131–232).
Gemini Live reconnect (gemini-live-bridge.mjs ws.on('close'), 926–1155): config remembered at connect (724–729, _userClosed). Cascade: model-rejected ⇒ rotate LIVE_MODEL_FALLBACKS (949–968); billing/rate-limit ⇒ per-model cooldown + key rotation through _keyList, then a single retry after KAI_LIVE_BILLING_COOLDOWN_MS≈15min (969–1073); connectivity-aware ⇒ if offline go dormant and rebuild via connectivity.whenOnline(), if online linear backoff 1000ms×attempt capped at 3 tries then a 30s recovery poll (1074–1114). Context preserved via session-resumption handles (873–878, 1157–1163) and GoAway clean-reconnect (1165–1190). connectivity.mjs uses raw TCP probes (not DNS) with dormant-not-die doctrine.
Other-API fallback. Leo's neural path is dual-provider (leo.mjs 6463–6497): Groq → Ollama fallback (and vice-versa). Voice degrade: the bridge fires onVoiceDegraded when all Live keys are depleted (1047–1051) but no bot registers that callback — so on full depletion Leo goes silent for the cooldown instead of dropping to edge-TTS.
Observers (diagnose, don't restart): heartbeat-monitor.mjs isolates a dead bot for 5 min (suppressBot, 112–123); resource-saver.mjs is a CPU/RAM/GPU governor that throttles loops but never restarts.
setTimeout offsets (539–561) with no check that Oracle/KAI actually finished booting. A slow Oracle boot lets dependents start against a not-ready parent.onVoiceDegraded emitted (1048) but unconsumed; Leo has no TTS-failover wiring.shared/*.mjs patches never auto-restart** (process-supervisor.mjs 58–59) — the bulk of the code base.kai_supervisor.py, up to ~5 min latency.Do not rip out the layered design — it is mature. Close the specific gaps:
1. Explicit dependency/readiness gating in ecosystem-manager.mjs (539–561). Before spawning each dependent tier, poll the parent's /health (Oracle, KAI, Dashboard already expose health per B.1) with a bounded wait + timeout, replacing blind setTimeout stagger with "start when parent is ready, else after cap." Encode an explicit dependency table (Leo depends on {Oracle, KAI, Dashboard}) so the order is data, not timing luck.
2. Wire the orphaned voice-degrade callback. Register bridge.onVoiceDegraded in leo.mjs to flip Leo to the existing edge-TTS path (he already has en-US/en-GB edge voices) for the cooldown window, then auto-restore when Live keys recover. This is a *voice-out-adjacent* change → treat as MEDIUM risk (see B.4); gate behind a flag and iterate.
3. Escalation after reconnect exhaustion. When the bridge's online reconnect (1092–1114) exhausts its 3 tries + 30s poll, emit a distinct "voice-stuck" signal the supervisor/heartbeat can act on (restart just Leo), so a wedged socket isn't invisible to the health layer.
4. Wedge-guard auto-retry with backoff. Add a single delayed auto-clear attempt (exponential backoff) before requiring human action (496–501), so transient port contention self-heals.
5. Codify startup/dependency + failure/recovery as a documented contract (a short FLEET-SUPERVISION.md) so "what depends on what / what recovers each" is explicit, matching the owner's ask.
Risk: MEDIUM. Items 1, 3, 4, 5 are supervisor/launcher-side and cannot affect audio playout. Item 2 (voice-degrade → TTS failover) is the only one that touches Leo's speaking path — it must reuse the *existing* edge-TTS engine (already proven) and must NOT alter onAudioChunk or the Live frame loop; it only chooses TTS when Live is fully unavailable. Flag-gate it (LEO_TTS_FAILOVER=1) and verify with the owner that switching mid-session doesn't stutter. Everything else is safe to ship independently.
---
Registration is a hardcoded 1:1 map, not the SQLite DB. Three redundant copies of "which channel is whose":
channel-rules.mjs 29 USER_TRANSCRIPT_MAP {userId → transcriptChannelId}; 37 TRANSCRIPT_USER_INFO reverse map {channelId → {userId,name,role,slotIdx}}.identities.mjs 6 HUMAN_REGISTRY carries transcriptChannelId + voiceSlot per human.voice-manager.mjs 7 FIXED_ASSIGNMENTS {userId → slotIndex}.userTranscriptChannels (Map userId→channelId, 1017), userToSlot/slotToUser (1014, 1040–1043).k (Ryan) and Taz are hardcoded: USER_TRANSCRIPT_MAP["1111106883135217665"] = "1500527640107417783" (Ryan, Slot 1, channel-rules.mjs 30) and ["1286110163505385523"] = "1500529928184008885" (Taz, Slot 2, 31), mirrored in identities.mjs and voice-manager.mjs. getTranscriptChannel(userId) (voice-manager.mjs 55) is a pure USER_TRANSCRIPT_MAP[userId] || null lookup.
The SQLite store (transcripts.db, transcript-memory.mjs) is episodic memory, not the registration map. transcript_fts (36) stores speaker, user_id, content, context, channel_id, timestamp; recall is keyed by channel_id/user_id. There is no slots/registration table.
Slots: fixed cap of 6. channel-rules.mjs 16 LEO_VOICE_SLOTS is a 6-element channel-ID array (Slot 1 Ryan, 2 Taz, 3–4 open/guest, 5–6 public). Dynamic assignment only fills idx 4–5: assignSlot loops for (let i=4; i<6; i++) and returns -1 (Full) if both taken (voice-manager.mjs 76–87). On -1, leo.mjs drops the join: No slots available for ${userId}. Ignoring. (2100–2107).
Speaker → channel resolution: resolveLiveTranscriptChannel (leo.mjs 3433–3441): runtime userTranscriptChannels.get(sid) → getTranscriptChannel(sid) → fallback CHANNEL_IDS.SUNDAY (public).
The "application sequence / gatekeeping" is essentially soft, not a hard gate. What exists is triggerVoiceLockOnboarding(user, profileName) (leo.mjs 4598–4622), called from the voice-join handler (2213–2222) for any non-bot whose name isn't in biometrics.profiles. It posts a [SECURITY ALERT] to CHANNEL_IDS.UNREGISTERED_SLOT and DMs an optional "voice lock signature" enrollment script ("You can still use the system without this"). The real verification tech (identity-vault.mjs Azure IdentityVault) has a stub verifyIdentity (empty body). So enrollment is offered but nothing blocks speech; access to a transcript channel is just a Discord permission overwrite granted automatically (updatePermissions 104, bootstrapPermissions 148). No queue/waitlist/apply-to-speak exists (applyToSpeak/waitlist grep empty).
The owner wants: register everyone to a transcript channel by user ID; verify who's registered to whom and when; only use an open transcript that belongs to that registered ID; retire the soft "application/voice-lock sequence" as the default; and fall back to a sequence only when all slots are full.
1. Make registration data, not code. Add a persistent registration record keyed by user ID (extend the existing data/voice_slots.json used by getSlotAssignments/assignSlot, or a small registrations table in transcripts.db) storing {userId, name, transcriptChannelId, slotIdx, registeredAt, registeredBy}. Keep the hardcoded USER_TRANSCRIPT_MAP for k/Taz as seed data, but resolve dynamic users from the persistent store so "who is registered to whom and when" is queryable. registeredAt satisfies the "and when" requirement.
2. Registration-verified resolution. Tighten resolveLiveTranscriptChannel (leo.mjs 3433–3441): only return a channel if the resolved channel's registered owner ID matches the current speaker ID; never silently fall through to SUNDAY for a registered user (that fallback is a likely source of cross-user leakage). Add an explicit verifyRegistration(userId, channelId) helper.
3. New default flow ("just converse → name → register"). When an unregistered person speaks and a slot is free: Leo converses normally, asks their name, calls a new registerUser(userId, name) that claims a free slot (reusing assignSlot, voice-manager.mjs 63–87) and writes the persistent record, then uses it exactly like k/Taz. Retire triggerVoiceLockOnboarding as the gate — keep the voice-signature enrollment as a *later, optional* security add-on, not an entry step.
4. Slots-full fallback = the retained sequence. Keep the current "application sequence" (the voice-lock / unregistered-slot path, plus the assignSlot === -1 branch at leo.mjs 2100–2107) alive but reachable only when assignSlot returns -1 (all dynamic slots taken). That is the owner's explicit "unless all transcript slots are full, fall back to the sequence" condition. Everywhere else, skip it.
Risk: MEDIUM. All changes are on the transcript-routing / identity side, not audio. The genuine hazard is mis-routing: a bug in registration-verified resolution could send one person's words to another's channel (the exact corruption the current code warns about at leo.mjs 3519–3523). Mitigate by keeping the change additive and logged, verifying against real speakers with the owner, and never removing the SUNDAY fallback for *unregistered* speakers (only for registered ones where a wrong-owner match should hard-stop rather than mis-route). No audio-path code is touched.
---
guildMemberAdd/GuildMembers across bots/*.mjs, root *.mjs, and shared/*.mjs returned zero matches (verified directly). The Discord clients don't register the GuildMembers intent.oracle-gateway.mjs (ecosystem-manager.mjs 107), and it has no voiceStateUpdate, no member.send, no join handler (verified directly). Oracle only DMs the owner (client.users.fetch(OWNER_ID) → owner.send) for EOD reports (oracle-gateway.mjs 692/777/784). Oracle's message entry is client.on('messageCreate') (891) which resolves identity (resolveIdentityFromMemory, 897) and fire-and-forgets each message to /api/digest-message (901–922).index.mjs (a separate voice client), not Oracle. index.mjs voice-join branch (657–697) DMs first-timers: member.send("Oracle: Welcome to the Roundtable. I've assigned you to Private Transcript #…") (682) and the capacity-full variant (667). So member.send(...) works; it just isn't wired into Oracle or to server-join.profile-memory.mjs recordProfile() (44) calls captureEntityFacts() (56) as a side-effect of recording, and posts a [PROFILE] line to the SENSITIVE channel. It is invoked for every non-bot human message in leo.mjs (1961–1962) and imported by native-bot.mjs (8). The extractor captureEntityFacts (transcript-memory.mjs 597–619) is conservative first-person regex (/\bi\s+(?:love|adore)\s+(.+)/i → like, etc., one fact/message, no LLM). Facts land in the entity_facts table; getEntityProfile() (627) returns a compact summary for prompt injection. A second store, user-warehouse.mjs (user_facts/user_fingerprints, keyed by user ID, setFact at 105), is defined but has no live caller in the message hot path.recordProfile|setFact|captureEntityFacts in oracle-gateway.mjs empty).1. Add a server-join handler. Register a new client.on('guildMemberAdd', …) in Oracle (oracle-gateway.mjs) and add the GuildMembers intent to Oracle's client. On join: (a) post a friendly welcome to CHANNEL_IDS.WORK (used at oracle-gateway.mjs 936/1105/1237) — warm, not "you can't reply here"; and (b) member.send(...) a concise DM tour (reuse the proven member.send pattern from index.mjs 682) explaining what the server is and what to expect — short, friendly, and without saying "I'm building your profile."
2. Author the copy, keep it short. The welcome + DM tour text needs writing (there's no existing string to reuse). Keep the DM to a few sentences; the owner explicitly does not want a wall of text or a dead-end tone.
3. Passive profile on Oracle's message path. Import recordProfile and call it right after the digest in oracle-gateway.mjs messageCreate (after 922, before command routing at 929+), mirroring leo.mjs 1961 — it already has message.author.id, resolved from, message.content, message.channelId in scope. This gives Oracle the same passive, non-announced fact capture Leo has, satisfying "passively register useful details through normal conversation." No new store needed; entity_facts + captureEntityFacts already do this conservatively.
4. Optionally enrich the extractor later (reaction-triggered facts for E), but v1 should just reuse the existing conservative regex path to stay safe.
Risk: LOW. Entirely text/DM/Discord-event side; no audio path involved. Risks are product-shaped, not stability-shaped: (a) DM spam / Discord rate limits if welcome fires repeatedly — guard with a "already welcomed" check against the registration store from C; (b) privacy tone — the DM must not read as surveillance, per the owner. member.send failures are already .catch()-safe in the existing pattern. Adding the GuildMembers intent is a config change on Oracle's client only.
---
drive-system.mjs readEngineState() (158–168) pulls valence, chi, phi_g, rho (aggregated over a window from source='rust-engine'). They're assembled and exposed by command-center-server.mjs buildVitals() (3089), served at :3001 GET /api/vitals (3758–3766). The engine is at :3334 (/api/session, /api/status); the KAI bot's in-process drives at :3401 GET /drives (kai.mjs 643).drive-system.mjs applyEngineSignals() (173–218) maps engine vitals → six drives (satisfaction ← valence+coherence, prediction_error ← chi, curiosity ← 1−phi_g/5, etc.). Public API: getDrives() (493) raw 0–1 drives; getDriveDirective() (436–476) turns levels into prose directives ([DRIVE: PAIN]…, [DRIVE: SOCIAL] Extended silence. Engage genuinely…, else [DRIVE: BALANCED]); getPredictionConfidenceDirective() (520–530) hedges from prediction accuracy. A separate autonomic-state.mjs getDrives() (31) computes anxiety/curiosity/urgency from CPU/RAM/latency with its own directive. No NeuralOscillator class exists in the JS tree — "valence" originates in Rust.buildLeoSystemPrompt (leo.mjs 4407; native-bot.mjs 2176) injects only getPredictionConfidenceDirective() (leo.mjs 4494, native-bot.mjs 2215). The social/text path start-bot.mjs (1382–1456) injects autonomic-state anxiety + prediction directive. getDriveDirective() (the mood directive) and raw engine vitals are computed but never injected into any spoken prompt — they only feed /drives and the dashboard.sentiment/userMood/detectEmotion/theoryOfMind/readReaction empty. metacognition.mjs (62–63, 308–321) models other bots ([EMPATHY]), not the human; getSnapReaction (native-bot.mjs 2970, leo.mjs 5519) is Leo's own filler ack, not a read of the user. epistemic-vault.mjs emotionalWeight is memory salience. biographies.mjs calls Leo "emotionally perceptive" as persona claim with no backing detection.The owner wants the speaking AI to factor in its own current vitals/emotional state plus a read of the other party's reaction, lightweight and only on turns that warrant it.
1. Inject own-state into the prompt (the easy, high-value half). In buildLeoSystemPrompt (leo.mjs 4407, at the existing hook line 4494) and its native twin (native-bot.mjs 2215), add getDriveDirective() and a compact vitals line (from the already-exposed /api/vitals or a cached readEngineState()), alongside the prediction directive that's already there. For Oracle, add the same at its prompt/status assembly (oracle-gateway.mjs 365–388). This immediately gives "factor in my own state before composing" using signals that already exist and are already rendered elsewhere — no new computation.
2. Gate it so it's not on every message. Only include the mood/vitals block when it "warrants it": e.g. when a drive directive is non-BALANCED, or on the first turn of a conversation, or every N turns, or when the vitals delta since last turn exceeds a threshold. This satisfies "lightweight, not on every message" and avoids prompt bloat.
3. Reaction-read of the other party (the new half). Add a lightweight readReaction(userText, priorContext) — start rule-based/cheap (punctuation, short affirmations vs. pushback, sentiment lexicon, response latency already available from the utterance-endpointing timer in A), optionally a tiny LLM classification reusing the getSnapReaction plumbing. Feed a one-line [THEORY-OF-MIND: user seems …] note into the same gated prompt block. Only run it on turns that warrant it (e.g. after Leo asked something, or when the user's utterance is emotionally loaded), not every message.
4. Source of truth: reuse getDriveDirective/getDrives (drive-system) and /api/vitals — do not invent a parallel emotion model.
Risk: LOW–MEDIUM. This is prompt-context assembly, not audio — it does not touch onAudioChunk or the frame loop. Risks: (a) prompt bloat / latency in the Live session if the block is large or added every turn — hence the gating in item 2; (b) behavioral drift — feeding mood into Leo could make him moody/erratic in ways the owner dislikes (this is the "paced producer made it worse" family of risk, but on the text-prompt side). Mitigate: flag-gate (LEO_SELF_STATE_IN_PROMPT=1, LEO_TOM=1), keep the injected text terse and directive-style (matching the existing [DRIVE: …] format), and eyeball-iterate with the owner one signal at a time (own-state first, reaction-read second). Do not let vitals injection change turn timing or interrupt behavior.
---
Phase 1 — A: utterance endpointing (highest leverage, most isolated). Fix the fragmented-voice bug: silence-based endpointing + per-speaker buffering (leo.mjs 3681–3703), flag-gated, iterate the silence window with the owner. Ship alone so its effect on turn-taking is cleanly observable. Re-point the response watchdog to fire from finalization.
Phase 2 — C: registration by ID. Persist registrations, registration-verified resolveLiveTranscriptChannel, new "converse → name → register" default, retire the voice-lock sequence except on slots-full. Do this before D/E because both rely on clean per-user identity and "who is registered when."
Phase 3 — B: supervision hardening. Dependency/readiness gating in the manager, wire the orphaned onVoiceDegraded → TTS failover (the one voice-adjacent item — flag-gate + owner verify), reconnect-exhaustion escalation, wedge auto-retry, and the FLEET-SUPERVISION.md contract. Independent of A/C; slotted third because it's the largest surface and partly overlaps the fragile path.
Phase 4 — D: onboarding + passive profile on Oracle. Add the guildMemberAdd handler + GuildMembers intent, welcome-to-WORK + DM tour (author copy), and recordProfile on Oracle's messageCreate. Depends on C for the "already registered/welcomed" check.
Phase 5 — E: vitals/theory-of-mind in prompts. Inject own-state (drive directive + vitals) into the prompt builders, gated; then add the reaction-read. Last because it's an enhancement layered on a calibrated, stable base, and it's the highest behavioral-drift risk.
Each phase is independently shippable and flag-gated; none requires the next to function.
---
1. A — silence window value. What LEO_UTTERANCE_SILENCE_MS feels right (700? 900? 1200?)? This is an eyeball-iterate knob; needs a live fly/talk session. Also: is a max-utterance cap of ~12s acceptable, or do you want longer monologues to land as one turn?
2. A — does Gemini Live emit a segment-final flag? I could not confirm from the code whether inputTranscription carries an isFinal/segment marker. One instrumented voice session (log the raw inputTranscription payloads) would tell us whether we can endpoint on a real signal instead of only silence. Worth doing before finalizing A.
3. C — where should registrations persist? Extend data/voice_slots.json, or add a registrations table to transcripts.db? And do you want registration to require your approval, or fully self-serve (Leo asks name → registers) for anyone in a free slot?
4. C — keep 6 slots? The cap of 6 (2 fixed + 2 guest + 2 public) is hardcoded in LEO_VOICE_SLOTS. Is 6 still right, or should the dynamic range grow before "slots full → sequence" triggers?
5. B — voice-degrade → TTS failover. Confirm you want Leo to auto-switch to edge-TTS when all Gemini Live keys deplete (vs. staying silent). This touches the speaking path, so it needs your sign-off and a live test.
6. D — welcome + DM tour copy. The exact wording needs your voice. Also: should the public welcome fire in WORK, or a dedicated welcome channel? And should the passive-profile capture on Oracle post [PROFILE] lines to the same SENSITIVE channel Leo uses, or somewhere else?
7. D — GuildMembers intent. Enabling guildMemberAdd requires the privileged GuildMembers gateway intent on Oracle's bot (and the toggle in the Discord Developer Portal). Confirm that's acceptable.
8. E — how often should own-state / theory-of-mind fire? "Turns that warrant it" needs a concrete rule: non-BALANCED drive only? Every N turns? After Leo asks a question? And how expressive should mood be in his replies — subtle, or clearly felt?
9. E — reaction-read fidelity. Start rule-based/cheap, or spend a small LLM call per warranted turn? The latter is richer but adds latency/cost to the Live loop.
---
*End of plan. No code was modified. Implementing any phase requires the usual surgical, flag-gated edits + a Codex version bump at that time.*