# LEO VOICE PIPELINE MAP — end-to-end **Generated:** 2026-06-30 · For the "fast-forward / time-compressed voice" investigation. **Scope:** every stage Leo's audio passes through (inbound mic → Gemini → playout) plus every module/dependency Leo relies on. File paths are relative to `C:\KAI\tools\oracle-discord`. > TL;DR of the bug (full detail in §6): `@discordjs/voice@0.19.2` paces a `StreamType.Raw` > resource with an **absolute-clock scheduler** (one 20 ms frame per timer tick, target time > advanced by a fixed +20 ms each frame). When the Node event loop stalls (sync `better-sqlite3` > flush into the 191 MB transcript DB, GC, Gemini session rebuild), the scheduler falls behind > wall-clock and then **fires back-to-back with a 1 ms delay until it catches up**, draining the > buffered PCM faster than real time = the audible speed-up. The producer hands Gemini's whole > utterance to the buffer faster than real time, which is the fuel the catch-up burst races > through. --- ## 0. Process & dependencies Leo loads | Dependency | Where | Role | |---|---|---| | `discord.js` + `@discordjs/voice@0.19.2` | `bots/leo.mjs:1-20` | Discord gateway + voice connection, `AudioPlayer`, `createAudioResource`, `StreamType`, `EndBehaviorType` | | `prism-media` (`prism.opus.Decoder/Encoder`) | `leo.mjs:4793`, `5805` | Opus decode of inbound mic; Opus encode is internal to `@discordjs/voice` on playout | | `ws` (WebSocket) | `shared/gemini-live-bridge.mjs:19` | Gemini Live bidi socket | | `better-sqlite3` | `shared/transcript-memory.mjs:1` | Episodic transcript / FTS5 memory (the 191 MB DB) — **synchronous** engine | | `@ricky0123/vad-node` + `onnxruntime-node` (optional) | `leo.mjs:118-155` | Neural Silero VAD; **OFF by default** (RMS gate used unless `LEO_NEURAL_VAD=1`) | | `stream.PassThrough` | `leo.mjs:104` | The jitter buffer between the audio producer and the player | | `ffmpeg` (spawned) | `leo.mjs:611-637`, `4580` | Only for the **soundboard / TTS-file / edge-reading** side-paths (OggOpus). The **Live conversational voice does NOT go through ffmpeg** — it is raw PCM. | | `tools/oracle-discord/.env` | secrets | `GEMINI_*` keys, `LEO_*` tuning vars. Never printed. | | The Rust engine / Oracle (`command-center-server.mjs` :3001) | separate procs | Not in the audio path; Leo only reads telemetry from it. | Three audio players are created up front (`leo.mjs:435-445`): - **`audioPlayer`** — Leo's primary voice (Gemini Live PCM). This is the one with the bug. - **`effectsPlayer`** — soundboard, thinking-cue, TTS files, edge-reading (OggOpus/Arbitrary). Swaps the voice subscription away and hands it back on Idle. - **`ambientPlayer`** — background ambience. --- ## 1. INBOUND — Discord mic → VAD/gate → Gemini 1. **Receiver subscription** — `getLeoVoiceReceiver()` (`leo.mjs:411`) returns `voiceConnection.receiver`. Per speaker, `_rxLive.subscribe(uid, { end: { behavior: AfterSilence, duration: END_OF_SPEECH_MS } })` (`leo.mjs:4792`, default **2200 ms**, env `LEO_END_OF_SPEECH_MS` / `LEO_SILENCE_FINALIZE_MS`). 2. **Opus decode** — `new prism.opus.Decoder({ frameSize: 960, channels: 2, rate: 48000 })`, `stream.pipe(decoder)` (`leo.mjs:4793-4795`). Inbound is 48 kHz stereo. 3. **Noise gate / VAD** — RMS energy gate per frame (`leo.mjs:~4810+`); thresholds `LEO_MIC_GATE` (idle, default 400) and `LEO_MIC_GATE_SPEAKING` (raised while Leo talks so a cough can't barge in). Neural Silero VAD only if `LEO_NEURAL_VAD=1`. **Echo half-duplex guard:** while `bridge._leoSpeakingUntil` is in the future (`leo.mjs:3456`, `LEO_ECHO_TAIL_MS` default 1200) the mic loop will not forward, so Leo's own speaker echo can't reach Gemini's VAD and cut his reply short. 4. **Forward to Gemini** — gated chunks go to `liveBridge.sendAudio(chunk)` (`leo.mjs:4873`, `4971` pre-roll flush, `5063`). 5. **Downsample for Gemini** — `sendAudio()` (`gemini-live-bridge.mjs:1326`) calls `_downsample48to16()` (`:1443`): 48 kHz stereo → **16 kHz mono s16le**, base64, sent as `realtimeInput.audio` `{ mimeType: "audio/pcm;rate=16000" }` (`:1334-1336`). Activity markers `activityStart` / `activityEnd` (`:1369`, `:1379`); idle silence keepalive (`:1390`); `sendAudioStreamEnd()` (`:1400`). --- ## 2. GEMINI LIVE — the websocket session - **URL:** `wss://generativelanguage.googleapis.com/.../BidiGenerateContent` (`gemini-live-bridge.mjs:29`). - **Model:** `models/gemini-2.5-flash-native-audio-latest` (PRIMARY, env `GEMINI_LIVE_MODEL`), with an ordered fallback list of native-audio preview models, Flash-Live last resort (`:41-52`). - **Voice:** `resolveGeminiVoice(botName)` → `generationConfig.speechConfig. prebuiltVoiceConfig.voiceName` (`:777-788`). - **Setup payload** (`:779-853`): `model`, `generationConfig` (audio response), `realtimeInputConfig` (`:792`), `sessionResumption` handle (`:845`). - **Session lifecycle / ~10-min cap:** - **GoAway** (`:1136-1154`): Gemini warns the session limit (~10 min) is near. Handler throttles (`minGap`), and on an idle session closes without reconnect; otherwise closes now and reconnects cleanly **with context preserved**. - **Session resumption** (`:1127-1129`): stores `newHandle` from `sessionResumptionUpdate`; the next setup resumes with it (fixes the "1008 GoAway abort loop" + memory loss across the boundary). - **Reconnect / key rotation** (`:758`, `:922-1037`): on `1011` billing/credits-depleted or `429`, rotates model then key with backoff; `_reconnectAttempts` budget reset per new model/key; long-cooldown + post-outage reconnect paths. - **Inbound messages** decoded in the WS `message` handler → dispatch to the callbacks below. --- ## 3. OUTBOUND audio generation → decode → resample - **Gemini emits** 24 kHz mono s16le base64 chunks (rate declared in the chunk `mimeType`). - **`bridge.onAudioChunk(base64, mimeType)`** assigned in `leo.mjs:3401`. Per chunk it: 1. Publishes the throttled fleet "Leo is speaking" heartbeat, **deferred via `setImmediate`** so the (now-debounced) `recordLeoVoiceConversation` write never blocks the frame loop (`leo.mjs:3412-3427`) — *prior hot-path fix, keep intact.* 2. **Decodes/resamples**: `GeminiLiveBridge.decodeAudioChunk(base64, mimeType)` (`gemini-live-bridge.mjs:1465`). **Rate-aware** (`srcRate` from `mimeType`, default 24000), **linear interpolation** upsample to **48 kHz stereo s16le**: `outFrames = floor(n * 48000 / srcRate)`, `step = srcRate/48000`, `pos = i*step` computed fresh per frame (no cumulative drift). ✅ Ratio is correct — *ruled out as a cause.* 3. **Echo reference RMS** — sparse stride-32 scan (`leo.mjs:3445-3451`) — *prior hot-path fix; was an O(n) full scan that stalled the loop, now cheap. Keep.* 4. **Writes PCM into the jitter buffer** — see §4. --- ## 4. JITTER BUFFER → AudioPlayer → Opus → UDP (the failure point) - **`bridge._liveAudioStream`** = `new PassThrough({ highWaterMark: 1<<22 })` (`leo.mjs:3458`). - **Start of playout (jitter prime):** a single ~200 ms cushion (`LEO_JITTER_MS`, default 200; `_jitterBytes = ms*192`) is accumulated in `bridge._prebuf`, then flushed into the stream and `audioPlayer.play(createAudioResource(stream, { inputType: StreamType.Raw }))` (`leo.mjs:3463-3484`). - **Steady playout (current logic):** each subsequent decoded chunk is written **immediately** to the PassThrough (`leo.mjs:3485-3508`). Optional backlog drop `LEO_MAX_PLAYOUT_BACKLOG_MS` (default 0 = never drop, because dropping truncates Leo's tail). **This is the producer that dumps the utterance faster than real time.** - **Turn end:** `bridge.onTurnComplete` (`leo.mjs:3512`) flushes any sub-jitter remainder, then `stream.end()` and resets `_playing` (`:3523-3535`); also writes the transcript and per-user channel post. - **Player → wire:** `@discordjs/voice` reads the Raw stream **one 20 ms / 3840-byte frame per scheduler tick** (`node_modules/@discordjs/voice/dist/index.js:600 _stepDispatch`, `:646 state.resource.read()`), Opus-encodes, and sends over the Discord UDP socket. **The scheduler is the only clock** (`index.js:120-145`). A 20 ms frame at 48 kHz / 16-bit / stereo = `48000 * 0.02 * 2ch * 2bytes` = **3840 bytes**. --- ## 5. CONTROL PATHS that touch a turn - **`sendText(text, turnComplete)`** (`gemini-live-bridge.mjs:1413`) — injects context/system lines. **Deferred** when the socket isn't ready into `this._pendingContext` (`:659`, `:1420-1422`), flushed on setup-complete (`:1235-1238`). *Prior fix — keep.* Many call sites in `leo.mjs` (join/leave context, contradiction notes, reading control, etc.). - **Interrupts / barge-in** — `audioPlayer.stop(true)` at `leo.mjs:1764`, `4051`; `.stop()` at `3797`, `4308`, `4428` (pre-empt current sentence on confirmed user speech). - **Tool calls** — handled in the WS message dispatch; `onTurnComplete` clears the narration watchdog and may be tool-only (no audio). - **Three players** — `audioPlayer` (voice), `effectsPlayer` (soundboard/TTS/reading, OggOpus via ffmpeg), `ambientPlayer` (ambience). effects/ambient temporarily `subscribe` away from the voice and hand back on Idle (`leo.mjs:586-661`, `728-757`). - **Transcript / memory DB writes** — `ingestMessage()` (`transcript-memory.mjs:171`) is called on turn-complete and message events (`leo.mjs:3318, 3360, 3572, 3849`). Writes are **already deferred/batched** (`transcript-memory.mjs:102-151`): rows queue in `_writeQueue`, a `setTimeout` (`TRANSCRIPT_FLUSH_MS`, default 150) fires `_flushQueue()` which runs ONE `db.transaction(...)`. **But `better-sqlite3` is synchronous** — when that flush fires it blocks the event loop for the duration of the FTS write, and `ingestMessage` still does two **inline synchronous SELECTs** (dedup `:188-191`, previous-message `:209-214`) on the caller's thread. These are the remaining event-loop stall sources that trigger the catch-up burst. - **Diagnostics** — `shared/voice-debug.mjs` (`LEO_VOICE_DEBUG=1`) wraps the path read-only and logs per-utterance **RTF (Real-Time Factor = audio-ms emitted / wall-ms elapsed)**; RTF > 1.05 = playing faster than real time = the speed-up caught in the act. --- ## 6. ROOT CAUSE (confirmed) — see the separate analysis in the Codex changelog The scheduler in `@discordjs/voice@0.19.2`: ``` // node_modules/@discordjs/voice/dist/index.js 120 var FRAME_LENGTH = 20; 124 function audioCycleStep() { 126 nextTime += FRAME_LENGTH; // target advances a FIXED +20ms/frame 128-130 for (player of available) player._stepDispatch(); // send ONE frame 131 prepareNextAudioFrame(available); 138 audioCycleInterval = setTimeout(audioCycleStep, Math.max(1, nextTime - Date.now())); ``` `nextTime` is an **absolute** wall-clock target, advanced by exactly +20 ms per frame regardless of how long the loop was actually blocked. After a stall of, say, 200 ms, the timer fires with `nextTime` 200 ms behind `Date.now()`, so `Math.max(1, nextTime - Date.now())` = **1**. It then dispatches a frame, advances `nextTime` by 20 ms (still behind), schedules again in 1 ms, and **bursts ~10 buffered frames in ~10 ms of wall-clock** until `nextTime` catches up — i.e. 200 ms of audio played in ~10 ms ≈ 20× = the **fast-forward**. `_stepPrepare` reads exactly one 3840-byte frame from the PassThrough each cycle (`index.js:646`), and because the producer has dumped the whole utterance into the buffer faster than real time, there is always a frame ready for the burst to consume. **Ruled out:** resample ratio bug (§3.2 — correct, no drift), Opus frame-size mismatch (Raw is framed internally at 3840 B; inbound decoder is a separate path), and producer-dump *alone* (it's the fuel, but without the scheduler catch-up burst a correctly-paced player would still play it at real time). **The fix** (implemented in `leo.mjs`, flag `LEO_PACED_PRODUCER`): pace the producer so the PassThrough never holds more than a tiny cushion — there is then no backlog for the catch-up burst to race through, so audio can never exceed real time. See the Codex changelog and `LEO-VOICE-FASTFORWARD-FIX` notes for the exact diff.