← all documents · raw markdown · 12 KB

LEO VOICE PIPELINE MAP — end-to-end

Generated: 2026-06-30 · For the "fast-forward / time-compressed voice" investigation.

Scope: every stage Leo's audio passes through (inbound mic → Gemini → playout) plus

every module/dependency Leo relies on. File paths are relative to C:\KAI\tools\oracle-discord.

> TL;DR of the bug (full detail in §6): @discordjs/[email protected] paces a StreamType.Raw

> resource with an absolute-clock scheduler (one 20 ms frame per timer tick, target time

> advanced by a fixed +20 ms each frame). When the Node event loop stalls (sync better-sqlite3

> flush into the 191 MB transcript DB, GC, Gemini session rebuild), the scheduler falls behind

> wall-clock and then fires back-to-back with a 1 ms delay until it catches up, draining the

> buffered PCM faster than real time = the audible speed-up. The producer hands Gemini's whole

> utterance to the buffer faster than real time, which is the fuel the catch-up burst races

> through.

---

0. Process & dependencies Leo loads

| Dependency | Where | Role |

|---|---|---|

| discord.js + @discordjs/[email protected] | bots/leo.mjs:1-20 | Discord gateway + voice connection, AudioPlayer, createAudioResource, StreamType, EndBehaviorType |

| prism-media (prism.opus.Decoder/Encoder) | leo.mjs:4793, 5805 | Opus decode of inbound mic; Opus encode is internal to @discordjs/voice on playout |

| ws (WebSocket) | shared/gemini-live-bridge.mjs:19 | Gemini Live bidi socket |

| better-sqlite3 | shared/transcript-memory.mjs:1 | Episodic transcript / FTS5 memory (the 191 MB DB) — synchronous engine |

| @ricky0123/vad-node + onnxruntime-node (optional) | leo.mjs:118-155 | Neural Silero VAD; OFF by default (RMS gate used unless LEO_NEURAL_VAD=1) |

| stream.PassThrough | leo.mjs:104 | The jitter buffer between the audio producer and the player |

| ffmpeg (spawned) | leo.mjs:611-637, 4580 | Only for the soundboard / TTS-file / edge-reading side-paths (OggOpus). The Live conversational voice does NOT go through ffmpeg — it is raw PCM. |

| tools/oracle-discord/.env | secrets | GEMINI_* keys, LEO_* tuning vars. Never printed. |

| The Rust engine / Oracle (command-center-server.mjs :3001) | separate procs | Not in the audio path; Leo only reads telemetry from it. |

Three audio players are created up front (leo.mjs:435-445):

audioPlayer — Leo's primary voice (Gemini Live PCM). This is the one with the bug.
effectsPlayer — soundboard, thinking-cue, TTS files, edge-reading (OggOpus/Arbitrary). Swaps the

voice subscription away and hands it back on Idle.

ambientPlayer — background ambience.

---

1. INBOUND — Discord mic → VAD/gate → Gemini

1. Receiver subscription — getLeoVoiceReceiver() (leo.mjs:411) returns

voiceConnection.receiver. Per speaker, `_rxLive.subscribe(uid, { end: { behavior:

AfterSilence, duration: END_OF_SPEECH_MS } }) (leo.mjs:4792`, default 2200 ms, env

LEO_END_OF_SPEECH_MS / LEO_SILENCE_FINALIZE_MS).

2. Opus decode — new prism.opus.Decoder({ frameSize: 960, channels: 2, rate: 48000 }),

stream.pipe(decoder) (leo.mjs:4793-4795). Inbound is 48 kHz stereo.

3. Noise gate / VAD — RMS energy gate per frame (leo.mjs:~4810+); thresholds

LEO_MIC_GATE (idle, default 400) and LEO_MIC_GATE_SPEAKING (raised while Leo talks so a

cough can't barge in). Neural Silero VAD only if LEO_NEURAL_VAD=1.

Echo half-duplex guard: while bridge._leoSpeakingUntil is in the future

(leo.mjs:3456, LEO_ECHO_TAIL_MS default 1200) the mic loop will not forward, so Leo's

own speaker echo can't reach Gemini's VAD and cut his reply short.

4. Forward to Gemini — gated chunks go to liveBridge.sendAudio(chunk)

(leo.mjs:4873, 4971 pre-roll flush, 5063).

5. Downsample for Gemini — sendAudio() (gemini-live-bridge.mjs:1326) calls

_downsample48to16() (:1443): 48 kHz stereo → 16 kHz mono s16le, base64, sent as

realtimeInput.audio { mimeType: "audio/pcm;rate=16000" } (:1334-1336).

Activity markers activityStart / activityEnd (:1369, :1379); idle silence keepalive

(:1390); sendAudioStreamEnd() (:1400).

---

2. GEMINI LIVE — the websocket session

URL: wss://generativelanguage.googleapis.com/.../BidiGenerateContent

(gemini-live-bridge.mjs:29).

Model: models/gemini-2.5-flash-native-audio-latest (PRIMARY, env GEMINI_LIVE_MODEL),

with an ordered fallback list of native-audio preview models, Flash-Live last resort

(:41-52).

Voice: resolveGeminiVoice(botName) → `generationConfig.speechConfig.

prebuiltVoiceConfig.voiceName (:777-788`).

Setup payload (:779-853): model, generationConfig (audio response),

realtimeInputConfig (:792), sessionResumption handle (:845).

Session lifecycle / ~10-min cap:
GoAway (:1136-1154): Gemini warns the session limit (~10 min) is near. Handler

throttles (minGap), and on an idle session closes without reconnect; otherwise closes

now and reconnects cleanly with context preserved.

Session resumption (:1127-1129): stores newHandle from

sessionResumptionUpdate; the next setup resumes with it (fixes the "1008 GoAway abort

loop" + memory loss across the boundary).

Reconnect / key rotation (:758, :922-1037): on 1011 billing/credits-depleted or

429, rotates model then key with backoff; _reconnectAttempts budget reset per new

model/key; long-cooldown + post-outage reconnect paths.

Inbound messages decoded in the WS message handler → dispatch to the callbacks below.

---

3. OUTBOUND audio generation → decode → resample

Gemini emits 24 kHz mono s16le base64 chunks (rate declared in the chunk mimeType).
bridge.onAudioChunk(base64, mimeType) assigned in leo.mjs:3401. Per chunk it:

1. Publishes the throttled fleet "Leo is speaking" heartbeat, deferred via setImmediate

so the (now-debounced) recordLeoVoiceConversation write never blocks the frame loop

(leo.mjs:3412-3427) — *prior hot-path fix, keep intact.*

2. Decodes/resamples: GeminiLiveBridge.decodeAudioChunk(base64, mimeType)

(gemini-live-bridge.mjs:1465). Rate-aware (srcRate from mimeType, default 24000),

linear interpolation upsample to 48 kHz stereo s16le:

outFrames = floor(n * 48000 / srcRate), step = srcRate/48000, pos = i*step computed

fresh per frame (no cumulative drift). ✅ Ratio is correct — *ruled out as a cause.*

3. Echo reference RMS — sparse stride-32 scan (leo.mjs:3445-3451) — *prior hot-path

fix; was an O(n) full scan that stalled the loop, now cheap. Keep.*

4. Writes PCM into the jitter buffer — see §4.

---

4. JITTER BUFFER → AudioPlayer → Opus → UDP (the failure point)

bridge._liveAudioStream = new PassThrough({ highWaterMark: 1<<22 })

(leo.mjs:3458).

Start of playout (jitter prime): a single ~200 ms cushion (LEO_JITTER_MS, default 200;

_jitterBytes = ms*192) is accumulated in bridge._prebuf, then flushed into the stream and

audioPlayer.play(createAudioResource(stream, { inputType: StreamType.Raw }))

(leo.mjs:3463-3484).

Steady playout (current logic): each subsequent decoded chunk is written immediately

to the PassThrough (leo.mjs:3485-3508). Optional backlog drop LEO_MAX_PLAYOUT_BACKLOG_MS

(default 0 = never drop, because dropping truncates Leo's tail). **This is the producer that

dumps the utterance faster than real time.**

Turn end: bridge.onTurnComplete (leo.mjs:3512) flushes any sub-jitter remainder,

then stream.end() and resets _playing (:3523-3535); also writes the transcript and

per-user channel post.

Player → wire: @discordjs/voice reads the Raw stream **one 20 ms / 3840-byte frame per

scheduler tick** (node_modules/@discordjs/voice/dist/index.js:600 _stepDispatch,

:646 state.resource.read()), Opus-encodes, and sends over the Discord UDP socket.

The scheduler is the only clock (index.js:120-145).

A 20 ms frame at 48 kHz / 16-bit / stereo = 48000 * 0.02 * 2ch * 2bytes = 3840 bytes.

---

5. CONTROL PATHS that touch a turn

sendText(text, turnComplete) (gemini-live-bridge.mjs:1413) — injects context/system

lines. Deferred when the socket isn't ready into this._pendingContext (:659,

:1420-1422), flushed on setup-complete (:1235-1238). *Prior fix — keep.* Many call

sites in leo.mjs (join/leave context, contradiction notes, reading control, etc.).

Interrupts / barge-in — audioPlayer.stop(true) at leo.mjs:1764, 4051; .stop() at

3797, 4308, 4428 (pre-empt current sentence on confirmed user speech).

Tool calls — handled in the WS message dispatch; onTurnComplete clears the narration

watchdog and may be tool-only (no audio).

Three players — audioPlayer (voice), effectsPlayer (soundboard/TTS/reading, OggOpus

via ffmpeg), ambientPlayer (ambience). effects/ambient temporarily subscribe away from

the voice and hand back on Idle (leo.mjs:586-661, 728-757).

Transcript / memory DB writes — ingestMessage() (transcript-memory.mjs:171) is called

on turn-complete and message events (leo.mjs:3318, 3360, 3572, 3849). Writes are **already

deferred/batched** (transcript-memory.mjs:102-151): rows queue in _writeQueue, a

setTimeout (TRANSCRIPT_FLUSH_MS, default 150) fires _flushQueue() which runs ONE

db.transaction(...). But better-sqlite3 is synchronous — when that flush fires it

blocks the event loop for the duration of the FTS write, and ingestMessage still does two

inline synchronous SELECTs (dedup :188-191, previous-message :209-214) on the caller's

thread. These are the remaining event-loop stall sources that trigger the catch-up burst.

Diagnostics — shared/voice-debug.mjs (LEO_VOICE_DEBUG=1) wraps the path read-only and

logs per-utterance RTF (Real-Time Factor = audio-ms emitted / wall-ms elapsed); RTF > 1.05

= playing faster than real time = the speed-up caught in the act.

---

6. ROOT CAUSE (confirmed) — see the separate analysis in the Codex changelog

The scheduler in @discordjs/[email protected]:

// node_modules/@discordjs/voice/dist/index.js
120  var FRAME_LENGTH = 20;
124  function audioCycleStep() {
126    nextTime += FRAME_LENGTH;                       // target advances a FIXED +20ms/frame
128-130  for (player of available) player._stepDispatch();   // send ONE frame
131    prepareNextAudioFrame(available);
138    audioCycleInterval = setTimeout(audioCycleStep, Math.max(1, nextTime - Date.now()));

nextTime is an absolute wall-clock target, advanced by exactly +20 ms per frame

regardless of how long the loop was actually blocked. After a stall of, say, 200 ms, the timer

fires with nextTime 200 ms behind Date.now(), so Math.max(1, nextTime - Date.now()) = 1.

It then dispatches a frame, advances nextTime by 20 ms (still behind), schedules again in 1 ms,

and bursts ~10 buffered frames in ~10 ms of wall-clock until nextTime catches up — i.e.

200 ms of audio played in ~10 ms ≈ 20× = the fast-forward. _stepPrepare reads exactly one

3840-byte frame from the PassThrough each cycle (index.js:646), and because the producer has

dumped the whole utterance into the buffer faster than real time, there is always a frame ready

for the burst to consume.

Ruled out: resample ratio bug (§3.2 — correct, no drift), Opus frame-size mismatch (Raw is

framed internally at 3840 B; inbound decoder is a separate path), and producer-dump *alone*

(it's the fuel, but without the scheduler catch-up burst a correctly-paced player would still

play it at real time).

The fix (implemented in leo.mjs, flag LEO_PACED_PRODUCER): pace the producer so the

PassThrough never holds more than a tiny cushion — there is then no backlog for the catch-up

burst to race through, so audio can never exceed real time. See the Codex changelog and

LEO-VOICE-FASTFORWARD-FIX notes for the exact diff.