← all documents · raw markdown · 28 KB

ORACLE-OS VOICE/VIDEO — Feasibility Report + Phased Build Plan

Status: SCOPE / PLAN ONLY — read-only investigation, no code written. Review with the owner before building.

Date: 2026-06-30

Goal (owner intent): Talk to the social bots by voice (and eventually video) directly through the Oracle-OS dashboard — a new item in the left icon rail that opens an in-dashboard voice/video session, as an *alternative path to Discord*. Video would be usable here (unlike Discord). Owner pictures "a separate similar file for the social bots" for this Oracle-OS voice path.

---

1. Verdict up front

Voice-first MVP is very achievable and lower-risk than it looks, because the hard part — the live conversational brain — is already built and is not Discord-specific. Leo's GeminiLiveBridge is a clean WebSocket client to Gemini Live native-audio; all the Discord coupling lives *outside* it in leo.mjs. We reuse the bridge unchanged and write two small adapters (browser-mic-in, browser-speaker-out).

The one genuinely new piece of infrastructure is a browser↔server realtime transport, because today the dashboard has zero streaming plumbing — no WebSocket server, no getUserMedia, no audio elements. It's a pure HTTP-polling REST app. That's the main build cost, and it's modest (the ws package is already a dependency).

Video is a real extension, not a toggle. The bridge does not send image/video frames to Gemini today (its only "image/video" code is *outbound* map tools). Adding vision means a new bridge method plus a Gemini-Live model that accepts video input plus camera-gating work. Worth planning, but clearly Phase 2+.

Recommended transport: a WebSocket audio bridge, NOT WebRTC. Rationale in §5.

---

2. How bot voice works today (the pipeline)

Traced through bots/leo.mjs (6,735 lines) and shared/gemini-live-bridge.mjs (2,493 lines).

                 ┌─────────────── leo.mjs (DISCORD-SPECIFIC) ───────────────┐
  Discord VC ──► receiver.subscribe(userId)  →  prism.opus.Decoder          │
  (user mic)     (48kHz stereo Opus)            (frameSize 960, 2ch, 48k)    │
                        │                                                     │
                        ▼  decoded 48k stereo PCM                             │
                 bridge.sendAudio(chunk) ───────────────────────────────────►│
                 └──────────────────────────────────────────────────────────┘
                                          │
   ┌────────────────── gemini-live-bridge.mjs (REUSABLE / Discord-free) ─────┐
   │  sendAudio(pcm): _downsample48to16() → 16k mono → WS realtimeInput.audio │
   │  { mimeType:"audio/pcm;rate=16000", data:<base64> }  ──► Gemini Live WS  │
   │                                                                          │
   │  Gemini streams back:  serverContent.modelTurn.parts[].inlineData        │
   │    (audio/pcm ~24kHz)  ──► this.onAudioChunk(base64, mimeType)           │
   │  Also: onTranscript, onInputTranscript, onTurnComplete, onToolCall       │
   └──────────────────────────────────────────────────────────────────────┬─┘
                                                                            │
                 ┌─────────────── leo.mjs (DISCORD-SPECIFIC) ───────────────┤
   Discord VC ◄── audioPlayer / paced feed  ◄── decodeAudioChunk()          │
   (Leo speaks)   createAudioResource(PassThrough, StreamType.Raw)          │
                  leoPacedFeed() upsamples 24k→48k stereo, 20ms frames      │
                 └──────────────────────────────────────────────────────────┘

Key files / anchors:

Discord-specific vs reusable — the important split:

| Concern | Where | Reusable off Discord? |
|---|---|---|
| Live conversational brain (WS to Gemini, VAD signalling, tools, transcripts, key rotation, keepalive) | gemini-live-bridge.mjs | ✅ 100% reusable — no Discord imports in the audio path |
| Mic input (Opus decode from a Discord VC) | leo.mjs receiver | ❌ Discord-only — replace with browser mic |
| Audio output (paced feed → @discordjs/voice player) | leo.mjs + gemini-live-voice.mjs | ❌ Discord-only — replace with browser speaker |
| Format contract: input = PCM16 (bridge downsamples 48k→16k mono itself); output = PCM16 ~24kHz base64 | bridge | ✅ this is the clean seam to tap |

The takeaway: the bridge already is the "audio-source-agnostic" core. It takes a PCM buffer in and hands a PCM buffer out. Discord is just one pair of adapters bolted onto that seam. Oracle-OS becomes a *second* pair of adapters.

---

3. Dashboard audit — existing browser↔bot audio/streaming paths

Traced command-center-server.mjs (:3001, 4,258 lines) and oracle.html (6,519 lines).

Finding: there is none. The dashboard is a pure HTTP-polling REST app.

So the browser realtime transport is net-new, but the library is present and the pattern is well-understood.

---

4. Left-rail integration (low-risk, well-understood)

The rail is trivially extensible. In oracle.html:

To add a Voice/Video panel: one new .rail-btn (icon + tip + onclick="setView('voice')"), one new <div class="view-pane" id="view-voice">, add 'voice' to the VIEW_TITLES map and the show/hide array in setView. Surgical, ~15 lines, no risk to existing views. This is the *easiest* part of the whole feature.

---

5. Recommended architecture — WebSocket audio bridge (not WebRTC)

Two credible transports for browser↔server audio:

Option A — WebSocket carrying raw PCM (RECOMMENDED).

Browser captures mic via getUserMedia → an AudioWorklet (or MediaRecorder) produces PCM16 → send binary frames over a WebSocket to the server → server hands each frame to bridge.sendAudio(). Gemini's reply arrives on bridge.onAudioChunk → server relays base64 PCM frames back over the same WebSocket → browser plays them via Web Audio API (AudioContext, scheduled buffer sources).

Option B — WebRTC.

Browser RTCPeerConnection streams Opus to the server; server decodes to PCM for the bridge and encodes bridge output back to Opus.

Recommendation: Option A. It matches the existing code's seams almost exactly and needs no new heavy dependencies. Echo cancellation is the one thing WebRTC gives for free — but for a headphones-on desktop session that's optional, and the browser's getUserMedia({audio:{echoCancellation:true}}) constraint covers most of it client-side.

Where does the bot process live? Two sub-options:

The "separate similar file": create shared/oracle-live-voice.mjs, modeled on gemini-live-voice.mjs, but with the audio sink/source swapped from @discordjs/voice to the WebSocket relay. Same bridge, same personas/resolveGeminiVoice, same tools — just a different mouth and ears. This is exactly the structure the owner described.

---

6. Video — what it additionally takes (Phase 2+)

Sending the browser camera to a bot is a real extension:

1. Bridge does not send vision today. Its only image/video code is *outbound* map tools (satellite/street/flyover posted to Discord). Gemini Live *does* accept video input (inline JPEG frames in realtimeInput), but there is no sendVideo/sendImage method — that's new code on the bridge (small, additive).

2. Model support. Must confirm the selected native-audio Live model accepts image/video input (some native-audio preview models are audio-only). May need a different Live model or a periodic-frame approach (send 1 JPEG/second rather than true video).

3. Browser side: getUserMedia({video:true}), grab frames from a <video>/canvas, JPEG-encode, send over the same WebSocket.

4. Camera gating (owner's explicit tie-in): the camera must turn on only for an active Oracle-OS video session and turn off when it ends. This is a browser-permission + explicit-state concern — a clear "session active" flag drives getUserMedia/track .stop(). This is net-new UI/state and should be built alongside the video panel, not retrofitted.

Video is clean to layer on *after* the voice transport exists, because it rides the same WebSocket.

---

7. Phased build plan

Effort is rough dev-time; risk is likelihood of breaking existing behavior or hitting unknowns.

| Phase | Deliverable | New vs reuse | Effort | Risk | Dependencies |
|---|---|---|---|---|---|
| 0 — Spike | Prove browser mic → WS → bridge.sendAudio → onAudioChunk → browser speaker end-to-end with a hardcoded persona. Throwaway HTML page, no rail. | New: WS server + tiny client. Reuse: bridge as-is. | S–M | Med (audio worklet + jitter buffer is the unknown) | Gemini Live key in server env; ws (already present) |
| 1 — Voice MVP (rail-integrated) | WebSocketServer in/around command-center-server.mjs; shared/oracle-live-voice.mjs (bridge + WS sink/source); left-rail "Voice" button + view-voice pane with connect/disconnect, mic level, live transcript, pick-a-bot. Audio only. | New: transport + voice panel + oracle-live-voice.mjs. Reuse: bridge, personas, tools, transcripts. | M–L | Med | Phase 0; server restart via Start-Dashboard.ps1; browser mic permission |
| 1.5 — Polish | Barge-in/half-duplex (port _leoSpeakingUntil echo-guard + manual-VAD activityStart/End), reconnect, key-rotation surfaced, per-bot voice selection, transcript persistence. | Reuse patterns from leo.mjs. | M | Low–Med | Phase 1 |
| 2 — Video send | sendVideo/sendImage on the bridge (additive); browser camera capture + JPEG frames over the same WS; camera gating (on only during active session). | New: bridge vision method + camera UI/state. | L | Med–High (model must accept video input; verify) | Phase 1; a vision-capable Live model |
| 2.5 — Bot "sees" UX | Show the bot what it's reacting to, wire vision into tool/transcript flow, tune frame rate for cost. | New UI. | M | Med | Phase 2 |
| 3 — Shared identity (optional) | Route Oracle-OS session into the *running* bot process (IPC) so it shares Discord memory/identity instead of a standalone session. | New: real-time cross-process audio IPC. | L | High | Phases 1–2; new IPC channel |

Dependencies summary (what it needs to run):

---

8. Voice-first MVP vs full voice+video

Voice-first MVP (Phases 0–1, optionally 1.5): Click the new rail icon → panel opens → "Connect" → pick a bot → talk, hear it reply, see the live transcript, "Disconnect." Audio only. This is the achievable, high-value first target and reuses ~everything that makes conversation *work*. The only real unknown is browser-side audio worklet + jitter buffering, de-risked by Phase 0.

Full voice+video (Phases 2+): adds camera→bot vision, camera-gating, and possibly a different Live model. Bigger, with a genuine unknown (model video-input support) — plan it, but gate it behind a working MVP.

---

9. Open questions for the owner before building

1. Session ownership: standalone Oracle-OS session (A1, simpler, no Discord memory) vs shared with the running bot (A2/Phase 3, faithful but much more work)? Recommend starting A1.

2. Which bots first in the dashboard voice panel — Leo, or the social four (Gemini/Claudey/Groq/X), or all?

3. Video model: OK to spend a spike confirming a native-audio + video-input Live model exists / picking a frame-rate strategy before committing to Phase 2?

4. Scope confirmation: this is all new surface area (transport + a new shared module + new panel) — none of it touches kaiverse.js or the flight controls, and the oracle.html change is additive. Confirm we're OK opening this track alongside the KAIVERSE visual work.

---

10. Bottom line

---

---

OWNER SPEC v2 — Full feature vision (captured verbatim-in-intent)

Added: 2026-06-30 · Status: SPEC + PLAN ONLY, no code written. This section captures the owner's fuller vision described after the feasibility report above, so nothing is lost. It supersedes/expands §7 where they conflict, and it stays reconciled with the feasibility findings (WebSocket-PCM transport, Discord-free gemini-live-bridge, [email protected] already a dep, new shared/oracle-live-voice.mjs, ~15-line left-rail add).

v2.1 The feature in one line

An in-dashboard, Discord-style VOICE + VIDEO call system on the Oracle-OS side — NOT Discord. You talk and (eventually) video-call the bots (and/or humans) from inside the Oracle dashboard itself, with a real call UI, participant tiles, active-speaker highlighting, picture-in-picture, device selection, and an invite/accept join flow. All signalling and media are directed to Oracle OS, not routed through Discord.

v2.2 Detailed requirements

A. Entry point — a "VIDEO" tab on voice channels.

On a voice channel (e.g. *Leo Voice*, *ai-social-chat Voice*, etc.), alongside the existing per-channel tabs CHAT / THREADS / METRICS / SETTINGS, add a new "VIDEO" tab. Despite the name, it means voice AND video. Clicking it opens the call.

B. Call view.

Opening the call turns the main window into a Discord-like call view. You can also chat inside the call (text chat remains available in the call view).

C. Invite / add participants.

You can add/invite whoever you want — bots and/or humans. Invitees receive a join REQUEST and must ACCEPT to join. All of this is directed to Oracle OS, not Discord.

D. Participant tiles.

Each participant gets a tile showing, per participant:

E. Picture-in-Picture (PiP) while chatting.

While in a call, if you click the CHAT tab, the video feed pops out into a small floating window pinned to the top/side (Twitch-stream style) so you can keep seeing who's talking while you chat.

F. Device settings.

A settings area to pick which CAMERA and MIC to use. Default selection = "Device Default." The user can change to any available device (enumerated from the browser).

G. RF device toggle.

If an RF camera/device is present, it gets an ENABLE/DISABLE toggle inside the Camera options area. This ties into camera-privacy gating — camera/RF are only on when a session is active AND the user enables it.

H. Behavior rule — bots don't reply to text during voice.

While you're in the VOICE call, the bots do NOT reply to your TEXT messages (they're on voice) — but they CAN SEE the text (it's in the chat). This is possible precisely because this is Oracle-OS-side, not Discord.

v2.3 How v2 maps onto the feasibility findings

Nothing in v2 changes the recommended architecture; it mostly adds UI surface and a join-flow:

v2.4 Phased build plan (v2)

Reconciled with the feasibility plan; refresh-testable frontend first, live media second, video third, multi-party last.

Phase 1 — UI foundation (frontend-only, hard-refresh testable, NO live media)

Build the entire interface and device-selection layer with no live media wired yet — pure UI + enumerateDevices.

Phase 2 — Voice MVP (live)

Make the call actually talk to a bot.

Phase 3 — Camera VIDEO + PiP + RF feed

Add real video on the transport built in Phase 2.

Phase 4 — Multi-participant INVITE / ACCEPT join flow

Turn it into a real multi-party call.

v2.5 Restart-vs-refresh quick reference (v2)

| Change | Apply how |
|---|---|
| oracle.html (tabs, call view, tiles, PiP, device pickers, RF toggle, all Phase-1 UI) | Hard-refresh the dashboard — no restart |
| command-center-server.mjs (WebSocketServer, signalling, text-suppression gating) | .\Start-Dashboard.ps1 — restarts :3001 only, leaves engine + bots + pipeline alone |
| new shared/oracle-live-voice.mjs | picked up on the same .\Start-Dashboard.ps1 |
| gemini-live-bridge.mjs (additive sendVideo/sendImage) | .\Start-Dashboard.ps1 (server owns the session in the A1 design) |

v2.6 Net-new beyond the original feasibility plan (flag for owner)

The original plan already covers the voice transport, oracle-live-voice.mjs, and video-send. v2 adds: the tabbed VIDEO entry + full Discord-style call view, participant tiles with avatar-fallback + active-speaker glow, PiP pop-out, device pickers with "Device Default," the RF enable/disable toggle, the text-suppression-during-voice rule, and the invite/accept multi-party join flow (incl. humans). None of this touches kaiverse.js or the flight controls; the oracle.html work is additive. Human-participant media is the one item with materially higher risk and is intentionally last.

*Documentation only — no code changed, no version bumped.*