LEO-VISION-BROWSE-DESIGN-GOAL.md

---

TL;DR — the four things the owner asked about

1. The "F" browser-view tool — Does not exist in the fleet. No Firecrawl, Puppeteer, Playwright, or any visual/screenshot browser. The "starts with F, browser view" is almost certainly Firecrawl (a hosted scrape/screenshot API) — which is NOT installed. What Leo *does* have today is open_page (fetches the raw text of a URL) and web_search (Google-grounded answer + source links). Both are text-only; neither renders or screenshots a page, and neither is wired to the live call UI.

2. Camera vision — Exists, real, flag-gated OFF. Leo can literally see the call camera via Gemini Live multimodal video (KAI_VIDEO_MODE=1), plus an optional face-recognition tap (KAI_FACE_RECOGNITION=1). What's missing is the *chain*: "capture what the camera sees → identify it → search it → show the results," and a surface to show them on.

3. Image generation — Exists, real. shared/gemi-image.mjs calls Google gemini-2.5-flash-image and is exposed as the generate_image tool; Leo already has a handler for it. Today it's text→image only. The same model is image-to-image capable, so restyling a camera frame into a t-shirt design is a small additive change (feed the frame as an input image). Needs GEMINI_API_KEY and has its own image quota.

4. "The CAD thing" — Does not exist as a garment/product/CAD viewer. The only 3D-model viewer in the repo is visualizer/ — a *BrainVisualizer* (loads brain.glb to show lattice metrics). BUT the ingredients for a 3D t-shirt mockup viewer are already here: Three.js r128 (kaiverse.js), the GLTFLoader (three-gltfloader-r128.js) is already vendored and already <script>-loaded in oracle.html, and a working OrbitControls+GLTFLoader viewer pattern is proven in the visualizer. So "the cad thing" = a to-be-built orbitable 3D t-shirt mockup that maps the generated design onto a shirt model. Pure browser WebGL — cheap on this box.

---

Detailed findings (with evidence)

1. Web-browsing / page-fetch / "F" tool

Tools that exist (defined in tools/oracle-discord/shared/native-tools.mjs, BASE_TOOLS_SCHEMA):

No F-named browser tool anywhere. Repo-wide search for firecrawl|puppeteer|playwright|browser_view|browserless|screenshot_url|surf_web found zero hits in the fleet (tools/oracle-discord/**). The only firecrawl string in the whole tree is inside src-donated/ (an unrelated donated TS codebase, not wired into KAI). The only tool whose name starts with "F" is find_place (Google Maps), which is not a browser view.

Conclusion: the owner's "F… browser view option" is not a current KAI tool. Best interpretation: Firecrawl (hosted crawl/scrape/screenshot API — fits "look it up and *show* me" because it can return a rendered page screenshot). It would be a new dependency + API key. Cheaper non-Firecrawl option: reuse the existing web_search + open_page for the *content*, and add a small rendered "show" panel (link cards, or an <iframe> of the page where the site's CSP allows, or a one-shot screenshot).

2. Camera vision / multimodal

The frame path is fully built and traceable end-to-end:

• Browser (oracle.html, ~L3043 vrCam()): getUserMedia({video}) → self-preview tile → every 2 s grabs a canvas toDataURL('image/jpeg', 0.6) and sends {type:'frame', data:<base64>} over the existing /ws/voice socket. The camera button only renders when the server announces video is enabled (m.event==='video', L3025).
• Server → session: /ws/voice forwards the frame to oracle-live-voice.sendFrame() (shared/oracle-live-voice.mjs L333).
• Session → model: sendFrame() → gemini-live-bridge.sendVideoFrame() (shared/gemini-live-bridge.mjs L1391) → realtimeInput.video on the Gemini Live socket. Gated by KAI_VIDEO_MODE=1 (default OFF; the bridge bounces frames when off).
• Face-recognition tap (additive, KAI_FACE_RECOGNITION=1, default OFF): oracle-live-voice._recognizeFrame() (L351) → person-recognition.ingestFrame() → embeds/matches against a shared person store and injects a non-spoken [WHO YOU SEE] context line so Leo greets a known person by name.

What this means: with KAI_VIDEO_MODE=1, Leo *already sees* the camera and can talk about what's in front of it. What's missing for the owner's ask: there is no "grab this frame → identify the object → web_search it → show the owner the results" chain, and no display surface in the call to show anything (the call UI is audio + camera tiles only). Note the Live-video model streams frames for conversational awareness; a reliable "identify + search" step is better done by capturing one still and sending it to gemini-2.5-flash (vision) for a labelled identification, then feeding that into the existing web_search.

3. Image generation

• shared/gemi-image.mjs: generateImage(prompt) → POST to gemini-2.5-flash-image :generateContent with responseModalities:['TEXT','IMAGE'], reads image bytes from candidates[].content.parts[].inlineData, returns a Buffer. Requires GEMINI_API_KEY (or GOOGLE_API_KEY). Header note in the file: image gen has its own free-tier quota.
• Exposed as the generate_image tool (native-tools.mjs schema L54, handler L583) which emits a generate_image ToolEvent; Leo already consumes it (bots/leo.mjs L750, posts the image to Discord).
• Current limitation: the wrapper is text→image only. gemini-2.5-flash-image also accepts an input image part, so image-to-image restyle (camera frame + "make this a bold single-color t-shirt graphic, vector, transparent bg") is a small additive function alongside generateImage() — not a new service.

Conclusion: image-gen is the one genuinely paid/quota-bound dependency in this chain, but it's already integrated and keyed. Restyle needs only a new "pass an input image" code path.

4. "The CAD thing" (3D / mockup / product viewer)

• No garment/CAD/product-mockup viewer exists. The only GLTFLoader instantiation in the repo is visualizer/src/main.js L97 — class BrainVisualizer, which loads a brain.glb (drei-assets) with OrbitControls + UnrealBloomPass and drives it from kai_ticks.csv lattice metrics. That's a metrics visualizer, not a product viewer.
• But the building blocks are already in place:
• Three.js r128 is already loaded and driving kaiverse.js.
• three-gltfloader-r128.js (a GLTFLoader build) is vendored at repo root and already <script src>-loaded in oracle.html (L2465) — so a GLTF model viewer can run inside the dashboard today.
• A proven OrbitControls + GLTFLoader + bloom viewer pattern exists in visualizer/.
• test_fetch.glb at repo root shows GLB loading has been exercised before.

Conclusion: "the cad thing" is best read as a 3D t-shirt/product mockup viewer to be built — an orbitable shirt model with the generated design applied as a texture/decal. It is pure in-browser WebGL (no server compute), so it's cheap on this box. It needs a CC0 t-shirt .glb/.obj and texture-mapping/decal code. This is build effort, not a compute or licensing problem.

---

Honest gap analysis — have vs. build

Feasibility on this Windows/CPU box:

• P1 (browse/show): light. Text + link cards are free-ish (Gemini key). A *visual* page screenshot means Firecrawl (paid/free-tier key) or a headless-browser dependency (Playwright/Puppeteer — heavier, adds an install; workable but avoid unless the owner wants true page screenshots).
• P2 (vision search): all Gemini API, no local GPU. Cost = Gemini vision + search API usage.
• P3 (image restyle): gemini-2.5-flash-image, no local compute — but its own image quota; heavy use likely needs a paid tier. This is the main cost item.
• P4 (3D viewer): trivial for the box — browser WebGL only, no server load.

---

Phased plan (flag-gated, reversible, non-breaking)

Cross-cutting first (enables P1/P2/P3/P4 to "show"): add a SHOW panel to the vr-room call UI in oracle.html — a flag-gated (window.KAI_SHOW_PANEL / server-announced) region that can render: text/source cards, an image, an <iframe>, or a <canvas> (for the 3D viewer). Additive DOM only; does not touch audio scheduling, VAD, the pacer, or the camera tiles.

P1 — Leo browses and shows on the call

• Wire the existing web_search + open_page results (and grounding source links) into the new SHOW panel as titled cards.
• Leo says it out loud (already can) and the panel displays the sources/snippets.
• Optional visual page render: only if the owner wants true screenshots — add Firecrawl (/scrape with screenshot) behind a new KAI_FIRECRAWL_KEY, or an <iframe> fallback for CSP-friendly sites. Decide before building (cost/dependency).
• *Touches:* oracle.html (panel), a small server route to relay results to /ws/voice. No control changes.

P2 — Camera-vision "what am I looking at → search → show"

• Add a "look at this" trigger (owner voice intent or a button) that grabs the current camera frame (the 2 s JPEG already exists) as a one-shot still.
• Send that still to gemini-2.5-flash (vision) for a short identification label → feed the label into the existing web_search → render results in the SHOW panel (P1 surface).
• Keep it independent of the KAI_VIDEO_MODE Live-video stream (that stays for conversational sight); this is a discrete capture→identify→search→show action, flag-gated (e.g. KAI_VISION_SEARCH).
• *Touches:* oracle-live-voice.mjs (a captureAndIdentify seam), server relay, oracle.html panel. No pacer/VAD changes.

P3 — Image-gen restyle → t-shirt design

• Add an image-to-image function beside generateImage() in gemi-image.mjs that accepts an input frame + a restyle prompt (t-shirt graphic: bold, vectorized, limited palette, transparent/flat bg).
• Trigger from a recognized camera still (P2) or any chosen image; show the returned design in the SHOW panel with a "use in mockup" button.
• Flag-gated (KAI_IMAGE_RESTYLE). Reuses the existing key + tool-event plumbing.
• *Touches:* gemi-image.mjs (additive fn), native-tools.mjs (optional new restyle_image tool or an arg on generate_image), panel. Cost note: image quota — see below.

P4 — View the design in the CAD/3D viewer

• Build a small Three.js r128 module that loads a CC0 t-shirt .glb, maps the P3 design as a texture/decal, and gives OrbitControls (rotate/zoom). Render into the SHOW panel <canvas>.
• Reuse the already-loaded three-gltfloader-r128.js and the visualizer/ pattern; keep it isolated from kaiverse.js (do not share the KAIVERSE scene/camera).
• Flag-gated (KAI_MOCKUP_VIEWER). Needs a shirt model asset (CC0) added to kaiverse_assets/ or similar.
• *Touches:* a new JS module + panel mount. No KAIVERSE / control changes.

Suggested order: SHOW panel → P1 → P2 → P3 → P4. Each ships behind its own flag so the owner can A/B and revert, and none of them alter the working first-person KAIVERSE controls, nsUpdateCamera, the voice pacer, or the existing call audio/camera behavior.

---

Cost / dependency flags (know before we build)

• GEMINI_API_KEY (already in .env): powers web_search, vision identify, and image-gen. Image generation (gemini-2.5-flash-image) has its own quota — P3 is the most likely to need a paid tier under real use.
• Firecrawl (optional, P1 only): needed *only* if the owner wants true rendered page screenshots rather than text + link cards. New dependency + paid/free-tier API key. If we skip it, P1 still works with text/sources + optional iframe.
• Headless browser (alt to Firecrawl): Playwright/Puppeteer would render pages locally but adds a heavy install and CPU load on this box — not recommended unless screenshots are a hard requirement.
• CC0 t-shirt 3D model (P4): a free asset to source and vendor; no license cost, just needs picking.
• No other paid services required. P2/P4 add no local compute cost of note.

---

What was NOT done here (per instructions)

No shipped code was edited, no server/fleet restart, no Codex/Cargo version bump. This document is discovery + plan only. Building begins only after the owner confirms scope (especially: Firecrawl yes/no for P1, and image-gen quota/paid tier for P3).