# LEO-VISION-BROWSE-DESIGN-GOAL.md

| | |
|---|---|
| **Status** | INVESTIGATION + PLAN ONLY — no code shipped, no restart, no version bump. |
| **Author** | kai-engineer discovery pass |
| **Date** | 2026-07-10 |
| **Scope** | A capability chain for the Oracle-OS LIVE VOICE/VIDEO call (Leo): browse-and-show → camera-vision search → image-gen restyle → 3D "CAD" mockup view. |
| **Focus rule** | This is a NEW additive track. It does NOT touch the KAIVERSE visual overhaul, `nsUpdateCamera`, the voice pacer, or any shipped call controls. Every phase is flag-gated + reversible. |

---

## TL;DR — the four things the owner asked about

1. **The "F" browser-view tool** — **Does not exist in the fleet.** No Firecrawl, Puppeteer, Playwright, or any visual/screenshot browser. The "starts with F, browser view" is almost certainly **Firecrawl** (a hosted scrape/screenshot API) — which is NOT installed. What Leo *does* have today is `open_page` (fetches the raw **text** of a URL) and `web_search` (Google-grounded answer + source links). Both are text-only; neither renders or screenshots a page, and neither is wired to the live call UI.

2. **Camera vision** — **Exists, real, flag-gated OFF.** Leo can literally see the call camera via Gemini Live multimodal video (`KAI_VIDEO_MODE=1`), plus an optional face-recognition tap (`KAI_FACE_RECOGNITION=1`). What's missing is the *chain*: "capture what the camera sees → identify it → search it → show the results," and a surface to show them on.

3. **Image generation** — **Exists, real.** `shared/gemi-image.mjs` calls Google `gemini-2.5-flash-image` and is exposed as the `generate_image` tool; Leo already has a handler for it. Today it's **text→image only**. The same model is image-to-image capable, so restyling a camera frame into a t-shirt design is a small additive change (feed the frame as an input image). Needs `GEMINI_API_KEY` and has its own image quota.

4. **"The CAD thing"** — **Does not exist as a garment/product/CAD viewer.** The only 3D-model viewer in the repo is `visualizer/` — a *BrainVisualizer* (loads `brain.glb` to show lattice metrics). BUT the ingredients for a 3D t-shirt mockup viewer are already here: Three.js r128 (kaiverse.js), the GLTFLoader (`three-gltfloader-r128.js`) is already vendored **and already `<script>`-loaded in `oracle.html`**, and a working OrbitControls+GLTFLoader viewer pattern is proven in the visualizer. So "the cad thing" = a **to-be-built** orbitable 3D t-shirt mockup that maps the generated design onto a shirt model. Pure browser WebGL — cheap on this box.

---

## Detailed findings (with evidence)

### 1. Web-browsing / page-fetch / "F" tool

**Tools that exist** (defined in `tools/oracle-discord/shared/native-tools.mjs`, `BASE_TOOLS_SCHEMA`):

| Tool | Line | What it actually does |
|---|---|---|
| `open_page` | schema 232, handler 718 | `readUrlContent(url)` from `url-reader.mjs` — fetches a URL and returns **raw text** (title + first 15k chars). Handles GitHub raw/repo specially, else generic page text. **No JS render, no screenshot, no images.** |
| `web_search` | schema 246, handler 724 | `webSearch(query)` → `google-search.mjs` `askGoogle()`. Uses **Gemini `google_search` grounding** (a live Google search summarized by `gemini-2.5-flash`) and returns an answer **plus source titles/URLs**. Text-only. Needs a Gemini key. |
| `query_wayback_machine` | schema 125, handler 607 | **Stub** — returns a canned string, not a real archive fetch. |
| `arxiv_search` | schema 139, handler 611 | **Stub** — canned string. |

**No F-named browser tool anywhere.** Repo-wide search for `firecrawl|puppeteer|playwright|browser_view|browserless|screenshot_url|surf_web` found **zero** hits in the fleet (`tools/oracle-discord/**`). The only `firecrawl` string in the whole tree is inside `src-donated/` (an unrelated donated TS codebase, not wired into KAI). The only tool whose name starts with "F" is `find_place` (Google Maps), which is not a browser view.

**Conclusion:** the owner's "F… browser view option" is not a current KAI tool. Best interpretation: **Firecrawl** (hosted crawl/scrape/**screenshot** API — fits "look it up and *show* me" because it can return a rendered page screenshot). It would be a new dependency + API key. Cheaper non-Firecrawl option: reuse the existing `web_search` + `open_page` for the *content*, and add a small rendered "show" panel (link cards, or an `<iframe>` of the page where the site's CSP allows, or a one-shot screenshot).

### 2. Camera vision / multimodal

**The frame path is fully built and traceable end-to-end:**

- **Browser (`oracle.html`, ~L3043 `vrCam()`):** `getUserMedia({video})` → self-preview tile → every **2 s** grabs a canvas `toDataURL('image/jpeg', 0.6)` and sends `{type:'frame', data:<base64>}` over the existing `/ws/voice` socket. The camera button only renders when the server announces video is enabled (`m.event==='video'`, L3025).
- **Server → session:** `/ws/voice` forwards the frame to `oracle-live-voice.sendFrame()` (`shared/oracle-live-voice.mjs` L333).
- **Session → model:** `sendFrame()` → `gemini-live-bridge.sendVideoFrame()` (`shared/gemini-live-bridge.mjs` L1391) → `realtimeInput.video` on the Gemini Live socket. **Gated by `KAI_VIDEO_MODE=1` (default OFF; the bridge bounces frames when off).**
- **Face-recognition tap (additive, `KAI_FACE_RECOGNITION=1`, default OFF):** `oracle-live-voice._recognizeFrame()` (L351) → `person-recognition.ingestFrame()` → embeds/matches against a shared person store and injects a non-spoken `[WHO YOU SEE]` context line so Leo greets a known person by name.

**What this means:** with `KAI_VIDEO_MODE=1`, Leo *already sees* the camera and can talk about what's in front of it. **What's missing for the owner's ask:** there is no "grab this frame → identify the object → `web_search` it → show the owner the results" chain, and **no display surface** in the call to show anything (the call UI is audio + camera tiles only). Note the Live-video model streams frames for conversational awareness; a reliable "identify + search" step is better done by capturing one still and sending it to `gemini-2.5-flash` (vision) for a labelled identification, then feeding that into the existing `web_search`.

### 3. Image generation

- `shared/gemi-image.mjs`: `generateImage(prompt)` → POST to `gemini-2.5-flash-image` `:generateContent` with `responseModalities:['TEXT','IMAGE']`, reads image bytes from `candidates[].content.parts[].inlineData`, returns a `Buffer`. Requires `GEMINI_API_KEY` (or `GOOGLE_API_KEY`). Header note in the file: **image gen has its own free-tier quota.**
- Exposed as the `generate_image` tool (`native-tools.mjs` schema L54, handler L583) which emits a `generate_image` ToolEvent; **Leo already consumes it** (`bots/leo.mjs` L750, posts the image to Discord).
- **Current limitation:** the wrapper is **text→image only**. `gemini-2.5-flash-image` also accepts an **input image** part, so image-to-image restyle (camera frame + "make this a bold single-color t-shirt graphic, vector, transparent bg") is a **small additive function** alongside `generateImage()` — not a new service.

**Conclusion:** image-gen is the one genuinely paid/quota-bound dependency in this chain, but it's already integrated and keyed. Restyle needs only a new "pass an input image" code path.

### 4. "The CAD thing" (3D / mockup / product viewer)

- **No garment/CAD/product-mockup viewer exists.** The only GLTFLoader instantiation in the repo is `visualizer/src/main.js` L97 — class `BrainVisualizer`, which loads a `brain.glb` (drei-assets) with `OrbitControls` + `UnrealBloomPass` and drives it from `kai_ticks.csv` lattice metrics. That's a metrics visualizer, not a product viewer.
- **But the building blocks are already in place:**
  - Three.js **r128** is already loaded and driving `kaiverse.js`.
  - **`three-gltfloader-r128.js`** (a GLTFLoader build) is vendored at repo root **and already `<script src>`-loaded in `oracle.html` (L2465)** — so a GLTF model viewer can run inside the dashboard today.
  - A proven OrbitControls + GLTFLoader + bloom viewer pattern exists in `visualizer/`.
  - `test_fetch.glb` at repo root shows GLB loading has been exercised before.

**Conclusion:** "the cad thing" is best read as a **3D t-shirt/product mockup viewer to be built** — an orbitable shirt model with the generated design applied as a texture/decal. It is pure in-browser WebGL (no server compute), so it's cheap on this box. It needs a CC0 t-shirt `.glb`/`.obj` and texture-mapping/decal code. This is build effort, not a compute or licensing problem.

---

## Honest gap analysis — have vs. build

| Capability | Already there | Must be built |
|---|---|---|
| Web content for the call | `web_search` (Gemini-grounded) + `open_page` (raw text), both working with a Gemini key | The **"show" surface** in the call; optionally a *visual* page render/screenshot (Firecrawl or headless browser) |
| Camera sight | Full frame path to Gemini Live (`KAI_VIDEO_MODE`), face-recog tap | Still-frame **capture-on-demand → identify → search → show** chain |
| Image generation | `gemi-image.mjs` + `generate_image` tool + Leo handler (text→image) | **Image-to-image restyle** path (feed input frame); render result on the call, not just Discord |
| 3D "CAD" viewer | Three.js r128, GLTFLoader vendored + loaded in `oracle.html`, viewer pattern in `visualizer/` | The **t-shirt mockup viewer** itself (model + texture/decal mapping + orbit UI) |
| **Shared blocker** | — | **The live-call UI has no display panel** — only audio + camera tiles. Every "show the owner" step needs one new flag-gated SHOW panel. |

**Feasibility on this Windows/CPU box:**
- P1 (browse/show): light. Text + link cards are free-ish (Gemini key). A *visual* page screenshot means Firecrawl (paid/free-tier key) or a headless-browser dependency (Playwright/Puppeteer — heavier, adds an install; workable but avoid unless the owner wants true page screenshots).
- P2 (vision search): all Gemini API, no local GPU. Cost = Gemini vision + search API usage.
- P3 (image restyle): `gemini-2.5-flash-image`, no local compute — but **its own image quota**; heavy use likely needs a paid tier. This is the main cost item.
- P4 (3D viewer): trivial for the box — browser WebGL only, no server load.

---

## Phased plan (flag-gated, reversible, non-breaking)

**Cross-cutting first (enables P1/P2/P3/P4 to "show"):** add a **SHOW panel** to the `vr-room` call UI in `oracle.html` — a flag-gated (`window.KAI_SHOW_PANEL` / server-announced) region that can render: text/source cards, an image, an `<iframe>`, or a `<canvas>` (for the 3D viewer). Additive DOM only; does not touch audio scheduling, VAD, the pacer, or the camera tiles.

### P1 — Leo browses and shows on the call
- Wire the existing `web_search` + `open_page` results (and grounding source links) into the new SHOW panel as titled cards.
- Leo says it out loud (already can) **and** the panel displays the sources/snippets.
- Optional visual page render: only if the owner wants true screenshots — add **Firecrawl** (`/scrape` with `screenshot`) behind a new `KAI_FIRECRAWL_KEY`, or an `<iframe>` fallback for CSP-friendly sites. Decide before building (cost/dependency).
- *Touches:* `oracle.html` (panel), a small server route to relay results to `/ws/voice`. No control changes.

### P2 — Camera-vision "what am I looking at → search → show"
- Add a "look at this" trigger (owner voice intent or a button) that grabs the **current** camera frame (the 2 s JPEG already exists) as a one-shot still.
- Send that still to `gemini-2.5-flash` (vision) for a short identification label → feed the label into the existing `web_search` → render results in the SHOW panel (P1 surface).
- Keep it independent of the `KAI_VIDEO_MODE` Live-video stream (that stays for conversational sight); this is a discrete capture→identify→search→show action, flag-gated (e.g. `KAI_VISION_SEARCH`).
- *Touches:* `oracle-live-voice.mjs` (a `captureAndIdentify` seam), server relay, `oracle.html` panel. No pacer/VAD changes.

### P3 — Image-gen restyle → t-shirt design
- Add an **image-to-image** function beside `generateImage()` in `gemi-image.mjs` that accepts an input frame + a restyle prompt (t-shirt graphic: bold, vectorized, limited palette, transparent/flat bg).
- Trigger from a recognized camera still (P2) or any chosen image; show the returned design in the SHOW panel with a "use in mockup" button.
- Flag-gated (`KAI_IMAGE_RESTYLE`). Reuses the existing key + tool-event plumbing.
- *Touches:* `gemi-image.mjs` (additive fn), `native-tools.mjs` (optional new `restyle_image` tool or an arg on `generate_image`), panel. **Cost note: image quota — see below.**

### P4 — View the design in the CAD/3D viewer
- Build a small Three.js r128 module that loads a CC0 t-shirt `.glb`, maps the P3 design as a texture/decal, and gives OrbitControls (rotate/zoom). Render into the SHOW panel `<canvas>`.
- Reuse the already-loaded `three-gltfloader-r128.js` and the `visualizer/` pattern; keep it isolated from `kaiverse.js` (do not share the KAIVERSE scene/camera).
- Flag-gated (`KAI_MOCKUP_VIEWER`). Needs a shirt model asset (CC0) added to `kaiverse_assets/` or similar.
- *Touches:* a new JS module + panel mount. No KAIVERSE / control changes.

**Suggested order:** SHOW panel → P1 → P2 → P3 → P4. Each ships behind its own flag so the owner can A/B and revert, and none of them alter the working first-person KAIVERSE controls, `nsUpdateCamera`, the voice pacer, or the existing call audio/camera behavior.

---

## Cost / dependency flags (know before we build)

- **`GEMINI_API_KEY`** (already in `.env`): powers `web_search`, vision identify, and image-gen. Image generation (`gemini-2.5-flash-image`) has **its own quota** — P3 is the most likely to need a **paid tier** under real use.
- **Firecrawl (optional, P1 only):** needed *only* if the owner wants true rendered page **screenshots** rather than text + link cards. New dependency + **paid/free-tier API key**. If we skip it, P1 still works with text/sources + optional iframe.
- **Headless browser (alt to Firecrawl):** Playwright/Puppeteer would render pages locally but adds a heavy install and CPU load on this box — not recommended unless screenshots are a hard requirement.
- **CC0 t-shirt 3D model (P4):** a free asset to source and vendor; no license cost, just needs picking.
- No other paid services required. P2/P4 add **no** local compute cost of note.

---

## What was NOT done here (per instructions)
No shipped code was edited, no server/fleet restart, no Codex/Cargo version bump. This document is discovery + plan only. Building begins only after the owner confirms scope (especially: Firecrawl yes/no for P1, and image-gen quota/paid tier for P3).