# PERSON-RECOGNITION-GOAL — Shared Face + Profile Memory for the KAI Fleet **Status:** PLAN / DESIGN + Phase-1 scaffold shipped behind a flag (default OFF). **Needs the owner's explicit go-ahead on the approach before any wiring into live agents** — this stores biometric face data of real people. **Date:** 2026-07-10 **Author track:** kai-engineer **Flag:** `KAI_FACE_RECOGNITION` (env, default OFF). Nothing in this subsystem runs unless it is `=1`. **Companion flag it rides on:** `KAI_VIDEO_MODE` (the camera path added to `oracle.html` / the Gemini-Live bridge on 2026-07-10, also default OFF). --- ## 0. Owner's vision (verbatim intent) The agents (Leo, KAI, etc.) should **recognize people**. When Leo sees the owner (via the call camera feed / any camera), he **learns that face**, and next time he sees them — anywhere — he knows it's them. Same for anyone he sees or talks with: he builds a **profile** of that person (who they are, what they talked about, memories made together) to know them better over time. This profile/memory is **shared across all agents** — any agent can access it and explore the memories they made with that person. And when multiple agents see/recognize the same person, they **confirm identity AI-to-AI**, which raises a **confidence score** against that face+person so recognition gets better over time. --- ## 1. Verdict up front This is **feasible and, importantly, it slots into infrastructure that already exists** — we are not building person-memory from scratch, we are adding a **face** modality to a person-memory system the fleet already runs. Three facts make this low-risk to design (higher-risk to *enable* — see Privacy, §7): 1. **There is already a biometric enrollment pattern to mirror.** `shared/voice-biometrics.mjs` + `state/biometric_profiles.json` + per-person `.npy` "DNA" files in `state/dna_signatures/` + a Python helper `shared/vocal_dna.py` (librosa → ~60-dim L2-normalized voiceprint, cosine match, 1-to-many identify). **Face recognition is the same shape with a different embedder.** We copy the proven pattern rather than invent one. 2. **There is already a shared, cross-agent person store.** Every agent (Leo, KAI, Groq, Gemini, Claudey, X, Oracle — separate Node processes) already reads/writes the same on-disk stores: `transcripts.db` (`user_profile_memories`, `entity_facts`, `user_facts`, `user_fingerprints`, `user_profiles`) via `shared/transcript-memory.mjs`, and the structured fact layer `shared/user-warehouse.mjs` (keyed by **Discord user ID**, alias-resolved). "Shared between all for memory" is already how the fleet works — we attach faces to the *same person key*. 3. **There is already a camera frame source.** As of 2026-07-10, `oracle.html` VIDEO MODE v1 captures the call camera and sends `{type:'frame', data:}` every ~2s over the existing `/ws/voice` socket → server → `OracleLiveVoiceSession.sendFrame()` → `gemini-live-bridge.sendFrame()` (`realtimeInput.video`), all gated by `KAI_VIDEO_MODE` (default OFF). That server-side frame choke point is exactly where a face-ingest hook belongs — one place, one guarded call. **The unifying design decision:** do **not** create a parallel identity namespace. Key face profiles by the **same canonical person id** the fleet already uses (Discord user id, alias-resolved via `user-warehouse.resolveAlias`). The moment a face is tied to person `P`, that face is automatically linked to `P`'s voice DNA, `P`'s `user_facts`, and `P`'s transcript memories — i.e. it is *already* "shared across all agents and all the memories they made with that person," for free. --- ## 2. Architecture ``` ┌──────────────────────── CAPTURE (browser, KAI_VIDEO_MODE) ────────────────────────┐ │ oracle.html getUserMedia(video) → → JPEG │ │ every ~2s: ws.send({type:'frame', data:}) over /ws/voice │ └───────────────────────────────────────┬───────────────────────────────────────────┘ ▼ ┌──────────────────────── SERVER frame choke point ────────────────────────────────┐ │ command-center-server.mjs ws 'frame' handler → OracleLiveVoiceSession.sendFrame │ │ │ │ ── PHASE-2 HOOK (guarded, no-op when flag OFF) ── │ │ if (KAI_FACE_RECOGNITION==='1') personRecognition.ingestFrame(base64Jpeg, { │ │ agent: session.botName, channelId, sessionUserId }) // fire-and-forget │ └───────────────────────────────────────┬───────────────────────────────────────────┘ ▼ ┌──────────────────── FACE PIPELINE (shared/person-recognition.mjs) ────────────────┐ │ 1. detect + embed: face_dna.py --embed → 512-d L2-normed vector (+ bbox, │ │ det-score). 0 faces → drop. >1 → take most-central/largest. │ │ 2. match: cosine vs every known person's stored embeddings │ │ best ≥ MATCH_HI → recognized(personId) │ │ MATCH_LO..HI → tentative (needs corroboration) │ │ < MATCH_LO → unknown → provisional person (unlabeled) │ │ 3. record sighting: append {ts, agent, channelId, cos, det} to person.sightings; │ │ store the embedding (cap N per person, keep most diverse) │ │ 4. confidence: recompute person.confidence from #embeddings, #sightings, │ │ #distinct agents, and cross-agent confirmations (§4) │ └───────────────────────────────────────┬───────────────────────────────────────────┘ ▼ ┌──────────────────── SHARED PERSON STORE (all agents read/write) ──────────────────┐ │ state/face_profiles.json ← canonical per-person face record (§3) │ │ state/face_signatures// ← that person's embedding vectors (.npy / .json) │ │ keyed by the SAME person id as user-warehouse / transcripts / voice DNA │ └───────────────────────────────────────┬───────────────────────────────────────────┘ ▼ ┌──────────────────── RETRIEVAL (mid-conversation, §5) ─────────────────────────────┐ │ leo.mjs / kai.mjs / oracle-live-voice: │ │ const who = personRecognition.whoIsOnCamera(channelId) │ │ → {personId, name, confidence, lastSeen, sharedNotes, recentMemories} │ │ → injected into the system/context so the agent greets by name + references │ │ past memories ("good to see you again, Ryan — last time we talked about …") │ └────────────────────────────────────────────────────────────────────────────────────┘ ``` **Face capture → detection + embedding → person store → match by similarity → confidence that rises with sightings and cross-agent confirmations.** Each stage above maps 1:1 to that requirement. --- ## 3. The shared person-profile store **One canonical record per person, keyed by the fleet's existing person id.** New file `state/face_profiles.json` (mirrors `state/biometric_profiles.json`), with heavy embedding vectors offloaded to `state/face_signatures//` (mirrors `state/dna_signatures/`). ```jsonc // state/face_profiles.json { "schemaVersion": 1, "people": { "": { // SAME id as user-warehouse / transcripts (Discord snowflake, // or "prov:" for an as-yet-unlabeled stranger) "personId": "", "displayName": "Ryan", // null until known / labeled by an agent or the owner "handles": ["nastermodx"], // known aliases (kept in sync w/ user-warehouse.resolveAlias) "provisional": false, // true = a stranger we can re-recognize but haven't named "firstSeen": "2026-07-10T19:50:15.514Z", "lastSeen": "2026-07-10T20:14:02.010Z", "embeddings": [ // vectors live in face_signatures//; this is the manifest { "file": "0001.npy", "dim": 512, "model": "insightface:buffalo_l", "capturedAt": "…", "agent": "Leo", "detScore": 0.94 } ], "sightings": [ // capped ring buffer (e.g. last 200), pruned by retention { "ts": "…", "agent": "Leo", "channelId": "…", "cos": 0.71, "detScore": 0.94 } ], "confidence": 0.0, // 0..1, see §4 (identity confidence for this face↔person link) "crossAgentConfirmations": [ // AI-to-AI agreements that raised confidence (§4) { "ts": "…", "byAgent": "KAI", "matchedPersonId": "", "cos": 0.69, "delta": +0.05 } ], "notes": { "shared": [ // notes any agent can read/append — "shared between all" { "ts": "…", "agent": "Leo", "text": "Owner. Prefers to be called Ryan." } ], "perAgent": { // an agent's own private impressions (still readable by all) "Leo": [ { "ts": "…", "text": "Talked about the KAIVERSE terminator work." } ] } } } } } ``` **Storage decision — JSON registry + `.npy` sidecars, NOT a new SQLite table. Justification:** - **Consistency beats novelty.** The fleet's *biometric* precedent is already JSON-registry + per-person `.npy` (`biometric_profiles.json` + `dna_signatures/`). Mirroring it means the face path reads exactly like the voice path — same mental model, same enrollment ergonomics, same backup story. - **Multi-process safe with a well-worn pattern.** Agents are separate Node processes; JSON files are the fleet's established cross-process shared medium (the whole `state/` "shoebox"). Writes use **temp-file + atomic rename** (the pattern already used in `shared/metrics-store.mjs`, `kaiverse-life.mjs`, etc.) to survive a crash mid-write. Concurrency is low (a sighting every ~2s per active call), so a short advisory lock + last-writer-merge is sufficient; no DB contention. - **Embeddings don't belong in a text row.** 512 floats × N per person is offloaded to `.npy` sidecars exactly like voice DNA, keeping the JSON small and diffable. - **The link, not a copy.** Face profiles reference the person by id; the rich *interaction* memory (what you talked about) already lives in `transcripts.db` / `user_facts` keyed by that id. We do not duplicate it — retrieval (§5) joins on the id. This is what makes the profile "shared across all agents" without any new sharing mechanism. *(If volume ever grows past what JSON comfortably holds — thousands of people, millions of sightings — the migration path is a `face_profiles`/`face_embeddings` table inside `transcripts.db` alongside `user_profile_memories`. Deferred; not needed for Phase 1–3.)* --- ## 4. Cross-agent confirmation protocol (AI-to-AI, raises confidence) Agents don't message each other synchronously about a face; they **corroborate through the shared store**, which is exactly how the fleet already coordinates. The confidence on a face↔person link is a function of independent agreement: ``` confidence = clamp01( 0.25 * f(numEmbeddings) // more enrolled views of this face + 0.25 * f(numSightings) // seen more often + 0.30 * f(numDistinctAgents) // ← the cross-agent term: how many DIFFERENT agents agreed + 0.20 * meanMatchCosineAboveHi ) // how strong the matches were ``` - **Agreement bumps.** When agent B ingests a frame and its best match is person `P` (cos ≥ MATCH_HI), and agent A had already labeled/recognized `P`, B writes a `crossAgentConfirmation` for `P`. A confirmation **from a distinct agent** applies a positive `delta` (with diminishing returns — the 5th agent to agree adds less than the 2nd). This is the "multiple agents see the same person → confidence goes up" mechanic. - **Disagreement penalizes.** If B's frame matches `P` strongly *and* `Q` non-trivially (an ambiguous face), or two agents attach conflicting `displayName`s to the same embedding cluster, confidence is **lowered** and the record is flagged `needsReview` rather than silently guessing. Better to be unsure than confidently wrong about who someone is. - **Provisional → named promotion.** A stranger starts as `prov:` with a low confidence. Once an agent (or the owner) attaches a real name/handle, the provisional record is merged into the canonical person id (embeddings + sightings carried over), and future cross-agent hits accrue to the named person. - **Thresholds are config, not magic** (`MATCH_HI`, `MATCH_LO`, per-agent decay, max embeddings/person) — tuned during Phase 2 with the owner, the same eyeball-iterate loop the visual work uses. --- ## 5. Retrieval — an agent greeting a recognized person mid-conversation The retrieval surface is a single read the voice/text paths can call cheaply: ```js // shared/person-recognition.mjs whoIsOnCamera(channelId) -> null | { personId, name, provisional, confidence, lastSeen, sharedNotes: [...], // notes.shared, newest first recentMemories:[...] // JOINED from the shared stores by personId: } ``` `recentMemories` is assembled by joining on the person id into the memory the fleet **already** keeps: `user-warehouse.listFacts(personId)` (home, preferences, relationships) + a recall query against `transcript-memory` (`user_profile_memories` / `transcript_fts`) for recent topics with that person. Nothing new is stored to answer "what did we talk about" — it's already there under the same key. **Hook points (all Phase 2+, guarded, additive — never on the audio pacer path):** - **Voice path:** in `oracle-live-voice.mjs` / `leo.mjs`, when a call starts or a face is first recognized on a channel, fetch `whoIsOnCamera(channelId)` and prepend a short, budgeted context line to the model turn: *"You can see Ryan on camera (confidence 0.82). Recent: KAIVERSE terminator work, prefers 'Ryan'."* The agent then naturally greets by name / references the memory. **This is a system-context injection only — it does not touch `leoPacedFeed`, the throttle, or the audio timing.** - **Text path / Discord:** the same `whoIsOnCamera`/`whoIs(personId)` read can enrich a reply when a recognized person is present. Optional, later. - **KAI (the lattice):** the sighting + name can be mirrored to the RSHL lattice via the existing `lattice-bridge` (best-effort, non-blocking) so KAI's associative recall also "knows the face," consistent with how user facts are already mirrored. --- ## 6. Face-detection + embedding approach (what's actually feasible on this stack) **Context:** the box is a limited Windows laptop (RTX 4050, but treat CPU as the floor). The existing biometric path already shells out to **Python** (`vocal_dna.py`, via `execSync('python …')`) using `numpy`/`librosa`/`scipy`. So a Python face helper is the *native* fit — same enrollment ergonomics, same `.npy` artifacts, same call pattern. **Options considered:** | Approach | Embed model | Where it runs | Pros | Cons | |---|---|---|---|---| | **A. Python `insightface` (ArcFace, `buffalo_l`) on `onnxruntime`** ← **recommended** | ArcFace, 512-d | Python helper `face_dna.py`, mirrors `vocal_dna.py` | CPU-feasible; SOTA-grade discrimination; `pip install insightface onnxruntime`; auto-downloads model; detection + embedding in one lib; produces `.npy` exactly like voice DNA | first-run model download (~300 MB); onnxruntime CPU install to verify on the box | | B. Python `face_recognition` (dlib) | 128-d | Python helper | very common, simple API | dlib build on Windows is painful (cmake/VS build tools); heavier to install | | C. Python `mediapipe` face embedder | ~192-d | Python helper | pip-installable, CPU-light, small model | slightly lower discrimination than ArcFace; API churn across versions | | D. `@vladmandic/face-api` (tfjs) in Node | 128-d | in-process in the fleet | no Python; keeps it all Node | `tfjs-node` native bindings finicky on Windows; adds a big dep to every bot | | E. **Browser-side** face-api.js / MediaPipe Tasks in `oracle.html` | 128–512-d | the browser, on the already-captured frame | **strong privacy win — raw face never has to leave the browser; only the embedding vector crosses the wire**; zero server model install | logic split across browser+server; only covers the dashboard camera path, not other cameras | **Recommendation:** **Option A (InsightFace ArcFace `buffalo_l`, 512-d, onnxruntime-CPU) as the server-side embedder**, because it is the most consistent with the proven `vocal_dna.py` pattern (Python helper → `.npy` → cosine match → 1-to-many identify), CPU-feasible, and the most discriminative of the easy-install options. **Strongly consider Option E as a Phase-3 privacy upgrade**: move embedding into the browser so raw biometric imagery never needs to reach the server for the dashboard camera — the server would then store only vectors. A + E compose (same 512-d space if the browser uses the same model family; otherwise keep them as separate model tags on the embeddings). **Phase-1 reality:** the scaffold ships `face_dna.py` as a **guarded stub** — it documents the InsightFace path and only attempts a real embed if the flag is on *and* the library imports successfully; otherwise it returns a clear "embedder-not-installed" status and the pipeline no-ops. **No model is downloaded and no dependency is installed until the owner approves the approach.** --- ## 7. PRIVACY / SAFETY (read this before enabling anything) **This subsystem stores biometric identifiers (face embeddings) of real people. Treat it accordingly.** - **Default OFF, owner-gated.** Everything is behind `KAI_FACE_RECOGNITION` (env, default OFF) and rides on `KAI_VIDEO_MODE` (also default OFF). With the flag unset, the module is inert: no capture, no embed, no store, no retrieval. Phase 1 changes **zero** live-agent behavior. - **Consent.** Enrollment of a *named* person should be a deliberate, consented act (the owner enrolling himself/known people), not silent background harvesting of every face on camera. Strangers may be stored only as **provisional, unlabeled** re-identification records, and only while the flag is on; naming a person is an explicit step. A consent/notice string should be surfaced in the dashboard when the camera + recognition are both on. *(Owner to confirm the consent UX before Phase 2.)* - **Fleet-only, never auto-shared.** Profiles live on-box under `state/` and are read only by the local fleet. **No face data, embedding, image, or profile is ever sent outside the fleet**, posted to Discord, put in a URL, or included in any external API payload beyond the already-consented Gemini-Live video frames that `KAI_VIDEO_MODE` itself sends. - **No faces in logs or URLs.** The pipeline logs only ids, cosine scores, and agent names — **never** base64 image data, never an embedding dump, never a face crop path in a URL. (Same discipline as the "no secrets in logs" rule.) - **No secrets.** Nothing here reads or emits `.env` values. - **Deletion / right to be forgotten.** A `forgetPerson(personId)` API removes the JSON record and the `.npy` sidecars (mirrors `voice-biometrics` deletion expectations). Retention prunes old sightings. - **Accuracy humility.** Face matching is probabilistic. The design *prefers `needsReview` over a confident wrong identification* (§4). An agent should never assert an identity below a comfortable confidence threshold — "I think this might be Ryan" vs. silently addressing the wrong person. - **This needs the owner's explicit go-ahead** on (a) the embedder choice and install, (b) the consent UX, and (c) enabling the flag, **before** any wiring into live agents. Flagged again in the report. --- ## 8. Phased plan **Phase 1 — Foundation (this deliverable; shipped, flag OFF, non-breaking).** - New self-contained module `shared/person-recognition.mjs`: the shared person-profile store (schema + atomic read/write API) and a `matchEmbedding` stub. All exports are **no-ops returning null/empty when `KAI_FACE_RECOGNITION!=='1'`**. - New `shared/face_dna.py`: guarded embedder stub (documents the InsightFace path; real embed only if lib present). - **Not imported by any shipped file yet** → zero effect on the running fleet. The exact one-line, guarded ingest hook and the retrieval hook are documented here for Phase 2 but **not** applied. **Phase 2 — Live recognition + cross-agent confirmation (needs owner go-ahead).** - Install/verify the embedder; replace the `face_dna.py` stub with the real InsightFace embed + the `matchEmbedding` cosine/identify implementation. - Apply the guarded ingest hook at the server frame choke point (`command-center-server.mjs` / `OracleLiveVoiceSession.sendFrame`) — fire-and-forget, no-op when flag OFF. - Implement the confidence model (§4) and cross-agent confirmation accrual. - Enrollment UX: owner enrolls himself / known people (mirrors `voice-biometrics` enrollment). **Phase 3 — Memory retrieval + agent greeting + privacy upgrade.** - Wire `whoIsOnCamera` retrieval into the voice/text context (system-context injection only; never the pacer). Agents greet by name and reference past memories joined from the existing shared stores. - Mirror sightings/names to the RSHL lattice via `lattice-bridge`. - Optional privacy upgrade: move embedding into the browser (Option E) so raw faces never reach the server for the dashboard camera path. **Phase 4 — Hardening.** - Retention/pruning of sightings + embedding-set curation (keep the most diverse N per person). - `needsReview` queue + owner tooling to merge/split/forget identities. - Liveness / anti-spoof consideration (a photo of a face vs. a live person), paralleling the voice liveness heuristic already in `vocal_dna.py`. --- ## 9. Files **Phase 1 (created):** - `shared/person-recognition.mjs` — shared person-profile store + gated read/write API + match stub. - `shared/face_dna.py` — guarded face-embedder stub (InsightFace path documented; inert without lib). - `state/face_profiles.json` — created lazily on first write (not committed with data). - `state/face_signatures/` — created lazily (per-person `.npy` sidecars). **Phase 2+ (documented, NOT yet edited):** - `command-center-server.mjs` — one guarded ingest line at the `type:'frame'` handler. - `shared/oracle-live-voice.mjs` / `bots/leo.mjs` / `bots/kai.mjs` — guarded retrieval context injection. --- ## 10. Open questions for the owner 1. **Approve InsightFace (ArcFace `buffalo_l`, onnxruntime-CPU) as the embedder?** (vs. dlib / mediapipe / browser-side.) This is the only real dependency install. 2. **Consent UX** for enrollment + a camera-on/recognition-on notice — what do you want it to say/show? 3. **Stranger policy:** store unknown faces as provisional re-id records while the flag is on, or only ever store people you explicitly enroll? 4. **Green light to wire the guarded hooks into live agents** once Phase 1 is reviewed? (Until then this subsystem is dormant.) ```