Voice UX: replies that arrive, in the language you speak

[luca-71]

Five changes from setting JARVIS up on a fresh Mac and trying to talk to him. Independent of #21 — this branch is cut from main.

Each commit stands alone and explains its own reasoning; the summary below is what they add up to.

The replies were not arriving

Nine call sites sent JARVIS's reply only inside if audio:. With no Fish Audio key — or a rate-limited one — the reply was dropped with no trace in the UI. The startup greeting and every proactive notification went straight to nothing.

Three of those sites also passed the raw bytes from synthesize_speech to send_json as "data": audio, without base64. Bytes are not JSON-serializable, so the send raised, and in two cases a bare except Exception: pass swallowed it. Those paths ("Fix Yourself", the calendar/mail lookups) delivered neither audio nor text even with a working Fish key, silently.

The client's stated "text fallback when TTS fails" then only ran console.log, so anything that did arrive reached the devtools console and nowhere else. There is now a caption below the orb, shown whether or not audio arrived, so it doubles as subtitles.

It could not understand anyone who is not American

recognition.lang was hardcoded to en-US. Spoken Italian came back as phonetic English nonsense — come stai arrived as comic Style. There is now a Spoken Language selector in the settings panel, persisted as SPEECH_LANG and read live.

Two things this required beyond a select element:

• Chrome reads lang at construction. Reassigning it on a session that has already run is silently ignored, so voice.ts rebuilds the recognition object. The retired session's onend checks whether it has been superseded, or it restarts and races the new one for the microphone.
• The system prompt is English and answered Italian questions in English, so a non-English selection appends a language directive.

Hesitation was read as a finished sentence

Chrome closes a recognition segment at every pause, and each "final" result was sent immediately. A sentence spoken with any hesitation reached JARVIS as several questions — he answered half a thought, then got the rest as a new one. The logs show single words like hey arriving alone.

Fragments are now collected and sent after a 1s gap, with a 6s ceiling (steady dictation never produces a gap, so an uncapped timer re-arms forever). Muting discards what is pending. Barge-in stays immediate — audio is cut on the first fragment, not at flush.

Dependencies

npm audit fix, no --force: vite 6.4.1 → 6.4.3 plus transitive bumps, all within the ranges package.json already declares. The one that mattered is the Vite dev server's arbitrary file read via WebSocket (GHSA-p9ff-h696-f583) — the documented workflow runs that dev server.

Known limitations, measured not guessed

• Action routing is less reliable outside English. On one repeated question, English picked the right action 4/4 while Italian misrouted to a screen capture 4/4. Three prompt wordings were tried; the one that halved the misrouting also broke language adherence on 2 of 4 replies, so the bare directive was kept. This is a weakness of the English prompt, not something a trailing sentence fixes reliably.
• 46 hardcoded English strings in server.py ("Right away, sir.") stay English in every language. Translating them is an i18n project, not a setting.
• Screen descriptions come back in English — they originate in a separate LLM call in screen.py that never sees the language directive.
• Fragment accumulation costs 1s before JARVIS starts thinking. That is the price of knowing the sentence ended; UTTERANCE_GAP_MS trades it back.

Testing

pytest tests/ gives 35 passed, 8 failed — identical to this branch's base, so no regressions. The 8 pre-existing failures are unrelated: 7 need playwright install, and test_browse_action_keywords imports ACTION_KEYWORDS, a symbol that no longer exists in server.py. tsc --noEmit is clean.

Verified by hand against a live server: language round-trips through the panel and applies without a restart, and replies arrive as captions with no Fish key configured.

🤖 Generated with Claude Code