fix: speak the configured language when relaying Claude Code - #30
Open
luca-71 wants to merge 12 commits into
Open
fix: speak the configured language when relaying Claude Code#30luca-71 wants to merge 12 commits into
luca-71 wants to merge 12 commits into
Conversation
npm audit fix, no --force needed: vite 6.4.1 -> 6.4.3, plus transitive bumps to postcss, nanoid, and picomatch. All within the ranges package.json already declares, so only the lockfile moves. The one that mattered here is the Vite dev server's arbitrary file read via WebSocket (GHSA-p9ff-h696-f583) — this project's documented workflow runs that dev server. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Nine call sites sent JARVIS's reply only inside `if audio:`, so whenever
Fish Audio was unconfigured, rate-limited, or failing, the reply was
dropped with no trace in the UI. The startup greeting and every proactive
notification — research complete, build finished, project connected — went
straight to nothing. Each site now falls back to a "text" message, which
the client can surface.
Three of those sites also passed the raw `bytes` from synthesize_speech
straight to send_json as `"data": audio`, without base64. Bytes are not
JSON-serializable, so the send raised — and in two cases the exception was
swallowed by a bare `except Exception: pass`. Those paths ("Fix Yourself"
and the calendar/mail lookups) delivered neither audio nor text even with
a working Fish key, and did so silently.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
The client's stated "text fallback when TTS fails" only ran console.log, so a reply that arrived without audio reached the devtools console and nowhere else. With no Fish Audio key the interface was not merely silent but blank — the orb reacted to nothing and said nothing. Add a caption below the orb, in the existing palette. It is shown whether or not audio arrived, so it doubles as subtitles for a working voice. It holds for 4s plus 60ms per character, capped at 20s. Reading time rather than a fixed delay, because without a voice the caption is the entire response and vanishing mid-sentence loses it. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
recognition.lang was hardcoded to en-US, so anyone speaking another
language was transcribed phonetically into English nonsense — "come stai"
arrived as "comic Style". Add a Spoken Language selector to the settings
panel, persisted as SPEECH_LANG and read live, so the choice applies
without a restart.
voice.ts rebuilds the recognition object on a language change rather than
reassigning .lang. Chrome reads that property at construction and ignores
it on a session that has already run, so the reassignment silently kept
the old language. The retired session's onend checks whether it has been
superseded, or it would restart and race the new one for the microphone.
main.ts resolves the language before opening the microphone, so the first
session is built correctly and the fragile switch path is reserved for
live changes from the panel.
The system prompt is English and would answer an Italian question in
English, so a non-English selection appends a language directive. Kept to
a bare directive deliberately: wordings that also discussed action
selection measurably weakened the language adherence they were added to
enforce.
Known limitation: action routing is less reliable outside English. On one
repeated question, English chose the right action 4/4 while Italian
misrouted to a screen capture 4/4. The 46 hardcoded English strings in
server.py ("Right away, sir.") also stay English — translating them is an
i18n project, not a setting.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Chrome closes a recognition segment at every pause, and the client sent each "final" result the instant it arrived. A sentence spoken with any hesitation reached JARVIS as several questions: he answered half a thought, then received the rest as a new one. The logs show single words like "hey" arriving alone. Collect fragments and send once the speaker has actually stopped, after a 1s gap. Two edges that would otherwise bite: - A 6s ceiling. Steady dictation never produces the gap, so without a cap the timer re-arms forever and the reply never comes. - Muting discards what is pending, so half a sentence cannot arrive a second after the user silenced him. Barge-in stays immediate: the audio is cut on the first fragment, not at flush, since interrupting is what the user wanted the moment they spoke. Costs 1s before JARVIS starts thinking. That is the price of knowing the sentence ended; UTTERANCE_GAP_MS trades it back for more split sentences. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Three complaints from one Italian session, three separate causes.
**He was called the wrong thing.** HONORIFIC was written to .env, served by
the preferences API and offered in the settings panel — and never read.
The prompt hardcoded `Address {user_name} as "sir"`, so the dropdown did
nothing. Worse, the model translated that "sir" freely and settled on
"signora": it addressed him with a feminine honorific, five times out of
five, while his configuration said otherwise.
The honorific is now a prompt variable, and the language directive states
it must be reproduced letter for letter rather than translated or
inflected. Placing that in the language block matters — the same
instruction in the personality section was ignored 5/5, and obeyed 5/5 at
the end. Measured, both ways round: "sir" and "signore" now each hold 5/5.
The panel field becomes free text with suggestions. It offered sir, ma'am
and none, so "signore" was not expressible in the first place.
**The greeting arrived in English.** It was hardcoded, so every Italian
session opened with "Good evening, sir." and switched language on the next
sentence. It is the one line a user hears every single time. Now built
from a small table for the languages the panel offers, falling back to
English.
**It could not hear its own name.** An Italian recogniser mangles an
English name: the logs show "arbiss", "e gli arbis" and "hey Yaris" where
he plainly said JARVIS. Added as corrections, taken from the transcripts
rather than invented.
"yaris" is handled only after a term of address. It is also a very common
Toyota, and a bare rule rewrote "la mia auto Yaris" into a summons.
Verified in both directions, including that "arbitro", "arbitri" and
"arbitrario" survive untouched.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Without a Fish Audio key JARVIS was mute — the whole voice half of a voice
assistant, gated behind a paid signup. macOS already ships voices for every
language the panel offers, so it can talk on a fresh install instead.
TTS_BACKEND picks the synthesiser: auto (default) uses Fish when its key is
set and the macOS voices otherwise, and fish/macos/none force the choice.
MACOS_VOICE names a voice; empty resolves the best installed one for the
configured language. The panel gains both, plus a preview button — choosing
blind from seventy-odd voices is not a choice.
Voices are discovered from `say -v ?` rather than hardcoded, since what is
installed varies by machine and macOS version. Output is WAV: the `say`
default is AIFF-C, which Chrome does not reliably decode.
Two defects found by testing rather than by reading:
The voice list parser keyed on the run of spaces before the locale tag, so
it saw 73 voices and one Italian. Classic voices are padded into a column
("Alice it_IT #"), newer ones carry a parenthesised language and
get a single space ("Eddy (Italiano (Italia)) it_IT #"). Anchored on the
"#" instead: 183 voices, nine Italian.
"auto" also resolved to Fish on an untouched install, because the
placeholder in .env.example is a non-empty string and passed a truth test.
Every line went to Fish for a 401 rather than falling back — precisely the
case this feature exists for. Both that check and the settings panel's now
share one _fish_configured helper.
`say` interprets [[...]] as embedded commands, [[volm 0]] among them, so
those are stripped: the text arrives from a speech transcript by way of an
LLM. Arguments go straight to exec, never a shell.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Acknowledgements, lookup results and error lines are built in Python, not written by the model, so they stayed English however the language was set. A single reply arrived as "Checking your calendar now, sir." followed by fluent Italian — the language setting looked half-finished because half of what JARVIS says never passed through it. Rather than ship hand-written translations for ten languages — inventing Japanese butler register is not something to do blind — the English catalogue is translated once per language by the model JARVIS already talks through, and cached on disk under data/. One Haiku call per language, ever. The cache is keyed by a hash of the catalogue, so editing a phrase re-translates instead of serving a stale set. Placeholders are named, never positional, because word order is exactly what changes between languages; the translator is told to reorder them freely but never rename or drop one. Translations are validated on the way out and again on the way in. A phrase that lost a placeholder does not raise — str.format ignores surplus arguments — so it would quietly drop the project name or the message count rather than fail visibly. Validating only at translation time would have left a hand-edited or truncated cache to do that silently, so load re-checks and falls back to English per phrase. Warm-up is backgrounded and English is used until it lands: a cold cache must not delay the first answer, and English is what that answer would have been anyway. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
data/phrases-<lang>.json is written at runtime by the translator and is keyed by a hash of the catalogue, so a committed copy would be both machine-specific and stale the moment a phrase changes. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
handle_research was pinned to claude-opus-4-6, three Opus generations behind. Swapping the model string alone would have broken it. Thinking is on by default on Opus 5, unlike 4.6 where omitting the parameter meant no thinking, so the response now arrives as ['thinking', 'text'] and content[0].text raises AttributeError — a ThinkingBlock has no .text. Verified against the live API rather than inferred. It now joins the text blocks and ignores the reasoning. max_tokens covers thinking and answer together on Opus 5, so 2000 — sized for prose on a non-thinking model — truncated the report. Raised to 16000, the ceiling before SDK HTTP timeouts constrain a non-streaming request. Opus 5's classifiers can decline a request, returning a successful HTTP 200 with stop_reason "refusal" and empty content; indexing content[0] would crash instead of reporting back. Checked first, and fallbacks="default" lets the API re-run a declined request elsewhere, routing by refusal category rather than pinning a model that will itself need migrating. Note handle_research has no callers: [ACTION:RESEARCH] dispatches to _execute_research, which spawns Claude Code. This changes no runtime behaviour today — it fixes code that would fail the moment anything reached it. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
An Italian session turned English the moment it entered work mode: 18:34:17 JARVIS: Ricerco Aspergillus per voi, sir. 18:35:36 Work mode -> claude -p: "sei ancora ricercando" 18:35:56 Claude Code response (756 chars) 18:35:58 JARVIS: Luca, I've just checked your workspace and confirmed... Three Haiku calls summarise Claude Code's output into something JARVIS can speak — prompted projects, self-work, and work mode. Each was handed English subprocess output with an English-only system prompt and no language directive, so it answered in English, which is the sensible reading of what it was given. This is a gap neither earlier change covered: the conversation gets its language from the system prompt in generate_response, and the fixed phrases come from the catalogue — but text relayed from a subprocess passes through neither. Add a shared _language_directive() and append it to all three prompts. It returns an empty string for English, so an English configuration renders byte-identical prompts. One helper rather than three copies, so the next summariser doesn't have to rediscover this. Verified on the same English input: Italian gets "Ho verificato il vostro workspace...", English is unchanged, and the honorific survives verbatim in both. Known limit: each summariser's except-branch passes Claude Code's raw output through when the Haiku call fails, so a failure there still surfaces English. Translating it would need the call that just failed. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Exercising work mode in Italian cut a summary off mid-word:
"...lo storico su 120 test ha escluso vantaggi statistici a 3+ sigma. I"
max_tokens on the three spoken summarisers was tuned against English
output. The same two sentences cost noticeably more tokens in Italian, so
a ceiling that fit in English truncates elsewhere — a limit this change
exposed by making those summaries non-English in the first place.
Raised to 400 and 250. The prompts already cap length in words ('2-3
sentences max'), so this is only the safety net behind them; the classifier
at line 826 is untouched, as it returns JSON rather than speech.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
An Italian session switched to English the moment it entered work mode, and stayed there.
What the log showed
Conversation is fine. The English begins one line after
claude -preturns.Why
Three Haiku calls summarise Claude Code's output into something JARVIS can speak — one for prompted projects, one for self-work, one for work mode. Each is handed English subprocess output with an English-only system prompt and no language directive: the main conversation gets its language from the system prompt in
generate_response, and these never pass through it.So the model does the sensible thing with what it was given, and answers in English. This is a distinct gap from #22 (conversation language) and #28 (fixed phrases) — those cover text JARVIS composes and text the repo hardcodes; this is text relayed from a subprocess, which neither touches.
The fix
A shared
_language_directive()appended to all three system prompts. It reuses the configuredSPEECH_LANGand the honorific from #26, and returns empty string for English — so an English configuration renders byte-identical prompts and nothing changes for the default.One helper rather than three copies, because the next summariser added should not have to rediscover this.
Verified
The same English input, summarised under each configuration:
it-IT— "Ho verificato il vostro workspace e confermato che non vi è nulla in esecuzione in background, sir. La cartella Aspergillus è completamente vuota…"en-US— "I've confirmed your Aspergillus workspace is entirely clear with no background processes or previous files…"Same source material, correct language each time, honorific preserved verbatim in both. Directive generation also checked across
it-IT,fr-FR(which correctly picks upmonsieur), anden-US(empty).Known limit
The
exceptfallback on each summariser passes Claude Code's raw output straight through when the Haiku call fails, so a failure there still surfaces English. Translating it would need the API call that just failed — the alternative is silence, which is worse.Testing
pytest tests/gives 35 passed, 8 failed — identical to this branch's base, so no regressions. The 8 pre-existing failures are unrelated: 7 needplaywright install, andtest_browse_action_keywordsimportsACTION_KEYWORDS, a symbol that no longer exists inserver.py.🤖 Generated with Claude Code