feat: speak with the macOS voices when Fish Audio is not configured - #27
Open
luca-71 wants to merge 8 commits into
Open
feat: speak with the macOS voices when Fish Audio is not configured#27luca-71 wants to merge 8 commits into
luca-71 wants to merge 8 commits into
Conversation
npm audit fix, no --force needed: vite 6.4.1 -> 6.4.3, plus transitive bumps to postcss, nanoid, and picomatch. All within the ranges package.json already declares, so only the lockfile moves. The one that mattered here is the Vite dev server's arbitrary file read via WebSocket (GHSA-p9ff-h696-f583) — this project's documented workflow runs that dev server. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Nine call sites sent JARVIS's reply only inside `if audio:`, so whenever
Fish Audio was unconfigured, rate-limited, or failing, the reply was
dropped with no trace in the UI. The startup greeting and every proactive
notification — research complete, build finished, project connected — went
straight to nothing. Each site now falls back to a "text" message, which
the client can surface.
Three of those sites also passed the raw `bytes` from synthesize_speech
straight to send_json as `"data": audio`, without base64. Bytes are not
JSON-serializable, so the send raised — and in two cases the exception was
swallowed by a bare `except Exception: pass`. Those paths ("Fix Yourself"
and the calendar/mail lookups) delivered neither audio nor text even with
a working Fish key, and did so silently.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
The client's stated "text fallback when TTS fails" only ran console.log, so a reply that arrived without audio reached the devtools console and nowhere else. With no Fish Audio key the interface was not merely silent but blank — the orb reacted to nothing and said nothing. Add a caption below the orb, in the existing palette. It is shown whether or not audio arrived, so it doubles as subtitles for a working voice. It holds for 4s plus 60ms per character, capped at 20s. Reading time rather than a fixed delay, because without a voice the caption is the entire response and vanishing mid-sentence loses it. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
recognition.lang was hardcoded to en-US, so anyone speaking another
language was transcribed phonetically into English nonsense — "come stai"
arrived as "comic Style". Add a Spoken Language selector to the settings
panel, persisted as SPEECH_LANG and read live, so the choice applies
without a restart.
voice.ts rebuilds the recognition object on a language change rather than
reassigning .lang. Chrome reads that property at construction and ignores
it on a session that has already run, so the reassignment silently kept
the old language. The retired session's onend checks whether it has been
superseded, or it would restart and race the new one for the microphone.
main.ts resolves the language before opening the microphone, so the first
session is built correctly and the fragile switch path is reserved for
live changes from the panel.
The system prompt is English and would answer an Italian question in
English, so a non-English selection appends a language directive. Kept to
a bare directive deliberately: wordings that also discussed action
selection measurably weakened the language adherence they were added to
enforce.
Known limitation: action routing is less reliable outside English. On one
repeated question, English chose the right action 4/4 while Italian
misrouted to a screen capture 4/4. The 46 hardcoded English strings in
server.py ("Right away, sir.") also stay English — translating them is an
i18n project, not a setting.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Chrome closes a recognition segment at every pause, and the client sent each "final" result the instant it arrived. A sentence spoken with any hesitation reached JARVIS as several questions: he answered half a thought, then received the rest as a new one. The logs show single words like "hey" arriving alone. Collect fragments and send once the speaker has actually stopped, after a 1s gap. Two edges that would otherwise bite: - A 6s ceiling. Steady dictation never produces the gap, so without a cap the timer re-arms forever and the reply never comes. - Muting discards what is pending, so half a sentence cannot arrive a second after the user silenced him. Barge-in stays immediate: the audio is cut on the first fragment, not at flush, since interrupting is what the user wanted the moment they spoke. Costs 1s before JARVIS starts thinking. That is the price of knowing the sentence ended; UTTERANCE_GAP_MS trades it back for more split sentences. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Three complaints from one Italian session, three separate causes.
**He was called the wrong thing.** HONORIFIC was written to .env, served by
the preferences API and offered in the settings panel — and never read.
The prompt hardcoded `Address {user_name} as "sir"`, so the dropdown did
nothing. Worse, the model translated that "sir" freely and settled on
"signora": it addressed him with a feminine honorific, five times out of
five, while his configuration said otherwise.
The honorific is now a prompt variable, and the language directive states
it must be reproduced letter for letter rather than translated or
inflected. Placing that in the language block matters — the same
instruction in the personality section was ignored 5/5, and obeyed 5/5 at
the end. Measured, both ways round: "sir" and "signore" now each hold 5/5.
The panel field becomes free text with suggestions. It offered sir, ma'am
and none, so "signore" was not expressible in the first place.
**The greeting arrived in English.** It was hardcoded, so every Italian
session opened with "Good evening, sir." and switched language on the next
sentence. It is the one line a user hears every single time. Now built
from a small table for the languages the panel offers, falling back to
English.
**It could not hear its own name.** An Italian recogniser mangles an
English name: the logs show "arbiss", "e gli arbis" and "hey Yaris" where
he plainly said JARVIS. Added as corrections, taken from the transcripts
rather than invented.
"yaris" is handled only after a term of address. It is also a very common
Toyota, and a bare rule rewrote "la mia auto Yaris" into a summons.
Verified in both directions, including that "arbitro", "arbitri" and
"arbitrario" survive untouched.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Without a Fish Audio key JARVIS was mute — the whole voice half of a voice
assistant, gated behind a paid signup. macOS already ships voices for every
language the panel offers, so it can talk on a fresh install instead.
TTS_BACKEND picks the synthesiser: auto (default) uses Fish when its key is
set and the macOS voices otherwise, and fish/macos/none force the choice.
MACOS_VOICE names a voice; empty resolves the best installed one for the
configured language. The panel gains both, plus a preview button — choosing
blind from seventy-odd voices is not a choice.
Voices are discovered from `say -v ?` rather than hardcoded, since what is
installed varies by machine and macOS version. Output is WAV: the `say`
default is AIFF-C, which Chrome does not reliably decode.
Two defects found by testing rather than by reading:
The voice list parser keyed on the run of spaces before the locale tag, so
it saw 73 voices and one Italian. Classic voices are padded into a column
("Alice it_IT #"), newer ones carry a parenthesised language and
get a single space ("Eddy (Italiano (Italia)) it_IT #"). Anchored on the
"#" instead: 183 voices, nine Italian.
"auto" also resolved to Fish on an untouched install, because the
placeholder in .env.example is a non-empty string and passed a truth test.
Every line went to Fish for a 401 rather than falling back — precisely the
case this feature exists for. Both that check and the settings panel's now
share one _fish_configured helper.
`say` interprets [[...]] as embedded commands, [[volm 0]] among them, so
those are stripped: the text arrives from a speech transcript by way of an
LLM. Arguments go straight to exec, never a shell.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
This was referenced Aug 11, 2026
Every reply crosses the WebSocket as base64, and WAV made that costly: a work-mode summary produced a 1.4MB frame, which exceeds the 1MB default frame limit in several WebSocket clients and closes the connection with 1009 'message too big'. say can emit AAC in an MP4 container, which every browser that runs this app decodes. Measured on the same spoken sentence: 91KB against 616KB — 119KB over the socket instead of 802KB. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Without a Fish Audio key JARVIS is mute. That is the entire voice half of a voice assistant, gated behind a paid signup — and the README's own quick start leaves you there until you have registered.
macOS already ships voices for every language the settings panel offers. This uses them, so a fresh clone talks.
What it adds
TTS_BACKENDpicks the synthesiser:auto(default)fishmacosnoneMACOS_VOICEnames a voice; left empty it resolves the best installed one for the configured language, preferring Daniel for English since the British voice suits the butler better than the American default.The settings panel gains both, plus a preview button — picking blind from seventy-odd voices is not really a choice. Voices for the current language are listed first, with a marker where the rest begin.
Voices are discovered from
say -v ?at runtime rather than hardcoded: which ones are installed varies by machine and by macOS version, and a hardcoded list would be wrong on somebody else's Mac. Output is WAV, because thesaydefault is AIFF-C and Chrome does not reliably decode it.Two defects that testing found and reading would not have
The voice list parser dropped most voices. It keyed on the run of spaces before the locale tag, and reported 73 voices with exactly one Italian. Classic voices are padded into a column, newer ones carry a parenthesised language in the name and get a single space:
Anchored on the
#instead. Now 183 voices, nine of them Italian.autoresolved to Fish on an untouched install. The placeholder in.env.exampleis a non-empty string, so it passed a plain truth test and counted as configured. Every line went to Fish for a 401 instead of falling back — precisely the case the feature exists for. The settings endpoint already compared against the placeholder; both now share one_fish_configuredhelper.Safety
sayinterprets[[...]]as embedded speech commands —[[volm 0]]silences it, among others. The text reaching the synthesiser comes from a speech transcript by way of an LLM, so those are stripped rather than trusted. Arguments are passed straight toexec, never through a shell.Verified
End to end with no Fish key configured, over the real WebSocket:
Language resolution checked across
it-IT→ Alice,en-US→ Daniel,fr-FR→ Thomas,ja-JP→ Kyoko,es-ES→ Mónica, and an unknown tag → no voice. An explicitly configured voice overrides all of it, including one from another language.Testing
pytest tests/gives 35 passed, 8 failed — identical to this branch's base, so no regressions. The 8 pre-existing failures are unrelated: 7 needplaywright install, andtest_browse_action_keywordsimportsACTION_KEYWORDS, a symbol that no longer exists inserver.py.tsc --noEmitclean.The quality is plainer than the Fish JARVIS model. The trade is that it costs nothing, needs no network, and works the moment you clone the repo.
🤖 Generated with Claude Code