fix: move deep research to Opus 5 - #29
Open
luca-71 wants to merge 10 commits into
Open
Conversation
npm audit fix, no --force needed: vite 6.4.1 -> 6.4.3, plus transitive bumps to postcss, nanoid, and picomatch. All within the ranges package.json already declares, so only the lockfile moves. The one that mattered here is the Vite dev server's arbitrary file read via WebSocket (GHSA-p9ff-h696-f583) — this project's documented workflow runs that dev server. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Nine call sites sent JARVIS's reply only inside `if audio:`, so whenever
Fish Audio was unconfigured, rate-limited, or failing, the reply was
dropped with no trace in the UI. The startup greeting and every proactive
notification — research complete, build finished, project connected — went
straight to nothing. Each site now falls back to a "text" message, which
the client can surface.
Three of those sites also passed the raw `bytes` from synthesize_speech
straight to send_json as `"data": audio`, without base64. Bytes are not
JSON-serializable, so the send raised — and in two cases the exception was
swallowed by a bare `except Exception: pass`. Those paths ("Fix Yourself"
and the calendar/mail lookups) delivered neither audio nor text even with
a working Fish key, and did so silently.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
The client's stated "text fallback when TTS fails" only ran console.log, so a reply that arrived without audio reached the devtools console and nowhere else. With no Fish Audio key the interface was not merely silent but blank — the orb reacted to nothing and said nothing. Add a caption below the orb, in the existing palette. It is shown whether or not audio arrived, so it doubles as subtitles for a working voice. It holds for 4s plus 60ms per character, capped at 20s. Reading time rather than a fixed delay, because without a voice the caption is the entire response and vanishing mid-sentence loses it. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
recognition.lang was hardcoded to en-US, so anyone speaking another
language was transcribed phonetically into English nonsense — "come stai"
arrived as "comic Style". Add a Spoken Language selector to the settings
panel, persisted as SPEECH_LANG and read live, so the choice applies
without a restart.
voice.ts rebuilds the recognition object on a language change rather than
reassigning .lang. Chrome reads that property at construction and ignores
it on a session that has already run, so the reassignment silently kept
the old language. The retired session's onend checks whether it has been
superseded, or it would restart and race the new one for the microphone.
main.ts resolves the language before opening the microphone, so the first
session is built correctly and the fragile switch path is reserved for
live changes from the panel.
The system prompt is English and would answer an Italian question in
English, so a non-English selection appends a language directive. Kept to
a bare directive deliberately: wordings that also discussed action
selection measurably weakened the language adherence they were added to
enforce.
Known limitation: action routing is less reliable outside English. On one
repeated question, English chose the right action 4/4 while Italian
misrouted to a screen capture 4/4. The 46 hardcoded English strings in
server.py ("Right away, sir.") also stay English — translating them is an
i18n project, not a setting.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Chrome closes a recognition segment at every pause, and the client sent each "final" result the instant it arrived. A sentence spoken with any hesitation reached JARVIS as several questions: he answered half a thought, then received the rest as a new one. The logs show single words like "hey" arriving alone. Collect fragments and send once the speaker has actually stopped, after a 1s gap. Two edges that would otherwise bite: - A 6s ceiling. Steady dictation never produces the gap, so without a cap the timer re-arms forever and the reply never comes. - Muting discards what is pending, so half a sentence cannot arrive a second after the user silenced him. Barge-in stays immediate: the audio is cut on the first fragment, not at flush, since interrupting is what the user wanted the moment they spoke. Costs 1s before JARVIS starts thinking. That is the price of knowing the sentence ended; UTTERANCE_GAP_MS trades it back for more split sentences. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Three complaints from one Italian session, three separate causes.
**He was called the wrong thing.** HONORIFIC was written to .env, served by
the preferences API and offered in the settings panel — and never read.
The prompt hardcoded `Address {user_name} as "sir"`, so the dropdown did
nothing. Worse, the model translated that "sir" freely and settled on
"signora": it addressed him with a feminine honorific, five times out of
five, while his configuration said otherwise.
The honorific is now a prompt variable, and the language directive states
it must be reproduced letter for letter rather than translated or
inflected. Placing that in the language block matters — the same
instruction in the personality section was ignored 5/5, and obeyed 5/5 at
the end. Measured, both ways round: "sir" and "signore" now each hold 5/5.
The panel field becomes free text with suggestions. It offered sir, ma'am
and none, so "signore" was not expressible in the first place.
**The greeting arrived in English.** It was hardcoded, so every Italian
session opened with "Good evening, sir." and switched language on the next
sentence. It is the one line a user hears every single time. Now built
from a small table for the languages the panel offers, falling back to
English.
**It could not hear its own name.** An Italian recogniser mangles an
English name: the logs show "arbiss", "e gli arbis" and "hey Yaris" where
he plainly said JARVIS. Added as corrections, taken from the transcripts
rather than invented.
"yaris" is handled only after a term of address. It is also a very common
Toyota, and a bare rule rewrote "la mia auto Yaris" into a summons.
Verified in both directions, including that "arbitro", "arbitri" and
"arbitrario" survive untouched.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Without a Fish Audio key JARVIS was mute — the whole voice half of a voice
assistant, gated behind a paid signup. macOS already ships voices for every
language the panel offers, so it can talk on a fresh install instead.
TTS_BACKEND picks the synthesiser: auto (default) uses Fish when its key is
set and the macOS voices otherwise, and fish/macos/none force the choice.
MACOS_VOICE names a voice; empty resolves the best installed one for the
configured language. The panel gains both, plus a preview button — choosing
blind from seventy-odd voices is not a choice.
Voices are discovered from `say -v ?` rather than hardcoded, since what is
installed varies by machine and macOS version. Output is WAV: the `say`
default is AIFF-C, which Chrome does not reliably decode.
Two defects found by testing rather than by reading:
The voice list parser keyed on the run of spaces before the locale tag, so
it saw 73 voices and one Italian. Classic voices are padded into a column
("Alice it_IT #"), newer ones carry a parenthesised language and
get a single space ("Eddy (Italiano (Italia)) it_IT #"). Anchored on the
"#" instead: 183 voices, nine Italian.
"auto" also resolved to Fish on an untouched install, because the
placeholder in .env.example is a non-empty string and passed a truth test.
Every line went to Fish for a 401 rather than falling back — precisely the
case this feature exists for. Both that check and the settings panel's now
share one _fish_configured helper.
`say` interprets [[...]] as embedded commands, [[volm 0]] among them, so
those are stripped: the text arrives from a speech transcript by way of an
LLM. Arguments go straight to exec, never a shell.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Acknowledgements, lookup results and error lines are built in Python, not written by the model, so they stayed English however the language was set. A single reply arrived as "Checking your calendar now, sir." followed by fluent Italian — the language setting looked half-finished because half of what JARVIS says never passed through it. Rather than ship hand-written translations for ten languages — inventing Japanese butler register is not something to do blind — the English catalogue is translated once per language by the model JARVIS already talks through, and cached on disk under data/. One Haiku call per language, ever. The cache is keyed by a hash of the catalogue, so editing a phrase re-translates instead of serving a stale set. Placeholders are named, never positional, because word order is exactly what changes between languages; the translator is told to reorder them freely but never rename or drop one. Translations are validated on the way out and again on the way in. A phrase that lost a placeholder does not raise — str.format ignores surplus arguments — so it would quietly drop the project name or the message count rather than fail visibly. Validating only at translation time would have left a hand-edited or truncated cache to do that silently, so load re-checks and falls back to English per phrase. Warm-up is backgrounded and English is used until it lands: a cold cache must not delay the first answer, and English is what that answer would have been anyway. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
data/phrases-<lang>.json is written at runtime by the translator and is keyed by a hash of the catalogue, so a committed copy would be both machine-specific and stale the moment a phrase changes. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
handle_research was pinned to claude-opus-4-6, three Opus generations behind. Swapping the model string alone would have broken it. Thinking is on by default on Opus 5, unlike 4.6 where omitting the parameter meant no thinking, so the response now arrives as ['thinking', 'text'] and content[0].text raises AttributeError — a ThinkingBlock has no .text. Verified against the live API rather than inferred. It now joins the text blocks and ignores the reasoning. max_tokens covers thinking and answer together on Opus 5, so 2000 — sized for prose on a non-thinking model — truncated the report. Raised to 16000, the ceiling before SDK HTTP timeouts constrain a non-streaming request. Opus 5's classifiers can decline a request, returning a successful HTTP 200 with stop_reason "refusal" and empty content; indexing content[0] would crash instead of reporting back. Checked first, and fallbacks="default" lets the API re-run a declined request elsewhere, routing by refusal category rather than pinning a model that will itself need migrating. Note handle_research has no callers: [ACTION:RESEARCH] dispatches to _execute_research, which spawns Claude Code. This changes no runtime behaviour today — it fixes code that would fail the moment anything reached it. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
This was referenced Aug 11, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
handle_researchwas pinned toclaude-opus-4-6, three Opus generations behind. This moves it toclaude-opus-5— which turns out not to be a one-line change.Swapping the model string alone would have broken it
Thinking is on by default on Opus 5 (unlike Opus 4.6, where omitting the parameter meant no thinking). So the response comes back as
['thinking', 'text'], and:Verified against the live API rather than inferred —
content[0].typeisthinking, andThinkingBlockhas no.text. Every research call would have raised.Now it joins the text blocks and ignores the reasoning.
Three more things Opus 5 requires
max_tokenscovers thinking and the answer. The old 2000 was sized for prose on a model that wasn't thinking; on Opus 5 the reasoning eats into the same budget and truncates the report. Raised to 16000 — the ceiling for a non-streaming request before SDK HTTP timeouts become the constraint.Refusals are a successful HTTP 200. Opus 5 ships elevated cybersecurity safeguards and its classifiers can decline a request, returning
stop_reason: "refusal"with an emptycontent. The old code indexedcontent[0]unconditionally, so a declined research request would crash rather than report back. Now checked before reading content.fallbacks: "default"is enabled. On a policy decline the API re-runs the request on another model server-side instead of handing back the refusal."default"routes by refusal category rather than pinning a specific fallback model that would itself need migrating later.Verified
Two real research calls end to end against the live API:
Also confirmed against the installed SDK (
anthropic0.121.0) thatfallbacks="default"with theserver-side-fallback-2026-07-01beta is accepted, rather than assuming the parameter exists.Testing
pytest tests/gives 35 passed, 8 failed — identical to this branch's base, so no regressions. The 8 pre-existing failures are unrelated: 7 needplaywright install, andtest_browse_action_keywordsimportsACTION_KEYWORDS, a symbol that no longer exists inserver.py.Note the two new phrase keys (
research.declined,research.empty) go through the #28 catalogue, so they are translated along with everything else rather than adding two more hardcoded English strings.🤖 Generated with Claude Code