Skip to content

fix: address the user the way he asked, and greet him in his language - #26

Open
luca-71 wants to merge 6 commits into
ethanplusai:mainfrom
luca-71:fix/addressing
Open

luca-71 wants to merge 6 commits into
ethanplusai:mainfrom
luca-71:fix/addressing

Conversation

@luca-71

@luca-71 luca-71 commented Aug 10, 2026

Copy link
Copy Markdown

Three complaints from one Italian session — "why does it call me by another name?", "I keep hearing English and then immediately Italian", "it often doesn't understand when I call it" — and three separate causes.

Stacked on #22. The localised greeting needs the SPEECH_LANG setting that PR introduces. Review #22 first.

He was called the wrong thing

HONORIFIC was written to .env, served by the preferences API, and offered as a dropdown in the settings panel. Nothing ever read it. The prompt hardcoded:

- Address {user_name} as "sir" naturally

So the dropdown was decorative. And in Italian the model translated that "sir" freely, settling on "signora" — addressing him with a feminine honorific, 5 times out of 5, while his configuration said sir.

The honorific is now a prompt variable, and the language directive states it must be reproduced letter for letter rather than translated or inflected for gender.

Where that instruction lives turned out to matter. In the personality section it was ignored 5/5. Moved into the language block appended at the end, it holds 5/5. Measured both ways round, so the fix is not merely "it stopped saying signora":

configured before after
sir "signora" 5/5 "sir" 5/5
signore "signore" 5/5 "signore" 5/5

The panel field also becomes free text with suggestions. It offered sir, ma'am and none — so signore was not expressible in the first place, and no amount of prompt work would have helped.

The greeting arrived in English

greeting = "Good evening, sir."

Hardcoded, sent on connect. Every Italian session therefore opened in English and switched language on the very next sentence — exactly the "first English, then immediately Italian" the user described. It is the one line heard every single time.

Now built from a small table covering the languages the panel offers, falling back to English for anything else. Thirty short strings, bounded — not an attempt to solve the wider hardcoded-string problem.

It could not hear its own name

An Italian recogniser mangles an English name. From the logs, where he was plainly saying JARVIS:

User: arbiss
User: e gli arbis
User: hey Yaris

Added to the existing correction table — taken from real transcripts rather than invented.

yaris is corrected only when preceded by a term of address. It is also a very common Toyota, and the obvious bare rule rewrote "la mia auto Yaris" into a summons. Verified in both directions, including that arbitro, arbitri and arbitrario pass through untouched — the word-boundary anchors hold.

Testing

pytest tests/ gives 35 passed, 8 failed — identical to this branch's base, so no regressions. The 8 pre-existing failures are unrelated: 7 need playwright install, and test_browse_action_keywords imports ACTION_KEYWORDS, a symbol that no longer exists in server.py. tsc --noEmit clean.

🤖 Generated with Claude Code

Luca Trisiello and others added 6 commits August 10, 2026 17:58
npm audit fix, no --force needed: vite 6.4.1 -> 6.4.3, plus transitive
bumps to postcss, nanoid, and picomatch. All within the ranges package.json
already declares, so only the lockfile moves.

The one that mattered here is the Vite dev server's arbitrary file read via
WebSocket (GHSA-p9ff-h696-f583) — this project's documented workflow runs
that dev server.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Nine call sites sent JARVIS's reply only inside `if audio:`, so whenever
Fish Audio was unconfigured, rate-limited, or failing, the reply was
dropped with no trace in the UI. The startup greeting and every proactive
notification — research complete, build finished, project connected — went
straight to nothing. Each site now falls back to a "text" message, which
the client can surface.

Three of those sites also passed the raw `bytes` from synthesize_speech
straight to send_json as `"data": audio`, without base64. Bytes are not
JSON-serializable, so the send raised — and in two cases the exception was
swallowed by a bare `except Exception: pass`. Those paths ("Fix Yourself"
and the calendar/mail lookups) delivered neither audio nor text even with
a working Fish key, and did so silently.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
The client's stated "text fallback when TTS fails" only ran console.log,
so a reply that arrived without audio reached the devtools console and
nowhere else. With no Fish Audio key the interface was not merely silent
but blank — the orb reacted to nothing and said nothing.

Add a caption below the orb, in the existing palette. It is shown whether
or not audio arrived, so it doubles as subtitles for a working voice.

It holds for 4s plus 60ms per character, capped at 20s. Reading time
rather than a fixed delay, because without a voice the caption is the
entire response and vanishing mid-sentence loses it.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
recognition.lang was hardcoded to en-US, so anyone speaking another
language was transcribed phonetically into English nonsense — "come stai"
arrived as "comic Style". Add a Spoken Language selector to the settings
panel, persisted as SPEECH_LANG and read live, so the choice applies
without a restart.

voice.ts rebuilds the recognition object on a language change rather than
reassigning .lang. Chrome reads that property at construction and ignores
it on a session that has already run, so the reassignment silently kept
the old language. The retired session's onend checks whether it has been
superseded, or it would restart and race the new one for the microphone.
main.ts resolves the language before opening the microphone, so the first
session is built correctly and the fragile switch path is reserved for
live changes from the panel.

The system prompt is English and would answer an Italian question in
English, so a non-English selection appends a language directive. Kept to
a bare directive deliberately: wordings that also discussed action
selection measurably weakened the language adherence they were added to
enforce.

Known limitation: action routing is less reliable outside English. On one
repeated question, English chose the right action 4/4 while Italian
misrouted to a screen capture 4/4. The 46 hardcoded English strings in
server.py ("Right away, sir.") also stay English — translating them is an
i18n project, not a setting.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Chrome closes a recognition segment at every pause, and the client sent
each "final" result the instant it arrived. A sentence spoken with any
hesitation reached JARVIS as several questions: he answered half a thought,
then received the rest as a new one. The logs show single words like "hey"
arriving alone.

Collect fragments and send once the speaker has actually stopped, after a
1s gap. Two edges that would otherwise bite:

- A 6s ceiling. Steady dictation never produces the gap, so without a cap
  the timer re-arms forever and the reply never comes.
- Muting discards what is pending, so half a sentence cannot arrive a
  second after the user silenced him.

Barge-in stays immediate: the audio is cut on the first fragment, not at
flush, since interrupting is what the user wanted the moment they spoke.

Costs 1s before JARVIS starts thinking. That is the price of knowing the
sentence ended; UTTERANCE_GAP_MS trades it back for more split sentences.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Three complaints from one Italian session, three separate causes.

**He was called the wrong thing.** HONORIFIC was written to .env, served by
the preferences API and offered in the settings panel — and never read.
The prompt hardcoded `Address {user_name} as "sir"`, so the dropdown did
nothing. Worse, the model translated that "sir" freely and settled on
"signora": it addressed him with a feminine honorific, five times out of
five, while his configuration said otherwise.

The honorific is now a prompt variable, and the language directive states
it must be reproduced letter for letter rather than translated or
inflected. Placing that in the language block matters — the same
instruction in the personality section was ignored 5/5, and obeyed 5/5 at
the end. Measured, both ways round: "sir" and "signore" now each hold 5/5.

The panel field becomes free text with suggestions. It offered sir, ma'am
and none, so "signore" was not expressible in the first place.

**The greeting arrived in English.** It was hardcoded, so every Italian
session opened with "Good evening, sir." and switched language on the next
sentence. It is the one line a user hears every single time. Now built
from a small table for the languages the panel offers, falling back to
English.

**It could not hear its own name.** An Italian recogniser mangles an
English name: the logs show "arbiss", "e gli arbis" and "hey Yaris" where
he plainly said JARVIS. Added as corrections, taken from the transcripts
rather than invented.

"yaris" is handled only after a term of address. It is also a very common
Toyota, and a bare rule rewrote "la mia auto Yaris" into a summons.
Verified in both directions, including that "arbitro", "arbitri" and
"arbitrario" survive untouched.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant