Fix the voice loop: a deaf microphone, his own voice interrupting him, and everything said three times

[ethanplusai]

Six fixes from a day of using JARVIS by voice on a room microphone rather than a headset. Every one was found in a log, and each ships with a test named for the failure it protects against.

The microphone could go deaf and say nothing about it

Chrome ends a continuous recognition session on its own, and safeStart swallowed the InvalidStateError thrown when the engine is between states — by which point onend has already fired, so nothing retried and it never listened again.

Worse, a session can report listening, attach audio, and simply return nothing. Measured live: 45 seconds of a live capture with no results, during which nothing anywhere indicated a problem.

createMicMonitor now holds its own getUserMedia stream open for the life of the page — no session, no timeout, no restarts — and measures the signal directly. Sound going in with nothing coming out is a named failure that rebuilds the recogniser within three seconds. A small meter in the corner answers "is it even hearing me?" without a log or a console.

What the engine is actually doing is tracked (onstart / onaudiostart) rather than assumed, and a watchdog covers the two shapes no event can report: nothing listening, and listening with no audio.

The recogniser is also rotated ahead of Chrome's ~8s no-speech teardown — but only in silence. Chrome permits one live recogniser, so starting a second aborts the first, and an aborted session never emits the final result for whatever was being said. Measured: interims for a whole sentence arrived, rotation fired 17ms after the last one, and nothing reached the server.

He interrupted himself

The echo rule matches whole tokens against what he just said. That catches his sentence coming back intact and misses it entirely when the recogniser mangles it — which, over a speaker, it always does.

A count of new words cannot separate a mis-hear from a real interruption; both clear two. The proportion can. While his audio is audible, a phrase half composed of words he is saying right now is his voice, not the user's. Cancel words are untouched, and "delete the file" over "Deleting the staging files" still cuts him off.

He said the same thing three times

Told to pass a message to a session, he would say "Will say that to it." — tool — "Saying this now." — tool — "Passed that to chitauri, sir." Three sentences for one instruction, because everything he writes is spoken the instant he writes it.

The persona already forbade it; he complied for a while and then didn't. So a turn that uses a tool now says exactly one thing, at the end, and everything written before the last tool call is dropped as narration of something not yet done. Ordinary conversation still streams as before.

A memory that could not be saved

A memory writer is refused while anything foreign sits in the generation composing it — correctly, since what it writes becomes trusted text later. The refusal told the user to say it again "in a fresh conversation", and there was no way to start one: no command, no tool, nothing in the persona. Live, that cost four turns and the fact went unsaved.

start fresh (and the other phrasings a person actually uses) now discards the generation, with no handover — carrying a summary across would carry the tainted text with it. It is tested to be hard to trigger by accident: "start the build" and "tell the session to start" do nothing.

There was no way to stop him

Every interrupt was a spoken word, and a spoken interrupt is exactly what the echo machinery makes unreliable. Escape, and a button shown only while he speaks, now stop him. A keystroke cannot be misheard. hush is deliberately not in CANCEL_WORDS, and there is a test asserting it stays out.

And so it can be seen next time

None of the above was visible. A barge-in, a resume, a dropped narration, and the recogniser's entire lifecycle produced no log line at all — so "he repeated himself" and "he could not hear me" were indistinguishable from "he was not asked". They all say so now, in the server log, next to the transcripts.

---

Testing: 2425 pass, tsc --noEmit clean, npm run build clean. All changed Python files parse under 3.12 (CI's version).

🤖 Generated with Claude Code