feat: voice dictation — Slack/Discord audio → text, supervised with the daemon - #111
Merged
Conversation
Transcribe inbound audio attachments (Discord voice notes, Slack audio clips) to text so users can dictate prompts to Claude alongside text and images. Claude has no native audio input, so the daemon transcribes first and merges the result into the message as a [voice transcript] block; the original audio file stays in downloaded_files. - daemon/transcription.ts: audio detection, transcript merging (pure, unit-tested), and an HTTP client to a self-hosted STT sidecar. Failures are logged and skipped — dictation never blocks message delivery. - daemon/router.ts: hook transcription into buildNotificationPayload after attachment download. - transcribe-server/: self-hosted sidecar serving NVIDIA Canary-Qwen 2.5B via NeMo (top of the Open ASR leaderboard for English accuracy), plus a GPU-free mock_server.py for testing the wiring locally. - Off by default; enable with HYDRA_TRANSCRIBE_ENABLED=1. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Make dictation work as a packaged default rather than a manual opt-in:
- Daemon transcription is now ON by default ("auto"): it's attempted
whenever audio arrives and silently no-ops if no sidecar is reachable
(the fetch fails fast). HYDRA_TRANSCRIBE_ENABLED=0 opts out.
- start-transcribe.sh: idempotent launcher for the sidecar in a tmux
session. Accepts a backend arg (`./start-transcribe.sh mock`) for a
zero-GPU end-to-end test; canary backend refuses cleanly until set up.
- watchdog.sh: revive the sidecar when HYDRA_TRANSCRIBE_AUTOSTART is set,
reusing the same supervision pattern as the bot session.
- transcribe-server/setup.sh: one-time venv + NeMo install.
- Docs/env updated for the packaged-default flow.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Pasting a doc line with an inline '# comment' into interactive zsh (which does not strip '#') passed the comment as args, making the launcher try backend '#'. Only accept mock|canary as the positional arg; ignore anything else with a warning and fall back to env/default. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Canary-Qwen runs via NeMo and needs a CUDA GPU, so it can't run on Mac. Add a Parakeet-MLX backend (NVIDIA Parakeet TDT on Apple's MLX runtime): native, ~50x realtime on M-series, ~6% English WER, no GPU. - transcribe-server/server_mlx.py — FastAPI server, same /transcribe contract, parakeet-mlx + ffmpeg resample. - transcribe-server/requirements-mlx.txt — light deps (no torch/NeMo). - start-transcribe.sh — `parakeet` backend; default by platform (Darwin -> parakeet, else canary). - setup.sh — installs the right requirements per platform / arg. - Docs updated across README, transcribe-server/README, .env.example. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
MLX ships arm64-only wheels, and on Apple Silicon the shell frequently runs under Rosetta (x86_64), where 'pip install mlx' fails with no matching wheel. setup.sh now uses an arm64 Homebrew python@3.12 and builds the venv via 'arch -arm64' for the parakeet backend; uv/system python paths remain for canary. Also relax the parakeet-mlx pin. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
tmux does not reliably inherit env, so a PARAKEET_MODEL override in .env never reached the server (it would fall back to the 0.6B default). Forward model-selection vars explicitly, only when set. Enables pinning the smaller 110M Parakeet on constrained networks where the 2.4GB 0.6B is impractical. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
…tests The module-top-level stub leaked into every test file loaded after it, swallowing bun test's per-test output and final summary for the rest of the suite (bun runs all files in one process). The stub was unnecessary: these tests only exercise pure helpers that never write to stderr. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Voice dictation now comes up with the daemon instead of needing a separate manual step or an explicit opt-in flag: - start-transcribe.sh gains an --auto mode used by every supervisor: explicit HYDRA_TRANSCRIBE_AUTOSTART wins in both directions; when unset it starts the sidecar iff the backend is ready (venv built, or mock chosen), and quietly no-ops otherwise so unconfigured machines don't log a failure every watchdog cycle. Honors HYDRA_TRANSCRIBE_ENABLED=0. - hydra up starts it right after the daemon (model loads while the byte comes up); hydra watchdog revives it each tick; hydra down stops it. - Legacy start-daemon.sh and watchdog.sh call the same --auto path; the watchdog's AUTOSTART=1 opt-in grep is replaced by the shared gate. - mock backend: resolve python3 up front and fall back to /usr/bin/python3 — asdf's shim fails when no python version is pinned for the dir, and mock_server.py is pure stdlib. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Slack voice clips arrive as files with mimetype audio/mp4 (m4a), or audio/webm;codecs=opus from browser recordings — assert detection for those shapes plus the extension fallback. Add transcribeDownloads tests with a stubbed fetch: only audio files are POSTed, sidecar failure skips the file instead of throwing, and the disabled flag short-circuits before the network. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…d 1) Structural: - ONE shared tmux session (hydra-transcribe) for all platform daemons — per-platform sessions raced for the same default port, and the loser reloaded the full model every watchdog tick. hydra down kills it; the other platform's watchdog revives it within a tick. - A crashed server PARKS its session (error on screen + in log) instead of exiting, so supervisors' has-session check holds: broken configs fail once, not as a model-load crash-loop every 120s. - The mock backend is excluded from --auto (manual or explicit AUTOSTART=1 only) — leftover test config must not keep canned transcripts flowing into real prompts. A remote HYDRA_TRANSCRIBE_URL also disables local autostart. Environment/robustness: - Extract only dictation keys from the state-dir .env instead of set -a sourcing the whole file — the model server has no business holding chat bot tokens (they leaked into the tmux server env when this script bootstrapped it). - Forward PATH into the tmux pane (launchd-frozen server env lacks /opt/homebrew/bin, breaking the servers' ffmpeg lookup) and shell-quote every interpolated value (shq) so an embedded quote can't break out of the tmux command string. - URL without an explicit port now binds the scheme default so a mismatch fails visibly instead of the sidecar silently serving a port the daemon never queries. - start-daemon.sh: sidecar refusal no longer fails the whole script under set -e after a successful daemon start. Legacy watchdog runs the sidecar step before the daemon branches' early exits. Daemon: - isAudioFile: a definitive non-audio MIME (video/mp4 screen recording) is no longer re-classified as audio by its extension; only generic types fall back. Codec suffixes (audio/webm;codecs=opus) parsed correctly. - transcribeFile checks size via statSync BEFORE reading — the cap now protects daemon memory, not just sidecar latency. - Tests: env save/restore moved into beforeEach/afterEach (the old describe-body restore ran at collection time and leaked env into later test files); network tests set HYDRA_TRANSCRIBE_ENABLED explicitly; new cases for video-MIME rejection and the pre-network size cap. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
- --auto with AUTOSTART unset now also requires BACKEND != mock — with a built venv, leftover BACKEND=mock in .env passed the venv-only gate and auto-supervised canned transcripts, the exact residue case the previous commit claimed to prevent. - .env key extraction now parses like shell sourcing: optional 'export' prefix, quoted values kept verbatim (a # inside quotes is not a comment), unquoted values lose trailing inline comments/whitespace. The grep|cut version kept ' 0 # never' whole, silently defeating explicit opt-outs and erroring per watchdog tick on commented backends. - Document that multi-platform machines must keep dictation config identical across platform .env files (shared session = first supervisor wins). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…port The daemon's .env parser used a bare `^(\w+)=(.*)$` regex that passed through inline `# comments` as part of the value — a malformed `SPAWN_CWD=~/work # my dir` would set SPAWN_CWD to `~/work # my dir` and break spawns. The sidecar's .env parser (start-transcribe.sh) already handled this correctly; this brings the daemon in line. Extracts `parseEnvLine` into `shared/env-parse.ts` (pure, no side effects) so both the daemon config loader and tests can use it without triggering config.ts's module-scope gateway/token side effects. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
sf8193
reviewed
Jul 9, 2026
…ield naming - Guard `tmuxKill(cfg.transcribeTmux)` in `hydra down`: only kill the shared `hydra-transcribe` session when the other platform's daemon isn't running, so `hydra down discord` doesn't break Slack's sidecar. - Replace sync `statSync`/`readFileSync` with async `stat`/`readFile` from `fs/promises` in `transcribeFile` — avoids blocking the event loop for files near the 25MB cap on the inbound-message hot path. - Rename `voice_transcript` metadata field to `voice_transcript_count` to match what it stores (a count, not the actual transcript text). - Extract `TRANSCRIBE_TMUX` constant to `shared/constants.ts` per project convention (cross-compilation-unit constants go there). Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
sf8193
reviewed
Jul 9, 2026
…chdog skip - Pass computed `content` (with '(attachment)' fallback) to mergeTranscripts instead of raw `msg.content`, so the fallback survives when all transcripts are empty. - Guard transcribeDownloads call on downloadedFiles.length > 0 to skip the promise allocation for text-only messages. - Replace two-platform assumption in sidecar kill guard with a general check against all known platforms. - Guard watchdog's startTranscribeAuto with tmuxExists to skip the shell fork when the sidecar is already running. - Use `catch (err: unknown)` with type guard instead of `any`. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
[voice transcript] .... Audio stays local — Parakeet-MLX on macOS (Apple Silicon), Canary-Qwen 2.5B on Linux+GPU.hydra upstarts the sidecar alongside the daemon,hydra downstops it (only when no other platform's daemon is running — shared session preserved), every watchdog tick revives it (skips the shell fork when already running). One shared tmux session (hydra-transcribe), park-on-crash (no model-load crash-loop), zero extra config after one-time./transcribe-server/setup.sh..envparser fix: strips inline# comments, handlesexportprefix and quoted values — the old regex passed trailing comments as part of the value. Extracted toshared/env-parse.tswith 11 unit tests.transcribeFileusesstat/readFilefromfs/promises— no event loop blocking for large audio files.execFileSynctimeout: sidecar autostart bounded to 15s;catch (err: unknown)with proper type guard.What changed
daemon/transcription.ts— audio detection + transcript merge (pure, unit-tested) + async HTTP client; failures logged and skipped, never block deliverydaemon/router.ts— hook intobuildNotificationPayloadafter attachment download;voice_transcript_countmeta marker; guarded ondownloadedFiles.length > 0cli/lifecycle.ts,cli/helpers.ts— sidecar supervision inhydra up/down/watchdog; shared sidecar kill guarded (iterates known platforms); watchdog skips shell fork when sidecar already runningshared/env-parse.ts—.envline parser (handles export, quotes, inline comments)shared/constants.ts—TRANSCRIBE_TMUXconstant (cross-compilation-unit per convention)daemon/config.ts— uses new parserstart-transcribe.sh—--autogate, shared session, park-on-crash, key-allowlisted.envextractionstart-daemon.sh,watchdog.sh— call the shared--autopathtranscribe-server/—server_mlx.py(Parakeet-MLX),server.py(Canary-Qwen/NeMo),mock_server.py(stdlib stub),setup.sh, requirements, READMEdaemon/__tests__/transcription.test.ts,shared/__tests__/env-parse.test.ts.env.example,README.mdTest plan
bun buildclean on all 3 entry points (daemon.ts, cli/hydra.ts, bridge.ts)bun test— 432/434 pass (2 pre-existing failures on origin/main, unrelated)[voice transcript]appearshydra up/hydra down/watchdog with sidecar configured — verify start/stop/revivehydra down discordwhile slack daemon is running — verify sidecar stays aliveOrigin
Rebased from kwliang1/hydra#1 onto upstream main. Improvements over the original PR: daemon
.envparser fix,execFileSynctimeout, async file I/O, shared-kill guard,TRANSCRIBE_TMUXconstant,mergeTranscriptsfallback fix, watchdog skip optimization,downloadedFiles.lengthguard,catch (err: unknown).🤖 Generated with Claude Code