Repository navigation
Add transcription API - #3
Merged
Merged
Conversation
`invoke_stream` refused everything but generate. It now opens an `rkwhisper-client` session for a transcribe call, forwards the audio arriving on later messages of the same call, and turns rkwhisperd's responses into events. `Asr` is the seam here that `Backend` is for generate: above it the session driver and the response mapping are ordinary Rust with tests, below it is a Unix socket and a shared-memory ring that only exist where rkwhisperd runs. Two things the design draft predates. rkwhisperd also sends `Cancelled` and the two VAD boundaries, which the table in MODEL_SERVER.md does not cover: the boundaries carry no event here, and a cancellation ends the stream rather than failing it, since this daemon is what asked for it. And `Done` carries no token counts, so usage stays zero rather than reporting a number invented here. `invoke` answers transcribe too, for a clip that arrives whole. The frontend will use the streaming call, since audio should reach rkwhisperd while the upload is still arriving, but refusing the unary one made the protocol dishonest. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01NQQRfiJcmrzXmnwKkJfAug
The upload arrives as multipart, is decoded with symphonia and resampled to 16 kHz mono s16le with rubato, and reaches the daemon as a stream rather than a buffer, so rkwhisperd starts on the first second of a recording while the rest is still being decoded. Probing is separate from decoding. Reading the container's headers is cheap, and doing it before the call makes an unreadable upload a 400 rather than a stream that fails after the daemon has begun reading it. All five response formats: json and text, then verbose_json, srt and vtt off the segment times. verbose_json carries the fields the official SDK declares as required, including the per-segment confidences rkwhisper does not report; those are zeros standing in for numbers that do not exist, which the comment there says plainly. Duration is counted from the samples the decoder produced rather than read from the upload's header, which may be absent, wrong, or describe a length the file does not contain. Two things worth knowing. rubato writes a whole output chunk even for a short final block, so a clip would arrive padded with the filter's own silence; `Resampling` tracks how much input it has taken to know how much output is owed and drops the rest. And symphonia has no Opus decoder, so webm/opus is the one format on OpenAI's list that this refuses. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01NQQRfiJcmrzXmnwKkJfAug
rkwhisper models load nothing here, so they have no loader thread to report progress, and `Registry::from_config` left them `unavailable` permanently. Every transcribe call was refused at validation before it reached the session code. The check is a session opened and dropped. That is more than reaching the socket, which only says something is listening: the handshake names the model, so a model rkwhisperd does not serve is caught too. It is less than a transcription, since no audio follows and rkwhisperd does no NPU work for it. MODEL_SERVER.md lists "how to tell whether rkwhisperd is up without starting a transcription" as an open question, so the interval is a constant with that noted rather than new config surface. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01NQQRfiJcmrzXmnwKkJfAug
`tests/sdk_conformance.py` now drives the official SDK against /v1/audio/transcriptions: the four response formats, a 44.1 kHz upload that exercises the resampler, and an upload that is not audio. It builds its own WAV, so the script still needs no fixture beside it. Set RKMODEL_WHISPER_MODEL to an empty string to skip it where no rkwhisperd is running. MODEL_SERVER.md's response table gains the three responses rkwhisperd sends that it predated, milestone 2 gets its checklist, and open question 9 records what the probe settled and what it did not. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01NQQRfiJcmrzXmnwKkJfAug
rubato reports a whole block consumed even for a partial one, since it reads the rest as silence. `Resampling` counted that, so the clip looked longer than it was and the padding it exists to drop went through instead: a second of 44.1 kHz arrived as 16347 samples rather than 16000, and half a second of 48 kHz as 8192 rather than 8000. The lengths are exact now, so the tests assert them exactly. The tolerance they had before was wide enough to hide this. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01NQQRfiJcmrzXmnwKkJfAug
It still described the workspace as a skeleton that loads no weights, which stopped being true when milestone 1 landed. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01NQQRfiJcmrzXmnwKkJfAug
Axum's DefaultBodyLimit is 2 MB. This endpoint documents OpenAI's 25 MB cap and checks the file field against it, but that check could never fire: anything over 2 MB was refused while the body was still being read, and arrived as "Error parsing `multipart/form-data` request" with a 400. A 3 MB recording was rejected as malformed. The route now sets its own limit, and a multipart failure reports the status the error itself carries, so an oversized upload is a 413 saying so rather than a 400 sending the caller to look at their encoding. The test that would have caught it posts 3 MB and expects it through; reverting the layer fails it. rkwhisperd's `Done` also carries `audio_s` and `rtf`, which were discarded. They are logged at debug now: that is the first thing to look at when a transcription feels slow, and it says whether the time went to the model or to this daemon. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01NQQRfiJcmrzXmnwKkJfAug
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Milestone 2. transcribe reaches rkwhisperd through rkwhisper-client in the daemon, and POST /v1/audio/transcriptions serves it in the frontend.
The path
The frontend takes the multipart upload, decodes the container with symphonia, resamples to 16 kHz mono s16le with rubato, and hands the daemon a stream rather than a buffer, so rkwhisperd starts on the first second of a recording while the rest is still being decoded and a long clip is never held in memory as PCM. The daemon opens an rkwhisper-client session and maps rkwhisperd's responses onto this protocol's events. All five response formats are served: json, text, srt, vtt and verbose_json.
Transcription goes through the daemon rather than straight from the frontend, because rkwhisper-client needs rkwhisperd's Unix socket and shared-memory audio ring — a frontend using it directly would have to run on the board. This way the frontend speaks one protocol and /v1/models has one source.
Also here
Model state for rkwhisper models. They load nothing locally, so they had no loader thread to report progress and sat unavailable permanently, which refused every transcribe call at validation. The probe is a session opened and dropped: more than reaching the socket, since the handshake names the model; less than a transcription, since no audio follows.
A body limit that was hiding a 25 MB cap. Axum's DefaultBodyLimit is 2 MB, so the endpoint's documented OpenAI limit could never fire and a 3 MB recording came back as "Error parsing multipart/form-data request". An oversized upload is now a 413 that says so.
A resampler count fix. rubato reports a whole block consumed even for a partial one, so clips looked longer than they were and the padding the count exists to drop went through instead. Lengths are exact now, and the tests assert them exactly rather than within a tolerance wide enough to hide it.
SDK coverage. tests/sdk_conformance.py drives the official SDK against the endpoint: the four response formats, a 44.1 kHz upload that exercises the resampler, and an upload that is not audio. It builds its own WAV, so it still needs no fixture beside it.
Verification
cargo fmt, cargo clippy --workspace --all-targets -- -D warnings and cargo test --workspace --all-targets (207 tests) are clean, and the release build cross-compiles for aarch64-unknown-linux-gnu. The frontend's tests run against a fake daemon, so they need no hardware.
Exercised end to end on an RK3588 against rkwhisperd with whisper-small-30s: a ten-minute clip returns a transcript, timed segments and verbose_json.
Known, and not addressed here
Transcript coverage on long audio depends on rkwhisper's own decode settings, which this PR does not expose. rkwhisperd's ClientHello defaults are what it ships with — beam 5, timestamps on, 128 new tokens per window. Measured on a 9.6-minute clip, dropping to greedy decoding (beam_size = 1) covered only 14% of the audio: a degenerate pass trips rkwhisper's repetition guard, and the stale timestamp it last emitted drives the seek loop past whole windows. So beam_size, notimestamps and max_new_tokens stay at rkwhisper's defaults and are not configurable yet. Whether they become per-model config or per-request fields is worth settling separately, with numbers.
verbose_json's avg_logprob, compression_ratio, no_speech_prob and tokens are zeros and empties standing in for values rkwhisper does not report. They are present because the SDK's schema requires the fields.
🤖 Generated with Claude Code