Skip to content

Add transcription API - #3

Merged
boydjohnson merged 7 commits into
mainfrom
transcription
Sep 12, 2026
Merged

boydjohnson merged 7 commits into
mainfrom
transcription

Conversation

@boydjohnson

Copy link
Copy Markdown
Member

Milestone 2. transcribe reaches rkwhisperd through rkwhisper-client in the daemon, and POST /v1/audio/transcriptions serves it in the frontend.

The path
The frontend takes the multipart upload, decodes the container with symphonia, resamples to 16 kHz mono s16le with rubato, and hands the daemon a stream rather than a buffer, so rkwhisperd starts on the first second of a recording while the rest is still being decoded and a long clip is never held in memory as PCM. The daemon opens an rkwhisper-client session and maps rkwhisperd's responses onto this protocol's events. All five response formats are served: json, text, srt, vtt and verbose_json.

Transcription goes through the daemon rather than straight from the frontend, because rkwhisper-client needs rkwhisperd's Unix socket and shared-memory audio ring — a frontend using it directly would have to run on the board. This way the frontend speaks one protocol and /v1/models has one source.

Also here
Model state for rkwhisper models. They load nothing locally, so they had no loader thread to report progress and sat unavailable permanently, which refused every transcribe call at validation. The probe is a session opened and dropped: more than reaching the socket, since the handshake names the model; less than a transcription, since no audio follows.
A body limit that was hiding a 25 MB cap. Axum's DefaultBodyLimit is 2 MB, so the endpoint's documented OpenAI limit could never fire and a 3 MB recording came back as "Error parsing multipart/form-data request". An oversized upload is now a 413 that says so.
A resampler count fix. rubato reports a whole block consumed even for a partial one, so clips looked longer than they were and the padding the count exists to drop went through instead. Lengths are exact now, and the tests assert them exactly rather than within a tolerance wide enough to hide it.
SDK coverage. tests/sdk_conformance.py drives the official SDK against the endpoint: the four response formats, a 44.1 kHz upload that exercises the resampler, and an upload that is not audio. It builds its own WAV, so it still needs no fixture beside it.
Verification
cargo fmt, cargo clippy --workspace --all-targets -- -D warnings and cargo test --workspace --all-targets (207 tests) are clean, and the release build cross-compiles for aarch64-unknown-linux-gnu. The frontend's tests run against a fake daemon, so they need no hardware.

Exercised end to end on an RK3588 against rkwhisperd with whisper-small-30s: a ten-minute clip returns a transcript, timed segments and verbose_json.

Known, and not addressed here
Transcript coverage on long audio depends on rkwhisper's own decode settings, which this PR does not expose. rkwhisperd's ClientHello defaults are what it ships with — beam 5, timestamps on, 128 new tokens per window. Measured on a 9.6-minute clip, dropping to greedy decoding (beam_size = 1) covered only 14% of the audio: a degenerate pass trips rkwhisper's repetition guard, and the stale timestamp it last emitted drives the seek loop past whole windows. So beam_size, notimestamps and max_new_tokens stay at rkwhisper's defaults and are not configurable yet. Whether they become per-model config or per-request fields is worth settling separately, with numbers.

verbose_json's avg_logprob, compression_ratio, no_speech_prob and tokens are zeros and empties standing in for values rkwhisper does not report. They are present because the SDK's schema requires the fields.

🤖 Generated with Claude Code

boydjohnson and others added 7 commits September 11, 2026 19:37
`invoke_stream` refused everything but generate. It now opens an
`rkwhisper-client` session for a transcribe call, forwards the audio
arriving on later messages of the same call, and turns rkwhisperd's
responses into events.

`Asr` is the seam here that `Backend` is for generate: above it the
session driver and the response mapping are ordinary Rust with tests,
below it is a Unix socket and a shared-memory ring that only exist
where rkwhisperd runs.

Two things the design draft predates. rkwhisperd also sends `Cancelled`
and the two VAD boundaries, which the table in MODEL_SERVER.md does not
cover: the boundaries carry no event here, and a cancellation ends the
stream rather than failing it, since this daemon is what asked for it.
And `Done` carries no token counts, so usage stays zero rather than
reporting a number invented here.

`invoke` answers transcribe too, for a clip that arrives whole. The
frontend will use the streaming call, since audio should reach
rkwhisperd while the upload is still arriving, but refusing the unary
one made the protocol dishonest.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01NQQRfiJcmrzXmnwKkJfAug
The upload arrives as multipart, is decoded with symphonia and
resampled to 16 kHz mono s16le with rubato, and reaches the daemon as a
stream rather than a buffer, so rkwhisperd starts on the first second of
a recording while the rest is still being decoded.

Probing is separate from decoding. Reading the container's headers is
cheap, and doing it before the call makes an unreadable upload a 400
rather than a stream that fails after the daemon has begun reading it.

All five response formats: json and text, then verbose_json, srt and vtt
off the segment times. verbose_json carries the fields the official SDK
declares as required, including the per-segment confidences rkwhisper
does not report; those are zeros standing in for numbers that do not
exist, which the comment there says plainly.

Duration is counted from the samples the decoder produced rather than
read from the upload's header, which may be absent, wrong, or describe a
length the file does not contain.

Two things worth knowing. rubato writes a whole output chunk even for a
short final block, so a clip would arrive padded with the filter's own
silence; `Resampling` tracks how much input it has taken to know how
much output is owed and drops the rest. And symphonia has no Opus
decoder, so webm/opus is the one format on OpenAI's list that this
refuses.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01NQQRfiJcmrzXmnwKkJfAug
rkwhisper models load nothing here, so they have no loader thread to
report progress, and `Registry::from_config` left them `unavailable`
permanently. Every transcribe call was refused at validation before it
reached the session code.

The check is a session opened and dropped. That is more than reaching
the socket, which only says something is listening: the handshake names
the model, so a model rkwhisperd does not serve is caught too. It is
less than a transcription, since no audio follows and rkwhisperd does no
NPU work for it.

MODEL_SERVER.md lists "how to tell whether rkwhisperd is up without
starting a transcription" as an open question, so the interval is a
constant with that noted rather than new config surface.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01NQQRfiJcmrzXmnwKkJfAug
`tests/sdk_conformance.py` now drives the official SDK against
/v1/audio/transcriptions: the four response formats, a 44.1 kHz upload
that exercises the resampler, and an upload that is not audio. It builds
its own WAV, so the script still needs no fixture beside it. Set
RKMODEL_WHISPER_MODEL to an empty string to skip it where no rkwhisperd
is running.

MODEL_SERVER.md's response table gains the three responses rkwhisperd
sends that it predated, milestone 2 gets its checklist, and open
question 9 records what the probe settled and what it did not.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01NQQRfiJcmrzXmnwKkJfAug
rubato reports a whole block consumed even for a partial one, since it
reads the rest as silence. `Resampling` counted that, so the clip looked
longer than it was and the padding it exists to drop went through
instead: a second of 44.1 kHz arrived as 16347 samples rather than
16000, and half a second of 48 kHz as 8192 rather than 8000.

The lengths are exact now, so the tests assert them exactly. The
tolerance they had before was wide enough to hide this.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01NQQRfiJcmrzXmnwKkJfAug
It still described the workspace as a skeleton that loads no weights,
which stopped being true when milestone 1 landed.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01NQQRfiJcmrzXmnwKkJfAug
Axum's DefaultBodyLimit is 2 MB. This endpoint documents OpenAI's 25 MB
cap and checks the file field against it, but that check could never
fire: anything over 2 MB was refused while the body was still being
read, and arrived as "Error parsing `multipart/form-data` request" with
a 400. A 3 MB recording was rejected as malformed.

The route now sets its own limit, and a multipart failure reports the
status the error itself carries, so an oversized upload is a 413 saying
so rather than a 400 sending the caller to look at their encoding.

The test that would have caught it posts 3 MB and expects it through;
reverting the layer fails it.

rkwhisperd's `Done` also carries `audio_s` and `rtf`, which were
discarded. They are logged at debug now: that is the first thing to look
at when a transcription feels slow, and it says whether the time went to
the model or to this daemon.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01NQQRfiJcmrzXmnwKkJfAug
@boydjohnson
boydjohnson merged commit c4d9e1d into main Sep 12, 2026
3 checks passed
@boydjohnson
boydjohnson deleted the transcription branch September 12, 2026 17:51
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant