feat: background HTTP pre-transcription while recording#252
Open
JaPossert wants to merge 2 commits into
Open
Conversation
Add a new "Begin transcribing while recording" setting that submits audio to the transcription provider in 28-second background chunks while the user speaks. By the time the user presses stop, the majority of the recording is already processed — only the tail (< 28 s) needs transcribing on stop, dramatically reducing perceived latency for long dictations. Unlike the existing realtime WebSocket mode this works with any standard OpenAI-compatible HTTP transcription endpoint (Groq, local Whisper/ faster-whisper, etc.) with no additional server requirements. Implementation: - PrefetchTranscriber.swift: lightweight class that accumulates raw PCM16 samples (24 kHz mono, from AudioRecorder.onPCM16Samples), writes 28-second chunks to temporary WAV files, and submits them to the existing TranscriptionService serially in the background. commitAndAwaitFinal() waits for in-flight chunks then transcribes the remaining tail, returning the joined transcript. - AppState: new @published prefetchTranscriptionEnabled property, wired into the recording start/stop/cancel flow alongside the existing realtimeService path. Realtime WebSocket takes priority if both settings are enabled. - SettingsView: new toggle above the existing realtime toggle, with a description clarifying it requires no special server support. Relates to zachlatta#250 (URLSession timeout fixes for local LLM inference). Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Contributor
📝 WalkthroughWalkthroughThis PR adds background prefetch transcription during recording, integrates it into ChangesPrefetch Transcription Feature
Estimated code review effort: 3 (Moderate) | ~25 minutes Sequence Diagram(s)sequenceDiagram
participant Recorder
participant AppState
participant PrefetchTranscriber
participant RealtimeService
participant TranscriptionService
participant FileService
AppState->>AppState: startPrefetchTranscriberIfEnabled()
AppState->>PrefetchTranscriber: init(TranscriptionService)
Recorder->>PrefetchTranscriber: appendPCM16(samples)
PrefetchTranscriber->>TranscriptionService: transcribe(fileURL) per chunk
AppState->>RealtimeService: final realtime transcript
alt realtime succeeds
RealtimeService-->>AppState: transcript
else realtime fails
AppState->>PrefetchTranscriber: commitAndAwaitFinal()
PrefetchTranscriber-->>AppState: merged transcript or empty
alt prefetch empty
AppState->>FileService: fallback transcription
FileService-->>AppState: transcript
end
end
AppState->>AppState: tearDownPrefetchTranscriber()
Possibly related PRs
🚥 Pre-merge checks | ✅ 5✅ Passed checks (5 passed)
✨ Finishing Touches🧪 Generate unit tests (beta)
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
![coderabbitai[bot]](https://avatars.githubusercontent.com/in/347564?s=60&v=4){kind=small}
Contributor
There was a problem hiding this comment.
Actionable comments posted: 2
Caution
Some comments are outside the diff and can’t be posted inline due to platform limitations.
⚠️ Outside diff range comments (1)
Sources/AppState.swift (1)
2637-2639: 🔒 Security & Privacy | 🟠 Major | ⚡ Quick winCancel prefetch uploads during teardown.
prefetchTranscriber = nilonly drops the reference; any already-started chunk task can keep uploading audio after cancellation/failure paths. Mirror realtime teardown by addingPrefetchTranscriber.cancel()and calling it from teardown/defer paths.Proposed direction
private func tearDownRealtimeService() { audioRecorder.onPCM16Samples = nil realtimeService?.cancel() realtimeService = nil - prefetchTranscriber = nil // discard any in-progress accumulation + prefetchTranscriber?.cancel() + prefetchTranscriber = nil // discard any in-progress accumulation } @@ private func tearDownPrefetchTranscriber() { audioRecorder.onPCM16Samples = nil + prefetchTranscriber?.cancel() prefetchTranscriber = nil }// In PrefetchTranscriber func cancel() { let task = lock.withLock { buffer.removeAll() let task = chainTail chainTail = Task { nil } return task } task.cancel() }Also applies to: 2900-2923
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@Sources/AppState.swift` around lines 2637 - 2639, The teardown path only nils out prefetch state, so any in-flight chunk upload can continue running after cancellation or failure. Add a cancel capability to PrefetchTranscriber, using its existing state/lock and chainTail handling to stop pending work and clear buffered audio. Then call that cancel method from the same teardown/defer paths that already cancel activeRealtime so both realtime and prefetch uploads are shut down consistently.
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In `@Sources/PrefetchTranscriber.swift`:
- Around line 59-61: `commitAndAwaitFinal()` in `PrefetchTranscriber` is
returning before all collected chunk results are guaranteed to be appended,
because the collector task writes to the merged transcript outside the serial
chain while `appendPCM16(_:)` only protects its own work. Move the
result-appending logic into the same serial chain behind `chainTail` (and keep
all `chainTail` mutations under the same lock/ordering used by
`appendPCM16(_:)`) so `commitAndAwaitFinal()` can safely await the full
transcript before returning.
- Around line 100-109: The transcribeWAV(_:) path in PrefetchTranscriber is
swallowing chunk upload/transcription failures by returning nil, which lets
callers skip failed segments and can produce a partial merged transcript. Update
PrefetchTranscriber to track any chunk failure during commitAndAwaitFinal() and
surface an explicit failure or empty result instead of silently dropping that
chunk, so AppState.resolveRawTranscript() falls back to the full-file
transcription when any prefetch chunk fails.
---
Outside diff comments:
In `@Sources/AppState.swift`:
- Around line 2637-2639: The teardown path only nils out prefetch state, so any
in-flight chunk upload can continue running after cancellation or failure. Add a
cancel capability to PrefetchTranscriber, using its existing state/lock and
chainTail handling to stop pending work and clear buffered audio. Then call that
cancel method from the same teardown/defer paths that already cancel
activeRealtime so both realtime and prefetch uploads are shut down consistently.
🪄 Autofix (Beta)
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: defaults
Review profile: CHILL
Plan: Pro Plus
Run ID: 4f668f48-5d67-403e-bb5b-18e5c9f6e727
📒 Files selected for processing (3)
Sources/AppState.swiftSources/PrefetchTranscriber.swiftSources/SettingsView.swift
Comment on lines
+100
to
+109
| private func transcribeWAV(_ wav: Data) async -> String? { | ||
| let url = FileManager.default.temporaryDirectory | ||
| .appendingPathComponent("prefetch_\(UUID().uuidString).wav") | ||
| do { | ||
| try wav.write(to: url) | ||
| defer { try? FileManager.default.removeItem(at: url) } | ||
| return try await service.transcribe(fileURL: url) | ||
| } catch { | ||
| return nil | ||
| } |
Contributor
There was a problem hiding this comment.
🎯 Functional Correctness | 🟠 Major | ⚡ Quick win
Do not silently accept partial transcripts after chunk failures.
Line 107 turns any upload/transcription failure into nil, and callers skip that segment. Since AppState.resolveRawTranscript() returns any non-empty merged prefetch result, one failed 28-second chunk can silently drop dictated audio instead of falling back to the full file.
Track chunk failures and return an empty result or explicit failure from commitAndAwaitFinal() so AppState uses the full-file fallback.
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
In `@Sources/PrefetchTranscriber.swift` around lines 100 - 109, The
transcribeWAV(_:) path in PrefetchTranscriber is swallowing chunk
upload/transcription failures by returning nil, which lets callers skip failed
segments and can produce a partial merged transcript. Update PrefetchTranscriber
to track any chunk failure during commitAndAwaitFinal() and surface an explicit
failure or empty result instead of silently dropping that chunk, so
AppState.resolveRawTranscript() falls back to the full-file transcription when
any prefetch chunk fails.
Each 28-second chunk is appended to a timestamped .txt file in the audio directory immediately after transcription — a crash loses at most one in-flight chunk rather than the entire session. Prefetch transcription now defaults to enabled on first launch so background transcription starts without requiring a manual settings toggle. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
.txtfile in the audio directory immediately after transcription — an app crash loses at most one 28-second chunkHow it works
A new
PrefetchTranscriberclass accumulates the raw PCM16 audio stream (AudioRecorder.onPCM16Samples, 24 kHz mono). Every 28 seconds it writes a temporary WAV file and submits it to the configuredTranscriptionServiceendpoint in a serial background task chain. When recording stops,commitAndAwaitFinal()waits for any in-flight chunk, transcribes the remaining tail, and returns the joined transcript. If the merged result is empty (network error, etc.) it falls back to the regular full-file upload path.Each chunk's transcript is appended via
FileHandleto a timestamped.txtfile in~/Library/Application Support/FreeFlow Dev/audio/. In a 3-hour session the transcriber holds at most ~2 × 1.35 MB of PCM16 in RAM at any time (one chunk being transcribed + partial next chunk buffer).The existing realtime WebSocket mode (
realtimeStreamingEnabled) takes priority when both settings are enabled, so the two paths are non-conflicting.Files changed
PrefetchTranscriber.swiftAppState.swiftprefetchTranscriptionEnabledproperty (default: on); recording start/stop/cancel wiring; durable save-URL generationSettingsView.swiftRelates to #250 (URLSession timeout fixes for local LLM inference).
Summary by CodeRabbit