Add ElevenLabs Scribe transcription#257
Conversation
📝 WalkthroughWalkthroughThis PR introduces provider selection for transcription, adding ElevenLabs Scribe v2 as an alternative to the existing OpenAI-compatible backend. Changes span batch transcription (TranscriptionService), realtime streaming (RealtimeTranscriptionService), AppState persistence/wiring, AudioRecorder sample-rate handling, Settings/Setup UI, and README documentation. ChangesElevenLabs Scribe transcription provider support
Estimated code review effort: 4 (Complex) | ~60 minutes Sequence Diagram(s)sequenceDiagram
participant User
participant AppState
participant AudioRecorder
participant RealtimeClient as RealtimeTranscriptionClient
participant TranscriptionService
User->>AppState: startRecording()
AppState->>AppState: transcriptionConfigurationErrorMessage()
alt config missing
AppState-->>User: show "Missing API Key" alert
else config valid
AppState->>RealtimeClient: start() (OpenAI-compatible or ElevenLabs)
AudioRecorder->>AppState: onPCM16Samples(data)
AppState->>RealtimeClient: appendPCM16(data)
AppState->>RealtimeClient: commitAndAwaitFinal()
RealtimeClient-->>AppState: partial transcript
AppState->>TranscriptionService: transcribe(fileURL, provider)
TranscriptionService-->>AppState: final transcript
AppState-->>User: display transcript
end
Possibly related PRs
🚥 Pre-merge checks | ✅ 4 | ❌ 1❌ Failed checks (1 warning)
✅ Passed checks (4 passed)
✨ Finishing Touches🧪 Generate unit tests (beta)
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
|
By the way, this was fully generated using AI. I’ve been looking forward to this feature for a long time, and I hope it’s helpful for you. |
![coderabbitai[bot]](https://avatars.githubusercontent.com/in/347564?s=60&v=4){kind=small}
There was a problem hiding this comment.
Actionable comments posted: 3
🧹 Nitpick comments (2)
Sources/SettingsView.swift (1)
320-398: 📐 Maintainability & Code Quality | 🔵 Trivial | ⚡ Quick winConsider consolidating the per-provider ternary duplication.
The provider-conditional title/placeholder/binding/focus/onSubmit/Clear logic is repeated ~6 times inline. Extracting small computed properties (e.g.
activeAPIKeyBinding,activeAPIKeyFocus,commitActiveAPIKey()) would reduce duplication and make future provider additions easier.🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@Sources/SettingsView.swift` around lines 320 - 398, The provider-specific API key UI in SettingsView is duplicating the same ternary logic across the SecureField, focus handling, submit action, and Clear button. Refactor this section by introducing small computed helpers in SettingsView such as activeAPIKeyBinding, activeAPIKeyFocus, activeAPIKeyPlaceholder, and commitActiveAPIKey() so the view uses one shared path for the selectedTranscriptionProvider, reducing repetition and making future provider additions simpler.README.md (1)
41-41: 📐 Maintainability & Code Quality | 🔵 Trivial | ⚡ Quick winAlign README wording with actual UI labels.
Two small mismatches between the docs and the actual Settings UI:
- Line 41 says "ElevenLabs Scribe v2" but the picker option (and line 87) is labeled "ElevenLabs Scribe".
- Line 87 says "expand Providers," but "Providers" is the static card title — the control users need to expand is the "Provider Settings" disclosure group inside it.
📝 Proposed wording fix
-- **Provider choices:** Use Groq by default, select ElevenLabs Scribe v2 for transcription, or configure OpenAI-compatible model IDs and API URLs in settings. +- **Provider choices:** Use Groq by default, select ElevenLabs Scribe for transcription, or configure OpenAI-compatible model IDs and API URLs in settings.-FreeFlow can use ElevenLabs Scribe v2 for speech-to-text while continuing to use your OpenAI-compatible provider for cleanup, Edit Mode, and context. Open Settings, expand Providers, choose ElevenLabs Scribe as the transcription provider, and enter an ElevenLabs API key. Realtime streaming uses Scribe v2 Realtime when the realtime toggle is enabled. +FreeFlow can use ElevenLabs Scribe v2 for speech-to-text while continuing to use your OpenAI-compatible provider for cleanup, Edit Mode, and context. Open Settings, expand Provider Settings under Providers, choose ElevenLabs Scribe as the transcription provider, and enter an ElevenLabs API key. Realtime streaming uses Scribe v2 Realtime when the realtime toggle is enabled.Also applies to: 85-88
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@README.md` at line 41, The README text does not match the actual Settings UI labels, so update the provider documentation wording to mirror the UI exactly. In the section describing transcription/model choices, change the “ElevenLabs Scribe v2” reference to “ElevenLabs Scribe” to match the picker option, and replace any instruction to expand “Providers” with “Provider Settings” to match the disclosure group name inside the Providers card. Keep the wording consistent in the README entries around the provider setup guidance so users can follow the same labels they see in the UI.
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In `@Sources/AppState.swift`:
- Around line 1019-1025: In makeTranscriptionService, the ElevenLabs path is
still using the shared transcriptionModel default instead of the ElevenLabs
batch model. Update the TranscriptionService initialization so that when
transcriptionProvider is ElevenLabs it passes the ElevenLabs-specific Scribe v2
batch model, while keeping the existing transcriptionModel for other providers.
Use the makeTranscriptionService and TranscriptionService symbols to locate the
change.
In `@Sources/RealtimeTranscriptionService.swift`:
- Around line 505-559: The audio send path is not serialized, so `appendPCM16`
and `commitAndAwaitFinal` can race and let the final commit overtake earlier
chunks. Update `RealtimeTranscriptionService` so both
`sendAudioChunk(_:commit:)` calls flow through a single serial send mechanism,
using the existing `stateQueue`/task path or a dedicated send queue, and ensure
queued PCM chunks are fully sent before the `commit: true` message. Preserve the
current chunking logic in `appendPCM16` and the finalization flow in
`commitAndAwaitFinal`, but make `URLSessionWebSocketTask.send` happen in order
from one path only.
In `@Sources/SettingsView.swift`:
- Around line 236-244: The new Picker in SettingsView is missing an
accessibility name, so VoiceOver cannot identify it. Update the Transcription
Provider Picker in SettingsView to include an explicit
.accessibilityLabel("Transcription Provider"), matching the pattern used by the
Transcription Language picker, and keep the existing .labelsHidden() so the
visual layout stays unchanged.
---
Nitpick comments:
In `@README.md`:
- Line 41: The README text does not match the actual Settings UI labels, so
update the provider documentation wording to mirror the UI exactly. In the
section describing transcription/model choices, change the “ElevenLabs Scribe
v2” reference to “ElevenLabs Scribe” to match the picker option, and replace any
instruction to expand “Providers” with “Provider Settings” to match the
disclosure group name inside the Providers card. Keep the wording consistent in
the README entries around the provider setup guidance so users can follow the
same labels they see in the UI.
In `@Sources/SettingsView.swift`:
- Around line 320-398: The provider-specific API key UI in SettingsView is
duplicating the same ternary logic across the SecureField, focus handling,
submit action, and Clear button. Refactor this section by introducing small
computed helpers in SettingsView such as activeAPIKeyBinding, activeAPIKeyFocus,
activeAPIKeyPlaceholder, and commitActiveAPIKey() so the view uses one shared
path for the selectedTranscriptionProvider, reducing repetition and making
future provider additions simpler.
🪄 Autofix (Beta)
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: defaults
Review profile: CHILL
Plan: Pro Plus
Run ID: 552125ef-bf0f-480f-becd-599628abad9c
📒 Files selected for processing (7)
README.mdSources/AppState.swiftSources/AudioRecorder.swiftSources/RealtimeTranscriptionService.swiftSources/SettingsView.swiftSources/SetupView.swiftSources/TranscriptionService.swift
| func makeTranscriptionService() throws -> TranscriptionService { | ||
| try TranscriptionService( | ||
| provider: transcriptionProvider, | ||
| apiKey: resolvedTranscriptionAPIKey, | ||
| baseURL: resolvedTranscriptionBaseURL, | ||
| transcriptionModel: transcriptionModel, | ||
| language: resolvedTranscriptionLanguage |
There was a problem hiding this comment.
🎯 Functional Correctness | 🟠 Major | ⚡ Quick win
Use the ElevenLabs batch model when the ElevenLabs provider is selected.
Line 1024 passes the shared transcriptionModel, which defaults to whisper-large-v3; that sends the OpenAI-compatible default into the ElevenLabs client instead of Scribe v2.
🐛 Proposed fix
+ private var resolvedTranscriptionModel: String {
+ switch transcriptionProvider {
+ case .openAICompatible:
+ return transcriptionModel
+ case .elevenLabs:
+ return TranscriptionService.defaultElevenLabsModel
+ }
+ }
+
func makeTranscriptionService() throws -> TranscriptionService {
try TranscriptionService(
provider: transcriptionProvider,
apiKey: resolvedTranscriptionAPIKey,
baseURL: resolvedTranscriptionBaseURL,
- transcriptionModel: transcriptionModel,
+ transcriptionModel: resolvedTranscriptionModel,
language: resolvedTranscriptionLanguage
)
}📝 Committable suggestion
‼️ IMPORTANT
Carefully review the code before committing. Ensure that it accurately replaces the highlighted code, contains no missing lines, and has no issues with indentation. Thoroughly test & benchmark the code to ensure it meets the requirements.
| func makeTranscriptionService() throws -> TranscriptionService { | |
| try TranscriptionService( | |
| provider: transcriptionProvider, | |
| apiKey: resolvedTranscriptionAPIKey, | |
| baseURL: resolvedTranscriptionBaseURL, | |
| transcriptionModel: transcriptionModel, | |
| language: resolvedTranscriptionLanguage | |
| private var resolvedTranscriptionModel: String { | |
| switch transcriptionProvider { | |
| case .openAICompatible: | |
| return transcriptionModel | |
| case .elevenLabs: | |
| return TranscriptionService.defaultElevenLabsModel | |
| } | |
| } | |
| func makeTranscriptionService() throws -> TranscriptionService { | |
| try TranscriptionService( | |
| provider: transcriptionProvider, | |
| apiKey: resolvedTranscriptionAPIKey, | |
| baseURL: resolvedTranscriptionBaseURL, | |
| transcriptionModel: resolvedTranscriptionModel, | |
| language: resolvedTranscriptionLanguage |
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
In `@Sources/AppState.swift` around lines 1019 - 1025, In
makeTranscriptionService, the ElevenLabs path is still using the shared
transcriptionModel default instead of the ElevenLabs batch model. Update the
TranscriptionService initialization so that when transcriptionProvider is
ElevenLabs it passes the ElevenLabs-specific Scribe v2 batch model, while
keeping the existing transcriptionModel for other providers. Use the
makeTranscriptionService and TranscriptionService symbols to locate the change.
| func appendPCM16(_ data: Data) { | ||
| guard !data.isEmpty else { return } | ||
| var chunks: [Data] = [] | ||
| stateQueue.sync { | ||
| guard task != nil, !commitSent, !closed else { return } | ||
| pendingAudio.append(data) | ||
| let targetBytes = Self.byteCount( | ||
| seconds: targetChunkSeconds, | ||
| sampleRate: pcmSampleRate, | ||
| bytesPerSample: bytesPerSample | ||
| ) | ||
| let heldBytes = Self.byteCount( | ||
| seconds: heldChunkSeconds, | ||
| sampleRate: pcmSampleRate, | ||
| bytesPerSample: bytesPerSample | ||
| ) | ||
| while pendingAudio.count >= targetBytes + heldBytes { | ||
| chunks.append(Data(pendingAudio.prefix(targetBytes))) | ||
| pendingAudio.removeFirst(targetBytes) | ||
| } | ||
| } | ||
| for chunk in chunks { | ||
| sendAudioChunk(chunk, commit: false) | ||
| } | ||
| } | ||
|
|
||
| func commitAndAwaitFinal() async throws -> String { | ||
| let currentTask: URLSessionWebSocketTask? = stateQueue.sync { | ||
| task | ||
| } | ||
| guard currentTask != nil else { | ||
| throw RealtimeTranscriptionError.notConnected | ||
| } | ||
|
|
||
| let finalChunk: Data? = stateQueue.sync { | ||
| if commitSent { return nil } | ||
| commitSent = true | ||
| let chunk: Data | ||
| if pendingAudio.isEmpty && hasSentAudio { | ||
| chunk = Data() | ||
| } else if pendingAudio.isEmpty { | ||
| chunk = Data(repeating: 0, count: Self.byteCount( | ||
| seconds: heldChunkSeconds, | ||
| sampleRate: pcmSampleRate, | ||
| bytesPerSample: bytesPerSample | ||
| )) | ||
| } else { | ||
| chunk = pendingAudio | ||
| } | ||
| pendingAudio.removeAll(keepingCapacity: false) | ||
| return chunk | ||
| } | ||
| if let finalChunk { | ||
| sendAudioChunk(finalChunk, commit: true) | ||
| } |
There was a problem hiding this comment.
🎯 Functional Correctness | 🟠 Major | 🏗️ Heavy lift
🧩 Analysis chain
🏁 Script executed:
set -euo pipefail
# Map the file structure first.
ast-grep outline Sources/RealtimeTranscriptionService.swift --view expanded
# Show the relevant line ranges with context.
nl -ba Sources/RealtimeTranscriptionService.swift | sed -n '1,260p'
echo '---'
nl -ba Sources/RealtimeTranscriptionService.swift | sed -n '260,760p'Repository: zachlatta/freeflow
Length of output: 2415
🏁 Script executed:
set -euo pipefail
# Read the relevant sections with line numbers using tools available in the sandbox.
sed -n '505,730p' Sources/RealtimeTranscriptionService.swift | cat -n
echo '---'
sed -n '730,820p' Sources/RealtimeTranscriptionService.swift | cat -nRepository: zachlatta/freeflow
Length of output: 12777
🏁 Script executed:
set -euo pipefail
# Check whether this file already serializes websocket sends through a dedicated queue/actor.
rg -n "sendQueue|serialize|DispatchQueue|actor|URLSessionWebSocketTask\.send|sendAudioChunk|appendPCM16|commitAndAwaitFinal" Sources/RealtimeTranscriptionService.swift
# Inspect the first class's send implementation and the second class' surrounding state.
sed -n '1,420p' Sources/RealtimeTranscriptionService.swift | cat -nRepository: zachlatta/freeflow
Length of output: 18174
🌐 Web query:
URLSessionWebSocketTask send concurrent calls ordering guarantee documentation
💡 Result:
Apple's official documentation for URLSessionWebSocketTask does not explicitly define a formal ordering guarantee for concurrent calls to the send(_:completionHandler:) method [1][2]. While there is no explicit guarantee for message sending, it is worth noting that for the sendPing(pongReceiveHandler:) method, Apple explicitly documents that the task guarantees the pongReceiveHandler is called in the same order that the pings were sent [3]. In practice, because URLSessionWebSocketTask manages a single underlying network connection, developers typically serialize outgoing messages manually to ensure the integrity of the WebSocket stream and the order of messages [2][4]. Relying on concurrent calls to send without manual synchronization can lead to unpredictable behavior, as the underlying implementation may not guarantee the delivery order of those specific concurrent requests. If you require strict ordering for your application, it is recommended to implement a serial queue or an async/await actor to serialize your send calls before they reach the task [5][6].
Citations:
- 1: https://developer.apple.com/documentation/foundation/urlsessionwebsockettask/send(_:completionhandler:)
- 2: https://developer.apple.com/documentation/foundation/urlsessionwebsockettask
- 3: https://developer.apple.com/documentation/foundation/urlsessionwebsockettask/sendping(pongreceivehandler:)
- 4: https://sosumi.ai/documentation/foundation/urlsessionwebsockettask
- 5: https://developer.apple.com/forums/thread/698065
- 6: https://developer.apple.com/forums/thread/653599
🌐 Web query:
URLSessionWebSocketTask send ordering concurrency guarantee Apple documentation
💡 Result:
Apple's documentation for URLSessionWebSocketTask does not explicitly define a concurrency guarantee for the internal ordering of send operations, though it is established that the WebSocket protocol itself, running over TCP, provides in-order delivery of data [1][2][3]. Because URLSessionWebSocketTask is a message-oriented protocol layered on TCP, messages sent sequentially over the same task should be delivered to the server in the order they were sent, provided the underlying TCP connection remains stable [4][5][2][3]. However, developers should note the following technical context: 1. Asynchronous Completion Handlers: The send(_:completionHandler:) method performs writes asynchronously [6][5]. While the messages themselves follow the TCP stream order, the completion handlers are called when the operation completes, which may not strictly guarantee the order in which these completion callbacks are executed if multiple send operations are initiated concurrently without awaiting the completion of the previous one. 2. Protocol-Level Ordering: WebSocket protocol guarantees the order of messages because it operates over a single TCP connection, which inherently preserves stream ordering [1][2][3]. 3. Implementation Responsibility: If your application logic requires strict serial processing (where send B must not be triggered until send A is confirmed), you must explicitly manage this synchronization in your code, for example, by awaiting the completion handler of each send operation before initiating the next [1]. In summary, while the underlying WebSocket transport ensures the ordering of data transmission, URLSessionWebSocketTask's asynchronous API design requires you to manage the synchronization of send operations if you depend on the order of their completion callbacks or logical execution order [6][5].
Citations:
- 1: Is the order of messages guaranteed? websockets/ws#542
- 2: https://www.sitongpeng.com/writing/websockets-guarantee-order-so-why-are-my-messages-scrambled
- 3: https://stackoverflow.com/questions/11804721/can-websocket-messages-arrive-out-of-order
- 4: https://developer.apple.com/documentation/foundation/urlsessionwebsockettask
- 5: https://sosumi.ai/documentation/foundation/urlsessionwebsockettask
- 6: https://developer.apple.com/documentation/foundation/urlsessionwebsockettask/send(_:completionhandler:)
Serialize audio sends before the final commit. appendPCM16 and commitAndAwaitFinal both call URLSessionWebSocketTask.send independently, so the commit: true message can outrun earlier audio and truncate the transcript. Route both through one serial send path.
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
In `@Sources/RealtimeTranscriptionService.swift` around lines 505 - 559, The audio
send path is not serialized, so `appendPCM16` and `commitAndAwaitFinal` can race
and let the final commit overtake earlier chunks. Update
`RealtimeTranscriptionService` so both `sendAudioChunk(_:commit:)` calls flow
through a single serial send mechanism, using the existing `stateQueue`/task
path or a dedicated send queue, and ensure queued PCM chunks are fully sent
before the `commit: true` message. Preserve the current chunking logic in
`appendPCM16` and the finalization flow in `commitAndAwaitFinal`, but make
`URLSessionWebSocketTask.send` happen in order from one path only.
| VStack(alignment: .leading, spacing: 6) { | ||
| Text("Transcription Provider") | ||
| .font(.caption.weight(.semibold)) | ||
| Picker("", selection: $appState.transcriptionProvider) { | ||
| ForEach(TranscriptionProvider.allCases) { provider in | ||
| Text(provider.displayName).tag(provider) | ||
| } | ||
| } | ||
| ) | ||
| .labelsHidden() |
There was a problem hiding this comment.
📐 Maintainability & Code Quality | 🟡 Minor | ⚡ Quick win
Add an accessibility label to the new Picker.
Unlike the "Transcription Language" picker below (which sets .accessibilityLabel("Transcription Language")), this new picker only has .labelsHidden(), so VoiceOver announces no name for it.
♿️ Proposed fix
Picker("", selection: $appState.transcriptionProvider) {
ForEach(TranscriptionProvider.allCases) { provider in
Text(provider.displayName).tag(provider)
}
}
.labelsHidden()
+ .accessibilityLabel("Transcription Provider")📝 Committable suggestion
‼️ IMPORTANT
Carefully review the code before committing. Ensure that it accurately replaces the highlighted code, contains no missing lines, and has no issues with indentation. Thoroughly test & benchmark the code to ensure it meets the requirements.
| VStack(alignment: .leading, spacing: 6) { | |
| Text("Transcription Provider") | |
| .font(.caption.weight(.semibold)) | |
| Picker("", selection: $appState.transcriptionProvider) { | |
| ForEach(TranscriptionProvider.allCases) { provider in | |
| Text(provider.displayName).tag(provider) | |
| } | |
| } | |
| ) | |
| .labelsHidden() | |
| VStack(alignment: .leading, spacing: 6) { | |
| Text("Transcription Provider") | |
| .font(.caption.weight(.semibold)) | |
| Picker("", selection: $appState.transcriptionProvider) { | |
| ForEach(TranscriptionProvider.allCases) { provider in | |
| Text(provider.displayName).tag(provider) | |
| } | |
| } | |
| .labelsHidden() | |
| .accessibilityLabel("Transcription Provider") |
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
In `@Sources/SettingsView.swift` around lines 236 - 244, The new Picker in
SettingsView is missing an accessibility name, so VoiceOver cannot identify it.
Update the Transcription Provider Picker in SettingsView to include an explicit
.accessibilityLabel("Transcription Provider"), matching the pattern used by the
Transcription Language picker, and keep the existing .labelsHidden() so the
visual layout stays unchanged.
1 participant
Closes #234.
This adds ElevenLabs as a first-class transcription provider without changing the default Groq/OpenAI-compatible path.
What changed:
I also stopped logging failed transcription response bodies from the upload path. Status, host, file name, and byte count are still logged, but provider response JSON is not.
Validation:
make clean && make CODESIGN_IDENTITY=-One small unrelated-looking change is in
AudioRecorder: the local CommandLineTools SDK here does not expose the newerAVCaptureSession.*Notificationnames, so I switched those three notification constants to the olderNSNotification.Name.*spellings. They are the same notifications and keep the local build green.Summary by CodeRabbit
New Features
Bug Fixes