Make dictation aware of the text surrounding the cursor (fixes #200)#267
Make dictation aware of the text surrounding the cursor (fixes #200)#267swimmer-303 wants to merge 1 commit into
Conversation
Spacing, capitalization, and punctuation were context-blind: dictating into the middle of a sentence produced a leading capital, a trailing period, and jammed against the existing text. Now the focused element's text around the insertion point (200 chars before, 80 after, read via AXStringForRange with an AXValue-slice fallback; secure fields are never read) is captured with the app context and: - flows into the post-processing prompt with rules to continue the sentence flow at the insertion point - drives a deterministic separating space when pasting directly after a word character or closing punctuation (skipped in Edit Mode) - suppresses the auto trailing space when the text after the cursor already provides separation Fixes zachlatta#200 (cherry picked from commit 500da9a)
📝 WalkthroughWalkthroughDictation now reads text around the cursor, includes that context in cleanup prompts, and applies cursor-aware leading and trailing spacing when pasting transcripts. Documentation and changelog entries describe the updated insertion behavior. ChangesCursor-aware dictation insertion
Estimated code review effort: 4 (Complex) | ~45 minutes Possibly related PRs
🚥 Pre-merge checks | ✅ 4 | ❌ 1❌ Failed checks (1 warning)
✅ Passed checks (4 passed)
✨ Finishing Touches🧪 Generate unit tests (beta)
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
![coderabbitai[bot]](https://avatars.githubusercontent.com/in/347564?s=60&v=4){kind=small}
There was a problem hiding this comment.
Actionable comments posted: 1
🧹 Nitpick comments (2)
Sources/AppState.swift (1)
3208-3211: 📐 Maintainability & Code Quality | 🔵 Trivial | ⚡ Quick winDoc comment undersells the actual join set.
The comment says "after a word character or closing punctuation," but the
joinableset on Line 3224 also includes.!?,:;(sentence/clause punctuation, not closing punctuation). The behavior itself is reasonable (e.g. inserting after a comma with no existing space), but the comment should describe it accurately to avoid confusing future maintainers.📝 Suggested comment fix
- /// Prepends a space when the dictation is inserted directly after a - /// word character or closing punctuation, so mid-text insertions do not - /// jam against the existing text (issue `#200`). Skipped for Edit Mode, - /// where the pasted text replaces the selection in place. + /// Prepends a space when the dictation is inserted directly after a + /// word character, sentence/clause punctuation (.!?,:;), or closing + /// punctuation, so mid-text insertions do not jam against the existing + /// text (issue `#200`). Skipped for Edit Mode, where the pasted text + /// replaces the selection in place.🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@Sources/AppState.swift` around lines 3208 - 3211, Update the doc comment above the dictation insertion logic to accurately state that a space is prepended when inserting after a word character or punctuation in the joinable set, including sentence and clause punctuation such as .!?,:;. Keep the existing behavior and use the joinable set near the insertion logic as the reference.Sources/AppContextService.swift (1)
409-422: 🚀 Performance & Scalability | 🔵 TrivialFallback path reads the entire field value (up to 1M chars) synchronously via AX.
When
kAXStringForRangeParameterizedAttributeisn't supported (common for non-native/Electron-style apps),fallbackSlicereads the wholekAXValueAttributevalue — up tocursorContextMaxFallbackLength(1,000,000) characters — for both the before- and after-cursor reads. For apps with large documents this could add noticeable latency to context capture, which is awaited before pasting completes.🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@Sources/AppContextService.swift` around lines 409 - 422, The fallback context path can synchronously fetch up to one million characters from AX, delaying paste completion. Update fallbackSlice and its callers to avoid reading the full kAXValueAttribute for large fields—prefer a bounded range/context query or skip the fallback when the value exceeds a smaller safe limit, while preserving UTF-16 range semantics and existing behavior for reasonably sized values.
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In `@Sources/AppContextService.swift`:
- Around line 360-372: The secure-field guard in cursorTextContext(from:) only
checks AXRole, so secure text fields identified by AXSubrole can pass through.
Retrieve AXSubrole from the focused element and return nil when it equals
AXSecureTextField, alongside the existing role check.
---
Nitpick comments:
In `@Sources/AppContextService.swift`:
- Around line 409-422: The fallback context path can synchronously fetch up to
one million characters from AX, delaying paste completion. Update fallbackSlice
and its callers to avoid reading the full kAXValueAttribute for large
fields—prefer a bounded range/context query or skip the fallback when the value
exceeds a smaller safe limit, while preserving UTF-16 range semantics and
existing behavior for reasonably sized values.
In `@Sources/AppState.swift`:
- Around line 3208-3211: Update the doc comment above the dictation insertion
logic to accurately state that a space is prepended when inserting after a word
character or punctuation in the joinable set, including sentence and clause
punctuation such as .!?,:;. Keep the existing behavior and use the joinable set
near the insertion logic as the reference.
🪄 Autofix (Beta)
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: defaults
Review profile: CHILL
Plan: Pro Plus
Run ID: b7ff9704-92df-405d-9a13-a7f91dd55d60
📒 Files selected for processing (5)
CHANGELOG.mdREADME.mdSources/AppContextService.swiftSources/AppState.swiftSources/PostProcessingService.swift
| func cursorTextContext(from appElement: AXUIElement) -> CursorTextContext? { | ||
| guard let focused = accessibilityElement(from: appElement, attribute: kAXFocusedUIElementAttribute as CFString) else { | ||
| return nil | ||
| } | ||
| if let role = accessibilityRawString(from: focused, attribute: kAXRoleAttribute as CFString), | ||
| role == "AXSecureTextField" { | ||
| return nil | ||
| } | ||
| guard let selectedRange = accessibilityRange(from: focused, attribute: kAXSelectedTextRangeAttribute as CFString), | ||
| selectedRange.location != kCFNotFound, | ||
| selectedRange.location >= 0 else { | ||
| return nil | ||
| } |
There was a problem hiding this comment.
🔒 Security & Privacy | 🟡 Minor | ⚡ Quick win
🧩 Analysis chain
🏁 Script executed:
#!/bin/bash
set -euo pipefail
# Inspect the target file around the cited lines and locate related accessibility helpers.
git ls-files 'Sources/AppContextService.swift' 'Sources/**/*.swift' | sed -n '1,120p'
echo '---'
ast-grep outline Sources/AppContextService.swift --view expanded
echo '---'
rg -n "AXSecureTextField|kAXSubroleAttribute|kAXRoleAttribute|cursorTextContext\\(|selectedRange" Sources -SRepository: zachlatta/freeflow
Length of output: 3576
🏁 Script executed:
#!/bin/bash
set -euo pipefail
sed -n '352,392p' Sources/AppContextService.swift
echo '---'
rg -n "CursorTextContext|cursorTextContext\\(|selectedText\\(|rawSelectedText\\(|inferActivityWithLLM|context prompt|LLM" Sources/AppContextService.swift Sources -S
echo '---'
web_search
Apple Accessibility secure text field AXRole AXSubrole NSSecureTextField password fieldRepository: zachlatta/freeflow
Length of output: 6196
🏁 Script executed:
#!/bin/bash
set -euo pipefail
sed -n '440,505p' Sources/AppContextService.swift
echo '---'
rg -n "AXSubrole|AXSecureTextField|secure field|password" Sources -S
echo '---'
sed -n '120,170p' Sources/AppContextService.swift
echo '---'
sed -n '180,290p' Sources/AppContextService.swiftRepository: zachlatta/freeflow
Length of output: 9497
🌐 Web query:
Apple Accessibility Inspector NSSecureTextField AXRole AXSubrole password field AXSecureTextFieldSubrole
💡 Result:
In macOS accessibility, NSSecureTextField is identified by the role AXTextField (using the constant NSAccessibilityTextFieldRole) and is distinguished from standard text fields by its subrole, which is AXSecureTextField (using the constant NSAccessibilitySecureTextFieldSubrole) [1][2][3][4]. The Accessibility Inspector tool allows developers to query the accessibility hierarchy, including these specific roles and subroles, to verify that UI elements are correctly identified by assistive technologies [5]. Key technical details regarding these constants include: - AXRole: For secure text fields, the standard role is AXTextField [2][4]. The role attribute provides the basic type of the accessibility object for programmatic identification [6][7][8]. - AXSubrole: The subrole AXSecureTextField is used to provide more specific information beyond the basic role, signaling to screen readers and other assistive tools that the input is sensitive (e.g., a password field) [6][1][4][8]. - Implementation: When creating custom accessibility wrappers, developers should map the protected state of a text field to this subrole [4]. Assistive technologies use this subrole to announce the element as a "Secure Text Field" [9]. These constants are fundamental to the Accessibility API on macOS, ensuring consistent behavior across different applications and web content [4][10].
Citations:
- 1: https://docs.rs/accessibility-sys/latest/accessibility_sys/constant.kAXSecureTextFieldSubrole.html
- 2: https://github.com/nkmr-jp/prompt-line/blob/1fb7cd16/native/text-field-detector.swift
- 3: https://github.com/tmandry/AXSwift/blob/master/Sources/Constants.swift
- 4: https://chromium.googlesource.com/chromium/src/+/df5ab817cb455c368ef3aeb147b2908d8247922b/ui/accessibility/platform/ax_platform_node_mac.mm
- 5: https://developer.apple.com/documentation/accessibility/accessibility-inspector
- 6: https://leopard-adc.pepas.com/documentation/Accessibility/Reference/AccessibilityLowlevel/AXAttributeConstants_h/CompositePage.html
- 7: https://leopard-adc.pepas.com/documentation/Cocoa/Conceptual/Accessibility/cocoaAXOverview/cocoaAXOverview.html
- 8: https://github.com/phracker/MacOSX-SDKs/blob/master/MacOSX10.7.sdk/System/Library/Frameworks/ApplicationServices.framework/Versions/A/Frameworks/HIServices.framework/Versions/A/Headers/AXAttributeConstants.h
- 9: https://github.com/cashapp/AccessibilitySnapshot/pull/327/files
- 10: https://github.com/WebKit/webkit/blob/master/Source/WebCore/accessibility/mac/WebAccessibilityObjectWrapperMac.mm
🏁 Script executed:
#!/bin/bash
set -euo pipefail
sed -n '1,80p' Sources/AppContextService.swift
echo '---'
rg -n "textBeforeCursor|textAfterCursor|cursorTextContext|AppContext\\(" Sources -SRepository: zachlatta/freeflow
Length of output: 5386
🏁 Script executed:
#!/bin/bash
set -euo pipefail
sed -n '3200,3265p' Sources/AppState.swift
echo '---'
sed -n '2700,2750p' Sources/AppState.swift
echo '---'
sed -n '1660,1705p' Sources/SettingsView.swiftRepository: zachlatta/freeflow
Length of output: 8084
🌐 Web query:
Safari Chrome password input AXRole AXSubrole AXSecureTextFieldSubrole accessibility macOS
💡 Result:
On macOS, password input fields in both Safari and Chrome are exposed to the accessibility API using a specific combination of role and subrole to ensure assistive technologies correctly identify them as secure input areas [1]. The standard mapping for an HTML <input type="password"> element on macOS is [1]: AXRole: AXTextField [1] AXSubrole: AXSecureTextField [1] (often represented by the constant kAXSecureTextFieldSubrole [2][3]) AXRoleDescription: "secure text field" [1] Browsers implement this by mapping their internal accessibility node properties to these macOS-specific values. For instance, in Chromium, the accessibility platform node layer (AXPlatformNodeCocoa) checks if a text field node has a "protected" state; if it does, it returns AXSecureTextFieldSubrole as the subrole [4]. Similarly, WebKit (the engine behind Safari) maintains logic within its AXObjectCache to handle secure text fields, ensuring that proper notifications and attributes are exposed to the system [5][6][7]. This consistent mapping allows screen readers like VoiceOver to announce the field as a "secure text field" and treat it appropriately, such as masking the input characters for privacy [8][1].
Citations:
- 1: https://github.com/Igalia/rfcs/blob/wpt-for-aams-rfc/rfcs/wpt-for-aams.md
- 2: https://docs.rs/accessibility-sys/latest/accessibility_sys/constant.kAXSecureTextFieldSubrole.html
- 3: https://sosumi.ai/documentation/appkit/nsaccessibility-swift.struct/subrole/securetextfield
- 4: https://chromium.googlesource.com/chromium/src/+/d6d648b6dc54c1c78936dc506e5ccebe23cee7a3/ui/accessibility/platform/ax_platform_node_cocoa.mm
- 5: AX: Fix for accessibility/password-selected-text-range.html in ITM. WebKit/WebKit#44487
- 6: https://github.com/WebKit/WebKit/blob/main/Source/WebCore/accessibility/AccessibilityRenderObject.cpp
- 7: https://github.com/WebKit/WebKit/blob/46334f674a96c704eabffdc76013c8bf0754c142/Source/WebCore/accessibility/mac/AXObjectCacheMac.mm
- 8: https://github.com/cashapp/AccessibilitySnapshot/pull/327/files
Check the secure-field subrole too. NSSecureTextField and browser password fields report AXRole = AXTextField plus AXSubrole = AXSecureTextField, so this role-only guard misses them. Add the subrole check so secure cursor context stays out of this path.
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
In `@Sources/AppContextService.swift` around lines 360 - 372, The secure-field
guard in cursorTextContext(from:) only checks AXRole, so secure text fields
identified by AXSubrole can pass through. Retrieve AXSubrole from the focused
element and return nil when it equals AXSecureTextField, alongside the existing
role check.
1 participant
Split out from #254.
Dictating into the middle of a sentence used to produce a stray leading capital, a trailing period, and text jammed against existing words. FreeFlow now reads up to 200 chars before / 80 after the insertion point (via
AXStringForRangewith anAXValue-slice fallback; secure fields are never read) and uses it three ways:Fixes #200
🤖 Generated with Claude Code
Summary by CodeRabbit
New Features
Documentation