Skip to content

feat(android): add a draft bar for editing cloud notes and dictation results - #1187

Merged
H-Chris233 merged 7 commits into
Open-Less:betafrom
DepengWang:feature/cloud-note-draft-confirm
Oct 19, 2026
Merged

H-Chris233 merged 7 commits into
Open-Less:betafrom
DepengWang:feature/cloud-note-draft-confirm

Conversation

@DepengWang

@DepengWang DepengWang commented Oct 13, 2026 •

Copy link
Copy Markdown
Contributor

Summary

Adds a draft bar to the Android IME: an editable box stacked above the keyboard panel, where a piece of text can be corrected with any input mode (stroke, pinyin, English, clipboard, voice) before it goes anywhere. Two things use it:

  • Cloud notes. Finishing a Cloud notes recording with a tap still posts it straight to the webhook. Swiping up on the mic during that recording ends it and opens the recognized text in the draft bar instead; nothing is sent until Confirm.
  • Editing a dictation result. The ✎ button on the voice panel (and the long-press on the clipboard panel's History key) now opens the text in the draft bar. Confirm writes the edited text back over the original in the host field.

The feature lives in openless-all/app/android/kotlin/OpenLessImeService.kt. The one piece of it that decides something without touching Android — where an edited text's original sits in the host field, and which edit replaces it — is the separate, pure ImeDraftReplacement.kt, with unit tests.

Behavior changes worth a look

These replace or remove existing behavior, so please say if any of them is not wanted:

  • The "speak to fix a word" panel is removed, together with its add-to-dictionary checkbox. ✎ used to open that panel and start recording at once; the replacement could only be dictated. With the draft bar open the mic still works and dictates into the draft at the caret, so voice correction is still possible, just not the only way.
  • The History key's second label on the clipboard panel changes from 字典 / Dict to 编辑 / Edit: its long-press no longer writes to the dictionary.
  • Swipe up during a Cloud notes recording used to switch to Raw and leave Cloud notes. It now ends the recording and opens the draft. Swipe up during an ordinary dictation is unchanged. The hint row reads "云笔记 · 上划编辑后发送" while this applies.

Relation to #1131 and #1146

Rebased onto both. #1131 guards dictation undo and the speak-to-fix whole-result replacement so that a length-based delete can no longer erase neighboring text.

How it works

  • Layout. The bar (draft box, then Cancel / status / Confirm) sits above the regular input view. The keyboard panel keeps its height, header and mode toggle; the IME grows taller by the height of the bar. It is not a dialog, so there is no focus or overlap problem with the mode toggle.
  • Where the keys go. Every key, candidate, backspace and dictation commit in this service writes through currentInputConnection. While a draft is open, getCurrentInputConnection() returns the draft box's own connection, so none of the existing keyboard code changes and nothing can reach the host field. If the draft's connection is unavailable it returns null rather than falling back to the host.
  • Key events. The IME window never has window focus, and the framework drops key events dispatched to an unfocused window. The draft connection therefore turns sendKeyEvent() into text edits itself (Return, Backspace, forward delete, cursor keys).
  • Mode switches. A mode switch rebuilds the whole input view. The draft box is a single instance re-parented into each rebuilt view, so text, caret, selection and scroll position carry over.
  • Confirm. A blank draft is refused. For Cloud notes both buttons are disabled while the request runs, the bar closes on success and stays with the error on failure so the text is not lost. For a dictation edit, the original is looked up in the text around the host caret (at the offset it had when the draft opened if the host reports offsets, otherwise the occurrence closest to the caret), selected and replaced; if it is gone, nothing is touched and the bar says so.
  • Leaving. Cancel, Back, hiding the keyboard, or the input target going away drops the draft. Nothing is sent and the host field is left as it was; after cancelling an edit the undo/redo/✎ controls come back.

submitCloudNoteText() gains an optional completion callback and is otherwise unchanged; the direct-send path calls it exactly as before.

Known limits

  • Long-press in the draft box is disabled, so there is no system copy/paste toolbar there. A tap places the caret; select all, copy and paste are available from the clipboard panel.
  • If the host app restarts input while a draft is open (for example the user taps another field), the draft is dropped.
  • After a dictation edit is confirmed, the host caret ends up after the replaced text, not where it was.
  • When the same text occurs more than once and the host does not report absolute offsets, the occurrence closest to the caret is replaced.
  • With the draft open, a Cloud notes recording finished with a plain tap is still sent directly rather than added to the draft.

Testing

  • ./gradlew :app:compileArm64DebugKotlin on this branch.
  • ./gradlew :app:testArm64DebugUnitTest: 85 tests pass (including fix(android): 听写撤销前确认原文仍在光标处 #1131's DictationSpanTest), 12 of them new in ImeDraftReplacementTest — the caret away from the dictated text (the bug found on the device), a selection, a lost selection found again by its remembered offset, repeated text, a host that reports no offsets, and an original that is gone. Breaking the offset comparison on purpose makes the matching test fail.
  • node --test scripts/*.test.mjs: 68 of 68 pass. android-cursor-context-contract.test.mjs reads the body of submitCloudNoteText() and still finds only the note text in the webhook payload.
  • Installed on a OnePlus CPH2573 and a Xiaomi M2011K2C (debug builds of this code merged onto current beta) and used by me there. The Cloud notes draft flow, including editing with the stroke and English panels and Confirm / Cancel, worked. The swipe-up rule and the "original text has changed" refusal when the host caret was not at the end of the text both came out of that use and are fixed in this branch.

Not tested:

  • The final host-text lookup (the caret fix) has been installed but I have not yet re-run every edit case on a device: caret elsewhere, a selection, repeated text.
  • Hosts that do not implement getExtractedText().
  • No automated tests for the rest: redirecting the input connection, the key-event handling, the bar's layout and the gesture need an input connection and a view tree, and are covered only by the device use above.

🤖 Generated with Claude Code

@H-Chris233 H-Chris233 added area:ime IME area area:input Input area enhancement New feature or request large-change Large or risky change needs-design Needs design discussion needs-review Needs review needs-tests Needs tests P2 Medium priority platform:android Android-specific labels Oct 13, 2026
@DepengWang
DepengWang force-pushed the feature/cloud-note-draft-confirm branch from 080ebce to b1e1ae2 Compare October 14, 2026 06:29
DepengWang and others added 3 commits October 14, 2026 16:28
A Cloud notes recording is still sent straight to the webhook when it
is finished with a tap. Swiping up on the mic while that recording is
in progress ends it and opens the recognized text in a draft bar
instead; nothing is sent until Confirm.

- The draft bar is stacked above the keyboard panel, which keeps its
  height, header and mode toggle, so every input mode can edit the
  draft.
- While the draft is open, getCurrentInputConnection() returns the
  draft box's own connection, so every key, candidate, backspace and
  dictation commit edits the draft and never the host app's field.
- The IME window has no window focus, so key events sent to the draft
  are turned into text edits (Return, Backspace, cursor keys).
- The draft box is one instance re-parented across input-view
  rebuilds, so a mode switch keeps the text, caret and selection.
- Confirm rejects a blank draft, sends once, and keeps the draft on a
  failed submit; Cancel, Back, hiding the keyboard or losing the input
  target drops the draft without a request.

Mid Cloud-notes recording the swipe up no longer switches to Raw.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The edit button on the voice panel, and the long-press on the
clipboard panel's History key, used to open a "speak to fix a word"
panel: the replacement could only be dictated. They now put the text
to fix (the host field's selection, or the whole last dictation
result) into the draft bar, so it can be edited with the stroke,
pinyin and English keyboards as well as by voice.

- A purpose decides whether Confirm posts to the Cloud notes webhook
  or writes back over the original text in the host field.
- The original is looked up in the text around the host caret — at
  the offset it had when the draft opened if the host reports offsets,
  otherwise the occurrence closest to the caret — then selected and
  replaced. If it is no longer there, nothing is touched.
- Cancel restores the undo/redo/edit controls as they were.
- The speak-to-fix panel, its state and its add-to-dictionary checkbox
  are removed; the History key's second label reads Edit, not Dict.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Moves the decision of where the original text sits in the host field,
and which edit replaces it, out of OpenLessImeService into the pure
ImeDraftReplacement, and adds JVM unit tests for it: the caret away
from the dictated text, a selection, a lost selection found again by
its remembered offset, repeated text, a host that reports no offsets,
and an original that is gone.

No behavior change; the service now only reads the host editor and
carries the plan out.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@DepengWang

Copy link
Copy Markdown
Contributor Author

@HKLHaoBin 你好,云笔记草稿栏这个 PR 已变基到 beta(解决了与 #1146 的冲突),CI 已全绿、可合并。麻烦有空时审核确认一下;标签里有 needs-design / needs-tests,如果需要补充设计说明或测试,请告诉我,谢谢!

@HKLHaoBin

Copy link
Copy Markdown
Collaborator

开始审查。

HKLHaoBin and others added 3 commits October 19, 2026 01:01
Prevent stale Cloud Notes callbacks and active dictation sessions from
closing or writing through a newer draft, and fail closed when a remembered
replacement anchor no longer matches the original text.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
HKLHaoBin
HKLHaoBin previously approved these changes Oct 18, 2026
@H-Chris233

Copy link
Copy Markdown
Collaborator

开始审查

@H-Chris233
H-Chris233 merged commit 4c66658 into Open-Less:beta Oct 19, 2026
7 checks passed
DepengWang added a commit to DepengWang/openless that referenced this pull request Oct 19, 2026
… open

With Open-Less#1187's draft bar open, typing goes to the draft box, so an empty
host field must not clear the stroke code row or the associations.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
DepengWang added a commit to DepengWang/openless that referenced this pull request Oct 19, 2026
… open

With Open-Less#1187's draft bar open, typing goes to the draft box, so an empty
host field must not clear the stroke code row or the associations.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
aeoform added a commit to aeoform/openless that referenced this pull request Oct 19, 2026
并入 origin/beta 剩下的 10 个上游 commit(此前分支已合过前 4 个):

- Windows:TSF 组字听写输入(Open-Less#1189)、明文插入模式开关(Open-Less#1182)、
  胶囊显示目标模式(Open-Less#1232)
- Android:arm64 本地 ASR + 云回退/自动切换(Open-Less#1231)、本地配置迁移(Open-Less#1199)、
  草稿条(Open-Less#1187)、悬浮层「永不显示」(Open-Less#1221)、英文键盘上滑符号重排(Open-Less#1188)
- 发布:稳定版直链别名(Open-Less#1198)
- coding-agent:dsh 补丁 YAML 缩进修复(Open-Less#1205)

代码层面零冲突,但本仓较新的依赖版本让上游新代码编译不过,手工收口两处
(都只在 Android 本地迁移包路径上):

1. `zip` 9 vs 2.x —— `ZipFile::name()` 现在返回 `Result<Cow<str>, ZipError>`:
   `.map_err(|_| BundleError::MalformedContainer)?.into_owned()`。
2. `cipher` 0.5(`aes` 钉 0.9.2 带来的版本)把 `XNonce::from_slice` 标成
   deprecated:改用 `&[u8; NONCE_BYTES] -> &XNonce` 的转换。

验证:core 与 linux-egui 全量测试通过、clippy `-D warnings` 干净、egui 四个
JS 契约测试通过、i18n 0 added/0 updated、deb/rpm 打包 + 校验通过
(2.0.2-Beta.1-137)。
DepengWang added a commit to DepengWang/openless that referenced this pull request Oct 22, 2026
… open

With Open-Less#1187's draft bar open, typing goes to the draft box, so an empty
host field must not clear the stroke code row or the associations.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
H-Chris233 added a commit that referenced this pull request Oct 28, 2026
…eed (stroke/English, rebased on Rime) (#1230)

* feat(android-ime): candidate, learning and speed improvements for stroke and English

Rebased onto the Rime pinyin backend (#1196): the LitePinyin changes
(full-pinyin phrase index, combo learning) are dropped; what remains
applies to the stroke and English keyboards and to the pinyin row's
association and candidate plumbing that Rime still uses.

Associations
- Drop the phrase the context already completes from the association
  results, so the row clears once the last association is picked.
- Clear associations (and the stroke code row) when the host empties the
  field, e.g. a chat app's send button.
- Rank associations the user picked before ahead of unpicked ones; typing
  an offered association out by hand counts as picking it (stroke, and
  Rime commits through RimePinyinController's association context).
  Replaces the stroke commit record whose key never matched an
  association lookup.

Stroke
- Mark, digit, space and Return keys drop the half-typed code, its
  segments and the code row before committing; a new input session clears
  the code row and candidates on screen too.
- Keep recording picks of a character that is on top only because of
  earlier picks, so its score stays fresh.

English
- Sentence punctuation replaces the space a candidate pick added.
- A quick double space after a word becomes ". ".
- Capitalize at the start of a sentence, line or field (not in email,
  URL, password or number fields); Shift turns an automatic capital off.

Speed
- Stroke dictionary sorted by frequency once at load; a query puts the
  user's picks first and stops at 36 matches instead of scoring and
  sorting up to ~48k entries (first stroke now 0-2 ms on a OnePlus).
- Usage stores parsed into one process-wide in-memory cache, kept current
  by a preference listener.
- Shift changes relabel the English keys in place instead of rebuilding
  the keyboard; the English/pinyin candidate row reuses its views.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* fix(android-ime): skip the field-emptied check while the draft box is open

With #1187's draft bar open, typing goes to the draft box, so an empty
host field must not clear the stroke code row or the associations.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* fix(android-ime): bind automatic edits to caret and learn split associations

Guard candidate spaces and double-space full stops with editor and selection anchors. Retain typed continuation boundaries so separately committed characters train the originally offered association. Add JVM and Android emulator regressions plus safe CI evidence collection.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

* fix(android-ime): reset pinyin learning context on editor deletion

Invalidate confirmed association text and continuation history when Pinyin Backspace deletes editor text without an active composition. Add an asynchronous service regression for stale-context learning.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

* test(android-ime): initialize Rime before backspace queue deadline

Complete cold dictionary deployment on the instrumentation test thread before exercising the editor fallback, matching the existing Rime integration test setup.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

* fix(android-ime): preserve learning and capitals

---------

Co-authored-by: DepengWang <2818245+DepengWang@users.noreply.github.com>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
Co-authored-by: HKLHaoBin <we3q@qq.com>
Co-authored-by: Chris233 <h-chris233@outlook.com>
H-Chris233 pushed a commit that referenced this pull request Oct 31, 2026
* feat(android-ime): candidate, learning and speed improvements for stroke and English

Rebased onto the Rime pinyin backend (#1196): the LitePinyin changes
(full-pinyin phrase index, combo learning) are dropped; what remains
applies to the stroke and English keyboards and to the pinyin row's
association and candidate plumbing that Rime still uses.

Associations
- Drop the phrase the context already completes from the association
  results, so the row clears once the last association is picked.
- Clear associations (and the stroke code row) when the host empties the
  field, e.g. a chat app's send button.
- Rank associations the user picked before ahead of unpicked ones; typing
  an offered association out by hand counts as picking it (stroke, and
  Rime commits through RimePinyinController's association context).
  Replaces the stroke commit record whose key never matched an
  association lookup.

Stroke
- Mark, digit, space and Return keys drop the half-typed code, its
  segments and the code row before committing; a new input session clears
  the code row and candidates on screen too.
- Keep recording picks of a character that is on top only because of
  earlier picks, so its score stays fresh.

English
- Sentence punctuation replaces the space a candidate pick added.
- A quick double space after a word becomes ". ".
- Capitalize at the start of a sentence, line or field (not in email,
  URL, password or number fields); Shift turns an automatic capital off.

Speed
- Stroke dictionary sorted by frequency once at load; a query puts the
  user's picks first and stops at 36 matches instead of scoring and
  sorting up to ~48k entries (first stroke now 0-2 ms on a OnePlus).
- Usage stores parsed into one process-wide in-memory cache, kept current
  by a preference listener.
- Shift changes relabel the English keys in place instead of rebuilding
  the keyboard; the English/pinyin candidate row reuses its views.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* fix(android-ime): skip the field-emptied check while the draft box is open

With #1187's draft bar open, typing goes to the draft box, so an empty
host field must not clear the stroke code row or the associations.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* fix(android-ime): bind automatic edits to caret and learn split associations

Guard candidate spaces and double-space full stops with editor and selection anchors. Retain typed continuation boundaries so separately committed characters train the originally offered association. Add JVM and Android emulator regressions plus safe CI evidence collection.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

* fix(android-ime): reset pinyin learning context on editor deletion

Invalidate confirmed association text and continuation history when Pinyin Backspace deletes editor text without an active composition. Add an asynchronous service regression for stale-context learning.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

* test(android-ime): initialize Rime before backspace queue deadline

Complete cold dictionary deployment on the instrumentation test thread before exercising the editor fallback, matching the existing Rime integration test setup.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

* feat(android-ime): keyboard data export, fuzzy pinyin and separate learning stores

Keyboard settings page:
- "Export / Import keyboard data": one JSON file with this page's
  preferences (keyboard size, haptics, switches, fuzzy pinyin), the Rime
  pinyin user dictionary, stroke ranking, association learning and the
  cloud notes webhook. Each part is chosen in a checklist; settings are
  replaced, learning data is merged (higher count and later use win).
  It is independent of the migration bundle and cloud sync.
- Fuzzy pinyin: seven pairs from rime-ice's own rules, applied to both
  pinyin schemas through a .custom.yaml patch and a redeploy.

Native Rime (shared with HarmonyOS): user_dict_export / user_dict_import
through librime levers backup/restore, and redeploy. Each closes every
session first; controllers start a new one after a session generation
bump. rime_levers_api.h is copied from librime 1.17.0, whose install step
only ships rime_api.h.

Stroke picks and association picks get separate stores (caps 5000 and
10000, trimmed to 90% in one pass), split once from the shared store.
The migration bundle keeps one combined "strokeFrequency" map.

LitePinyin's backup adapter and stores are removed. The bundle keeps
"pinyinLearnedPhrases" (schema 1 requires it), always empty and ignored.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* fix(android-ime): preserve export scope and serialize Rime maintenance

Restore export choices across activity recreation and reject unsafe dictionary metadata. Import only learned entries so snapshot tombstones cannot delete local words.

Serialize session-closing maintenance with complete input tasks, preserving undelivered text across session recreation. Add snapshot, queue, activity lifecycle and editor retry regressions.

* fix(android-tests): register keyboard settings in debug manifests

CI stages Kotlin without the production application manifest. Register the settings activity in debug builds and launch it explicitly in the target app so the recreation regression can run.

Test the generated manifest through the scaffolding CLI and drain settings dictionary exports before later native-session tests.

* fix(ci): free Android runner disk before emulator restore

Remove unused hosted .NET/Haskell tools and the Rime build cache after JNI libraries and dictionary assets have been staged. Make room for the pinned emulator image without deleting packaged inputs.

Exercise the cleanup command against an isolated fixture and verify it runs after APK compilation and before emulator extraction.

---------

Co-authored-by: DepengWang <2818245+DepengWang@users.noreply.github.com>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
Co-authored-by: HKLHaoBin <we3q@qq.com>
Co-authored-by: Chris Harris <chris23346579@gmail.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

area:ime IME area area:input Input area enhancement New feature or request large-change Large or risky change needs-design Needs design discussion needs-review Needs review needs-tests Needs tests P2 Medium priority platform:android Android-specific

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants