Skip to content

feat(keyboard): report the return key in keyboard status - #4602

Open
thymikee wants to merge 1 commit into
mainfrom
feat/keyboard-status-return-key
Open

thymikee wants to merge 1 commit into
mainfrom
feat/keyboard-status-return-key

Conversation

@thymikee

@thymikee thymikee commented Nov 16, 2026 •

Copy link
Copy Markdown
Member

Summary

keyboard status now answers what the visible keyboard's return key offers, on both platforms.

Before / after, same command:

$ agent-device keyboard status          # iOS, before: UNSUPPORTED_OPERATION
Keyboard visible: yes                   # after
Return key: done
Return key label: done

$ agent-device keyboard status --json   # Android, Settings search field
{ "platform": "android", "action": "status", "visible": true,
  "returnKey": "search", "inputType": "0x300001", "inputOwner": "app" }

Android parses the focused editor's imeOptions out of the dumpsys input_method output the visibility read already fetched, so no extra adb call; IME_FLAG_NO_ENTER_ACTION outranks the action bits, with the app's actionLabel beside it. iOS gains a read-only keyboardStatus runner command that reads the keyboard's own key elements, so a localized return key keeps its wording; keyboard enter's probe order is unchanged. Both project into one vocabulary (KEYBOARD_RETURN_KEYS): only Android establishes none; unknown means a key this read cannot name; a hidden keyboard reports neither field. Limrun iOS now serves status too.

Touched files: 41.

Validation

Tested at fd62e5fd79. pnpm typecheck, format, lint, generate:capabilities green; affected unit-core projects: 521 files / 5270 tests. Swift runner unit tests, built with AGENT_DEVICE_XCUITEST_INCLUDE_UNIT_TESTS=1: 16 cases, 0 failures. No local check:affected receipt for this head (the branch moved while it waited on the host validation lock), so gates are CI's.

Device evidence (helper 0.21.38-dev, androidSnapshot.backend: android-helper):

  • iOS Simulator 26.2, at this head: Settings search → search/search; Polish keyboard → unknown, label szukaj.
  • Android emulator, Android 17: Settings search → search; tester email → done.

@github-actions

github-actions Bot commented Nov 16, 2026 •

Copy link
Copy Markdown
Contributor

Size Report

Metric Base Current Diff
Installed (including dependencies) 6.33 MB 6.34 MB +8.5 kB
Package (unpacked) 6.33 MB 6.34 MB +8.5 kB
Package (download) 1.92 MB 1.92 MB +2.5 kB

Startup median (7 runs, lower is better):

Scenario Base Current Diff
CLI --version 31.1 ms 32.7 ms +1.6 ms
CLI --help 71.2 ms 76.8 ms +5.5 ms

@cubic-dev-ai cubic-dev-ai Bot left a comment •

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

All reported issues were addressed across 40 files

You’re at about 96% of the monthly reviewed-line limit. You may want to disable incremental reviews to conserve quota. Reviews will continue until that limit is exceeded. If you need help avoiding interruptions, please contact contact@cubic.dev.

Reply with feedback, questions, or to request a fix.

View guided diff | Turn on auto-fix | Re-trigger cubic

Comment thread website/docs/docs/commands.md Resolved
Comment thread packages/contracts/src/keyboard.ts Resolved
@apex-sentinel-staging

apex-sentinel-staging Bot commented Nov 16, 2026 •

Copy link
Copy Markdown

Apex Sentinel · ✅ Approve

Since the last review: 0 new · 0 still open · 1 no longer raised (1 marked fixed)

No open findings.

1 finding below minor hidden (user_settings.min_severity).

Reviewed with this repository's own Sentinel configuration (ea414582b7fa).

💬 Talking to Sentinel

In a finding's thread, reply with:

Command What it does
@apex-sentinel-staging note <context> Adds context; Sentinel re-checks the cited code
@apex-sentinel-staging pivot <why> Says the finding is wrong; Sentinel re-checks the cited code
@apex-sentinel-staging like Marks the finding as useful
@apex-sentinel-staging dislike <why> Marks it as not useful, so it is not raised again

On this conversation, comment @apex-sentinel-staging review to run a full review once again.

Earlier reviews

Review of dbe52010 · 16.11.2026 17:31 UTC

Apex Sentinel · ✅ Approve

Since the last review: 0 new · 1 still open · 1 no longer raised

1 finding(s): 1 minor.

🟡 Minor (1)

Where Finding
apple/runner/AgentDeviceRunner/AgentDeviceRunnerUITests/RunnerTests+Keyboard.swift:207 iOS return-key label can only ever be one of nine English probe labels

2 findings below minor hidden (user_settings.min_severity).

Reviewed with this repository's own Sentinel configuration (cb335952017e).

Review of 3a4d56c3 · 16.11.2026 16:56 UTC

Apex Sentinel · ✅ Approve

2 finding(s): 2 minor.

🟡 Minor (2)

Where Finding
apple/runner/AgentDeviceRunner/AgentDeviceRunnerUITests/RunnerTests+Keyboard.swift:207 iOS return-key read can only name the nine probe labels
apple/runner/AgentDeviceRunner/AgentDeviceRunnerUITests/UnitTests/RunnerTests+ModelsTests.swift:272 Merge-base exemption now skips witness assertion for volumeUp/volumeDown

Reviewed with this repository's own Sentinel configuration (b3c1a3c8abd9).

@apex-sentinel-staging apex-sentinel-staging Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Apex Sentinel: 2 new findings on 3a4d56c. The verdict and the full list are in the summary comment.

@thymikee

Copy link
Copy Markdown
Member Author

The keyboard help text describes unknown wrongly, so this needs one fix before merge (reviewed at 3a4d56c).

The help string at https://github.com/callstack/agent-device/blob/3a4d56c/packages/command-surface/src/system/index.ts#L322 says unknown means "a key is there that this read cannot name". On iOS, a visible keyboard with no return control (number pad, phone pad) or with a key outside the probe labels also answers unknown. Agents that read this help will assume a submit key exists and try keyboard enter on a number pad. The same claim is open in the commands.md thread, so fixing only the website leaves the help agents read. Please state one rule on every surface that describes unknown (the help string, commands.md, the capabilities note and the contracts doc in keyboard.ts): unknown means the read could not name the return key, including when it found none, and only Android can establish none.

Not blocking, take or leave: the iOS status read classifies the element's localized label instead of the matched probe label or identifier, so a Polish "szukaj" key answers unknown (RunnerTests+Keyboard.swift:213); route and continue in KEYBOARD_RETURN_KEYS have no producer, so the vocabulary could match what producers emit (packages/contracts/src/keyboard.ts:19); and the daemon defaults a visible keyboard's missing return key to unknown, which could move into the WebDriver provider with the field required on visible results (keyboard-runtime.ts:156).

The open inline threads still apply. The review threads on visionOS eligibility (#4602 (comment)), comma truncation of the action label (#4602 (comment)) and the ModelsTests early return (#4602 (comment)) hold, as do the threads on the none type (#4602 (comment)) and the help wording (#4602 (comment)). Four lower-priority threads also still apply.

The live device runs are reported in the PR body. The checks show 16 of 17 passing, and the one pending is a review bot that does not exercise this code. There are no conflicts. Before merge, please fix the unknown wording on every surface and resolve the holding threads.

@thymikee
thymikee added this pull request to stack #4606 November 16, 2026 17:21
Base automatically changed from refactor/apple-runner-return-key-lookup to main November 16, 2026 17:22
@thymikee
thymikee force-pushed the feat/keyboard-status-return-key branch 5 times, most recently from 6c78740 to 5981d97 Compare November 16, 2026 18:22
`keyboard status` now answers what the visible keyboard's return key offers, on both
platforms. Android parses the focused editor's `imeOptions` action out of the `dumpsys
input_method` output the visibility read already fetched, so no extra adb call is made;
`IME_FLAG_NO_ENTER_ACTION` outranks the action bits, and an app's `actionLabel` is reported
beside its action kind. iOS gains a runner status command that reads the same return control
`keyboard enter` presses, so the two commands cannot disagree about the same key.

Both readings project into one vocabulary (`KEYBOARD_RETURN_KEYS` in the contracts package):
`none` is only Android's to establish, `unknown` says a key is there this read cannot name,
and a hidden keyboard reports neither field. Snapshot output and its device calls are
unchanged.

- Apple: `keyboardStatus` runner command (read-only, app-windows surface), `RunnerKeyboardStatus`
- Android: `parseAndroidReturnKey` in `device-input-state.ts`
- Daemon: projects `returnKey`/`returnKeyLabel`; a visibility-only owner publishes `unknown`
- Output: `Return key:` and `Return key label:` lines, plus the JSON fields
@thymikee
thymikee force-pushed the feat/keyboard-status-return-key branch from 5981d97 to fd62e5f Compare November 16, 2026 18:23
@thymikee

Copy link
Copy Markdown
Member Author

Review round addressed at fd62e5fd79 — 10 threads, each replied to and resolved.

Contracts and platform mapping

  • The Apple keyboard status variant excludes returnKey: 'none', which only Android's IME_ACTION_NONE reading establishes; the Apple mapper accepts the shared vocabulary minus that word and degrades anything else to unknown.
  • keyboard status is gated on the handheld Apple leaf. visionOS previously would have answered "keyboard hidden" for a keyboard that may be up, because the runner's visibility read is #if os(iOS). dismiss and enter keep their existing eligibility.
  • The owner matrix now names each owner's legs from its own declarations: Android and the Apple runner three (the Apple runner on iOS/iPadOS), HarmonyOS dismiss/enter, WebDriver all three on Android and dismiss only on iOS.

iOS status read no longer needs English
It reads the keyboard's own key elements: a control whose wording kind(of:) maps is the return key in any language; otherwise the bottom-trailing measurable key is, the slot Apple's keyboards give it. Its trimmed wording is always reported, so the Polish keyboard answers unknown with returnKeyLabel: "szukaj" on-device, and the unknown docs now say what each platform's read actually proved. findKeyboardReturnControl/probeLabels are unchanged press-only code.

Android
A custom actionLabel keeps its commas and stops at the next key=value field, }/], or end of line, so Save, now is no longer truncated to Save.

Help, docs and tests
Help text and the output schema each have their own assertion over the twelve returnKey words; returnKeyLabel is described as trimmed everywhere users read it. The merge-base witness exemption is now per assertion and covers only keyboardStatus, so volumeUp/volumeDown are back in the witness column (they were never in the replay column).

Also: the Swift trailing-comma guard CI flagged is fixed — that syntax needs Swift 6.1+ — verified across apple/runner.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant