Skip to content

Commit fd62e5f

Browse files
committed
feat(keyboard): report the return key in keyboard status
`keyboard status` now answers what the visible keyboard's return key offers, on both platforms. Android parses the focused editor's `imeOptions` action out of the `dumpsys input_method` output the visibility read already fetched, so no extra adb call is made; `IME_FLAG_NO_ENTER_ACTION` outranks the action bits, and an app's `actionLabel` is reported beside its action kind. iOS gains a runner status command that reads the same return control `keyboard enter` presses, so the two commands cannot disagree about the same key. Both readings project into one vocabulary (`KEYBOARD_RETURN_KEYS` in the contracts package): `none` is only Android's to establish, `unknown` says a key is there this read cannot name, and a hidden keyboard reports neither field. Snapshot output and its device calls are unchanged. - Apple: `keyboardStatus` runner command (read-only, app-windows surface), `RunnerKeyboardStatus` - Android: `parseAndroidReturnKey` in `device-input-state.ts` - Daemon: projects `returnKey`/`returnKeyLabel`; a visibility-only owner publishes `unknown` - Output: `Return key:` and `Return key label:` lines, plus the JSON fields
1 parent cb33595 commit fd62e5f

41 files changed

Lines changed: 1081 additions & 121 deletions

Some content is hidden

Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.

‎apple/runner/AgentDeviceRunner/AgentDeviceRunnerUITests/RunnerKeyboardReturnKey.swift‎

Lines changed: 5 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -16,9 +16,11 @@ import Foundation
1616
/// `Emergency Call` places a call, which no caller of this should ever report as a submit. Every
1717
/// other Apple type has a case above.
1818
///
19-
/// `probeLabels` is the order `findKeyboardReturnControl(app:)` presses in. It is the runner's own
20-
/// vocabulary, so it is not derived from the cases: it carries the two spellings of the plain
21-
/// return key, in the order that found a hittable control before this rule existed.
19+
/// `probeLabels` is the order `findKeyboardReturnControl(app:)` presses in. It is the
20+
/// runner's own vocabulary, so it is not derived from the cases: it carries the two spellings of the
21+
/// plain return key, in the order that found a hittable control before this rule existed. The
22+
/// `keyboard status` read does not use it: it reads the keyboard's own key elements, so it needs no
23+
/// English to find the return key.
2224
enum RunnerKeyboardReturnKey: String {
2325
case go
2426
case search
Lines changed: 87 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,87 @@
1+
import Foundation
2+
import CoreGraphics
3+
4+
/// One `keyboard status` answer: whether the software keyboard is on screen and, when it is, what
5+
/// its return key offers.
6+
///
7+
/// `returnKey` is a `RunnerKeyboardReturnKey` raw value, or `nil` when the keyboard is hidden — a
8+
/// hidden keyboard has no return key to report, which is a different fact from one this read cannot
9+
/// name. `returnKeyLabel` is the wording the key itself carries, with surrounding whitespace
10+
/// trimmed, so a localized or an app's custom key stays readable even where the kind is `unknown`.
11+
/// It is present only when the read saw a key to read it from; a keyboard that exposes no readable
12+
/// key answers `unknown` with no label.
13+
struct RunnerKeyboardStatus: Equatable {
14+
let visible: Bool
15+
let returnKey: String?
16+
let returnKeyLabel: String?
17+
}
18+
19+
extension RunnerKeyboardStatus {
20+
/// One control the status read observed on the visible keyboard: the wording it carries in the
21+
/// accessibility tree, and the frame it occupies. `frame` decides which control is the
22+
/// keyboard's bottom-trailing key, which is where Apple's keyboards place the return key.
23+
struct ObservedKeyboardKey {
24+
let text: String?
25+
let frame: CGRect
26+
}
27+
28+
/// Builds the answer from what the runner observed. The keyboard decides whether a return key is
29+
/// reported at all: a hidden one has none, which is a different fact from one this read cannot
30+
/// name. Only Android's `IME_ACTION_NONE` can establish that a field offers no action key, so
31+
/// `none` is never this runner's answer.
32+
///
33+
/// The key's own wording decides the kind, through the one rule `keyboard enter` probes with
34+
/// (`RunnerKeyboardReturnKey.kind(of:)`), so the two commands can never disagree about the same
35+
/// key. The label itself is kept as the accessibility tree carries it apart from its surrounding
36+
/// whitespace, so a localized or an app's custom key stays readable even where the kind is
37+
/// `unknown`.
38+
static func observed(
39+
visible: Bool,
40+
returnControlText: String?
41+
) -> RunnerKeyboardStatus {
42+
guard visible else {
43+
return RunnerKeyboardStatus(visible: false, returnKey: nil, returnKeyLabel: nil)
44+
}
45+
guard let returnControlText else {
46+
return RunnerKeyboardStatus(
47+
visible: true,
48+
returnKey: RunnerKeyboardReturnKey.unknown.rawValue,
49+
returnKeyLabel: nil
50+
)
51+
}
52+
let label = returnControlText.trimmingCharacters(in: .whitespacesAndNewlines)
53+
return RunnerKeyboardStatus(
54+
visible: true,
55+
returnKey: RunnerKeyboardReturnKey.kind(of: label).rawValue,
56+
returnKeyLabel: label.isEmpty ? nil : label
57+
)
58+
}
59+
60+
/// Builds the answer from every control the read saw on the visible keyboard, so it does not need to
61+
/// read English to find the return key. A control whose wording `kind(of:)` names a kind for is the
62+
/// return key on any keyboard in any language, and its wording is the label. When none maps — a
63+
/// fully localized keyboard — the return key is the measurable control in the keyboard's
64+
/// bottom-trailing corner: the slot Apple's keyboards give it, beside and right of the space bar,
65+
/// which no letter, shift or delete key reaches. A keyboard's padding elements report an empty
66+
/// frame and never win on position. An unmatched key's wording is still reported, with kind
67+
/// `unknown`: that is what the label is for. Only a keyboard exposing no worded, measurable control
68+
/// answers `unknown` with no label.
69+
static func observed(
70+
visible: Bool,
71+
keyboardKeys: [ObservedKeyboardKey]
72+
) -> RunnerKeyboardStatus {
73+
guard visible else {
74+
return RunnerKeyboardStatus(visible: false, returnKey: nil, returnKeyLabel: nil)
75+
}
76+
let worded = keyboardKeys.filter { key in
77+
!(key.text ?? "").trimmingCharacters(in: .whitespacesAndNewlines).isEmpty
78+
}
79+
if let named = worded.first(where: { RunnerKeyboardReturnKey.kind(of: $0.text ?? "") != .unknown }) {
80+
return observed(visible: true, returnControlText: named.text)
81+
}
82+
let returnKey = worded
83+
.filter { !$0.frame.isEmpty }
84+
.max { lhs, rhs in (lhs.frame.maxY, lhs.frame.maxX) < (rhs.frame.maxY, rhs.frame.maxX) }
85+
return observed(visible: true, returnControlText: returnKey?.text)
86+
}
87+
}

‎apple/runner/AgentDeviceRunner/AgentDeviceRunnerUITests/RunnerSurfacePolicy.swift‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -16,7 +16,7 @@ extension CommandType {
1616
return .none
1717
case .tap, .longPress, .drag, .type, .swipe, .scroll, .findText, .querySelector, .readText,
1818
.snapshot, .backInApp, .gesture, .gestureViewport, .appSwitcher,
19-
.keyboardDismiss, .sequence:
19+
.keyboardDismiss, .keyboardStatus, .sequence:
2020
return .appWindows
2121
case .screenshot, .recordStart:
2222
return .observedScreen

‎apple/runner/AgentDeviceRunner/AgentDeviceRunnerUITests/RunnerTests+CommandExecution.swift‎

Lines changed: 17 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -636,6 +636,23 @@ extension RunnerTests {
636636
wasVisible: result.wasVisible
637637
)
638638
)
639+
case .keyboardStatus:
640+
var status: RunnerKeyboardStatus?
641+
withBoundedInteractionIdleTimeoutIfSupported(activeApp, waits: .bothSkipped) {
642+
status = keyboardStatus(app: activeApp)
643+
}
644+
guard let result = status else {
645+
return Response(ok: false, error: ErrorPayload(message: "keyboardStatus produced no result"))
646+
}
647+
return Response(
648+
ok: true,
649+
data: DataPayload(
650+
message: "keyboardStatus",
651+
visible: result.visible,
652+
keyboardReturnKey: result.returnKey,
653+
keyboardReturnKeyLabel: result.returnKeyLabel
654+
)
655+
)
639656
case .alert:
640657
let action = (command.action ?? "get").lowercased()
641658
let deadline = alertDeadline ?? Date().addingTimeInterval(

‎apple/runner/AgentDeviceRunner/AgentDeviceRunnerUITests/RunnerTests+Keyboard.swift‎

Lines changed: 37 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -201,6 +201,43 @@ extension RunnerTests {
201201
return visibleKeyboardFrame(app: app) != nil
202202
}
203203

204+
/// What `keyboard status` answers with: the keyboard's visibility and, when it is up, the return
205+
/// key the focused field offers. It reads the keyboard's own key elements rather than looking a
206+
/// control up by English name, so a localized keyboard (a return key reading `szukaj`) is reported
207+
/// with its own wording instead of a label-less `unknown`. `RunnerKeyboardStatus` decides which
208+
/// observed control is the return key, through the same `kind(of:)` rule `keyboard enter` probes
209+
/// with, so the two commands can never disagree about the wording they can name.
210+
func keyboardStatus(app: XCUIApplication) -> RunnerKeyboardStatus {
211+
guard isKeyboardVisible(app: app) else {
212+
return RunnerKeyboardStatus.observed(visible: false, returnControlText: nil)
213+
}
214+
return RunnerKeyboardStatus.observed(visible: true, keyboardKeys: observedKeyboardKeys(app: app))
215+
}
216+
217+
/// Every control the visible keyboard exposes with its wording and frame. The keyboard is another
218+
/// process, so every element read is a round trip: one pass over its keys and one over its buttons
219+
/// is what a read can afford, and it is the same two queries a snapshot's keyboard band makes. An
220+
/// element that answers no wording is dropped — it is neither a kind nor a label. Off iOS nothing
221+
/// is queried and nothing is reported, so a caller must not read the empty answer as a keyboard
222+
/// with no return key; `keyboard status` is not advertised there.
223+
func observedKeyboardKeys(app: XCUIApplication) -> [RunnerKeyboardStatus.ObservedKeyboardKey] {
224+
#if os(iOS)
225+
let keyboard = app.keyboards.firstMatch
226+
let elements: [XCUIElement] =
227+
safely("KEYBOARD_STATUS_KEYS", [XCUIElement]()) { keyboard.keys.allElementsBoundByIndex }
228+
+ safely("KEYBOARD_STATUS_BUTTONS", [XCUIElement]()) { keyboard.buttons.allElementsBoundByIndex }
229+
return elements.compactMap { element in
230+
guard let label = safely("KEYBOARD_STATUS_KEY_LABEL", { element.label }) else { return nil }
231+
let text = label.trimmingCharacters(in: .whitespacesAndNewlines)
232+
guard !text.isEmpty else { return nil }
233+
let frame = safely("KEYBOARD_STATUS_KEY_FRAME", CGRect.zero) { element.frame }
234+
return RunnerKeyboardStatus.ObservedKeyboardKey(text: label, frame: frame)
235+
}
236+
#else
237+
return []
238+
#endif
239+
}
240+
204241
func dismissKeyboard(app: XCUIApplication) -> RunnerKeyboardDismissOutcome {
205242
let keyboard = app.keyboards.firstMatch
206243
let wasVisible = isKeyboardVisible(app: app)

‎apple/runner/AgentDeviceRunner/AgentDeviceRunnerUITests/RunnerTests+Models.swift‎

Lines changed: 8 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -26,6 +26,7 @@ enum CommandType: String, Codable, CaseIterable {
2626
case volumeDown
2727
case keyboardDismiss
2828
case keyboardReturn
29+
case keyboardStatus
2930
case alert
3031
case sequence
3132
case gesture
@@ -300,7 +301,7 @@ extension Command {
300301
.keyboardDismiss, .keyboardReturn, .sequence, .gesture:
301302
return .interaction
302303

303-
case .findText, .readText, .gestureViewport:
304+
case .findText, .readText, .gestureViewport, .keyboardStatus:
304305
return .appRead(retainsJournalResponseJson: true)
305306

306307
case .snapshot:
@@ -536,6 +537,12 @@ struct DataPayload: Codable {
536537
var wasVisible: Bool?
537538
var dismissed: Bool?
538539
var keyboardDismissMechanism: String?
540+
/// The kind of return key the visible keyboard offers, on a `keyboard status` answer. Absent when
541+
/// the keyboard is hidden, which is not the same claim as a key this read could not name.
542+
var keyboardReturnKey: String?
543+
/// The wording that return key carries, verbatim, so a localized or an app's custom key stays
544+
/// readable. Absent when the key carries none, or there is no key to read.
545+
var keyboardReturnKeyLabel: String?
539546
var orientation: String?
540547
var gestureFallback: String?
541548
var gestureFallbackMessage: String?

‎apple/runner/AgentDeviceRunner/AgentDeviceRunnerUITests/UnitTests/RunnerTests+KeyboardReturnKeyTests.swift‎

Lines changed: 139 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -82,5 +82,144 @@ extension RunnerTests {
8282
XCTAssertEqual(RunnerKeyboardReturnKey.kind(of: label), expected[label], label)
8383
}
8484
}
85+
86+
/// What a `keyboard status` read builds from each observation. The visible keyboard's return
87+
/// control decides both fields: its wording names the kind through the same rule `keyboard enter`
88+
/// probes with, and it is kept as the accessibility tree carries it apart from surrounding
89+
/// whitespace, so a key this read cannot name is still readable.
90+
func testKeyboardStatusReportsTheObservedReturnControl() {
91+
let status = RunnerKeyboardStatus.observed(visible: true, returnControlText: " Search ")
92+
XCTAssertEqual(status, RunnerKeyboardStatus(visible: true, returnKey: "search", returnKeyLabel: "Search"))
93+
}
94+
95+
/// The kinds `kind(of:)` names but the press list never probes are reported by the read, because
96+
/// the read looks at the keyboard's own keys rather than looking a control up by English name.
97+
func testKeyboardStatusReportsAKindThePressProbeListOmits() {
98+
for (label, kind) in [("Route", "route"), ("Continue", "continue")] {
99+
XCTAssertEqual(
100+
RunnerKeyboardStatus.observed(visible: true, returnControlText: label),
101+
RunnerKeyboardStatus(visible: true, returnKey: kind, returnKeyLabel: label),
102+
label
103+
)
104+
XCTAssertFalse(
105+
RunnerKeyboardReturnKey.probeLabels.contains(label),
106+
"\(label) must stay out of the press probe list, which the read does not use"
107+
)
108+
}
109+
}
110+
111+
/// A hidden keyboard reports no return key at all. That is a different fact from one it could not
112+
/// name, so both fields stay out rather than answering `unknown` for a key nobody can press. The
113+
/// keyboard's keys are ignored: nothing is being submitted to.
114+
func testKeyboardStatusReportsNoReturnKeyWhileTheKeyboardIsHidden() {
115+
XCTAssertEqual(
116+
RunnerKeyboardStatus.observed(visible: false, returnControlText: "Search"),
117+
RunnerKeyboardStatus(visible: false, returnKey: nil, returnKeyLabel: nil)
118+
)
119+
XCTAssertEqual(
120+
RunnerKeyboardStatus
121+
.observed(visible: false, keyboardKeys: [RunnerKeyboardStatus.ObservedKeyboardKey(text: "Done", frame: CGRect(x: 0, y: 0, width: 40, height: 40))]),
122+
RunnerKeyboardStatus(visible: false, returnKey: nil, returnKeyLabel: nil)
123+
)
124+
}
125+
126+
/// The English keyboard, as the accessibility tree carries it: the search field's return key reads
127+
/// `Search` in the bottom-trailing slot. The picker names the kind from its wording.
128+
func testKeyboardStatusPicksTheReturnKeyFromTheKeyboardKeysByWording() {
129+
let keys = observedKeyboardKeysFixture([
130+
("Q", 5, 959, 39), ("W", 44, 959, 40), ("shift", 5, 1067, 51), ("Delete", 348, 1067, 52),
131+
("Numbers", 5, 1121, 49), ("Emoji", 54, 1121, 49), ("Space", 103, 1121, 197),
132+
("Search", 301, 1121, 99), ("Padding-Right", 0, 952, 0),
133+
])
134+
XCTAssertEqual(
135+
RunnerKeyboardStatus.observed(visible: true, keyboardKeys: keys),
136+
RunnerKeyboardStatus(visible: true, returnKey: "search", returnKeyLabel: "Search")
137+
)
138+
}
139+
140+
/// The whole point of reading the keyboard's own keys: a fully localized keyboard names no kind, so
141+
/// the return key is the bottom-trailing control and its wording is reported even though the kind
142+
/// is `unknown`. These are the frames this read measured on a Polish iOS keyboard, where `szukaj`
143+
/// sits beside a wide space bar in the last row and the letter, shift and delete keys are higher or
144+
/// further left. The headless test IME, which exposes keys with no wording at all, is the same
145+
/// shape with nothing to report.
146+
func testKeyboardStatusReportsALocalizedReturnKeyWithItsOwnWording() {
147+
let keys = observedKeyboardKeysFixture([
148+
("Padding-Left", 0, 952, 0), ("Q", 5, 959, 39), ("P", 360, 959, 40), ("shift", 5, 1067, 51),
149+
("M", 301, 1067, 40), ("usuń", 348, 1067, 52), ("cyfry", 5, 1121, 49), ("Emoji", 54, 1121, 49),
150+
(" ", 103, 1121, 197), ("szukaj", 301, 1121, 99),
151+
])
152+
XCTAssertEqual(
153+
RunnerKeyboardStatus.observed(visible: true, keyboardKeys: keys),
154+
RunnerKeyboardStatus(visible: true, returnKey: "unknown", returnKeyLabel: "szukaj")
155+
)
156+
}
157+
158+
/// The picker takes a control whose wording names a kind even when another control sits further
159+
/// down-right, so a keyboard that localizes some keys and not others is still named by wording.
160+
/// Padding elements, which carry no usable frame, never win on position.
161+
func testKeyboardStatusPrefersNamedWordingOverKeyboardGeometry() {
162+
let keys = observedKeyboardKeysFixture([
163+
("Done", 5, 959, 39), ("ż", 301, 1121, 99), ("Padding-Right", 0, 952, 0),
164+
])
165+
XCTAssertEqual(
166+
RunnerKeyboardStatus.observed(visible: true, keyboardKeys: keys),
167+
RunnerKeyboardStatus(visible: true, returnKey: "done", returnKeyLabel: "Done")
168+
)
169+
}
170+
171+
/// Only a visible keyboard that exposes no control worth reporting answers `unknown` with no label:
172+
/// an empty layout, a keyboard whose keys all carry no wording (a headless test IME), and one whose
173+
/// only wording sits on a zero-frame padding element.
174+
func testKeyboardStatusReportsUnknownForAVisibleKeyboardWithNoReadableKey() {
175+
let expected = RunnerKeyboardStatus(visible: true, returnKey: "unknown", returnKeyLabel: nil)
176+
XCTAssertEqual(RunnerKeyboardStatus.observed(visible: true, keyboardKeys: []), expected)
177+
XCTAssertEqual(
178+
RunnerKeyboardStatus.observed(
179+
visible: true,
180+
keyboardKeys: observedKeyboardKeysFixture([(" ", 301, 1121, 99)])
181+
),
182+
expected
183+
)
184+
XCTAssertEqual(
185+
RunnerKeyboardStatus.observed(
186+
visible: true,
187+
keyboardKeys: observedKeyboardKeysFixture([("szukaj", 0, 952, 0)])
188+
),
189+
expected
190+
)
191+
}
192+
193+
/// The press path is unchanged by what the read can see: it still looks its own labels up in its
194+
/// own order, and the read never consults that list.
195+
func testKeyboardEnterProbeListStillOwnsItsOrder() {
196+
XCTAssertEqual(
197+
RunnerKeyboardReturnKey.probeLabels,
198+
["return", "Return", "Enter", "Go", "Search", "Next", "Done", "Send", "Join"]
199+
)
200+
}
201+
202+
/// A key whose wording the rule does not recognize keeps its wording, which is the whole point of
203+
/// reporting the label: a localized or a custom key stays readable where the kind is `unknown`.
204+
func testKeyboardStatusKeepsAnUnrecognizedLabelVerbatim() {
205+
for label in ["Send Message", "搜索", "完了"] {
206+
let status = RunnerKeyboardStatus.observed(visible: true, returnControlText: label)
207+
XCTAssertEqual(status.returnKey, "unknown", label)
208+
XCTAssertEqual(status.returnKeyLabel, label)
209+
}
210+
}
211+
212+
/// Builds a keyboard's observed controls from the frames this read measured; a zero width is an
213+
/// element with no usable frame, such as the padding entries a keyboard exposes.
214+
private func observedKeyboardKeysFixture(
215+
_ entries: [(text: String, x: CGFloat, y: CGFloat, width: CGFloat)]
216+
) -> [RunnerKeyboardStatus.ObservedKeyboardKey] {
217+
entries.map { entry in
218+
RunnerKeyboardStatus.ObservedKeyboardKey(
219+
text: entry.text,
220+
frame: CGRect(x: entry.x, y: entry.y, width: entry.width, height: entry.width > 0 ? 54 : 0)
221+
)
222+
}
223+
}
85224
}
86225
#endif

0 commit comments

Comments
 (0)