Repository navigation
Description
Request: none
Problem
Feature #1013 lets a lease holder drive its device's screen over the lease's input socket. On an iOS simulator three things the person sends are thrown away:
- hardware keys, on an iOS 27 runtime: the socket says
keyboardUnavailable, so the person can only tap the on-screen keyboard; - the app switcher button;
- the Lock button.
Simlock dropped them on serve-sim's word ("the iOS 27 guest ignores injected keys", "Xcode 26+ drops the injected Home press"), never on a measurement. On 2026-11-17 the helper Simlock already ships sent all three, and they worked on iOS 26.4, 27.0 and 27.2 runtimes (ADR 0085 holds the table).
Rotation, also named when this issue was opened, shipped in #1080 (ADR 0052).
Who it is for
A person taking over an agent's iOS simulator through the lease's input socket, for example in T3 Code's device panel.
Outcome
On an iOS simulator of any runtime, the person types with a hardware keyboard, opens the app switcher, and locks or wakes the screen, as they already do on Android.
How it works for the user
flowchart LR
P[Person in the device panel] -->|types a key| K[key reaches the simulator]
P -->|app switcher button| A[the app switcher opens]
P -->|Lock button| L[the screen locks, or wakes if locked]
P -->|Home| H[home screen, as today]
Before, on an iOS 27 simulator, the socket's screen message:
0x82 {"width":1206,"height":2622,"orientation":"portrait","keyboardUnavailable":true}
After:
0x82 {"width":1206,"height":2622,"orientation":"portrait"}
Examples
| Message on an iOS simulator | Today | After |
|---|---|---|
0x06 key, iOS 26.4 runtime |
typed | typed |
0x06 key, iOS 27.0 or 27.2 runtime |
dropped | typed |
0x06 key, a runtime id that cannot be read |
dropped | typed |
0x04 home |
home screen (simctl) |
same |
0x04 app_switcher |
dropped | the app switcher opens |
0x04 lock |
dropped | the screen locks; on a locked screen, it wakes |
0x04 back |
dropped | dropped |
0x04 on a physical iPhone |
no input at all | same |
What could go wrong
- On a worker with an older Xcode than the ones measured, a key or button could do nothing and nothing says so. Unlikely: Xcode 26.4.1, 27.0 and 27.2 all worked.
- Right after a fresh boot the first key or two may be lost. Seen once in about ten tries.
- An app switcher press that iOS reads as one Home press shows the home screen instead. Not seen in five tries.
Words used
- Input socket:
WS /v1/leases/{id}/input, where a lease holder sends taps, keys and buttons in expo serve-sim's messages. - Helper:
simlock-screen-capture <udid> --input, the vendored program that sends Indigo HID messages into a simulator. - Indigo message: the simulator's private HID message, e.g. a key down for usage
0x04(a). keyboardUnavailable: a flag in the screen message (0x82) that tells the client to hide hardware typing.
Non-goals
- Back on iOS: an iPhone has no Back button.
- Input on a physical iPhone.
- Moving Home off
simctl. - Detecting a press the simulator ignored.
- New CLI or MCP commands for buttons.
Completion conditions
- On a leased iOS 27 simulator, through the input socket: a focused text field receives typed hardware keys,
app_switcheropens the app switcher, andlocklocks the screen. - The socket's
0x82for an iOS simulator never carrieskeyboardUnavailable. - The website's input page says keys, the app switcher and Lock work on every iOS simulator runtime.
Open questions
Decisions
- ADR 0085 — An iOS simulator takes keys on every runtime, and the app switcher and Lock: narrows ADR 0049 §4; fixes how each button is sent.
- Decided in the session: Home stays
simctl launch com.apple.springboard, because it already works and the new presses do not need it. - Decided in the session: keys are not gated by Xcode or runtime version, because every measured Xcode and runtime typed, and a gate would rest on a version nobody showed to fail.
- Decided in the session: the
keyboardUnavailablefield stays in core's0x82shape though no device sets it, because it is serve-sim's message. - Decided in the session: the device check is a slow-lane test that reads typed keys with an agent-device snapshot and the app switcher and Lock with agent-device
appstate(a snapshot would bring the app back), because the simulator's own lock state cannot be read (com.apple.springboard.lockstatestays 0 with no passcode).
Behaviours
| Id | Behaviour (invoker, action, what it sees) | Seam | Cases | Task |
|---|---|---|---|---|
| B1 | A socket client on an iOS simulator sends a key: the helper gets it on every runtime, and the config never says keyboardUnavailable |
createIosDeviceInput (src/drivers/ios/input/device-input.ts), helper faked |
runtime iOS 26.4; iOS 27.0; iOS 28.1; unreadable id garbage; config after a foldable's size change |
#1659 |
| B2 | A socket client on an iOS simulator presses app_switcher or lock: the helper gets 0x04 with that name; home still runs simctl launch and writes nothing to the helper; back is dropped (I1) |
createIosDeviceInput, helper faked |
app_switcher; lock; home; back |
#1659 |
| B3 | The helper reads 0x04: lock becomes one Indigo button press (source 1, down then up); app_switcher two Indigo Home presses (source 0) 50 ms apart; another name nothing, logged (I1) |
the helper's press plan for a button name, a pure function in vendor/screen-capture/Sources/simlock-screen-capture/, in a Swift test (swift test --package-path vendor/screen-capture) |
lock; app_switcher; back; siri; empty name — the 0x04 reader sends exactly this plan, never through sendButton |
#1659 |
Stored state
none
Interfaces
| Id | What | Shape and bounds | Errors and limits | Where | Task | Used by |
|---|---|---|---|---|---|---|
| I1 | Helper stdin message 0x04 button |
envelope as the other tags (4-byte big-endian length counting the tag, tag 0x04, JSON); body {"button": "app_switcher" | "lock"}, required |
an unknown name is logged by the helper and dropped; the helper does not answer | InputMode.swift (reader), device-input.ts (writer) |
#1659 | — |
Existing code
Commit c9d82e3.
Searches: keyboardUnavailable in src and website (5 files), app_switcher in src (4 files), buttonSource in vendor/screen-capture (1 file), --input in src (2 production files).
| Id | Code (path:line) | Today | With this feature | Handled by |
|---|---|---|---|---|
| X1 | src/drivers/ios/input/device-input.ts:80-87 keysAvailable |
keys only below iOS 27, and none for an unreadable runtime | removed; keys always go | B1 |
| X2 | src/drivers/ios/input/device-input.ts:213-215 #withKeys |
adds keyboardUnavailable when keys are off |
removed; the config is the helper's size | B1 |
| X3 | src/drivers/ios/input/device-input.ts:227-240 send |
a button other than home is dropped; a key is dropped when keys are off |
app_switcher and lock are written to the helper; keys always written |
B1, B2 |
| X4 | src/drivers/ios/input/device-input.ts:162-178 frameOfMessage |
no case for button |
a button frame for app_switcher and lock |
B2 |
| X5 | vendor/screen-capture/Sources/simlock-screen-capture/InputMode.swift:21-25, 97-115 |
reads tags 0x03, 0x06, 0x07, 0x0b; others skipped | also reads 0x04 | B3 |
| X6 | vendor/screen-capture/Sources/simlock-screen-capture/HIDInjector.swift:238-263, 442-466 |
home and lock sources defined, sendHIDButton private; sendButton knows swipe_home, siri, side_button |
the 0x04 reader sends B3's plan through sendHIDButton; sendButton is unchanged and not called by it |
B3 |
| X7 | src/drivers/ios/input/device-input.test.ts:86-180 |
asserts app_switcher and lock dropped, and keyboardUnavailable for iOS 27, 28.1 and garbage |
rewritten to B1 and B2 | B1, B2 |
| X8 | src/core/input.ts:34, 44 |
INPUT_BUTTONS holds all four names; keyboardUnavailable? in the config |
unchanged | — |
| X9 | vendor/screen-capture/NOTICE.md:137-147, HIDInjector.swift:1-8, vendor/screen-capture/Package.swift header |
say the app switcher and Lock paths are deleted | say they are back, sent as ADR 0085 says | #1659 |
| X10 | website/src/content/docs/docs/using-devices/watching-the-screen.mdx "On an iOS simulator" |
app_switcher, lock dropped; keys dropped on iOS 27 |
they work on every runtime; back dropped |
#1659 |
| X11 | src/drivers/ios/input/AGENTS.md rules and known gaps |
keys rule, dropped buttons, #1024 gap | rules for B1–B3; gap: a press iOS ignores is not detected | #1659 |
| X13 | src/drivers/ios/input/device-input.ts:242-243 #home comment |
says the app switcher and Lock are dropped | says Home alone uses simctl |
#1659 |
| X14 | vendor/screen-capture/Sources/simlock-screen-capture/InputMode.swift:3-17 header |
lists tags 0x03, 0x06, 0x07, 0x0b; "Home is not here" | adds 0x04 app_switcher and lock; Home still not here |
B3 |
| X15 | vendor/screen-capture/NOTICE.md:54 HIDInjector row |
"Home, app switcher and Lock paths are deleted" | upstream's paths stay deleted; app switcher and Lock are simlock's own press plan (ADR 0085) | #1659 |
| X12 | docs/internal/adr/0049-a-lease-holder-watches-and-drives-its-devices-screen.md status line, docs/internal/adr/README.md row |
no pointer to 0085 | status narrowed by 0085 | #1659 |
Tasks
- iOS simulator input: keys on every runtime, app switcher and Lock #1659 iOS simulator input: keys on every runtime, app switcher and Lock — a person on a leased iOS simulator types hardware keys, opens the app switcher and locks the screen through the input socket. Risk: a key or button does nothing on the simulator, and nothing says so.
Completion conditions: 1 by #1659's slow-lane Done when line, 2 by its pnpm test line, 3 by its website line.
Open work in the same area: draft PR #1499 (backend ownership) touches src/drivers/ios/input/device-input.test.ts, and PR #1256 (console screens side by side) touches the website input page. Neither changes the lines this task changes, and both are stale (last updated 2026-11-10 and 2026-11-04), so no Depends on.
Written by an agent.
Activity
feat(input): iOS simulator input through the HID injector (#1037)
[-]iOS simulator input on Xcode 27: hardware keys, app switcher, Lock and rotation[/-][+]iOS simulator input: hardware keys on every runtime, app switcher and Lock[/+]V3RON commentedon Nov 17, 2026
Design posted: Problem rewritten (rotation shipped in #1080; keys, app switcher and Lock measured working 2026-11-17), all business sections, Decisions (ADR 0085, PR #1658), Behaviours B1–B4, Interfaces I1, Existing code X1–X12.
Written by an agent.
docs(adr): accept 0085 (#1024)
V3RON commentedon Nov 17, 2026
Spec updated: task #1659 posted; T1 replaced by #1659 in every row; Tasks section filled; the device check is now the slow-lane test
e2e/slow-ios-input.test.tsreading the screen back with agent-device (Decisions).Written by an agent.
docs(adr): 0085 credits the Home-press claim to serve-sim's notes (#1024
docs(adr): 0085, an iOS simulator takes keys on every runtime and the…
github-actions commentedon Nov 17, 2026
Closing: all 1 sub-issues are closed. The PR that closed the last one walks the completion conditions (delivery rule 9); reopen this if one does not hold on
main. Flip its ADRs to Accepted if delivery has not.V3RON commentedon Nov 17, 2026
Delivery run
Merged: #1658 (ADR 0085), #1660 (#1659)
Parked: none
Waiting: none
Decided: #1659 — how the slow-lane test reads
app_switcherandlock→ by screenshot change (over 30% of pixels), because agent-deviceappstatestill reports Settings in front behind the app switcher (PR #1660, "Decision taken")Written by an agent.