Skip to content

iOS simulator input: hardware keys on every runtime, app switcher and Lock #1024

Closed
1 of 1 issue completed

Description

@V3RON

Request: none

Problem

Feature #1013 lets a lease holder drive its device's screen over the lease's input socket. On an iOS simulator three things the person sends are thrown away:

  • hardware keys, on an iOS 27 runtime: the socket says keyboardUnavailable, so the person can only tap the on-screen keyboard;
  • the app switcher button;
  • the Lock button.

Simlock dropped them on serve-sim's word ("the iOS 27 guest ignores injected keys", "Xcode 26+ drops the injected Home press"), never on a measurement. On 2026-11-17 the helper Simlock already ships sent all three, and they worked on iOS 26.4, 27.0 and 27.2 runtimes (ADR 0085 holds the table).

Rotation, also named when this issue was opened, shipped in #1080 (ADR 0052).

Who it is for

A person taking over an agent's iOS simulator through the lease's input socket, for example in T3 Code's device panel.

Outcome

On an iOS simulator of any runtime, the person types with a hardware keyboard, opens the app switcher, and locks or wakes the screen, as they already do on Android.

How it works for the user

flowchart LR
  P[Person in the device panel] -->|types a key| K[key reaches the simulator]
  P -->|app switcher button| A[the app switcher opens]
  P -->|Lock button| L[the screen locks, or wakes if locked]
  P -->|Home| H[home screen, as today]
Loading

Before, on an iOS 27 simulator, the socket's screen message:

0x82 {"width":1206,"height":2622,"orientation":"portrait","keyboardUnavailable":true}

After:

0x82 {"width":1206,"height":2622,"orientation":"portrait"}

Examples

Message on an iOS simulator Today After
0x06 key, iOS 26.4 runtime typed typed
0x06 key, iOS 27.0 or 27.2 runtime dropped typed
0x06 key, a runtime id that cannot be read dropped typed
0x04 home home screen (simctl) same
0x04 app_switcher dropped the app switcher opens
0x04 lock dropped the screen locks; on a locked screen, it wakes
0x04 back dropped dropped
0x04 on a physical iPhone no input at all same

What could go wrong

  • On a worker with an older Xcode than the ones measured, a key or button could do nothing and nothing says so. Unlikely: Xcode 26.4.1, 27.0 and 27.2 all worked.
  • Right after a fresh boot the first key or two may be lost. Seen once in about ten tries.
  • An app switcher press that iOS reads as one Home press shows the home screen instead. Not seen in five tries.

Words used

  • Input socket: WS /v1/leases/{id}/input, where a lease holder sends taps, keys and buttons in expo serve-sim's messages.
  • Helper: simlock-screen-capture <udid> --input, the vendored program that sends Indigo HID messages into a simulator.
  • Indigo message: the simulator's private HID message, e.g. a key down for usage 0x04 (a).
  • keyboardUnavailable: a flag in the screen message (0x82) that tells the client to hide hardware typing.

Non-goals

  • Back on iOS: an iPhone has no Back button.
  • Input on a physical iPhone.
  • Moving Home off simctl.
  • Detecting a press the simulator ignored.
  • New CLI or MCP commands for buttons.

Completion conditions

  • On a leased iOS 27 simulator, through the input socket: a focused text field receives typed hardware keys, app_switcher opens the app switcher, and lock locks the screen.
  • The socket's 0x82 for an iOS simulator never carries keyboardUnavailable.
  • The website's input page says keys, the app switcher and Lock work on every iOS simulator runtime.

Open questions

Decisions

  • ADR 0085 — An iOS simulator takes keys on every runtime, and the app switcher and Lock: narrows ADR 0049 §4; fixes how each button is sent.
  • Decided in the session: Home stays simctl launch com.apple.springboard, because it already works and the new presses do not need it.
  • Decided in the session: keys are not gated by Xcode or runtime version, because every measured Xcode and runtime typed, and a gate would rest on a version nobody showed to fail.
  • Decided in the session: the keyboardUnavailable field stays in core's 0x82 shape though no device sets it, because it is serve-sim's message.
  • Decided in the session: the device check is a slow-lane test that reads typed keys with an agent-device snapshot and the app switcher and Lock with agent-device appstate (a snapshot would bring the app back), because the simulator's own lock state cannot be read (com.apple.springboard.lockstate stays 0 with no passcode).

Behaviours

Id Behaviour (invoker, action, what it sees) Seam Cases Task
B1 A socket client on an iOS simulator sends a key: the helper gets it on every runtime, and the config never says keyboardUnavailable createIosDeviceInput (src/drivers/ios/input/device-input.ts), helper faked runtime iOS 26.4; iOS 27.0; iOS 28.1; unreadable id garbage; config after a foldable's size change #1659
B2 A socket client on an iOS simulator presses app_switcher or lock: the helper gets 0x04 with that name; home still runs simctl launch and writes nothing to the helper; back is dropped (I1) createIosDeviceInput, helper faked app_switcher; lock; home; back #1659
B3 The helper reads 0x04: lock becomes one Indigo button press (source 1, down then up); app_switcher two Indigo Home presses (source 0) 50 ms apart; another name nothing, logged (I1) the helper's press plan for a button name, a pure function in vendor/screen-capture/Sources/simlock-screen-capture/, in a Swift test (swift test --package-path vendor/screen-capture) lock; app_switcher; back; siri; empty name — the 0x04 reader sends exactly this plan, never through sendButton #1659

Stored state

none

Interfaces

Id What Shape and bounds Errors and limits Where Task Used by
I1 Helper stdin message 0x04 button envelope as the other tags (4-byte big-endian length counting the tag, tag 0x04, JSON); body {"button": "app_switcher" | "lock"}, required an unknown name is logged by the helper and dropped; the helper does not answer InputMode.swift (reader), device-input.ts (writer) #1659 —

Existing code

Commit c9d82e3.

Searches: keyboardUnavailable in src and website (5 files), app_switcher in src (4 files), buttonSource in vendor/screen-capture (1 file), --input in src (2 production files).

Id Code (path:line) Today With this feature Handled by
X1 src/drivers/ios/input/device-input.ts:80-87 keysAvailable keys only below iOS 27, and none for an unreadable runtime removed; keys always go B1
X2 src/drivers/ios/input/device-input.ts:213-215 #withKeys adds keyboardUnavailable when keys are off removed; the config is the helper's size B1
X3 src/drivers/ios/input/device-input.ts:227-240 send a button other than home is dropped; a key is dropped when keys are off app_switcher and lock are written to the helper; keys always written B1, B2
X4 src/drivers/ios/input/device-input.ts:162-178 frameOfMessage no case for button a button frame for app_switcher and lock B2
X5 vendor/screen-capture/Sources/simlock-screen-capture/InputMode.swift:21-25, 97-115 reads tags 0x03, 0x06, 0x07, 0x0b; others skipped also reads 0x04 B3
X6 vendor/screen-capture/Sources/simlock-screen-capture/HIDInjector.swift:238-263, 442-466 home and lock sources defined, sendHIDButton private; sendButton knows swipe_home, siri, side_button the 0x04 reader sends B3's plan through sendHIDButton; sendButton is unchanged and not called by it B3
X7 src/drivers/ios/input/device-input.test.ts:86-180 asserts app_switcher and lock dropped, and keyboardUnavailable for iOS 27, 28.1 and garbage rewritten to B1 and B2 B1, B2
X8 src/core/input.ts:34, 44 INPUT_BUTTONS holds all four names; keyboardUnavailable? in the config unchanged —
X9 vendor/screen-capture/NOTICE.md:137-147, HIDInjector.swift:1-8, vendor/screen-capture/Package.swift header say the app switcher and Lock paths are deleted say they are back, sent as ADR 0085 says #1659
X10 website/src/content/docs/docs/using-devices/watching-the-screen.mdx "On an iOS simulator" app_switcher, lock dropped; keys dropped on iOS 27 they work on every runtime; back dropped #1659
X11 src/drivers/ios/input/AGENTS.md rules and known gaps keys rule, dropped buttons, #1024 gap rules for B1–B3; gap: a press iOS ignores is not detected #1659
X13 src/drivers/ios/input/device-input.ts:242-243 #home comment says the app switcher and Lock are dropped says Home alone uses simctl #1659
X14 vendor/screen-capture/Sources/simlock-screen-capture/InputMode.swift:3-17 header lists tags 0x03, 0x06, 0x07, 0x0b; "Home is not here" adds 0x04 app_switcher and lock; Home still not here B3
X15 vendor/screen-capture/NOTICE.md:54 HIDInjector row "Home, app switcher and Lock paths are deleted" upstream's paths stay deleted; app switcher and Lock are simlock's own press plan (ADR 0085) #1659
X12 docs/internal/adr/0049-a-lease-holder-watches-and-drives-its-devices-screen.md status line, docs/internal/adr/README.md row no pointer to 0085 status narrowed by 0085 #1659

Tasks

Completion conditions: 1 by #1659's slow-lane Done when line, 2 by its pnpm test line, 3 by its website line.

Open work in the same area: draft PR #1499 (backend ownership) touches src/drivers/ios/input/device-input.test.ts, and PR #1256 (console screens side by side) touches the website input page. Neither changes the lines this task changes, and both are stale (last updated 2026-11-10 and 2026-11-04), so no Depends on.

Written by an agent.

Activity

  1. added
    feature:specSpec in progress, business and technical together.
    on Oct 26, 2026
  2. changed the title [-]iOS simulator input on Xcode 27: hardware keys, app switcher, Lock and rotation[/-] [+]iOS simulator input: hardware keys on every runtime, app switcher and Lock[/+] on Nov 17, 2026
  3. V3RON commented on Nov 17, 2026

    @V3RON
    ContributorAuthor

    Design posted: Problem rewritten (rotation shipped in #1080; keys, app switcher and Lock measured working 2026-11-17), all business sections, Decisions (ADR 0085, PR #1658), Behaviours B1–B4, Interfaces I1, Existing code X1–X12.

    Written by an agent.

  4. added a commit that references this issue on Nov 17, 2026
  5. V3RON commented on Nov 17, 2026

    @V3RON
    ContributorAuthor

    Spec updated: task #1659 posted; T1 replaced by #1659 in every row; Tasks section filled; the device check is now the slow-lane test e2e/slow-ios-input.test.ts reading the screen back with agent-device (Decisions).

    Written by an agent.

  6. added
    feature:plannedSplit into tasks. Never picked up itself.
    and removed
    feature:specSpec in progress, business and technical together.
    on Nov 17, 2026
  7. github-actions commented on Nov 17, 2026

    @github-actions
    Contributor

    Closing: all 1 sub-issues are closed. The PR that closed the last one walks the completion conditions (delivery rule 9); reopen this if one does not hold on main. Flip its ADRs to Accepted if delivery has not.

  8. V3RON commented on Nov 17, 2026

    @V3RON
    ContributorAuthor

    Delivery run

    Merged: #1658 (ADR 0085), #1660 (#1659)
    Parked: none
    Waiting: none
    Decided: #1659 — how the slow-lane test reads app_switcher and lock → by screenshot change (over 30% of pixels), because agent-device appstate still reports Settings in front behind the app switcher (PR #1660, "Decision taken")

    Written by an agent.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    feature:plannedSplit into tasks. Never picked up itself.

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions