Skip to content

iOS simulator input: keys on every runtime, app switcher and Lock #1659

Closed

Description

@V3RON

Part of #1024.

Scope

A person on a leased iOS simulator's input socket types with hardware keys on every runtime, opens the app switcher and locks or wakes the screen. The website says so.

If it goes wrong, a key or button the person sends does nothing on the simulator, and nothing says so.

Technical spec

Commit c9d82e3.

Rows

  • Behaviours: B1, B2, B3
  • Interfaces: I1
  • Existing code: X1, X2, X3, X4, X5, X6, X7, X8, X9, X10, X11, X12, X13, X14, X15

Facts

# Fact Source
F1 The Swift test target depends on the simlock-screen-capture executable target, so a test can reach its internal functions vendor/screen-capture/Package.swift (SimlockCaptureTests dependencies)
F2 IndigoHIDMessageForButton(source, direction, target) with direction 1 down and 2 up, target 0x33 vendor/screen-capture/Sources/simlock-screen-capture/HIDInjector.swift:238-263
F3 Measured 2026-11-17: Lock (source 1) locked iOS 26.4, 27.0 and 27.2 simulators; a second press woke 26.4 and 27.2; two Home presses (source 0) 0.05–0.2 s apart opened the app switcher; Indigo keys typed on all three runtimes ADR 0085, Context table
F4 The slow lane already opens a lease's input socket, taps through it and reads the result with an agent-device snapshot e2e/slow-agent-device.test.ts:250-280
F5 At agent-device 0.21.36 a snapshot brings the session's app to the front when it is in the background; appstate and screenshot bring nothing forward agent-device RunnerTests+Models.swift:153-158, 174-180, 293-302 (spec check of #1659)

Modules touched

Searches: keyboardUnavailable, app_switcher, keysAvailable, buttonSource, tagOrientation over src, vendor/screen-capture, website, docs/internal.

  • src/drivers/ios/input/ (device input, its test, its manual)
  • vendor/screen-capture/ (InputMode.swift, HIDInjector.swift, a new press plan file and its Swift test, NOTICE.md, Package.swift header comment)
  • e2e/slow-ios-input.test.ts (new slow-lane test) and its row in docs/internal/agent-rules/toolchain.md
  • website/src/content/docs/docs/using-devices/watching-the-screen.mdx
  • docs/internal/adr/0049-…md status line, docs/internal/adr/README.md row for 0049

Rules in play

  • ADR 0085 §1–4: the decision this task builds.
  • ADR 0049 §4: narrowed by 0085; its status line says so (X12), as it does for 0052.
  • vendor/screen-capture/NOTICE.md: every change to a MODIFIED FILE keeps its header and the deviation text true (X9, X15).
  • src/drivers/ios/input/AGENTS.md: Home has one owner, simctl in the driver; the helper never sends Home on its own, only as part of the app switcher plan.
  • Documentation rule 9: the input socket is public surface, so the website page changes in this PR (X10).

Retired words

  • keyboardUnavailable on an iOS simulator: no device sets it; the field stays in core's 0x82 shape only.

Tests

The seam for B1 and B2 is createIosDeviceInput with the helper faked, modelled on src/drivers/ios/input/device-input.test.ts; B3 is a Swift test modelled on vendor/screen-capture/Tests/SimlockCaptureTests/PanelTouchTests.swift.

  • B1: a socket client sends a key on an iOS simulator: runtime iOS 26.4; iOS 27.0; iOS 28.1; unreadable id garbage; the config after a foldable's size change has no keyboardUnavailable
  • B2: a socket client presses a button on an iOS simulator: app_switcher; lock; home; back
  • B3: the helper's press plan for a button name: lock; app_switcher; back; siri; empty name

Done when

  • pnpm test passes, with B1 and B2 in src/drivers/ios/input/device-input.test.ts.
  • swift test --package-path vendor/screen-capture passes, with B3.
  • On a real simulator, the new slow-lane test e2e/slow-ios-input.test.ts passes: on a leased iOS 27 simulator with capabilities.input.enabled, through WS /v1/leases/{id}/input (helper e2e/helpers/input-socket.ts): with Settings (com.apple.Preferences) open in an agent-device session and its search field focused, keys sent on the socket appear in an agent-device snapshot; app_switcher changes the screen; Settings is reopened with agent-device open; then lock changes the screen again. Each is read as more than 30% of the pixels changed between xcrun simctl io <udid> screenshot before and after the press, because agent-device appstate still reports Settings in front behind the app switcher and its screenshot hangs there (decided in delivery, pending the maintainer). This proves each press changed the screen, not which screen came up; the screenshots are kept for a reader. Model: the input socket and snapshot in e2e/slow-agent-device.test.ts:250-280.
  • docs/internal/agent-rules/toolchain.md has a slow-lane machine row for e2e/slow-ios-input.test.ts (an iOS 27 runtime and SIMLOCK_E2E_AGENT_DEVICE), modelled on the iOS foldable row.
  • The website page watching-the-screen.mdx, section "On an iOS simulator", says keys work on every runtime, app_switcher opens the app switcher, lock locks or wakes the screen, and back is dropped (X10).

Out of scope

  • Home through the helper; Back on iOS; physical iPhones; CLI or MCP button commands.

Depends on

Written by an agent.

Activity

  1. added
    task:draftScope written; technical spec or its spec check missing.
    on Nov 17, 2026
  2. added
    task:readySpec checked. An agent may implement it once its Depends on allow.
    and removed
    task:draftScope written; technical spec or its spec check missing.
    on Nov 17, 2026
  3. self-assigned this
    on Nov 17, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

Labels

task:readySpec checked. An agent may implement it once its Depends on allow.

Type

No type

Projects

No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions