Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
# 0049. A lease holder watches and drives its device's screen

- **Status:** Accepted; §3's rotation narrowed by [ADR 0052](0052-an-ios-simulator-takes-serve-sims-rotation.md): an iOS simulator takes `0x07`
- **Status:** Accepted; §3's rotation narrowed by [ADR 0052](0052-an-ios-simulator-takes-serve-sims-rotation.md): an iOS simulator takes `0x07`; §4's iOS 27 keys, app switcher and Lock narrowed by [ADR 0085](0085-an-ios-simulator-takes-keys-on-every-runtime-and-the-app-switcher-and-lock.md)
- **Date:** 2026-10-26
- **Issue:** [#1013](https://github.com/callstackincubator/simlock/issues/1013)
- **Narrows:** [ADR 0045](0045-a-devices-screen-streams-to-operators-not-a-leases.md)
Expand Down
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
# 0085. An iOS simulator takes keys on every runtime, and the app switcher and Lock

- **Status:** Accepted — not yet implemented
- **Status:** Accepted
- **Date:** 2026-11-17
- **Issue:** [#1024](https://github.com/callstackincubator/simlock/issues/1024)
- **Narrows:** [ADR 0049](0049-a-lease-holder-watches-and-drives-its-devices-screen.md)
Expand Down
4 changes: 2 additions & 2 deletions docs/internal/adr/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -87,7 +87,7 @@ the status is stale.
| [0046](0046-a-standalone-worker-serves-agent-device-by-default-with-a-private-profile.md) | A standalone worker serves agent-device by default and hands local agents a private profile | Accepted — not yet implemented; narrows 0023 §1, §3–4, 0030 §2, 0039 and 0043 §4 |
| [0047](0047-device-exec-fills-in-the-leases-device.md) | `device.exec` fills in the lease's device, and refuses another | Accepted — not yet implemented; extends 0005 §19a; narrows 0003 §11 for one MCP tool |
| [0048](0048-an-ios-worker-may-keep-its-simulators-in-xcodes-device-set.md) | An iOS worker may keep its simulators in Xcode's device set | Accepted; narrows 0001 §1, §2, §3, §5, §6 and §7 for iOS under `deviceSet: "default"` |
| [0049](0049-a-lease-holder-watches-and-drives-its-devices-screen.md) | A lease holder watches and drives its device's screen | Accepted; §3 narrowed by 0052 |
| [0049](0049-a-lease-holder-watches-and-drives-its-devices-screen.md) | A lease holder watches and drives its device's screen | Accepted; §3 narrowed by 0052; §4 narrowed by 0085 |
| [0050](0050-an-operator-can-forget-a-learned-warm-kind.md) | An operator can forget a learned warm kind | Accepted; how `warm forget` matches a baseline kind narrowed by 0062 |
| [0051](0051-configured-warm-targets-come-first-and-learning-can-be-turned-off.md) | Configured warm targets come first, and learning can be turned off | Accepted |
| [0052](0052-an-ios-simulator-takes-serve-sims-rotation.md) | An iOS simulator takes serve-sim's rotation | Accepted; narrows 0049 §3 on iOS simulators; §4 narrowed by 0075 |
Expand Down Expand Up @@ -117,4 +117,4 @@ the status is stale.
| [0080](0080-an-android-worker-may-enrol-phones-reached-over-the-network.md) | An Android worker may enrol phones reached over the network | Accepted; narrows 0022 §6 and §7, and 0073 §2 and §5 |
| [0083](0083-a-worker-may-turn-single-slim-services-on-or-off-beside-its-categories.md) | A worker may turn single slim services on or off beside its categories | Accepted; narrows 0060 §1, §2 and 0066 §2 |
| [0084](0084-a-worker-may-serve-a-platforms-enrolled-phones-only.md) | A worker may serve a platform's enrolled phones only | Accepted; narrows 0022 and 0009 §4 |
| [0085](0085-an-ios-simulator-takes-keys-on-every-runtime-and-the-app-switcher-and-lock.md) | An iOS simulator takes keys on every runtime, and the app switcher and Lock | Accepted — not yet implemented; narrows 0049 §4 |
| [0085](0085-an-ios-simulator-takes-keys-on-every-runtime-and-the-app-switcher-and-lock.md) | An iOS simulator takes keys on every runtime, and the app switcher and Lock | Accepted; narrows 0049 §4 |
1 change: 1 addition & 0 deletions docs/internal/agent-rules/toolchain.md
Original file line number Diff line number Diff line change
Expand Up @@ -211,6 +211,7 @@ The machine can run a line only if:
| iOS window | the iOS check; `mdfind "kMDItemCFBundleIdentifier == 'com.apple.dt.Devices'"` prints a path; `stat -f %Su /dev/console` is not `root` (someone is logged in); `pgrep -x DeviceHub` finds nothing (the test starts DeviceHub itself with `-shutdownStartedDevicesOnQuit NO` and quits it at the end). Run `scripts/slow-e2e.sh e2e/slow-ios-window.test.ts` |
| Physical iOS | `uname -s` is `Darwin`; `SIMLOCK_PHYSICAL_IOS` is the UDID of an iPhone on USB (trusted, Developer Mode on, unlocked, Auto-Lock Never); `SIMLOCK_PHYSICAL_IOS_APP` is a `.app` signed for it. `/usr/libexec/PlistBuddy -c "Print :CFBundleShortVersionString" /Library/Developer/PrivateFrameworks/CoreDevice.framework/Resources/Info.plist` prints a version whose major is 642 or later (streaming a phone needs the CoreDevice of Xcode 27 or later; the framework is machine-wide and does not follow `xcode-select`). DeviceHub and Xcode are not running during the lane. The stream cases need the iPhone on iOS 27 or later; an older iPhone runs the refusal case instead. Run `scripts/slow-e2e.sh --physical e2e/slow-ios-physical.test.ts` |
| iOS foldable | the iOS check; `xcrun simctl list runtimes` lists iOS 27.1 or later and `xcrun simctl list devicetypes` lists `iPhone Duo`; `SIMLOCK_E2E_AGENT_DEVICE` names agent-device 0.21.35 or later; `command -v ffmpeg` finds it. Run `scripts/slow-e2e.sh e2e/slow-ios-foldable.test.ts` |
| iOS simulator input | the iOS check; `xcrun simctl list runtimes` lists an iOS 27 or later runtime; `uname -m` is `arm64`; `SIMLOCK_E2E_AGENT_DEVICE` names the agent-device the lane pins (agent-device through simlock row). Run `scripts/slow-e2e.sh e2e/slow-ios-input.test.ts` |
| agent-device through simlock | `node -v` is 22.12.0 or later (agent-device's own minimum); the iOS check and `uname -m` is `arm64` (the fixture app is arm64 only), or the Android check; `SIMLOCK_E2E_AGENT_DEVICE` (in `.slow-lane.env`) is the absolute path of `bin/agent-device.mjs` in an install of the agent-device version `package.json` pins for `@agent-device/proxy` (`npm install -g agent-device@<version>`, or a checkout at tag `v<version>` built with `pnpm install && pnpm build && pnpm build:android`, the last building the Android snapshot helper); room for two devices: `node -p "os.totalmem() / 2**30"` prints at least 7 for iOS or 12 for Android (simlock's 4 GiB OS reserve plus two of its default 1.5 GiB iOS or 4 GiB Android per-device budgets). Run `scripts/slow-e2e.sh e2e/slow-agent-device.test.ts` or `scripts/slow-e2e.sh e2e/slow-agent-device-gateway.test.ts` |
| Physical Android | `command -v adb` finds it; `adb devices -l` lists `$SIMLOCK_PHYSICAL_ANDROID` on a USB transport (a `usb:` field) in state `device`; `SIMLOCK_PHYSICAL_ANDROID_APK` names an APK; `SIMLOCK_PHYSICAL_ANDROID_NETWORK` beside it names an address `adb devices -l` lists in state `device` with no `usb:` field, for the network case. Run `scripts/slow-e2e.sh --physical e2e/slow-android-physical.test.ts` |
| agent-device with a physical phone | the agent-device through simlock checks above (the Android or iOS one), and the Physical Android checks with `SIMLOCK_PHYSICAL_ANDROID`, or the Physical iOS checks with `SIMLOCK_PHYSICAL_IOS`; for iOS also `SIMLOCK_E2E_IOS_TEAM_ID` and `SIMLOCK_E2E_IOS_BUNDLE_ID` (in `.slow-lane.env`); set `SIMLOCK_E2E_IOS_PROVISIONING_PROFILE` and `SIMLOCK_E2E_IOS_SIGNING_IDENTITY` only for manual signing, which an Xcode-managed profile fails. Run `scripts/slow-e2e.sh --physical e2e/slow-agent-device-physical.test.ts` |
Expand Down
2 changes: 2 additions & 0 deletions e2e/helpers/agent-device.ts
Original file line number Diff line number Diff line change
Expand Up @@ -145,6 +145,8 @@ export async function makeAgent(
AGENT_DEVICE_STATE_DIR: home,
AGENT_DEVICE_DAEMON_AUTH_TOKEN: token.secret,
AGENT_DEVICE_NO_UPDATE_NOTIFIER: "1",
// why: the pinned agent-device refuses a leased remote connection without a named session (REMOTE_SESSION_REQUIRED).
AGENT_DEVICE_SESSION: name,
...(options.direct ? { AGENT_DEVICE_DAEMON_BASE_URL: proxyUrl(env) } : {}),
};
const background = new Set<CliBackgroundHandle>();
Expand Down
228 changes: 228 additions & 0 deletions e2e/slow-ios-input.test.ts
Original file line number Diff line number Diff line change
@@ -0,0 +1,228 @@
import { execFileSync } from "node:child_process";
import { mkdtemp, readFile } from "node:fs/promises";
import { tmpdir } from "node:os";
import { join } from "node:path";
import { PNG } from "pngjs";
import { afterAll, beforeAll, describe, expect, it } from "vitest";

import {
assertPinnedAgentDevice,
cleanupAgents,
FIRST_SNAPSHOT,
freeLoopbackPort,
listLeases,
listRequests,
makeAgent,
newAgentToken,
ok,
openInputSocket,
progress,
waitFor,
waitForAgentDevice,
withSharedDaemon,
type Agent,
type SharedTestEnv,
} from "./helpers/index.js";

const AGENT_DEVICE = process.env.SIMLOCK_E2E_AGENT_DEVICE;
const SETTINGS = "com.apple.Preferences";
const INPUT_BUTTON = 0x04;
const INPUT_KEY = 0x06;
const INPUT_TOUCH = 0x03;
const INPUT_ADMITTED = 0x83;
const INPUT_CONFIG = 0x82;
// why: USB HID keyboard usages for the letters a, b and c.
const TYPED = [0x04, 0x05, 0x06] as const;

// why: the file proves an iOS 27 runtime, where ADR 0049 once dropped keys, the app switcher and Lock.
function hasIos27Runtime(): boolean {
if (process.platform !== "darwin" || process.arch !== "arm64") return false;
try {
return /^iOS 2[7-9]|^iOS [3-9]\d/m.test(
execFileSync("xcrun", ["simctl", "list", "runtimes"], { encoding: "utf8" }),
);
} catch {
return false;
}
}

interface Box {
readonly x: number;
readonly y: number;
readonly width: number;
readonly height: number;
}

function* nodesOf(node: unknown): Generator<Record<string, unknown>> {
if (typeof node !== "object" || node === null) return;
if (!Array.isArray(node)) yield node as Record<string, unknown>;
for (const child of Object.values(node)) yield* nodesOf(child);
}

function boxOf(node: Record<string, unknown>): Box | undefined {
const rect = node["rect"] as Box | undefined;
return rect !== undefined && typeof rect.width === "number" ? rect : undefined;
}

// why: the search field is the node agent-device types a SearchField (Settings' list also holds buttons
// labelled Search, and a cell named "Optimising Search and Siri"); the screen is the largest box in the
// same snapshot, so the tap's fraction is measured against one size.
function searchFieldAndScreen(json: unknown): { field?: Box; screen?: Box } {
const boxed = [...nodesOf(json)].flatMap((node) => {
const box = boxOf(node);
return box === undefined ? [] : [{ box, node }];
});
const area = (box: Box) => box.width * box.height;
const screen = boxed.map((entry) => entry.box).sort((left, right) => area(right) - area(left))[0];
const field = boxed.find(({ node }) => node["type"] === "SearchField")?.box;
return {
...(field === undefined ? {} : { field }),
...(screen === undefined ? {} : { screen }),
};
}

// why: the share of pixels that moved between two screenshots of one simulator; the app switcher and
// Lock repaint most of a screen, so a press counts when more than CHANGED of it moved.
const CHANGED = 0.3;
function changedFraction(before: PNG, after: PNG): number {
if (before.width !== after.width || before.height !== after.height) return 1;
let moved = 0;
for (let i = 0; i < before.data.length; i += 4) {
const delta =
Math.abs(before.data[i]! - after.data[i]!) +
Math.abs(before.data[i + 1]! - after.data[i + 1]!) +
Math.abs(before.data[i + 2]! - after.data[i + 2]!);
if (delta > 48) moved++;
}
return moved / (before.data.length / 4);
}
const sleep = (ms: number) => new Promise((resolve) => setTimeout(resolve, ms));

describe.skipIf(AGENT_DEVICE === undefined || !hasIos27Runtime())(
"iOS simulator input on an iOS 27 runtime",
{ tags: ["slow", "ios"] },
() => {
let env: SharedTestEnv;

beforeAll(async () => {
const pinned = await assertPinnedAgentDevice("ios");
env = await withSharedDaemon({
driver: "real",
configOverrides: {
http: { enabled: true, host: "127.0.0.1", port: await freeLoopbackPort() },
capabilities: {
"agent-device": { enabled: true, command: pinned },
input: { enabled: true },
},
capacity: { config: { limits: { ios: { maxDevices: 1, maxRunning: 1 } } } },
},
});
const probe = await newAgentToken(env);
await waitForAgentDevice(env, probe.secret);
}, 420_000);

afterAll(async () => {
await cleanupAgents();
if (env !== undefined) {
await env.cli(["release", "--all", "--yes"], { timeout: 60_000 });
// why: no warm pool is configured here, so a released simulator is destroyed and no replacement turns ready; waitForFree would wait for one; only the lease and the request have to be gone before the daemon stops.
await waitFor(
async () =>
(await listLeases(env)).length === 0 && (await listRequests(env)).length === 0,
{ interval: 1_000, label: "no lease and no request after the release", timeout: 120_000 },
);
}
await env?.teardown();
}, 420_000);

it(
"on a leased iOS 27 simulator, keys sent on the input socket reach Settings' search field, app_switcher takes Settings out of the foreground, and so does lock after Settings is reopened",
{ timeout: 900_000 },
async () => {
const a: Agent = await makeAgent(env, "ios-input");
ok(await a.run(["open", "--platform", "ios", "--os", ">=27"], { timeout: 330_000 }));
ok(await a.run(["open", SETTINGS]));
const shots = await mkdtemp(join(tmpdir(), "simlock-ios-input-"));
progress(`screenshots of the iOS input test are saved in ${shots}`);

const snapshot = await a.run(["snapshot", "--json"], FIRST_SNAPSHOT);
ok(snapshot);
const { field, screen } = searchFieldAndScreen(snapshot.json);
expect(field, "Settings' search field in the snapshot").toBeDefined();
expect(screen, "the screen in the snapshot").toBeDefined();
const lease = (await listLeases(env))[0];
const socket = await openInputSocket(env.httpPort, lease?.id ?? "", a.token);
const appState = (await a.run(["appstate", "--json"])).data as
| { device_udid?: string; ios_simulator_device_set?: string }
| undefined;
expect(appState?.device_udid, "appstate names the simulator").toBeDefined();
const frame = async (name: string): Promise<PNG> => {
const file = join(shots, `${name}.png`);
execFileSync("xcrun", [
"simctl",
"--set",
appState!.ios_simulator_device_set!,
"io",
appState!.device_udid!,
"screenshot",
file,
]);
return PNG.sync.read(await readFile(file));
};
const waitForScreenChange = async (before: PNG, name: string): Promise<PNG> => {
let latest = before;
await waitFor(
async () => {
latest = await frame(name);
return changedFraction(before, latest) > CHANGED;
},
{ interval: 2_000, label: `the screen changed for ${name}`, timeout: 30_000 },
);
return latest;
};
try {
await socket.waitForTag(INPUT_ADMITTED);
const config = (await socket.waitForTag(INPUT_CONFIG)) as Record<string, unknown>;
expect(config["keyboardUnavailable"], "no keyboardUnavailable on iOS 27").toBeUndefined();

const x = (field!.x + field!.width / 2 - screen!.x) / screen!.width;
const y = (field!.y + field!.height / 2 - screen!.y) / screen!.height;
socket.send(INPUT_TOUCH, { type: "begin", x, y });
socket.send(INPUT_TOUCH, { type: "end", x, y });
// why: the first keys on a freshly booted simulator can be lost (ADR 0085 Context), so
// the keys are sent again until they show, up to a bound.
await waitFor(
async () => {
for (const usage of TYPED) {
socket.send(INPUT_KEY, { type: "down", usage });
socket.send(INPUT_KEY, { type: "up", usage });
}
const typed = await a.run(["snapshot"]);
return typed.code === 0 && /abc/i.test(typed.stdout);
},
{ interval: 3_000, label: "typed keys in the snapshot", timeout: 60_000 },
);
const frameA = await frame("1-typed");

// why: agent-device's appstate keeps answering runningForeground behind the app switcher and
// its screenshot hangs there, so the presses are read off the simulator's own screen.
socket.send(INPUT_BUTTON, { button: "app_switcher" });
const switcher = await waitForScreenChange(frameA, "2-app-switcher");

ok(await a.run(["open", SETTINGS]));
await sleep(3_000);
const reopened = await frame("2b-reopened");
expect(
changedFraction(switcher, reopened),
"Settings is back after the switcher",
).toBeGreaterThan(CHANGED);
socket.send(INPUT_BUTTON, { button: "lock" });
await waitForScreenChange(reopened, "3-lock");
} finally {
socket.close();
}
ok(await a.run(["close"]));
},
);
},
);
10 changes: 5 additions & 5 deletions src/drivers/ios/input/AGENTS.md
Original file line number Diff line number Diff line change
@@ -1,13 +1,13 @@
# iOS device input

Does: taps, scrolls, Home, keys and rotations on an iOS simulator, through `simlock-screen-capture <udid> --input`; a physical iPhone answers unavailable.
Does: taps, scrolls, Home, the app switcher, Lock, keys and rotations on an iOS simulator, through `simlock-screen-capture <udid> --input`; a physical iPhone answers unavailable.
Flows: [iOS driver](../AGENTS.md), [input](../../../input/AGENTS.md).

## Rules

- The helper's HID client and the runtime rule for keys stay here; core and the input module name neither.
- Home is `xcrun simctl` here and nowhere in the helper (one owner). Xcode 26 and later silently drop the HID home-button press, so Home launches `com.apple.springboard`; a failure is only logged. The app switcher, Lock and Back are dropped.
- Keys are dropped, and the config says `keyboardUnavailable: true`, when the runtime is iOS 27 or later or cannot be read: the iOS 27 guest ignores injected keys.
- The helper's HID client stays here; core and the input module do not name it.
- Home is `xcrun simctl` here and nowhere in the helper (one owner). Home launches `com.apple.springboard`; a failure is only logged. `app_switcher` and `lock` go to the helper (ADR 0085). Back is dropped.
- Keys go to the helper on every runtime, and the config never says `keyboardUnavailable`. A press the guest ignores is not detected.
- The helper follows the driver's device-set rule: `SIMLOCK_IOS_DEVICE_SET` names the owned set under `simlock` and is absent under `default`. The daemon's own value never reaches it.
- The open waits for the helper's first size frame (`READY_DEADLINE_MS` in `device-input.ts`), which it writes once its HID client is set up. A missing program, an early exit or no frame rejects the open, and the helper is stopped.
- The config is always `portrait`, a rotation included: it is the size of the panel the taps land on. A foldable's helper sends the size again when the panel in use changes (ADR 0075 §4); a size equal to the held one is no change, and a frame that is not a size is warned about and kept out.
Expand All @@ -17,7 +17,7 @@ Flows: [iOS driver](../AGENTS.md), [input](../../../input/AGENTS.md).

## Known gaps

- **No app switcher or Lock, and no keys on iOS 27 runtimes (#1024).**
- **A CoreSimulator older than 1174.21 was not measured** for the Home and Lock presses (ADR 0085); a worker on one may ignore them silently.
- **A screen that does not rotate ignores a rotation, and nothing tells the client** (the iPhone home screen, upside down on a Face ID iPhone). Rotation was checked on Xcode 26.4 (iOS 26.4 runtime) and Xcode 27.0 (iOS 26.4 and 27.0 runtimes) (#1080).
- **A one-panel device's input follows its native frame**, so after a rotation taps and picture stay in the same frame; a foldable's follow the upright picture.
- **Taps may vanish after a reboot under the lease.** After the health monitor reboots a crashed simulator the helper's HID client is bound to the old boot, until the lease's input reopens.
Loading
Loading