Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
24 changes: 19 additions & 5 deletions docs/adr/0014-session-ref-frame-lifetime.md
Original file line number Diff line number Diff line change
Expand Up @@ -378,11 +378,25 @@ Each of these is one logical command:
- Jitter.
- `swipe --count`.
- Platform sequence chunks.
- The multi-ref press/click plan (`press @e13,@e14`, and the same `click` form), under the plan rules below.
Comment thread
strzhao marked this conversation as resolved.
Resolved

Each one resolves or plans once, expires once before the first repetition, and executes the original physical point/plan for every repetition.
It does not re-resolve the target or reactivate the frame between repetitions.
Failure in any later repetition or chunk leaves the frame expired.
Each one resolves or plans once and expires the frame once at its first mutating dispatch.
The repeated-action commands — `--count`, double-tap, hold, jitter, `swipe --count`, platform sequence chunks — execute the original physical point/plan for every repetition.
They do not re-resolve the target or reactivate the frame between repetitions, and a failure in any later repetition or chunk leaves the frame expired.
This preserves the explicit repeated-physical-interaction contract, and it prevents later commands from reuse of the frame.
The multi-ref press/click plan is not a repetition of one physical point: it admits its ordered keys once before dispatch, expires the frame at key 1, and dispatches each later key at its admitted point under the amendment rules below.

#### 2026-11-14 amendment: the multi-ref press/click plan is one logical command

`press @e13,@e14,...` (2 to 16 snapshot refs, and the same form on `click`) is one logical command with four rules:

1. Every member resolves and admits against the current frame before dispatch.
2. The frame expires once, at key 1.
3. Later keys dispatch their admitted points; no member re-resolves.
4. A failure after key 1 is post-seam: it reports `dispatchedSteps`, `refPlanKey`, and `refPlanPending`, and it never restores the frame.

This widens the earlier one-resolved-target-per-command limit to N pre-admitted targets for this plan shape only.
Every other one-target constraint in this decision stays unchanged.

### Partial issuance and MCP

Expand Down Expand Up @@ -434,7 +448,7 @@ Callers can avoid unnecessary captures with:
- Selectors.
- Direct selector fast paths.
- A settled partial result whose emitted ref is pinned.
- `--count` for a repeated physical action at one resolved target.
- `--count` repeats one resolved target (the multi-ref plan in the 2026-11-14 amendment above dispatches its N pre-admitted keys under one logical command, not this row).

The daemon never captures automatically for the sole purpose of a ref renewal.

Expand Down Expand Up @@ -470,7 +484,7 @@ This decision composes with existing ADRs as follows:

- A stale ref cannot silently mutate a different element after navigation.
The caller receives a typed failure and must choose a fresh observation or a selector.
- Ref-oriented workflows that perform multiple logical mutations must capture again between the mutations, consume an honestly issued settled ref, use selectors, or use one repeated-action command such as `--count`.
- Ref-oriented workflows that perform multiple logical mutations must capture again between the mutations, consume an honestly issued settled ref, use selectors, use one repeated-action command such as `--count`, or issue one multi-ref press/click plan whose members all resolve and admit before key 1 (the 2026-11-14 amendment above).
Legacy hand-written `.ad` scripts that reuse several bare refs from one snapshot must change.
- Conservative fused requests may consume ref authority even when the platform later reports that no action was necessary.
This favors the avoidance of a wrong-screen action over the preservation of refs after an ambiguous dispatch.
Expand Down
104 changes: 104 additions & 0 deletions docs/evidence/multi-ref-plan-ios-simulator-2026-11-12.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,104 @@
# Multi-ref press plan on an iOS simulator PIN pad, 2026-11-12

Evidence for #4296 (built and run from the contribution worktree, daemon
restarted clean). The plan is one logical command that resolves and admits
every member against the active frame and then dispatches the admitted
points in order; this run exercises that lowering on real simulator dispatch: a PIN pad page served
to Safari on the booted iPhone 17 Pro simulator (`C1C05FFF-29AD-40D8-94E5-D9CBB25EC6BB`,
iOS 26.5), whose digit keys are real accessibility buttons (`1` `2` `3` `4` `C`)
and whose state row (`ENTERED=... RESULT=...`) plus dot row (`●●●●`) give a
machine-readable oracle for what landed.

## Positive: `press @e12,@e13,@e12,@e13` — the repeated-digit PIN

Fresh `snapshot -i`, then one command (timed):

```json
{
"sequence": {
"pressed": 4,
"refs": ["@e12", "@e13", "@e12", "@e13"],
"points": [
{ "x": 111, "y": 300 },
{ "x": 201, "y": 300 },
{ "x": 111, "y": 300 },
{ "x": 201, "y": 300 }
]
},
"ref": "e13"
}
```

Wall time `2104ms`. The page's own state after the run:
`●●●●` and `ENTERED=1212 RESULT=PASS` — the repeated ref pressed the same
key twice, and the screenshot (`positive-final.png`) shows four dots with the
pass row. Every member resolved and admitted once against the snapshot frame
before key 1; the four dispatches then ran against the admitted points, which
is why the run costs about what four presses cost and not less.

## Negative: a member the frame cannot resolve

Fresh snapshot, then `press @e13,@e999` (`@e999` names nothing), at the
resolve-once head:

```json
{
"success": false,
"code": "COMMAND_FAILED",
"message": "Ref @e999 not found",
"details": {
"dispatched": "no",
"refPlanKey": "@e999",
"refPlanPending": [],
"ref": "e999"
}
}
```

Every member resolves before key 1, so the whole plan refuses pre-dispatch:
Comment thread
strzhao marked this conversation as resolved.
Resolved
`dispatched: 'no'`, no key sent, and the page held `ENTERED=-`. An earlier
iteration of this PR resolved members lazily at dispatch time; there the same
command sent key 1 first and disclosed `dispatched: 'unknown'` with
`dispatchedSteps: 1` — the ledger-owned post-dispatch disclosure the request
boundary still carries for backend failures. The refusal site moved with the
resolve-once shape; the disclosure contract did not.

## Later-key mutation semantics, observed rather than claimed

Three page variants put a mid-plan mutation between key 1 and key 2 (a
removed key, a disabled key, a system alert raised 50ms after key 1). In all
three the second member dispatched onto whatever occupied the key's
snapshot-time point, and the command reported success. This is the
resolve-once contract working as specified: key 1 expires the ref frame, so
later keys tap where the snapshot showed them and no member re-resolves —
Comment thread
cubic-dev-ai[bot] marked this conversation as resolved.
Resolved
`press --count` has the same property across its repetitions. Callers whose
screen can move between keys take a fresh snapshot and rebuild the plan;
callers who need landed verification on a fixed key use `--settle`/`--verify`.

## Wall time against the alternatives

Same page, same PIN, three routes:

| Route | Wall time | Outcome |
| --- | --- | --- |
| `press @e12,@e13,@e12,@e13` (one command) | 2104ms | PASS |
| `batch` with four `label=` selector steps | 3058ms | PASS |
| four `(snapshot; press)` caller rounds | 3724ms | PASS |

A fourth route — `batch` with four `ref` steps — failed at digit 2:
`Ref @e13 belongs to an expired ref frame` with `dispatched: 'unknown'`,
`executed: 2` (`batch.json`), because key 1's own mutation expires the frame
the next ref step resolves against. The comma-list plan resolves every member once before key 1 and then
dispatches the admitted points, so a repeated ref keeps working across
per-key mutations where a ref batch dies on the expired frame.

## Notes

- iOS simulator, XCUITest dispatch; no web backend involved (the page is the
app surface Safari exposes).
- Sessions opened for the run were closed; the local HTTP server ended.
- Local `check:affected --run --base old-origin/main` passes every stage
except `unit`, which fails only `eager-closure-budgets` — this clone's
`origin` names a private fork, so the gate reads the wrong merge base;
pointed at the upstream head for one `pnpm run check:unit`, the chain exits
0. CI on the pushed head is the authoritative run.
Original file line number Diff line number Diff line change
Expand Up @@ -8,6 +8,7 @@ import { localCommandPolicy } from '@agent-device/contracts/command-policy';
import type { Point } from '@agent-device/kernel/snapshot';
import type { SnapshotState } from '@agent-device/contracts/snapshot-model';
import { summarizeAxEvidence } from '@agent-device/capture-kit/snapshot-evidence';
import { AppError } from '@agent-device/kernel/errors';
import { iosSystemSurfaceDisclosure } from '@agent-device/contracts/ios-system-surface';
import { makeSnapshotState } from '@agent-device/selectors/snapshot-geometry-fixtures';
import {
Expand Down Expand Up @@ -799,3 +800,136 @@ test('coordinate tap with no session snapshot has no warning', async () => {
assert.equal(result.kind, 'point');
assert.equal('warning' in result, false);
});

// --- multi-ref plan lowering (press @e1,@e2,...) ---

function pinPadSnapshot(): SnapshotState {
return makeSnapshotState([
{
index: 0,
depth: 0,
type: 'Button',
label: '1',
value: '1',
rect: { x: 10, y: 20, width: 60, height: 60 },
hittable: true,
},
{
index: 1,
depth: 0,
type: 'Button',
label: '2',
value: '2',
rect: { x: 80, y: 20, width: 60, height: 60 },
hittable: true,
},
{
index: 2,
depth: 0,
type: 'Button',
label: '3',
value: '3',
rect: { x: 150, y: 20, width: 60, height: 60 },
hittable: true,
},
]);
}

test('runtime press plan dispatches every member as an ordinary press and reports the summary', async () => {
const taps: Array<{ point: Point; count?: number }> = [];
const device = createInteractionDevice(pinPadSnapshot(), {
tap: async (_context, point, options) => {
taps.push({ point, count: options?.count });
return { ok: true };
},
});

const result = await device.interactions.press(ref('@e1'), {
session: 'default',
refs: ['e1', 'e2', 'e1'],
});

assert.equal(taps.length, 3);
assert.deepEqual(result.sequence?.pressed, 3);
assert.deepEqual(result.sequence?.refs, ['@e1', '@e2', '@e1']);
// Each member dispatches its own resolved center: @e1 at (40, 50), @e2 at
// (110, 50), and the repeated @e1 again at (40, 50) — asserted literally so
// the test fails if any member resolves against a different member's point.
assert.deepEqual(
taps.map((tap) => tap.point),
Comment thread
strzhao marked this conversation as resolved.
Resolved
[
{ x: 40, y: 50 },
{ x: 110, y: 50 },
{ x: 40, y: 50 },
],
);
assert.deepEqual(result.sequence?.points, [
{ x: 40, y: 50 },
{ x: 110, y: 50 },
{ x: 40, y: 50 },
]);
});

test('runtime press plan sleeps intervalMs between keys and passes count to every member', async () => {
const taps: string[] = [];
const sleeps: number[] = [];
const device = createInteractionDevice(pinPadSnapshot(), {
tap: async () => {
taps.push('tap');
return { ok: true };
},
// The test clock observes every inter-key wait the plan asks for.
clock: {
now: () => 0,
sleep: async (ms: number) => {
sleeps.push(ms);
},
},
});

const result = await device.interactions.press(ref('@e1'), {
session: 'default',
refs: ['e1', 'e1'],
intervalMs: 1,
Comment thread
cubic-dev-ai[bot] marked this conversation as resolved.
Resolved
count: 2,
});

assert.equal(taps.length, 2);
assert.deepEqual(sleeps, [1]);
assert.deepEqual(result.sequence?.pressed, 2);
});

test('runtime press plan refuses a non-ref target', async () => {
const device = createInteractionDevice(pinPadSnapshot(), {
tap: async () => ({ ok: true }),
});

await assert.rejects(
device.interactions.press({ kind: 'point', x: 1, y: 2 }, { refs: ['e1'] }),
/multi-ref press plan requires ref targets/,
);
});

test('runtime press plan names the refused key and the pending keys when a member fails', async () => {
const taps: Point[] = [];
const device = createInteractionDevice(pinPadSnapshot(), {
tap: async (_context, point) => {
taps.push(point);
if (taps.length === 2) {
throw new AppError('COMMAND_FAILED', 'the second key refused on the backend');
}
return { ok: true };
},
});

await assert.rejects(
device.interactions.press(ref('@e1'), { session: 'default', refs: ['e1', 'e2'] }),
(error: AppError) => {
assert.equal(error.code, 'COMMAND_FAILED');
assert.equal(error.details?.refPlanKey, '@e2');
assert.deepEqual(error.details?.refPlanPending, []);
return true;
},
);
assert.equal(taps.length, 2);
});
Loading
Loading