diff --git a/docs/adr/0014-session-ref-frame-lifetime.md b/docs/adr/0014-session-ref-frame-lifetime.md index 01a35056ef..d375affb46 100644 --- a/docs/adr/0014-session-ref-frame-lifetime.md +++ b/docs/adr/0014-session-ref-frame-lifetime.md @@ -378,11 +378,25 @@ Each of these is one logical command: - Jitter. - `swipe --count`. - Platform sequence chunks. +- The multi-ref press/click plan (`press @e13,@e14`, and the same `click` form), under the plan rules below. -Each one resolves or plans once, expires once before the first repetition, and executes the original physical point/plan for every repetition. -It does not re-resolve the target or reactivate the frame between repetitions. -Failure in any later repetition or chunk leaves the frame expired. +Each one resolves or plans once and expires the frame once at its first mutating dispatch. +The repeated-action commands — `--count`, double-tap, hold, jitter, `swipe --count`, platform sequence chunks — execute the original physical point/plan for every repetition. +They do not re-resolve the target or reactivate the frame between repetitions, and a failure in any later repetition or chunk leaves the frame expired. This preserves the explicit repeated-physical-interaction contract, and it prevents later commands from reuse of the frame. +The multi-ref press/click plan is not a repetition of one physical point: it admits its ordered keys once before dispatch, expires the frame at key 1, and dispatches each later key at its admitted point under the amendment rules below. + +#### 2026-11-14 amendment: the multi-ref press/click plan is one logical command + +`press @e13,@e14,...` (2 to 16 snapshot refs, and the same form on `click`) is one logical command with four rules: + +1. Every member resolves and admits against the current frame before dispatch. +2. The frame expires once, at key 1. +3. Later keys dispatch their admitted points; no member re-resolves. +4. A failure after key 1 is post-seam: it reports `dispatchedSteps`, `refPlanKey`, and `refPlanPending`, and it never restores the frame. + +This widens the earlier one-resolved-target-per-command limit to N pre-admitted targets for this plan shape only. +Every other one-target constraint in this decision stays unchanged. ### Partial issuance and MCP @@ -434,7 +448,7 @@ Callers can avoid unnecessary captures with: - Selectors. - Direct selector fast paths. - A settled partial result whose emitted ref is pinned. -- `--count` for a repeated physical action at one resolved target. +- `--count` repeats one resolved target (the multi-ref plan in the 2026-11-14 amendment above dispatches its N pre-admitted keys under one logical command, not this row). The daemon never captures automatically for the sole purpose of a ref renewal. @@ -470,7 +484,7 @@ This decision composes with existing ADRs as follows: - A stale ref cannot silently mutate a different element after navigation. The caller receives a typed failure and must choose a fresh observation or a selector. -- Ref-oriented workflows that perform multiple logical mutations must capture again between the mutations, consume an honestly issued settled ref, use selectors, or use one repeated-action command such as `--count`. +- Ref-oriented workflows that perform multiple logical mutations must capture again between the mutations, consume an honestly issued settled ref, use selectors, use one repeated-action command such as `--count`, or issue one multi-ref press/click plan whose members all resolve and admit before key 1 (the 2026-11-14 amendment above). Legacy hand-written `.ad` scripts that reuse several bare refs from one snapshot must change. - Conservative fused requests may consume ref authority even when the platform later reports that no action was necessary. This favors the avoidance of a wrong-screen action over the preservation of refs after an ambiguous dispatch. diff --git a/docs/evidence/multi-ref-plan-ios-simulator-2026-11-12.md b/docs/evidence/multi-ref-plan-ios-simulator-2026-11-12.md new file mode 100644 index 0000000000..83071699d6 --- /dev/null +++ b/docs/evidence/multi-ref-plan-ios-simulator-2026-11-12.md @@ -0,0 +1,104 @@ +# Multi-ref press plan on an iOS simulator PIN pad, 2026-11-12 + +Evidence for #4296 (built and run from the contribution worktree, daemon +restarted clean). The plan is one logical command that resolves and admits +every member against the active frame and then dispatches the admitted +points in order; this run exercises that lowering on real simulator dispatch: a PIN pad page served +to Safari on the booted iPhone 17 Pro simulator (`C1C05FFF-29AD-40D8-94E5-D9CBB25EC6BB`, +iOS 26.5), whose digit keys are real accessibility buttons (`1` `2` `3` `4` `C`) +and whose state row (`ENTERED=... RESULT=...`) plus dot row (`●●●●`) give a +machine-readable oracle for what landed. + +## Positive: `press @e12,@e13,@e12,@e13` — the repeated-digit PIN + +Fresh `snapshot -i`, then one command (timed): + +```json +{ + "sequence": { + "pressed": 4, + "refs": ["@e12", "@e13", "@e12", "@e13"], + "points": [ + { "x": 111, "y": 300 }, + { "x": 201, "y": 300 }, + { "x": 111, "y": 300 }, + { "x": 201, "y": 300 } + ] + }, + "ref": "e13" +} +``` + +Wall time `2104ms`. The page's own state after the run: +`●●●●` and `ENTERED=1212 RESULT=PASS` — the repeated ref pressed the same +key twice, and the screenshot (`positive-final.png`) shows four dots with the +pass row. Every member resolved and admitted once against the snapshot frame +before key 1; the four dispatches then ran against the admitted points, which +is why the run costs about what four presses cost and not less. + +## Negative: a member the frame cannot resolve + +Fresh snapshot, then `press @e13,@e999` (`@e999` names nothing), at the +resolve-once head: + +```json +{ + "success": false, + "code": "COMMAND_FAILED", + "message": "Ref @e999 not found", + "details": { + "dispatched": "no", + "refPlanKey": "@e999", + "refPlanPending": [], + "ref": "e999" + } +} +``` + +Every member resolves before key 1, so the whole plan refuses pre-dispatch: +`dispatched: 'no'`, no key sent, and the page held `ENTERED=-`. An earlier +iteration of this PR resolved members lazily at dispatch time; there the same +command sent key 1 first and disclosed `dispatched: 'unknown'` with +`dispatchedSteps: 1` — the ledger-owned post-dispatch disclosure the request +boundary still carries for backend failures. The refusal site moved with the +resolve-once shape; the disclosure contract did not. + +## Later-key mutation semantics, observed rather than claimed + +Three page variants put a mid-plan mutation between key 1 and key 2 (a +removed key, a disabled key, a system alert raised 50ms after key 1). In all +three the second member dispatched onto whatever occupied the key's +snapshot-time point, and the command reported success. This is the +resolve-once contract working as specified: key 1 expires the ref frame, so +later keys tap where the snapshot showed them and no member re-resolves — +`press --count` has the same property across its repetitions. Callers whose +screen can move between keys take a fresh snapshot and rebuild the plan; +callers who need landed verification on a fixed key use `--settle`/`--verify`. + +## Wall time against the alternatives + +Same page, same PIN, three routes: + +| Route | Wall time | Outcome | +| --- | --- | --- | +| `press @e12,@e13,@e12,@e13` (one command) | 2104ms | PASS | +| `batch` with four `label=` selector steps | 3058ms | PASS | +| four `(snapshot; press)` caller rounds | 3724ms | PASS | + +A fourth route — `batch` with four `ref` steps — failed at digit 2: +`Ref @e13 belongs to an expired ref frame` with `dispatched: 'unknown'`, +`executed: 2` (`batch.json`), because key 1's own mutation expires the frame +the next ref step resolves against. The comma-list plan resolves every member once before key 1 and then +dispatches the admitted points, so a repeated ref keeps working across +per-key mutations where a ref batch dies on the expired frame. + +## Notes + +- iOS simulator, XCUITest dispatch; no web backend involved (the page is the + app surface Safari exposes). +- Sessions opened for the run were closed; the local HTTP server ended. +- Local `check:affected --run --base old-origin/main` passes every stage + except `unit`, which fails only `eager-closure-budgets` — this clone's + `origin` names a private fork, so the gate reads the wrong merge base; + pointed at the upstream head for one `pnpm run check:unit`, the chain exits + 0. CI on the pushed head is the authoritative run. diff --git a/packages/command-runtime/src/commands/interaction/runtime/interactions.test.ts b/packages/command-runtime/src/commands/interaction/runtime/interactions.test.ts index 1621abe83f..6324caeb28 100644 --- a/packages/command-runtime/src/commands/interaction/runtime/interactions.test.ts +++ b/packages/command-runtime/src/commands/interaction/runtime/interactions.test.ts @@ -8,6 +8,7 @@ import { localCommandPolicy } from '@agent-device/contracts/command-policy'; import type { Point } from '@agent-device/kernel/snapshot'; import type { SnapshotState } from '@agent-device/contracts/snapshot-model'; import { summarizeAxEvidence } from '@agent-device/capture-kit/snapshot-evidence'; +import { AppError } from '@agent-device/kernel/errors'; import { iosSystemSurfaceDisclosure } from '@agent-device/contracts/ios-system-surface'; import { makeSnapshotState } from '@agent-device/selectors/snapshot-geometry-fixtures'; import { @@ -799,3 +800,136 @@ test('coordinate tap with no session snapshot has no warning', async () => { assert.equal(result.kind, 'point'); assert.equal('warning' in result, false); }); + +// --- multi-ref plan lowering (press @e1,@e2,...) --- + +function pinPadSnapshot(): SnapshotState { + return makeSnapshotState([ + { + index: 0, + depth: 0, + type: 'Button', + label: '1', + value: '1', + rect: { x: 10, y: 20, width: 60, height: 60 }, + hittable: true, + }, + { + index: 1, + depth: 0, + type: 'Button', + label: '2', + value: '2', + rect: { x: 80, y: 20, width: 60, height: 60 }, + hittable: true, + }, + { + index: 2, + depth: 0, + type: 'Button', + label: '3', + value: '3', + rect: { x: 150, y: 20, width: 60, height: 60 }, + hittable: true, + }, + ]); +} + +test('runtime press plan dispatches every member as an ordinary press and reports the summary', async () => { + const taps: Array<{ point: Point; count?: number }> = []; + const device = createInteractionDevice(pinPadSnapshot(), { + tap: async (_context, point, options) => { + taps.push({ point, count: options?.count }); + return { ok: true }; + }, + }); + + const result = await device.interactions.press(ref('@e1'), { + session: 'default', + refs: ['e1', 'e2', 'e1'], + }); + + assert.equal(taps.length, 3); + assert.deepEqual(result.sequence?.pressed, 3); + assert.deepEqual(result.sequence?.refs, ['@e1', '@e2', '@e1']); + // Each member dispatches its own resolved center: @e1 at (40, 50), @e2 at + // (110, 50), and the repeated @e1 again at (40, 50) — asserted literally so + // the test fails if any member resolves against a different member's point. + assert.deepEqual( + taps.map((tap) => tap.point), + [ + { x: 40, y: 50 }, + { x: 110, y: 50 }, + { x: 40, y: 50 }, + ], + ); + assert.deepEqual(result.sequence?.points, [ + { x: 40, y: 50 }, + { x: 110, y: 50 }, + { x: 40, y: 50 }, + ]); +}); + +test('runtime press plan sleeps intervalMs between keys and passes count to every member', async () => { + const taps: string[] = []; + const sleeps: number[] = []; + const device = createInteractionDevice(pinPadSnapshot(), { + tap: async () => { + taps.push('tap'); + return { ok: true }; + }, + // The test clock observes every inter-key wait the plan asks for. + clock: { + now: () => 0, + sleep: async (ms: number) => { + sleeps.push(ms); + }, + }, + }); + + const result = await device.interactions.press(ref('@e1'), { + session: 'default', + refs: ['e1', 'e1'], + intervalMs: 1, + count: 2, + }); + + assert.equal(taps.length, 2); + assert.deepEqual(sleeps, [1]); + assert.deepEqual(result.sequence?.pressed, 2); +}); + +test('runtime press plan refuses a non-ref target', async () => { + const device = createInteractionDevice(pinPadSnapshot(), { + tap: async () => ({ ok: true }), + }); + + await assert.rejects( + device.interactions.press({ kind: 'point', x: 1, y: 2 }, { refs: ['e1'] }), + /multi-ref press plan requires ref targets/, + ); +}); + +test('runtime press plan names the refused key and the pending keys when a member fails', async () => { + const taps: Point[] = []; + const device = createInteractionDevice(pinPadSnapshot(), { + tap: async (_context, point) => { + taps.push(point); + if (taps.length === 2) { + throw new AppError('COMMAND_FAILED', 'the second key refused on the backend'); + } + return { ok: true }; + }, + }); + + await assert.rejects( + device.interactions.press(ref('@e1'), { session: 'default', refs: ['e1', 'e2'] }), + (error: AppError) => { + assert.equal(error.code, 'COMMAND_FAILED'); + assert.equal(error.details?.refPlanKey, '@e2'); + assert.deepEqual(error.details?.refPlanPending, []); + return true; + }, + ); + assert.equal(taps.length, 2); +}); diff --git a/packages/command-runtime/src/commands/interaction/runtime/interactions.ts b/packages/command-runtime/src/commands/interaction/runtime/interactions.ts index 6c19cdc82f..dccc021be8 100644 --- a/packages/command-runtime/src/commands/interaction/runtime/interactions.ts +++ b/packages/command-runtime/src/commands/interaction/runtime/interactions.ts @@ -4,6 +4,7 @@ import type { PreresolvedInteractionTarget, PressCommandResult, RepeatedInput, + ResolvedInteractionTarget, ResolvedTarget, } from '@agent-device/contracts/interaction'; import { AppError } from '@agent-device/kernel/errors'; @@ -12,7 +13,7 @@ import type { Point } from '@agent-device/kernel/snapshot'; import type { AgentDeviceRuntime, CommandContext } from '@agent-device/contracts/runtime-contract'; import { isFillableType } from '@agent-device/contracts/snapshot'; import { attachResolvedInteractionTarget } from '@agent-device/host-kit/interaction-outcome'; -import { toBackendContext } from '../../runtime-common.ts'; +import { sleep, toBackendContext } from '../../runtime-common.ts'; import { toBackendResult, type RuntimeCommand } from '../../runtime-types.ts'; import { applyPostActionObservation, @@ -41,6 +42,14 @@ export type PressCommandOptions = CommandContext & RepeatedInput & { target: InteractionTarget; button?: ClickButton; + /** + * Multi-ref press plan (`press @e13,@e14,...`): every member in dispatch + * order, bare ref bodies; `target` repeats member 0 for the response's + * ref identity. Absent for a single-ref press. One ADR 0014 logical + * command: every member resolves and admits against the active frame + * before the first press; see tapRefPlanCommand. + */ + refs?: readonly string[]; /** ADR 0012 step 4: replay-only post-resolution guard; see resolution.ts. */ expectedResolvedTarget?: ExpectedResolvedTarget; /** #1654: a mutating `find`'s already-resolved node; see resolution.ts. */ @@ -139,14 +148,17 @@ async function tapCommand( options: PressCommandOptions, action: 'click' | 'press', ): Promise { + if (options.refs?.length) { + return await tapRefPlanCommand(runtime, options, action); + } const observation = planPostActionObservation(options); const nativeRefTap = observation.needsPreActionBaseline ? null : await maybeTapRefTarget(runtime, options, action); if (nativeRefTap) return nativeRefTap; - // One request object for the resolution and for the looks admission may spend, so a second look is - // the same question asked again rather than a differently-configured one. + // One request object for the resolution and for the re-resolutions #3471's gate may spend, so the + // second look is the same question asked again rather than a differently-configured one. const resolutionRequest: ResolveInteractionTargetParams = { action, requireInteractive: true, @@ -200,6 +212,142 @@ function requireResolvedPoint(result: { point?: Point }): Point { return result.point; } +/** + * Resolves and admits every plan member against the active frame, before key 1. Member 0's + * resolution captures the plan's pre-command evidence baseline. Every refusal here is + * pre-dispatch: nothing has crossed to the device, so the verdict stays `no`. + */ +async function resolveRefPlanMembers( + runtime: AgentDeviceRuntime, + options: PressCommandOptions, + action: 'click' | 'press', + plannedRefs: readonly string[], + captureEvidenceBaseline: boolean, +): Promise> { + // One request object for the resolution and for the looks admission may spend, so a + // second look is the same question asked again rather than a differently-configured one. + const resolutionRequest: ResolveInteractionTargetParams = { + action, + requireInteractive: true, + pipeline: SELECTOR_PIPELINE_POLICIES.promotedTarget, + }; + const members: Array<{ ref: string; atRest: ResolvedInteractionTarget; point: Point }> = []; + for (const [index, ref] of plannedRefs.entries()) { + const member: PressCommandOptions = { ...options, target: { kind: 'ref', ref } }; + try { + const resolved = await resolveInteractionTarget(runtime, member, { + ...resolutionRequest, + // One pre-command baseline serves the plan's evidence: member 0's resolution + // captures it, and the final result carries it forward. + captureEvidenceBaseline: captureEvidenceBaseline && index === 0, + }); + const atRest = await admitResolvedTarget(runtime, member, resolutionRequest, resolved); + members.push({ ref, atRest, point: requireResolvedPoint(atRest) }); + } catch (error) { + throw attachRefPlanContext(error, plannedRefs, index); + } + } + return members; +} + +/** + * A multi-ref plan is one ADR 0014 logical command that resolves and admits every member + * once against the active frame, then dispatches the admitted points in order — `press + * --count` with N points. Key 1's dispatch expires the ref frame, so later keys tap where + * the snapshot showed them: no member re-resolves, and the motion screen judges the whole + * plan once, before key 1. Every resolution and admission refusal is pre-dispatch + * (`dispatched: 'no'`); after key 1, the request dispatch ledger owns the dispatched-steps + * disclosure and these details name the key that refused and the keys that did not run. + */ +async function tapRefPlanCommand( + runtime: AgentDeviceRuntime, + options: PressCommandOptions, + action: 'click' | 'press', +): Promise { + if (options.target.kind !== 'ref') { + throw new AppError('INVALID_ARGS', 'A multi-ref press plan requires ref targets.'); + } + if (!runtime.backend.tap) { + throw new AppError('UNSUPPORTED_OPERATION', 'tap is not supported by this backend'); + } + // The parser emits the full plan in `refs` with `target` kept as member 0's + // response identity, so the plan iterates `refs` alone. + const plannedRefs = (options.refs ?? []).map((ref) => (ref.startsWith('@') ? ref : `@${ref}`)); + const observation = planPostActionObservation(options); + const members = await resolveRefPlanMembers( + runtime, + options, + action, + plannedRefs, + observation.needsPreActionBaseline, + ); + let lastBackendResult; + for (const [index, member] of members.entries()) { + if (index > 0 && options.intervalMs !== undefined && options.intervalMs > 0) { + await sleep(runtime, options.intervalMs, options.signal); + } + const pointObservation = pointObservationOf(member.atRest); + try { + lastBackendResult = await runtime.backend.tap( + toBackendContext(runtime, options), + member.point, + { + button: options.button, + count: options.count, + intervalMs: options.intervalMs, + holdMs: options.holdMs, + jitterPx: options.jitterPx, + doubleTap: options.doubleTap, + ...(pointObservation ? { pointObservation } : {}), + }, + ); + } catch (error) { + // Resolution is complete before the backend call. Preserve it out of + // band so a daemon-level failure corroboration can still record the same + // target identity if the backend reports an ambiguous tap outcome. + attachResolvedInteractionTarget(error, member.atRest); + throw attachRefPlanContext(error, plannedRefs, index); + } + } + // The response keeps member 0's identity — the first key's ref, point, and + // its captured evidence baseline — while every key's landing stays in + // sequence.points. + const finalTarget: ResolvedInteractionTarget = members[0]!.atRest; + const formattedBackendResult = toBackendResult(lastBackendResult); + const result = await applyPostActionObservation( + runtime, + options, + finalTarget, + { + ...finalTarget, + ...(formattedBackendResult ? { backendResult: formattedBackendResult } : {}), + }, + observation, + ); + return { + ...result, + sequence: { + pressed: members.length, + refs: plannedRefs, + points: members.map((member) => member.point), + }, + }; +} + +function attachRefPlanContext( + error: unknown, + refs: readonly string[], + failedIndex: number, +): unknown { + if (!(error instanceof AppError)) return error; + error.details = { + ...error.details, + refPlanKey: refs[failedIndex], + refPlanPending: refs.slice(failedIndex + 1), + }; + return error; +} + async function maybeTapRefTarget( runtime: AgentDeviceRuntime, options: PressCommandOptions, diff --git a/packages/command-surface/src/command-input.ts b/packages/command-surface/src/command-input.ts index b2e96478ea..f18c2a27b6 100644 --- a/packages/command-surface/src/command-input.ts +++ b/packages/command-surface/src/command-input.ts @@ -392,7 +392,10 @@ export function selectorSnapshotFields() { export function repeatedFields() { return { count: integerField('Number of press/click repetitions.', { min: 1 }), - intervalMs: integerField('Delay between repeated press/click actions.', { min: 0 }), + intervalMs: integerField( + 'Delay between repeated press/click actions, or between the keys of a comma-separated ref sequence.', + { min: 0 }, + ), holdMs: integerField('Hold duration for each action.', { min: 0 }), jitterPx: integerField( 'Deterministic per-axis coordinate jitter in pixels (each press offsets x and y by up to this amount).', diff --git a/packages/command-surface/src/interaction/index.ts b/packages/command-surface/src/interaction/index.ts index 93986f641a..d6d31713f5 100644 --- a/packages/command-surface/src/interaction/index.ts +++ b/packages/command-surface/src/interaction/index.ts @@ -215,12 +215,33 @@ const tapInteractionResponseDataSchema = interactionResponseDataSchema({ evidence: interactionEvidenceSchema, button: enumSchema(['secondary', 'middle']), count: numberSchema('Number of press/click repetitions.'), - intervalMs: numberSchema('Delay between repeated press/click actions.'), + intervalMs: numberSchema( + 'Delay between repeated press/click actions, or between the keys of a comma-separated ref sequence.', + ), holdMs: numberSchema('Hold duration for each action.'), jitterPx: numberSchema( 'Deterministic per-axis coordinate jitter in pixels (each press offsets x and y by up to this amount).', ), doubleTap: booleanSchema('Whether the command requested a double-tap action.'), + sequence: objectSchema( + { + pressed: numberSchema('How many keys of the plan were dispatched.'), + refs: stringArraySchema('The planned refs in dispatch order, each with its @ prefix.'), + points: { + type: 'array', + description: 'Where each dispatched key landed, in plan order.', + items: objectSchema( + { + x: numberSchema(), + y: numberSchema(), + }, + ['x', 'y'], + ), + }, + }, + ['pressed', 'refs', 'points'], + 'Present when the command ran a multi-ref plan (press @e13,@e14,...): one action per key, in order.', + ), }, }); @@ -362,7 +383,7 @@ const interactionCliSchemas = { allowedFlags: [...SELECTOR_SNAPSHOT_FLAGS, 'record'], }, click: { - usageOverride: 'click ', + usageOverride: 'click ', usageFlags: [], positionalArgs: ['target'], allowsExtraPositionals: true, @@ -374,7 +395,7 @@ const interactionCliSchemas = { ], }, press: { - usageOverride: 'press ', + usageOverride: 'press ', usageFlags: [], positionalArgs: ['targetOrX', 'y?'], allowsExtraPositionals: true, @@ -460,6 +481,8 @@ const clickCommandFacet = defineCommandFacet({ text: { summary: 'Click or tap a UI target', searchHint: 'press button link web mouse', + cliDetail: + "A comma-separated ref sequence follows the press form: click @e13,@e14 clicks each key in order as one command, with press's plan contract. See press for the plan semantics and recovery fields. Script recording refuses the sequence form: record one click per key.", }, metadata: metadata('click'), run: (client, input) => client.interactions.click(toClickOptions(input)), @@ -473,8 +496,9 @@ const pressCommandFacet = defineCommandFacet({ effect: 'destructive', text: { summary: 'Short-press a UI target', - searchHint: 'tap click touch button', - cliDetail: 'The hold duration is positional on longpress, not press --hold-ms.', + searchHint: 'click touch button', + cliDetail: + 'The hold duration is positional on longpress, not press --hold-ms. A comma-separated ref plan (press @e13,@e14,@e15,@e16; 2 to 16 refs, repeats allowed) resolves and admits every key against the current frame, then dispatches the admitted points: key 1 expires the frame, later keys dispatch where the snapshot showed them; a later refusal carries details.dispatchedSteps, details.refPlanKey, and details.refPlanPending, and sequence.points holds the dispatched points. --interval-ms sleeps between keys; --count, --hold-ms, --jitter-px, and --double-tap apply to every key; the response ref is the first key. Recording refuses the plan.', }, metadata: metadata('press'), run: (client, input) => client.interactions.press(toPressOptions(input)), diff --git a/packages/command-surface/src/interaction/metadata.ts b/packages/command-surface/src/interaction/metadata.ts index 306586d230..74b5869629 100644 --- a/packages/command-surface/src/interaction/metadata.ts +++ b/packages/command-surface/src/interaction/metadata.ts @@ -67,7 +67,7 @@ const interactionCommandDescriptions = { click: "Activate a UI target by snapshot ref, selector, or coordinates. This can change app state. Success reports dispatch only; settle returns the resulting UI diff and verify returns cheap post-action evidence. A ref or selector target is re-checked for motion before dispatch when the session's previous action just animated the screen, and is dispatched at its resting position; one that has not stopped refuses with target_still_moving rather than acting where it was. With settle, continue from the returned diff. Copy refs exactly, including any ~sN suffix.", press: - "Short-press a UI target by snapshot ref, selector, or coordinates. Success reports dispatch only; settle returns the resulting UI diff and verify returns cheap post-action evidence. A ref or selector target is re-checked for motion before dispatch when the session's previous action just animated the screen, and is dispatched at its resting position; one that has not stopped refuses with target_still_moving rather than acting where it was.", + "Short-press a UI target by snapshot ref, selector, or coordinates. Success reports dispatch only; settle returns the resulting UI diff and verify returns cheap post-action evidence. A ref or selector target is re-checked for motion before dispatch when the session's previous action just animated the screen, and is dispatched at its resting position; one that has not stopped refuses with target_still_moving rather than acting where it was. A comma-separated ref list (`@e13,@e14,@e15,@e16`) dispatches its keys in order as one command; 2-16 refs, a repeated ref repeats that key, and the response carries a sequence summary of refs and points. The plan admits every key before the first press; later keys dispatch where the snapshot showed them, and a later refusal carries details.dispatchedSteps, details.refPlanKey, and details.refPlanPending.", fill: 'Replace text in a UI input selected by snapshot ref, selector, or coordinates. An empty text clears the field. Unconfirmed evidence does not establish the resulting value or screen. The recordAs option keeps sensitive text out of a recorded replay while sending it to the live app. To append instead of replacing, use type.', longpress: 'Hold a UI target by snapshot ref, selector, or coordinates to open a context menu or perform another hold gesture. The durationMs input overrides the default hold duration.', diff --git a/packages/contracts/src/daemon-command-surface.ts b/packages/contracts/src/daemon-command-surface.ts index dd04171334..5e5ab04ea5 100644 --- a/packages/contracts/src/daemon-command-surface.ts +++ b/packages/contracts/src/daemon-command-surface.ts @@ -196,6 +196,14 @@ export type DaemonPressOptions = CommandContext & Readonly<{ target: InteractionTarget; button?: ClickButton; + /** + * Multi-ref press plan (`press @e13,@e14,...`): every member in dispatch + * order, bare ref bodies; `target` repeats member 0 so the response's ref + * identity keeps the single-ref shape. Absent for a single-ref press. + * The plan is not recordable and cannot carry a replay target guard — + * daemon admission refuses both shapes. + */ + refs?: readonly string[]; expectedResolvedTarget?: ReplayTargetGuardDenotation; /** #1654: a mutating `find`'s already-resolved node. */ preresolvedTarget?: PreresolvedInteractionTarget; diff --git a/packages/contracts/src/interaction.ts b/packages/contracts/src/interaction.ts index fd30cea822..edd81f0787 100644 --- a/packages/contracts/src/interaction.ts +++ b/packages/contracts/src/interaction.ts @@ -476,6 +476,20 @@ export type HoverCommandResponseData = | (TouchResponseRef & TouchHoverExtras) | (TouchResponseSelector & TouchHoverExtras); +/** + * What a multi-ref press sequence (`press @e13,@e14,@e15,@e16`) reports about + * the plan it executed. Present only when the sequence form ran; a single-ref + * press carries no `sequence` field. + */ +export type PressSequenceSummary = { + /** How many keys of the plan were dispatched. Equals the plan length on success. */ + pressed: number; + /** The planned refs in dispatch order, each with its `@` prefix. */ + refs: string[]; + /** Where each dispatched key landed, in plan order. */ + points: Point[]; +}; + /** * Internal runtime result for press/click. The daemon response layer turns * this into `PressCommandResponseData` via `buildInteractionResponseData`. @@ -486,6 +500,7 @@ export type PressCommandResult = ResolvedInteractionTarget & { warning?: string; evidence?: InteractionEvidence; settle?: SettleObservation; + sequence?: PressSequenceSummary; }; /** diff --git a/packages/daemon/src/execution/interaction/internal/__tests__/interaction-touch-targets.test.ts b/packages/daemon/src/execution/interaction/internal/__tests__/interaction-touch-targets.test.ts index ead2ca11dd..eed853245e 100644 --- a/packages/daemon/src/execution/interaction/internal/__tests__/interaction-touch-targets.test.ts +++ b/packages/daemon/src/execution/interaction/internal/__tests__/interaction-touch-targets.test.ts @@ -192,3 +192,148 @@ test('parseFillTarget keeps the established whitespace-only rules', () => { // … while a blank selector-fill argument stays a mistake. expect(parseFillTarget(['label="Email"', ' ']).ok).toBe(false); }); + +// --- Multi-ref press sequence (`press @e13,@e14,...`) --- + +test('parseTouchTarget decodes a comma-separated ref sequence for press', () => { + const parsed = parseTouchTarget(['@e13,@e14,@e15,@e16'], 'press'); + + expect(parsed).toEqual({ + ok: true, + target: { kind: 'ref', ref: '@e13' }, + refs: ['e13', 'e14', 'e15', 'e16'], + refGenerations: [undefined, undefined, undefined, undefined], + }); +}); + +test('parseTouchTarget decodes pinned members and keeps their generations', () => { + const parsed = parseTouchTarget(['@e1~s4,@e2'], 'press'); + + expect(parsed.ok).toBe(true); + if (!parsed.ok) return; + expect(parsed.target).toEqual({ kind: 'ref', ref: '@e1' }); + expect(parsed.refs).toEqual(['e1', 'e2']); + expect(parsed.refGenerations).toEqual([4, undefined]); +}); + +test('parseTouchTarget accepts the sequence form for click and rejects it for other commands', () => { + const clickParsed = parseTouchTarget(['@e1,@e2'], 'click'); + expect(clickParsed.ok).toBe(true); + // Each non-tap-family command refuses through its own branch. + const longpressParsed = parseTouchTarget(['@e1,@e2'], 'longpress'); + expect(longpressParsed.ok).toBe(false); + if (!longpressParsed.ok && !longpressParsed.response.ok) { + expect(longpressParsed.response.error.code).toBe('INVALID_ARGS'); + expect(longpressParsed.response.error.message).toBe( + 'longpress takes one @ref — only press and click accept a comma-separated ref sequence', + ); + } + const hoverParsed = parseTouchTarget(['@e1,@e2'], 'hover'); + expect(hoverParsed.ok).toBe(false); + if (!hoverParsed.ok && !hoverParsed.response.ok) { + expect(hoverParsed.response.error.code).toBe('INVALID_ARGS'); + expect(hoverParsed.response.error.message).toBe( + 'hover takes one @ref — only press and click accept a comma-separated ref sequence', + ); + } +}); + +test('parseTouchTarget rejects a sequence with an empty member', () => { + for (const positional of ['@e1,,@e2', '@e1,', '@e1,,']) { + const parsed = parseTouchTarget([positional], 'press'); + expect(parsed.ok, positional).toBe(false); + if (parsed.ok || parsed.response.ok) continue; + expect(parsed.response.error.code).toBe('INVALID_ARGS'); + expect(parsed.response.error.message).toContain('empty member between commas'); + } + // A single @ref without a comma stays the plain single-ref path. + expect(parseTouchTarget(['@e1'], 'press').ok).toBe(true); +}); + +test('parseTouchTarget keeps a comma token without a leading @ on the selector path', () => { + // A malformed list that never opens with @ is today's selector text, not a sequence. + expect(parseTouchTarget([',@e1'], 'press')).toEqual({ + ok: true, + target: { kind: 'selector', selector: ',@e1' }, + }); +}); + +test('parseTouchTarget keeps a repeated member and rejects non-ref members', () => { + const empty = parseTouchTarget(['@e1,,@e2'], 'press'); + expect(empty.ok).toBe(false); + if (!empty.ok && !empty.response.ok) { + expect(empty.response.error.message).toContain('empty member between commas'); + } + // Selector semantics re-resolve per press, so a repeated ref is the PIN + // digit case, not a mistake: @e1,@e1 presses the same key twice. + const repeated = parseTouchTarget(['@e1,@e1'], 'press'); + expect(repeated.ok).toBe(true); + if (!repeated.ok) return; + expect(repeated.refs).toEqual(['e1', 'e1']); + const nonRef = parseTouchTarget(['@e1,label'], 'press'); + expect(nonRef.ok).toBe(false); + if (!nonRef.ok && !nonRef.response.ok) { + expect(nonRef.response.error.code).toBe('INVALID_ARGS'); + expect(nonRef.response.error.message).toContain('expected a snapshot ref'); + expect(nonRef.response.error.message).toContain('"label"'); + } +}); + +test('parseTouchTarget rejects a sequence above the 16-ref bound', () => { + const positional = Array.from({ length: 17 }, (_, index) => `@e${index + 1}`).join(','); + const overBound = parseTouchTarget([positional], 'press'); + expect(overBound.ok).toBe(false); + if (!overBound.ok && !overBound.response.ok) { + expect(overBound.response.error.code).toBe('INVALID_ARGS'); + expect(overBound.response.error.message).toContain('lists 17 refs'); + expect(overBound.response.error.message).toContain('the maximum is 16'); + } + const atBound = Array.from({ length: 16 }, (_, index) => `@e${index + 1}`).join(','); + expect(parseTouchTarget([atBound], 'press').ok).toBe(true); +}); + +test('parseTouchTarget keeps the fallback label path for a plain ref with trailing text', () => { + const parsed = parseTouchTarget(['@e4', 'Email field'], 'press'); + expect(parsed).toEqual({ + ok: true, + target: { kind: 'ref', ref: '@e4', fallbackLabel: 'Email field' }, + }); +}); + +test('parseTouchTarget refuses a space-separated second ref on press and click', () => { + // `press @e13 @e14` is the comma plan typed with spaces; on the single-ref + // path the second positional would become a fallback label and only the + // first key would ever be pressed. + for (const command of ['press', 'click'] as const) { + const parsed = parseTouchTarget(['@e13', '@e14'], command); + expect(parsed.ok, command).toBe(false); + if (parsed.ok || parsed.response.ok) continue; + expect(parsed.response.error.code).toBe('INVALID_ARGS'); + expect(parsed.response.error.message).toContain('Separate the keys'); + expect(parsed.response.error.hint).toBeTruthy(); + } + // A pinned second ref is the same mistake. + expect(parseTouchTarget(['@e13', '@e14~s4'], 'press').ok).toBe(false); + // Commands outside the tap family keep today's trailing-text behavior: the + // second positional stays a fallback label there. + expect(parseTouchTarget(['@e1', '@e2'], 'longpress').ok).toBe(true); +}); + +test('parseTouchRefSequence refuses a trailing positional after the plan', () => { + // A plan is one positional; a trailing token must not be dropped silently + // while the single-ref path reads the same shape as a fallback label. + const parsed = parseTouchTarget(['@e13,@e14', 'stray'], 'press'); + expect(parsed.ok).toBe(false); + if (parsed.ok || parsed.response.ok) return; + expect(parsed.response.error.code).toBe('INVALID_ARGS'); + expect(parsed.response.error.message).toContain('takes no trailing label'); +}); + +test('parseTouchTarget refuses a malformed generation suffix as a second key', () => { + // `@e14~s` is generation-shaped enough that treating it as a fallback label + // would dispatch only the first key. + const parsed = parseTouchTarget(['@e13', '@e14~s'], 'press'); + expect(parsed.ok).toBe(false); + if (parsed.ok || parsed.response.ok) return; + expect(parsed.response.error.message).toContain('Separate the keys'); +}); diff --git a/packages/daemon/src/execution/interaction/internal/interaction-touch-press-admission.test.ts b/packages/daemon/src/execution/interaction/internal/interaction-touch-press-admission.test.ts index 3c579684c2..410c2cdf41 100644 --- a/packages/daemon/src/execution/interaction/internal/interaction-touch-press-admission.test.ts +++ b/packages/daemon/src/execution/interaction/internal/interaction-touch-press-admission.test.ts @@ -2,6 +2,7 @@ import { test, expect, beforeEach } from 'vitest'; import { attachRefs } from '@agent-device/kernel/snapshot'; import { makeSessionStore } from '../../../__tests__/support/store-factory.ts'; import { activateCompleteRefFrame, refFrameState } from '../../../core/ref-frame.ts'; +import { armAuthoring } from '../../../core/session-script-publication-state.ts'; import { setSessionSnapshot, STALE_SNAPSHOT_REFS_WARNING } from '../../session-snapshot.ts'; import { installInteractorSnapshotResolution, @@ -351,3 +352,160 @@ test('after a session reopen, a pin from the previous lifetime rejects (reseeded } expect(mockTapPoint).not.toHaveBeenCalled(); }); + +// --- Multi-ref press plan: whole-plan admission and the recording guard --- + +test('a plan one member cannot authorize refuses whole and leaves the frame untouched', async () => { + const sessionStore = makeSessionStore(); + const sessionName = 'plan-member-mismatch'; + const session = makeStaleRefSession(sessionName); + session.snapshotGeneration = 15; + sessionStore.publish(sessionName, session); + mockTapPoint.mockRejectedValue(new Error('touch should not be called for an unauthorized plan')); + + // Member 2 pins an epoch the active frame never issued: the whole plan + // refuses before the first dispatch, and the frame keeps its state. + const response = await runInteraction(sessionStore, sessionName, 'press', ['@e1,@e2~s12']); + expect(response?.ok).toBe(false); + if (response && !response.ok) { + expect(response.error.code).toBe('COMMAND_FAILED'); + expect(response.error.details?.reason).toBe('ref_generation_mismatch'); + expect(response.error.details?.dispatched).toBe('no'); + } + expect(mockTapPoint).not.toHaveBeenCalled(); + expect(refFrameState(sessionStore.get(sessionName)!)).toBe('active'); +}); + +test('an expired frame refuses a plan on its first member, before any dispatch', async () => { + const sessionStore = makeSessionStore(); + const sessionName = 'plan-expired-frame'; + const session = makeStaleRefSession(sessionName); + sessionStore.publish(sessionName, session); + mockInteractorSnapshot.mockResolvedValue({ + nodes: makeTwoButtonNodes(), + backend: 'xctest', + producer: 'apple-runner', + }); + + // A selector press crosses the side-effect seam and expires the frame. + const selectorPress = await runInteraction(sessionStore, sessionName, 'press', [ + 'label=Continue', + ]); + expect(selectorPress?.ok).toBe(true); + expect(refFrameState(sessionStore.get(sessionName)!)).toBe('expired'); + const touchCallsBefore = mockTapPoint.mock.calls.length; + + const response = await runInteraction(sessionStore, sessionName, 'press', ['@e1,@e2']); + expect(response?.ok).toBe(false); + if (response && !response.ok) { + expect(response.error.details?.reason).toBe('ref_frame_expired'); + expect(response.error.details?.dispatched).toBe('no'); + } + expect(mockTapPoint).toHaveBeenCalledTimes(touchCallsBefore); +}); + +test('a plan with distinct stale pins joins its warnings in dispatch order', async () => { + const sessionStore = makeSessionStore(); + const sessionName = 'plan-warning-join'; + const session = makeStaleRefSession(sessionName); + session.snapshotGeneration = 15; + sessionStore.publish(sessionName, session); + mockTapPoint.mockRejectedValue(new Error('touch should not be called for a stale plan')); + + const response = await runInteraction(sessionStore, sessionName, 'press', ['@e1~s11,@e2~s12']); + expect(response?.ok).toBe(false); + if (response && !response.ok) { + // The refusal carries the typed admission details beside the joined hint: + // pre-dispatch, and the frame admission refused a member's stale pin. + expect(response.error.details?.dispatched).toBe('no'); + expect(response.error.details?.reason).toBe('ref_generation_mismatch'); + // The refusal hint carries both members' warnings, in plan order. + expect(String(response.error.details?.hint)).toBe( + "Ref @e1 was minted from snapshot s11 but the session's ref frame is now s15 — re-run snapshot -i. " + + "Ref @e2 was minted from snapshot s12 but the session's ref frame is now s15 — re-run snapshot -i.", + ); + } + expect(mockTapPoint).not.toHaveBeenCalled(); +}); + +test('a plan on an expired frame warns once, not once per member', async () => { + const sessionStore = makeSessionStore(); + const sessionName = 'plan-warning-dedupe'; + const session = makeStaleRefSession(sessionName); + sessionStore.publish(sessionName, session); + mockInteractorSnapshot.mockResolvedValue({ + nodes: makeTwoButtonNodes(), + backend: 'xctest', + producer: 'apple-runner', + }); + await runInteraction(sessionStore, sessionName, 'press', ['label=Continue']); + expect(refFrameState(sessionStore.get(sessionName)!)).toBe('expired'); + mockTapPoint.mockRejectedValue(new Error('touch should not be called for an expired plan')); + + // Every member's staleness warning is the same expired-frame sentence, so + // the join collapses to one instance instead of one per key. + const response = await runInteraction(sessionStore, sessionName, 'press', ['@e1,@e2']); + expect(response?.ok).toBe(false); + if (response && !response.ok) { + expect(String(response.error.details?.hint)).toBe(STALE_SNAPSHOT_REFS_WARNING); + expect(response.error.details?.dispatched).toBe('no'); + expect(response.error.details?.reason).toBe('ref_frame_expired'); + } +}); + +test('script recording refuses a multi-ref press plan, before any dispatch', async () => { + const sessionStore = makeSessionStore(); + const sessionName = 'plan-recording-refused'; + const session = makeStaleRefSession(sessionName); + session.scriptPublication = armAuthoring({ path: '/tmp/ad-plan-recording.ad', force: false }); + sessionStore.publish(sessionName, session); + mockTapPoint.mockRejectedValue(new Error('touch should not be called for a recorded plan')); + + const response = await runInteraction(sessionStore, sessionName, 'press', ['@e1,@e2']); + expect(response?.ok).toBe(false); + if (response && !response.ok) { + expect(response.error.code).toBe('INVALID_ARGS'); + expect(response.error.message).toMatch(/cannot be recorded/); + expect(response.error.details?.dispatched).toBe('no'); + } + expect(mockTapPoint).not.toHaveBeenCalled(); + expect(refFrameState(sessionStore.get(sessionName)!)).toBe('active'); +}); + +test('a multi-ref press plan refuses a replay target guard instead of downgrading to one key', async () => { + const sessionStore = makeSessionStore(); + const sessionName = 'plan-replay-guard-refused'; + sessionStore.publish(sessionName, makeStaleRefSession(sessionName)); + mockTapPoint.mockRejectedValue(new Error('touch should not be called for a guarded plan step')); + + const response = await handleInteractionCommands({ + createCommandSurface: touchCreateCommandSurface, + req: { + token: 't', + session: sessionName, + command: 'press', + positionals: ['@e1,@e2'], + flags: {}, + internal: { + replayTargetGuard: { + identity: { role: 'Button', label: 'Continue' }, + structural: { documentOrder: 1, sibling: 0 }, + }, + }, + }, + sessionName, + sessionStore, + contextFromFlags, + ...getRuntimeBindings(), + }); + + expect(response?.ok).toBe(false); + if (response === null || response.ok) { + throw new Error(`expected a refusal, got ${JSON.stringify(response)}`); + } + expect(response.error.code).toBe('INVALID_ARGS'); + expect(response.error.message).toMatch(/cannot carry a replay target guard/); + expect(response.error.details?.reason).toBe('sequence-replay-guard-unsupported'); + expect(response.error.details?.dispatched).toBe('no'); + expect(mockTapPoint).not.toHaveBeenCalled(); +}); diff --git a/packages/daemon/src/execution/interaction/internal/interaction-touch-press-admission.ts b/packages/daemon/src/execution/interaction/internal/interaction-touch-press-admission.ts index 7172f514f0..f36a507e15 100644 --- a/packages/daemon/src/execution/interaction/internal/interaction-touch-press-admission.ts +++ b/packages/daemon/src/execution/interaction/internal/interaction-touch-press-admission.ts @@ -8,6 +8,7 @@ import { } from '@agent-device/contracts/click-button'; import { publicPlatformString } from '@agent-device/kernel/device'; import { resolveRefStalenessWarning } from '../../session-snapshot.ts'; +import { isSessionRecording } from '../../../core/session-script-publication-capability.ts'; import { readRefMutationFrame } from '../../../core/ref-frame.ts'; import type { SessionState } from '../../../core/session-state.ts'; import { refMutationAdmissionResponse } from './interaction-ref-policy.ts'; @@ -54,6 +55,8 @@ export type AdmittedTargetedTouch = { durationMs: number | undefined; staleRefsWarning: string | undefined; refContext: RefAdmissionContext | undefined; + /** Multi-ref press plan members in dispatch order, bare bodies; absent for a single-ref target. */ + refs: readonly string[] | undefined; }; type ParsedTargetedTouch = Extract; @@ -86,6 +89,8 @@ async function readTargetedTouchAdmission( ? parseLongPressTarget(positionals) : parseTouchTarget(positionals, commandLabel); if (!parsedTarget.ok) return { response: parsedTarget.response }; + const sequenceGuardResponse = sequenceRecordingGuardResponse(session, req, parsedTarget); + if (sequenceGuardResponse) return { response: sequenceGuardResponse }; const staleRefsWarning = readTargetedTouchStalenessWarning(session, req, parsedTarget); const refAdmission = admitTargetedTouchRef( params, @@ -107,6 +112,7 @@ async function readTargetedTouchAdmission( durationMs: command === 'longpress' ? parsedTarget.durationMs : undefined, staleRefsWarning, refContext: readRefAdmissionContext(req, parsedTarget, staleRefsWarning), + refs: parsedTarget.refs, }, }; } @@ -158,6 +164,7 @@ function clickButtonValidationResponse( * while the frame is expired. A mutating `find`'s internal dispatch supplies a * locator-minted ref (`internal.findResolvedTarget`), so it carries no * user-facing staleness — the caller never consumed a `@ref` (ADR 0014). + * A multi-ref plan warns per member; distinct warnings join in dispatch order. */ function readTargetedTouchStalenessWarning( session: SessionState, @@ -166,6 +173,18 @@ function readTargetedTouchStalenessWarning( ): string | undefined { if (parsedTarget.target.kind !== 'ref') return undefined; if (req.internal?.findResolvedTarget !== undefined) return undefined; + if (parsedTarget.refs) { + const warnings = new Set(); + parsedTarget.refs.forEach((ref, index) => { + const warning = resolveRefStalenessWarning({ + session, + ref: `@${ref}`, + mintedGeneration: parsedTarget.refGenerations?.[index], + }); + if (warning) warnings.add(warning); + }); + return warnings.size > 0 ? Array.from(warnings).join(' ') : undefined; + } return resolveRefStalenessWarning({ session, ref: parsedTarget.target.ref, @@ -173,6 +192,37 @@ function readTargetedTouchStalenessWarning( }); } +/** + * A multi-ref press plan can be neither recorded nor replayed through a target + * guard. The recorder binds one target (and its recording-time evidence) to + * each recorded step, and a guarded replay dispatch resolves through the + * single-ref path — a plan that slipped into either shape would replay its + * first key only and report the remaining keys as sent. Refusing at admission + * keeps the refusal pre-dispatch: the frame keeps its pre-command state. + */ +function sequenceRecordingGuardResponse( + session: SessionState, + req: InteractionRouteInput['req'], + parsedTarget: ParsedTargetedTouch, +): DaemonResponse | undefined { + if (!parsedTarget.refs) return undefined; + if (isSessionRecording(session) && !req.flags?.noRecord) { + return errorResponse( + 'INVALID_ARGS', + 'A multi-ref press sequence cannot be recorded: the recorder binds one target per step, so the recorded step would press only the first key. Record one press per key, or pass --no-record on this step.', + { reason: 'sequence-recording-unsupported' }, + ); + } + if (req.internal?.replayTargetGuard !== undefined) { + return errorResponse( + 'INVALID_ARGS', + 'A multi-ref press sequence cannot carry a replay target guard: the guard binds one target, so the replayed step would press only the first key. Re-record the journey as one press per key.', + { reason: 'sequence-replay-guard-unsupported' }, + ); + } + return undefined; +} + function readRefAdmissionContext( req: InteractionRouteInput['req'], parsedTarget: ParsedTargetedTouch, @@ -201,18 +251,33 @@ function admitTargetedTouchRef( req.flags, ); if (invalidRefFlagsResponse) return { response: invalidRefFlagsResponse }; - const admissionResponse = req.internal?.findResolvedTarget - ? null - : refMutationAdmissionResponse({ - ref: parsedTarget.target.ref, - mintedGeneration: parsedTarget.refGeneration, - staleRefsWarning, - frame: readRefMutationFrame({ - session, + if (req.internal?.findResolvedTarget) return {}; + // ADR 0014: every member of a multi-ref plan is admitted against the active + // frame before the first dispatch, so a plan one key cannot authorize + // refuses whole and the frame keeps its pre-command state. + const members: { ref: string; mintedGeneration: number | undefined }[] = parsedTarget.refs + ? parsedTarget.refs.map((ref, index) => ({ + ref: `@${ref}`, + mintedGeneration: parsedTarget.refGenerations?.[index], + })) + : [ + { ref: parsedTarget.target.ref, mintedGeneration: parsedTarget.refGeneration, - }), - }); - if (admissionResponse) return { response: admissionResponse }; + }, + ]; + for (const member of members) { + const admissionResponse = refMutationAdmissionResponse({ + ref: member.ref, + mintedGeneration: member.mintedGeneration, + staleRefsWarning, + frame: readRefMutationFrame({ + session, + ref: member.ref, + mintedGeneration: member.mintedGeneration, + }), + }); + if (admissionResponse) return { response: admissionResponse }; + } return {}; } diff --git a/packages/daemon/src/execution/interaction/internal/interaction-touch-press.ts b/packages/daemon/src/execution/interaction/internal/interaction-touch-press.ts index ec0127dfbc..bd70e4641a 100644 --- a/packages/daemon/src/execution/interaction/internal/interaction-touch-press.ts +++ b/packages/daemon/src/execution/interaction/internal/interaction-touch-press.ts @@ -113,6 +113,7 @@ function buildTargetedRuntimeOptions( runtime, command, target, + refs: admitted.refs, sessionName, requestId: req.meta?.requestId, clickButton: admitted.clickButton, @@ -158,6 +159,7 @@ async function runTargetedTouchInteraction(params: { runtime: Awaited>; command: TargetedTouchCommand; target: InteractionTarget; + refs: readonly string[] | undefined; sessionName: string; requestId: string | undefined; clickButton: ReturnType; @@ -194,6 +196,7 @@ function pressRuntimeOptions( params: { clickButton: ReturnType; flags: CommandFlags | undefined; + refs: readonly string[] | undefined; preresolvedTarget?: PreresolvedInteractionTarget; }, shared: { @@ -214,6 +217,9 @@ function pressRuntimeOptions( jitterPx: flags?.jitterPx, doubleTap: flags?.doubleTap, verify: flags?.verify, + // The multi-ref press plan: present only when the request named a + // comma-separated ref sequence. + ...(params.refs ? { refs: params.refs } : {}), // Only click/press take it: `find` dispatches click and fill, never // longpress or hover, so declaring it on their options would be an // unconsumed claim (#1649 review). diff --git a/packages/daemon/src/execution/interaction/internal/interaction-touch-targets.ts b/packages/daemon/src/execution/interaction/internal/interaction-touch-targets.ts index 7e87b66394..926c6dcb7d 100644 --- a/packages/daemon/src/execution/interaction/internal/interaction-touch-targets.ts +++ b/packages/daemon/src/execution/interaction/internal/interaction-touch-targets.ts @@ -14,10 +14,25 @@ import { parseCoordinateTarget } from './interaction-targeting.ts'; import { parseVersionedRefPositional } from '../../ref-positionals.ts'; import { errorResponse } from '@agent-device/kernel/contracts'; +/** What a decoded touch target carries: the target plus optional ref-plan provenance. */ +type ParsedTouchTargetOk = { + target: InteractionTarget; + refGeneration?: number; + /** Multi-ref press sequence members in dispatch order, bare ref bodies (`e13`). */ + refs?: string[]; + refGenerations?: (number | undefined)[]; +}; + export type ParsedTouchTarget = - | { ok: true; target: InteractionTarget; refGeneration?: number; durationMs?: never } + | (ParsedTouchTargetOk & { + ok: true; + /** Only `longpress` decodes a duration. */ durationMs?: never; + }) | { ok: false; response: DaemonResponse }; +/** The tap family members a multi-ref sequence is admitted on: the commands that own `--count`. */ +const SEQUENCE_TOUCH_COMMANDS: ReadonlySet = new Set(['press', 'click']); + export function parseTouchTarget(positionals: string[], commandLabel: string): ParsedTouchTarget { const coordinates = parseCoordinateTarget(positionals); if (coordinates) { @@ -25,6 +40,38 @@ export function parseTouchTarget(positionals: string[], commandLabel: string): P } const first = positionals[0] ?? ''; if (first.startsWith('@')) { + if (first.includes(',')) { + if (!SEQUENCE_TOUCH_COMMANDS.has(commandLabel)) { + return { + ok: false, + response: errorResponse( + 'INVALID_ARGS', + `${commandLabel} takes one @ref — only press and click accept a comma-separated ref sequence`, + ), + }; + } + return parseTouchRefSequence(positionals); + } + const spaceSeparatedRef = positionals[1]; + if ( + SEQUENCE_TOUCH_COMMANDS.has(commandLabel) && + spaceSeparatedRef !== undefined && + /^@e\d+(~s?\d*)?$/.test(spaceSeparatedRef.trim()) + ) { + // `press @e13 @e14` is a comma-separated plan typed with spaces. On the + // single-ref path the extra positional would become a fallback label and + // every key after the first would be lost to a silent success — the + // exact failure class the sequence form exists to remove. + return { + ok: false, + response: errorResponse( + 'INVALID_ARGS', + `"${spaceSeparatedRef.trim()}" looks like a second key. Separate the keys of a ref sequence with commas: @e13,@e14.`, + undefined, + { hint: REF_SEQUENCE_HINT }, + ), + }; + } const versioned = parseVersionedRefPositional(first); if (!versioned.ok) return { ok: false, response: versioned.response }; return { @@ -50,8 +97,92 @@ export function parseTouchTarget(positionals: string[], commandLabel: string): P return { ok: true, target: { kind: 'selector', selector } }; } +const TOUCH_SEQUENCE_MAX_REFS = 16; + +/** + * The multi-ref press plan (`press @e13,@e14,@e15,@e16`): every comma-separated + * token must parse as a versioned snapshot ref and normalize to the `e` + * body the frame issues, and the plan names at least two distinct keys. The + * first member stays the command's `target` so the response's ref identity + * keeps the single-ref shape; the full plan rides beside it. The form is not + * recordable — the recorder binds one target per step — so daemon admission + * refuses it while a script recording is armed, and a replay dispatch that + * still carries a target guard for it refuses instead of downgrading to the + * first key. + */ +function parseTouchRefSequence(positionals: string[]): ParsedTouchTarget { + if (positionals.length > 1) { + // A plan is one positional. A trailing token would otherwise be dropped + // silently, while the same command on the single-ref path reads it as a + // fallback label — one trailing rule per form, stated instead of guessed. + return { + ok: false, + response: errorResponse( + 'INVALID_ARGS', + `Unexpected argument "${positionals.slice(1).join(' ')}" after the ref sequence — a multi-ref plan takes no trailing label.`, + undefined, + { hint: REF_SEQUENCE_HINT }, + ), + }; + } + const members = positionals[0]!.split(',').map((token) => token.trim()); + if (members.some((token) => token.length === 0)) { + return { + ok: false, + response: errorResponse( + 'INVALID_ARGS', + `Invalid ref sequence "${positionals[0]}" — empty member between commas.`, + undefined, + { hint: REF_SEQUENCE_HINT }, + ), + }; + } + if (members.length > TOUCH_SEQUENCE_MAX_REFS) { + return { + ok: false, + response: errorResponse( + 'INVALID_ARGS', + `Ref sequence "${positionals[0]}" lists ${members.length} refs; the maximum is ${TOUCH_SEQUENCE_MAX_REFS}.`, + undefined, + { hint: REF_SEQUENCE_HINT }, + ), + }; + } + const parsed: { ref: string; generation?: number }[] = []; + for (const member of members) { + const versioned = parseVersionedRefPositional(member); + if (!versioned.ok) return { ok: false, response: versioned.response }; + const body = versioned.ref.startsWith('@') ? versioned.ref.slice(1) : versioned.ref; + if (!/^e\d+$/.test(body)) { + return { + ok: false, + response: errorResponse( + 'INVALID_ARGS', + `Invalid ref "${member}" in sequence — expected a snapshot ref.`, + undefined, + { hint: REF_SEQUENCE_HINT }, + ), + }; + } + parsed.push({ + ref: body, + ...(versioned.generation !== undefined ? { generation: versioned.generation } : {}), + }); + } + const bodies = parsed.map((member) => member.ref); + return { + ok: true, + target: { kind: 'ref', ref: `@${bodies[0]}` }, + refs: bodies, + refGenerations: parsed.map((member) => member.generation), + }; +} + +const REF_SEQUENCE_HINT = + 'List each snapshot ref in press order, for example press @e13,@e14,@e15,@e16; a repeated ref presses the same key twice (a PIN digit). The whole plan resolves and admits against the current frame before the first press, so a member that cannot resolve refuses before anything dispatches; after the first press later keys tap where the snapshot showed them, and a later key that refuses carries its reason in details.refPlanKey, the keys already sent in details.dispatchedSteps, and the keys that did not run in details.refPlanPending.'; + export type ParsedLongPressTarget = - | { ok: true; target: InteractionTarget; refGeneration?: number; durationMs?: number } + | (ParsedTouchTargetOk & { ok: true; durationMs?: number }) | { ok: false; response: DaemonResponse }; export function parseLongPressTarget(positionals: string[]): ParsedLongPressTarget { @@ -161,9 +292,12 @@ export function interactionResultExtra( // `evidence` (#1047, opt-in via --verify) is additive on press/fill only — // LongPressCommandResult has no evidence field, so it reads as undefined // (and gets dropped by the response layer) for longpress. `settle` (#1101, - // opt-in via --settle) is additive on all four touch commands. + // opt-in via --settle) is additive on all four touch commands. `sequence` + // is additive on press only (the multi-ref plan summary). const evidence = 'evidence' in result ? result.evidence : undefined; const settle = result.settle; + const sequence = 'sequence' in result ? result.sequence : undefined; + const shared = sequence ? { sequence } : {}; if (result.kind === 'ref') { return { ref: stripAtPrefix(result.target?.kind === 'ref' ? result.target.ref : undefined), @@ -174,6 +308,7 @@ export function interactionResultExtra( evidence, settle, resolution: result.resolution, + ...shared, }; } if (result.kind === 'selector') { @@ -186,9 +321,10 @@ export function interactionResultExtra( evidence, settle, resolution: result.resolution, + ...shared, }; } - return { evidence, settle }; + return { evidence, settle, ...shared }; } export function formatTouchTargetLabel( diff --git a/src/cli/__tests__/command-explain.test.ts b/src/cli/__tests__/command-explain.test.ts index e9d84afce0..472dbd639e 100644 --- a/src/cli/__tests__/command-explain.test.ts +++ b/src/cli/__tests__/command-explain.test.ts @@ -88,7 +88,7 @@ describe('explainCommand', () => { mcp: true, postActionObservation: 'settle-and-verify', }, - cli: { usage: 'click ' }, + cli: { usage: 'click ' }, }, }); if (!result.found) return; diff --git a/src/commands/schema/cli-help-gestures.ts b/src/commands/schema/cli-help-gestures.ts index e8debbd501..e6806a88a0 100644 --- a/src/commands/schema/cli-help-gestures.ts +++ b/src/commands/schema/cli-help-gestures.ts @@ -23,7 +23,7 @@ Shapes: agent-device gesture tap 80 300 200 450 310 600 --count 20 --pause-ms 100 scroll --until repeats scroll-and-check passes until that element is on screen, then stops: one request instead of a scroll-then-snapshot loop, and it stops on the target rather than overshooting it. It reports the passes it spent; vertical searches compare repeated captures of descendants in the selected scroll container and fail when those captures show no progress. If no container is found, they compare the visible tree. It is refused on top/bottom, which already stop themselves. The default limit is ${DEFAULT_SCROLL_UNTIL_PASSES} gestures. For a longer list, add --max-passes (1–${SCROLL_UNTIL_MAX_PASSES}); it requires --until. Large pass budgets can hit the 90-second request timeout; inspect the current screen and continue with another search if that happens. longpress accepts coordinates, @refs, or selectors; prefer @ref/selector, coordinates only as a fallback. Duration and gesture scale/center are positional. gesture pan is one finger by default; add --pointer-count 2 for a parallel two-finger pan. Keep count/pause/pattern on one swipe: --count (cap 200), --pause-ms (cap 10000ms), --pattern ping-pong; the combined swipe/pause schedule is capped at 60000ms. - For repeated iOS smoke checks: press --count --jitter-px for tap series, swipe --count for drag series. gesture tap touches 2 to 5 distinct points and holds them for --hold-ms (default 50, min 16); --count repeats the same contacts (cap 200) with --pause-ms between, the series must fit 60000ms counting about 500ms per repetition, and a repetition costs about 450ms on a physical iPhone. Use press for one finger. + For repeated iOS smoke checks: press --count --jitter-px for tap series, swipe --count for drag series. gesture tap touches 2 to 5 distinct points and holds them for --hold-ms (default 50, min 16); --count repeats the same contacts (cap 200) with --pause-ms between, the series must fit 60000ms counting about 500ms per repetition, and a repetition costs about 450ms on a physical iPhone. Use press for one finger. For repeated keys (PIN pad), enter the plan in one command: press @e13,@e14,@e15,@e16 (2 to 16 refs, one press per key, a repeated ref presses the same key twice, click takes the same form); the whole plan resolves and admits against the current frame before the first press, key 1 expires the frame, later keys tap where the snapshot showed them, so re-snapshot and rebuild the plan when the screen can move between keys; a later key that refuses carries details.dispatchedSteps counting the keys already sent, --interval-ms sets the key delay, and --count/--hold-ms/--jitter-px/--double-tap apply to every key. Platform quirks: If rotation stops repeated iOS presses, inspect the screen before continuing with the remaining presses. diff --git a/src/mcp/__tests__/tool-ref-pins.test.ts b/src/mcp/__tests__/tool-ref-pins.test.ts index 4315010ae1..e7399824b6 100644 --- a/src/mcp/__tests__/tool-ref-pins.test.ts +++ b/src/mcp/__tests__/tool-ref-pins.test.ts @@ -564,3 +564,55 @@ test('ref-pin store leaves pins untouched for a scroll with no settle payload', ref: '@e2~s7', }); }); + +test('ref-pin store pins every member of a comma-separated ref plan from one snapshot', () => { + const pins = makeStore(); + + pins.mergeCommandResult( + 'snapshot', + { + nodes: [{ ref: 'e13' }, { ref: 'e14' }, { ref: 'e15' }, { ref: 'e16' }], + truncated: false, + refsGeneration: 500020, + }, + undefined, + 'demo', + ); + + const pinned = pins.pinInput('press', { session: 'demo', ref: '@e13,@e14,@e15,@e16' }, undefined); + assert.deepEqual(pinned, { + session: 'demo', + ref: '@e13~s500020,@e14~s500020,@e15~s500020,@e16~s500020', + }); +}); + +test('ref-pin store pins a repeated plan member twice and leaves unknown members bare', () => { + const pins = makeStore(); + + pins.mergeCommandResult( + 'snapshot', + { nodes: [{ ref: 'e1' }, { ref: 'e2' }], truncated: false, refsGeneration: 6 }, + undefined, + 'demo', + ); + + const repeated = pins.pinInput('press', { session: 'demo', ref: '@e1,@e2,@e1' }, undefined); + assert.deepEqual(repeated, { session: 'demo', ref: '@e1~s6,@e2~s6,@e1~s6' }); + + const unknownMember = pins.pinInput('press', { session: 'demo', ref: '@e1,@e99' }, undefined); + assert.deepEqual(unknownMember, { session: 'demo', ref: '@e1~s6,@e99' }); +}); + +test('ref-pin store pins the unpinned members of a partly pinned plan', () => { + const pins = makeStore(); + + pins.mergeCommandResult( + 'snapshot', + { nodes: [{ ref: 'e13' }, { ref: 'e14' }], truncated: false, refsGeneration: 500021 }, + undefined, + 'demo', + ); + + const pinned = pins.pinInput('press', { session: 'demo', ref: '@e13~s4,@e14' }, undefined); + assert.deepEqual(pinned, { session: 'demo', ref: '@e13~s4,@e14~s500021' }); +}); diff --git a/src/mcp/tool-ref-pins.ts b/src/mcp/tool-ref-pins.ts index 0810b511bf..98fe7a7800 100644 --- a/src/mcp/tool-ref-pins.ts +++ b/src/mcp/tool-ref-pins.ts @@ -320,8 +320,17 @@ function pinRef(ref: string, pins: Map): string { // Only pin the canonical plain form `@e12`: an existing `~` means the ref is // already pinned (or malformed — the daemon owns rejecting that), and a // missing `@` prefix is not a ref the daemon would accept anyway. Refs with - // no recorded provenance pass through unpinned — never guess. - if (!ref.startsWith('@') || ref.includes('~')) return ref; + // no recorded provenance pass through unpinned — never guess. The comma + // split runs first so a plan with one already-pinned member still pins the + // others: `@e13~s4,@e14` forwards `@e13~s4` untouched and pins `@e14`. + if (!ref.startsWith('@')) return ref; + if (ref.includes(',')) { + return ref + .split(',') + .map((member) => pinRef(member.trim(), pins)) + .join(','); + } + if (ref.includes('~')) return ref; const generation = pins.get(ref.slice(1)); return generation === undefined ? ref : `${ref}~s${generation}`; } diff --git a/website/docs/docs/commands.md b/website/docs/docs/commands.md index f0abcae1df..0d88f74953 100644 --- a/website/docs/docs/commands.md +++ b/website/docs/docs/commands.md @@ -468,6 +468,8 @@ agent-device alert dismiss ```bash agent-device click @e1 agent-device click @e1 --button secondary # macOS secondary click / context menu +agent-device click @e13,@e14 # one plan, one admission pass +agent-device press @e13,@e14 # a two-key plan; up to 16 refs agent-device focus @e2 agent-device fill @e2 "text" # Replace the field value agent-device fill @e2 "search" --delay-ms 100 @@ -507,6 +509,8 @@ Standalone `--verify` captures in the pre-action tree's view, which can differ f A `click` or `press` whose target the daemon resolves to a point (`@ref` or selector) re-checks that target's position before dispatching when the session's previous action animated the screen, and dispatches where it came to rest. When that check cannot confirm a stable position — the target was still moving, or a re-check could not measure it — the command refuses with `target_still_moving` and `dispatched: no`: nothing was dispatched, the point the command resolved is in `error.details.rect`, and where the target was last seen is in `error.details.movedToRect` when that last look measured one. Run `wait ` on the target and retry the same command, or add `--settle` so the action waits for the UI to go quiet. A refused `@ref` press needs a fresh `snapshot -i` and a selector, because the wait retires the ref. Two dispatch paths hand off before the check: a press at raw coordinates has no resolved target to re-check, and a dispatch the backend performs on the element itself — `click @ref` on a backend with element dispatch, or a Maestro non-hittable coordinate fallback — goes to that backend rather than through a resolved point. The re-check stops when the request is canceled or reaches its deadline, and nothing is dispatched (`dispatched: no`): a canceled request fails with `details.reason: request_canceled`, and a request out of time with `details.reason: daemon_request_timeout`. +A comma-separated ref plan — `press @e13,@e14` or the same form on `click` — dispatches 2 to 16 snapshot refs as one command. The whole plan resolves and admits against the current ref frame before the first press (run `snapshot -i` first and copy the refs exactly); a repeated ref dispatches the same key twice. The first press expires the frame, and the later keys dispatch at the points the snapshot admitted — no member re-resolves. A refusal before the first press leaves the frame active with `dispatched: no`; a refusal after the first press reports `details.dispatchedSteps` (keys already dispatched), `details.refPlanKey` (the refused key), and `details.refPlanPending` (the keys that did not run) — replan from a fresh `snapshot -i`, because the frame stays expired; a failed plan returns those error details, not `sequence.points`. On a successful response, `ref` stays the first key and `sequence.points` holds every dispatched point. While script recording is armed, the plan refuses unless the step passes `--no-record`, because the recorder binds one target per step: record one press per key. This is ADR 0014's multi-ref logical command; see [ADR 0014](https://github.com/callstack/agent-device/blob/main/docs/adr/0014-session-ref-frame-lifetime.md). + ### Text entry - `fill` replaces the field value. `type` appends to the focused field and accepts text only: use `fill @ref "text"` to target a field, or `press @ref` then `type "text"`.