Skip to content

Mobile taps omit the snapshot version from control references #1406

Closed

Description

@gabimoncha

Note

This issue was reported by Codex (GPT-6.1 Sol) with @gabimoncha's permission.

What happened

The mobile engine sends control references without their snapshot version, so agent-device cannot check whether a tap uses an old screen capture.

A control reference such as @e1 identifies an element in a captured screen. refsGeneration identifies that capture's version. For version 7, the command must send @e1~s7.

Expected: a tap includes the version that identified its control.

Actual: the engine receives refsGeneration: 7 but sends plain @e1.

Minimal reproduction

This HTTP fixture changes the button to Reopen only when the tap includes version 7. It needs no simulator or model. The final assertion fails on @e2e-dev/mobile 0.12.4.

// e2e.config.ts
import type { E2EConfig } from 'e2e';

export default {
	tests: '*.e2e.ts',
	targets: [{ name: 'protocol', platform: 'api' }],
	workers: 1,
} satisfies E2EConfig;

// generation.e2e.ts
import { test } from 'e2e';
import { mobile } from '@e2e-dev/mobile';
import { strictEqual } from 'node:assert/strict';
import { randomUUID } from 'node:crypto';
import { createServer } from 'node:http';
import type { AddressInfo } from 'node:net';

test('taps include the snapshot version', async () => {
	const simulatorId = randomUUID();
	let completed = false;
	const server = createServer(async (request, response) => {
		response.setHeader('content-type', 'application/json');
		if (request.method === 'GET') {
			response.end(JSON.stringify({
				ok: true, service: 'agent-device', version: '0.21.29', rpcProtocolVersion: 2,
			}));
			return;
		}
		let body = '';
		for await (const chunk of request) body += chunk;
		const { id, params } = JSON.parse(body);
		if (params.command === 'press') completed = params.positionals[0] === '@e1~s7';
		const data = params.command === 'snapshot' ? {
			refsGeneration: 7,
			nodes: [
				{ index: 0, ref: 'e0', type: 'Application', label: 'Example', depth: 0,
					rect: { x: 0, y: 0, width: 400, height: 800 } },
				{ index: 1, ref: 'e1', type: 'Button', label: completed ? 'Reopen' : 'Complete',
					depth: 1, parentIndex: 0, enabled: true, hittable: true,
					rect: { x: 10, y: 10, width: 60, height: 44 } },
			],
		} : {};
		response.end(JSON.stringify({ jsonrpc: '2.0', id, result: { ok: true, data } }));
	});
	await new Promise<void>((resolve) => server.listen(0, '127.0.0.1', resolve));
	const previous = process.env.AGENT_DEVICE_DAEMON_BASE_URL;
	process.env.AGENT_DEVICE_DAEMON_BASE_URL =
		`http://127.0.0.1:${(server.address() as AddressInfo).port}`;
	try {
		const engine = mobile({ platform: 'ios', device: simulatorId });
		const signal = AbortSignal.timeout(5000);
		await engine.init?.({
			runId: 'repro', targetName: 'protocol', projectRoot: process.cwd(),
			app: { bundleId: 'com.example.app' }, env: {}, headed: false, workerSlot: 0, signal, log() {},
		});
		const operation = {
			signal, timeoutMs: 5000, runId: 'repro', attemptId: 'repro', origin: 'test' as const,
		};
		const before = await engine.observe?.(operation);
		const button = before?.root.children?.[0]?.children?.[0];
		if (!button) throw new Error('The Complete button is missing');
		await engine.perform?.(button.ref, { kind: 'tap' }, operation);
		const after = await engine.observe?.(operation);
		strictEqual(after?.root.children?.[0]?.children?.[0]?.name, 'Reopen');
	} finally {
		if (previous === undefined) delete process.env.AGENT_DEVICE_DAEMON_BASE_URL;
		else process.env.AGENT_DEVICE_DAEMON_BASE_URL = previous;
		server.closeAllConnections();
		await new Promise<void>((resolve) => server.close(() => resolve()));
	}
});

Versions

  • e2e CLI: 0.21.3
  • @e2e-dev/mobile: 0.12.4
  • agent-device: 0.21.29
  • Node: 24.22.1
  • macOS: 27.2

Engine

@e2e-dev/mobile

Anything else

The test raises Node's ERR_ASSERTION error. The E2E runner prints:

ERROR: Expected values to be strictly equal:
+ actual - expected
+ 'Complete'
- 'Reopen'

The installed agent-device types state: "pair a ref with it (@e12~s<refsGeneration>) before a mutation."

The proven cause is that AgentDeviceSurface.project() drops refsGeneration, and actionTarget() sends a plain reference. A local patch retains the version with each control reference and includes it in the command. The reproduction passes with that patch.

Activity

  1. changed the title [-]Mobile engine discards snapshot ref generation before mutations[/-] [+]Mobile taps omit the snapshot version from control references[/+] on Oct 27, 2026
  2. okwasniewski commented on Oct 27, 2026

    @okwasniewski
    Member

    Thanks, confirmed on main (agent-device 0.21.31). actionTarget (packages/mobile/src/surface.ts:1106) sends plain @eN, and RawSnapshot never reads refsGeneration, so the daemon cannot tell which snapshot a ref came from.

    On a live simulator: a ref from an interactive snapshot, sent plain after a newer full snapshot (as locate takes), tapped a different element. The same ref pinned as @e5~sN was rejected with ref_generation_mismatch, which the engine already maps to NODE_STALE (errors.ts:85).

    Fix: keep each node's refsGeneration through project() / bind() and send @eN~sG on every ref action. #1300 adds a separate e2e-side stale check and does not cover this.

    This was an automated review.

  3. samanyudas commented on Oct 27, 2026

    @samanyudas
    Contributor

    I'm working on a fix for this.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions