Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
17 commits
Select commit Hold shift + click to select a range
650c325
feat(harmonyos): inject gesture pan as a densified touch-panel trajec…
strzhao Nov 6, 2026
658dd02
fix(harmonyos): align transform test fixtures with the resolver's sha…
strzhao Nov 6, 2026
6be869b
refactor(harmonyos): keep the injection timeout budget constants modu…
strzhao Nov 6, 2026
da0af4b
docs(help): fold the HarmonyOS pan transport line into the existing q…
strzhao Nov 6, 2026
59a2bb5
fix(harmonyos): pace the injection per spawned process and re-probe t…
strzhao Nov 6, 2026
df4ecc3
docs(evidence): record a densified pan dragging a real slider to a 3p…
strzhao Nov 8, 2026
c3999af
refactor(contracts): own the densified gesture path next to its sampler
strzhao Nov 12, 2026
7a994b6
refactor(harmonyos): drop the sendevent pan transport until a readabl…
strzhao Nov 12, 2026
9778249
docs(evidence): correct the slider pass to the endpoint swipe it took
strzhao Nov 12, 2026
d41e77e
fix(contracts): keep both path endpoints clear of the progress hook
strzhao Nov 12, 2026
181935c
refactor(contracts): take the gesture duration from its last canonica…
strzhao Nov 13, 2026
15c872b
chore: retrigger CI after the live-emulator back-navigation flake
strzhao Nov 13, 2026
e32297c
test(gestures): pin typed short-path refusal and qualify live evidence
thymikee Nov 13, 2026
37700e5
test(sdk): load plist module before timed host-scope assertions
thymikee Nov 13, 2026
a19ef99
docs(gestures): limit live scroll evidence to reported success
thymikee Nov 13, 2026
f047762
fix(gestures): enforce canonical timelines and qualify Android evidence
thymikee Nov 13, 2026
1a3e15e
chore: retrigger CI after the live-emulator transition flake
strzhao Nov 13, 2026
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
52 changes: 52 additions & 0 deletions docs/evidence/android-densified-lowering-2026-11-13.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,52 @@
# Android densified lowering through the shared densifyGesturePath, 2026-11-13

Contributor-reported evidence for #4239's Android side records one pan and one
scroll on hardware. The retained pan notes name the Android helper. The scroll
notes report success without identifying a backend. Raw CLI responses were not
preserved, so the excerpts below are not verbatim response artifacts.

## Device and build

- Redmi K60 (Android), adb serial `5p4hqglvpzxshiuw`, Settings app foreground
- Contributor-reported tested commit: `181935c951`, built in the contribution
worktree, daemon restarted clean; the Android automation helper (0.32.0 build, packaged under
this tree's version name) was installed with `adb install -t` before the run

## Retained contributor notes

The original pan notes included `executionProfile: timed-pan`. Android's command
does not emit that field. It is omitted below; it describes the plan selected by
the code, not a field observed in the response.

```json
{
"kind": "pan",
"backend": "android-helper",
"message": "Panned (500, 1400) by (0, -600)"
}
```

```json
{
"message": "Scrolled down",
Comment thread
cubic-dev-ai[bot] marked this conversation as resolved.
Resolved
"success": true
}
```

The pan notes name `backend: android-helper`. For a pan plan, the shared planner
selects `timed-pan` and Android lowers its samples before helper dispatch; this is
an inference from the code. The scroll notes do not identify a backend or sampled
route. The contributor reported visible scrolling in both runs.

## What this evidences

The contributor reports Android helper pan dispatch and a successful scroll.
The retained notes cannot establish a response-reported execution profile or
which backend or lowering executed the scroll. Neither result shows the cadence
of individual timed samples. Cadence equivalence rests on the pure-function and
Android-lowering tests rather than these reported runs.

The retained notes name `gesture pan --json` and `scroll down`, but do not
preserve the complete argument lists. The pan response records the start point
`(500, 1400)` and displacement `(0, -600)`. These are response facts, not a
reconstruction of the missing invocation.
97 changes: 96 additions & 1 deletion packages/contracts/src/gesture-plan.test.ts
Original file line number Diff line number Diff line change
@@ -1,6 +1,10 @@
import assert from 'node:assert/strict';
import { test } from 'vitest';
import { buildDragGesturePlan, singlePointerPlanEndpoints } from './gesture-plan.ts';
import {
buildDragGesturePlan,
densifyGesturePath,
singlePointerPlanEndpoints,
} from './gesture-plan.ts';

const DRAG_PROPERTY_RUNS = 100;

Expand Down Expand Up @@ -77,5 +81,96 @@ test('every drag plan preserves ordered samples and exact hold/move endpoints',
for (let index = 1; index < samples.length; index += 1) {
assert.ok(samples[index]!.offsetMs >= samples[index - 1]!.offsetMs);
}
const dense = densifyGesturePath(samples, 'android');
assert.deepEqual(dense[0]?.point, from);
assert.deepEqual(dense.at(-1)?.point, to);
}
});

test('densifyGesturePath merges the profile grid into the canonical path and keeps its endpoints', () => {
const canonical = [
{ offsetMs: 0, point: { x: 0, y: 0 } },
{ offsetMs: 100, point: { x: 100, y: 0 } },
{ offsetMs: 200, point: { x: 100, y: 100 } },
];
const dense = densifyGesturePath(canonical, 'default');

// The 16 ms grid and the canonical offsets merge into one ascending timeline.
const offsets = dense.map(({ offsetMs }) => offsetMs);
assert.equal(offsets[0], 0);
assert.equal(offsets.at(-1), 200);
assert.ok([...offsets].sort((left, right) => left - right).every((o, i) => o === offsets[i]));
assert.deepEqual(dense[0]?.point, { x: 0, y: 0 });
assert.deepEqual(dense.at(-1)?.point, { x: 100, y: 100 });
// A merged grid offset inside the first segment interpolates on that segment's line.
const at50 = dense.find(({ offsetMs }) => offsetMs === 50);
Comment thread
thymikee marked this conversation as resolved.
Resolved
assert.deepEqual(at50?.point, { x: 50, y: 0 });
const at150 = dense.find(({ offsetMs }) => offsetMs === 150);
assert.deepEqual(at150?.point, { x: 100, y: 50 });
// Canonical offsets survive verbatim.
const at100 = dense.find(({ offsetMs }) => offsetMs === 100);
assert.deepEqual(at100?.point, { x: 100, y: 0 });
});

test('densifyGesturePath hands within-segment pacing to the progress hook and keeps its endpoint rule', () => {
const canonical = [
{ offsetMs: 0, point: { x: 0, y: 0 } },
{ offsetMs: 100, point: { x: 100, y: 0 } },
];
const seen: Array<[number, number]> = [];
const dense = densifyGesturePath(canonical, 'default', (progress, segmentDurationMs) => {
seen.push([progress, segmentDurationMs]);
return progress * progress;
Comment thread
cubic-dev-ai[bot] marked this conversation as resolved.
Resolved
});

const at50 = dense.find(({ offsetMs }) => offsetMs === 50);
assert.deepEqual(at50?.point, { x: 25, y: 0 });
assert.ok(
seen.some(([progress, segmentDurationMs]) => progress === 0.5 && segmentDurationMs === 100),
);
// The hook only shapes interior offsets: every progress it saw lies strictly
// between 0 and 1, so the endpoints never passed through it.
assert.ok(seen.length >= 1);
for (const [progress] of seen) {
assert.ok(progress > 0 && progress < 1, `hook saw progress ${progress}`);
}
assert.deepEqual(dense[0]?.point, { x: 0, y: 0 });
assert.deepEqual(dense.at(-1)?.point, { x: 100, y: 0 });
});

test('densifyGesturePath refuses a path of fewer than two timed samples', () => {
assert.throws(() => densifyGesturePath([{ offsetMs: 0, point: { x: 0, y: 0 } }], 'default'), {
code: 'INVALID_ARGS',
details: { reason: 'GESTURE_PATH_TOO_SHORT' },
});
assert.throws(
() =>
densifyGesturePath(
[
{ offsetMs: 0, point: { x: 0, y: 0 } },
{ offsetMs: 0, point: { x: 1, y: 0 } },
],
'default',
),
{ code: 'INVALID_ARGS', details: { reason: 'GESTURE_PATH_TOO_SHORT' } },
);
});

for (const [name, offsets] of [
['a nonzero first offset', [5, 100]],
['tied first offsets', [0, 0, 100]],
['tied last offsets', [0, 100, 100]],
['reversed offsets', [0, 75, 50, 100]],
['a non-finite interior offset', [0, Number.NaN, 100]],
] as const) {
test(`densifyGesturePath refuses ${name}`, () => {
const canonical = offsets.map((offsetMs, index) => ({
offsetMs,
point: { x: index * 10, y: 0 },
}));
assert.throws(() => densifyGesturePath(canonical, 'default'), {
code: 'INVALID_ARGS',
details: { reason: 'GESTURE_PATH_INVALID_TIMELINE' },
});
});
}
75 changes: 75 additions & 0 deletions packages/contracts/src/gesture-plan.ts
Original file line number Diff line number Diff line change
Expand Up @@ -22,6 +22,7 @@ import type {
GestureExecutionProfile,
GestureSemanticInput,
PointerTrajectory,
PointerTrajectorySample,
SinglePointerGesturePlan,
SinglePointerTrajectory,
TapGesturePlan,
Expand Down Expand Up @@ -467,6 +468,80 @@ export function sampleGestureOffsets(
);
}

/**
* Densifies a plan's own timed path onto one merged sampling grid: the profile grid and the
* canonical offsets merge, every merged offset interpolates inside its canonical segment, and the
* endpoints and duration stay the plan's — the duration IS the last canonical offset, so a
* disagreeing duration cannot be written. This is the one owner of the sampled-path rule; every
* transport lowering calls it instead of keeping its own merge-and-interpolate loop. The optional
* progress hook lets a transport reshape within-segment pacing (Android's controlled scroll) while
* the segment walk and the endpoint rule stay shared.
*/
export function densifyGesturePath(
canonical: readonly PointerTrajectorySample[],
profile: GestureSamplingProfile,
progress: (baseProgress: number, segmentDurationMs: number) => number = (p) => p,
): PointerTrajectorySample[] {
// The walk reads each segment's end, so a path of fewer than two timed samples has no
// interpolable segment; the last offset is the duration the sampling grid builds on.
const durationMs = canonical.at(-1)?.offsetMs ?? 0;
if (canonical.length < 2 || !Number.isFinite(durationMs) || durationMs <= 0) {
Comment thread
thymikee marked this conversation as resolved.
Resolved
throw new AppError('INVALID_ARGS', 'gesture path needs at least two timed samples', {
reason: 'GESTURE_PATH_TOO_SHORT',
});
}
if (
canonical[0]!.offsetMs !== 0 ||
canonical.some(
({ offsetMs }, index) =>
!Number.isFinite(offsetMs) || (index > 0 && offsetMs <= canonical[index - 1]!.offsetMs),
)
) {
throw new AppError(
'INVALID_ARGS',
'gesture path needs a strictly increasing timeline from zero',
{
reason: 'GESTURE_PATH_INVALID_TIMELINE',
},
);
}
const offsets = [
...new Set([
...sampleGestureOffsets(durationMs, profile),
...canonical.map(({ offsetMs }) => offsetMs),
]),
].sort((left, right) => left - right);
let segmentIndex = 0;
return offsets.map((offsetMs) => {
while (
segmentIndex < canonical.length - 2 &&
offsetMs > canonical[segmentIndex + 1]!.offsetMs
) {
segmentIndex += 1;
}
const start = canonical[segmentIndex]!;
const end = canonical[segmentIndex + 1]!;
const segmentDurationMs = end.offsetMs - start.offsetMs;
const baseProgress =
segmentDurationMs === 0 ? 0 : (offsetMs - start.offsetMs) / segmentDurationMs;
return {
offsetMs,
point:
// Both endpoints and every zero-length segment keep the canonical
// point, so the hook only ever reshapes strictly interior offsets.
offsetMs === end.offsetMs || segmentDurationMs === 0
? end.point
: baseProgress === 0
? start.point
: interpolateGesturePoint(
start.point,
end.point,
progress(baseProgress, segmentDurationMs),
),
};
});
}

function assertSamplesInViewport(
samples: readonly { offsetMs: number; point: Point }[],
viewport: Rect,
Expand Down
40 changes: 6 additions & 34 deletions packages/platform-android/src/touch-plan-lowering.ts
Original file line number Diff line number Diff line change
@@ -1,8 +1,7 @@
import type { ScrollReleaseBehavior } from '@agent-device/contracts/scroll-command';
import {
GESTURE_SAMPLE_INTERVAL_MS,
interpolateGesturePoint,
sampleGestureOffsets,
densifyGesturePath,
} from '@agent-device/contracts/gesture-plan';
import type {
GesturePlan,
Expand Down Expand Up @@ -63,38 +62,11 @@ export function lowerAndroidTouchPlan(plan: AndroidTouchPlan): AndroidLoweredTou

const { releaseBehavior, ...gesturePlan } = plan;
const [{ pointerId, samples: canonicalSamples }] = gesturePlan.pointers;
const offsets = [
...new Set([
...sampleGestureOffsets(plan.durationMs, 'android'),
...canonicalSamples.map(({ offsetMs }) => offsetMs),
]),
].sort((left, right) => left - right);
let segmentIndex = 0;
const dense = offsets.map((offsetMs) => {
while (
segmentIndex < canonicalSamples.length - 2 &&
offsetMs > canonicalSamples[segmentIndex + 1]!.offsetMs
) {
segmentIndex += 1;
}
const start = canonicalSamples[segmentIndex]!;
const end = canonicalSamples[segmentIndex + 1]!;
const segmentDurationMs = end.offsetMs - start.offsetMs;
const progress = (offsetMs - start.offsetMs) / segmentDurationMs;
return {
offsetMs,
point:
offsetMs === end.offsetMs || segmentDurationMs === 0
? end.point
: interpolateGesturePoint(
start.point,
end.point,
releaseBehavior === 'controlled'
? controlledScrollProgress(progress, segmentDurationMs)
: progress,
),
};
});
const dense = densifyGesturePath(canonicalSamples, 'android', (progress, segmentDurationMs) =>
releaseBehavior === 'controlled'
? controlledScrollProgress(progress, segmentDurationMs)
: progress,
);
// The sampler floors the frame count at three, so `dense` always holds at least four samples.
const samples: AndroidTransportSamples = [dense[0]!, dense[1]!, dense[2]!, ...dense.slice(3)];

Expand Down
2 changes: 1 addition & 1 deletion src/commands/schema/cli-help-gestures.ts
Original file line number Diff line number Diff line change
Expand Up @@ -21,7 +21,7 @@ Shapes:
agent-device gesture rotate 35 200 420
agent-device gesture transform 200 420 80 -40 2 35 700
agent-device gesture tap 80 300 200 450 310 600 --count 20 --pause-ms 100
scroll <dir> --until <selector> repeats scroll-and-check passes until that element is on screen, then stops: one request instead of a scroll-then-snapshot loop, and it stops on the target rather than overshooting it. It reports the passes it spent, fails when the content runs out first, and is refused on top/bottom, which already stop themselves. The default limit is ${DEFAULT_SCROLL_UNTIL_PASSES} gestures. For a longer list, add --max-passes <n> (1–${SCROLL_UNTIL_MAX_PASSES}); it requires --until. Large pass budgets can hit the 90-second request timeout; inspect the current screen and continue with another search if that happens.
scroll <dir> --until <selector> repeats scroll-and-check passes until that element is on screen, then stops: one request instead of a scroll-then-snapshot loop, and it stops on the target rather than overshooting it. It reports the passes it spent; vertical searches compare repeated captures of descendants in the selected scroll container and fail when those captures show no progress. If no container is found, they compare the visible tree. It is refused on top/bottom, which already stop themselves. The default limit is ${DEFAULT_SCROLL_UNTIL_PASSES} gestures. For a longer list, add --max-passes <n> (1–${SCROLL_UNTIL_MAX_PASSES}); it requires --until. Large pass budgets can hit the 90-second request timeout; inspect the current screen and continue with another search if that happens.
Comment thread
thymikee marked this conversation as resolved.
Resolved
Comment thread
thymikee marked this conversation as resolved.
Resolved
longpress accepts coordinates, @refs, or selectors; prefer @ref/selector, coordinates only as a fallback. Duration and gesture scale/center are positional. gesture pan is one finger by default; add --pointer-count 2 for a parallel two-finger pan. Keep count/pause/pattern on one swipe: --count (cap 200), --pause-ms (cap 10000ms), --pattern ping-pong; the combined swipe/pause schedule is capped at 60000ms.
For repeated iOS smoke checks: press <x> <y> --count <n> --jitter-px <n> for tap series, swipe <x1> <y1> <x2> <y2> --count <n> for drag series. gesture tap touches 2 to 5 distinct points and holds them for --hold-ms (default 50, min 16); --count repeats the same contacts (cap 200) with --pause-ms between, the series must fit 60000ms counting about 500ms per repetition, and a repetition costs about 450ms on a physical iPhone. Use press for one finger.

Expand Down
6 changes: 5 additions & 1 deletion src/sdk/limrun-host-scope.test.ts
Original file line number Diff line number Diff line change
@@ -1,11 +1,15 @@
import assert from 'node:assert/strict';
import { chmodSync, mkdirSync, writeFileSync } from 'node:fs';
import path from 'node:path';
import { test } from 'vitest';
import { beforeAll, test } from 'vitest';
import { mkdtempForTestSync } from '@agent-device/test-kit';
import { withCommandHost } from '@agent-device/host-kit/command';
import { createLimrunRuntimeDependencies } from './limrun-runtime-dependencies.ts';

beforeAll(async () => {
if (process.platform !== 'win32') await import('@agent-device/platform-apple/install-artifact');
});

test.skipIf(process.platform === 'win32')(
'Limrun ADB callbacks retain their owning host outside and across concurrent foreign scopes',
async () => {
Expand Down
Loading