A drive step can aim: moveMouse, look and aimAt move the real crosshair

.claude/skills/jgengine-verify/SKILL.md

@@ -41,6 +41,7 @@ Classify each acceptance claim before scheduling proof. A plan that uses screens
 - `GameHost` publishes declared `capture.probe` as `window.__jgProbe` after game initialization and staging. Reads use the live context; replacement/unmount retires the hook. An absent or throwing probe supplies no progress evidence.
 - Placeholder-vs-authored: `debug_snapshot().probes.fallbacks` reports which render seams resolved to fallbacks (green ground, primitive actors, proxy scatter) and why — a non-empty count proves content is unauthored/misconfigured rather than an intended placeholder. `probes.textureErrors` (`{ url, count }[]`) is its twin for the failure `fallbacks` cannot see: a model that resolves but whose GLB textures 404 — a non-empty list is a visibly-broken (untextured) scene, treat it exactly like a model fallback.
 - Portable `drive --click-at x,y` sends trusted native left-click input to a canvas or control in CSS viewport pixels. Steps preserve argument order; coordinates must be finite, nonnegative and strictly inside the selected device or overridden viewport (mobile DPR does not multiply them). Use the game’s visible world target; coordinate clicking does not project an entity id, run a game command directly, or advance fixed simulation time. Older project-owned `scripts/drive.mjs` keep their installed behavior; update the scaffold harness to use this option.
+- Aiming in a step file: `{ "moveMouse": [x, y] }` hovers the pointer at viewport pixels, `{ "look": [dx, dy] }` turns a pointer-locked view by that mouse movement (the step takes pointer lock on the play canvas inside a CDP user gesture, so it never fires the weapon), and `{ "aimAt": "<page expression>" }` turns until the target's `{x,y}` projection sits on the crosshair within `tolerancePx` (default 6), measuring the page's own response instead of assuming a sensitivity. A shooter proof asserts the arrival every time — a shot before the aim that leaves `capture.probe` health flat, then a shot after it that drops it — because "fired six taps" proves only that the trigger works.
 - A route a proof needs once belongs in `<game>/proofs/flows/<name>.json`, not copied into every proof: `bun run drive <game> --flow <name>` runs it, `--flows` lists each name with its step count and asserts, and `{ "flow": "<name>" }` includes it (4 levels deep, a repeated name is a cycle). A flow asserts arrival — the match started, N units present — because a flow that only clicks proves nothing. Every step prints one line and `--log <path>` writes JSONL, so a failed run names the step, the expected and actual value, and stops before the next screenshot.
 - UI-flow logic (a command/intent handler, not its pixels): drive it canvas-free with `createHeadlessRunner(...).ui.invoke("intent.name", input)`, then assert the resulting reactive state off `ctx` (store, stats, scene). No renderer, no pointer simulation — leave pixels to `shoot`; prove the logic headlessly.
 - Play itself, as a script: `bun run simulate <game> <script>` boots the game's real `PlayableGame` in-process with no browser and plays a committed `<game>/scripts/*.play.ts` through `createPlayScript`/`runPlayScript` (`@jgengine/shell/play`). Steps are tick-based (`move`, `aim`, `fire`, `use`, `wait`, `command`) plus optional `setup`/`onTick` hooks; `expect` assertions read world state twice — right after setup and on their own tick — so "the enemy lost health" and "the wave is 2" are both one line. The runner replays the shell's own frame pipeline (held movement through the walk controller, a bound action resolving to its command on press and repeat, `interact` prompts, hotbar), prints a tick timeline, names any failed assertion and exits 1; the timeline digest is the replay comparison (a fresh process with the same script and seed repeats it). Prefer this over playing for minutes to reach a wave, a boss phase, a lap or a shootout; record a clip only for what a script cannot judge (look, feel, camera).

packages/jgengine/src/templates/captureArgs.test.ts

@@ -100,6 +100,32 @@ describe("parseStepFile", () => {
       .toBe("steps[4].rightClickAt: expected [x, y] numbers, got \"420,220\"");
   });
 
+  test("parses aim steps and lists them in --check with their conditions", () => {
+    const steps = parseStepFile({
+      steps: [
+        { moveMouse: [640, 360], before: { is: "menu.ready" } },
+        { look: [-120, 40] },
+        { aimAt: "crosshair.target()", tolerancePx: 4, after: { is: "aim.onTarget", timeoutMs: 3000 } },
+      ],
+    }, "aim.json");
+    expect(steps[0]).toMatchObject({ kind: "move-mouse", x: 640, y: 360, before: { is: "menu.ready" } });
+    expect(steps[1]).toEqual({ kind: "look", dx: -120, dy: 40 });
+    expect(steps[2]).toMatchObject({ kind: "aim-at", expression: "crosshair.target()", tolerancePx: 4, after: { is: "aim.onTarget", timeoutMs: 3000 } });
+    expect(parseStepFile({ steps: [{ aimAt: "[10, 20]" }] }, "aim.json")[0]).toMatchObject({ tolerancePx: 6 });
+    const listed = describeStepFile(steps, "aim.json");
+    expect(listed).toContain("#1 moveMouse 640,360 before<=15000ms");
+    expect(listed).toContain("#2 look -120,40 px");
+    expect(listed).toContain("#3 aimAt \"crosshair.target()\" within 4px after<=3000ms");
+  });
+
+  test("names the index and the field for a malformed aim coordinate or target", () => {
+    expect(error([{ click: "Start" }, { look: [10, "20"] }])).toBe("steps[1].look: expected [x, y] numbers, got [10,\"20\"]");
+    expect(error([{ moveMouse: [1, 2, 3] }])).toBe("steps[0].moveMouse: expected [x, y] numbers, got [1,2,3]");
+    expect(error([{ aimAt: "" }])).toContain("steps[0].aimAt: expected the page expression");
+    expect(error([{ aimAt: "target()", tolerancePx: 0 }])).toContain("steps[0].tolerancePx: expected a positive number");
+    expect(error([{ look: [1, 2], tolerancePx: 3 }])).toContain("steps[0]: \"tolerancePx\" sizes an aimAt target");
+  });
+
   test("names the index for an unknown key code", () => {
     const message = error([{ click: "Start" }, { wait: 1 }, { key: "KeyHH", ms: 120 }]);
     expect(message).toContain("steps[2].key: unknown key code \"KeyHH\"");

packages/jgengine/src/templates/captureArgs.ts

@@ -108,6 +108,9 @@ export type CaptureStep = (
   | { kind: "click"; text: string }
   | { kind: "double-click"; text: string }
   | { kind: "click-at"; spec: string; button: "left" | "right" }
+  | { kind: "move-mouse"; x: number; y: number }
+  | { kind: "look"; dx: number; dy: number }
+  | { kind: "aim-at"; expression: string; tolerancePx: number }
   | { kind: "fill"; label: string; value: string }
   | { kind: "key"; code: string; holdMs: number }
   | { kind: "wait"; ms: number }
@@ -128,10 +131,12 @@ export type CaptureStep = (
 export const DRIVE_CONDITION_TIMEOUT_MS = 15_000;
 /** How often a condition is re-evaluated while it is failing. */
 export const DRIVE_CONDITION_POLL_MS = 50;
+/** How close `aimAt` must bring the target to the crosshair before the step counts as aimed. */
+export const AIM_TOLERANCE_PX = 6;
 
 /** The step kinds a JSON step file accepts, in the order the help lists them. */
 export function driveStepKinds(): string[] {
-  return ["click", "doubleClick", "clickAt", "rightClickAt", "fill", "key", "wait", "waitFor", "reload", "shot", "rpc", "probe", "assert", "flow"];
+  return ["click", "doubleClick", "clickAt", "rightClickAt", "moveMouse", "look", "aimAt", "fill", "key", "wait", "waitFor", "reload", "shot", "rpc", "probe", "assert", "flow"];
 }
 
 /** A named flow file's parsed JSON and the path its errors name. */
@@ -358,7 +363,7 @@ export function parseStepFile(input: unknown, source: string, options: ParseStep
   }
   if (!Array.isArray(top.steps)) throw new Error(source + ": expected {\"steps\":[...]}, got " + valueType(top.steps));
   if (top.steps.length === 0) throw new Error(source + ": the steps list is empty, so the run would prove nothing");
-  const modifiers = ["before", "after", "ms", "timeoutMs", "note", "is"];
+  const modifiers = ["before", "after", "ms", "timeoutMs", "tolerancePx", "note", "is"];
   return top.steps.flatMap((raw, index) => {
     const at = "steps[" + index + "]";
     if (raw === null || typeof raw !== "object" || Array.isArray(raw)) throw new Error(at + ": expected a step object, got " + valueType(raw));
@@ -371,6 +376,7 @@ export function parseStepFile(input: unknown, source: string, options: ParseStep
     const kind = actions[0]!;
     const value = step[kind];
     if (step.is !== undefined && kind !== "assert") throw new Error(at + ": \"is\" is the assert expectation, so it cannot sit on a " + kind + " step");
+    if (step.tolerancePx !== undefined && kind !== "aimAt") throw new Error(at + ": \"tolerancePx\" sizes an aimAt target, so it cannot sit on a " + kind + " step");
     if (kind === "flow") return expandFlow(flowNameAt(at + ".flow", value), at, source, options, step);
     const before = step.before === undefined ? undefined : conditionAt(at + ".before", step.before);
     const after = step.after === undefined ? undefined : conditionAt(at + ".after", step.after);
@@ -386,6 +392,24 @@ export function parseStepFile(input: unknown, source: string, options: ParseStep
       case "clickAt":
       case "rightClickAt":
         return [shape({ kind: "click-at", spec: coordinatePair(at + "." + kind, value), button: kind === "rightClickAt" ? "right" : "left" })];
+      case "moveMouse": {
+        const [x, y] = coordinatePair(at + ".moveMouse", value).split(",").map(Number) as [number, number];
+        return [shape({ kind: "move-mouse", x, y })];
+      }
+      case "look": {
+        const [dx, dy] = coordinatePair(at + ".look", value).split(",").map(Number) as [number, number];
+        return [shape({ kind: "look", dx, dy })];
+      }
+      case "aimAt": {
+        if (typeof value !== "string" || value.trim() === "") {
+          throw new Error(at + ".aimAt: expected the page expression that returns the target's screen point { x, y }, got " + JSON.stringify(value));
+        }
+        const tolerance = step.tolerancePx;
+        if (tolerance !== undefined && (typeof tolerance !== "number" || !Number.isFinite(tolerance) || tolerance <= 0)) {
+          throw new Error(at + ".tolerancePx: expected a positive number of CSS pixels, got " + JSON.stringify(tolerance));
+        }
+        return [shape({ kind: "aim-at", expression: value, tolerancePx: tolerance ?? AIM_TOLERANCE_PX })];
+      }
       case "fill": {
         if (value === null || typeof value !== "object" || Array.isArray(value)) throw new Error(at + ".fill: expected {\"label\": \"<label>\", \"value\": \"<text>\"}, got " + JSON.stringify(value));
         const field = value as Record<string, unknown>;
@@ -442,7 +466,7 @@ function flowNameAt(path: string, value: unknown): string {
 
 /** Resolve one `{ "flow": name }` include into its steps, refusing a cycle or a fifth level. */
 function expandFlow(name: string, at: string, includeSource: string, options: ParseStepOptions, step: Record<string, unknown>): CaptureStep[] {
-  const extra = ["before", "after", "ms", "timeoutMs", "note", "is"].filter((key) => step[key] !== undefined);
+  const extra = ["before", "after", "ms", "timeoutMs", "tolerancePx", "note", "is"].filter((key) => step[key] !== undefined);
   if (extra.length > 0) throw new Error(at + ".flow: an include carries no fields of its own (got " + extra.join(", ") + "); put them on the flow's steps");
   const stack = options.stack ?? [];
   if (stack.includes(name)) throw new Error(at + ".flow: flow cycle by name: " + [...stack, name].join(" -> "));
@@ -474,6 +498,9 @@ export function describeStep(step: CaptureStep): string {
     case "click": return "click \"" + step.text + "\"";
     case "double-click": return "doubleClick \"" + step.text + "\"";
     case "click-at": return (step.button === "right" ? "rightClickAt " : "clickAt ") + step.spec;
+    case "move-mouse": return "moveMouse " + step.x + "," + step.y;
+    case "look": return "look " + step.dx + "," + step.dy + " px";
+    case "aim-at": return "aimAt " + JSON.stringify(step.expression) + " within " + step.tolerancePx + "px";
     case "fill": return "fill \"" + step.label + "\" = \"" + step.value + "\"";
     case "key": return "key " + step.code + " " + step.holdMs + "ms";
     case "wait": return "wait " + step.ms + "ms (last resort: a guessed sleep)";

packages/jgengine/src/templates/driveAim.test.ts

@@ -0,0 +1,133 @@
+import { describe, expect, test } from "bun:test";
+
+import { acquireDrivePointerLock, aimDriveAt, lookDrivePointer, moveDrivePointer, type DriveAimSession } from "./driveAim";
+
+const WIDTH = 1600;
+const HEIGHT = 900;
+const CENTRE = { x: WIDTH / 2, y: HEIGHT / 2 };
+
+/**
+ * A page whose target is the projection of a yaw offset: a pointer-locked view turns by
+ * `movementX / pxPerRad` pixels of projection, so the harness has to learn that cost.
+ */
+function fakePage(options: { pxPerRad?: number; locked?: boolean; deliverMovement?: "cdp" | "none" } = {}) {
+  const pxPerRad = options.pxPerRad ?? 900;
+  const state = { yaw: 0.75, sent: [] as { x: number; y: number }[], synthetic: 0, events: [] as string[], last: { x: 0, y: 0 }, acc: { x: 0, y: 0 } };
+  const session: DriveAimSession = {
+    async send(method, params) {
+      state.events.push(method);
+      if (method === "Input.dispatchMouseEvent") {
+        state.last = { x: Number(params?.x), y: Number(params?.y) };
+        state.sent.push(state.last);
+        if (options.locked === true && options.deliverMovement !== "none") state.yaw -= (state.last.x - CENTRE.x) / pxPerRad;
+      }
+      if (method === "Runtime.evaluate" && JSON.stringify(params).includes("requestPointerLock")) {
+        return { result: { value: options.locked === true ? "locked" : "refused" } };
+      }
+      return {};
+    },
+    async evaluate(expression) {
+      if (expression.includes("pointerLockElement")) return options.locked === true;
+      if (expression.includes("getBoundingClientRect")) return { x: CENTRE.x, y: CENTRE.y };
+      if (expression.includes("__jgAimMove")) {
+        if (expression.includes("state.acc = { x: 0, y: 0 }")) {
+          state.acc = { x: 0, y: 0 };
+          return true;
+        }
+        const delivered = options.deliverMovement === "none" ? state.acc : { x: state.last.x - CENTRE.x + state.acc.x, y: state.last.y - CENTRE.y + state.acc.y };
+        const seen = options.deliverMovement === "none" ? state.synthetic : 1 + state.synthetic;
+        return { x: delivered.x, y: delivered.y, seen };
+      }
+      if (expression.includes("Object.defineProperty")) {
+        state.synthetic += 1;
+        const dx = Number(/movementX', \{ value: (-?[\d.]+)/.exec(expression)?.[1] ?? 0);
+        state.acc.x += dx;
+        state.yaw -= dx / pxPerRad;
+        return true;
+      }
+      if (expression.includes("crosshair.target")) {
+        return { x: CENTRE.x + state.yaw * pxPerRad, y: CENTRE.y };
+      }
+      return null;
+    },
+  };
+  return { session, state };
+}
+
+describe("aim steps", () => {
+  test("moveMouse dispatches an absolute pointer move and no button", async () => {
+    const { session, state } = fakePage();
+    await moveDrivePointer(session, { x: 240, y: 180 });
+    expect(state.sent).toEqual([{ x: 240, y: 180 }]);
+    expect(state.events).toEqual(["Input.dispatchMouseEvent"]);
+  });
+
+  test("look turns a locked view and reports the movement the page saw", async () => {
+    const { session, state } = fakePage({ locked: true });
+    const before = state.yaw;
+    const turned = await lookDrivePointer(session, 300, 0);
+    expect(turned.movement).toEqual({ x: 300, y: 0 });
+    expect(state.yaw).toBeLessThan(before);
+  });
+
+  test("look chunks a long turn from the pinned centre instead of leaving the viewport", async () => {
+    const { session, state } = fakePage({ locked: true });
+    await lookDrivePointer(session, 500, 0);
+    expect(state.sent.map((point) => point.x)).toEqual([CENTRE.x + 200, CENTRE.x + 200, CENTRE.x + 100]);
+  });
+
+  test("look falls back to a page move carrying the deltas when CDP delivers none", async () => {
+    const { session, state } = fakePage({ locked: true, deliverMovement: "none" });
+    const before = state.yaw;
+    await lookDrivePointer(session, 120, 0);
+    expect(state.synthetic).toBe(1);
+    expect(state.yaw).toBeLessThan(before);
+  });
+
+  test("look refuses a page that takes the move and never reports movement", async () => {
+    const { session } = fakePage({ locked: true, deliverMovement: "none" });
+    const silent: DriveAimSession = {
+      async send(method, params) {
+        return session.send(method, params);
+      },
+      async evaluate(expression) {
+        if (expression.includes("__jgAimMove")) {
+          return expression.includes("state.acc = { x: 0, y: 0 }") ? true : { x: 0, y: 0, seen: 0 };
+        }
+        if (expression.includes("Object.defineProperty")) return false;
+        if (expression.includes("getBoundingClientRect")) return { x: CENTRE.x, y: CENTRE.y };
+        return null;
+      },
+    };
+    await expect(lookDrivePointer(silent, 40, 0)).rejects.toThrow("reported no movement");
+  });
+
+  test("acquireDrivePointerLock asks the canvas for the lock, never a mouse button", async () => {
+    const { session, state } = fakePage({ locked: true });
+    expect(await acquireDrivePointerLock(session)).toBe(true);
+    expect(state.events).toEqual([]);
+    const late = fakePage({ locked: true });
+    expect(await acquireDrivePointerLock(late.session)).toBe(true);
+    expect(late.state.events.some((method) => method.includes("MouseEvent"))).toBe(false);
+  });
+
+  test("aimAt learns a fast page's pixel cost and lands inside the tolerance", async () => {
+    const { session } = fakePage({ locked: true, pxPerRad: 2200 });
+    const aimed = await aimDriveAt(session, null, "crosshair.target()", 6);
+    expect(aimed.moves).toBeGreaterThan(0);
+    expect(Math.abs(aimed.point.x - CENTRE.x)).toBeLessThanOrEqual(6);
+  });
+
+  test("aimAt lands on a slow page too, without being told the sensitivity", async () => {
+    const { session } = fakePage({ locked: true, pxPerRad: 400 });
+    const aimed = await aimDriveAt(session, null, "crosshair.target()", 6);
+    expect(Math.abs(aimed.point.x - CENTRE.x)).toBeLessThanOrEqual(6);
+  });
+
+  test("aimAt moves the plain pointer when the page is not pointer-locked", async () => {
+    const { session, state } = fakePage({ locked: false });
+    const aimed = await aimDriveAt(session, null, "crosshair.target()", 6);
+    expect(aimed.moves).toBe(1);
+    expect(state.sent.at(-1)?.x).toBeCloseTo(CENTRE.x + 0.75 * 900, 0);
+  });
+});