Skip to content

Mobile: delayed keyboard movement can send the next tap to stale bounds #1296

Closed

Description

@shamilovtim

What happened

After text entry, a deterministic tap can use bounds captured before the keyboard moves the form. In an iOS simulator recording, a password-visibility control was resolved at y=298; the keyboard then moved it to about y=211, but the tap landed at y=298. The subsequent assertion failed.

Expected: after text entry, honor the remaining configured transition budget and resolve the next control again before dispatch.

The mobile engine records the pre-entry index after type/fill/clear. settled() skips its transition wait when the first post-entry snapshot has the same bounds. That snapshot can precede keyboard movement, so an unchanged first frame is not sufficient.

Separately, agent-device can refuse a moving target with reason=target_still_moving and dispatched=no. The engine currently classifies that as nonretryable ENGINE_FAILURE, even though re-observing is safe when no input was dispatched. Unknown dispatch status must remain nonretryable.

Minimal reproduction

// Deterministic regression on current main, with no device or model needed:
// https://github.com/shamilovtim/e2e/blob/codex/mobile-keyboard-transitions/packages/mobile/tests/unit/engine.test.ts
// Test: "reobserves after %s when the keyboard moves a control after an unchanged snapshot"
//
// 1. Observe a textbox and an existing button at position A.
// 2. Type, fill, or clear the textbox.
// 3. Return a first post-entry snapshot with the button still at A.
// 4. Return the next snapshot with the same button at B and a new ref.
// 5. Tap the button: main dispatches the old ref; the patch dispatches the new ref.
//
// Run from repo root:
// pnpm --filter e2e run build
// pnpm --filter @e2e-dev/mobile exec vitest run tests/unit/engine.test.ts tests/unit/errors.test.ts -t "reobserves after|retries a moving target"
//
// The real-device symptom is intermittent; a standalone public app reproduction
// has not been prepared. The regression uses the existing scripted client.

Versions

e2e 0.21.2; @e2e-dev/mobile 0.12.3; agent-device 0.21.28; Node 24.22.0; iOS 27.0 simulator, macOS 27.0.1. Regression also reproduced against upstream main 93f77d8.

Engine

@e2e-dev/mobile

Anything else

Candidate fix: main...shamilovtim:e2e:codex/mobile-keyboard-transitions

The three delayed-layout cases and the no-dispatch motion refusal fail on unmodified main and pass with the patch. The full mobile package suite passes 349 tests. pnpm check passes.

Equivalent package patch was exercised in focused downstream iOS login and billing-eligibility journeys: each passed three times with zero test retries. One additional cold-start stress run encountered an unrelated native crash before credential entry, so this is not a claim that every run was green or that Android was verified.

Tradeoff: after type/fill/clear, the next deterministic action may wait the remainder of transition even when no keyboard animation actually occurs. No new timeout, retry loop, coordinate workaround, or Return-key action is introduced.

Activity

  1. tadeumaia commented on Oct 22, 2026

    @tadeumaia

    We hit the same no-dispatch refusal on iOS in two situations, so here is another real-world case for the second part of this issue.

    After moving @e2e-dev/mobile from 0.12.2 to 0.12.3 (agent-device 0.21.25 to 0.21.28), four of our iOS tests failed with:

    ENGINE_FAILURE: perform tap failed: Ref @e47 had no confirmed resting position when the transition window ran out
    

    In agent-device 0.21.28 that is reason: target_still_moving with dispatched: no, from the branch where the first re-measure comes back unmeasured: for a ref target the measure needs adoptedFrom === 'observation', and the mobile engine always presses by ref. Each refusal took about 280 ms.

    • After text entry, as in this issue: a scripted screen.getByTestId("set-1-reps-field").first().tap() right after a digit typed on the iOS number pad (device.locator('role=Key label="0"').tap()). Three tests.
    • With no keyboard: a replayed agent action, tap button "Completar" (1 of 2), right after tap button "Fechar" closed an inline sheet. One test. The same replay passed three times in another test of the same run.

    Both targets' names appear more than once on screen (once per exercise card), which may be a coincidence.

    On 0.12.2, with the same app build, simulator and recordings, all four pass (6 of 6 and 3 of 3 in the two affected files), so we have pinned 0.12.2 for now. We have not tried #1297 against these.

    Versions: e2e 0.21.2, @e2e-dev/mobile 0.12.3 (agent-device 0.21.28), iPhone 17 simulator on iOS 26.5, Expo dev client (React Native, new architecture), macOS.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    bugSomething isn't working

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions