Skip to content

fix(mobile): revalidate input bindings before fill and clear - #1410

Merged
okwasniewski merged 7 commits into
e2e-inc:mainfrom
bheemreddy-samsara:fix/mobile-input-freshness
Oct 28, 2026
Merged

okwasniewski merged 7 commits into
e2e-inc:mainfrom
bheemreddy-samsara:fix/mobile-input-freshness

Conversation

@bheemreddy-samsara

@bheemreddy-samsara bheemreddy-samsara commented Oct 27, 2026 •

Copy link
Copy Markdown
Contributor

What and why

  • fill and clear always dispatch through one fresh mobile snapshot and refind(), even when capture latency consumed the transition budget. Existing transition waits and nearest-node matching remain in use.
  • Follow moved fields and identical inputs using the fresh binding. Missing fields, changed security facts, or incomplete captures return NODE_STALE before mutation so existing retries re-observe and rerun secret authorization.
  • Reuse TEXT_INPUT_ROLES; keep the non-input refusal and shorten the docs, skill, and changeset. No dependency or agent-device changes. Replacement/empty-clear behavior is separate.

Issue

Fixes #1409

Workflow

flowchart LR
  A[Resolved target] --> R{Input role?}
  R -- no --> U[UNSUPPORTED_CAPABILITY]
  R -- yes --> B[Existing transition wait when needed]
  B --> C[One fresh snapshot and refind]
  C --> D{Target present and security facts unchanged?}
  D -- yes --> E[Fill or clear with fresh binding]
  D -- no --> F[NODE_STALE before mutation]
  F --> G[Existing re-observation and authorization retry]
  G --> C
Loading

Verified

Ran it locally: yes, scripted mobile command stream and built packages, not a device.

Validation head: 7d710946a1d7175e5742b4167a97dec59aa97f6c.

main 1a7a331b this branch
Four slow-capture freshness cases fail All four dispatch the fresh ref with one capture
Six duplicate/sliding compatibility cases pass All six still pass; no uniqueness or rect-equality refusal
  • Owning-boundary tests reproduced 13 failures on the previous PR implementation. The revised engine test file and full mobile unit suite pass, 383 tests total, including cancelled fill/clear transition waits.
  • Built mobile smoke:
    Built mobile smoke: moved input -> fresh @e42, 1 capture, 1 fill.
    
  • The four core fake-engine retry cases are supplemental guards, not regression proof for this fix. They pass, as do 3,060 core unit tests, with one skipped.
  • Local workspace typecheck, lint, docs, mobile dead-code analysis, and package-policy checks pass.
  • Exact-head spec, docs, and benchmark gate passed; benchmark suites were skipped. iOS and Android deterministic suites remain pending.
  • Cubic completed its exact-head incremental review with zero issues across the five cleanup files; all review threads remain resolved.
  • The earlier duplicate-identity finding was resolved after the maintainer's policy clarification. The latest cleanup keeps the original settled() signature and separates waiting from the single input relocation.

Active blockers and limits

@mintlify

mintlify Bot commented Oct 27, 2026 •

Copy link
Copy Markdown

Preview approval required

This pull request comes from a fork, so Mintlify won't build a preview until someone with write access approves commit 11cffff.

Approve preview

@bheemreddy-samsara

Copy link
Copy Markdown
Contributor Author

@cubic-dev-ai review this PR

@cubic-dev-ai

cubic-dev-ai Bot commented Oct 27, 2026

Copy link
Copy Markdown
Contributor

@cubic-dev-ai review this PR

@bheemreddy-samsara I have started the AI code review. It will take a few minutes to complete.

@cubic-dev-ai cubic-dev-ai Bot left a comment •

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

All reported issues were addressed across 7 files

Heads up: you’ve reached your flex budget. Increase your flex budget or wait for usage to reset.

View guided diff | Re-trigger cubic

Comment thread packages/mobile/src/surface.ts Resolved
Comment thread packages/mobile/src/surface.ts Resolved
@bheemreddy-samsara

Copy link
Copy Markdown
Contributor Author

@cubic-dev-ai review this PR

@cubic-dev-ai

cubic-dev-ai Bot commented Oct 27, 2026

Copy link
Copy Markdown
Contributor

@cubic-dev-ai review this PR

@bheemreddy-samsara I have started the AI code review. It will take a few minutes to complete.

@cubic-dev-ai cubic-dev-ai Bot left a comment •

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

All reported issues were addressed across 7 files

Heads up: you’ve reached your flex budget. Increase your flex budget or wait for usage to reset.

View guided diff | Re-trigger cubic

Comment thread packages/mobile/src/surface.ts Resolved
Comment thread docs/reference/mobile.mdx Resolved
@bheemreddy-samsara

Copy link
Copy Markdown
Contributor Author

@cubic-dev-ai review this PR

@cubic-dev-ai

cubic-dev-ai Bot commented Oct 27, 2026

Copy link
Copy Markdown
Contributor

@cubic-dev-ai review this PR

@bheemreddy-samsara I have started the AI code review. It will take a few minutes to complete.

@cubic-dev-ai cubic-dev-ai Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

No issues found across 7 files

Heads up: you’ve reached your flex budget. Increase your flex budget or wait for usage to reset.

View guided diff | Re-trigger cubic

@bheemreddy-samsara
bheemreddy-samsara marked this pull request as ready for review October 27, 2026 15:18

@cubic-dev-ai cubic-dev-ai Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

No issues found across 7 files

Heads up: you’ve reached your flex budget. Increase your flex budget or wait for usage to reset.

View guided diff | Re-trigger cubic

@bheemreddy-samsara

Copy link
Copy Markdown
Contributor Author

@okwasniewski Can you give a quick look ?

@okwasniewski okwasniewski left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for the issue and the clear scripted repro. I confirmed the gap on main 1a7a331. If the observation's capture outlasts the transition budget, settled() returns early (surface.ts:1199-1200) and fill goes out with no fresh snapshot. Your freshness tests catch it: with surface.ts reverted to main, all 26 in input freshness fail. pnpm check and the 377 mobile unit tests pass on 8d56213.

The bug is real, but the fix is broader than it needs to be, and it breaks cases that work today. I checked each one below with a scratch test against the scripted client, running this head and then main's surface.ts.

Must fix

  1. Identical fields can no longer be filled. uniqueInput (surface.ts:1151) plus !entry.uniqueInput (:1233) means any input that shares role/testId/name/text/placeholder with another one always gets NODE_STALE. Re-observing does not help, so the retry loops until timeout. Repro: two text fields labelled Digit with no identifier (an OTP row). This head gives NODE_STALE on 3 of 3 attempts. Main fills @e20 on the first try. e2e has to work on any user app, and "use distinct test ids" is not an option for apps users don't control. refind() already handles duplicates by picking the nearest one.

  2. The rect equality check breaks the transition path for inputs. freshInput compares the fresh rect with the original entry (:1234), not with the node settled() just relocated. A field that arrived with the last action and was first seen mid-slide (the Android case settled() exists for) now always fails once. Repro, adapted from the existing "acts on a control that came with the last action..." test with a text field: main fills the landed @e42 with 1 capture, this head throws NODE_STALE after 2 captures. A moved field with the same identity and the same security facts is still the same sink, so geometry change should not be an error. Following the fresh binding is the fix, and that is how the reported bug actually gets fixed. Rect comparison is a heuristic we would rather not add.

  3. Every input action in the budget window now takes two captures. settled() -> relocated() takes one, then freshInput takes another and throws away the first result (:1289-1293). I counted 2 captures per fill on this head and 1 on main.

Suggested shape, which deletes most of the diff. For fill/clear, skip the elapsed-budget shortcut and always relocate: one fresh snapshot, refind(), act on the fresh ref. After #1300 lands, relocated() already throws a retryable NODE_STALE when the target is gone. On top of that, keep only the check your secret-authorization argument needs: NODE_STALE when secure / inputPurpose / disabled / hidden changed, or when the fresh tree is truncated. That removes uniqueInput from NodeBinding, sameInputIdentity (it duplicates refind's predicate), freshInput, and the rect check.

  1. isInputRole (surface.ts:240) duplicates TEXT_INPUT_ROLES in nodes.ts:97. Export and reuse that one.

Nits

  1. packages/e2e/tests/integration/agent-input-stale.test.ts exercises the existing core NODE_STALE retry through a fake engine that throws on its own. This PR doesn't change packages/e2e/src, so the test passes on main and can't fail for this change. It's fine to keep as a guard, but it isn't evidence for the fix.
  2. The docs/reference/mobile.mdx:240-252 paragraph and its skill copy list internal comparison fields. Once 1-3 are done, one sentence is enough: "fill and clear re-read the field before typing; if it disappeared or its secure state changed, the action retries after a fresh observation." The changeset can be shortened the same way.
  3. The non-input UNSUPPORTED_CAPABILITY guard (:1277) is fine and matches Playwright's fill. Consider sharing it with the focus role check at :1300 instead of adding a second one.

Overlap: #1300 (strict relocated(), Ready) should merge first, and this PR should then build on it. #1297 changes the settled() skip rule as well. Rebase after both.

This was an automated review.

@bheemreddy-samsara

Copy link
Copy Markdown
Contributor Author

I'll wait for #1300 and #1297 to land, then rebase this PR as requested.

The requested fixes are still pending: use one fresh relocation for fill/clear, remove the uniqueness and rect-equality guards so identical and sliding inputs keep working, and retain the security-fact/truncated-tree checks. I'll also reuse TEXT_INPUT_ROLES, shorten the docs and changeset, and add duplicate/sliding-input regressions. The existing core retry tests are supplemental guards, not proof of the mobile fix.

@bheemreddy-samsara

Copy link
Copy Markdown
Contributor Author

Implemented the independent code fixes in c6f49aa. Fill/clear now use one fresh relocation, including after the elapsed-budget shortcut, and follow refind()'s existing nearest-node matching. Removed uniqueInput, sameInputIdentity, freshInput, and geometry-equality rejection; retained the original binding's security-fact checks in that single relocation.

TEXT_INPUT_ROLES is exported and reused, and the docs, skill, and changeset are shortened. Duplicate OTP, sliding-input, and capture-count regressions reproduce the old PR's failures and pass with this change. The core fake-engine tests remain supplemental guards, not proof of the mobile fix. Focus behavior is unchanged.

#1300 and #1297 are still open. Their changes have not been copied into this branch; the final rebase remains pending until both land.

@cubic-dev-ai cubic-dev-ai Bot left a comment •

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

All reported issues were addressed across 7 files (changes from recent commits).

Heads up: you’ve reached your flex budget. Increase your flex budget or wait for usage to reset.

View guided diff | Re-trigger cubic

Comment thread skills/e2e/references/setup.md Resolved
Comment thread packages/mobile/src/surface.ts Resolved
@okwasniewski

Copy link
Copy Markdown
Member

Thanks, this is a lot smaller now and it fixes #1409 the right way. I re-checked c6f49aa (2dbda1c only changes docs). pnpm check passes and all 381 mobile unit tests pass. With surface.ts reverted to main, 25 tests fail: 24 input freshness tests plus the updated perform mapping test. So the new tests can catch a regression.

Earlier asks

  1. Identical fields: done. Two Digit text fields with no identifier: filling either one lands on its own ref (@e20 / @e21) for both test and agent origin. At 8d56213 the same scratch test failed.
  2. Sliding field: done. A text field first seen at y=2000 and landed as @e42 (adapted from the existing transition test) fills @e42 with 1 capture, same as main.
  3. Capture count: done. Fill/clear takes exactly 1 capture, both inside the budget window and after it. Main takes 0 after the window. That one extra capture is the cost of always relocating, and it's the shape we suggested.
  4. TEXT_INPUT_ROLES reused: done.
  5. Agreed, it stays as a supplemental guard.
  6. Docs/changeset shortened: done.
  7. The focus check (surface.ts:1273, role !== 'textbox') is still separate from the new TEXT_INPUT_ROLES guard, so focus and fill now accept different role sets. Not blocking.

Cubic on c6f49aa

  • surface.ts:1207 (nearest duplicate could get a secret meant for another field): not a real issue. Secret authorization checks the field's purpose, not which node it is (packages/e2e/src/agent/secrets.ts:52). The relocated node must also match inputPurpose / secure / disabled / hidden, so two fields it could confuse are indistinguishable to the model and to the policy. A uniqueness check would bring back ask 1.
  • The two setup.md points are covered by 2dbda1c.

Nits

  1. settled(entry, op, refreshInput) and relocated(entry, op, input) thread a boolean through both helpers, and refreshInput ? this.relocated(entry, operation, true) : entry is repeated at :1188 and :1192. A simpler shape for fill/clear: wait out the transition if needed (test origin only), then always call one input relocation. settled() would then keep its original signature.
  2. Drop "A coordinate-only backend can still race movement after this read" / "A fresh read does not make coordinate dispatch atomic" from the docs and skill. It describes internals, and users can't act on it.

Rebase: as you said, this conflicts in surface.ts with #1300, #1297 and #1413. All three are textual. After #1300, relocated() already throws NODE_STALE when nothing is found, so drop found === undefined from your condition. After #1413, pass raw.refsGeneration to bind() in relocated(). You already keep raw there, so that's a one-line change. Your fill path always acts on the newest capture, so it fits #1413's ref pinning. After #1297, bind() takes the projected screen.

This was an automated review.

@okwasniewski
okwasniewski dismissed their stale review October 27, 2026 16:11

Asks addressed at c6f49aa, see re-check comment. Automated review.

@bheemreddy-samsara

Copy link
Copy Markdown
Contributor Author

Addressed both cleanup nits in 7d71094. waitForTransition() only waits; fill/clear then perform one input relocation, while settled() keeps its original signature. The threaded refreshInput flag and repeated relocation ternaries are gone.

Removed the implementation-level coordinate-race wording from the docs and consumer skill, including the generated plugin copy. Added fill/clear cancellation guards to keep a cancelled transition wait from capturing or dispatching input.

The final rebase remains pending on #1300, #1297, and #1413.

@bheemreddy-samsara

Copy link
Copy Markdown
Contributor Author

@okwasniewski Can you give a another look ?

@okwasniewski

Copy link
Copy Markdown
Member

Re-check at 7d710946a: both nits are done. waitForTransition() only waits, settled() keeps its original signature, fill/clear do one input relocation, and the coordinate-race wording is gone from the docs and both skill copies. The new cancellation test covers fill and clear. CI is green. Re-labeled. Only the rebase after #1300 / #1297 / #1413 is left, as you said.

This was an automated review.

# Conflicts:
#	packages/mobile/src/surface.ts
@okwasniewski
okwasniewski added this pull request to the merge queue Oct 28, 2026
Merged via the queue into e2e-inc:main with commit b309bcd Oct 28, 2026
26 checks passed
kacperkapusciak added a commit that referenced this pull request Oct 28, 2026
An empty Compose Multiplatform field on iOS reports no placeholder; it
appends the text it draws to its label, so "Name, Ada Lovelace" read as a
field holding a name and the examples nightly's agent tried to clear it.
Read that trailing text as the field's placeholder when it is the field's
only child. It applies once the field reads editable: agent-device's XCTest
tree reports it so today, its simulator AX tree once
callstack/agent-device types text-entry-trait elements (follow-up bump).

#1410 refused fill and clear on anything but a text input role, which broke
the Compose Multiplatform and Flutter iOS examples: agent-device 0.21.33's
simulator AX tree reports their fields as an unclassified Other. Type into
an Other that holds nothing but text; a container of controls stays refused.
@bheemreddy-samsara
bheemreddy-samsara deleted the fix/mobile-input-freshness branch October 28, 2026 13:27
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Mobile fill can use stale input geometry after the transition budget expires

3 participants