Skip to content

fix(scroll): clamp WebDriver and HarmonyOS swipes inside the container - #4554

Open
aribaskagan wants to merge 2 commits into
callstack:mainfrom
aribaskagan:fix-2714-webdriver-harmony-scroll-container
Open

aribaskagan wants to merge 2 commits into
callstack:mainfrom
aribaskagan:fix-2714-webdriver-harmony-scroll-container

Conversation

@aribaskagan

@aribaskagan aribaskagan commented Nov 14, 2026 •

Copy link
Copy Markdown

Summary

WebDriver and HarmonyOS directional scrolls ignored containerRect, so a swipe could start or end outside the resolved list. Both owners now convert that rect into the swipe frame and clamp it with buildScrollGesturePlan. A shortened swipe reports pixels (dispatched) beside requestedPixels. A span that already fits, and a scroll with no container, stay unchanged.

Part of #2714 (WebDriver + HarmonyOS slice; Android done in #3746, local iOS runner left out since it plans in Swift).

AI assistance (Cursor) was used to prepare this change; I reviewed and tested it.

Validation

Tested SHA c03127a4958a7b0aecff57f6525f03e6ba3f6c4c, base d78be6c88715ad80fc658497f23f91c86bf9489b.

  • Before the production edit, pnpm exec vitest run --project unit-core on the two scroll test files: 3 failed, 8 passed (inset-container swipes ended outside the list). After: 11 passed.
  • pnpm check:affected --run --head c03127a4958a7b0aecff57f6525f03e6ba3f6c4c passed (format, lint, typecheck, layering, fallow, build, vitest-related, unit).
  • No live WebDriver grid or HarmonyOS device; coverage and device replay lanes are CI-only.

Autofix with [code]smith
Open in [code]smith · Need help on this PR? Tag @codesmith-bot with what you need.

View guided diff

Directional scroll on those owners ignored containerRect, so a swipe could start or end outside the resolved list. The shared planner now shortens that gesture and reports the dispatched distance beside the requested one.
Copilot AI balanced review requested due to automatic review settings November 14, 2026 15:41

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copilot was unable to review this pull request because the user who requested the review has insufficient remaining quota. Try requesting a review at a lower effort level.

An organization owner can choose the repository organization as the billing source for reviews requested by members with a paid Copilot license. Learn about Copilot code review billing.

@cubic-dev-ai cubic-dev-ai Bot left a comment •

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

All reported issues were addressed across 5 files

You’re at about 92% of the monthly reviewed-line limit. You may want to disable incremental reviews to conserve quota. Reviews will continue until that limit is exceeded. If you need help avoiding interruptions, please contact contact@cubic.dev.

Reply to a comment to ask cubic a question or push back. It learns from your replies.

View guided diff | Re-trigger cubic

@thymikee

Copy link
Copy Markdown
Member

Thanks for this. At c03127a the clamp logic looks right for WebDriver, but the HarmonyOS route still sends the swipe in the wrong coordinate frame, and its only regression test locks that in. The check run is green (one check, none failing), but no CI lane has a HarmonyOS device, so it does not cover the changed device path. Before merge, fix the Harmony translate-out and add one live run.

The test 'translates an offset window into the swipe frame before clamping' in https://github.com/callstack/agent-device/blob/c03127a/packages/platform-harmonyos/src/__tests__/input-actions.test.ts#L154 adds OFFSET_WINDOW.x/y back to the dispatched swipe before it checks the swipe is inside the container. That asserts the coordinates sent to uitest uiInput swipe are window-local. hdc takes screen coordinates, and the snapshot container and the Application rect are absolute. The test passes without the missing translate-out and fails once the open thread's fix is applied. As written, a non-origin window gets a swipe shifted by the window origin, and the test gives false confidence. Please assert that the raw dispatched coordinates are strictly inside the absolute container, with no offset added back, and make the production change that satisfies it.

The frame rule now has several owners. relativeToViewport in https://github.com/callstack/agent-device/blob/c03127a/packages/platform-harmonyos/src/input-actions.ts#L85 is a copy of the Android helper at platform-android/src/input-actions.ts:378-380, and WebDriver adds a third as relativeToFrame (webdriver-interactor.ts:89-92). Revyl has a fourth variant (provider-revyl/src/interactor.ts:411). Harmony is the site that forgot the translate-out, which shows the translate-in and translate-out pair belongs in the planner, not in each caller. The rule to satisfy: the planner takes the absolute container and returns absolute endpoints, so no backend can forget the way back. If a smaller change is preferred, export one translate helper from @agent-device/kernel/rect and import it in all three packages instead of adding a copy per package.

This changes swipe dispatch on two backends, and the PR body says no live WebDriver grid or HarmonyOS device was used. Every test uses a stubbed client or a mocked runHarmonyShell, so the shared snapshot, source/window and Application-rect frame is assumed, not shown. Please run one backend live at c03127a and put the output in the PR body. For WebDriver, use a real Appium or cloud-grid session on an inset scrollable (for example the tester app's inset /automation route, scrollable near rect=(18,178,366,662) on 402x874) and run scroll down --amount 0.75 --json. The output must show x1,y1,x2,y2 strictly inside the printed containerRect, pixels below requestedPixels, and movement: 'moved' instead of container-outside-swipe. Do the same on HarmonyOS if you have hardware, and repeat with a non-origin window after the fix if you can produce one. If neither backend is available, please say so as an unresolved risk.

Could the planner take a referenceOrigin (or a frame Rect) in ScrollGestureOptions, plan in the zero-origin frame, and return absolute x1..y2? Each backend would then pass { referenceOrigin: frame, container: containerRect } and dispatch the plan unchanged. That would delete Android's relativePlan/scrollPlan offset block and relativeToViewport, avoid a relativeToFrame in WebDriver, remove the Harmony helper, and make the translate-out omission impossible. The production diff would likely end up smaller than the current +25 lines. It needs a small change to platform-android/src/input-actions.ts first so Android uses the origin-aware planner. Revyl's clamped-screen variant can stay separate or follow.

The open inline thread on the Harmony translate-out (#4554 (comment)) still applies and matches the first finding above.

Not blocking, and fine to take or leave: scrollHarmony uses an inline {amount,pixels,durationMs} & ScrollContainerTarget type while WebDriver uses ScrollExecutionOptions, which Interactor.scroll also uses, so using it there would be more consistent. The 'fits unchanged' and 'without a container' tests in webdriver-interactor-scroll.test.ts pass without the fix, so they are non-regression guards, and only the inset-list tests fail without it.

I did not run the tests, so the author's 3 failed / 8 passed before-the-fix count is unverified. I also could not confirm that a HarmonyOS Application rect has a non-zero origin on shipping hardware, so the reach of the translate-out problem is limited to non-origin windows. I did not trace repeated scroll-until passes for these two backends.

HarmonyOS translated the container into a zero-origin frame and then sent those local endpoints to hdc, which takes screen coordinates. The planner now accepts the frame origin, plans in that zero-origin frame, and returns absolute endpoints, so HarmonyOS, WebDriver, and Android dispatch the plan instead of translating it themselves.

@cubic-dev-ai cubic-dev-ai Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

1 issue found across 7 files (changes from recent commits).

You’re at about 94% of the monthly reviewed-line limit. You may want to disable incremental reviews to conserve quota. Reviews will continue until that limit is exceeded. If you need help avoiding interruptions, please contact contact@cubic.dev.

Prompt for AI agents (unresolved issues)

Check if these issues are valid — if so, understand the root cause of each and fix them. When an issue isn't valid or won't be fixed in this PR, reply in its thread with the reason and then resolve the thread. If appropriate, use sub-agents to investigate and fix each issue separately.


<file name="packages/contracts/src/scroll-gesture.ts">

<violation number="1" location="packages/contracts/src/scroll-gesture.ts:158">
P2: When the cross-axis band is empty, `keepSwipeInContainer` returns the padded lane without applying the scrolled-axis clamp. A narrow container (under about 17px across) can therefore receive a swipe that starts outside its vertical range. Keep the axis clamp and fall back only the cross coordinate, or document this case explicitly.</violation>
</file>

Reply with feedback, questions, or to request a fix.

View guided diff | Re-trigger cubic

const crossPadding = scrollEdgePadding(crossLength);
const lane = options.container
? keepSwipeInContainer(unclamped, options.container, vertical, {
const lane = container

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P2: When the cross-axis band is empty, keepSwipeInContainer returns the padded lane without applying the scrolled-axis clamp. A narrow container (under about 17px across) can therefore receive a swipe that starts outside its vertical range. Keep the axis clamp and fall back only the cross coordinate, or document this case explicitly.

Prompt for AI agents
Check if this issue is valid — if so, understand the root cause and fix it. When an issue isn't valid or won't be fixed in this PR, reply in its thread with the reason and then resolve the thread. At packages/contracts/src/scroll-gesture.ts, line 158:

<comment>When the cross-axis band is empty, `keepSwipeInContainer` returns the padded lane without applying the scrolled-axis clamp. A narrow container (under about 17px across) can therefore receive a swipe that starts outside its vertical range. Keep the axis clamp and fall back only the cross coordinate, or document this case explicitly.</comment>

<file context>
@@ -138,8 +155,8 @@ export function buildScrollGesturePlan(options: ScrollGestureOptions): ScrollGes
   const crossPadding = scrollEdgePadding(crossLength);
-  const lane = options.container
-    ? keepSwipeInContainer(unclamped, options.container, vertical, {
+  const lane = container
+    ? keepSwipeInContainer(unclamped, container, vertical, {
         axis: { min: edgePadding, max: axisLength - edgePadding },
</file context>

@aribaskagan

Copy link
Copy Markdown
Author

Thanks for the review. Pushed ed6665a on top of c03127a.

HarmonyOS now sends screen coordinates. buildScrollGesturePlan takes referenceOrigin, subtracts it from the container, plans in the zero-origin frame, and adds that origin to every endpoint. scrollHarmony passes the Application rect origin and the absolute container, sends plan.x1 through plan.y2 to uitest uiInput swipe, and returns those endpoints.

The test is now HarmonyOS scroll dispatches an offset-window swipe inside the absolute container. It checks that the raw dispatched points are strictly inside the absolute container and does not add OFFSET_WINDOW back. On c03127a it fails (swipe was {"x1":201,"y1":186,"x2":201,"y2":669}, 1 failed, 7 passed in that file); it passes on ed6665a. The unclamped offset-window test had the same frame issue, so it now expects the zero-origin plan plus the window origin.

Android and WebDriver pass the same origin and absolute container and dispatch the plan's endpoints, so the Harmony and Android relativeToViewport copies and WebDriver's relativeToFrame are gone. Android still reports referenceWidth/referenceHeight as origin plus size (existing test expects 1090 by 1940); the planner leaves those alone, so WebDriver still reports the window size. Revyl's helper also clamps the container into the screen, so that variant stays in Revyl. scrollHarmony keeps its own options type since ScrollExecutionOptions includes releaseBehavior, which this owner does not implement.

No live WebDriver grid or HarmonyOS device was available, so there is no live run. That remains an unresolved risk; the tests use a mocked runHarmonyShell and a stubbed WebDriver client.

pnpm check:affected --run --head ed6665a passed with exactCommit: true (format, lint, typecheck, layering, fallow, build, vitest-related 407 passed, unit 19741 passed, 70 skipped). This also addresses the cubic inline thread.

@thymikee

Copy link
Copy Markdown
Member

Thanks for the update. At ed6665a the Harmony path now passes the Application origin and dispatches the planner's absolute endpoints, and the offset-window test checks the raw points against the absolute container, so that earlier gap is fixed. One thing still needs live evidence before merge.

The change alters the swipe coordinates sent on real device paths (input-actions.ts#L61). Harmony now sends Application-origin-shifted screen coordinates to uitest uiInput swipe, and WebDriver and Android now rely on the planner for absolute endpoints. Every test stubs runHarmonyShell or the WebDriver client, so nothing shows that the Application rect and the snapshot container rects share hdc's screen space. If they do not, the fix could do nothing or over-shift, and the mocked tests cannot catch it. The PR body Validation block still names c03127a, and the author comment says no device was available. Please run one backend live at ed6665a and paste the output in the PR body. For WebDriver, use a real Appium or cloud-grid session on an inset scrollable (for example the tester app's inset /automation route, container near rect=(18,178,366,662) on 402x874). scroll down --amount 0.75 --json should print x1,y1,x2,y2 strictly inside the printed containerRect, pixels below requestedPixels, and movement moved (not container-outside-swipe). For HarmonyOS, if you have hardware, run the same command in a non-origin Application window (split-screen or floating) and show that the raw swipe points sent to hdc lie strictly inside the absolute containerRect. Android output now goes through the new origin path too, so one existing-adb scroll down --json run on an offset or inset surface with unchanged x1..y2 would close that question. If no backend is reachable, the maintainer needs to accept the unproven risk explicitly.

The review thread on the empty cross band (#4554 (comment)) still applies. It only reaches sliver-width containers, so it is not blocking. The thread on Harmony origin (#4554 (comment)) no longer applies because input-actions.ts now passes the origin and dispatches absolute endpoints, with a test for it, so you can resolve it.

Not blocking, take or leave: the changeset and the PR body Summary do not say that an unclamped HarmonyOS scroll in a non-origin Application window now dispatches screen coordinates instead of window-local ones, so one clause in the changeset plus a refreshed Validation SHA when you add the live evidence would cover it.

CI shows one check, passing. It only runs unit tests with mocked Harmony and WebDriver clients, so no CI lane exercises these device paths. No conflicts. Merge needs the live run above.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants