Repository navigation
feat(locator): show how many candidates each filter step left - #1416
Conversation
There was a problem hiding this comment.
2 issues found across 6 files
Prompt for AI agents (unresolved issues)
Check if these issues are valid — if so, understand the root cause of each and fix them. When an issue isn't valid or won't be fixed in this PR, reply in its thread with the reason and then resolve the thread. If appropriate, use sub-agents to investigate and fix each issue separately.
<file name="docs/reference/errors.mdx">
<violation number="1" location="docs/reference/errors.mdx:109">
P2: The docs say a filtered locator's message always lists the per-step counts, but the counts are omitted when the post-failure diagnostic pass cannot read the page or when the message would exceed the 8192-character cap. Qualify both rows so readers know the counts can be absent.</violation>
</file>
<file name="packages/e2e/src/locator/expression.ts">
<violation number="1" location="packages/e2e/src/locator/expression.ts:136">
P2: A filter applied to `locator.or(...)` still produces candidate-count diagnostics because the walk accepts the union as its base. Preserve the existing union-root message by returning `undefined` when `current.kind === 'union'` as well as when no filter was found.</violation>
</file>
Heads up: you’ve reached your flex budget. Increase your flex budget or wait for usage to reset.
View guided diff | Re-trigger cubic
| | `APP_NOT_OPEN` | An operation needed an open app: `app.open()` had not run, or the engine reported invalid state | test | 1 | | ||
| | `LOCATOR_NOT_FOUND` | No matching node within the deadline, or an agent action refers to a node that is no longer available | test | 1 | | ||
| | `LOCATOR_AMBIGUOUS` | More than one match for a single-node operation. Fails at once, with no polling. | test | 1 | | ||
| | `LOCATOR_NOT_FOUND` | No matching node within the deadline, or an agent action refers to a node that is no longer available. For a filtered locator the message lists how many candidates each filter step left, counted after the failure | test | 1 | |
There was a problem hiding this comment.
P2: The docs say a filtered locator's message always lists the per-step counts, but the counts are omitted when the post-failure diagnostic pass cannot read the page or when the message would exceed the 8192-character cap. Qualify both rows so readers know the counts can be absent.
Prompt for AI agents
Check if this issue is valid — if so, understand the root cause and fix it. When an issue isn't valid or won't be fixed in this PR, reply in its thread with the reason and then resolve the thread. At docs/reference/errors.mdx, line 109:
<comment>The docs say a filtered locator's message always lists the per-step counts, but the counts are omitted when the post-failure diagnostic pass cannot read the page or when the message would exceed the 8192-character cap. Qualify both rows so readers know the counts can be absent.</comment>
<file context>
@@ -106,8 +106,8 @@ category a blocked verdict reports as the obstacle.
| `APP_NOT_OPEN` | An operation needed an open app: `app.open()` had not run, or the engine reported invalid state | test | 1 |
-| `LOCATOR_NOT_FOUND` | No matching node within the deadline, or an agent action refers to a node that is no longer available | test | 1 |
-| `LOCATOR_AMBIGUOUS` | More than one match for a single-node operation. Fails at once, with no polling. | test | 1 |
+| `LOCATOR_NOT_FOUND` | No matching node within the deadline, or an agent action refers to a node that is no longer available. For a filtered locator the message lists how many candidates each filter step left, counted after the failure | test | 1 |
+| `LOCATOR_AMBIGUOUS` | More than one match for a single-node operation. Fails at once, with no polling, and for a filtered locator lists how many candidates each filter step left, counted after the failure | test | 1 |
| `ACTION_FAILED` | Actionability failure, operation timeout, or an action that may already have committed | test | 1 |
</file context>
There was a problem hiding this comment.
Fixed in a11af50. Both rows now qualify the listing: the counts appear when the post-failure diagnostic pass can read the chain. The 8192-character report bound the message is serialized under drops the lines on an over-long chain, which keeps the failure valid without them.
| current = current.source; | ||
| } | ||
| if (!filtered) return undefined; | ||
| return [current, ...steps.toReversed()]; |
There was a problem hiding this comment.
P2: A filter applied to locator.or(...) still produces candidate-count diagnostics because the walk accepts the union as its base. Preserve the existing union-root message by returning undefined when current.kind === 'union' as well as when no filter was found.
Prompt for AI agents
Check if this issue is valid — if so, understand the root cause and fix it. When an issue isn't valid or won't be fixed in this PR, reply in its thread with the reason and then resolve the thread. At packages/e2e/src/locator/expression.ts, line 136:
<comment>A filter applied to `locator.or(...)` still produces candidate-count diagnostics because the walk accepts the union as its base. Preserve the existing union-root message by returning `undefined` when `current.kind === 'union'` as well as when no filter was found.</comment>
<file context>
@@ -116,6 +116,26 @@ export function indexExpression(
+ current = current.source;
+ }
+ if (!filtered) return undefined;
+ return [current, ...steps.toReversed()];
+}
+
</file context>
| return [current, ...steps.toReversed()]; | |
| if (!filtered || current.kind === 'union') return undefined; | |
| return [current, ...steps.toReversed()]; |
There was a problem hiding this comment.
Fixed in a11af50. filterSpine now returns undefined when the chain bottoms out in a union as well, so a filter over or() keeps today's message, and a test covers a filter sitting on a union.
There was a problem hiding this comment.
Fixed in 8c5b55b (amended): filterSpine now returns the union root as a single base entry, so a filter over or() counts the union once and never descends into its sides, with a unit test.
Preview approval requiredThis pull request comes from a fork, so Mintlify won't build a preview until someone with write access approves commit |
2517f76 to
a11af50
Compare
okwasniewski
left a comment
There was a problem hiding this comment.
Thanks @DeryFerd. The diagnostic is useful and the tests do real work: with engine.ts reverted to main, 4 of the new engine tests fail. Against apps/testbed on web I got the output the PR describes:
locator matched no nodes within getByRole("listitem").filter({ hasText: "urgent" }).filter({ has: getByRole("button", name: "Close C") }); candidates left after each filter step, counted after the failure:
getByRole("listitem"): 4
getByRole("listitem").filter({ hasText: "urgent" }): 2
...: 0
On web the walk added about 75 ms to a 1 s not-found, and an ambiguous failure still came back in about 23 ms. pnpm check passes. The cubic points about waitedMs and the union root are handled.
Must fix
-
packages/e2e/src/locator/engine.ts:267: the "fresh 500 ms deadline" does not limit the walk on mobile.packages/mobile/src/surface.tslocate()takes a full agent-device snapshot on every call, listening only tooperation.signaland never readingoperation.timeoutMs, and the sessionguard(engine/session.ts) adds no timeout. So a 3-step chain costs 3 extra full snapshots after every filtered failure, each running to completion. That matters most onLOCATOR_AMBIGUOUS, which the docs promise "fails at once". Please either:- bound the walk as a whole (one shared deadline for the whole walk, not a fresh one per prefix), or
- state the mobile cost and add a unit test pinning the walk's total time against a session that ignores
timeoutMs.
Either way, please measure one failure on an iOS or Android sim and post the number.
-
engine.ts:617MESSAGE_MAX_LENGTH: remove it.internal/errors.ts:417MAX_MESSAGE_BYTESalready caps every message inserializeError. The new copy counts characters where the existing cap counts bytes, and is applied only on the ambiguous path.
Nits
candidateCountsreturnsstring[] | undefinedand both callers writesteps === undefined ? '' : stepLines(steps). Return the finished suffix (''when there is nothing to show) andstepLines, the optional and both ternaries go away.expression.tsfilterSpine: dropping the lines when a filter sits on a union (a.or(b).filter(...)) has no clear reason. The union is one base expression and can get one count like any other base; only the walk itself needs to stop there.screenshot-capture.ts:60catchesLOCATOR_NOT_FOUNDfromperformUntil(scrollIntoView)and discards it, so on that path the walk is pure cost. Fine once (1) bounds the walk.- The
catch {}incandidateCountsalso swallows cancellation. Harmless today, butif (this.signal.aborted) throwwould make the intent explicit.
For the maintainer
#1293 is a feature request with no maintainer reply yet. Community policy asks for agreement on the issue before a feature PR.
This was an automated review.
a11af50 to
8c5b55b
Compare
|
Thanks for the careful review — all must-fix items landed in
On measuring a failure on an iOS or Android sim: I can't run simulators on this Windows machine, and this PR does not touch mobile engine code, so I can't post a real number from a device here. The bound is structural instead — one shared deadline caps the whole walk regardless of chain length, and each prefix read is a single immediate snapshot pass with no polling. Happy to post a number if there is a sim-backed run you would like me to compare against, or if CI's mobile deterministic suite exercises the path. |
There was a problem hiding this comment.
All reported issues were addressed across 3 files (changes from recent commits).
Heads up: you’ve reached your flex budget. Increase your flex budget or wait for usage to reset.
View guided diff | Re-trigger cubic
okwasniewski
left a comment
There was a problem hiding this comment.
Thanks @DeryFerd. I re-checked at 8c5b55b7a. Status of the earlier asks:
- (1) Walk bound: done. One shared deadline now covers the whole walk (
engine.ts:266). I accept that you can't measure on a simulator. With the shared deadline the cost is now bounded by design. - (2)
MESSAGE_MAX_LENGTH: removed. - (3)
candidateCountsreturns the finished suffix: done. - (4) A union at the base counts once: done.
- (6) Cancellation is rethrown: done.
pnpm check fails, though, and the new bound has no test.
Must fix
-
packages/e2e/src/locator/engine.ts:274: oxlint fails withCatch parameter 'cause' is caught but never used, so CIcheckandspec gateare red. Use} catch {. -
Add a test for the shared deadline. I mutated
candidateCountsin three ways, and all 72 engine tests still passed each time:- a fresh
new Deadline(STEP_COUNT_BUDGET_MS)per prefix - the
if (deadline.expired()) break;line deleted - the abort rethrow deleted
The current test only checks that each prefix's
timeoutMsis <= 500, which a fresh per-prefix deadline also passes. Please add a session whoselocate()ignorestimeoutMsand takes, say, 300 ms (fake time is fine), and assert that a 3-step chain stops after 2 resolves. Also add a test where the signal is aborted mid-walk and the error isCANCELLED. - a fresh
-
A walk that stops early prints only part of the list. If the deadline breaks the loop, or a later prefix throws (the point cubic raised at
:276), the message still says "candidates left after each filter step" but lists only the steps it read. On mobile this will be the common case: one full snapshot can use up the 500 ms budget, so usually only the base count shows. Either drop the lines when the walk did not reach the full chain, or say that it stopped (for example "walk stopped after 1 of 3 steps").
Nits
engine.ts:64: theSTEP_COUNT_BUDGET_MSJSDoc still says "fresh deadline one diagnostic prefix resolve ... gets". It is now the budget for the whole walk.- The
candidateCountsJSDoc says the walk "cannot cost more than one short budget even on mobile". A snapshot that starts just before the deadline still runs to completion, so the real bound is the budget plus one snapshot. Please say that. filterSpinecould return[]instead ofundefinedwhen there is no filter step. ThencandidateCountsneeds only one emptiness check, and the return type loses its optional.
This was an automated review.
8c5b55b to
38c1dad
Compare
|
All three must-fixes and all three nits are in the new head Must fix
Nits
Verified locally: |
Asks addressed, see re-review comment.
|
Re-review at
Optional nits:
This was an automated review. |
|
[claude-opus-5-5] automated review on behalf of @okwasniewski @DeryFerd heads up: this now conflicts with main in This was an automated review. |
A filtered locator's LOCATOR_NOT_FOUND and LOCATOR_AMBIGUOUS now list how many candidates each filter step left, read in one immediate pass after the failure on a fresh short deadline, and the message says the counts were read then. A chain whose diagnostic walk cannot read keeps the plain failure. The prefixes come from filterSpine in the expression module. Fixes e2e-inc#1293
38c1dad to
c57d113
Compare
|
Rebased onto latest main ( Verified after the rebase: |
|
[claude-opus-5-5] automated review on behalf of @okwasniewski Re-reviewed the rebase (c57d113 on main 720947d). The Small cleanups left over from the rebase (optional, not blocking):
|
What and why
A filtered locator fails with one line that names the whole expression, and that is all it says. With a chain like the one below,
LOCATOR_NOT_FOUNDcan mean any of three things: there are no list items yet, the text is spelled differently, or the button is named differently. Working out which step emptied the candidate set means reading the tree by hand, or rewriting the locator one step at a time and rerunning.LOCATOR_AMBIGUOUSonly says how many nodes the whole chain matched, so the step that failed to narrow stays anonymous there too.For a filtered locator, the two failures now carry the count each step left, in the error message itself:
The counts are read after the failure, one immediate resolve per prefix on a short fresh deadline, and the message says so, because a page that changes between the failing attempt and the diagnostic pass can disagree with it. The step labels reuse the existing expression rendering, so nothing new learns to describe a locator, and the text goes through the run's redaction at record time like any error message.
A few boundaries keep the message quiet when the lines would say nothing. A locator with no filter step, an index-only chain, and a union root keep today's message. If the diagnostic pass cannot read, the failure stands without the lines, since a diagnostic must not fail the failure.
LOCATOR_AMBIGUOUSkeeps itsmatchesdetail and gains the same lines.No new API and no wire change. The counts live in the message text,
report-v1bounds the message throughserializeErroras before, and error codes, exit codes, and the public type surface are unchanged. CONTRIBUTING.md's stability policy does not cover the exact text of error messages.What changed
packages/e2e/src/locator/expression.tsgainsfilterSpine(expression): the prefixes of a filtered chain, innermost first, followingfilterandindexsteps down theirsource. It returns undefined when no filter sits on the spine, which is what keeps plain queries andor()roots out of the message.packages/e2e/src/locator/engine.tsbuilds the lines where the two failures are raised. The not-found path inresolveCountedand the ambiguity path both callcandidateCounts, which resolves each prefix once throughresolveNowon a fresh 500 ms deadline capped by the attempt budget, and returns undefined when any resolve throws, so the message stays as it is today. The old module-levelassertSinglebecame the asyncsingleOfmethod, so both sites share one ambiguity error.packages/e2e/tests/unit/expression.test.tscovers the spine: the prefixes in order, an index step between filters, the no-filter cases, and a union root.packages/e2e/tests/unit/locator-engine.test.tsadds a fake session that answers per expression, keyed by the rendered locator, so the tests pin exactly which expressions a failure path resolves and in which order. They cover the per-step lines onLOCATOR_NOT_FOUNDandLOCATOR_AMBIGUOUS, a locator with no filter step left alone, the plain failure kept when the walk cannot read, and an index step between filters counted under its own label.docs/reference/errors.mdx: theLOCATOR_NOT_FOUNDandLOCATOR_AMBIGUOUSentries name the per-step listing.e2e, since the locator engine ships in the published package.Issue
Fixes #1293
Verified
Ran it locally: yes
packages/e2efails the same 22 files on this branch as on the commit the branch was cut from, checked by diffing the failing-file sets of twotests/unitruns:Those failures are this machine's Windows baseline, CRLF checkouts, symlink privilege, and path separators, and they are identical on both sides. The locator-adjacent files all pass: expression, locator-engine, resolve-expression, locate, locator-action-options, locator-basics, locator-union, and locator-wait-for, 296 tests together.
tsc --noEmitinpackages/e2eis clean, which includes thetests/types/sdk-types.tsassertions.oxlintreports 0 warnings and 0 errors on the four touched files.node scripts/check-error-codes.tsreports 87 codes in source, all documented, with the two updated entries in the set.node scripts/check-peer-ranges.tspasses, andnode scripts/check-changeset-bumps.tsreports the new changeset in the pending set with no unannounced 0.x minor or major.dist, which needs a full rebuild this box has not done. They run inspec.ymlon every pull request.Checklist
pnpm changeset) for any change to a published package, or the change is testbed, docs, or CI only.!, the changeset body names what breaks and what replaces it, and the deprecation warning landed a window earlier (stability policy in CONTRIBUTING.md).docs/**/*.mdxpage for the behavior, andskills/e2e/if the skill describes it..d.tsreviewed,tests/types/sdk-types.tsupdated; wire change edits the schema and both fixtures.e2e/engine): changesets fore2e,@e2e-dev/web, and@e2e-dev/mobile.