From ecfc854773f4bad931d5f1e7dd2b7580867641f4 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Dar=C3=ADo=20Kondratiuk?= Date: Tue, 3 Nov 2026 18:51:04 -0300 Subject: [PATCH 1/2] List the candidates each filter step left on a failed filtered locator (upstream candidateCounts) Refs tester-army/e2e#1416 (79e2e13d3365). Co-Authored-By: Claude Sonnet 5.5 Claude-Session: https://claude.ai/code/session_01Ls2e1yaZFgZ3FEJkYScfkK --- COMPATIBILITY.md | 2 +- docs-site/src/content/docs/docs/index.mdx | 2 + skills/e2e-dotnet/SKILL.md | 5 +- skills/e2e-dotnet/references/debugging.md | 4 +- src/E2E/Internal/Time.cs | 3 + src/E2E/Locator/LocatorEngine.cs | 87 ++++++- src/E2E/Locator/Screen.cs | 29 ++- .../Expression/LocatorExpressionsTests.cs | 49 ++++ .../LocatorFailureCandidateCountsTests.cs | 220 ++++++++++++++++++ 9 files changed, 386 insertions(+), 15 deletions(-) create mode 100644 tests/E2E.Tests/LocatorEngine/LocatorFailureCandidateCountsTests.cs diff --git a/COMPATIBILITY.md b/COMPATIBILITY.md index 53341c33a..c67b271fe 100644 --- a/COMPATIBILITY.md +++ b/COMPATIBILITY.md @@ -121,7 +121,7 @@ Names are C# versions of the JavaScript API: `agent.act` is `ActAsync`, `screen. ## Error codes that differ -A locator reports an engine failure as upstream's `translateLocatorError` (`LocatorEngine.TranslateLocatorError`, over `ErrorSerializer.TranslateEngineError`, upstream's `translateEngineError`, `internal/errors.ts`) does, with the engine error as `InnerException` and the locator's description added to the message (#846): `NOT_ACTIONABLE` and `ACTION_MAY_HAVE_COMMITTED` are `ACTION_FAILED`, `OPERATION_TIMEOUT` is `ACTION_FAILED` (`operation timed out: `), `NODE_STALE` and `FRAME_NOT_FOUND` are `LOCATOR_NOT_FOUND`, `FRAME_AMBIGUOUS` is `LOCATOR_AMBIGUOUS`, and `INVALID_STATE` is `APP_NOT_OPEN`. `CANCELLED` is an `EngineException` of that code (`operation cancelled`), `UNSUPPORTED_CAPABILITY` is a `ConfigurationException` (category `configuration`), and `ENGINE_FAILURE`, any other engine code, and an error that is no e2e error are an `EngineException` `ENGINE_FAILURE` (category `infrastructure`), each with the original error as `InnerException`, as upstream's `translateEngineError` wraps them (#885). Differences: an `OperationCanceledException` is returned as it is, so a caller's cancel stays a cancel. The port applies the translation to the locator's resolve (including the resolve over an observation) and to its actions; it has no read by reference to translate. `LocatorEngine` (`Locator/LocatorEngine.cs`, upstream's `locator/engine.ts`, #994) owns the resolve, the retries, the deadlines, the engine events, and the operation context a call carries (`OperationContext.Test`); `Screen` and `Locator` pass it their query. Differences: its session is the set of calls in `LocatorEngineOptions` (.NET-only: the attempt reads the screen through its scope), a locate answers the nodes themselves so `read`, `readNow`, `readAllNow`, and `tryRead` read a resolve and have no `session.read` to go stale, `fillSecret` and `pressSequentially` are not engine methods (a secret fill and a typed text are engine actions), and `DeadlineFor` is upstream's `deadline` (`Deadline` is the type's name). `LocatorEngine/` ports the `read retry`, `resolve retry`, `perform`, `readAllNow`, `reads that do not wait`, `translateLocatorError`, `visible queries`, and `isNodeVisible` describes; not ported: the its that need `session.read` to go stale (read retry 1 to 3, read contract, readAllNow stale and failure, readNow stale between resolve and read), `secret fill` (no engine `fillSecret`), `AttemptBudget.capped` (no `AttemptBudget`), the cross-realm describe (no second module registry in .NET), and the `locator failure candidate counts` describe (#1001). An action whose engine call finds a retryable stale node resolves the locator again and repeats the action while its deadline remains, as upstream's `performUntil` does; an action that may have committed is never repeated. An action whose deadline passed before it was sent is not sent: it fails with `ACTION_FAILED` (`operation timed out before the tap was sent: `), as upstream's `lateAction` (#869). Tests: `LocatorEngine/TranslateLocatorErrorMappingTableTests` (the `translateLocatorError mapping table` describe; the seven rows with the port's codes are `Partial_`) and the perform `it`s of `LocatorEngine/LocatorEnginePerformContractTests`. All six perform `it`s are ported; `marks every action it performs with the test origin` runs on the `LocatorEngine` itself (`ScriptedLocatorEngine`, upstream's `makeEngine`), and a .NET-only test checks that `E2ESession` wires that origin to the engine a test's step reaches (`OriginRecordingEngine`). +A locator reports an engine failure as upstream's `translateLocatorError` (`LocatorEngine.TranslateLocatorError`, over `ErrorSerializer.TranslateEngineError`, upstream's `translateEngineError`, `internal/errors.ts`) does, with the engine error as `InnerException` and the locator's description added to the message (#846): `NOT_ACTIONABLE` and `ACTION_MAY_HAVE_COMMITTED` are `ACTION_FAILED`, `OPERATION_TIMEOUT` is `ACTION_FAILED` (`operation timed out: `), `NODE_STALE` and `FRAME_NOT_FOUND` are `LOCATOR_NOT_FOUND`, `FRAME_AMBIGUOUS` is `LOCATOR_AMBIGUOUS`, and `INVALID_STATE` is `APP_NOT_OPEN`. `CANCELLED` is an `EngineException` of that code (`operation cancelled`), `UNSUPPORTED_CAPABILITY` is a `ConfigurationException` (category `configuration`), and `ENGINE_FAILURE`, any other engine code, and an error that is no e2e error are an `EngineException` `ENGINE_FAILURE` (category `infrastructure`), each with the original error as `InnerException`, as upstream's `translateEngineError` wraps them (#885). Differences: an `OperationCanceledException` is returned as it is, so a caller's cancel stays a cancel. The port applies the translation to the locator's resolve (including the resolve over an observation) and to its actions; it has no read by reference to translate. `LocatorEngine` (`Locator/LocatorEngine.cs`, upstream's `locator/engine.ts`, #994) owns the resolve, the retries, the deadlines, the engine events, and the operation context a call carries (`OperationContext.Test`); `Screen` and `Locator` pass it their query. Differences: its session is the set of calls in `LocatorEngineOptions` (.NET-only: the attempt reads the screen through its scope), a locate answers the nodes themselves so `read`, `readNow`, `readAllNow`, and `tryRead` read a resolve and have no `session.read` to go stale, `fillSecret` and `pressSequentially` are not engine methods (a secret fill and a typed text are engine actions), and `DeadlineFor` is upstream's `deadline` (`Deadline` is the type's name). `LocatorEngine/` ports the `read retry`, `resolve retry`, `perform`, `readAllNow`, `reads that do not wait`, `translateLocatorError`, `visible queries`, `isNodeVisible`, and `locator failure candidate counts` (`LocatorFailureCandidateCountsTests`) describes; not ported: the its that need `session.read` to go stale (read retry 1 to 3, read contract, readAllNow stale and failure, readNow stale between resolve and read), `secret fill` (no engine `fillSecret`), `AttemptBudget.capped` (no `AttemptBudget`), and the cross-realm describe (no second module registry in .NET). A filtered locator that matches nothing or too many lists the candidates each filter step left (tester-army/e2e#1416, `79e2e13d3365`): `LocatorExpression.FilterSpine()` is `filterSpine` (`locator/expression.ts`), and `LocatorEngine.CandidateCountsAsync` is `candidateCounts`, read by `NotFoundAsync` (`notFound` in `resolveCounted`) and `AmbiguousAsync` (`singleOf`, which replaces `assertSingle`), one immediate resolve per prefix of the chain under one 500 ms budget (`StepCountBudget`, `STEP_COUNT_BUDGET_MS`; `Deadline.Min` is `Deadline.min`). A walk that cannot finish says `walk stopped after N of M steps`, a read that fails drops the list, and a cancel surfaces as cancellation. Differences: the port's failure messages are its own (` matched 0 nodes.`), so the list follows on a new line (`candidates left after each filter step, counted after the failure:`) where upstream continues after `; `. The five `filterSpine` its are in `Expression/LocatorExpressionsTests`. An action whose engine call finds a retryable stale node resolves the locator again and repeats the action while its deadline remains, as upstream's `performUntil` does; an action that may have committed is never repeated. An action whose deadline passed before it was sent is not sent: it fails with `ACTION_FAILED` (`operation timed out before the tap was sent: `), as upstream's `lateAction` (#869). Tests: `LocatorEngine/TranslateLocatorErrorMappingTableTests` (the `translateLocatorError mapping table` describe; the seven rows with the port's codes are `Partial_`) and the perform `it`s of `LocatorEngine/LocatorEnginePerformContractTests`. All six perform `it`s are ported; `marks every action it performs with the test origin` runs on the `LocatorEngine` itself (`ScriptedLocatorEngine`, upstream's `makeEngine`), and a .NET-only test checks that `E2ESession` wires that origin to the engine a test's step reaches (`OriginRecordingEngine`). ## Locator actions diff --git a/docs-site/src/content/docs/docs/index.mdx b/docs-site/src/content/docs/docs/index.mdx index f80162b87..c036984fa 100644 --- a/docs-site/src/content/docs/docs/index.mdx +++ b/docs-site/src/content/docs/docs/index.mdx @@ -125,4 +125,6 @@ await Screen.GetByTestId("feed").ScrollUntilVisibleAsync(row, new ScrollUntilVis A locator reports an engine failure in the test's own codes, with the engine's error as `InnerException`: `NOT_ACTIONABLE`, `ACTION_MAY_HAVE_COMMITTED`, and `OPERATION_TIMEOUT` are `ACTION_FAILED`, a stale node or a missing frame is `LOCATOR_NOT_FOUND`, an ambiguous frame is `LOCATOR_AMBIGUOUS`, and `INVALID_STATE` is `APP_NOT_OPEN`. `CANCELLED` and `ENGINE_FAILURE` keep their codes (category `infrastructure`), and `UNSUPPORTED_CAPABILITY` is a `ConfigurationException`. A `locator.WaitForAsync` that runs out fails with `LOCATOR_NOT_FOUND` (`locator did not become : `), not `TIMEOUT`, and a pass verifies the acts before it. A stale node found by an action is looked up again, within the action timeout, before the action fails. +When a filtered locator (`Filter(...)`, `First()`, `Nth(n)`) matches nothing or too many, the `LOCATOR_NOT_FOUND` or `LOCATOR_AMBIGUOUS` message ends with `candidates left after each filter step, counted after the failure:` and one line per step, innermost first, with the locator so far and how many nodes it left. A step that narrowed too much or too little names itself. The counts come from one extra read per step after the failure, so a page that changed since can disagree with the failing attempt. All the reads share one 500 ms budget: a chain that cannot finish says `walk stopped after N of M steps`, and a read that fails leaves the list out. A locator without a filter step gets no list. + A read or navigation that times out fails with `OPERATION_TIMEOUT`. It is not retried, even when the call log mentions a detached element or a navigation race. On a device, an input (a tap, key, swipe, or `Back`) that the device does not answer within `actionTimeout` fails with `ACTION_MAY_HAVE_COMMITTED` and is never repeated, because it may have landed; an input the timeout ran out before is not sent (`OPERATION_TIMEOUT`), and a long press that holds at least as long as the time left is refused (`NOT_ACTIONABLE`). A locate, an observation, or a screenshot on a device runs within the timeout of the call that takes it, so a device that stops answering fails the step with `ACTION_FAILED` instead of holding it until the test timeout. `WaitForAsync`, `Expect`, and `ToHaveScreenshotAsync` give their last read extra time, up to their slowest read so far and within the test timeout, so a screen that shows up during it still passes. See `EngineErrorCodes` in the [API reference](/api/). diff --git a/skills/e2e-dotnet/SKILL.md b/skills/e2e-dotnet/SKILL.md index 9cf4caf8a..d1b05dcb4 100644 --- a/skills/e2e-dotnet/SKILL.md +++ b/skills/e2e-dotnet/SKILL.md @@ -282,7 +282,10 @@ lines you changed: - Text matches are exact by default. A locator that matches two nodes fails with `LOCATOR_AMBIGUOUS`; narrow it, or add `First()` to an `a.Or(b)` - whose branches both match (topic `writing-tests`). + whose branches both match (topic `writing-tests`). When a filtered locator + matches nothing or too many, the message lists the candidates each filter + step left, counted after the failure; the step where the count jumps or + drops to 0 is the one to change. - Narrow a locator with `Filter(hasText:, hasNotText:, has:, hasNot:)`; any of the four, at least one. `hasText` and `hasNotText` are case-insensitive substrings (or a `Regex`) over the subtree text. `has` keeps, and `hasNot` diff --git a/skills/e2e-dotnet/references/debugging.md b/skills/e2e-dotnet/references/debugging.md index 53e74cae2..d1491dce5 100644 --- a/skills/e2e-dotnet/references/debugging.md +++ b/skills/e2e-dotnet/references/debugging.md @@ -243,8 +243,8 @@ failed before it read a screen has `observationRevision` `unrecorded`. See | `INVALID_ARGUMENT` | A step argument failed validation (an unknown agent, a budget above the agent's) | Fix the call | | `SECRET_UNAVAILABLE` | `Secrets.Get` named a secret the config does not declare | Add it under `secrets` | | `AUTH_CREDENTIAL_UNAVAILABLE` | `Credentials.User(name)` found no `E2E_USER__USERNAME` or `_PASSWORD` | Set both variables | -| `LOCATOR_NOT_FOUND` | Wrong role or name, inexact text, element off screen, page not open, or a `locator.WaitForAsync` state that never held (`locator did not become `) | Read the markup for the accessible name; `exact: false` or a `Regex`; `App.OpenAsync` first | -| `LOCATOR_AMBIGUOUS` | Two matches (hidden duplicate, repeated label) | A name, a container scope, `Filter(...)`, or `First()`, `Nth(i)` | +| `LOCATOR_NOT_FOUND` | Wrong role or name, inexact text, element off screen, page not open, or a `locator.WaitForAsync` state that never held (`locator did not become `) | Read the markup for the accessible name; `exact: false` or a `Regex`; `App.OpenAsync` first. For a filtered locator the message lists how many nodes each filter step left, counted after the failure | +| `LOCATOR_AMBIGUOUS` | Two matches (hidden duplicate, repeated label) | A name, a container scope, `Filter(...)`, or `First()`, `Nth(i)`. For a filtered locator the message lists how many nodes each step left (`candidates left after each filter step, counted after the failure:`): the step where the count jumps or drops to 0 is the one to change | | `APP_NOT_OPEN` | A `Screen` call before `App.OpenAsync()` | Open the app first | | `ASSERTION_FAILED` | Wrong expectation, or the state settles later than 5 s; for `AssertAsync`, a false judgment | Compare with the actual text in the message; `timeout` on the matcher; rewrite the statement. Do not loosen it to pass | | `ASSERTION_INCONCLUSIVE` | The judge could not decide from what the screen shows | Reach the right screen first; make the statement concrete and visible. When the answer is in pixels (a chart, a canvas), pass `Vision = VisionMode.On` | diff --git a/src/E2E/Internal/Time.cs b/src/E2E/Internal/Time.cs index 99e558f5e..9ca851841 100644 --- a/src/E2E/Internal/Time.cs +++ b/src/E2E/Internal/Time.cs @@ -20,6 +20,9 @@ public Deadline(TimeSpan timeout, TimeProvider clock) public TimeSpan Remaining(DateTimeOffset now) => now >= EndsAt ? TimeSpan.Zero : EndsAt - now; public bool Expired(DateTimeOffset now) => now >= EndsAt; + + /// The earlier of two deadlines (upstream's Deadline.min). + public static Deadline Min(Deadline first, Deadline second) => first.EndsAt <= second.EndsAt ? first : second; } /// diff --git a/src/E2E/Locator/LocatorEngine.cs b/src/E2E/Locator/LocatorEngine.cs index a98dbacbb..2c8538a7a 100644 --- a/src/E2E/Locator/LocatorEngine.cs +++ b/src/E2E/Locator/LocatorEngine.cs @@ -69,6 +69,12 @@ internal sealed class LocatorEngine(LocatorEngineOptions options) internal TimeSpan PollInterval { get; } = TimeSpan.FromMilliseconds(50); + /// + /// The budget the whole diagnostic failure walk shares: best effort, never a wait (upstream's STEP_COUNT_BUDGET_MS). One deadline + /// bounds every prefix resolve, so a multi-step chain cannot cost more than one short budget; the real bound is the budget plus one read. + /// + private static readonly TimeSpan StepCountBudget = TimeSpan.FromMilliseconds(500); + /// The end of the attempt's test timeout, which no read may outlast (upstream's budget.deadline); null outside a session or with no test timeout. internal Deadline? BudgetDeadline { get; set; } @@ -345,7 +351,7 @@ internal Task> ReadAllNowAsync(LocatorExpression exp internal async Task TryReadAsync(LocatorExpression expression, CancellationToken token, ReadWindow? reads = null, MissingFrame missingFrame = MissingFrame.Wait) { var matches = await ResolveOnceAsync(expression, token, reads: reads, missingFrame: missingFrame).ConfigureAwait(false); - return matches.Count > 1 ? throw Ambiguous(expression, matches.Count) : matches.Count == 1 ? matches[0] : null; + return matches.Count > 1 ? throw await AmbiguousAsync(expression, matches.Count, token).ConfigureAwait(false) : matches.Count == 1 ? matches[0] : null; } /// @@ -370,7 +376,7 @@ internal async Task ReadAsync(LocatorExpression expression, Cancel if (Clock.GetUtcNow() >= deadline) { - throw NotFound(expression, Clock.GetUtcNow() - started); + throw await NotFoundAsync(expression, Clock.GetUtcNow() - started, token).ConfigureAwait(false); } await Task.Delay(PollInterval, token).ConfigureAwait(false); @@ -397,7 +403,7 @@ internal async Task ResolveExactlyOneAsync(LocatorExpression expre if (deadline.Expired(Clock.GetUtcNow())) { - throw NotFound(expression, Clock.GetElapsedTime(started)); + throw await NotFoundAsync(expression, Clock.GetElapsedTime(started), token).ConfigureAwait(false); } await Task.Delay(PollInterval, token).ConfigureAwait(false); @@ -416,7 +422,7 @@ internal async Task ResolveExactlyOneAsync(LocatorExpression expre } catch (Exception ex) when (sampled && !token.IsCancellationRequested && reads.CutOff(ex)) { - throw new TestException("LOCATOR_NOT_FOUND", expression.Describe() + " matched 0 nodes.", ex) { Details = LocatorDetails(expression, elapsed()) }; + throw await NotFoundAsync(expression, elapsed(), token, ex).ConfigureAwait(false); } } @@ -483,7 +489,7 @@ internal async Task PerformAsync(LocatorExpression expression, Func= deadline) { throw node is null - ? NotFound(expression, Clock.GetUtcNow() - started) + ? await NotFoundAsync(expression, Clock.GetUtcNow() - started, token).ConfigureAwait(false) : new TestException("NOT_ACTIONABLE", expression.Describe() + (node.States.Hidden ? " is hidden." : " is disabled.")); } @@ -537,7 +543,7 @@ internal async Task PerformUntilAsync(LocatorExpression expression, LocatorActio if (deadline.Expired(Clock.GetUtcNow())) { throw node is null - ? NotFound(expression, Clock.GetElapsedTime(started)) + ? await NotFoundAsync(expression, Clock.GetElapsedTime(started), token).ConfigureAwait(false) : new TestException("NOT_ACTIONABLE", expression.Describe() + (node.States.Hidden ? " is hidden." : " is disabled.")); } @@ -700,9 +706,70 @@ private void RecordAction(SemanticNode node, LocatorAction action, DateTimeOffse }); } - internal static TestException Ambiguous(LocatorExpression expression, int count) => - new("LOCATOR_AMBIGUOUS", expression.Describe() + " matched " + count.ToString(CultureInfo.InvariantCulture) + " nodes.") { Details = LocatorDetails(expression) with { Matches = count } }; + /// + /// LOCATOR_AMBIGUOUS for a resolve of many nodes, with the per-step counts of a filtered locator (upstream's singleOf). + /// + internal async Task AmbiguousAsync(LocatorExpression expression, int count, CancellationToken token) + { + var lines = await CandidateCountsAsync(expression, token).ConfigureAwait(false); + return new("LOCATOR_AMBIGUOUS", expression.Describe() + " matched " + count.ToString(CultureInfo.InvariantCulture) + " nodes." + lines) { Details = LocatorDetails(expression) with { Matches = count } }; + } + + /// + /// LOCATOR_NOT_FOUND, with the per-step counts of a filtered locator (upstream's notFound in resolveCounted). + /// is the wait as it happened, read before the diagnostic pass runs: the prefix resolves are not part of it. + /// + private async Task NotFoundAsync(LocatorExpression expression, TimeSpan waited, CancellationToken token, Exception? cause = null) + { + var message = expression.Describe() + " matched 0 nodes." + await CandidateCountsAsync(expression, token).ConfigureAwait(false); + var details = LocatorDetails(expression, waited); + return cause is null ? new TestException("LOCATOR_NOT_FOUND", message) { Details = details } : new TestException("LOCATOR_NOT_FOUND", message, cause) { Details = details }; + } + + /// + /// The diagnostic on a filtered locator's failure: one immediate resolve per prefix of its chain, innermost first, and how many + /// candidates each left, as the message's per-step lines (upstream's candidateCounts). One deadline bounds the whole walk, so even + /// on a session whose locate ignores its budget, a chain whose walk cannot finish lists only the prefixes it read and says so, or lists + /// nothing when a read failed outright. The counts are read after the failure, so a page that changed since can disagree with the failing + /// attempt, and the message says so. + /// + private async Task CandidateCountsAsync(LocatorExpression expression, CancellationToken token) + { + var spine = expression.FilterSpine(); + if (spine.Count == 0) + { + return ""; + } + + var deadline = Deadline.Min(new Deadline(StepCountBudget, Clock), BudgetDeadline ?? new Deadline(DateTimeOffset.MaxValue)); + var reads = new ReadWindow(deadline, Clock); + List lines = []; + try + { + foreach (var prefix in spine) + { + if (deadline.Expired(Clock.GetUtcNow())) + { + break; + } - private static TestException NotFound(LocatorExpression expression, TimeSpan waited) => - new("LOCATOR_NOT_FOUND", expression.Describe() + " matched 0 nodes.") { Details = LocatorDetails(expression, waited) }; + var nodes = await ResolveNowAsync(prefix, token, reads).ConfigureAwait(false); + lines.Add(prefix.Describe() + ": " + nodes.Count.ToString(CultureInfo.InvariantCulture)); + } + } + catch (Exception) + { + // A diagnostic read failure must not fail the failure: partial counts must not read as a completed walk, so return nothing. + token.ThrowIfCancellationRequested(); + return ""; + } + + if (lines.Count == 0) + { + return ""; + } + + var stopped = lines.Count < spine.Count ? ", walk stopped after " + lines.Count.ToString(CultureInfo.InvariantCulture) + " of " + spine.Count.ToString(CultureInfo.InvariantCulture) + " steps" : ""; + return "\ncandidates left after each filter step" + stopped + ", counted after the failure:\n" + string.Join('\n', lines); + } } diff --git a/src/E2E/Locator/Screen.cs b/src/E2E/Locator/Screen.cs index 9b39675be..9f6d76cd6 100644 --- a/src/E2E/Locator/Screen.cs +++ b/src/E2E/Locator/Screen.cs @@ -934,7 +934,7 @@ await samples.RecordAsync(_screen.Steps, token, () => Time.PollReadsAsync(new Po var matches = await ResolveAsync(token, includeHidden: true, reads, state is WaitForState.Detached or WaitForState.Hidden ? MissingFrame.Empty : MissingFrame.Wait).ConfigureAwait(false); if (matches.Count > 1) { - throw LocatorEngine.Ambiguous(Query, matches.Count); + throw await _screen.Engine.AmbiguousAsync(Query, matches.Count, token).ConfigureAwait(false); } samples.Add(matches.Count == 0 ? "absent" : matches[0].States.Hidden ? "hidden" : "visible"); @@ -1359,6 +1359,33 @@ internal ErrorDetails ExpressionHints() } } + /// + /// The prefixes of a filtered locator's chain, innermost first: the base and every filter or index step applied over it + /// (upstream's filterSpine). Empty when the chain has no filter step, since a plain query, selector, or frame has no step a + /// candidate count can explain. A union at the base counts once like any other base; the walk stops there and never descends into its + /// two sides. + /// + internal IReadOnlyList FilterSpine() + { + List steps = []; + var current = this; + var filtered = false; + while (current.Kind is "filter" or "index") + { + filtered |= current.Kind == "filter"; + steps.Add(current); + current = current.Source!; + } + + if (!filtered) + { + return []; + } + + steps.Reverse(); + return [current, .. steps]; + } + public string Describe() { switch (Kind) diff --git a/tests/E2E.Tests/Expression/LocatorExpressionsTests.cs b/tests/E2E.Tests/Expression/LocatorExpressionsTests.cs index 86206459f..4a9dd38c3 100644 --- a/tests/E2E.Tests/Expression/LocatorExpressionsTests.cs +++ b/tests/E2E.Tests/Expression/LocatorExpressionsTests.cs @@ -124,4 +124,53 @@ public void Rejects_empty_filters() Assert.Equal("INVALID_LOCATOR", error.Code); Assert.Contains("hasText, hasNotText, has, and/or hasNot", error.Message, StringComparison.Ordinal); } + + [Fact] + public void Lists_every_prefix_of_a_filtered_chain_innermost_first() + { + var baseQuery = _screen.GetByRole("listitem").Query; + var first = new Locator(_screen, baseQuery).Filter("Order").Query; + var second = new Locator(_screen, first).Filter(has: _screen.GetByRole("button")).Query; + + Assert.Equal([baseQuery, first, second], second.FilterSpine()); + } + + [Fact] + public void Follows_an_index_step_between_filters() + { + var baseQuery = _screen.GetByRole("listitem").Query; + var filtered = new Locator(_screen, baseQuery).Filter("Order").Query; + var indexed = new Locator(_screen, filtered).First().Query; + var chained = new Locator(_screen, indexed).Filter("1042").Query; + + Assert.Equal([baseQuery, filtered, indexed, chained], chained.FilterSpine()); + } + + [Fact] + public void Filter_spine_is_empty_without_a_filter_step() + { + var baseLocator = _screen.GetByRole("listitem"); + + Assert.Empty(baseLocator.Query.FilterSpine()); + Assert.Empty(baseLocator.Nth(0).Query.FilterSpine()); + Assert.Empty(_screen.GetByText("Pro").Query.FilterSpine()); + } + + [Fact] + public void Filter_spine_is_empty_for_a_union_of_chains() + { + var baseLocator = _screen.GetByRole("listitem"); + var union = baseLocator.Filter("Order").Or(baseLocator); + + Assert.Empty(union.Query.FilterSpine()); + } + + [Fact] + public void Treats_a_union_as_a_single_base_to_count_once_never_descending_into_its_sides() + { + var union = _screen.GetByRole("listitem").Or(_screen.GetByRole("button")); + var filtered = union.Filter("Order").Query; + + Assert.Equal([union.Query, filtered], filtered.FilterSpine()); + } } diff --git a/tests/E2E.Tests/LocatorEngine/LocatorFailureCandidateCountsTests.cs b/tests/E2E.Tests/LocatorEngine/LocatorFailureCandidateCountsTests.cs new file mode 100644 index 000000000..f0e6f44c2 --- /dev/null +++ b/tests/E2E.Tests/LocatorEngine/LocatorFailureCandidateCountsTests.cs @@ -0,0 +1,220 @@ +// Copyright 2026 TesterArmy. +// Modified by Dario Kondratiuk. +// SPDX-License-Identifier: Apache-2.0 + +using E2E.Engine; +using Microsoft.Extensions.Time.Testing; + +namespace E2E.Tests.LocatorEngine; + +/// +/// Upstream's locator failure candidate counts (tester-army/e2e#1416, 79e2e13d3365). The engine answers per expression, keyed by +/// the locator as it is described, so a test pins which expressions a failure resolves and how much budget each resolve was given. +/// +public sealed class LocatorFailureCandidateCountsTests +{ + private const string Heading = "candidates left after each filter step, counted after the failure"; + + private static readonly SemanticNode Ref = new() { Ref = "node-1", Role = "listitem", Name = "Order 1042" }; + + private readonly Screen _screen = ScreenFixture.Create(); + private readonly LocatorExpression _base; + private readonly LocatorExpression _texted; + private readonly LocatorExpression _chain; + + public LocatorFailureCandidateCountsTests() + { + var baseLocator = _screen.GetByRole("listitem"); + var texted = baseLocator.Filter("Order 1042"); + _base = baseLocator.Query; + _texted = texted.Query; + _chain = texted.Filter(has: _screen.GetByRole("button", "Cancel order")).Query; + } + + [Fact] + public async Task Lists_how_many_candidates_each_filter_step_left_counted_after_the_failure() + { + var counting = Counting(Answers((_chain, Nodes(0)), (_base, Nodes(6)), (_texted, Nodes(1)))); + + var failure = await Assert.ThrowsAsync(() => counting.Engine.ReadAsync(_chain, CancellationToken.None)); + + Assert.Equal("LOCATOR_NOT_FOUND", failure.Code); + Assert.Contains(Heading, failure.Message, StringComparison.Ordinal); + Assert.Contains($"{_base.Describe()}: 6", failure.Message, StringComparison.Ordinal); + Assert.Contains($"{_texted.Describe()}: 1", failure.Message, StringComparison.Ordinal); + Assert.Contains($"{_chain.Describe()}: 0", failure.Message, StringComparison.Ordinal); + + // The walk re-resolves the prefixes innermost first, after the failing resolve's polls. + Assert.Equal(_chain.Describe(), counting.Resolved[0]); + Assert.Equal([_base.Describe(), _texted.Describe(), _chain.Describe()], counting.Resolved.TakeLast(3)); + + // The failing resolve polled on the action deadline; the diagnostic resolves arrived after it, each bounded by the 500 ms diagnostic budget. + Assert.InRange(counting.Timeouts[0], TimeSpan.FromMilliseconds(19), TimeSpan.FromMilliseconds(20)); + var diagnostic = counting.Timeouts.TakeLast(3).ToList(); + Assert.Equal(3, diagnostic.Count); + Assert.All(diagnostic, timeout => Assert.InRange(timeout, TimeSpan.FromTicks(1), TimeSpan.FromMilliseconds(500))); + } + + [Fact] + public async Task Keeps_a_locator_with_no_filter_step_unlisted() + { + var counting = Counting(Answers((_base, Nodes(0)))); + + var failure = await Assert.ThrowsAsync(() => counting.Engine.ReadAsync(_base, CancellationToken.None)); + + Assert.Equal("LOCATOR_NOT_FOUND", failure.Code); + Assert.DoesNotContain("counted after the failure", failure.Message, StringComparison.Ordinal); + + // Only the failing resolve polled; no walk ran. + Assert.All(counting.Resolved, key => Assert.Equal(_base.Describe(), key)); + } + + [Fact] + public async Task Keeps_the_plain_failure_when_the_diagnostic_pass_cannot_read() + { + var counting = Counting(Answers((_chain, Nodes(0)), (_base, new EngineException(EngineErrorCodes.EngineFailure, "engine died", retryable: false)))); + + var failure = await Assert.ThrowsAsync(() => counting.Engine.ReadAsync(_chain, CancellationToken.None)); + + Assert.Equal("LOCATOR_NOT_FOUND", failure.Code); + Assert.DoesNotContain("counted after the failure", failure.Message, StringComparison.Ordinal); + + // The walk stopped at the prefix that refused to read; only the full chain polled. + Assert.Equal([_base.Describe()], counting.Resolved.Where(key => key != _chain.Describe())); + } + + [Fact] + public async Task Lists_the_step_counts_on_an_ambiguous_filtered_locator() + { + var counting = Counting(Answers((_chain, Nodes(2)), (_base, Nodes(6)), (_texted, Nodes(3)))); + + var failure = await Assert.ThrowsAsync(() => counting.Engine.ReadAsync(_chain, CancellationToken.None)); + + Assert.Equal("LOCATOR_AMBIGUOUS", failure.Code); + Assert.Equal(2, failure.Details?.Matches); + Assert.Contains($"{_base.Describe()}: 6", failure.Message, StringComparison.Ordinal); + Assert.Equal([_chain.Describe(), _base.Describe(), _texted.Describe(), _chain.Describe()], counting.Resolved); + } + + [Fact] + public async Task Counts_an_index_step_between_filters_under_its_own_label() + { + var indexed = new Locator(_screen, _texted).First().Query; + var chained = new Locator(_screen, indexed).Filter("1042").Query; + var counting = Counting(Answers((chained, Nodes(0)), (_base, Nodes(2)), (_texted, Nodes(1)), (indexed, Nodes(1)))); + + var failure = await Assert.ThrowsAsync(() => counting.Engine.ReadAsync(chained, CancellationToken.None)); + + Assert.Contains($"{_base.Describe()}: 2", failure.Message, StringComparison.Ordinal); + Assert.Contains($"{_texted.Describe()}: 1", failure.Message, StringComparison.Ordinal); + Assert.Contains($"{indexed.Describe()}: 1", failure.Message, StringComparison.Ordinal); + Assert.Contains($"{chained.Describe()}: 0", failure.Message, StringComparison.Ordinal); + } + + /// + /// A locate the way mobile's behaves: it ignores its budget and every call runs to completion (here by moving a fake clock 300 ms). The walk + /// shares one deadline, so a three-step chain reads two prefixes and then says it stopped, rather than starting a third full read the budget + /// cannot cover. + /// + [Fact] + public async Task Bounds_the_whole_walk_by_one_shared_deadline_when_locate_ignores_the_operation_timeout() + { + var clock = new FakeTimeProvider(); + List resolved = []; + List timeouts = []; + var engine = Engine( + (expression, budget, _) => + { + resolved.Add(expression.Describe()); + timeouts.Add(budget); + clock.Advance(TimeSpan.FromMilliseconds(300)); + return Task.FromResult>([]); + }, + () => CancellationToken.None, + clock); + + var failure = await Assert.ThrowsAsync(() => engine.ReadAsync(_chain, CancellationToken.None)); + + Assert.Equal("LOCATOR_NOT_FOUND", failure.Code); + + // The failing resolve polls once, then the walk resolves two of the three prefixes inside the shared budget and refuses the third. + Assert.Equal([_chain.Describe(), _base.Describe(), _texted.Describe()], resolved); + Assert.Contains("walk stopped after 2 of 3 steps", failure.Message, StringComparison.Ordinal); + Assert.All(timeouts.Skip(1), timeout => Assert.InRange(timeout, TimeSpan.FromTicks(1), TimeSpan.FromMilliseconds(500))); + } + + [Fact] + public async Task Surfaces_cancellation_when_the_signal_aborts_mid_walk() + { + using var cancellation = new CancellationTokenSource(); + var engine = Engine( + (expression, _, _) => + { + // Abort during the walk's second prefix, as a mobile session would when the attempt's signal fires between two reads. + if (expression.Describe() == _texted.Describe()) + { + cancellation.Cancel(); + throw new InvalidOperationException("boom"); + } + + return Task.FromResult>([]); + }, + () => cancellation.Token, + TimeProvider.System); + + await Assert.ThrowsAnyAsync(() => engine.ReadAsync(_chain, cancellation.Token)); + } + + [Fact] + public async Task Drops_the_counts_when_a_later_prefix_cannot_be_read() + { + var counting = Counting(Answers((_chain, Nodes(0)), (_base, Nodes(1)), (_texted, new EngineException(EngineErrorCodes.EngineFailure, "engine died", retryable: false)))); + + var failure = await Assert.ThrowsAsync(() => counting.Engine.ReadAsync(_chain, CancellationToken.None)); + + Assert.Equal("LOCATOR_NOT_FOUND", failure.Code); + Assert.DoesNotContain("counted after the failure", failure.Message, StringComparison.Ordinal); + + // The walk read the base, then the next prefix refused: a partial list must not present itself as the per-step counts of a completed walk. + Assert.Equal([_base.Describe(), _texted.Describe()], counting.Resolved.Where(key => key != _chain.Describe())); + } + + private static IReadOnlyList Nodes(int count) => [.. Enumerable.Repeat(Ref, count)]; + + private static Dictionary Answers(params (LocatorExpression Expression, object Answer)[] entries) => + entries.ToDictionary(entry => entry.Expression.Describe(), entry => entry.Answer); + + private static E2E.LocatorEngine Engine(Func>> locate, Func cancellation, TimeProvider clock) => + new(new LocatorEngineOptions + { + Observe = _ => Task.FromResult(new Observation { Route = "/", Roots = [] }), + Locate = locate, + Perform = (_, _, _, _) => throw new InvalidOperationException("unexpected perform"), + Cancellation = cancellation, + ActionTimeout = TimeSpan.FromMilliseconds(20), + AssertionTimeout = TimeSpan.FromMilliseconds(1_000), + Clock = clock, + }); + + /// An engine over , keyed by the described locator: nodes to answer, or an exception to throw. + private static CountingEngine Counting(Dictionary answers) + { + List resolved = []; + List timeouts = []; + var engine = Engine( + (expression, budget, _) => + { + var key = expression.Describe(); + resolved.Add(key); + timeouts.Add(budget); + return answers.TryGetValue(key, out var answer) + ? answer is Exception failure ? Task.FromException>(failure) : Task.FromResult((IReadOnlyList)answer) + : throw new InvalidOperationException("unexpected locate: " + key); + }, + () => CancellationToken.None, + TimeProvider.System); + return new CountingEngine(engine, resolved, timeouts); + } + + private sealed record CountingEngine(E2E.LocatorEngine Engine, List Resolved, List Timeouts); +} From 0253bcfa5bd5c4487176c8bf84283e6c53b40b17 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Dar=C3=ADo=20Kondratiuk?= Date: Tue, 3 Nov 2026 19:51:49 -0300 Subject: [PATCH 2/2] Loosen the budget assertion and say a filter step is needed for the list Co-Authored-By: Claude Sonnet 5.5 Claude-Session: https://claude.ai/code/session_01Ls2e1yaZFgZ3FEJkYScfkK --- docs-site/src/content/docs/docs/index.mdx | 2 +- .../LocatorEngine/LocatorFailureCandidateCountsTests.cs | 2 +- 2 files changed, 2 insertions(+), 2 deletions(-) diff --git a/docs-site/src/content/docs/docs/index.mdx b/docs-site/src/content/docs/docs/index.mdx index c036984fa..52c5e5915 100644 --- a/docs-site/src/content/docs/docs/index.mdx +++ b/docs-site/src/content/docs/docs/index.mdx @@ -125,6 +125,6 @@ await Screen.GetByTestId("feed").ScrollUntilVisibleAsync(row, new ScrollUntilVis A locator reports an engine failure in the test's own codes, with the engine's error as `InnerException`: `NOT_ACTIONABLE`, `ACTION_MAY_HAVE_COMMITTED`, and `OPERATION_TIMEOUT` are `ACTION_FAILED`, a stale node or a missing frame is `LOCATOR_NOT_FOUND`, an ambiguous frame is `LOCATOR_AMBIGUOUS`, and `INVALID_STATE` is `APP_NOT_OPEN`. `CANCELLED` and `ENGINE_FAILURE` keep their codes (category `infrastructure`), and `UNSUPPORTED_CAPABILITY` is a `ConfigurationException`. A `locator.WaitForAsync` that runs out fails with `LOCATOR_NOT_FOUND` (`locator did not become : `), not `TIMEOUT`, and a pass verifies the acts before it. A stale node found by an action is looked up again, within the action timeout, before the action fails. -When a filtered locator (`Filter(...)`, `First()`, `Nth(n)`) matches nothing or too many, the `LOCATOR_NOT_FOUND` or `LOCATOR_AMBIGUOUS` message ends with `candidates left after each filter step, counted after the failure:` and one line per step, innermost first, with the locator so far and how many nodes it left. A step that narrowed too much or too little names itself. The counts come from one extra read per step after the failure, so a page that changed since can disagree with the failing attempt. All the reads share one 500 ms budget: a chain that cannot finish says `walk stopped after N of M steps`, and a read that fails leaves the list out. A locator without a filter step gets no list. +When a filtered locator (a chain with at least one `Filter(...)`, and any `First()` or `Nth(n)` after it) matches nothing or too many, the `LOCATOR_NOT_FOUND` or `LOCATOR_AMBIGUOUS` message ends with `candidates left after each filter step, counted after the failure:` and one line per step, innermost first, with the locator so far and how many nodes it left. A step that narrowed too much or too little names itself. The counts come from one extra read per step after the failure, so a page that changed since can disagree with the failing attempt. All the reads share one 500 ms budget: a chain that cannot finish says `walk stopped after N of M steps`, and a read that fails leaves the list out. A locator without a filter step gets no list. A read or navigation that times out fails with `OPERATION_TIMEOUT`. It is not retried, even when the call log mentions a detached element or a navigation race. On a device, an input (a tap, key, swipe, or `Back`) that the device does not answer within `actionTimeout` fails with `ACTION_MAY_HAVE_COMMITTED` and is never repeated, because it may have landed; an input the timeout ran out before is not sent (`OPERATION_TIMEOUT`), and a long press that holds at least as long as the time left is refused (`NOT_ACTIONABLE`). A locate, an observation, or a screenshot on a device runs within the timeout of the call that takes it, so a device that stops answering fails the step with `ACTION_FAILED` instead of holding it until the test timeout. `WaitForAsync`, `Expect`, and `ToHaveScreenshotAsync` give their last read extra time, up to their slowest read so far and within the test timeout, so a screen that shows up during it still passes. See `EngineErrorCodes` in the [API reference](/api/). diff --git a/tests/E2E.Tests/LocatorEngine/LocatorFailureCandidateCountsTests.cs b/tests/E2E.Tests/LocatorEngine/LocatorFailureCandidateCountsTests.cs index f0e6f44c2..dcc6c3799 100644 --- a/tests/E2E.Tests/LocatorEngine/LocatorFailureCandidateCountsTests.cs +++ b/tests/E2E.Tests/LocatorEngine/LocatorFailureCandidateCountsTests.cs @@ -49,7 +49,7 @@ public async Task Lists_how_many_candidates_each_filter_step_left_counted_after_ Assert.Equal([_base.Describe(), _texted.Describe(), _chain.Describe()], counting.Resolved.TakeLast(3)); // The failing resolve polled on the action deadline; the diagnostic resolves arrived after it, each bounded by the 500 ms diagnostic budget. - Assert.InRange(counting.Timeouts[0], TimeSpan.FromMilliseconds(19), TimeSpan.FromMilliseconds(20)); + Assert.InRange(counting.Timeouts[0], TimeSpan.FromTicks(1), TimeSpan.FromMilliseconds(20)); var diagnostic = counting.Timeouts.TakeLast(3).ToList(); Assert.Equal(3, diagnostic.Count); Assert.All(diagnostic, timeout => Assert.InRange(timeout, TimeSpan.FromTicks(1), TimeSpan.FromMilliseconds(500)));