Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
3 changes: 2 additions & 1 deletion COMPATIBILITY.md
Original file line number Diff line number Diff line change
Expand Up @@ -45,7 +45,8 @@ Names are C# versions of the JavaScript API: `agent.act` is `ActAsync`, `screen.
- `toHaveScreenshot` as `Expect.That(Screen).ToHaveScreenshotAsync(name, options)` and the same on a locator, with `Not` and `Expect.Soft(locator)`; see [Visual comparison](#visual-comparison)
- `expect.soft` for every locator expectation, as `Expect.Soft(locator)`, including `.Not` and the per-matcher `timeout`. An `ASSERTION_FAILED` is kept and the body runs on; any other error still throws. `E2ETest` records each one on the NUnit result, as inside `Assert.EnterMultipleScope`, so the test fails when the body ends and lists every failure. `E2E.XUnit.V3.E2ETest` keeps them on the session and fails the test when it is disposed. As upstream, a failed soft assertion outranks a skip: `Assert.Ignore` or `Assert.Inconclusive` after one fails the test in NUnit, and the message keeps the skip reason beside the failures. In xUnit v3, `Assert.Skip` after one fails the test the same way. Other hosts set `E2ESessionOptions.OnSoftFailure` or call `E2ESession.CloseSoftFailures`
- `expect(browser)` as `Expect.That(Browser)`: `ToHaveURLAsync` (a string or a `Regex`, with `ignoreCase` and `timeout`), `ToHaveTitleAsync` (a `TextMatch`, with `timeout`), and `ToHaveClassAsync(locator, TextMatch)`, `.Not`, and `Expect.Soft(Browser)`. Like the locator matchers, a passing one verifies the previous `act`, and `.Not` passes after 1000 ms of the opposite. A read that outlasts the matcher's deadline ends it, so a page that stops answering fails at the assertion timeout
- `expect.poll` as `Expect.Poll(read, options)`, with `Timeout`, `Interval`, and `Message`. A read that throws is retried. The matchers are `ToBeAsync`, `ToSatisfyAsync`, and `.Not`; `E2E.NUnit` adds `ToMatchAsync` for any NUnit constraint, such as `Is.GreaterThan(3)`. The default timeout is 5 seconds, not the configured assertion timeout, because the poll does not see the running test, and it does not stop at the test deadline unless a cancellation token is passed
- `expect.poll` as `Expect.Poll(read, options)`, with `Timeout`, `Interval`, and `Message`. A read that throws is retried. The matchers are `ToBeAsync`, `ToSatisfyAsync`, and `.Not`; `E2E.NUnit` adds `ToMatchAsync` for any NUnit constraint, such as `Is.GreaterThan(3)`. The default timeout is 5 seconds, not the configured assertion timeout, because the poll does not see the running test, and it does not stop at the test deadline unless a cancellation token is passed (a `Timeout` of zero waits for that token)
- A call's `Timeout` of `TimeSpan.Zero` sets no deadline of its own, as `timeout: 0` does upstream (`feat(e2e): timeout 0 means no step deadline`, `d2953bb`, tester-army/e2e#1554). Locator actions, `WaitForAsync`, locator and browser matchers, `ToHaveScreenshotAsync`, `Screen.TreeAsync` and `DebugAsync`, `Browser.GotoAsync`, `WaitForResponseAsync`, `WaitForDownloadAsync`, and `WaitForDownloadsAsync` wait until the test deadline instead of failing at once. A negative `TimeSpan` is `TestException` `INVALID_ARGUMENT` on every one of them (it was `ArgumentOutOfRangeException` on some); an infinite `TimeSpan` is negative, so it is refused too. `Time.CheckStepTimeout`, `StepDeadline`, and `StepBudget` (`Internal/Time.cs`, .NET-only; upstream's `checkStepTimeout` and `stepTimeout`) hold the rule. Upstream hands an engine author `stepTimeout` and builds the operation from it; the port hands an engine a `TimeSpan` budget, so `StepBudget` gives it what is left of the test deadline (at most a day, and a day when nothing bounds the test). `Expect.Poll` has no attempt: with `Timeout = TimeSpan.Zero` it polls until the cancellation token ends, and with a token that cannot end it throws `ConfigurationException` `INVALID_CONFIG` (upstream: outside a test). Not ported: the contributed-fixture `FixtureOperation.timeout` rule and `withTimeout` taking an infinite budget, because the port has neither.
- `agent.act`, `agent.assert`, `agent.waitFor`, `agent.extract`. `ActOptions` has `Params`, `Timeout`, `MaxSteps`, `MaxModelCalls`, and `Agent`; `AssertOptions` has `Params`, `Timeout`, and `Agent`; `WaitForOptions` has `Params`, `Timeout`, `Interval`, `MaxModelCalls`, and `Agent`; `ExtractOptions` has `Timeout` and `Agent`. See [Agents](#agents)
- Act tools `observe`, `scroll`, `scroll_to`, and `back`. They are offered when the engine declares `EngineCapabilities.Scroll` or `EngineCapabilities.History`. `scroll_to` with a target scrolls it into view. With text, it pages the viewport, or the target list, until a node reading the text is listed, then scrolls it into view. It stops when the screen stops moving
- Replay cache for a verified `act`: role, name, test id, and path. Modes are `self-finalized`, `agent-concluded`, and `missed`
Expand Down
2 changes: 1 addition & 1 deletion docs-site/src/content/docs/api/E2E/ActionOptions.md
Original file line number Diff line number Diff line change
Expand Up @@ -40,6 +40,6 @@ public TimeSpan? Timeout { get; init; }
```


How long to wait for exactly one enabled match. Defaults to the configured action timeout.
How long to wait for exactly one enabled match. Defaults to the configured action timeout. Zero sets no deadline of its own: the test timeout bounds it. A negative value is `INVALID_ARGUMENT`.

**Value:** [TimeSpan?](https://learn.microsoft.com/dotnet/api/system.timespan)
8 changes: 4 additions & 4 deletions docs-site/src/content/docs/api/E2E/Browser.md
Original file line number Diff line number Diff line change
Expand Up @@ -664,7 +664,7 @@ public Task<BrowserDownload> WaitForDownloadAsync(Func<Task> trigger, TimeSpan?


Runs &lt;code class="paramref"&gt;trigger&lt;/code&gt; and waits for the download it starts, within &lt;code class="paramref"&gt;timeout&lt;/code&gt; (the action timeout by
default). The file is kept under the attempt's `downloads` artifact folder, as an artifact of this step. A text download
default; zero waits for the test timeout). The file is kept under the attempt's `downloads` artifact folder, as an artifact of this step. A text download
(`text/*` or `application/json`) is rewritten in place through the session's secrets when it holds any, and the report
labels it `complete`; any other download stays as the app served it, `incomplete`. A trigger that
throws fails the call with its own error, and one that starts no download fails with `ACTION_FAILED`, naming the wait and
Expand Down Expand Up @@ -692,7 +692,7 @@ public Task<IReadOnlyList<BrowserDownload>> WaitForDownloadsAsync(Func<Task> tri


Runs &lt;code class="paramref"&gt;trigger&lt;/code&gt; once and waits for the first &lt;code class="paramref"&gt;count&lt;/code&gt; downloads it starts, within
&lt;code class="paramref"&gt;timeout&lt;/code&gt; (the action timeout by default), and returns them in start order. Each file is kept under the attempt's
&lt;code class="paramref"&gt;timeout&lt;/code&gt; (the action timeout by default; zero waits for the test timeout), and returns them in start order. Each file is kept under the attempt's
`downloads` artifact folder as its own artifact of this step, and a text download is rewritten through the session's secrets
as `WaitForDownloadAsync` does. One timeout bounds the start, completion and
save of every file. When fewer start in time the call fails with `ACTION_FAILED` naming both counts; extra downloads are
Expand Down Expand Up @@ -724,7 +724,7 @@ public Task<WebResponse> WaitForResponseAsync(string pattern, TimeSpan? timeout
Waits for a response whose complete URL matches the glob &lt;code class="paramref"&gt;pattern&lt;/code&gt;:
`*` matches within one path segment, `**` crosses `/`, `?`
matches one character, and `\` escapes the next one. The timeout bounds
the match only, and defaults to the action timeout.
the match only, and defaults to the action timeout; zero waits for the test timeout.

**Parameters**

Expand All @@ -746,7 +746,7 @@ public Task<WebResponse> WaitForResponseAsync(Regex pattern, TimeSpan? timeout =
```


Waits for a response whose URL matches &lt;code class="paramref"&gt;pattern&lt;/code&gt;. The timeout bounds the match only, and defaults to the action timeout.
Waits for a response whose URL matches &lt;code class="paramref"&gt;pattern&lt;/code&gt;. The timeout bounds the match only, and defaults to the action timeout; zero waits for the test timeout.

**Parameters**

Expand Down
2 changes: 1 addition & 1 deletion docs-site/src/content/docs/api/E2E/GotoOptions.md
Original file line number Diff line number Diff line change
Expand Up @@ -40,7 +40,7 @@ public TimeSpan? Timeout { get; init; }
```


The navigation budget. Defaults to the action timeout.
The navigation budget. Defaults to the action timeout. Zero sets no deadline of its own: the test timeout bounds it. A negative value is `INVALID_ARGUMENT`.

**Value:** [TimeSpan?](https://learn.microsoft.com/dotnet/api/system.timespan)

Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -51,6 +51,6 @@ public TimeSpan? Timeout { get; init; }
```


How long to wait. Defaults to the configured action timeout.
How long to wait. Defaults to the configured action timeout. Zero sets no deadline of its own: the test timeout bounds it. A negative value is `INVALID_ARGUMENT`.

**Value:** [TimeSpan?](https://learn.microsoft.com/dotnet/api/system.timespan)
3 changes: 2 additions & 1 deletion docs-site/src/content/docs/api/E2E/PollOptions.md
Original file line number Diff line number Diff line change
Expand Up @@ -68,6 +68,7 @@ public TimeSpan? Timeout { get; init; }
```


Deadline for the matcher to hold. The default is 5 seconds.
Deadline for the matcher to hold. The default is 5 seconds. Zero sets no deadline: the poll runs until the cancellation token it is
given ends, so it needs a token that can end (`INVALID_CONFIG` otherwise).

**Value:** [TimeSpan?](https://learn.microsoft.com/dotnet/api/system.timespan)
2 changes: 1 addition & 1 deletion docs-site/src/content/docs/api/E2E/ScreenshotOptions.md
Original file line number Diff line number Diff line change
Expand Up @@ -113,6 +113,6 @@ public TimeSpan? Timeout { get; init; }
```


Assertion budget. Defaults to the assertion timeout.
Assertion budget. Defaults to the assertion timeout. Zero sets no deadline of its own: the test timeout bounds it. A negative value is `INVALID_ARGUMENT`.

**Value:** [TimeSpan?](https://learn.microsoft.com/dotnet/api/system.timespan)
2 changes: 1 addition & 1 deletion docs-site/src/content/docs/features/assertions.md
Original file line number Diff line number Diff line change
Expand Up @@ -9,7 +9,7 @@ sidebar:

- `Expect.Soft(locator)` and `Expect.Soft(screen)` record each failure and let the test go on. On NUnit the failures land on the test result, as inside `Assert.EnterMultipleScope`, and the test fails when it ends with every failure listed. On xUnit v3 they stay on the session, and the test fails when it is disposed, with every failure in one message.
- `Expect.Soft(Browser)` keeps the failures of `ToHaveURLAsync`, `ToHaveTitleAsync`, and `ToHaveClassAsync` the same way.
- `Expect.Poll(read)` retries a value. `ToBeAsync` and `ToSatisfyAsync(predicate)` work on both adapters. `E2E.NUnit` also adds `ToMatchAsync` with any NUnit constraint: `await Expect.Poll(() => ReadCountAsync()).ToMatchAsync(Is.GreaterThan(3));`. A read that throws, or a matcher that refuses the value's type, is one failing sample, not the end of the poll.
- `Expect.Poll(read)` retries a value. `ToBeAsync` and `ToSatisfyAsync(predicate)` work on both adapters. `E2E.NUnit` also adds `ToMatchAsync` with any NUnit constraint: `await Expect.Poll(() => ReadCountAsync()).ToMatchAsync(Is.GreaterThan(3));`. A read that throws, or a matcher that refuses the value's type, is one failing sample, not the end of the poll. A `Timeout` of `TimeSpan.Zero` sets no deadline: the poll runs until the cancellation token you pass ends.

## Evidence screenshots

Expand Down
2 changes: 1 addition & 1 deletion docs-site/src/content/docs/migrate/playwright.mdx
Original file line number Diff line number Diff line change
Expand Up @@ -252,7 +252,7 @@ The agent does the checkout steps and adapts when the form changes. A test can a
| `Page.UnrouteAsync()` | `Browser.UnrouteAsync(pattern)` | web only |
| `Page.UnrouteAllAsync()` | `Browser.UnrouteAllAsync()`; configured site headers stay | web only |
| `Page.WaitForResponseAsync(pattern)`, `RunAndWaitForResponseAsync` | `Browser.WaitForResponseAsync(pattern)`: a string or `Regex`, no predicate; `Url`, `Status`, `Headers`, `BodyAsync()`, `JsonAsync<T>()`, `TextAsync()`. A string is the glob of `Browser.RouteAsync`, tested against the whole URL, query included: Playwright's `"/api/orders"` is the full URL or `"**/api/orders"`, and `"\\?"` is a literal `?`. Start it before the step that sends the request, and await it after | web only |
| a `Timeout` of `0` for no timeout | a positive `TimeSpan`. `TimeSpan.Zero` on `Browser.GotoAsync`, `WaitForDownloadAsync`, or `WaitForDownloadsAsync` throws `ArgumentOutOfRangeException`. A locator action takes `Timeout.InfiniteTimeSpan` for no timeout | missing |
| a `Timeout` of `0` for no timeout | the same: `TimeSpan.Zero` sets no deadline of its own on a locator action, `WaitForAsync`, a matcher, `ToHaveScreenshotAsync`, `Browser.GotoAsync`, `WaitForResponseAsync`, `WaitForDownloadAsync`, and `WaitForDownloadsAsync`; the test timeout still bounds it. A negative `TimeSpan` throws `TestException` with `INVALID_ARGUMENT`. `Expect.Poll` with `Timeout = TimeSpan.Zero` polls until the cancellation token you pass ends, and without one it throws `ConfigurationException` with `INVALID_CONFIG` | same |
| `response.Ok`, `response.Request` | `Status` | missing |
| `Page.WaitForRequestAsync()`, `request.PostDataJSON()` | none yet for `WaitForRequestAsync`; `route.Request.PostData` is a string | missing |
| `RouteFromHARAsync()`, `RouteWebSocketAsync()` | not yet | missing |
Expand Down
6 changes: 6 additions & 0 deletions skills/e2e-dotnet/SKILL.md
Original file line number Diff line number Diff line change
Expand Up @@ -226,6 +226,12 @@ lines you changed:
step, so keep it longer than the slowest screen takes to load. A call's own
timeout still wins. An `Expect` already gives up after 5 s by default
(`assertionTimeout`).
- A call's `Timeout` of `TimeSpan.Zero` sets no deadline of its own: the call
waits until the test timeout instead of failing at once. It works on locator
actions, `WaitForAsync`, `Expect` matchers, `ToHaveScreenshotAsync`,
`Browser.GotoAsync`, `WaitForResponseAsync`, and `WaitForDownloadAsync`. A
negative `TimeSpan` fails with `INVALID_ARGUMENT`. `Expect.Poll` with a zero
`Timeout` polls until the cancellation token you pass ends.
- Keep the replay cache on. A failed run keeps the recording of every goal
whose check passed, and the next run replays those with no model call. New,
changed, and failed goals run live, and so does a replay the trace page
Expand Down
9 changes: 2 additions & 7 deletions src/E2E/Expect/BrowserExpect.cs
Original file line number Diff line number Diff line change
Expand Up @@ -161,14 +161,9 @@ private Task PollAsync(string matcher, string describeExpected, TimeSpan? timeou

private async Task PollCoreAsync(string matcher, string describeExpected, TimeSpan? timeout, Func<CancellationToken, Task<(bool? Holds, string Observed)>> read, CancellationToken cancellationToken)
{
if (timeout is { } given)
{
ArgumentOutOfRangeException.ThrowIfLessThan(given, TimeSpan.Zero, nameof(timeout));
}

var token = _browser.Token(cancellationToken);
var clock = TimeProvider.System;
var deadline = new Deadline(timeout ?? _browser.AssertionTimeout, clock);
var deadline = Time.StepDeadline(timeout, "expect." + (_negated ? "not." : "") + matcher, _browser.AssertionTimeout, _browser.BudgetDeadline, clock);
string? observed = null;
try
{
Expand All @@ -182,7 +177,7 @@ await Time.PollConditionAsync(new PollConditionOptions
{
var left = deadline.Remaining(clock.GetUtcNow());
using var bounded = CancellationTokenSource.CreateLinkedTokenSource(token);
bounded.CancelAfter(left > Time.PollInterval ? left : Time.PollInterval);
bounded.CancelAfter(left > Time.Unbounded ? Time.Unbounded : left > Time.PollInterval ? left : Time.PollInterval);
try
{
var sample = await read(bounded.Token).ConfigureAwait(false);
Expand Down
7 changes: 1 addition & 6 deletions src/E2E/Expect/Expect.cs
Original file line number Diff line number Diff line change
Expand Up @@ -405,19 +405,14 @@ private async Task PollCoreAsync(
MissingFrame missingFrame,
CancellationToken cancellationToken)
{
if (timeout is { } budget)
{
ArgumentOutOfRangeException.ThrowIfLessThan(budget, TimeSpan.Zero, nameof(timeout));
}

var token = _locator.Screen.Token(cancellationToken);
var clock = _locator.Screen.Clock;
Verdict last = default;
IReadOnlyList<SemanticNode> lastMatches = [];
var samples = new SampleHistory("expect", _locator.Screen.Redact);
try
{
var deadline = new Deadline(timeout ?? _locator.Screen.AssertionTimeout, clock);
var deadline = _locator.Screen.Engine.AssertionDeadlineFor(timeout, "expect." + (_negated ? "not." : "") + matcher);
await samples.RecordAsync(_locator.Screen.Steps, token, () => Time.PollReadsAsync(new PollReadsOptions
{
Deadline = deadline,
Expand Down
30 changes: 27 additions & 3 deletions src/E2E/Expect/Poll.cs
Original file line number Diff line number Diff line change
Expand Up @@ -3,13 +3,17 @@
// SPDX-License-Identifier: Apache-2.0

using System.Globalization;
using E2E.Internal;

namespace E2E;

/// <summary>How <see cref="Expect.Poll{T}(Func{CancellationToken, Task{T}}, PollOptions?)"/> re-reads a value.</summary>
public sealed class PollOptions
{
/// <summary>Deadline for the matcher to hold. The default is 5 seconds.</summary>
/// <summary>
/// Deadline for the matcher to hold. The default is 5 seconds. Zero sets no deadline: the poll runs until the cancellation token it is
/// given ends, so it needs a token that can end (<c>INVALID_CONFIG</c> otherwise).
/// </summary>
public TimeSpan? Timeout { get; init; }

/// <summary>Pause between reads. The default is 100 milliseconds.</summary>
Expand Down Expand Up @@ -74,7 +78,7 @@ private async Task PollAsync(string matcher, Func<T, bool> check, Func<T, string
var interval = _options.Interval ?? DefaultInterval;
var clock = _options.Clock;
var started = clock.GetUtcNow();
var deadline = started + timeout;
var deadline = PollDeadline(timeout, started, cancellationToken);
string? last = null;
while (true)
{
Expand All @@ -88,7 +92,7 @@ private async Task PollAsync(string matcher, Func<T, bool> check, Func<T, string
T value;
try
{
value = await _read(cancellationToken).WaitAsync(remaining, clock, cancellationToken).ConfigureAwait(false);
value = await _read(cancellationToken).WaitAsync(remaining > Time.Unbounded ? Time.Unbounded : remaining, clock, cancellationToken).ConfigureAwait(false);
}
catch (OperationCanceledException) when (cancellationToken.IsCancellationRequested)
{
Expand Down Expand Up @@ -142,6 +146,26 @@ private async Task PollAsync(string matcher, Func<T, bool> check, Func<T, string
throw new TestException("ASSERTION_FAILED", ErrorRedaction.Redact(string.Join('\n', lines)));
}

/// <summary>
/// The poll's own deadline. A <c>Timeout</c> of zero sets none, as upstream's <c>timeout: 0</c> waits for the phase's, and the port's
/// phase is the caller's <paramref name="cancellationToken"/>: it needs one that can end, or nothing would bound the poll
/// (upstream's <c>INVALID_CONFIG</c> outside a test).
/// </summary>
private static DateTimeOffset PollDeadline(TimeSpan timeout, DateTimeOffset started, CancellationToken cancellationToken)
{
if (timeout != TimeSpan.Zero)
{
return started + timeout;
}

if (!cancellationToken.CanBeCanceled)
{
throw new ConfigurationException("INVALID_CONFIG", "expect.poll timeout 0 waits for the test deadline, so it needs a cancellation token that ends with the test; pass a positive timeout or the test's token");
}

return Time.Never;
}

private static Task PauseAsync(DateTimeOffset deadline, TimeSpan interval, TimeProvider clock, CancellationToken cancellationToken)
{
var remaining = deadline - clock.GetUtcNow();
Expand Down
2 changes: 1 addition & 1 deletion src/E2E/Expect/ScreenshotAssertion.cs
Original file line number Diff line number Diff line change
Expand Up @@ -54,7 +54,7 @@ private static async Task ExpectScreenshotCoreAsync(Screen screen, Locator? subj
var file = Path.Combine(directory, call.Stem + store.Suffix + ".png");
var shown = Shown(store, file);
var token = screen.Token(cancellationToken);
var deadline = new Deadline(call.Timeout ?? screen.AssertionTimeout, screen.Clock);
var deadline = screen.Engine.AssertionDeadlineFor(call.Timeout, api + " option");

var reads = screen.Reads(deadline);
var capturer = new Capturer(screen, store, api, subject, call, reads);
Expand Down
Loading
Loading