Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 1 addition & 0 deletions COMPATIBILITY.md
Original file line number Diff line number Diff line change
Expand Up @@ -162,6 +162,7 @@ Differences:
- Upstream's docs site pages that have no counterpart in the port: the page objects page (`docs/page-objects.mdx`, moved to `/page-object-model` in `d30114a`), `e2e feedback` and the `docs/reference/cli.mdx` text about its opt-out variables (`4c1e2e8`), and `examples/` with its registry (`b3ae3a4`, `38b802d`, `a29edcb`, `b54646f`). The migration guides (`docs/migrate/*.mdx`, `8537abc`) are ported: see [Migration guides](#migration-guides). The docs site has .NET-only guides for upstream's five example apps of `caf51eb` (tester-army/e2e#1497: i18n, file upload, location, biometrics, deep links), under [Examples](docs-site/src/content/docs/examples/index.mdx). They are not ports: that commit only adds registry entries that link to separate repositories, so there was no guide text. Each page writes C# tests for the upstream app. The self-healing deep dive (`6fa99ea`) is ported as the [Self-healing tests](README.md#self-healing-tests) section of the guide, since the port has no docs pages for the cache, the trace page, or executors to link to. Known gap: only `e2e explore` prints a run summary, since it is the one command that feeds a run to `ListReporter` (see [List reporter](#list-reporter)): the `Cache 4 replayed · 1 handed off · 1 missed` row is `Format.CacheText` over `Format.StepsCacheTally` (`E2E.Cli`, upstream's `cacheText` and `stepsCacheTally` of `report/format.ts`), which an exploration, with its cache off, never fills, and a test run does not print ([#799](https://github.com/hardkoded/e2e-dotnet/issues/799))
- Parallel workers and serial suites
- `test.fail` and the `fail` test option (tester-army/e2e#1539, `732ea8a`): a test expected to fail, whose product failure passes and whose pass fails with `UNEXPECTED_PASS`. It lives in upstream's test registry, scheduler, retry, and reporters (`settleExpectedFailure` runs on a finished attempt), and the port has none of them. NUnit and xUnit v3 own the verdict and the retries, and neither has an expected-failure outcome, so a port would need a new public API of its own
- Playwright's page tests in the testbed suite (tester-army/e2e#1505, `bdf769b`) are ported in slices (hardkoded/e2e-dotnet#1060), as the `playwright` category of `tests/E2E.NUnit.Tests`, against the playground's `/content` route and Playwright's test assets (Apache-2.0, `Testbed/playwright/`, with Playwright's `LICENSE` and `NOTICE`). A test upstream marks `{ fail: 'reason' }` keeps its assertions inside the test-only `Helpers.ExpectedToFailAsync`, which passes while the body fails and fails once it passes. Ported so far: `check`, `wait-for-url`, `locator-list`, and `selectors-text` (one test is such a known difference, hardkoded/e2e-dotnet#1062).
- Serial groups (upstream tester-army/e2e#1056, #1470): serial runs, group retries, and skipping the rest of a group after a failure belong to the test runner, which is NUnit or xUnit here, so the port keeps no serial groups. The schema has `serialAttempt.cutRetry` and the serial verdict is read from it, but nothing in the port writes a serial attempt: see [Report document](#report-document)
- The out-of-process engine protocol (upstream tester-army/e2e#1232, [#873](https://github.com/hardkoded/e2e-dotnet/issues/873)): the port has no runner and no worker pool, so there is no engine process to talk to. An engine is written in .NET against `IEngine`
- Diff-only observations for a custom executor's own model turns. The built-in agent has them (#1022): `#id` lines on the opening screen and on `observe`, and a diff after each action (see [MCP server](#mcp-server) for the presenter). `ExecutorObservation.Text`, which an executor reads from `Observe`, keeps the full `[ref=…]` snapshot, cut at the observation byte budget (see [Agents](#agents)), and a node is addressed by its ref there (`new ExecutorTarget("e12")`), so a custom executor has no presenter of its own. Upstream gives every executor the `#id` view.
Expand Down
72 changes: 72 additions & 0 deletions tests/E2E.Agent.Tests/PlaywrightRoutes.cs
Original file line number Diff line number Diff line change
@@ -0,0 +1,72 @@
// Copyright 2026 TesterArmy.
// Modified by Dario Kondratiuk.
// SPDX-License-Identifier: Apache-2.0

using System.Net;
using System.Text;

namespace E2E.Playground;

/// <summary>
/// The routes of the tests ported from Playwright, a port of upstream's <c>apps/testbed/app/playwright/routes.mjs</c>:
/// its test assets (copied from microsoft/playwright <c>tests/assets</c> at v1.64.1, Apache-2.0, under
/// <c>Testbed/playwright/assets</c> next to the test assembly) and the dynamic routes its test server sets per test.
/// The playground serves them for any path it has no page for. This port has <c>/content</c> and the assets so far.
/// </summary>
internal static class PlaywrightRoutes
{
private static readonly string Root = Path.Combine(AppContext.BaseDirectory, "Testbed", "playwright", "assets");

private static readonly Dictionary<string, string> Types = new(StringComparer.Ordinal)
{
[".html"] = "text/html",
[".js"] = "text/javascript",
[".css"] = "text/css",
[".json"] = "application/json",
[".png"] = "image/png",
[".pdf"] = "application/pdf",
[".svg"] = "image/svg+xml",
[".txt"] = "text/plain",
};

/// <summary>Serves one request: a dynamic route, else an asset, else a 404.</summary>
public static async Task HandleAsync(HttpListenerContext context)
{
var response = context.Response;
var path = context.Request.Url!.AbsolutePath;
if (path == "/content")
{
var html = context.Request.QueryString["html"] ?? "";
response.ContentType = "text/html; charset=utf-8";
await WriteAsync(response, FromBase64Url(html)).ConfigureAwait(false);
return;
}

var file = Path.GetFullPath(Path.Combine(Root, Uri.UnescapeDataString(path).TrimStart('/')));
if (!file.StartsWith(Root + Path.DirectorySeparatorChar, StringComparison.Ordinal) || !File.Exists(file))
{
response.StatusCode = 404;
response.ContentType = "text/plain";
await WriteAsync(response, Encoding.UTF8.GetBytes("Not found")).ConfigureAwait(false);
return;
}

response.ContentType = Types.GetValueOrDefault(Path.GetExtension(file), "application/octet-stream");
response.AddHeader("Cache-Control", "no-store");
await WriteAsync(response, await File.ReadAllBytesAsync(file).ConfigureAwait(false)).ConfigureAwait(false);
}

private static byte[] FromBase64Url(string value)
{
var padded = value.Replace('-', '+').Replace('_', '/');
padded = padded.PadRight(padded.Length + ((4 - (padded.Length % 4)) % 4), '=');
var bytes = new byte[padded.Length];
return Convert.TryFromBase64String(padded, bytes, out var written) ? bytes[..written] : [];
}

private static async Task WriteAsync(HttpListenerResponse response, byte[] bytes)
{
response.ContentLength64 = bytes.Length;
await response.OutputStream.WriteAsync(bytes).ConfigureAwait(false);
}
}
2 changes: 1 addition & 1 deletion tests/E2E.Agent.Tests/Testbed.cs
Original file line number Diff line number Diff line change
Expand Up @@ -1014,7 +1014,7 @@ await WriteHtmlAsync(response, "Dashboard", """
}
else
{
response.StatusCode = 404;
await PlaywrightRoutes.HandleAsync(context).ConfigureAwait(false);
}

response.Close();
Expand Down
160 changes: 160 additions & 0 deletions tests/E2E.NUnit.Tests/Check/PageCheckTests.cs
Original file line number Diff line number Diff line change
@@ -0,0 +1,160 @@
// Copyright 2026 TesterArmy.
// Modified by Dario Kondratiuk.
// SPDX-License-Identifier: Apache-2.0

using static E2E.NUnit.Tests.Helpers;

namespace E2E.NUnit.Tests.Check;

/// <summary>
/// <c>test.describe('page check')</c> of upstream's <c>apps/testbed/tests/check.e2e.ts</c> (Playwright's <c>page-check.spec.ts</c>).
/// Not ported, as upstream: should check the label with position (no <c>position</c>), trial run should not check and trial run
/// should not uncheck (no <c>trial</c>), should check the box using setChecked (no <c>setChecked</c>).
/// </summary>
public sealed class PageCheckTests : PlaywrightTest
{
private static readonly string[] AriaRoles = ["checkbox", "menuitemcheckbox", "option", "radio", "switch", "menuitemradio", "treeitem"];

[Test]
public async Task Should_check_the_box()
{
await Browser.GotoAsync(Content("<input id='checkbox' type='checkbox'></input>"));
await Browser.Locator("input").CheckAsync();
Assert.That(await Browser.EvaluateAsync<bool>("() => window.checkbox.checked"), Is.True);
}

[Test]
public async Task Should_not_check_the_checked_box()
{
await Browser.GotoAsync(Content("<input id='checkbox' type='checkbox' checked></input>"));
await Browser.Locator("input").CheckAsync();
Assert.That(await Browser.EvaluateAsync<bool>("() => window.checkbox.checked"), Is.True);
}

[Test]
public async Task Should_uncheck_the_box()
{
await Browser.GotoAsync(Content("<input id='checkbox' type='checkbox' checked></input>"));
await Browser.Locator("input").UncheckAsync();
Assert.That(await Browser.EvaluateAsync<bool>("() => window.checkbox.checked"), Is.False);
}

[Test]
public async Task Should_not_uncheck_the_unchecked_box()
{
await Browser.GotoAsync(Content("<input id='checkbox' type='checkbox'></input>"));
await Browser.Locator("input").UncheckAsync();
Assert.That(await Browser.EvaluateAsync<bool>("() => window.checkbox.checked"), Is.False);
}

[Test]
public async Task Should_check_radio()
{
await Browser.GotoAsync(Content("""
<input type='radio'>one</input>
<input id='two' type='radio'>two</input>
<input type='radio'>three</input>
"""));
await Browser.Locator("#two").CheckAsync();
Assert.That(await Browser.EvaluateAsync<bool>("() => window.two.checked"), Is.True);
}

[Test]
public async Task Should_check_radio_by_aria_role()
{
await Browser.GotoAsync(Content("""
<div role='radio' id='checkbox'>CHECKBOX</div>
<script>
checkbox.addEventListener('click', () => checkbox.setAttribute('aria-checked', 'true'));
</script>
"""));
await Browser.Locator("div").CheckAsync();
Assert.That(await Browser.EvaluateAsync<string>("() => window.checkbox.getAttribute('aria-checked')"), Is.EqualTo("true"));
}

// Upstream: { fail: '`uncheck` refuses a `role=radio`, where Playwright clicks it and refuses only a native radio' }.
[Test]
public async Task Should_uncheck_radio_by_aria_role()
{
await ExpectedToFailAsync("uncheck", async () =>
{
await Browser.GotoAsync(Content("""
<div role='radio' id='checkbox' aria-checked="true">CHECKBOX</div>
<script>
checkbox.addEventListener('click', () => checkbox.setAttribute('aria-checked', 'false'));
</script>
"""));
await Browser.Locator("div").UncheckAsync();
Equal("false", await Browser.EvaluateAsync<string>("() => window.checkbox.getAttribute('aria-checked')"));
});
}

[Test]
public async Task Should_check_the_box_by_aria_role()
{
foreach (var role in AriaRoles)
{
await Browser.GotoAsync(Content($$"""
<div role='{{role}}' id='checkbox'>CHECKBOX</div>
<script>
checkbox.addEventListener('click', () => checkbox.setAttribute('aria-checked', 'true'));
</script>
"""));
await Browser.Locator("div").CheckAsync();
Assert.That(await Browser.EvaluateAsync<string>("() => window.checkbox.getAttribute('aria-checked')"), Is.EqualTo("true"));
}
}

// Upstream: { fail: '`uncheck` refuses a `role=radio`, where Playwright clicks it and refuses only a native radio' }.
[Test]
public async Task Should_uncheck_the_box_by_aria_role()
{
await ExpectedToFailAsync("uncheck", async () =>
{
foreach (var role in AriaRoles)
{
await Browser.GotoAsync(Content($$"""
<div role='{{role}}' id='checkbox' aria-checked="true">CHECKBOX</div>
<script>
checkbox.addEventListener('click', () => checkbox.setAttribute('aria-checked', 'false'));
</script>
"""));
await Browser.Locator("div").UncheckAsync();
Equal("false", await Browser.EvaluateAsync<string>("() => window.checkbox.getAttribute('aria-checked')"));
}
});
}

[Test]
public async Task Should_throw_when_not_a_checkbox()
{
await Browser.GotoAsync(Content("<div>Check me</div>"));
var error = await Assert.CatchAsync(() => Browser.Locator("div").CheckAsync());
Assert.That(error!.Message, Does.Contain("Not a checkbox or radio button"));
}

[Test]
public async Task Should_throw_when_not_a_checkbox_2()
{
await Browser.GotoAsync(Content("<div role=button>Check me</div>"));
var error = await Assert.CatchAsync(() => Browser.Locator("div").CheckAsync());
Assert.That(error!.Message, Does.Contain("Not a checkbox or radio button"));
}

[Test]
public async Task Should_check_the_box_inside_a_button()
{
await Browser.GotoAsync(Content("<div role='button'><input type='checkbox'></div>"));
await Browser.Locator("input").CheckAsync();
Assert.That(await Browser.EvaluateAsync<bool>("() => document.querySelector('input').checked"), Is.True);
Assert.That(await Browser.Locator("input").IsCheckedAsync(), Is.True);
}

[Test]
public async Task Should_throw_when_trying_to_uncheck_radio_button()
{
await Browser.GotoAsync(Content("<input type='radio' name='test' checked id='radio'>"));
var error = await Assert.CatchAsync(() => Browser.Locator("#radio").UncheckAsync());
Assert.That(error!.Message, Does.Contain("uncheck cannot clear a radio button"));
}
}
2 changes: 2 additions & 0 deletions tests/E2E.NUnit.Tests/E2E.NUnit.Tests.csproj
Original file line number Diff line number Diff line change
Expand Up @@ -16,7 +16,9 @@
</ItemGroup>
<ItemGroup>
<None Include="Testbed\control-inventory.html" CopyToOutputDirectory="PreserveNewest" />
<None Include="Testbed\playwright\**" CopyToOutputDirectory="PreserveNewest" />
<Compile Include="..\E2E.Agent.Tests\Testbed.cs" Link="Testbed\Playground.cs" />
<Compile Include="..\E2E.Agent.Tests\PlaywrightRoutes.cs" Link="Testbed\PlaywrightRoutes.cs" />
</ItemGroup>
<ItemGroup>
<ProjectReference Include="..\..\src\E2E.NUnit\E2E.NUnit.csproj" />
Expand Down
70 changes: 70 additions & 0 deletions tests/E2E.NUnit.Tests/Helpers/Helpers.cs
Original file line number Diff line number Diff line change
@@ -0,0 +1,70 @@
// Copyright 2026 TesterArmy.
// Modified by Dario Kondratiuk.
// SPDX-License-Identifier: Apache-2.0

using System.Text;

namespace E2E.NUnit.Tests;

/// <summary>
/// The helpers of the tests ported from Playwright, upstream's <c>apps/testbed/tests/helpers.ts</c>:
/// <c>content</c> and <c>PREFIX</c>. <see cref="ExpectedToFailAsync"/> is .NET-only (see its summary).
/// </summary>
internal static class Helpers
{
/// <summary>The app's origin, Playwright's <c>server.PREFIX</c> in the tests ported from it.</summary>
public static string PREFIX => E2E.Playground.Testbed.Url;

/// <summary>A path that serves <paramref name="html"/> as a page, for Playwright's <c>page.setContent(html)</c>.</summary>
public static string Content(string html) =>
"/content?html=" + Convert.ToBase64String(Encoding.UTF8.GetBytes(html)).TrimEnd('=').Replace('+', '-').Replace('/', '_');

/// <summary>
/// An equality check for a body run by <see cref="ExpectedToFailAsync"/>. It throws: an NUnit assertion inside a test body is
/// recorded as a failure of the test even when the body catches it.
/// </summary>
public static void Equal<T>(T expected, T actual)
{
if (!EqualityComparer<T>.Default.Equals(expected, actual))
{
throw new InvalidOperationException($"Expected {expected}, but was {actual}.");
}
}

/// <summary>
/// Runs the body of a test upstream marks <c>{ fail: 'reason' }</c>, a known difference from Playwright: the test passes while
/// <paramref name="body"/> still fails with a message that contains <paramref name="failure"/>, and fails once it passes, so the marker is removed. The port has no <c>fail</c> test option
/// (see COMPATIBILITY.md), and NUnit has no expected-failure outcome.
/// </summary>
public static async Task ExpectedToFailAsync(string failure, Func<Task> body)
{
try
{
await body().ConfigureAwait(false);
}
catch (Exception ex) when (ex is not OperationCanceledException)
{
Assert.That(ex.Message, Does.Contain(failure), "the body failed, but not with the known difference");
return;
}

Assert.Fail("UNEXPECTED_PASS: the test is marked as a known difference from Playwright, and it passed. Remove the marker.");
}
}

/// <summary>
/// The base of the classes ported from Playwright's page tests: the playground as the base URL, the cache off, and the <c>playwright</c>
/// category (upstream's <c>tags: ['playwright']</c>).
/// </summary>
[Category("playwright")]
public abstract class PlaywrightTest : E2ETest
{
private static readonly E2EConfig FixtureConfig = E2EConfig.Parse(
"""{ "cache": { "mode": "off" } }""",
AppContext.BaseDirectory,
_ => null);

protected override E2EConfig Config => FixtureConfig;

protected override string? BaseUrl => E2E.Playground.Testbed.Url;
}
27 changes: 27 additions & 0 deletions tests/E2E.NUnit.Tests/LocatorList/LocatorListTests.cs
Original file line number Diff line number Diff line change
@@ -0,0 +1,27 @@
// Copyright 2026 TesterArmy.
// Modified by Dario Kondratiuk.
// SPDX-License-Identifier: Apache-2.0

using static E2E.NUnit.Tests.Helpers;

namespace E2E.NUnit.Tests.LocatorList;

/// <summary>
/// <c>test.describe('locator list')</c> of upstream's <c>apps/testbed/tests/locator-list.e2e.ts</c> (Playwright's
/// <c>locator-list.spec.ts</c>). <c>div &gt;&gt; p</c> is the CSS <c>div p</c>.
/// </summary>
public sealed class LocatorListTests : PlaywrightTest
{
[Test]
public async Task Locator_all_should_work()
{
await Browser.GotoAsync(Content("<div><p>A</p><p>B</p><p>C</p></div>"));
var texts = new List<string?>();
foreach (var p in await Browser.Locator("div p").AllAsync())
{
texts.Add(await p.TextContentAsync());
}

Assert.That(texts, Is.EqualTo(new[] { "A", "B", "C" }));
}
}
Loading
Loading