diff --git a/COMPATIBILITY.md b/COMPATIBILITY.md index 7ed296adb..d5038a4d4 100644 --- a/COMPATIBILITY.md +++ b/COMPATIBILITY.md @@ -162,7 +162,7 @@ Differences: - Upstream's docs site pages that have no counterpart in the port: the page objects page (`docs/page-objects.mdx`, moved to `/page-object-model` in `d30114a`), `e2e feedback` and the `docs/reference/cli.mdx` text about its opt-out variables (`4c1e2e8`), and `examples/` with its registry (`b3ae3a4`, `38b802d`, `a29edcb`, `b54646f`). The migration guides (`docs/migrate/*.mdx`, `8537abc`) are ported: see [Migration guides](#migration-guides). The docs site has .NET-only guides for upstream's five example apps of `caf51eb` (tester-army/e2e#1497: i18n, file upload, location, biometrics, deep links), under [Examples](docs-site/src/content/docs/examples/index.mdx). They are not ports: that commit only adds registry entries that link to separate repositories, so there was no guide text. Each page writes C# tests for the upstream app. The self-healing deep dive (`6fa99ea`) is ported as the [Self-healing tests](README.md#self-healing-tests) section of the guide, since the port has no docs pages for the cache, the trace page, or executors to link to. Known gap: only `e2e explore` prints a run summary, since it is the one command that feeds a run to `ListReporter` (see [List reporter](#list-reporter)): the `Cache 4 replayed · 1 handed off · 1 missed` row is `Format.CacheText` over `Format.StepsCacheTally` (`E2E.Cli`, upstream's `cacheText` and `stepsCacheTally` of `report/format.ts`), which an exploration, with its cache off, never fills, and a test run does not print ([#799](https://github.com/hardkoded/e2e-dotnet/issues/799)) - Parallel workers and serial suites - `test.fail` and the `fail` test option (tester-army/e2e#1539, `732ea8a`): a test expected to fail, whose product failure passes and whose pass fails with `UNEXPECTED_PASS`. It lives in upstream's test registry, scheduler, retry, and reporters (`settleExpectedFailure` runs on a finished attempt), and the port has none of them. NUnit and xUnit v3 own the verdict and the retries, and neither has an expected-failure outcome, so a port would need a new public API of its own -- Playwright's page tests in the testbed suite (tester-army/e2e#1505, `bdf769b`) are ported in slices (hardkoded/e2e-dotnet#1060), as the `playwright` category of `tests/E2E.NUnit.Tests`, against the playground's `/content` route and Playwright's test assets (Apache-2.0, `Testbed/playwright/`, with Playwright's `LICENSE` and `NOTICE`). A test upstream marks `{ fail: 'reason' }` keeps its assertions inside the test-only `Helpers.ExpectedToFailAsync`, which passes while the body fails and fails once it passes. Ported so far: `check`, `wait-for-url`, `locator-list`, and `selectors-text` (one test is such a known difference, hardkoded/e2e-dotnet#1062). +- Playwright's page tests in the testbed suite (tester-army/e2e#1505, `bdf769b`) are ported in slices (hardkoded/e2e-dotnet#1060), as the `playwright` category of `tests/E2E.NUnit.Tests`, against the playground's `/content` route and Playwright's test assets (Apache-2.0, `Testbed/playwright/`, with Playwright's `LICENSE` and `NOTICE`). A test upstream marks `{ fail: 'reason' }` keeps its assertions inside the test-only `Helpers.ExpectedToFailAsync`, which passes while the body fails and fails once it passes. Ported so far: `check`, `wait-for-url`, `locator-list`, `selectors-text`, `fill`, `select-option`, `set-input-files`, and `filechooser`. Upstream's `select-option` test marked `fail` (a bare string selects by label only) passes in the port, which matches a value or a label, so it has no marker. A missing file reports .NET's `Could not find file`, where upstream shows Node's `ENOENT`. - Serial groups (upstream tester-army/e2e#1056, #1470): serial runs, group retries, and skipping the rest of a group after a failure belong to the test runner, which is NUnit or xUnit here, so the port keeps no serial groups. The schema has `serialAttempt.cutRetry` and the serial verdict is read from it, but nothing in the port writes a serial attempt: see [Report document](#report-document) - The out-of-process engine protocol (upstream tester-army/e2e#1232, [#873](https://github.com/hardkoded/e2e-dotnet/issues/873)): the port has no runner and no worker pool, so there is no engine process to talk to. An engine is written in .NET against `IEngine` - Diff-only observations for a custom executor's own model turns. The built-in agent has them (#1022): `#id` lines on the opening screen and on `observe`, and a diff after each action (see [MCP server](#mcp-server) for the presenter). `ExecutorObservation.Text`, which an executor reads from `Observe`, keeps the full `[ref=…]` snapshot, cut at the observation byte budget (see [Agents](#agents)), and a node is addressed by its ref there (`new ExecutorTarget("e12")`), so a custom executor has no presenter of its own. Upstream gives every executor the `#id` view. @@ -282,7 +282,7 @@ Upstream's `4982d8d` (#1333) documents that the browser engine's semantic tree l ## Migration guides -`docs-site/src/content/docs/migrate/` ports upstream's `docs/migrate/*.mdx` (`index`, `playwright`, `selenium`, `cypress`, `detox`, `maestro`, `testerarmy`) as read at `7b1b8b27` (`8537abc7`, tester-army/e2e#1441; `b81b7d75`, tester-army/e2e#1411; `d52e9b01`, tester-army/e2e#1468; `a60310f9`, tester-army/e2e#1508; `0852e61`, tester-army/e2e#1545; `f6d0679f`, tester-army/e2e#1361). They keep upstream's file names, headings, and order. Each guide's tables use the port's C# names, and the sidebar has a Migrate group. Differences: +`docs-site/src/content/docs/migrate/` ports upstream's `docs/migrate/*.mdx` (`index`, `playwright`, `selenium`, `cypress`, `detox`, `maestro`, `testerarmy`) as read at `7b1b8b27` (`8537abc7`, tester-army/e2e#1441; `b81b7d75`, tester-army/e2e#1411; `d52e9b01`, tester-army/e2e#1468; `a60310f9`, tester-army/e2e#1508; `f6d0679f`, tester-army/e2e#1361). They keep upstream's file names, headings, and order. Each guide's tables use the port's C# names, and the sidebar has a Migrate group. Differences: - The `playwright` guide maps Playwright for .NET (`PageTest`, `ContextOptions()`, `Page.GetByRole(AriaRole...)`), where upstream maps Node Playwright. Playwright .NET has no `playwright.config.ts`, so the config tables map `BrowserNewContextOptions`, `.runsettings`, and the environment variables instead of `testDir`, `webServer`, `projects`, and `reporter`. The `selenium` guide maps Selenium for .NET, where upstream's example is Java. The `cypress`, `detox`, `maestro`, and `testerarmy` guides keep their source runners and write the target in C#, in a new .NET test project beside the old suite. - Every code sample has an NUnit form, and an xUnit v3 form where the two differ. A short NUnit and xUnit table in the overview replaces upstream's `test`, `describe`, hook, tag, and retry names. @@ -291,7 +291,6 @@ Upstream's `4982d8d` (#1333) documents that the browser engine's semantic tree l - No counterpart in the port, marked `missing` or `by design` in the tables: `e2e run` and `e2e list` (use `dotnet test`), the `tests`, `workers`, `output`, `maxDuration`, `--shard`, `--watch`, `--changed-since`, `--repeat-each`, `--max-failures`, and `--update-snapshots` flags and keys (the framework owns selection, retries, and parallelism; `--last-failed` is the `E2E_LAST_FAILED` variable and the `LastFailed` member of the run fixture, see [Test filters](#test-filters)), `describe` options (`serial`, `skip`, `retries`, `timeout`), `test.extend` fixtures and `base.extend` page objects as fixtures (a page object takes `Screen` and `Browser` in a `[SetUp]` or `InitializeAsync`), several targets in one run (the port runs one), the HTML and markdown run summaries, `unique()` as a free function (`Values.Unique`), `credentials` entries for TOTP (`secrets` hold values, not code), and the `inbox` credential. A row names a feature with `Not yet` where the port has a symbol missing from its locator set, such as the tap `position`, `TapAt`, and the screen-level `Swipe`. - The `detox`, `maestro`, and `testerarmy` rows that use `ToHaveScreenshotAsync` on a device say the port documents the matcher for the web engine only. Nothing in the port runs it on a device. - The prompt's step to port `.detoxrc.js` and `config.yaml` into one device target relies on `E2E_MOBILE_PLATFORM` and `E2E_MOBILE_APP_PATH` to switch between iOS and Android, since `e2e.config.json` holds one target (see [MobileAppTest](#mobileapptest-not-in-upstream)). -- From `0852e61` (tester-army/e2e#1545) the `playwright` guide mirrors only what the port does. Not mirrored, because the port has no such feature: the `workers` and `retries` limits, `test.skip`, `test.fail`, `test.use`, hook and `beforeAll` rules, the `--grep`, `--reporter`, `--trace`, `-u`, and `file:line` flag rows, `expect.closeTo`, and the value-matcher list (the framework owns them). Not verified in the port, so left out: the `**/` route glob rule and the hidden `iframe` visibility note. - The snippets were written against the port's public API and compiled by reading, not run. The docs site build checks the pages, not the C#. - No tests: upstream added none for these pages, and no `src/` symbol changed. @@ -642,7 +641,7 @@ Not ported: - `e2e explore [goal]` (`E2E.Cli`, `ExploreCommand`) takes `--config`, `--target`, `--agent`, `--max-steps`, `--timeout` (milliseconds), `--headed`, `--output`, `--debug`, `--ai-trace`, `--trace`, and `--video`, with upstream's bounds, and exits with the outcome's code: 0, 1, 2, 3, and 130 for an interrupt. A bad flag exits 2 with commander's message (`option '--max-steps ' argument '101' is invalid. must be an integer from 1 through 100`). Each `E2E_USER__USERNAME` and `_PASSWORD` pair in the environment that no configured credential owns is passed as `Credentials` (named `` in lower case; .NET-only, upstream carries the config's `credentials` alone). A username with no password, or a short one, exits 2. - The output is upstream's list reporter, fed by the run's events (see [List reporter](#list-reporter) and [Run events](#run-events)): `explore-text.ts` is `ExploreText` (the severity words, why-it-ended text, step counts, and the order of findings), and `list-explore.ts` is `ExploreView` in `E2E.Cli`. `ExploreCommand` subscribes a `ListReporter` to the process's events, `Explore.ExploreAsync` writes the run's start (`HostRun.Started`, with the explored target and agent) and plan, and the command ends the run with `HostRun.FinishAsync`. The output is the `RUN` header, the header of the exploration, each finding and each step as they happen, then the `Findings` section, the `Assessment`, and the summary rows (`Findings`, `Steps`, `AI`, `Start at`, `Duration`). Differences: no live window (the command always prints the output a pipe gets, where upstream paints a live window on a terminal), the model and the target text the command redacted before are now redacted where the run writes its events (the explore state and the session's results), and a failure that is not the findings' verdict prints as `CODE: message` below the record. An interrupt (exit 130) prints no summary. An error thrown after the run started (the engine failed to start) is one `error CODE: message` line on standard error, below the header. - `--debug`, `--ai-trace`, `--trace [mode]`, and `--video [mode]` are upstream's, with its parser: a bare `--trace` or `--video` is `on`, the optional value is greedy (`--trace "Explore checkout"` reads the goal as the mode and is refused with `write --trace=, or put test files before --trace`), and the old spellings `all` and `retries` are refused with the mode they meant. `ExploreOptions.Trace`, `Video`, `Debug`, and `AiTrace` carry them; a `Trace` or `Video` is a `Run` source over the target's and the config's, else the target's mode. `E2E_TRACE`, `E2E_VIDEO`, `E2E_DEBUG`, and `E2E_AI_TRACE` set them without the flag. An exploration is one attempt, so a retry mode keeps nothing, and `ExploreOptions.Notice` says to pass `--trace on` or `--video on` (upstream's `recordingNotices` for `explore`, as `Explore.RecordingNotices`, internal), as it says that a `--video` the engine cannot record is skipped. An exploration's verdict (`ASSERTION_FAILED` with an issue finding) gets no trace page (`isExploreVerdict`, as `TracePages.IsExploreVerdict`), and `ReportRun.Explore` carries the record for that. `e2e mcp` takes none of the four, as upstream. -- Not ported: `--session` and `--reporter` (the command refuses them with exit 2, since running without a session or a report would change what the run means). `--session` needs a runner that finds and runs the setup test that saves the session and restores it into the exploration's attempt; the CLI has no such runner (a setup test is a compiled `[E2ESetup]` test in the user's test assembly), and a saved session lives in the store of one process (`SessionRun`) and is deleted when the run ends, so another process cannot restore it. `ExploreOptions.Session` (upstream's `session`) is ported for the part that does not need the runner: it names a session the host has already restored (for example through an `Engine` of its own), and then the planner and the explorer are told the app starts signed in (`ExplorePlan.SignedInContext`, upstream's `signedInContext`, added to the explorer's agent context as upstream's `agentContext`), and the planner plans no sign-in and offers no other-account lens. It does not run a setup or restore state itself, and an unknown name is not `COLLECTION_ERROR`. Also not ported: `--reporter json` (the port has no reporters; `.e2e/report.json` has the record under `run.explore`, see [Report document](#report-document)), a start from a restored session by the command, and the `--artifacts` refusal (the port never had the flag). +- Not ported: `--session` and `--reporter` (the command refuses them with exit 2, since running without a session or a report would change what the run means). `--session` needs a runner that finds and runs the setup test that saves the session and restores it into the exploration's attempt; the CLI has no such runner (a setup test is a compiled `[E2ESetup]` test in the user's test assembly), and a saved session lives in the store of one process (`SessionRun`) and is deleted when the run ends, so another process cannot restore it. `ExploreOptions.Session` (upstream's `session`) is ported for the part that does not need the runner: it names a session the host has already restored (for example through an `Engine` of its own), and then the planner and the explorer are told the app starts signed in (`ExplorePlan.SignedInContext`, upstream's `signedInContext`, added to the explorer's agent context as upstream's `agentContext`), and the planner plans no sign-in and offers no other-account lens. It does not run a setup or restore state itself, and an unknown name is not `COLLECTION_ERROR`. Also not ported: `--reporter json` (the port has no reporters; `.e2e/report.json` has the record under `run.explore`, see [Report document](#report-document)), `--reporter json` (the port has no reporters; `.e2e/report.json` has the record under `run.explore`, see [Report document](#report-document)), a start from a restored session by the command, and the `--artifacts` refusal (the port never had the flag). - The planner prompt (tester-army/e2e#1523 `f4be1c4`, #1522 `73734c6`): `ExplorePlan.PlanInstruction` lists summaries for the latest 8 steps only (`SummarizedSteps`, upstream's `SUMMARIZED_STEPS`), and carries upstream's lenses and finish rule. The "another account or role" lens needs several accounts and no `PlanRequest.Session` (the session of `ExploreOptions.Session`), as upstream. `PlanInstruction` adds the `signedInContext` line and the "work as the signed-in user" text for a session. Tests: `ExplorePlan/PlanInstructionTests` (upstream's `explore-plan.test.ts`, whole). `ExploreRules` in `Executor.cs` has upstream's edge-case rule and its "same defect on another screen is a new finding" sentence, word for word. - Tests: `E2eExploreTests` ports the `it` of `explore.test.ts` that says to pass `--video on` when a retry mode would record nothing (partial: the notice comes through `ExploreOptions.Notice`, and a scripted engine that declares the video capability stands in for the web engine). The `--trace` and `--video` parser tests of `cli.test.ts` are about `e2e run`, which the port lacks; `ExploreFlagsTests` (.NET-only) covers the same parser on `e2e explore`, and `ExploreRecordingTests` (.NET-only) covers the files. @@ -770,8 +769,6 @@ Selected is `aria-selected="true"` or a selected `