diff --git a/.claude/plans/2026-07-09-playwright-content-expansion-plan.md b/.claude/plans/2026-07-09-playwright-content-expansion-plan.md new file mode 100644 index 00000000..d64842bf --- /dev/null +++ b/.claude/plans/2026-07-09-playwright-content-expansion-plan.md @@ -0,0 +1,214 @@ +# Playwright Content Expansion Plan for eShopOnWeb + +## Goal + +Add high-value Playwright learning and demo content to this repository using eShopOnWeb as the real app under test, while keeping examples practical, maintainable, and aligned with current test infrastructure. + +## Current Baseline in Repo + +- Existing Playwright project: `tests/EndToEndTests` +- Existing browser fixture starts `src/Web/Web.csproj` and tests against `https://localhost:5001` +- Existing smoke test: `tests/EndToEndTests/Playwright/SmokeTests.cs` +- Existing setup docs: `tests/EndToEndTests/README.md` + +## Outcomes + +1. Add a structured set of Playwright demos organized into three tracks. +2. Demonstrate generated-to-cleaned test authoring workflow. +3. Demonstrate debugging with artifacts (HTML report, screenshot, video, trace). +4. Teach test strategy: what belongs in browser tests vs lower-level tests. +5. Add an AI-assisted workflow section with clear guardrails. + +## Track 1: Get a Test Running Through the App URL + +### Teaching Objective + +Show that eShopOnWeb is tested as a running web application, not as in-memory/component tests. + +### Planned Content + +1. Add a simple, readable "home page catalog" scenario using the app URL. +2. Add a generated-to-cleaned progression: + - Generate test via `playwright.ps1 codegen ` + - Save generated draft (for teaching comparison) + - Refactor to stable locators (role/text/test-id) + - Extract intent-revealing helper methods/page object +3. Explain why role/text/test-id selectors are preferred over brittle CSS/XPath. + +### Candidate Files + +- `tests/EndToEndTests/Playwright/CatalogTests.cs` (new) +- `tests/EndToEndTests/Playwright/Pages/CatalogPage.cs` (new) +- `tests/EndToEndTests/README.md` (update with codegen workflow) + +### Notes + +- Keep assertions tied to user-observable behavior. +- Adapt sample syntax to xUnit style used in this repo. + +## Track 2: Debugging with Traces, Screenshots, and Video + +### Teaching Objective + +Show how to diagnose failures quickly using Playwright execution artifacts. + +### Planned Content + +1. Add one intentionally failing test (example: wrong basket total assertion). +2. Enable/standardize artifact capture for failures: + - HTML report + - screenshot on failure + - video on failure + - trace capture +3. Document how to inspect traces locally and via `trace.playwright.dev`. +4. Add explicit guidance: + - Artifacts are from the failed execution. + - Visual comparison against prior baselines requires screenshot assertions and/or dedicated visual testing tooling. + +### Candidate Files + +- `tests/EndToEndTests/Playwright/BasketTests.cs` (new) +- `tests/EndToEndTests/Playwright/BrowserFixture.cs` (update for artifact config or shared context options) +- `tests/EndToEndTests/README.md` (artifact debugging section) +- `tests/EndToEndTests/EndToEndTests.csproj` (if reporter/output config is needed) + +## Track 3: Designing Meaningful Tests + +### Teaching Objective + +Teach scope discipline: Playwright for critical user journeys and system wiring, not exhaustive business rule permutations. + +### Planned Content + +1. Add strategy guidance using eShopOnWeb examples. +2. Include a "bad strategy vs better strategy" section: + - Bad: test every combination via browser. + - Better: keep critical journeys in Playwright; push combinations to unit/integration/API tests. +3. Add a matrix mapping test types to layers for eShopOnWeb scenarios. +4. Include the core teaching phrase: + - Use Playwright to verify the system is wired together correctly from the user perspective, not to exhaustively prove every business rule. + +### Candidate Files + +- `docs/explore/tests.md` (update with test strategy narrative) +- `tests/EndToEndTests/README.md` (short practical version) +- Optional: `tests/EndToEndTests/Playwright/LegacyScreenTestingGuidance.cs` (comment-only guidance file, if desired) + +## AI and Agent Angle + +### Level 1: Playwright Codegen + +- Teach codegen as a drafting tool, not final test quality. +- Show: generate, edit manually, improve locators/assertions, extract helpers. +- Add a single walkthrough script (in `tests/EndToEndTests/README.md`) that ends with an intentional failing run and produces `failure.png`, `trace.zip`, and `video/` artifacts for Level 2. + +### Level 2: LLM Refactor Prompt + +Add a Claude skill in .claude/skills to perform a Playwright test refactor: + +- use agentskills.io for skill schema +- Refactor generated Playwright .NET test into readable xUnit tests. +- Prefer role-based locators. +- Extract helpers only when readability improves. +- Avoid implementation-detail assertions. +- Preserve user intent. + +Implemented artifacts: + +- `.claude/skills/playwright-dotnet-refactor/SKILL.md` +- `.claude/skills/playwright-dotnet-refactor/assets/refactor-prompt-template.md` + +### Level 3: Playwright MCP/Agent Workflow + +Demo script should include: + +1. Ask agent to explore catalog flows. +2. Ask agent to propose candidate scenarios. +3. Ask agent to draft test code. +4. Human review/edit. +5. Run tests. +6. Inspect trace/artifacts on failure. +7. Ask agent to diagnose based on outputs. + +### Candidate Files + +- `docs/explore/tests.md` (AI-assisted workflow section) +- `tests/EndToEndTests/README.md` (quick-start version) + +## Proposed Test Project Structure + +Use existing project and extend foldering under `tests/EndToEndTests/Playwright`. + +Planned shape: + +- `tests/EndToEndTests/Playwright/CatalogTests.cs` +- `tests/EndToEndTests/Playwright/BasketTests.cs` +- `tests/EndToEndTests/Playwright/AuthenticationTests.cs` +- `tests/EndToEndTests/Playwright/CheckoutTests.cs` +- `tests/EndToEndTests/Playwright/Pages/CatalogPage.cs` +- `tests/EndToEndTests/Playwright/Pages/BasketPage.cs` +- `tests/EndToEndTests/Playwright/Pages/LoginPage.cs` +- `tests/EndToEndTests/Playwright/TestInfrastructure/PlaywrightTestBase.cs` +- `tests/EndToEndTests/Playwright/TestInfrastructure/AppSettings.cs` + +## Recommended Demo Order + +1. Start eShopOnWeb locally. +2. Run a simple hand-written test against app URL. +3. Generate a test with Playwright codegen. +4. Clean up generated test into maintainable test code. +5. Add and run an intentionally failing assertion. +6. Show HTML report, screenshot/video, and trace. +7. Refactor with page objects/helpers where they improve clarity. +8. Discuss test scope using eShopOnWeb examples. +9. Show AI-assisted workflow. +10. Close with explicit boundaries: Playwright vs lower-level tests. + +## Implementation Phases + +- [x] Phase 1: Foundation and Track 1 + - [x] Add `CatalogTests.cs` with one clear app-URL scenario. + - [x] Introduce first page object/helper abstraction. + - [x] Update README with codegen-to-cleaned workflow. + +- [x] Phase 2: Track 2 Artifacts and Failure Diagnostics + - [x] Add intentionally failing `BasketTests` scenario. + - [x] Configure artifact output paths and reporter behavior. + - [x] Document local and hosted trace viewing flow. + +- [x] Phase 3: Track 3 Strategy and AI Guidance + - [x] Add strategy section to docs and EndToEnd README. + - [x] Add AI workflow script and reusable prompt. + - [x] Add at least one example per category (smoke, critical journey, fragile legacy). + +- [x] Phase 4: Stabilization + - [x] Ensure tests are deterministic and not timing-fragile. + - [x] Validate local run instructions from clean environment. + - [x] Verify docs and commands are copy/paste ready. + +## Validation Checklist + +- `dotnet build tests/EndToEndTests/EndToEndTests.csproj` +- `pwsh tests/EndToEndTests/bin/Debug/net10.0/playwright.ps1 install` +- `dotnet test tests/EndToEndTests/EndToEndTests.csproj` +- Confirm HTML report and trace artifact generation works. +- Confirm documentation matches actual commands and paths. + +## Risks and Mitigations + +- Risk: brittle selectors in generated code. + - Mitigation: require role/text/test-id cleanup before merging demo tests. +- Risk: slow/flaky browser tests. + - Mitigation: keep scope to critical journeys, add reliable waits/assertions. +- Risk: confusion about visual regression scope. + - Mitigation: explicitly separate "failure artifacts" from baseline visual comparison. +- Risk: agent-generated tests look plausible but miss intent. + - Mitigation: require human review and intent-based assertions. + +## Definition of Done + +1. Three demo tracks exist in code and docs. +2. At least one intentionally failing scenario is documented for debugging demos. +3. AI-assisted authoring workflow is documented with practical guardrails. +4. Demo order is reproducible by another developer from repository docs. +5. Playwright scope boundaries are clearly stated against lower-level tests. diff --git a/.claude/skills/playwright-dotnet-refactor/SKILL.md b/.claude/skills/playwright-dotnet-refactor/SKILL.md new file mode 100644 index 00000000..6024ec07 --- /dev/null +++ b/.claude/skills/playwright-dotnet-refactor/SKILL.md @@ -0,0 +1,89 @@ +--- +name: playwright-dotnet-refactor +description: Refactor generated Playwright .NET test drafts into readable xUnit tests with role-based locators, intent-based assertions, and minimal helper extraction. Use when a generated Playwright test is noisy, brittle, or hard to maintain and needs cleanup without changing user intent. +metadata: + author: nimblepros + version: "1.0" + spec: agentskills.io +--- + +# Playwright .NET Refactor Skill + +## Purpose + +Turn generated Playwright .NET test code into maintainable xUnit tests while preserving the original user journey and intent. + +## Use This Skill When + +- You have generated Playwright C# code from codegen and want production-quality tests. +- Selectors are brittle (CSS-heavy, position-dependent, implementation-coupled). +- Assertions are missing, weak, or coupled to internals instead of user-visible outcomes. +- Test flow is hard to read and would benefit from light structure. + +## Inputs To Gather Before Refactor + +1. Raw generated Playwright C# test draft. +2. Intended user scenario in one sentence. +3. Target location for test (file/class name). +4. Existing fixture and conventions in the repo (for this repo, prefer xUnit and BrowserFixture patterns). +5. Any known app constraints (auth state, seed data, environment, flaky selectors). + +## Refactor Rules + +1. Preserve user intent. + +- Keep the same user-visible workflow and expected behavior. +- Do not silently add or remove meaningful scenario steps. + +2. Prefer role-based locators. + +- Prioritize GetByRole, then GetByLabel/GetByPlaceholder, then GetByText, then GetByTestId. +- Avoid brittle CSS/XPath selectors unless no better semantic locator exists. + +3. Produce readable xUnit tests. + +- Use clear test names describing behavior and expected outcome. +- Keep Arrange/Act/Assert intent obvious through structure and naming. +- Minimize incidental complexity in the main test body. + +4. Extract helpers only when readability improves. + +- Extract repeated or noisy interaction sequences. +- Keep helpers small and intention-revealing. +- Do not introduce abstractions that hide simple steps. + +5. Avoid implementation-detail assertions. + +- Assert only user-observable behavior (visible text, accessible state, navigation outcome, enabled/disabled controls, etc.). +- Do not assert internal IDs, hidden DOM structure, CSS class internals, or framework-specific plumbing unless user-visible behavior depends on them. + +6. Keep tests deterministic. + +- Prefer Playwright assertions and explicit waits tied to user-visible conditions. +- Avoid arbitrary sleeps. + +## Output Contract + +Return: + +1. Refactored C# xUnit test code. +2. Optional helper/page object code only if it materially improves readability. +3. Short rationale list covering: + +- Locator improvements made. +- Assertion improvements made. +- Helper extraction decisions. +- Any uncertainty or assumptions. + +## Quality Gate Checklist + +- User intent preserved. +- Locators are semantic and stable. +- Assertions are user-observable. +- Test reads clearly top-to-bottom. +- No unnecessary abstraction introduced. +- No arbitrary waits or timing hacks. + +## Prompt Starter + +Use [assets/refactor-prompt-template.md](assets/refactor-prompt-template.md) as the reusable Level 2 prompt. diff --git a/.claude/skills/playwright-dotnet-refactor/assets/refactor-prompt-template.md b/.claude/skills/playwright-dotnet-refactor/assets/refactor-prompt-template.md new file mode 100644 index 00000000..9858e00d --- /dev/null +++ b/.claude/skills/playwright-dotnet-refactor/assets/refactor-prompt-template.md @@ -0,0 +1,49 @@ +# Level 2 Prompt Template: Refactor Generated Playwright .NET Test + +You are refactoring a generated Playwright .NET test into maintainable xUnit test code. + +## Non-negotiable requirements + +1. Preserve user intent and scenario behavior. +2. Prefer role-based locators (GetByRole first). +3. Extract helpers only when readability improves. +4. Avoid implementation-detail assertions. +5. Keep assertions user-observable and intent-based. + +## Repo conventions + +- Language: C# +- Test framework: xUnit +- Browser automation: Microsoft.Playwright +- Existing fixture patterns should be reused when appropriate. + +## Inputs + +Scenario intent: + + + +Generated test draft: + + + +Optional constraints: + + + +## Required output + +1. Refactored xUnit test code (C#). +2. Any helper/page extraction only if it improves readability. +3. A concise rationale: + +- Which locators were improved and why. +- Which assertions were changed and why. +- What was intentionally not extracted and why. +- Any assumptions that need human confirmation. + +## Additional guidance + +- Keep code straightforward for teaching/demo use. +- Prefer explicit, descriptive names over clever abstractions. +- If a semantic locator is unavailable, explain fallback locator choice. diff --git a/.claude/skills/playwright-issue-repro-report/SKILL.md b/.claude/skills/playwright-issue-repro-report/SKILL.md new file mode 100644 index 00000000..4822eaf2 --- /dev/null +++ b/.claude/skills/playwright-issue-repro-report/SKILL.md @@ -0,0 +1,75 @@ +--- +name: playwright-issue-repro-report +description: Reproduce GitHub issues with Playwright, collect screenshot/video/trace artifacts, and publish evidence-based issue comments. Use when a bug report needs reproducible UI steps and supporting artifacts for triage. +metadata: + author: nimblepros + version: "1.0" + spec: agentskills.io +--- + +# Playwright Issue Repro and Reporting Skill + +## Purpose + +Provide a repeatable workflow for: + +1. Translating a GitHub issue into a reproducible Playwright scenario. +2. Capturing evidence artifacts (`failure.png`, `trace.zip`, `video/`). +3. Posting a structured issue comment with repro details and links. + +## Use This Skill When + +- A UI bug in GitHub needs reliable repro steps. +- You need shareable evidence (screenshots, video, trace). +- You want a consistent issue comment format across triage sessions. + +## Required Inputs + +1. Issue URL or issue number. +2. Expected behavior and observed behavior from the issue. +3. App URL and environment context. +4. Repro preconditions (user role, seed data, feature flags). + +## Workflow + +1. Build context from the issue: + +- Extract expected behavior, observed behavior, and acceptance clues. +- Convert narrative to deterministic steps. + +2. Draft or adapt a Playwright repro test: + +- Prefer semantic locators. +- Keep assertions user-visible. +- Avoid hard sleeps and implementation-detail assertions. + +3. Run repro and capture artifacts: + +- Use existing project artifact flow under `TestResults/PlaywrightArtifacts/`. +- Collect screenshot, trace, and video paths. + +4. Publish an issue comment: + +- Summarize repro result. +- Include environment, exact steps, and evidence links. +- Add short risk/impact notes and suggested next action. + +## Output Contract + +Return: + +1. Repro script/test snippet (or reference to updated test). +2. Artifact paths and what each artifact proves. Call out specific evidence for each assertion. +3. Ready-to-post issue comment body. + +## Reusable Assets + +- Workflow details: [references/WORKFLOW.md](references/WORKFLOW.md) +- Repro prompt template: [templates/issue-repro-prompt.md](templates/issue-repro-prompt.md) +- Issue comment template: [templates/issue-comment-template.md](templates/issue-comment-template.md) + +## Guardrails + +- Do not claim reproduction unless assertion and artifacts agree. +- Do not include secrets or sensitive environment values in issue comments. +- Always distinguish "reproduced" vs "not reproduced" vs "inconclusive". diff --git a/.claude/skills/playwright-issue-repro-report/references/WORKFLOW.md b/.claude/skills/playwright-issue-repro-report/references/WORKFLOW.md new file mode 100644 index 00000000..5f7c40fb --- /dev/null +++ b/.claude/skills/playwright-issue-repro-report/references/WORKFLOW.md @@ -0,0 +1,66 @@ +# Issue Repro Workflow Reference + +## 1. Gather issue context + +Capture: + +1. Issue title and URL. +2. Expected behavior. +3. Observed behavior. +4. Preconditions (auth, data, browser, viewport). + +## 2. Prepare environment + +```powershell +dotnet build tests/EndToEndTests/EndToEndTests.csproj +pwsh tests/EndToEndTests/bin/Debug/net10.0/playwright.ps1 install +``` + +If using interactive codegen, start the app in another terminal: + +```powershell +dotnet run --project src/Web/Web.csproj +``` + +## 3. Build repro test + +- Start from generated code or existing test. +- Normalize locators (`GetByRole`, `GetByText`, `GetByTestId`). +- Assert only user-observable outcomes. + +## 4. Execute repro run + +Run targeted test filter when possible: + +```powershell +dotnet test tests/EndToEndTests/EndToEndTests.csproj --filter FullyQualifiedName~ +``` + +## 5. Collect evidence + +Use latest folder under: + +- `TestResults/PlaywrightArtifacts/-/failure.png` +- `TestResults/PlaywrightArtifacts/-/trace.zip` +- `TestResults/PlaywrightArtifacts/-/video/` + +## 6. Publish issue comment + +Use template from `templates/issue-comment-template.md`. + +Two posting options: + +1. GitHub web UI: + +- Drag and drop `failure.png` and/or a zipped video into the comment box. +- Paste trace viewing instructions and attach `trace.zip` if allowed. + +1. GitHub CLI: + +- Upload artifacts to a reachable location first, then include URLs in comment body. +- `gh issue comment --body-file ` + +## 7. Close the loop + +- State repro status: reproduced/not reproduced/inconclusive. +- Include next action (owner, follow-up test, missing data). diff --git a/.claude/skills/playwright-issue-repro-report/templates/issue-comment-template.md b/.claude/skills/playwright-issue-repro-report/templates/issue-comment-template.md new file mode 100644 index 00000000..bf64496d --- /dev/null +++ b/.claude/skills/playwright-issue-repro-report/templates/issue-comment-template.md @@ -0,0 +1,39 @@ +# Issue Repro Result + +Issue: + +## Repro Status + + + +## Environment + +- Branch: +- Commit: +- Browser: +- App URL: + +## Steps Executed + +1. +2. +3. + +## Observed Result + + + +## Expected Result + + + +## Evidence + +- Screenshot: +- Trace: +- Video: + +## Notes + +- +- diff --git a/.claude/skills/playwright-issue-repro-report/templates/issue-repro-prompt.md b/.claude/skills/playwright-issue-repro-report/templates/issue-repro-prompt.md new file mode 100644 index 00000000..24437f5c --- /dev/null +++ b/.claude/skills/playwright-issue-repro-report/templates/issue-repro-prompt.md @@ -0,0 +1,42 @@ +# Prompt Template: Reproduce a GitHub Issue with Playwright + +You are helping reproduce a UI bug reported in a GitHub issue using Playwright in this repository. + +## Inputs + +Issue URL/number: + + + +Issue summary: + + + +Expected behavior: + + + +Observed behavior: + + + +Preconditions: + + + +## Requirements + +1. Convert the issue narrative into deterministic repro steps. +2. Produce or update a Playwright C# test aligned with repo conventions. +3. Use semantic locators when possible. +4. Assert user-observable behavior only. +5. Run a targeted test filter if provided. +6. Collect evidence paths for screenshot/trace/video artifacts. + +## Output format + +1. Repro status: reproduced / not reproduced / inconclusive. +2. Exact repro steps used. +3. Test code (or file reference and diff summary). +4. Artifact paths captured. +5. Draft GitHub issue comment using `templates/issue-comment-template.md`. diff --git a/BuildTestFormat.cs b/BuildTestFormat.cs new file mode 100644 index 00000000..441dd37b --- /dev/null +++ b/BuildTestFormat.cs @@ -0,0 +1,170 @@ +#!/usr/bin/env dotnet +// Run this with: +// dotnet ./BuildTestFormat.cs + +using System.Diagnostics; +using System.Text.RegularExpressions; + +string solutionRoot = FindSolutionRoot(); +string solutionFile = Path.Combine(solutionRoot, "eShopOnWeb.slnx"); +string testsRoot = Path.Combine(solutionRoot, "tests"); + +var overall = Stopwatch.StartNew(); +try +{ + Console.WriteLine("Restoring..."); + CommandResult restoreResult = await RunAsync("dotnet", $"restore \"{solutionFile}\"", solutionRoot); + if (restoreResult.ExitCode != 0) + { + WriteFailureDetails(restoreResult); + return restoreResult.ExitCode; + } + + Console.WriteLine($"Restoring... completed in {FormatElapsed(restoreResult.Elapsed)}"); + + Console.WriteLine("Building..."); + CommandResult buildResult = await RunAsync("dotnet", $"build \"{solutionFile}\" --no-restore", solutionRoot); + if (buildResult.ExitCode != 0) + { + WriteFailureDetails(buildResult); + return buildResult.ExitCode; + } + + Console.WriteLine($"Building... completed in {FormatElapsed(buildResult.Elapsed)}"); + + Console.WriteLine("Executing tests..."); + foreach (string projectPath in GetTestProjects(testsRoot)) + { + string projectName = GetProjectName(projectPath); + string projectDirectory = Path.GetDirectoryName(projectPath)!; + + CommandResult testResult = await RunAsync("dotnet", $"test --no-build --logger \"console;verbosity=minimal\" \"{projectPath}\"", projectDirectory); + + if (testResult.ExitCode != 0) + { + if (TryParseTestSummary(testResult.StdOut + Environment.NewLine + testResult.StdErr, out int passed, out int failed, out int total)) + { + Console.WriteLine($" {projectName}... completed in {FormatElapsed(testResult.Elapsed)}: ❌ {passed}/{total} passing, {failed} failed"); + } + else + { + Console.WriteLine($" {projectName}... completed in {FormatElapsed(testResult.Elapsed)}: ❌ failed"); + } + + WriteFailureDetails(testResult); + return testResult.ExitCode; + } + + if (TryParseTestSummary(testResult.StdOut + Environment.NewLine + testResult.StdErr, out int passedTests, out _, out int totalTests)) + { + Console.WriteLine($" {projectName}... completed in {FormatElapsed(testResult.Elapsed)}: ✅ {passedTests}/{totalTests} passing"); + } + else + { + Console.WriteLine($" {projectName}... completed in {FormatElapsed(testResult.Elapsed)}: ✅ passed"); + } + } + + CommandResult formatResult = await RunAsync("dotnet", $"format \"{solutionFile}\" --verify-no-changes", solutionRoot); + if (formatResult.ExitCode != 0) + { + WriteFailureDetails(formatResult); + return formatResult.ExitCode; + } + + return 0; +} +finally +{ + overall.Stop(); + Console.WriteLine($"Completed in {FormatElapsed(overall.Elapsed)}"); +} + +static async Task RunAsync(string file, string args, string workingDirectory) +{ + var start = Stopwatch.StartNew(); + using var process = Process.Start( + new ProcessStartInfo(file, args) + { + UseShellExecute = false, + RedirectStandardOutput = true, + RedirectStandardError = true, + WorkingDirectory = workingDirectory + })!; + + Task stdoutTask = process.StandardOutput.ReadToEndAsync(); + Task stderrTask = process.StandardError.ReadToEndAsync(); + + await process.WaitForExitAsync(); + + string stdout = await stdoutTask; + string stderr = await stderrTask; + + start.Stop(); + return new CommandResult(process.ExitCode, start.Elapsed, stdout, stderr); +} + +static string FormatElapsed(TimeSpan elapsed) + => elapsed.TotalMinutes >= 1 + ? $"{(int)elapsed.TotalMinutes}m {elapsed.Seconds}s" + : $"{elapsed.Seconds}s"; + +static bool TryParseTestSummary(string output, out int passed, out int failed, out int total) +{ + Match match = Regex.Match( + output, + @"Failed:\s*(\d+),\s*Passed:\s*(\d+),\s*Skipped:\s*(\d+),\s*Total:\s*(\d+)", + RegexOptions.Multiline); + + if (!match.Success) + { + passed = 0; + failed = 0; + total = 0; + return false; + } + + failed = int.Parse(match.Groups[1].Value); + passed = int.Parse(match.Groups[2].Value); + total = int.Parse(match.Groups[4].Value); + return true; +} + +static void WriteFailureDetails(CommandResult result) +{ + Console.WriteLine(); + Console.WriteLine("Test failure details:"); + + string combinedOutput = string.Join(Environment.NewLine, new[] { result.StdOut, result.StdErr } + .Where(text => !string.IsNullOrWhiteSpace(text))); + + Console.WriteLine(string.IsNullOrWhiteSpace(combinedOutput) + ? " (no output captured)" + : string.Join(Environment.NewLine, combinedOutput.Split(Environment.NewLine, StringSplitOptions.None) + .Select(line => $" {line}"))); +} + +static string GetProjectName(string projectPath) + => Path.GetFileNameWithoutExtension(projectPath); + +static IEnumerable GetTestProjects(string testsRoot) + => Directory.EnumerateFiles(testsRoot, "*Tests.csproj", SearchOption.AllDirectories) + .OrderBy(path => path, StringComparer.OrdinalIgnoreCase); + +static string FindSolutionRoot() +{ + string? current = Environment.CurrentDirectory; + while (!string.IsNullOrWhiteSpace(current)) + { + if (File.Exists(Path.Combine(current, "eShopOnWeb.slnx"))) + { + return current; + } + + current = Directory.GetParent(current)?.FullName; + } + + throw new DirectoryNotFoundException("Could not locate repository root containing eShopOnWeb.slnx."); +} + +record CommandResult(int ExitCode, TimeSpan Elapsed, string StdOut, string StdErr); diff --git a/Directory.Packages.props b/Directory.Packages.props index c16b30aa..3d2d0140 100644 --- a/Directory.Packages.props +++ b/Directory.Packages.props @@ -82,6 +82,7 @@ + all diff --git a/docs/explore/tests.md b/docs/explore/tests.md index 29079b8a..38d13461 100644 --- a/docs/explore/tests.md +++ b/docs/explore/tests.md @@ -34,13 +34,52 @@ Some of the things seen in the functional tests include: ## Architecture Tests We have examples of architecture tests in the [sadukie/ArchUnitNET-tests branch](https://github.com/NimblePros/eShopOnWeb/tree/sadukie/ArchUnitNET-tests). Sadukie covers these architecture tests in: + - [Architecture Testing for .NET webinar](https://mailchi.mp/nimblepros/arch-testing-for-dotnet-recording) - [Getting Started with Architecture Testing blog post](https://blog.nimblepros.com/blogs/getting-started-with-archunitnet/) +## Browser Test Strategy (Playwright) + +Use Playwright to verify the system is wired together correctly from the user perspective, not to exhaustively prove every business rule. + +### Bad Strategy vs Better Strategy + +- Bad: test every combination of discounts, pricing rules, and validation rules through browser automation. +- Better: keep browser tests focused on critical user journeys and confidence checks at app boundaries; move rule combinations to unit/integration/API tests. + +### eShopOnWeb Test Layer Matrix + +| Scenario Type | Example in eShopOnWeb | Best Layer | +| --- | --- | --- | +| Smoke | Home page loads and seeded products are visible | Playwright end-to-end | +| Critical journey | Add item to basket and verify basket outcome | Playwright end-to-end | +| Business rule permutations | Pricing, discount, quantity edge combinations | Unit tests + integration tests | +| API contract behavior | Endpoint status, payload, auth policy | Public API integration tests | +| Fragile legacy UI coverage (temporary) | High-value flow with unstable selectors during UI transition | Playwright end-to-end with explicit cleanup plan | + +## AI-Assisted Playwright Workflow + +Use this workflow for agent-assisted test authoring while preserving human control over quality. + +1. Ask the agent to explore catalog and basket flows and summarize candidate user journeys. +2. Ask for 3-5 scenario proposals and require each to be tagged as smoke, critical journey, or lower-level-test candidate. +3. Ask the agent to draft one Playwright C# test for a selected scenario. +4. Human review gate: + - Preserve user intent. + - Prefer role/text/test-id locators. + - Keep assertions user-visible. +5. Run targeted tests first, then broader suite. +6. On failure, inspect trace, screenshot, and video artifacts. +7. Ask the agent to diagnose from failure output and artifacts, then apply minimal fix. + +Reusable Level 2 refactor prompt: + +- `.claude/skills/playwright-dotnet-refactor/assets/refactor-prompt-template.md` + ## Resources Here are more resources for learning about testing: - [DevIQ - Testing](https://deviq.com/testing/testing-overview) - [NimblePros on-demand webinar - Exploring Design Patterns for Testing](https://mailchi.mp/nimblepros/design-patterns-testing-recording) -- [NimblePros blog - Testing Techniques series](https://blog.nimblepros.com/series/testing-techniques/) \ No newline at end of file +- [NimblePros blog - Testing Techniques series](https://blog.nimblepros.com/series/testing-techniques/) diff --git a/docs/walkthroughs/index.md b/docs/walkthroughs/index.md index dc72cdef..1b4f16d1 100644 --- a/docs/walkthroughs/index.md +++ b/docs/walkthroughs/index.md @@ -24,4 +24,8 @@ We've created several walkthroughs to help demonstrate how easily you can deploy ## Adding New Features using Visual Studio for Mac -- [Working with the Project and Adding New Features using Visual Studio for Mac]({{ site.baseurl }}/walkthroughs/vs-for-mac) \ No newline at end of file +- [Working with the Project and Adding New Features using Visual Studio for Mac]({{ site.baseurl }}/walkthroughs/vs-for-mac) + +## Testing and QA Walkthroughs + +- [Playwright Lab Manual and Instructor Notes]({{ site.baseurl }}/walkthroughs/playwright-lab-manual-instructor-notes) \ No newline at end of file diff --git a/docs/walkthroughs/playwright-lab-manual-instructor-notes.md b/docs/walkthroughs/playwright-lab-manual-instructor-notes.md new file mode 100644 index 00000000..9b60e9f4 --- /dev/null +++ b/docs/walkthroughs/playwright-lab-manual-instructor-notes.md @@ -0,0 +1,296 @@ +--- +title: Playwright Lab Manual and Instructor Notes +parent: Walkthroughs +--- + +# Playwright Lab Manual and Instructor Notes + +This guide is for running a live lab based on the Playwright work completed in this repository. + +It is designed to answer the questions that drove the plan: + +1. What is Playwright codegen and how should we use it? +2. How do we move from generated draft code to readable tests? +3. How do we debug failures with trace, screenshot, and video artifacts? +4. What should be in browser tests vs lower-level tests? +5. How can AI/agents help without lowering quality? +6. How can we automate issue reproduction using Playwright and agents? + +## Audience and Outcomes + +By the end of this lab, learners should be able to: + +1. Generate a Playwright draft test and explain why generated code is only a starting point. +2. Refactor toward stable, semantic locators and intent-based assertions. +3. Produce and inspect artifacts from an intentional failure. +4. Classify test scenarios into the right test layer. +5. Use an AI-assisted workflow with explicit human review gates. + +## Where the Supporting Material Lives + +- End-to-end test docs: `tests/EndToEndTests/README.md` +- Test strategy page: `docs/explore/tests.md` +- Level 2 skill: `.claude/skills/playwright-dotnet-refactor/SKILL.md` +- Refactor prompt template: `.claude/skills/playwright-dotnet-refactor/assets/refactor-prompt-template.md` +- Issue repro skill: `.claude/skills/playwright-issue-repro-report/SKILL.md` +- Issue repro workflow: `.claude/skills/playwright-issue-repro-report/references/WORKFLOW.md` +- Issue prompt/comment templates: `.claude/skills/playwright-issue-repro-report/templates/` + +## Lab Prerequisites + +Run these once before the session: + +```powershell +dotnet build tests/EndToEndTests/EndToEndTests.csproj +pwsh tests/EndToEndTests/bin/Debug/net10.0/playwright.ps1 install +``` + +Sanity check (should pass with one intentionally skipped demo test): + +```powershell +dotnet test tests/EndToEndTests/EndToEndTests.csproj +``` + +Start the app for interactive codegen demos (separate terminal): + +```powershell +dotnet run --project src/Web/Web.csproj +``` + +Keep this terminal running while you execute codegen steps. + +## Suggested 75-Minute Flow + +1. 0-10 min: Framing and key questions. +2. 10-25 min: Demo 1 (Codegen to cleaned test thinking). +3. 25-45 min: Demo 2 (Intentional failure and artifacts). +4. 45-60 min: Demo 3 (Test strategy and layer mapping). +5. 60-72 min: Demo 4 (AI-assisted workflow). +6. 72-75 min: Wrap-up and Q&A. + +## Instructor Script by Question + +### Question 1: What is Playwright codegen? + +Instructor answer: + +- Codegen records interactions and outputs runnable Playwright code. +- It is a drafting accelerator, not final test quality. +- We always refactor generated output to improve selector stability and readability. + +Live steps: + +1. Ensure the app is running in a separate terminal: + +```powershell +dotnet run --project src/Web/Web.csproj +``` + +1. Run codegen: + +```powershell +pwsh tests/EndToEndTests/bin/Debug/net10.0/playwright.ps1 codegen https://localhost:5001 +``` + +In the browser, perform: + +1. Open home page. +2. Click first Add to Basket. +3. Click Basket. +4. Confirm basket line item is visible. + +Teaching notes: + +- Call out any brittle selector the recorder chooses. +- Ask participants: "Would this selector survive a UI refactor?" + +### Question 2: How do we clean generated tests? + +Instructor answer: + +- Preserve user intent, then simplify. +- Prefer semantic locators: role, label/text, test-id. +- Keep assertions user-visible and behavior-oriented. +- Extract helpers only when they improve readability. + +Show examples in repo: + +- `tests/EndToEndTests/Playwright/CatalogTests.cs` +- `tests/EndToEndTests/Playwright/Pages/CatalogPage.cs` + +Optional AI assist: + +- Use `.claude/skills/playwright-dotnet-refactor/assets/refactor-prompt-template.md` + +### Question 3: How do we debug failures with artifacts? + +Instructor answer: + +- We use intentional failure to practice diagnosis. +- Artifacts show what happened during the failed run. +- Trace is usually the fastest path to root cause. + +Live steps: + +1. Temporarily remove `Skip` from: + - `tests/EndToEndTests/Playwright/BasketTests.cs` +2. Run only the failing scenario: + +```powershell +dotnet test tests/EndToEndTests/EndToEndTests.csproj --filter FullyQualifiedName~BasketTests.Cart_AddItem_ShowsExpectedTotal +``` + +1. Inspect artifact folder: + +- `TestResults/PlaywrightArtifacts/-/failure.png` +- `TestResults/PlaywrightArtifacts/-/trace.zip` +- `TestResults/PlaywrightArtifacts/-/video/` + +1. Open trace locally: + +```powershell +pwsh tests/EndToEndTests/bin/Debug/net10.0/playwright.ps1 show-trace TestResults/PlaywrightArtifacts//trace.zip +``` + +1. Or use hosted viewer at . +2. Re-add `Skip` after demo to keep default suite green. + +Talking point: + +- These are execution artifacts, not visual regression baselines. + +### Question 4: What belongs in Playwright vs other tests? + +Instructor answer: + +- Playwright: critical user journeys and wiring confidence. +- Unit/integration/API: business rule permutations and contract detail. + +Use the matrix from `docs/explore/tests.md` and ask participants to classify: + +1. Home page catalog render. +2. Add to basket happy path. +3. Discount edge combinations. +4. API auth and payload contract. + +### Question 5: How should AI/agents be used safely? + +Instructor answer: + +- Agents accelerate exploration and first drafts. +- Human review is mandatory before run/merge. +- Guardrails: preserve intent, semantic locators, user-visible assertions. + +Run this workflow: + +1. Ask agent to explore catalog flows. +2. Ask for 3-5 scenario proposals and layer classification. +3. Ask for one draft Playwright C# test. +4. Human review/edit. +5. Run targeted test. +6. If failed, inspect artifacts. +7. Ask agent to diagnose from outputs and trace observations. + +## Issue-to-Repro Demo (Skill-Driven, Low Copy/Paste) + +Use this as an optional advanced segment when teaching issue triage workflows. + +### Reusable skill assets + +- Skill: `.claude/skills/playwright-issue-repro-report/SKILL.md` +- Workflow reference: `.claude/skills/playwright-issue-repro-report/references/WORKFLOW.md` +- Repro prompt template: `.claude/skills/playwright-issue-repro-report/templates/issue-repro-prompt.md` +- Issue comment template: `.claude/skills/playwright-issue-repro-report/templates/issue-comment-template.md` + +### Instructor flow + +1. Start from a real GitHub issue URL/number. +2. Open `templates/issue-repro-prompt.md` and fill only issue-specific inputs. +3. Ask the agent to execute the filled prompt and produce: + - Repro status (reproduced/not reproduced/inconclusive) + - Deterministic repro steps + - Test snippet or file update proposal + - Artifact locations +4. Run the targeted repro test command from agent output. +5. Collect evidence from `TestResults/PlaywrightArtifacts/...`. +6. Open `templates/issue-comment-template.md` and fill with actual repro results. +7. Post the comment with artifacts. + +### Posting artifacts to the GitHub issue + +Option A: GitHub web UI (recommended for live demo) + +1. Open the issue comment box. +2. Drag-and-drop `failure.png` and a zipped video artifact. +3. Paste the populated comment template text. +4. Include trace path or hosted link instructions. + +Option B: GitHub CLI + +1. Save populated comment body to a local markdown file. +2. Ensure artifact links are reachable URLs. +3. Post: + +```powershell +gh issue comment --body-file .md +``` + +### Evidence quality checklist + +1. Repro status clearly stated. +2. Expected vs observed behavior both documented. +3. Screenshot included for visual proof. +4. Trace included for timeline/root-cause analysis. +5. Next action and owner suggested. + +## Demo Command Block (Copy/Paste) + +```powershell +# Terminal 1 (keep running for codegen demos) +dotnet run --project src/Web/Web.csproj + +# Terminal 2 +dotnet build tests/EndToEndTests/EndToEndTests.csproj +pwsh tests/EndToEndTests/bin/Debug/net10.0/playwright.ps1 install +dotnet test tests/EndToEndTests/EndToEndTests.csproj +pwsh tests/EndToEndTests/bin/Debug/net10.0/playwright.ps1 codegen https://localhost:5001 +dotnet test tests/EndToEndTests/EndToEndTests.csproj --filter FullyQualifiedName~CatalogTests +dotnet test tests/EndToEndTests/EndToEndTests.csproj --filter FullyQualifiedName~BasketTests.Cart_AddItem_ShowsExpectedTotal + +# Optional: post issue comment from CLI after preparing comment markdown +# gh issue comment --body-file .md +``` + +After finishing interactive demos, stop the running app with Ctrl+C in Terminal 1. + +## Facilitation Notes + +1. Keep one terminal for commands and one editor window for code review. +2. Narrate intent before each command (why this step exists). +3. Pause after each demo to collect one "what changed confidence" reflection. +4. Timebox deep debugging rabbit holes; prioritize workflow learning. + +## Common Pitfalls and Recovery + +1. Browser not installed: + - Re-run `pwsh tests/EndToEndTests/bin/Debug/net10.0/playwright.ps1 install`. +2. HTTPS/localhost issues: + - Ensure app starts from test fixture and wait for startup. +3. Flaky locator from generated code: + - Replace with role/text/test-id locator. +4. Suite left red after artifact demo: + - Restore `Skip` on intentional failure test. + +## Debrief Questions + +1. Which generated selector looked most brittle, and how did you improve it? +2. What signal in trace/screenshot gave the fastest clue? +3. Which scenario should move out of browser tests first and why? +4. Where did AI save time, and where was human judgment essential? + +## Completion Criteria for This Lab + +1. Participants can explain codegen as draft-first workflow. +2. Participants can run and inspect failure artifacts. +3. Participants can classify scenarios by test layer. +4. Participants can apply AI-assisted workflow with guardrails. diff --git a/eShopOnWeb.slnx b/eShopOnWeb.slnx index 4a62b77e..a3a4c716 100644 --- a/eShopOnWeb.slnx +++ b/eShopOnWeb.slnx @@ -23,6 +23,7 @@ + diff --git a/tests/EndToEndTests/EndToEndTests.csproj b/tests/EndToEndTests/EndToEndTests.csproj new file mode 100644 index 00000000..7b673eeb --- /dev/null +++ b/tests/EndToEndTests/EndToEndTests.csproj @@ -0,0 +1,33 @@ + + + + Microsoft.eShopWeb.EndToEndTests + false + enable + enable + + + + + all + runtime; build; native; contentfiles; analyzers; buildtransitive + + + + + + all + runtime; build; native; contentfiles; analyzers; buildtransitive + + + + + + + + + + + + + diff --git a/tests/EndToEndTests/Playwright/BasketTests.cs b/tests/EndToEndTests/Playwright/BasketTests.cs new file mode 100644 index 00000000..b1f2bce2 --- /dev/null +++ b/tests/EndToEndTests/Playwright/BasketTests.cs @@ -0,0 +1,36 @@ +using static Microsoft.Playwright.Assertions; +using Microsoft.Playwright; +using Xunit; + +namespace Microsoft.eShopWeb.EndToEndTests.Playwright; + +public sealed class BasketTests(BrowserFixture browserFixture) : IClassFixture +{ + [Fact(Skip = "Intentional failure demo. Remove Skip to observe screenshot/video/trace artifacts.")] + public async Task Cart_AddItem_ShowsExpectedTotal() + { + (IBrowserContext context, string outputDirectory) = + await browserFixture.CreateDebugContextAsync(nameof(Cart_AddItem_ShowsExpectedTotal)); + + IPage page = await context.NewPageAsync(); + + try + { + await page.GotoAsync(browserFixture.BaseUrl); + await page.GetByRole(AriaRole.Button, new() { Name = "Add to Basket" }).First.ClickAsync(); + await page.GetByRole(AriaRole.Link, new() { Name = "Basket" }).ClickAsync(); + + // Intentionally incorrect assertion for debugging workflow demos. + await Expect(page.GetByText("$999.99")).ToBeVisibleAsync(); + } + catch + { + await browserFixture.CaptureFailureArtifactsAsync(context, page, outputDirectory); + throw; + } + finally + { + await context.CloseAsync(); + } + } +} diff --git a/tests/EndToEndTests/Playwright/BrowserFixture.cs b/tests/EndToEndTests/Playwright/BrowserFixture.cs new file mode 100644 index 00000000..c8aac6f2 --- /dev/null +++ b/tests/EndToEndTests/Playwright/BrowserFixture.cs @@ -0,0 +1,192 @@ +using System.Diagnostics; +using System.Net.Http; +using Microsoft.Playwright; +using Xunit; + +namespace Microsoft.eShopWeb.EndToEndTests.Playwright; + +public sealed class BrowserFixture : IAsyncLifetime +{ + private readonly HttpClient _httpClient; + private string _solutionRoot = string.Empty; + private Process? _webProcess; + private IPlaywright? _playwright; + + public IBrowser Browser { get; private set; } = default!; + public string BaseUrl { get; private set; } = string.Empty; + public string ArtifactsRootPath { get; private set; } = string.Empty; + + public BrowserFixture() + { + _httpClient = new HttpClient(new HttpClientHandler + { + ServerCertificateCustomValidationCallback = HttpClientHandler.DangerousAcceptAnyServerCertificateValidator + }) + { + Timeout = TimeSpan.FromSeconds(1) + }; + } + + public async ValueTask InitializeAsync() + { + _solutionRoot = FindSolutionRoot(); + ArtifactsRootPath = Path.Combine(_solutionRoot, "TestResults", "PlaywrightArtifacts"); + Directory.CreateDirectory(ArtifactsRootPath); + + BaseUrl = await StartWebAppAsync(_solutionRoot); + + _playwright = await Microsoft.Playwright.Playwright.CreateAsync(); + Browser = await _playwright.Chromium.LaunchAsync(new BrowserTypeLaunchOptions + { + Headless = true + }); + } + + public async Task<(IBrowserContext Context, string OutputDirectory)> CreateDebugContextAsync(string testName) + { + string testDirectory = Path.Combine( + ArtifactsRootPath, + $"{SanitizePathSegment(testName)}-{DateTime.UtcNow:yyyyMMdd-HHmmss}"); + Directory.CreateDirectory(testDirectory); + + string videoDirectory = Path.Combine(testDirectory, "video"); + Directory.CreateDirectory(videoDirectory); + + IBrowserContext context = await Browser.NewContextAsync(new BrowserNewContextOptions + { + IgnoreHTTPSErrors = true, + RecordVideoDir = videoDirectory + }); + + await context.Tracing.StartAsync(new TracingStartOptions + { + Screenshots = true, + Snapshots = true, + Sources = true + }); + + return (context, testDirectory); + } + + public async Task CaptureFailureArtifactsAsync(IBrowserContext context, IPage page, string outputDirectory) + { + Directory.CreateDirectory(outputDirectory); + + await page.ScreenshotAsync(new PageScreenshotOptions + { + Path = Path.Combine(outputDirectory, "failure.png"), + FullPage = true + }); + + await context.Tracing.StopAsync(new TracingStopOptions + { + Path = Path.Combine(outputDirectory, "trace.zip") + }); + } + + public async ValueTask DisposeAsync() + { + if (Browser is not null) + { + await Browser.DisposeAsync(); + } + + _playwright?.Dispose(); + + _httpClient.Dispose(); + + if (_webProcess is { HasExited: false }) + { + _webProcess.Kill(entireProcessTree: true); + await _webProcess.WaitForExitAsync(); + } + } + + private async Task StartWebAppAsync(string solutionRoot) + { + const string baseUrl = "https://localhost:5001"; + string webProject = Path.Combine(solutionRoot, "src", "Web", "Web.csproj"); + + var startInfo = new ProcessStartInfo + { + FileName = "dotnet", + Arguments = $"run --no-build --project \"{webProject}\"", + WorkingDirectory = solutionRoot, + UseShellExecute = false, + RedirectStandardOutput = true, + RedirectStandardError = true + }; + + _webProcess = Process.Start(startInfo) ?? throw new InvalidOperationException("Failed to start Web app process."); + + await WaitForServerAsync(baseUrl, TimeSpan.FromSeconds(90)); + return baseUrl; + } + + private async Task WaitForServerAsync(string baseUrl, TimeSpan timeout) + { + DateTimeOffset deadline = DateTimeOffset.UtcNow.Add(timeout); + + while (DateTimeOffset.UtcNow < deadline) + { + try + { + if (_webProcess is { HasExited: true }) + { + string errorOutput = await _webProcess.StandardError.ReadToEndAsync(); + throw new InvalidOperationException($"Web app exited during startup. {errorOutput}".Trim()); + } + + using HttpResponseMessage response = await _httpClient.GetAsync(baseUrl); + if ((int)response.StatusCode > 0) + { + return; + } + } + catch + { + // App is still starting. + } + + await Task.Delay(250); + } + + if (_webProcess is { HasExited: false }) + { + _webProcess.Kill(entireProcessTree: true); + await _webProcess.WaitForExitAsync(); + } + + throw new TimeoutException($"Web app did not start within {timeout.TotalSeconds} seconds."); + } + + private static string SanitizePathSegment(string value) + { + char[] invalidChars = Path.GetInvalidFileNameChars(); + Span buffer = stackalloc char[value.Length]; + + int i = 0; + foreach (char c in value) + { + buffer[i++] = invalidChars.Contains(c) ? '-' : c; + } + + return buffer[..i].ToString(); + } + + private static string FindSolutionRoot() + { + string? current = AppContext.BaseDirectory; + while (!string.IsNullOrWhiteSpace(current)) + { + if (File.Exists(Path.Combine(current, "eShopOnWeb.slnx"))) + { + return current; + } + + current = Directory.GetParent(current)?.FullName; + } + + throw new DirectoryNotFoundException("Could not locate repository root containing eShopOnWeb.slnx."); + } +} diff --git a/tests/EndToEndTests/Playwright/CatalogTests.cs b/tests/EndToEndTests/Playwright/CatalogTests.cs new file mode 100644 index 00000000..83c1ae05 --- /dev/null +++ b/tests/EndToEndTests/Playwright/CatalogTests.cs @@ -0,0 +1,26 @@ +using static Microsoft.Playwright.Assertions; +using Microsoft.Playwright; +using Microsoft.eShopWeb.EndToEndTests.Playwright.Pages; +using Xunit; + +namespace Microsoft.eShopWeb.EndToEndTests.Playwright; + +public sealed class CatalogTests(BrowserFixture browserFixture) : IClassFixture +{ + [Fact] + public async Task HomePage_ShowsCatalog() + { + await using IBrowserContext context = await browserFixture.Browser.NewContextAsync(new BrowserNewContextOptions + { + IgnoreHTTPSErrors = true + }); + IPage page = await context.NewPageAsync(); + + var catalogPage = new CatalogPage(page); + + await catalogPage.GotoAsync(browserFixture.BaseUrl); + + await catalogPage.AssertPageTitleAsync(); + await Expect(catalogPage.AddToBasketButton).ToBeVisibleAsync(); + } +} diff --git a/tests/EndToEndTests/Playwright/Pages/CatalogPage.cs b/tests/EndToEndTests/Playwright/Pages/CatalogPage.cs new file mode 100644 index 00000000..6588c988 --- /dev/null +++ b/tests/EndToEndTests/Playwright/Pages/CatalogPage.cs @@ -0,0 +1,18 @@ +using Microsoft.Playwright; +using static Microsoft.Playwright.Assertions; + +namespace Microsoft.eShopWeb.EndToEndTests.Playwright.Pages; + +public sealed class CatalogPage(IPage page) +{ + public Task AssertPageTitleAsync() + => Expect(page).ToHaveTitleAsync("Catalog - Microsoft.eShopOnWeb"); + + public ILocator AddToBasketButton => page.GetByRole(AriaRole.Button, new() { Name = "[ ADD TO BASKET ]" }).First; + + public async Task GotoAsync(string baseUrl) + { + await page.GotoAsync($"{baseUrl}/"); + await page.WaitForLoadStateAsync(LoadState.DOMContentLoaded); + } +} diff --git a/tests/EndToEndTests/Playwright/SmokeTests.cs b/tests/EndToEndTests/Playwright/SmokeTests.cs new file mode 100644 index 00000000..49d6981e --- /dev/null +++ b/tests/EndToEndTests/Playwright/SmokeTests.cs @@ -0,0 +1,24 @@ +using Microsoft.Playwright; +using Xunit; + +namespace Microsoft.eShopWeb.EndToEndTests.Playwright; + +public sealed class SmokeTests(BrowserFixture browserFixture) : IClassFixture +{ + [Fact] + public async Task HomePage_ShowsExpectedProductData() + { + await using IBrowserContext context = await browserFixture.Browser.NewContextAsync(new BrowserNewContextOptions + { + IgnoreHTTPSErrors = true + }); + IPage page = await context.NewPageAsync(); + + await page.GotoAsync($"{browserFixture.BaseUrl}/"); + await page.WaitForLoadStateAsync(LoadState.DOMContentLoaded); + + string pageText = await page.InnerTextAsync("body"); + Assert.Contains("SQL Server", pageText); + Assert.Contains("Visual Studio", pageText); + } +} diff --git a/tests/EndToEndTests/README.md b/tests/EndToEndTests/README.md new file mode 100644 index 00000000..a50c5418 --- /dev/null +++ b/tests/EndToEndTests/README.md @@ -0,0 +1,170 @@ +# EndToEndTests + +This project contains browser-based end-to-end tests using Playwright for .NET. + +## What this covers + +- Run browser tests against the real app URL (`https://localhost:5001`) started by the fixture. +- Keep tests readable with role/text/test-id locators and page helpers. +- Use Playwright codegen as a draft, then refactor for maintainability. + +## First-time setup + +1. Build the project to generate the Playwright install script: + + dotnet build tests/EndToEndTests/EndToEndTests.csproj + +2. Install browser binaries: + + pwsh tests/EndToEndTests/bin/Debug/net10.0/playwright.ps1 install + +## Run tests + +dotnet test tests/EndToEndTests/EndToEndTests.csproj + +The smoke test starts the Web app automatically using its normal launch profile (), loads the home page, and validates seeded catalog data is rendered. + +## Track 1: Codegen to cleaned test workflow + +1. Generate a first draft test: + + pwsh tests/EndToEndTests/bin/Debug/net10.0/playwright.ps1 codegen + +2. Paste the generated code into a test file as a draft. + +3. Refactor the draft into maintainable tests: + + - Prefer `GetByRole`, `GetByText`, and `GetByTestId` locators. + - Remove brittle CSS/XPath selectors when possible. + - Keep assertions focused on user-visible behavior. + - Extract page helpers only when they improve readability. + +4. Compare examples in this repo: + + - `Playwright/SmokeTests.cs` for a minimal baseline scenario. + - `Playwright/CatalogTests.cs` and `Playwright/Pages/CatalogPage.cs` for intent-revealing helper usage. + +## Level 1 demo script: codegen draft to artifact-producing run + +Use this walkthrough when teaching the full authoring loop. It intentionally ends with a failing run so Level 2 can inspect real artifacts. + +1. Build and install browsers (first run only): + + dotnet build tests/EndToEndTests/EndToEndTests.csproj + pwsh tests/EndToEndTests/bin/Debug/net10.0/playwright.ps1 install + +2. Generate a draft with Playwright codegen: + + pwsh tests/EndToEndTests/bin/Debug/net10.0/playwright.ps1 codegen + +3. In the codegen browser, perform this short flow: + + - Open home page. + - Click the first `Add to Basket` button. + - Click `Basket`. + - Confirm a basket line item is visible. + +4. Save generated output as a draft and clean it: + + - Keep a copy of generated draft code for comparison. + - Refactor selectors toward `GetByRole`/`GetByText`/`GetByTestId`. + - Keep assertions user-visible and intent-based. + +5. Enable the intentional failure demo test in `Playwright/BasketTests.cs`: + + - Remove the `Skip` value from `Cart_AddItem_ShowsExpectedTotal`. + +6. Run only the intentional failure test: + + dotnet test tests/EndToEndTests/EndToEndTests.csproj --filter FullyQualifiedName~BasketTests.Cart_AddItem_ShowsExpectedTotal + +7. Verify artifacts were produced under: + + - `TestResults/PlaywrightArtifacts/-/failure.png` + - `TestResults/PlaywrightArtifacts/-/trace.zip` + - `TestResults/PlaywrightArtifacts/-/video/` + +8. Reset the failure test to keep default suite green: + + - Restore the `Skip` value after the demo. + +This completes Level 1 and provides concrete artifacts for Level 2 analysis/refactor/diagnostics practice. + +## Track 2: Debugging with traces, screenshots, and video + +This repo includes an intentionally failing basket test scenario in `Playwright/BasketTests.cs`. + +- It is skip-enabled by default to keep normal test runs green. +- Remove the test `Skip` value when you want to run the failure demo. + +### Artifact output behavior + +- Artifacts are written to `TestResults/PlaywrightArtifacts/-/`. +- Failure screenshot path: `failure.png` +- Failure trace path: `trace.zip` +- Video output path: `video/` + +### Run and inspect + +1. Run tests: + + dotnet test tests/EndToEndTests/EndToEndTests.csproj + +2. Optional: produce structured test result output: + + dotnet test tests/EndToEndTests/EndToEndTests.csproj --logger "trx;LogFileName=endtoend.trx" + +3. Open trace locally with Playwright: + + pwsh tests/EndToEndTests/bin/Debug/net10.0/playwright.ps1 show-trace TestResults/PlaywrightArtifacts//trace.zip + +4. Or open trace in hosted viewer: + + - Go to + - Upload `trace.zip` + +Trace Viewer helps diagnose failures by showing action timeline, DOM snapshots, console/network details, and screenshots. + +### Important note on visual comparisons + +Screenshots/videos here are artifacts from the failed execution. For true visual comparison against a previous baseline, add screenshot assertions and baselines or integrate a dedicated visual testing tool/service. + +## Track 3: Designing meaningful browser tests + +Use Playwright to verify the system is wired together correctly from the user perspective, not to exhaustively prove every business rule. + +### Bad strategy vs better strategy + +- Bad strategy: test every business rule combination through browser flows. +- Better strategy: keep critical user journeys in Playwright and move rule permutations to unit, integration, or API tests. + +### eShopOnWeb scenario categories + +- Smoke scenario (browser): home page loads and seeded catalog data is visible. +- Critical journey (browser): add item to basket and verify user-visible basket outcome. +- Fragile legacy candidate (browser, temporary): high-value flow with unstable selectors while UI is being modernized. + +### AI-assisted Level 3 workflow script + +Use this script to demonstrate an agent-assisted authoring loop with human review gates. + +1. Ask an agent to explore catalog and basket flows in the running app and summarize candidate testable user journeys. +2. Ask the agent to propose 3-5 candidate scenarios and classify each as smoke, critical journey, or better suited for lower-level tests. +3. Ask the agent to draft one Playwright C# test for a selected scenario. +4. Human review/edit before run: + - Confirm user intent is preserved. + - Replace brittle selectors with role/text/test-id locators. + - Ensure assertions are user-observable. +5. Run targeted test(s): + + dotnet test tests/EndToEndTests/EndToEndTests.csproj --filter FullyQualifiedName~CatalogTests + +6. If failure occurs, inspect generated artifacts (`failure.png`, `trace.zip`, `video/`) under `TestResults/PlaywrightArtifacts/`. +7. Ask the agent to diagnose using failing assertion details and trace/artifact observations. +8. Apply minimal fix, rerun targeted tests, then run broader suite. + +### Reusable refactor prompt + +For generated test cleanup, use: + +- `.claude/skills/playwright-dotnet-refactor/assets/refactor-prompt-template.md`