From 5f35ee0ddbafd28ab954b1df593e825752c15f25 Mon Sep 17 00:00:00 2001 From: Engel Nyst Date: Sat, 11 Jul 2026 03:33:15 +0200 Subject: [PATCH] feat(qa-changes): capture visual evidence for frontend PRs Add a Phase 3 step: when a PR has frontend work, run the affected screens from the branch's final state and capture a screenshot of each relevant state (empty, loading, error, populated) plus a GIF of the key interaction end to end. Where behavior changes, include before/after. Attach it all to the PR so the change can be reviewed by observation, not by reading the diff. Adds a matching 'Visual Evidence' block to the report format and a key principle, and regenerates the skills catalog. Co-authored-by: smolpaws --- skills/index.js | 2 +- skills/qa-changes/SKILL.md | 35 +++++++++++++++++++++++++++++++++++ 2 files changed, 36 insertions(+), 1 deletion(-) diff --git a/skills/index.js b/skills/index.js index c9af0c9f..3b711cad 100644 --- a/skills/index.js +++ b/skills/index.js @@ -397,7 +397,7 @@ export const SKILLS_CATALOG = [ "triggers": [ "/qa-changes" ], - "content": "# QA Changes\n\nValidate pull request changes by actually running the code — not just reading it. The goal is to verify that new behavior works as the PR claims, existing behavior is not broken, and the repository remains healthy after the change.\n\nThe bar is high: test the way a thorough human QA engineer would. If the PR changes a web UI, spin up the server and verify it in a real browser. If it changes a CLI, run the CLI with real inputs. Do not settle for \"the tests pass\" — actually use the software.\n\n## Core Methodology\n\nQA proceeds in four phases. Complete each phase in order. If a phase fails, report the failure and stop.\n\n### Phase 1: Understand the Change\n\nRead the PR diff, title, and description. **Identify the goal of this PR** — this is the single most important thing to understand before proceeding. A PR might fix a bug, add a feature, refactor code, improve performance, update documentation, or something else entirely. Check:\n\n1. **The PR description \"Why\" / \"Summary\" section** — what is the author trying to accomplish?\n2. **Linked issues** — if the PR references an issue, read it. But note: the PR may address the issue differently than expected, or only partially. The PR description is the real specification for what *this PR* intends to deliver.\n3. **The PR title** — often summarizes the intent (e.g., \"fix: X not working when Y\", \"feat: add Z capability\", \"refactor: consolidate duplicated X logic\").\n\nThen classify every changed file:\n\n- **New feature**: User-visible behavior that did not exist before.\n- **Bug fix**: Corrects existing behavior to match intended behavior.\n- **Refactor**: Restructuring that should not change external behavior.\n- **Configuration / CI / docs**: Non-functional changes.\n\nFor each change, identify the *entry point* — the concrete way a user would interact with it (CLI command, API endpoint, UI page, function call). This drives what to exercise in Phase 3.\n\nFinally, form a clear hypothesis: \"This PR should [achieve stated goal] by [approach taken in the diff].\" Phase 3 will test that hypothesis.\n\n### Phase 2: Set Up the Environment\n\nBootstrap the repository so the project builds and runs successfully.\n\n1. **Read the repo's bootstrap instructions.** Check `AGENTS.md`, `README.md`, `Makefile`, `package.json`, `pyproject.toml`, `Cargo.toml`, or equivalent. Always prefer the project's own documented setup commands.\n2. **Install dependencies.** Use the project's dependency manager (`uv sync`, `npm install`, `pip install -r requirements.txt`, `bundle install`, `cargo build`, etc.).\n3. **Build the project** if a build step is required (compile, transpile, bundle).\n4. **Note CI status.** Glance at the PR's CI checks and note whether they pass or fail. Do NOT re-run the test suite yourself — that is CI's job, not yours. Your job starts in Phase 3.\n\nIf setup fails, report the failure with the exact error output and stop.\n\n### Phase 3: Exercise the Changed Behavior\n\nThis is the most important phase. **Actually use the software** the way a real user would to verify the change works as the PR claims. This is what distinguishes QA from CI (which runs tests) and code review (which reads code).\n\n**Do NOT:**\n- Run the test suite (`pytest`, `npm test`, `cargo test`, etc.) — that is CI's job.\n- Analyze code by reading files and commenting on style, structure, or logic — that is code review's job.\n- Run linters, formatters, type checkers, or pre-commit hooks — that is CI's job.\n\n**DO:**\n- Run the actual application, CLI, or server and interact with it as a user would.\n- Make real HTTP requests, run real commands, open real browser pages.\n- Always attempt real execution first. Running `--help`, `--dry-run`, or `--version` is NOT functional verification — it only proves argument parsing works. If real execution fails due to missing credentials, external services, or environment constraints, report what you tried and what could not be verified. Do not substitute `--help` output for evidence the software works.\n- Reproduce bugs and verify fixes end-to-end.\n- Test user-facing behavior that automated tests cannot or do not cover.\n\n**Start by verifying the PR achieves its stated goal.** Use the hypothesis from Phase 1. For example:\n- If the PR claims to \"fix crash when X is empty\", reproduce the crash scenario and confirm it no longer occurs.\n- If the PR claims to \"add support for Y\", actually use Y end-to-end and confirm it works.\n- If the PR claims to \"add a new dashboard page\", navigate to the page and verify it renders and functions correctly.\n- If the PR claims to \"add a new CLI flag\", run the CLI with that flag and verify the output.\n\n\"Tests pass\" is not a QA finding. The question is: does the software actually do what the PR says it does?\n\n**For frontend / UI changes:**\n- Start the development server.\n- Use a real browser (via Playwright, browser automation tools, or the built-in browser) to navigate to the affected pages.\n- Verify the visual change renders correctly. Take screenshots as evidence.\n- Test user interactions (clicks, form submissions, navigation).\n- Try at least one edge case (empty state, long text, missing data).\n\n**For CLI changes:**\n- Run the CLI command with realistic arguments. Capture stdout and stderr.\n- Verify the output matches the PR's claimed behavior.\n- Try at least one edge case (invalid input, missing flags, empty input).\n\n**For API / backend changes:**\n- Start the server.\n- Make actual HTTP requests (`curl`, `httpie`, or a test client) to affected endpoints.\n- Verify response status codes, response bodies, and side effects (database writes, file creation).\n- Test error cases (bad input, missing auth, not found).\n\n**For bug fixes — use a before/after comparison:**\n1. **Reproduce the bug without the fix.** Check out the base branch (or revert the PR's changes) and run a concrete command or code path that triggers the reported failure. Show the exact command and its output.\n2. **Interpret the baseline result.** Explain what the output means — e.g., \"This confirms the bug exists: the resolver cannot find the package because the lockfile's cutoff date is too old.\"\n3. **Apply the PR's changes.** Check out the PR branch, apply the patch, or set the environment variable — whatever the fix entails.\n4. **Re-run the same verification.** Run the same command or exercise the same code path with the fix in place. Show the exact command and its output.\n5. **Interpret the result.** Explain what the new output means — e.g., \"The resolver now finds the package, confirming the fix works.\"\n6. **Check for side effects.** Confirm the fix does not break related functionality.\n\n**For library / SDK changes:**\n- Write a short script that imports and calls the changed functions.\n- Verify the return values and behavior match the PR's claims.\n- Test edge cases the PR author may have missed.\n\n**For refactors:**\n- If the refactor touches a critical or user-facing path, manually exercise that path to confirm behavior is unchanged.\n- For pure internal refactors where CI passes and no user-facing path is affected, Phase 2's CI check is sufficient.\n\n**For configuration / CI / docs:**\n- Validate syntax (YAML lint, JSON parse, markdown render).\n- If it is a build change, confirm the build still succeeds.\n- For doc changes, confirm the documentation renders correctly if a preview is available.\n\n**Always show your work with a before/after narrative.** For every verification, the report must include: (a) the exact command you ran, (b) the actual output you observed, and (c) your interpretation of that output. For bug fixes and behavioral changes, demonstrate BOTH the broken/old state AND the fixed/new state so the reviewer can see the delta. Present this evidence inside collapsible `
` blocks — the core deliverable is the verdict and summary, not raw logs.\n\n### Knowing When to Give Up\n\nSome verification approaches will fail due to environment constraints, missing system dependencies, or tooling limitations. That is expected.\n\n**The rule: if the same general approach fails after three materially different attempts, stop trying that approach.** For example, if three different Playwright configurations all fail to connect to the dev server, do not try a fourth Playwright variation. Switch to a fundamentally different approach (e.g., `curl` + manual HTML inspection instead of browser automation). If two fundamentally different approaches both fail, give up on that specific verification and say so in the report.\n\nWhen giving up on a verification:\n- State clearly what was attempted and why it failed.\n- State what *could not* be verified as a result.\n- Suggest the human add guidance to `AGENTS.md` (or a custom `/qa-changes` skill) that would help future QA runs succeed — for example: which port the dev server runs on, what system packages are required, how to configure browser automation, or what the expected test output looks like.\n\nDo not silently skip verification. An honest \"I could not verify X because Y\" is far more valuable than a false \"everything works.\"\n\n### Phase 4: Report Results\n\nPost a structured report as a PR review using the GitHub API. **Keep the report scannable.** A reviewer should grasp the verdict and key results in under 10 seconds. Put lengthy evidence (logs, code snippets, full command output) inside collapsible `
` blocks so the top-level report stays compact.\n\n#### Report format\n\n```markdown\n## {verdict_emoji} QA Report: {VERDICT}\n\n{One-sentence summary of what was verified and the outcome.}\n\n### Does this PR achieve its stated goal?\n\n{Direct answer: Yes / Partially / No.}\n{2-3 sentences explaining WHY, referencing specific evidence from\nexercising the software. For bug fixes: is the bug actually fixed?\nFor features: does the new capability work end-to-end? For refactors:\nis the restructuring achieved without changing behavior? Be specific\nabout what the goal was and whether the changes deliver on it.}\n\n| Phase | Result |\n|-------|--------|\n| Environment Setup | {emoji} {one-line status} |\n| CI Status | {emoji} {one-line note from CI checks, e.g. \"all green\" or \"2 checks failing\"} |\n| Functional Verification | {emoji} {one-line status} |\n\n
Functional Verification\n\n{Structure each verification as a before/after narrative:\n\n### Test N: {Description}\n\n**Step 1 — Reproduce / establish baseline (without the fix):**\nRan `{exact command}`:\n```\n{actual output}\n```\nThis shows {interpretation — what the output means, e.g. \"the bug\nexists because...\"}.\n\n**Step 2 — Apply the PR's changes:**\n{What was done — e.g. checked out the PR branch, set env var, etc.}\n\n**Step 3 — Re-run with the fix in place:**\nRan `{same or equivalent command}`:\n```\n{actual output}\n```\nThis shows {interpretation — e.g. \"the fix works because the error\nis gone and the expected result appears\"}.\n\nRepeat for each changed behavior. For non-bug-fix changes\n(features, refactors), the baseline step may simply describe the\nprior state rather than reproducing a failure.}\n\n
\n\n
Unable to Verify\n\n{What could not be verified, what was attempted, and suggested\nAGENTS.md guidance. Omit this section entirely if everything\nwas verified.}\n\n
\n\n### Issues Found\n\n{List concrete problems, or \"None.\" if clean.}\n\n- 🔴 **Blocker**: ...\n- 🟠 **Issue**: ...\n- 🟡 **Minor**: ...\n```\n\n#### Formatting rules\n\n- **Verdict line + summary** come first. One emoji, one sentence. No preamble.\n- **Status table** gives the at-a-glance overview. One row per phase, one-line status.\n- **Evidence goes in `
` blocks.** Any code block, log excerpt, or command output longer than ~4 lines belongs inside a collapsible. Reviewers who want proof can expand; others can skip.\n- **Do not repeat information.** The summary, table, and details should each add new information — not restate the same facts in different formats.\n- **Issues Found** is always visible (not collapsible). If there are no issues, write \"None.\"\n- **Omit empty sections.** If there is nothing unable to verify, drop that `
` block entirely.\n\n#### Verdict values\n\n- ✅ **PASS**: Change works as described, no regressions.\n- ⚠️ **PASS WITH ISSUES**: Change mostly works, but issues were found (list them).\n- ❌ **FAIL**: Change does not work as described, or introduces regressions.\n- 🟡 **PARTIAL**: Some behavior verified, some could not be (list what was and was not verified).\n\n## Key Principles\n\n- **Answer the core question first: does this PR achieve its stated goal?** This is the primary deliverable. Explicitly state whether the changes deliver on what the PR description promises — whether that is a bug fix, a new feature, a refactor, or anything else.\n- **Fail fast.** If setup fails, stop and report. Do not spend tokens on later phases with a broken environment.\n- **Run the code, not the tests.** Execute the actual software — start servers, run CLI commands, make HTTP requests, open browsers. Do not run `pytest`, `npm test`, or equivalent test suites. That is CI's job.\n- **Do not analyze code.** Reading files and commenting on style, structure, or logic is code review's job. Your job is to exercise behavior, not read source files.\n- **Set a high bar.** If the change affects a UI, open it in a real browser. If it affects a CLI, run the actual CLI with real inputs. If it affects an API, make real HTTP requests.\n- **Test what the PR claims.** The PR description is the specification. Verify the claim, not hypothetical scenarios.\n- **Leave CI to CI.** Do not re-run tests, linters, formatters, or type checkers. Note CI status, then focus entirely on functional verification that CI cannot do.\n- **Report evidence, not opinions.** Include exact commands, outputs, and error messages — inside collapsible blocks.\n- **Keep it scannable.** The report is for busy reviewers. Verdict and summary up top, evidence collapsed below. Do not repeat information across sections.\n- **Give up gracefully.** If a verification approach does not work after three materially different attempts, switch approaches. If two different approaches fail, give up and report honestly. Suggest `AGENTS.md` improvements.\n- **Respect the project's conventions.** Use the project's own tools and build commands for setup." + "content": "# QA Changes\n\nValidate pull request changes by actually running the code — not just reading it. The goal is to verify that new behavior works as the PR claims, existing behavior is not broken, and the repository remains healthy after the change.\n\nThe bar is high: test the way a thorough human QA engineer would. If the PR changes a web UI, spin up the server and verify it in a real browser. If it changes a CLI, run the CLI with real inputs. Do not settle for \"the tests pass\" — actually use the software.\n\n## Core Methodology\n\nQA proceeds in four phases. Complete each phase in order. If a phase fails, report the failure and stop.\n\n### Phase 1: Understand the Change\n\nRead the PR diff, title, and description. **Identify the goal of this PR** — this is the single most important thing to understand before proceeding. A PR might fix a bug, add a feature, refactor code, improve performance, update documentation, or something else entirely. Check:\n\n1. **The PR description \"Why\" / \"Summary\" section** — what is the author trying to accomplish?\n2. **Linked issues** — if the PR references an issue, read it. But note: the PR may address the issue differently than expected, or only partially. The PR description is the real specification for what *this PR* intends to deliver.\n3. **The PR title** — often summarizes the intent (e.g., \"fix: X not working when Y\", \"feat: add Z capability\", \"refactor: consolidate duplicated X logic\").\n\nThen classify every changed file:\n\n- **New feature**: User-visible behavior that did not exist before.\n- **Bug fix**: Corrects existing behavior to match intended behavior.\n- **Refactor**: Restructuring that should not change external behavior.\n- **Configuration / CI / docs**: Non-functional changes.\n\nFor each change, identify the *entry point* — the concrete way a user would interact with it (CLI command, API endpoint, UI page, function call). This drives what to exercise in Phase 3.\n\nFinally, form a clear hypothesis: \"This PR should [achieve stated goal] by [approach taken in the diff].\" Phase 3 will test that hypothesis.\n\n### Phase 2: Set Up the Environment\n\nBootstrap the repository so the project builds and runs successfully.\n\n1. **Read the repo's bootstrap instructions.** Check `AGENTS.md`, `README.md`, `Makefile`, `package.json`, `pyproject.toml`, `Cargo.toml`, or equivalent. Always prefer the project's own documented setup commands.\n2. **Install dependencies.** Use the project's dependency manager (`uv sync`, `npm install`, `pip install -r requirements.txt`, `bundle install`, `cargo build`, etc.).\n3. **Build the project** if a build step is required (compile, transpile, bundle).\n4. **Note CI status.** Glance at the PR's CI checks and note whether they pass or fail. Do NOT re-run the test suite yourself — that is CI's job, not yours. Your job starts in Phase 3.\n\nIf setup fails, report the failure with the exact error output and stop.\n\n### Phase 3: Exercise the Changed Behavior\n\nThis is the most important phase. **Actually use the software** the way a real user would to verify the change works as the PR claims. This is what distinguishes QA from CI (which runs tests) and code review (which reads code).\n\n**Do NOT:**\n- Run the test suite (`pytest`, `npm test`, `cargo test`, etc.) — that is CI's job.\n- Analyze code by reading files and commenting on style, structure, or logic — that is code review's job.\n- Run linters, formatters, type checkers, or pre-commit hooks — that is CI's job.\n\n**DO:**\n- Run the actual application, CLI, or server and interact with it as a user would.\n- Make real HTTP requests, run real commands, open real browser pages.\n- Always attempt real execution first. Running `--help`, `--dry-run`, or `--version` is NOT functional verification — it only proves argument parsing works. If real execution fails due to missing credentials, external services, or environment constraints, report what you tried and what could not be verified. Do not substitute `--help` output for evidence the software works.\n- Reproduce bugs and verify fixes end-to-end.\n- Test user-facing behavior that automated tests cannot or do not cover.\n\n**Start by verifying the PR achieves its stated goal.** Use the hypothesis from Phase 1. For example:\n- If the PR claims to \"fix crash when X is empty\", reproduce the crash scenario and confirm it no longer occurs.\n- If the PR claims to \"add support for Y\", actually use Y end-to-end and confirm it works.\n- If the PR claims to \"add a new dashboard page\", navigate to the page and verify it renders and functions correctly.\n- If the PR claims to \"add a new CLI flag\", run the CLI with that flag and verify the output.\n\n\"Tests pass\" is not a QA finding. The question is: does the software actually do what the PR says it does?\n\n**For frontend / UI changes:**\n- Start the development server.\n- Use a real browser (via Playwright, browser automation tools, or the built-in browser) to navigate to the affected pages.\n- Verify the visual change renders correctly. Take screenshots as evidence.\n- Test user interactions (clicks, form submissions, navigation).\n- Try at least one edge case (empty state, long text, missing data).\n- Then capture visual evidence and attach it to the PR (see next step).\n\n**Capture visual evidence (frontend PRs only):**\n\nIf the PR has frontend work, run the affected screens from the branch's final\nstate and record what a reviewer would otherwise have to imagine from the diff.\nThe goal: the change can be reviewed **by observation, not by reading code.**\n\n1. **Screenshot each relevant state** of every affected screen — whichever\n apply: *empty*, *loading*, *error*, and *populated*. Skip states the screen\n genuinely cannot reach.\n2. **Record a GIF (or short video) of the key interaction end to end** — the\n main flow the PR changes, driven the way a user would (e.g. via Playwright's\n video/tracing or a screen recorder).\n3. **Where behavior changes, capture before/after** — the same state/interaction\n on the base branch and on the PR branch, side by side, so the delta is\n visible.\n4. **Attach it all to the PR.** Embed the images and GIF directly in the QA\n report so they render inline. If your environment can't upload attachments\n through the API, commit the media to the branch under `.pr/` and reference\n them by their raw URLs. Label each clearly (screen → state, or before/after).\n\nKeep it proportional: capture the screens the PR actually touches, not the whole\napp. If you cannot render a screen (missing data, an unreachable state, no\nbrowser), say so in the report rather than faking it.\n\n**For CLI changes:**\n- Run the CLI command with realistic arguments. Capture stdout and stderr.\n- Verify the output matches the PR's claimed behavior.\n- Try at least one edge case (invalid input, missing flags, empty input).\n\n**For API / backend changes:**\n- Start the server.\n- Make actual HTTP requests (`curl`, `httpie`, or a test client) to affected endpoints.\n- Verify response status codes, response bodies, and side effects (database writes, file creation).\n- Test error cases (bad input, missing auth, not found).\n\n**For bug fixes — use a before/after comparison:**\n1. **Reproduce the bug without the fix.** Check out the base branch (or revert the PR's changes) and run a concrete command or code path that triggers the reported failure. Show the exact command and its output.\n2. **Interpret the baseline result.** Explain what the output means — e.g., \"This confirms the bug exists: the resolver cannot find the package because the lockfile's cutoff date is too old.\"\n3. **Apply the PR's changes.** Check out the PR branch, apply the patch, or set the environment variable — whatever the fix entails.\n4. **Re-run the same verification.** Run the same command or exercise the same code path with the fix in place. Show the exact command and its output.\n5. **Interpret the result.** Explain what the new output means — e.g., \"The resolver now finds the package, confirming the fix works.\"\n6. **Check for side effects.** Confirm the fix does not break related functionality.\n\n**For library / SDK changes:**\n- Write a short script that imports and calls the changed functions.\n- Verify the return values and behavior match the PR's claims.\n- Test edge cases the PR author may have missed.\n\n**For refactors:**\n- If the refactor touches a critical or user-facing path, manually exercise that path to confirm behavior is unchanged.\n- For pure internal refactors where CI passes and no user-facing path is affected, Phase 2's CI check is sufficient.\n\n**For configuration / CI / docs:**\n- Validate syntax (YAML lint, JSON parse, markdown render).\n- If it is a build change, confirm the build still succeeds.\n- For doc changes, confirm the documentation renders correctly if a preview is available.\n\n**Always show your work with a before/after narrative.** For every verification, the report must include: (a) the exact command you ran, (b) the actual output you observed, and (c) your interpretation of that output. For bug fixes and behavioral changes, demonstrate BOTH the broken/old state AND the fixed/new state so the reviewer can see the delta. Present this evidence inside collapsible `
` blocks — the core deliverable is the verdict and summary, not raw logs.\n\n### Knowing When to Give Up\n\nSome verification approaches will fail due to environment constraints, missing system dependencies, or tooling limitations. That is expected.\n\n**The rule: if the same general approach fails after three materially different attempts, stop trying that approach.** For example, if three different Playwright configurations all fail to connect to the dev server, do not try a fourth Playwright variation. Switch to a fundamentally different approach (e.g., `curl` + manual HTML inspection instead of browser automation). If two fundamentally different approaches both fail, give up on that specific verification and say so in the report.\n\nWhen giving up on a verification:\n- State clearly what was attempted and why it failed.\n- State what *could not* be verified as a result.\n- Suggest the human add guidance to `AGENTS.md` (or a custom `/qa-changes` skill) that would help future QA runs succeed — for example: which port the dev server runs on, what system packages are required, how to configure browser automation, or what the expected test output looks like.\n\nDo not silently skip verification. An honest \"I could not verify X because Y\" is far more valuable than a false \"everything works.\"\n\n### Phase 4: Report Results\n\nPost a structured report as a PR review using the GitHub API. **Keep the report scannable.** A reviewer should grasp the verdict and key results in under 10 seconds. Put lengthy evidence (logs, code snippets, full command output) inside collapsible `
` blocks so the top-level report stays compact.\n\n#### Report format\n\n```markdown\n## {verdict_emoji} QA Report: {VERDICT}\n\n{One-sentence summary of what was verified and the outcome.}\n\n### Does this PR achieve its stated goal?\n\n{Direct answer: Yes / Partially / No.}\n{2-3 sentences explaining WHY, referencing specific evidence from\nexercising the software. For bug fixes: is the bug actually fixed?\nFor features: does the new capability work end-to-end? For refactors:\nis the restructuring achieved without changing behavior? Be specific\nabout what the goal was and whether the changes deliver on it.}\n\n| Phase | Result |\n|-------|--------|\n| Environment Setup | {emoji} {one-line status} |\n| CI Status | {emoji} {one-line note from CI checks, e.g. \"all green\" or \"2 checks failing\"} |\n| Functional Verification | {emoji} {one-line status} |\n\n
Functional Verification\n\n{Structure each verification as a before/after narrative:\n\n### Test N: {Description}\n\n**Step 1 — Reproduce / establish baseline (without the fix):**\nRan `{exact command}`:\n```\n{actual output}\n```\nThis shows {interpretation — what the output means, e.g. \"the bug\nexists because...\"}.\n\n**Step 2 — Apply the PR's changes:**\n{What was done — e.g. checked out the PR branch, set env var, etc.}\n\n**Step 3 — Re-run with the fix in place:**\nRan `{same or equivalent command}`:\n```\n{actual output}\n```\nThis shows {interpretation — e.g. \"the fix works because the error\nis gone and the expected result appears\"}.\n\nRepeat for each changed behavior. For non-bug-fix changes\n(features, refactors), the baseline step may simply describe the\nprior state rather than reproducing a failure.}\n\n
\n\n
Visual Evidence\n\n{Frontend PRs only. Embed the screenshots (per screen → state: empty,\nloading, error, populated) and the GIF/video of the key interaction so\nthey render inline. Where behavior changed, show before/after. Label\neach clearly. Omit this section entirely for non-frontend PRs.}\n\n
\n\n
Unable to Verify\n\n{What could not be verified, what was attempted, and suggested\nAGENTS.md guidance. Omit this section entirely if everything\nwas verified.}\n\n
\n\n### Issues Found\n\n{List concrete problems, or \"None.\" if clean.}\n\n- 🔴 **Blocker**: ...\n- 🟠 **Issue**: ...\n- 🟡 **Minor**: ...\n```\n\n#### Formatting rules\n\n- **Verdict line + summary** come first. One emoji, one sentence. No preamble.\n- **Status table** gives the at-a-glance overview. One row per phase, one-line status.\n- **Evidence goes in `
` blocks.** Any code block, log excerpt, or command output longer than ~4 lines belongs inside a collapsible. Reviewers who want proof can expand; others can skip.\n- **Do not repeat information.** The summary, table, and details should each add new information — not restate the same facts in different formats.\n- **Issues Found** is always visible (not collapsible). If there are no issues, write \"None.\"\n- **Omit empty sections.** If there is nothing unable to verify, drop that `
` block entirely.\n\n#### Verdict values\n\n- ✅ **PASS**: Change works as described, no regressions.\n- ⚠️ **PASS WITH ISSUES**: Change mostly works, but issues were found (list them).\n- ❌ **FAIL**: Change does not work as described, or introduces regressions.\n- 🟡 **PARTIAL**: Some behavior verified, some could not be (list what was and was not verified).\n\n## Key Principles\n\n- **Answer the core question first: does this PR achieve its stated goal?** This is the primary deliverable. Explicitly state whether the changes deliver on what the PR description promises — whether that is a bug fix, a new feature, a refactor, or anything else.\n- **Fail fast.** If setup fails, stop and report. Do not spend tokens on later phases with a broken environment.\n- **Run the code, not the tests.** Execute the actual software — start servers, run CLI commands, make HTTP requests, open browsers. Do not run `pytest`, `npm test`, or equivalent test suites. That is CI's job.\n- **Do not analyze code.** Reading files and commenting on style, structure, or logic is code review's job. Your job is to exercise behavior, not read source files.\n- **Set a high bar.** If the change affects a UI, open it in a real browser. If it affects a CLI, run the actual CLI with real inputs. If it affects an API, make real HTTP requests.\n- **Make frontend changes reviewable by observation.** For UI PRs, attach screenshots of each relevant state and a GIF of the key interaction (before/after where behavior changes) so a reviewer can see the change, not infer it from the diff.\n- **Test what the PR claims.** The PR description is the specification. Verify the claim, not hypothetical scenarios.\n- **Leave CI to CI.** Do not re-run tests, linters, formatters, or type checkers. Note CI status, then focus entirely on functional verification that CI cannot do.\n- **Report evidence, not opinions.** Include exact commands, outputs, and error messages — inside collapsible blocks.\n- **Keep it scannable.** The report is for busy reviewers. Verdict and summary up top, evidence collapsed below. Do not repeat information across sections.\n- **Give up gracefully.** If a verification approach does not work after three materially different attempts, switch approaches. If two different approaches fail, give up and report honestly. Suggest `AGENTS.md` improvements.\n- **Respect the project's conventions.** Use the project's own tools and build commands for setup." }, { "name": "release-notes", diff --git a/skills/qa-changes/SKILL.md b/skills/qa-changes/SKILL.md index 12807c27..e3f16abb 100644 --- a/skills/qa-changes/SKILL.md +++ b/skills/qa-changes/SKILL.md @@ -75,6 +75,31 @@ This is the most important phase. **Actually use the software** the way a real u - Verify the visual change renders correctly. Take screenshots as evidence. - Test user interactions (clicks, form submissions, navigation). - Try at least one edge case (empty state, long text, missing data). +- Then capture visual evidence and attach it to the PR (see next step). + +**Capture visual evidence (frontend PRs only):** + +If the PR has frontend work, run the affected screens from the branch's final +state and record what a reviewer would otherwise have to imagine from the diff. +The goal: the change can be reviewed **by observation, not by reading code.** + +1. **Screenshot each relevant state** of every affected screen — whichever + apply: *empty*, *loading*, *error*, and *populated*. Skip states the screen + genuinely cannot reach. +2. **Record a GIF (or short video) of the key interaction end to end** — the + main flow the PR changes, driven the way a user would (e.g. via Playwright's + video/tracing or a screen recorder). +3. **Where behavior changes, capture before/after** — the same state/interaction + on the base branch and on the PR branch, side by side, so the delta is + visible. +4. **Attach it all to the PR.** Embed the images and GIF directly in the QA + report so they render inline. If your environment can't upload attachments + through the API, commit the media to the branch under `.pr/` and reference + them by their raw URLs. Label each clearly (screen → state, or before/after). + +Keep it proportional: capture the screens the PR actually touches, not the whole +app. If you cannot render a screen (missing data, an unreachable state, no +browser), say so in the report rather than faking it. **For CLI changes:** - Run the CLI command with realistic arguments. Capture stdout and stderr. @@ -181,6 +206,15 @@ prior state rather than reproducing a failure.}
+
Visual Evidence + +{Frontend PRs only. Embed the screenshots (per screen → state: empty, +loading, error, populated) and the GIF/video of the key interaction so +they render inline. Where behavior changed, show before/after. Label +each clearly. Omit this section entirely for non-frontend PRs.} + +
+
Unable to Verify {What could not be verified, what was attempted, and suggested @@ -221,6 +255,7 @@ was verified.} - **Run the code, not the tests.** Execute the actual software — start servers, run CLI commands, make HTTP requests, open browsers. Do not run `pytest`, `npm test`, or equivalent test suites. That is CI's job. - **Do not analyze code.** Reading files and commenting on style, structure, or logic is code review's job. Your job is to exercise behavior, not read source files. - **Set a high bar.** If the change affects a UI, open it in a real browser. If it affects a CLI, run the actual CLI with real inputs. If it affects an API, make real HTTP requests. +- **Make frontend changes reviewable by observation.** For UI PRs, attach screenshots of each relevant state and a GIF of the key interaction (before/after where behavior changes) so a reviewer can see the change, not infer it from the diff. - **Test what the PR claims.** The PR description is the specification. Verify the claim, not hypothetical scenarios. - **Leave CI to CI.** Do not re-run tests, linters, formatters, or type checkers. Note CI status, then focus entirely on functional verification that CI cannot do. - **Report evidence, not opinions.** Include exact commands, outputs, and error messages — inside collapsible blocks.