Test the Level A image alternative issue in live prepublish

docs/live-testing.md

@@ -2,7 +2,7 @@
 
 The manual **Live Siteimprove smoke test** workflow builds and tests a candidate, installs those exact bytes into a fresh disposable CMS, then opens the real overlay. Live tests run only on the newer CMS profile. It does not replace the overlay, manufacture report data or invoke its command queue from the test.
 
-The first smoke test checks existing API entitlement, authenticates through the direct-login popup, opens the panel and requires an authenticated successful report response for the exact mapped public URL. The selectors and response contract follow the AEM/WordPress test approach; they still need confirmation with the first live account run. A second live scenario scans a persisted synthetic draft with a missing image alternative, saves a correction as a new draft revision, and scans again. Both checks must carry their distinct run markers, enter the running state, and return to the recheck control with the active-check indicator cleared. As in the WordPress live suite, the issue-result assertion is explicitly skipped until its live result mapping is verified. Loading-state exit does not prove scan success or issue detection. No live run is claimed yet.
+The first smoke test checks existing API entitlement, authenticates through the direct-login popup, opens the panel and requires an authenticated successful report response for the exact mapped public URL. The selectors and response contract follow the AEM/WordPress test approach; they still need confirmation with the first live account run. A second live scenario scans a persisted synthetic draft with a missing image alternative, saves a correction as a new draft revision, and scans again. Both checks must carry their distinct run markers, enter the running state, and return to the recheck control with the active-check indicator cleared. Live run 35017790911 passed the report and draft-handoff smoke checks. The suite now also requires the documented WCAG 1.1.1 Level A check, SIA-R2 (Image without a text alternative), before the fix and its absence in the visible accessibility results after the fix. These new result assertions still require live validation; they fail instead of skipping when the result cannot be found.
 
 ## Configure before enabling
 
@@ -51,7 +51,7 @@ Do not pass live credentials to `test:cms`. The shared runner rejects Siteimprov
 
 Unit tests verify configuration rejection, URL separation, report-readiness predicates, request-domain restrictions safe reporting, and marker isolation. Ordinary CMS CI validates six scenarios on each of the two locked CMS profiles, including persisted draft corrections and unchanged published content. Neither proves the real login selectors, live response schema, environment protections or the first live run. Those remain prerequisites before marking this smoke test operational or making it a release gate.
 
-The outcome JSON contains an explicit `skipped` entry for missing-image-alternative result validation. Passing smoke checks establish authentication, report lookup, fresh draft handoff and loading-state exit; they do not establish correct prepublish results. This follows the [WordPress live runner](https://github.com/Siteimprove/CMS-plugin-Wordpress/blob/master/tests/live/run.js).
+The earlier smoke baseline followed the [WordPress live runner](https://github.com/Siteimprove/CMS-plugin-Wordpress/blob/master/tests/live/run.js). The new active image-alternative assertions go beyond its skipped result check. The rule name and WCAG mapping come from [Siteimprove's documented checks](https://help.siteimprove.com/support/solutions/articles/80000448514) and [API check IDs](https://help.siteimprove.com/support/solutions/articles/80000448497). The fixture contains one synthetic image, with no alternative name before the fix and a persisted descriptive alternative afterward. No live page assets or report contents are copied into the repository.
 
 After login, the test waits for authenticated report data for the mapped URL before opening the report panel, matching the WordPress flow. Diagnostic fields are independently filtered by the reporter; they contain no raw response fields, account text or URLs.
 

docs/test-coverage-plan.md

@@ -6,12 +6,12 @@ Use synthetic fixtures to reproduce reported behavior. Keep customer names, acco
 | --- | --- | --- | --- |
 | 1 | Settings save and prepublish redirects with conventional host routing | Backend HTTP tests and real CMS | Separate redirect PR; add the conventional route to a CMS fixture after integration |
 | 1 | Regional language URLs, fallback, host and port mapping | Backend plus multilingual CMS fixture | Planned; coordinate with existing regional-language work |
-| 1 | Real login and the report for an explicitly mapped crawled URL | Protected live smoke | Implemented; first live execution pending |
+| 1 | Real login and the report for an explicitly mapped crawled URL | Protected live smoke | Passed in live run 35017790911 |
 | 1 | Page → Block → Page navigation without block URL calls or exceptions | Backend/browser plus real CMS | Existing backend/browser coverage; real-CMS block fixture planned. Block support remains out of scope |
 | 2 | Parallel settings reads, token requests and saves without data loss or exceptions | Backend with controlled request timing | Planned; reproduce the failure before selecting a concurrency fix |
 | 2 | Upgrade an older supported package to the candidate and preserve settings | Disposable CMS, two actual package versions | Implemented: published 4.3.3 → candidate on the newer CMS profile; same database, identity, settings and draft |
-| 2 | Prepublish checks consume the current synthetic draft and complete successfully | Protected live suite | Partial: fresh markers, running state and loading-state exit implemented; issue-result assertion explicitly skipped pending verified live mapping; first live execution pending |
-| 2 | Persist an edit, recheck, verify changed results and highlight the current page | Protected live suite | Draft image correction and recheck implemented; first live execution pending. Actual highlighting remains planned |
+| 2 | Prepublish checks consume the current synthetic draft and complete successfully | Protected live suite | Draft handoff and loading-state exit passed in live run 35017790911; active WCAG 1.1.1/SIA-R2 before-and-after result assertions added, pending live validation |
+| 2 | Persist an edit, recheck, verify changed results and highlight the current page | Protected live suite | Draft image correction and recheck passed in live run 35017790911; rule-result assertions await live validation. Actual highlighting remains planned |
 | 3 | Session expiry, login recovery and restricted users | Controlled tests, then live where appropriate | Planned; direct-login smoke does not cover SSO/MFA |
 | 3 | Legacy overlay presentation | Separate live acceptance | Planned only if retained in release scope |
 

tests/CmsHost/Views/StandardPage/Index.cshtml

@@ -10,6 +10,7 @@
 }
 @if (Model.IncludeTestImage)
 {
+    <p>The illustration below identifies the test symbol.</p>
     <img id="live-test-image" width="80" height="80" src="data:image/svg+xml,%3Csvg xmlns='http://www.w3.org/2000/svg' width='80' height='80'%3E%3Crect width='80' height='80' fill='blue'/%3E%3C/svg%3E" alt="@(string.IsNullOrEmpty(Model.ImageAlternative) ? null : Model.ImageAlternative)" />
 }
 </main></body>

tests/live/accessibility-rule.mjs

@@ -0,0 +1,10 @@
+// Official check name and mapping:
+// https://help.siteimprove.com/support/solutions/articles/80000448514
+// https://help.siteimprove.com/support/solutions/articles/80000448497
+export const imageAlternativeRule = Object.freeze({
+  id: 'SIA-R2',
+  apiId: 2,
+  criterion: '1.1.1',
+  level: 'A',
+  label: 'Image without a text alternative',
+});

tests/live/prepublish.spec.mjs

@@ -2,6 +2,7 @@ import { test, expect } from '@playwright/test';
 import { openLiveEditor } from './editor.mjs';
 import { settings } from './settings.mjs';
 import { observeDraft } from './prepublish.mjs';
+import { imageAlternativeRule } from './accessibility-rule.mjs';
 
 async function scan(page, evidence, marker) {
   const overlay = page.frameLocator('iframe.si-iframe-element');
@@ -23,7 +24,7 @@ async function scan(page, evidence, marker) {
   });
 }
 
-test('prepublish hands off both saved draft revisions and exits the loading state', async ({ page, context }) => {
+test('prepublish detects WCAG 1.1.1 image alternative issue and clears it after a saved fix', async ({ page, context }) => {
   test.setTimeout(780_000);
   const config = settings(process.env);
   const marker = process.env.CMS_DRAFT_MARKER;
@@ -33,13 +34,18 @@ test('prepublish hands off both saved draft revisions and exits the loading stat
   await openLiveEditor(page, context);
   const preview = page.frameLocator('iframe[name="sitePreview"]');
   await expect(preview.locator('#live-test-marker')).toHaveText(marker);
+  await expect(preview.locator('img')).toHaveCount(1);
   expect(await preview.locator('#live-test-image').getAttribute('alt')).toBeNull();
   const publishedPath = new URL(config.crawledUrl).pathname;
   const published = await page.request.get(publishedPath);
   expect(published.ok()).toBe(true);
   expect(await published.text()).not.toContain(marker);
   await test.step('fresh draft is handed to the SDK and leaves the running state', async () => {
     await scan(page, evidence, marker);
+    await test.step('live: WCAG 1.1.1 issue detected', async () => {
+      const overlay = page.frameLocator('iframe.si-iframe-element');
+      await expect(overlay.getByText(imageAlternativeRule.label, { exact: true })).toBeVisible();
+    });
   });
   await test.step('saved draft correction is handed to the SDK and leaves the running state', async () => {
     const response = await page.request.post('/test/live-draft/fix', { headers: { 'X-Cms-Test': 'prepublish' } });
@@ -52,11 +58,13 @@ test('prepublish hands off both saved draft revisions and exits the loading stat
     if (!await panel.isVisible()) await page.locator('.si-smallbox button.si-button').click();
     await expect(panel).toBeVisible();
     await scan(page, evidence, fixedMarker);
+    await test.step('live: WCAG 1.1.1 issue cleared', async () => {
+      const overlay = page.frameLocator('iframe.si-iframe-element');
+      // Require the results section to remain visible and reject an error alert.
+      await expect(overlay.getByText('Accessibility', { exact: true }).first()).toBeVisible();
+      await expect(overlay.getByRole('alert')).toHaveCount(0);
+      await expect(overlay.getByText(imageAlternativeRule.label, { exact: true })).toHaveCount(0);
+    });
     expect(await (await page.request.get(publishedPath)).text()).not.toContain(fixedMarker);
   });
 });
-
-// As in WordPress, loading-state exit alone does not establish scan-result correctness.
-test.skip('prepublish reports the missing image alternative and clears it after the fix', async () => {
-  // Restore this assertion after verifying the live result schema or UI mapping.
-});

tests/live/safe-reporter.mjs

@@ -6,7 +6,8 @@ import { mkdirSync, writeFileSync } from 'node:fs';
 const stages = new Set(['live: entitlement', 'live: CMS login', 'live: public URL mapping',
   'live: draft preview', 'live: open login popup', 'live: identity username',
   'live: identity password', 'live: submit login', 'live: report panel', 'live: mapped report data',
-  'live: start prepublish', 'live: draft handoff', 'live: loading-state exit']);
+  'live: start prepublish', 'live: draft handoff', 'live: loading-state exit',
+  'live: WCAG 1.1.1 issue detected', 'live: WCAG 1.1.1 issue cleared']);
 
 export default class SafeReporter {
   results = [];