Skip to content

CMS - E2E fixtures: seed dedicated test pages instead of depending on editable sample content - #447

Merged
dfe-lance merged 4 commits into
mainfrom
e2e-fixture-content
Sep 16, 2026
Merged

dfe-lance merged 4 commits into
mainfrom
e2e-fixture-content

Conversation

@dfe-lance

@dfe-lance dfe-lance commented Sep 15, 2026

Copy link
Copy Markdown
Contributor

Nine E2E tests failed against any database whose sample content had been edited. Two
symptoms, one cause: the tests asserted on content editors are free to change.

Seven BackToTopTests navigated to /wiki/wiki-sandbox and /guidance/short-page and
never got past their first assertion — both page rows existed with zero PageNodeVersions,
so the app returned 404. Two admin analytics tests searched for widget and asserted the
prior-search panel lists hit links; whether that matched anything was a matter of luck, and
the luck ran out.

The sample seeder cannot repair the first case. It imports with ContentImportMode.Skip,
and the collision test is the page identity rather than whether the page has anything to
render, so an emptied page is walked past on every run and the route stays 404 for good.
Skip is correct there — the seed button must never overwrite an editor's work — it just
cannot be the recovery mechanism.

What changed

A second content-staging bundle, a second seeder, and a pinned /development-testing root
that nothing else creates:

Fixture Type In search Purpose
long-page wiki no The half of the back-to-top contract that scrolls
short-page content no Fits the viewport; the link must never reveal
search-fixture content yes Carries the keyword testfixture so a search is guaranteed a hit
  • Imported with Replace, which is what brings an emptied fixture back. Nothing under that
    root is anyone's work, so a fixture is always in the state the tests expect rather than
    whatever the last person to open it left behind.
  • The root is a folder and is hidden from the menu, so an ordinary visitor has no route to
    it, while an editor browsing the page tree sees one plainly-labelled container instead of
    test pages interleaved with their own.
  • Seeded behind the existing Seed sample CMS pages button — the pages are useful to
    develop against, not only to test against — and at start-up wherever SeedDevelopmentData
    is set, so the review app the suite runs against always has them. Never in Production.
  • Runtime-created E2E fixture … pages move under the same root. Teardown there is
    best-effort and the delete route is a soft delete, so a leaked page stays in the tree and
    travels to the next environment inside a content export — eighteen are in a preproduction
    bundle already. They now leak somewhere nobody has to ask what they are. Their seeding
    helper was duplicated verbatim across two test classes; it moves to CmsSeedHelpers.

wiki-sandbox and short-page are removed from the sample bundle. The rows are left where
they are on environments that already have them — no longer seeded, and an editor can delete
them.

Acceptance criteria

  • A development-testing root exists only where it has been seeded, and appears in
    neither the site navigation nor search results
  • Re-running the seed repairs a fixture page whose versions have been deleted
  • BackToTopTests target fixture pages and pass against a database whose sample content
    has been edited
  • The two analytics tests no longer depend on widget matching the corpus
  • Seeding is unavailable in production
  • The full E2E suite runs green locally

One deliberate deviation: the search-fixture page is visible to search. The two analytics
tests assert that fixture content appears in search results, so suppressing it would suppress
the thing under test. The root and the other two fixtures are excluded.

Testing

  • Unit: 5256 passed, 0 failed
  • Integration: 811 passed; the single failure is
    SearchTelemetryE2ETests.SearchAsync_WidenedQuery_P95LatencyBelow300ms, which fails on
    unmodified main and is excluded by the CI filter
  • E2E (Category!=VisualRegression): 206 passed, 0 failed, 4 skipped
  • Full Release build of the solution: clean, no new warnings

Written test-first. The repair guarantee and the production gate were each verified to fail
when the corresponding code is removed.

Not in scope

The 18 stray E2E fixture … pages already in a preproduction bundle are not swept up.
PageNodeService.DeleteAsync only soft-deletes, so removing them needs a genuine purge —
worth a follow-up alongside a sweeper for the cancelled-run case. New strays now land under
the fixture root rather than /help.

Closes #443
Closes #412
AB#303114

The browser suite navigated to sample CMS pages that live in the ordinary
content tree. Nothing marked them as test scaffolding, so to an editor they
looked like any other page — and on the development environment their content
had been removed. Seven back-to-top tests then failed on their first assertion,
before exercising any behaviour, because /wiki/wiki-sandbox and
/guidance/short-page returned 404.

The sample seeder cannot repair that. It imports with skip-on-collision, and
the collision test is the page identity rather than whether the page has
anything to render, so a page whose versions were deleted is walked past on
every run and the route stays 404 for good. Skip is the right behaviour there —
the seed button must never overwrite an editor's work — it just means it cannot
be the recovery mechanism.

So the fixtures move out. A second content-staging bundle, a second seeder, and
a pinned /development-testing root that nothing else creates:

  * long-page  — wiki-typed, the half of the back-to-top contract that scrolls
  * short-page — fits the viewport, where the link must never reveal
  * search-fixture — carries the keyword "testfixture" so a site search can be
    relied on to return a hit

Imported with replace rather than skip, which is what brings back an emptied
fixture. Nothing under that root is anyone's work, so there is nothing to lose
by overwriting it, and a fixture is always in the state the tests expect rather
than whatever the last person to open it left behind.

The root is a folder and is hidden from the menu, so an ordinary visitor has no
route to it, while an editor browsing the page tree sees one plainly-labelled
container instead of test pages interleaved with their own. The two back-to-top
fixtures are kept out of search as well; the search fixture is deliberately in
it, since being findable is the thing under test.

Seeded behind the existing "seed sample CMS pages" button — the pages are
useful to develop against, not only to test against — and at start-up wherever
SeedDevelopmentData is set, so the review app the suite runs against always has
them. Never in production, where nothing runs the suite and nobody wants a tree
of test pages in the CMS.

The old /wiki/wiki-sandbox and /guidance/short-page rows are left where they
are on environments that already have them; they are no longer seeded, and an
editor can delete them.

AB#303114
Refs #412
Refs #443
Nine tests failed against any database whose sample content had been edited,
and all nine came down to the same thing: they asserted on content editors are
free to change.

Seven back-to-top tests navigated to /wiki/wiki-sandbox and /guidance/short-page
and never got past the first assertion — both page rows existed with no
versions, so the app returned 404. They now navigate to the fixtures, which are
re-imported on every seed and so come back from exactly that state.

Two admin analytics tests searched for "widget" and then asserted the
prior-search panel lists hit links. Whether that search matched anything was a
matter of luck, and the luck ran out; the panel was behaving correctly with
nothing to list. They now search for the keyword the search fixture carries,
which is guaranteed to match it and nothing an editor wrote.

The runtime-created fixtures move under the same root. Teardown there is
best-effort and the delete route is a soft delete, so a leaked page stays in the
tree and travels to the next environment inside a content export — eighteen of
them are sitting in a preproduction bundle already. They now leak somewhere
nobody has to ask what they are. Their seeding helper was duplicated verbatim
between two test classes; it moves to CmsSeedHelpers alongside the rest.

AB#303114
Refs #412
Refs #443
The button now refreshes the fixture pages as well as adding the sample ones,
so the tile's description should say so rather than leaving someone to discover
it from the banner afterwards.

AB#303114
Refs #412
@github-actions

github-actions Bot commented Sep 15, 2026

Copy link
Copy Markdown

Review app for PR 447 was deleted

@dfe-lance
dfe-lance merged commit 174ae99 into main Sep 16, 2026
8 checks passed
@dfe-lance
dfe-lance deleted the e2e-fixture-content branch September 16, 2026 19:43
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

bug Something isn't working deploy

Projects

None yet

2 participants