CMS - E2E fixtures: seed dedicated test pages instead of depending on editable sample content - #447
Merged
Merged
Conversation
The browser suite navigated to sample CMS pages that live in the ordinary
content tree. Nothing marked them as test scaffolding, so to an editor they
looked like any other page — and on the development environment their content
had been removed. Seven back-to-top tests then failed on their first assertion,
before exercising any behaviour, because /wiki/wiki-sandbox and
/guidance/short-page returned 404.
The sample seeder cannot repair that. It imports with skip-on-collision, and
the collision test is the page identity rather than whether the page has
anything to render, so a page whose versions were deleted is walked past on
every run and the route stays 404 for good. Skip is the right behaviour there —
the seed button must never overwrite an editor's work — it just means it cannot
be the recovery mechanism.
So the fixtures move out. A second content-staging bundle, a second seeder, and
a pinned /development-testing root that nothing else creates:
* long-page — wiki-typed, the half of the back-to-top contract that scrolls
* short-page — fits the viewport, where the link must never reveal
* search-fixture — carries the keyword "testfixture" so a site search can be
relied on to return a hit
Imported with replace rather than skip, which is what brings back an emptied
fixture. Nothing under that root is anyone's work, so there is nothing to lose
by overwriting it, and a fixture is always in the state the tests expect rather
than whatever the last person to open it left behind.
The root is a folder and is hidden from the menu, so an ordinary visitor has no
route to it, while an editor browsing the page tree sees one plainly-labelled
container instead of test pages interleaved with their own. The two back-to-top
fixtures are kept out of search as well; the search fixture is deliberately in
it, since being findable is the thing under test.
Seeded behind the existing "seed sample CMS pages" button — the pages are
useful to develop against, not only to test against — and at start-up wherever
SeedDevelopmentData is set, so the review app the suite runs against always has
them. Never in production, where nothing runs the suite and nobody wants a tree
of test pages in the CMS.
The old /wiki/wiki-sandbox and /guidance/short-page rows are left where they
are on environments that already have them; they are no longer seeded, and an
editor can delete them.
AB#303114
Refs #412
Refs #443
Nine tests failed against any database whose sample content had been edited, and all nine came down to the same thing: they asserted on content editors are free to change. Seven back-to-top tests navigated to /wiki/wiki-sandbox and /guidance/short-page and never got past the first assertion — both page rows existed with no versions, so the app returned 404. They now navigate to the fixtures, which are re-imported on every seed and so come back from exactly that state. Two admin analytics tests searched for "widget" and then asserted the prior-search panel lists hit links. Whether that search matched anything was a matter of luck, and the luck ran out; the panel was behaving correctly with nothing to list. They now search for the keyword the search fixture carries, which is guaranteed to match it and nothing an editor wrote. The runtime-created fixtures move under the same root. Teardown there is best-effort and the delete route is a soft delete, so a leaked page stays in the tree and travels to the next environment inside a content export — eighteen of them are sitting in a preproduction bundle already. They now leak somewhere nobody has to ask what they are. Their seeding helper was duplicated verbatim between two test classes; it moves to CmsSeedHelpers alongside the rest. AB#303114 Refs #412 Refs #443
The button now refreshes the fixture pages as well as adding the sample ones, so the tile's description should say so rather than leaving someone to discover it from the banner afterwards. AB#303114 Refs #412
|
Review app for PR 447 was deleted |
davidgouge
approved these changes
Sep 16, 2026
5 tasks
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Nine E2E tests failed against any database whose sample content had been edited. Two
symptoms, one cause: the tests asserted on content editors are free to change.
Seven
BackToTopTestsnavigated to/wiki/wiki-sandboxand/guidance/short-pageandnever got past their first assertion — both page rows existed with zero
PageNodeVersions,so the app returned 404. Two admin analytics tests searched for
widgetand asserted theprior-search panel lists hit links; whether that matched anything was a matter of luck, and
the luck ran out.
The sample seeder cannot repair the first case. It imports with
ContentImportMode.Skip,and the collision test is the page identity rather than whether the page has anything to
render, so an emptied page is walked past on every run and the route stays 404 for good.
Skip is correct there — the seed button must never overwrite an editor's work — it just
cannot be the recovery mechanism.
What changed
A second content-staging bundle, a second seeder, and a pinned
/development-testingrootthat nothing else creates:
long-pageshort-pagesearch-fixturetestfixtureso a search is guaranteed a hitReplace, which is what brings an emptied fixture back. Nothing under thatroot is anyone's work, so a fixture is always in the state the tests expect rather than
whatever the last person to open it left behind.
it, while an editor browsing the page tree sees one plainly-labelled container instead of
test pages interleaved with their own.
develop against, not only to test against — and at start-up wherever
SeedDevelopmentDatais set, so the review app the suite runs against always has them. Never in Production.
E2E fixture …pages move under the same root. Teardown there isbest-effort and the delete route is a soft delete, so a leaked page stays in the tree and
travels to the next environment inside a content export — eighteen are in a preproduction
bundle already. They now leak somewhere nobody has to ask what they are. Their seeding
helper was duplicated verbatim across two test classes; it moves to
CmsSeedHelpers.wiki-sandboxandshort-pageare removed from the sample bundle. The rows are left wherethey are on environments that already have them — no longer seeded, and an editor can delete
them.
Acceptance criteria
development-testingroot exists only where it has been seeded, and appears inneither the site navigation nor search results
BackToTopTeststarget fixture pages and pass against a database whose sample contenthas been edited
widgetmatching the corpusOne deliberate deviation: the
search-fixturepage is visible to search. The two analyticstests assert that fixture content appears in search results, so suppressing it would suppress
the thing under test. The root and the other two fixtures are excluded.
Testing
SearchTelemetryE2ETests.SearchAsync_WidenedQuery_P95LatencyBelow300ms, which fails onunmodified
mainand is excluded by the CI filterCategory!=VisualRegression): 206 passed, 0 failed, 4 skippedWritten test-first. The repair guarantee and the production gate were each verified to fail
when the corresponding code is removed.
Not in scope
The 18 stray
E2E fixture …pages already in a preproduction bundle are not swept up.PageNodeService.DeleteAsynconly soft-deletes, so removing them needs a genuine purge —worth a follow-up alongside a sweeper for the cancelled-run case. New strays now land under
the fixture root rather than
/help.Closes #443
Closes #412
AB#303114