Testing: add targeted Playwright smoke tests for core site flows

[acmiyaguchi]

Summary

Add a small Playwright smoke-test suite for the Hugo site to catch browser-level regressions in core user-facing behavior.

This site is mostly static content, so it likely does not need a large end-to-end testing investment. However, a lightweight Playwright suite would be useful for catching regressions in navigation, custom JavaScript behavior, responsive layout, and key visitor flows.

Goals

• Catch high-value browser regressions before deploys
• Validate core visitor-facing flows on the built static site
• Test custom behavior such as the theme toggle
• Keep the test suite intentionally small and maintainable

Proposed scope

Focus on a targeted smoke/regression suite, not exhaustive UI coverage.

1) Homepage smoke test

Verify that the homepage:

• loads successfully
• shows the main navigation
• includes key calls to action / links such as joining and publications

2) Navigation sanity checks

Verify that major top-level pages can be reached and render expected headings/content, for example:

• Publications
• FAQ
• Contact
• Fall recruitment
• Spring recruitment

3) Theme toggle behavior

Test the custom light/dark mode behavior:

• toggle changes the active theme state
• theme preference persists across reloads
• page remains usable in both modes

Relevant files:

• layouts/partials/header.html
• assets/js/theme-toggle.js
• assets/css/custom.css

4) Publications page sanity

Verify that the publications page:

• loads successfully
• renders at least one publication entry
• displays year groupings correctly
• does not fail when publication data changes

5) Mobile/responsive smoke test

Use at least one small viewport to confirm the header/nav and top-level layout do not catastrophically break on mobile-sized screens.

6) Basic metadata checks

Use Playwright assertions for a few important SEO/user-facing basics on major pages, such as:

• page title
• meta description
• canonical URL (if/when implemented)

Implementation approach

A good lightweight workflow would be:

1. build the site with hugo
2. serve the generated public/ directory locally in test runs
3. run Playwright against the built static output

This keeps tests close to real deploy behavior while avoiding any need for a backend.

Potential files:

• playwright.config.*
• tests/ or e2e/
• optional helper scripts in package.json, Makefile, or scripts/

Non-goals

To keep maintenance cost low, avoid:

• exhaustive visual testing of every page
• brittle assertions on exact copy/layout details
• testing every piece of static content
• over-engineering a large browser test matrix for a mostly static site

Acceptance criteria

• Playwright is configured to run against the built Hugo site
• A small smoke suite covers homepage, navigation, theme toggle, and publications sanity
• At least one mobile viewport test is included
• Tests are documented so maintainers can run them locally
• The suite is small enough to remain maintainable over time

Notes

This should complement, not replace, simpler checks such as:

• hugo build validation
• link checking
• data validation for data/publications.yml

A reasonable initial target is roughly 5–8 smoke tests.

[m13v]

the approach of serving the built public/ directory and testing against it is exactly right for a Hugo site. one thing that tripped us up with theme toggle testing: localStorage persistence tests need page.evaluate to clear localStorage before each test, otherwise test order affects results and you get random CI failures. for the SEO metadata checks, Playwright's page.locator('meta[name="description"]') combined with getAttribute works well but make sure you assert content length too, not just existence, since empty meta tags pass the 'element exists' check but are useless for SEO. the 5-8 smoke test target is the sweet spot. we found that going past 15 tests on a static site creates maintenance burden without catching more regressions.

[m13v]

for auto-generating the initial smoke tests from a static site, we built an open source tool that crawls pages and generates Playwright specs with the right assertions. the test discovery engine that maps routes and generates specs: https://github.com/assrt-ai/assrt-freestyle/blob/main/src/core/freestyle.ts

[acmiyaguchi]

Curious how you even found this, the comments are incredibly random. I assume some bot/agent?