[CGBassPlayer]

Note

#1155 (test dependency upgrade) is built on this PR. Merge this first.

With agents become more and more popular, I figured its more important than ever to add a large suite of tests to the website.

closes #286
fixes #230

• 404 Page
  • Check for unicorn image
  • Check for "Page Not Found" text
  • Random missing URL returns a real 404 status
• Home
  • Logo and tagline
  • Pagination loads a different page of episodes
  • Show slider: one slide per active show, newest episode first
  • Show slider: next/prev buttons move it
  • Show slider: a held mouse drag moves it (all 3 browsers)
• Navigation (desktop)
  • Navbar items and their links
  • Shows / People / Community dropdowns match exactly, and every item links correctly
  • RSS feeds menu
• Episode Page (5 newest + 5 oldest per active show, 2 + 2 per archived show, by date, plus predefined edge-case episodes: LUP 434, Coder Radio 343 & 443). Expected values come from each episode's front matter. Active shows sample 5 because the scraper can rewrite an active show's last 5 episodes.
  • Tab title contains show name, episode number, and JB (fixed regex that matched any title)
  • Podverse Player (iframe points at the episode's GUID; falls back to <audio> when there's no podcast_file)
  • Support and Subscribe Buttons
  • Show Image
  • Hosts / Guests
  • Sponsors
  • Links (same-site links in the episode body resolve)
  • Tags (every tag shown and its tag page resolves; emoji tags skipped for 🦒 giraffe emoji tags page silently failing to build #516)
• Show Page (All Shows)
  • pagination (similar to home page test)
  • Icons
  • Hosts
• Show List and Archive
  • /show/ lists exactly the active shows
  • /archive/ lists exactly the archived shows
• Subscribe Pages (All Shows, e.g. LUP)
  • RSS button points at the show's feed
  • One icon link per direct_links entry in subscribe.md, with its icon from data/podcast_player.json
• Tags (one real tag per tricky character type: spaces, apostrophe, &, -, +, #, ., curly quotes, emoji)
  • /tags/ lists the tag with the right episode count, and its tag page lists those episodes
  • The tag link on an episode page leads to a tag page listing that episode
  • Tag page heading and episode cards
• People (Hosts and Guests)
  • Guest Pagination
  • All Hosts are present
  • Every host/guest card on /people/ links to a page that exists
  • Host pages (every active host): name heading, shows they host, one card per hosted episode
  • Guest pages (3 most frequent guests): name heading, shows they've been on, one card per episode
• Sponsors
  • Network sponsors, then each show with sponsors on an episode in the last 90 days (sponsors.lookbackdays), with the right sponsor links
• Community
  • One card per community page, linking to its direct_link or its own page
  • Community pages without a direct_link (IRC, Matrix, Mumble, Picks) load
• Picks
  • One card per data/picks file, with its link and episode badges
  • Every episode a pick links to exists
• Footer
  • Community links match the community pages
  • "Get Involved" links match config.toml, and "Edit this page on GitHub" points at the page's source
• Contact
  • Submit button stays visible after validation errors
  • Embedded Wufoo form loads with email and message fields
• Feeds and Meta
  • sitemap.xml lists the core pages, every show, and every episode
  • index.xml is valid RSS 2.0, newest first, starting at the newest episode
  • Open Graph / Twitter tags (title, URL, canonical, description, image) on home, show, episode, host, and community pages
• llms.txt (from Add llms.txt support: auto-updating indexes + markdown page twins #1145, reworked to match the rest of the suite)
  • /llms.txt structure; Shows section matches the navbar; every show listed in the right section with its episode count and RSS link
  • /llms-full.txt has no front matter or HTML escaping
  • Per-show llms.txt and episodes.json (all 8 shows) match the episode files in content/
  • Markdown twins (index.md) for shows, sampled episodes, hosts, people, and guests, served as text/markdown
  • describedby / markdown alternate links in page heads
  • llms.txt only exists on show sections (/show/, /hosts/, /guests/ return 404)
• Calendar
  • Calendar defaults to the browser's timezone
  • Choosing a timezone updates the calendar; re-selecting the placeholder keeps the current one
• Membership
  • Every link in content/membership/index.md renders with the same text and href
• Matrix
  • /.well-known/matrix files exist with the federation details
• Live
  • Embedded player (HLS player: off-air overlay, "Try Again" reconnect; stream is stubbed so tests don't depend on JB being live)
  • Audio/video toggle swaps the players and relabels itself, clicked as a button by its accessible name
  • Way to mock JB being live to check live light? Obsolete: the navbar live indicator is no longer loaded since the live page moved to HLS. Replaced by stubbing the HLS stream above.
• Mobile and tablet (Galaxy S9+ 320px, iPhone 15, Pixel 7, iPad Mini, iPad Pro 11, iPad Pro 11 landscape; each on its real engine: iOS on WebKit, Android on Chromium)
  • Burger menu opens and closes, shows the nav links, and updates aria-expanded
  • No sideways scrolling on home, show, episode and /people/
  • Episode page shows exactly one sponsor block, the mobile one below 769px
  • Episode player fits the screen

Also in this PR

• Tests run in parallel (-n auto) with colored output; screenshots only on failure
• Per-episode tests are marked episodes. On PRs, CI runs them unless every changed file is on a known-safe list (docs, picks, non-episode content/layouts, other tests). Pushes to main/develop and manual runs always run everything
• Expected values come from content/, data/ and config rather than hardcoded counts, so tests keep passing as content changes
• make dev-tests: dev server with --disableFastRender. Plain make dev renders pages one at a time on first request (~2s each), which makes parallel runs time out. Run make dev-tests, then make tests
• Mobile tests save a screenshot on failure, and CI now uploads screenshots even when the test step fails (it was skipped before)
• Fixes found by the new tests:
  • /people/ scrolled sideways at 320px: the 2-per-row grid now only applies from 376px to 599px, and narrower phones get one card per row
  • Home show slider sprang back on a held drag at most widths: longSwipesRatio: 0.1 in main.js
  • Navbar burger never updated aria-expanded
  • Live audio/video toggle was a role="switch" with an inverted aria-checked; it's now a plain button, since video vs audio isn't on/off
• Default og:image changed to /images/logo.png; the old /images/jupiter-broadcasting.png didn't exist
• Bumped actions/checkout (v7), actions/upload-artifact (v7) and dorny/test-reporter (v3)

Known failures (marked xfail, will flag when fixed)

• Coder Radio show page links host Michael to /guests/michael/ because content/people/michael.md has type: guest
• Tags containing # link to a URL fragment instead of the tag page (partials/episode/tags.html doesn't escape #)
• Tags containing & or - link to a double-hyphen URL that doesn't exist (partials/episode/tags.html builds the URL by hand)
• /people/ links joe, jnsgruk, joeres and michaetunnell to pages that don't exist: they have no episodes. They were re-added by the people scraper; removal is handled separately

Skipped

• Emoji tag tests: emoji-only tags have no tag page at all (🦒 giraffe emoji tags page silently failing to build #516)

[gerbrent]

• Should we iterate over all shows or just pick 1 since it is templated?

Feels asthough we don't need to iterate over EVERY episode, but only 1 seems almost too restrictive and could make dangerous assumptions (i.e. if $latest_episode is OK, they're all OK!). Suggestion: testing the latest 5 episodes and the oldest 5 episodes?

• Way to mock JB being live to check live light?

Maybe this is a helpful starting point? #378 (comment)

[CGBassPlayer]

Suggestion: testing the latest 5 episodes and the oldest 5 episodes?

Do you think 5 newest across all shows or per show?

Also I would advocate for the possibility of picking a specific set of episodes that we should test because not all episodes have guests and I want make sure the Guests section of the episode page is check every time when E2E is run.

[gerbrent]

great points @CGBassPlayer !

• 5 newest and 5 oldest of every show (since some shows are unique in their strangeness.. who knows what we'll find!)
• in addition, we could/should also pick some known-useful episodes, as you mentioned. great idea.

[CGBassPlayer]

...known-useful episodes...

I think that LUP 434 is a great candidate and one I use when working on the episode page layout

[elreydetoda]

Also, I've listed out a bunch of other things/info here: #286 (comment)

You've got most of it with your initial comment for this PR, but just figured I'd toss it in to help with some of the things like mocking the request.

[elreydetoda]

...known-useful episodes...

I think that LUP 434 is a great candidate and one I use when working on the episode page layout

Also, here are some episodes that were particularly tricky from the scrapers perspective. We might want to include those as well:

https://github.com/JupiterBroadcasting/show-scraper/blob/48f0833e63a8473eb8afa6d2297942aeb487b139/scraper.py#L524-L537

[CGBassPlayer]

So I managed to find a way to build the urls for the episodes we want. I grabs the 5 oldest, and 5 newest episodes per show as well as a whatever we populate in a dictionary in the file. I will look through to see what other edge cases in the issues @elreydetoda pointed out. But the part I think would be the hardest is over. Testing time locally took like 7 minutes through the first run, looking at GH Actions it seems to have taken only 30 seconds.

[elreydetoda]

BTW @CGBassPlayer, once this stuff gets merged: ChanceM#1

We'll have the live page indicator + mocking (mainly just intercepting the live request and responding with an event and not actual mocking) taken care of.

[CGBassPlayer]

Not sure why they PR closed when I pulled the latest changes in... My best guess would be because I rebased my changes on top and so the commit history for my branch has changed? No totally sure. But seems like it good again