Skip to content

The sitemap said all 53 pages changed today, every day - #182

Merged
ChelseaKR merged 1 commit into
mainfrom
fix/sitemap-lastmod-from-content-change
Sep 18, 2026
Merged

ChelseaKR merged 1 commit into
mainfrom
fix/sitemap-lastmod-from-content-change

Conversation

@ChelseaKR

Copy link
Copy Markdown
Owner

What was wrong

Measured on the live host, not in the generator. On 2026-09-13 https://sprout.chelseakr.com/sitemap.xml carried <lastmod>2026-09-13</lastmod> on all 53 of 53 entries — the day the docs were built. It would have said 2026-09-14 on all 53 the next morning, whether or not a single page changed.

The pages do not agree. /PUBLICATION-READINESS/'s source had not been touched since 2026-08-16; /ARCHITECTURE/'s since 2026-06-22. <lastmod> is the one element in a sitemap that is a factual claim about the content, and this one was a claim about the CI runner — this repository's own "absence rendered as a value" shape, with "we did not measure this per page" published as today's date.

A note on the issue's table: the 2026-05-01 it reports as the newest date visible on /ADAPT/ is a fetch_date inside an example config snippet, not a date the page states about itself. The honest per-page date is the commit that last changed the page's source, and that is what this uses.

What changed

docs_hooks/sitemap_lastmod.py dates each page from git log -1 --format=%cs -- <source>, and theme_overrides/sitemap.xml (a one-file custom_dir, mkdocs' own template with one element changed) emits <lastmod> only for a page it could date. <lastmod> is optional in the sitemaps protocol, so an omitted element is a valid sitemap that says nothing false, which a stamped-today one is not.

The same build now publishes 52 dates across 16 distinct days, 2026-06-22 … 2026-09-07 — and none of them is the build's own date.

/ is deliberately left undated. docs-pages copies web-static/public/ over the mkdocs output, so the document served at the root is the zero-server reference surface, not docs/index.md. Dating that URL from docs/index.md's history would publish a change date for a document nobody receives — the same defect in different clothes.

A shallow checkout is refused rather than consulted. git log -1 -- <path> does not fail in a depth-1 clone: it answers for every path with the tip commit's date, because relative to no parent that single commit introduces the whole tree. Nothing about the reply is malformed, so asking-then-sanity-checking would have reinstated the build date under a new name. The hook asks git rev-parse --is-shallow-repository first, once, and publishes nothing from a checkout that cannot answer. The three jobs that build the docs (ci.yml's docs, pages.yml, live-integrity.yml) now check out with full history and say in place why.

Two voices, because an omission is honest and silent. The build prints 52 of 53 pages state when they last changed and names every page it could not date. sprout site-check — a merge gate via make verify, and a step of docs-pages — now fails when a published <lastmod> is malformed, impossible, or in the future, and when none of the entries carries one, which is exactly what a lost fetch-depth: 0 produces. Its success line carries both numbers.

The count is checked against the file rather than trusted. The hook computes dates; the template publishes them; the first cannot see the second. Measured while writing this: with the override removed and the hook left in place, the build printed 52 of 53 pages state when they last changed while the file it had just written carried the build date on all 53 — a reassuring number beside the exact artifact it was wrong about. on_post_build now reads the sitemap back and fails the build unless its dates are precisely the derived ones, compared as multisets so a fallback that repeats one date cannot pass on a count.

And the live sentinel can now see it. scripts/verify_live_site.py used to strip every <lastmod> before comparing the deployed sitemap.xml, because those elements were the build's own date. They are now a property of the commit, so the file is compared byte for byte, dates included — the elements the old exclusion removed are precisely the ones this change corrects, and a sentinel blind to them could not tell an honest deployed sitemap from a build-stamped one. sitemap.xml.gz keeps its one documented exclusion (a gzip MTIME) and is still checked by decompressing it.

How it was verified

make verify ran green through the pre-push hook (lint · mypy strict · tests ≥90% · security · eval · smoke · a11y · site-check · claims · calibrate · gate-inventory · slo · corpus-report · propose-check · docs · zizmor · ci-parity · web-static). sprout check-tuning-scope --base origin/main reports no tunable-surface change — nothing here touches src/sprout/providers/ or any other tuned path, so no Tunes-Against: trailer is claimed.

Four negative controls, each with a git-object baseline taken before the sabotage and re-asserted byte-identical after, and __pycache__ cleared between runs. The fixture's content dates (2026-05-01, 2026-06-22) are deliberately months older than the day it is read on, so a gate that only checks the shape of a date cannot pass by accident.

control sabotage result
the zero-coverage rule dated == 0 → dated == -1 test_a_sitemap_that_dates_nothing_at_all_is_reported red
the future-date rule stamped > today → stamped > date(2999, 1, 1) test_a_date_in_the_future_is_reported red
the shallow-checkout refusal the literal "false" → "true" the shallow-clone test and the happy-path test both red
the whole fix, end to end remove theme_overrides/sitemap.xml the rebuilt sitemap carried 2026-09-13 on all 53; the build aborted under --strict naming all 16 derived dates it did not publish

The fourth is the one worth reading: it is what exposed that the hook's count and the published file are two different things, and it is why the read-back check exists. The shallow-clone test is a real git clone --depth 1 and asserts both halves — that the naive per-path question does come back with the tip's date there (naive == REPO_TIP != PAGE_CHANGED), and that the hook refuses it.

Closes #181.

Nothing about the web-static root page's own date is settled here; it is listed above as deliberately undated.

Prepared with AI assistance; reviewed before submission.

@ChelseaKR

Copy link
Copy Markdown
Owner Author

Independent verification of this PR's central claim — reproduced, with one correction to the body

Arrived here from #181 under a separate SEO sweep, found this open, and verified rather than duplicated. No competing branch was created and nothing here was pushed.

The defect is still live and still on main. Built origin/main (dc85850) from a clean worktree and a fresh venv:

site/sitemap.xml -> 53 <loc>, 53 <lastmod>, all of them 2026-09-13 (today)

and the deployed file agrees — https://sprout.chelseakr.com/sitemap.xml, fetched today, 53 of 53 entries reading 2026-09-13.

This branch fixes it, and the numbers reproduce exactly. Built 4583d60 the same way:

INFO - sitemap <lastmod>: 52 of 53 pages state when they last changed
INFO - sitemap <lastmod>: no recorded change date for index.md

52 URLs carry a defensible lastmod; 53 URLs are examinable. 16 distinct days, 2026-06-22 … 2026-09-07, and zero entries carrying the build's own date. The single undated URL is https://sprout.chelseakr.com/ — the deliberate web-static root exclusion this PR documents.

Spot-checked four entries against git rather than against the emitted file, and checked committer and author dates separately, since a rebase resets the former and would have made a stale page look freshly changed:

page git log -1 --format=%cs %as sitemap
/PUBLICATION-READINESS/ 2026-08-16 2026-08-16 2026-08-16
/ARCHITECTURE/ 2026-09-06 2026-09-06 2026-09-06
/ADAPT/ 2026-09-06 2026-09-06 2026-09-06
/RESEARCH-ROADMAP/ 2026-09-06 2026-09-06 2026-09-06

Committer and author agree throughout, so these are real content changes, not rebase artifacts.

The headline negative control, re-run from scratch

step result
baseline tree hash 9c8b3a1f; theme_overrides/sitemap.xml object 8a0312fd; custom_dir present once in mkdocs.yml; 5 lastmod references in the override
sabotage override removed — file absent, git status shows D theme_overrides/sitemap.xml, 0 files under theme_overrides/ containing lastmod
result build aborted, exit 1 under --strict: "the sitemap this build wrote carries 53 date(s) but this hook derived 52 … Published and not derived: 2026-09-13. Derived and not published: 2026-06-22, … 2026-09-07". The file it wrote before aborting carried 2026-09-13 on all 53.
restore object hash back to 8a0312fd, tree hash back to 9c8b3a1f, byte-identical; __pycache__ cleared either side

The read-back check is load-bearing: it caught the exact state where the hook's own count (52 of 53) was reassuring and the file beside it was wrong on all 53. That is the strongest part of this change.

One correction to the PR body

/ARCHITECTURE/'s since 2026-06-22

Measured, its source last changed on 2026-09-06 (865e074, #135), which is what this branch's own build publishes. 2026-06-22 is the newest date visible on the rendered page, which is the column #181's table reported — a different quantity, and the one the PR body elsewhere correctly declines to use. The code is right; only this example is. /PUBLICATION-READINESS/'s 2026-08-16 in the same sentence checks out.

State, for whoever picks this up

mergeStateStatus is DIRTY and no checks have reported. The branch is behind main by fa9f4a7 (#183) and dc85850 (#184); the overlap is .github/workflows/ci.yml, CHANGELOG.md and src/sprout/cli.py. Worth noting for the rebase: dc85850 touched docs/PUBLICATION-READINESS.md, so that page's honest date legitimately becomes 2026-09-13 afterwards, and the 16-distinct-days figure in the body will shift. The two headline numbers (52 / 53) should not.

Prepared with AI assistance; reviewed before submission.

mkdocs fills <lastmod> from get_build_date(), so every entry carried the day the
docs were built. Measured live 2026-09-13: 53 of 53 read 2026-09-13, including a
page whose source had not moved since 2026-08-16.

Each page is now dated from the commit that last changed its own source, and a
page with no such commit carries no element at all. A shallow checkout is refused
outright rather than consulted: `git log -1 -- <path>` answers there for every
path with the tip's date, which would have reinstated the build date under a new
name.

The build prints how many of how many pages it could date and names the rest;
`sprout site-check` fails on a malformed, impossible or future date, and on a
sitemap that dates nothing at all. The live sentinel no longer strips <lastmod>
before comparing, so the deployed dates are now compared byte for byte.
@ChelseaKR
ChelseaKR force-pushed the fix/sitemap-lastmod-from-content-change branch from 4583d60 to 2870bcc Compare September 18, 2026 19:16
@ChelseaKR
ChelseaKR merged commit 0b79b1b into main Sep 18, 2026
16 checks passed
@ChelseaKR
ChelseaKR deleted the fix/sitemap-lastmod-from-content-change branch September 18, 2026 19:24
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

sitemap.xml stamps the build date as lastmod on all 53 pages, up to 4.5 months of freshness the pages do not have

1 participant