Produce the two files a crawler asks for, and refuse a sitemap that drifts [#69]

[iderex]

Part of #69, and short of its whole done condition. The last section says what
is left and why.

What was wrong

Two files get requested whether or not any page links them, and the build wrote
neither, so each request was answered with the not-found page:

go run . build | grep -c -E 'robots|sitemap'
0

Run 2026-08-13 at 269d93e. The reader's host serves a request for nothing, and
the client that asked gets none of what it asked for.

What the change does

The build writes robots.txt and sitemap.xml, last, after everything that can
put a page into the output:

go run . build | grep '^wrote'
wrote dist/index.html (1977 bytes)
wrote dist/privacy/index.html (4199 bytes, 2 checked, 3 promised, 2 residual)
wrote dist/legal/index.html (3489 bytes, 0 of 3 answered)
wrote dist/404.html (2051 bytes, 1 way(s) onward)
wrote dist/.well-known/security.txt (379 bytes)
wrote dist/sitemap.xml (259 bytes, 3 address(es) listed)
wrote dist/robots.txt (216 bytes, 1 sitemap(s) named)

cat dist/sitemap.xml
<?xml version="1.0" encoding="UTF-8"?>
<urlset xmlns="http://www.sitemaps.org/schemas/sitemap/0.9">
  <url><loc>https://flowfin.dev/</loc></url>
  <url><loc>https://flowfin.dev/legal/</loc></url>
  <url><loc>https://flowfin.dev/privacy/</loc></url>
</urlset>

cat dist/robots.txt
# Nothing on this site is kept out of an index. This file exists so that a
# crawler asking for it is answered rather than served the not-found page.
User-agent: *
Disallow:

Sitemap: https://flowfin.dev/sitemap.xml

Three decisions are in the change rather than under it, each with its reason at
the code.

The not-found page is the one produced page the sitemap leaves out. A sitemap is
a list of addresses a crawler is invited to fetch and index, and that page is
served in answer to addresses that are not its own, so listing it asks for an
index entry that sends a reader to an error page under an address this site
claims to have.

Neither file carries a date. A sitemap may state when each page last changed,
and a build writing today's date into one produces different bytes from
unchanged source on two days. That is checked rather than asserted:

go run . reproduce | tail -1
  7 file(s), identical in both builds

The exclusion file names the sitemap only where one was written, because a
robots file pointing at an address the build did not produce sends the one
client that reads it to the not-found page.

Which pages a sitemap carries is one exported function, read by the writer and
by the leg that compares the two. Stating the rule in both places would state it
twice, and the second statement would agree with the first on the day it was
written and never be checked against it again.

The leg, and what it refuses

go run . ci | head -1
gate: 7 legs, in order: format, vet, test, build, links, sitemap, invariants
go run . ci | grep '  sitemap:'
  sitemap: every page the build wrote is listed once, and every entry has a page behind it

The file is generated, so on the day it is written the two agree by construction
and a comparison over them proves nothing. What it refuses is the day after, and
both directions are shown against this tree rather than against a fixture.

A page listed nowhere, produced by changing the writer to drop one address. This
is the failure every other check passes: the page is valid, it is served, and
nothing tells a crawler it is there.

go test ./internal/sitemap -run TestTheTreeAgreesWithItsOwnSitemap -count=1 -v
=== RUN   TestTheTreeAgreesWithItsOwnSitemap
    sitemap_test.go:111: sitemap: 1 disagreement(s) between the list and the output
        sitemap: 2 address(es) listed, against 3 page(s) the build produced
          the build produced a page served at https://flowfin.dev/ and sitemap.xml lists no entry for it, so nothing ever tells a crawler the page is there
--- FAIL: TestTheTreeAgreesWithItsOwnSitemap (0.01s)

An address with no page behind it, produced by appending one to the writer:

go test ./internal/sitemap -run TestTheTreeAgreesWithItsOwnSitemap -count=1 -v
=== RUN   TestTheTreeAgreesWithItsOwnSitemap
    sitemap_test.go:111: sitemap: 1 disagreement(s) between the list and the output
        sitemap: 4 address(es) listed, against 3 page(s) the build produced
          sitemap.xml lists https://flowfin.dev/install/, and the build wrote no page served at that address, so a crawler following it is answered with the not-found page
--- FAIL: TestTheTreeAgreesWithItsOwnSitemap (0.02s)

Both edits were made to the writer and taken out again; the branch carries
neither. The same two directions are decided cases in the suite, together with a
page listed twice, which both directions above pass over because each entry has
a page behind it.

The leg walks the output directory rather than reading the list of paths the
build reported. A walk that trusted the writer's own account of what it wrote
would agree with the writer about a page the writer never mentioned.

What this moves in the existing suite

Five cases asserted how many paths a build reported, which is a statement about
the whole set of files the build writes. Every new writer moves that number and
none of those cases is about it, so each now asks whether the build reported the
file the case is about. The one about an absolute output directory asks instead
that every reported path is under the directory it was given, which is what that
case was for.

One count is deliberately left as a count. The reproducibility neighbour states
three files, because the two new ones are the likeliest in this tree to become
unreproducible and that case is where it would be caught.

The gate

go run . ci
gate: 7 legs, in order: format, vet, test, build, links, sitemap, invariants
  needs-network was not asked for. Asking costs a request to the public name from whatever machine runs it, and a verdict that moves when somebody else's service does rather than when this tree changes. Ask with: go run ./harness/needs-network
  format: ok, 48 file(s)
  vet: ok
  test: ok, 23 test file(s)
  build: ok, 7 file(s)
  links: every reference that stays inside this site resolves to a file the build wrote
  sitemap: every page the build wrote is listed once, and every entry has a page behind it
  invariants: ok, 34 rule(s) decided, 1 owed and not decided
7 of 7 legs ran. None was skipped.

go run . hygiene origin/main HEAD | tail -1
1 commit(s) judged, none refused.

All commands run 2026-08-13 at fe91b05.

One verb reds on this branch and it reds identically on main, for a reason
this change does not touch:

go run . changelog | tail -1
CHANGELOG.md carries no section for 0.1.0, so this release would be one nobody described. Write the section, moving what belongs to it out of Unreleased, before the run that creates the tag

Run 2026-08-13 on this branch and again with the branch stashed. It is not a leg
of the gate and it is about the release run, which #53 holds.

The means

Go, and the same shapes the tree already carries: a writer beside the other page
writers, a package with a pure decision the suite drives directly, and a leg in
the gate. It adds no language, no runtime and no dependency, and every claim
above is a command in a suite that already exists.

What is not covered

The icon. #69 asks for three files and this lands two. What the icon shows is
entry 7 of #7 and is not settled, so shipping one here would answer that entry
by drawing it. The clause about serving the produced files over a real host is
also untouched, and the byte budget clause reaches the icon rather than these
two.

Reader

This change carries no second reader. The evidence above is in its place: every
claim is a command run at fe91b05, and the two directions the new leg refuses
are shown reddening this tree and then passing with the same bytes restored.