Produce the files a browser and a crawler ask for without being linked

[iderex]

Depends on the landing page issue and on the not-found page issue.

Three files get requested whether or not any page links them, and none of them
is in this plan. A browser asks for the icon on the first visit to any page. A
crawler asks for the exclusion file before it asks for anything else. Both ask
for them again on every site they have not seen answer.

None of them answers today:

for p in /robots.txt /favicon.ico /sitemap.xml; do
  printf "%-36s " "https://flowfin.dev$p"
  curl -sS -o /dev/null -w "%{http_code}\n" "https://flowfin.dev$p"
done
https://flowfin.dev/robots.txt       404
https://flowfin.dev/favicon.ico      404
https://flowfin.dev/sitemap.xml      404

Run 2026-08-08. The tree behind that domain holds three files and no not-found
page:

gh api 'repos/Flowfin/hub/git/trees/HEAD?recursive=1' --jq '[.tree[].path|select(startswith("docs/"))]'
["docs/CNAME","docs/design-system.html","docs/index.html"]

so each of those requests is answered with the host's default page, which is
somebody else's markup on this project's domain. That half is #30. This issue is
the other half, which is producing the files rather than improving what happens
when they are missing.

What the build produces.

An icon, self-hosted, in the output and referenced from the head of every page.
Referencing it is what stops the browser guessing at /favicon.ico, and a
reference plus a file is one request either way. It is one of the few binary
assets in the tree, so it is counted against the byte budget in #34 like any
other image rather than being exempt because it is small.

What the icon shows is entry 7 of #7 and is not settled here. The project
publishes no image at all today, so this issue lands the plumbing against
whichever answer comes back: the file is produced by the build, referenced from
one place in the template, and replaced by changing one asset rather than every
page.

An exclusion file. This site has nothing it wants kept out of an index, so the
file is short, and its job is to say that plainly and to name the sitemap rather
than to hide anything. Not having one does not stop a crawler; it makes the
crawler ask and get an error page, which is a request the reader's host serves
for no reason.

A sitemap, generated. Twelve of the pages come from the roster, so a
hand-written list is wrong the day a row is added. It lists exactly the pages the
build produced and nothing else, which makes it checkable by the same walk over
the output that #31 uses to refuse a dead link: a sitemap entry with no produced
page behind it, and a produced page missing from the sitemap, are both failures.

All three are produced rather than committed. A committed file that describes
generated output is a second copy of the output that drifts from it, and the
reproducibility check in #42 covers what the build writes rather than what
somebody remembered to update.

A fourth file of the same class is #86, which is asked for at a fixed path by
somebody who found a security problem in the pages. It is a separate issue
because what it says comes from the security policy rather than from anything
here, and it is produced the same way for the same reason.

The head the icon is referenced from is the frame in #73, which is what makes that reference one line rather than one per page.

Done when

go run . build writes the icon, robots.txt and sitemap.xml into the output;
every produced page references the icon from its head; the sitemap lists exactly
the produced pages, with a test proving that an extra entry and a missing entry
each red the check; requesting each of the three on a served build returns the
file rather than the not-found page; and the icon is counted in the byte budget.