Rebuild the landmark data every week, and put it on the site

.github/workflows/README.md

@@ -6,9 +6,11 @@
 | `release-python.yml` | a `v*` tag | Builds and publishes the Python package to PyPI |
 | `release-npm.yml` | a `v*` tag | Builds and publishes the TypeScript package to npm |
 | `release-nuget.yml` | a `v*` tag | Builds and publishes the C# package to NuGet |
+| `landmarks.yml` | Mondays at 05:43 UTC, and by hand | Builds the landmark archive, the name index and the app's place-name packs from the day's GeoNames dump, and publishes them as the `landmarks` and `place-packs` releases, unless they shrank against what is published |
+| `pages.yml` | a push to main that touches the site, a Landmarks run that published, and by hand | Builds the website with the published landmark data and deploys it to GitHub Pages |
 
-Nothing publishes on a push to a branch. A release happens because a version
-tag was pushed, and for no other reason.
+No package publishes on a push to a branch. A package release happens because a
+version tag was pushed, and for no other reason.
 
 ## What CI proves that a single port's tests cannot
 

.github/workflows/landmarks.yml

@@ -20,11 +20,20 @@
 # say which references are near enough -- and outside that range recovery does
 # not fail, it returns somewhere else entirely.
 #
-# It is a job of its own, run by hand, for two reasons. It downloads 400 MB and
-# spends five minutes on it, which is not a thing to do on every push to a
-# stylesheet. And the data changing is a decision rather than a side effect: a
-# new archive here does not reach the site until the next Pages run, which is
-# the right way round for something a reader is asked to trust.
+# It is a job of its own because it downloads 400 MB and spends five minutes on
+# it, which is not a thing to do on every push to a stylesheet. It runs once a
+# week, since GeoNames rewrites its dump every day and a week of edits is worth
+# having, and by hand whenever the data is wanted sooner. A run that publishes
+# starts the Pages workflow, so the site and the packs move together.
+#
+# Nobody reads a scheduled run before it publishes, so the run reads itself:
+# the new data is measured against what is published and refused if it shrank
+# (see "Compare with what is published"). Shrinking on purpose stays possible,
+# by hand, as a decision rather than a side effect.
+#
+# GitHub stops the schedule of a public repository after 60 days with no
+# activity in it, and says so by email first. Turn it back on under Actions,
+# Landmarks.
 #
 # Actions are pinned to a commit rather than a tag: a tag can be moved to point
 # at different code, a commit cannot. The release itself is made with `gh`,
@@ -43,6 +52,15 @@ on:
         description: 'Passes over the dump; more passes, less memory'
         required: false
         default: '8'
+      shrink:
+        description: 'Publish even if the data shrank'
+        type: boolean
+        required: false
+        default: false
+  schedule:
+    # Mondays, two hours after GeoNames writes the day's dump (about 03:35 UTC),
+    # and clear of the hour, when scheduled runs queue longest.
+    - cron: '43 5 * * 1'
 
 # Enough to write the release, and nothing else.
 permissions:
@@ -88,14 +106,18 @@ jobs:
           echo "rows: $(wc -l < allCountries.txt)"
 
       # Slices keep the peak under about 2 GB. Without them this needs more
-      # than 4 and is one runner change away from failing.
+      # than 4 and is one runner change away from failing. A scheduled run has
+      # no inputs, so the defaults are said twice, here and above.
       - name: Build the shards
+        env:
+          LEVEL: ${{ inputs.level || '4' }}
+          SLICES: ${{ inputs.slices || '8' }}
         run: |
           node scripts/build-landmarks.mjs \
             --geonames ../geonames \
             --out landmarks \
-            --level '${{ inputs.level }}' \
-            --slices '${{ inputs.slices }}' | tee ../report.txt
+            --level "$LEVEL" \
+            --slices "$SLICES" | tee ../report.txt
         working-directory: web
 
       # The other direction. The archive answers "what is near this point"; this
@@ -132,6 +154,55 @@ jobs:
           tar -czf names.tar.gz -C web/names .
           ls -l landmarks.tar.gz names.tar.gz
 
+      # A dump cut short, or a change in its format, builds without an error
+      # and would publish a world with most of its places missing, to the site
+      # and to every phone that keeps a pack. So the new data is measured
+      # against what is published now: the landmarks, their shards, the names
+      # packed for the app, and every country with a pack. A week of GeoNames
+      # edits moves these by thousands in millions; a fall of one in twenty is
+      # not an edit, and nor is a country with no pack. Failing here publishes
+      # nothing, and the release stays as it was.
+      - name: Compare with what is published
+        if: ${{ !inputs.shrink }}
+        env:
+          GH_TOKEN: ${{ github.token }}
+        run: |
+          mkdir before
+          if ! gh release download place-packs --pattern packs.json --dir before; then
+            echo 'Nothing published yet, so nothing to compare with.' >> "$GITHUB_STEP_SUMMARY"
+            exit 0
+          fi
+          gh release download landmarks --pattern landmarks.tar.gz --output - \
+            | tar -xzO ./manifest.json > before/manifest.json
+          node -e '
+            const fs = require("fs");
+            const read = (file) => JSON.parse(fs.readFileSync(file, "utf8"));
+            const lines = (packs) => Object.values(packs.countries).reduce((sum, c) => sum + c.lines, 0);
+            const was = { archive: read("before/manifest.json"), packs: read("before/packs.json") };
+            const now = { archive: read("web/landmarks/manifest.json"), packs: read("web/packs/packs.json") };
+            const measured = [
+              ["landmarks", was.archive.landmarks, now.archive.landmarks],
+              ["shards", was.archive.shards, now.archive.shards],
+              ["packed names", lines(was.packs), lines(now.packs)],
+              ["country packs", Object.keys(was.packs.countries).length, Object.keys(now.packs.countries).length],
+            ];
+            const summary = ["| | published | built | change |", "| --- | ---: | ---: | ---: |"];
+            const refused = [];
+            for (const [what, before, after] of measured) {
+              const change = ((after - before) / before) * 100;
+              summary.push(`| ${what} | ${before.toLocaleString("en")} | ${after.toLocaleString("en")} | ${change.toFixed(2)}% |`);
+              if (after < before * 0.95) refused.push(`${what} fell from ${before} to ${after}`);
+            }
+            const gone = Object.keys(was.packs.countries).filter((cc) => !(cc in now.packs.countries));
+            if (gone.length) refused.push(`no pack any more for ${gone.join(", ")}`);
+            fs.appendFileSync(process.env.GITHUB_STEP_SUMMARY, summary.join("\n") + "\n\n");
+            console.log(summary.join("\n"));
+            for (const reason of refused) {
+              console.log(`::error::${reason}. Nothing was published. If the data really did shrink, run this by hand with "Publish even if the data shrank".`);
+            }
+            if (refused.length) process.exit(1);
+          '
+
       # One moving tag rather than a dated one, so the Pages build has a fixed
       # name to ask for. Asking for `latest` would hand it whichever release is
       # newest, and the ports publish releases here too.

.github/workflows/pages.yml

@@ -53,6 +53,13 @@ on:
       - 'typescript/README.md'
       - '.github/workflows/pages.yml'
   workflow_dispatch:
+  # The landmark archive and the name index are fetched from a release, so a
+  # new one changes nothing in this tree and no push would bring it here. The
+  # Landmarks workflow rebuilds them weekly; this follows each run, and builds
+  # only after one that published (see `if:` on the build job).
+  workflow_run:
+    workflows: [Landmarks]
+    types: [completed]
 
 # Read the tree, mint the token Pages needs to accept a deployment, and nothing
 # else. The build job below drops even the read permission it does not use.
@@ -71,6 +78,9 @@ jobs:
 
   build:
     name: Build
+    # A Landmarks run that failed or was refused published nothing, and the
+    # site already has what is published.
+    if: github.event_name != 'workflow_run' || github.event.workflow_run.conclusion == 'success'
     runs-on: ubuntu-latest
     permissions:
       contents: read

android/PLAY.md

@@ -46,7 +46,7 @@ changes that page in the same pull request; it changes the
   already apart, the phone from 1, the watch from 1001 and the car from 2001,
   and each is raised with every upload of that bundle.
 - [ ] **The place-name packs** are live as the `place-packs` release. They are
-  published by the Landmarks workflow, run by hand.
+  published by the Landmarks workflow, every Monday and by hand.
 - [ ] **Store listing, graphics and screenshots**, as [below](#store-listing).
   The icon, the feature graphic and the screenshots, phone, 7-inch and 10-inch
   tablet, desktop, watch and Android Automotive OS, are made. The Android XR

web/README.md

@@ -322,13 +322,17 @@ integration that serves the files in development and copies them into the output
 at the end of a build.
 
 `landmarks/` is generated and **not committed**. It is built by
-[`.github/workflows/landmarks.yml`](../.github/workflows/landmarks.yml), run by
-hand, and published as a release asset under the `landmarks` tag. The Pages
-build downloads that asset; if there is none it warns and carries on, and the
-reference list says it could not be loaded.
-
-Rebuilding the data is therefore a decision rather than a side effect: a new
-archive does not reach the site until the next Pages run.
+[`.github/workflows/landmarks.yml`](../.github/workflows/landmarks.yml), every
+Monday and by hand, and published as a release asset under the `landmarks` tag.
+The Pages build downloads that asset; if there is none it warns and carries on,
+and the reference list says it could not be loaded.
+
+A run that publishes starts the Pages workflow, so a new archive reaches the site
+the same morning. Before publishing, the run measures what it built against what
+is published, and refuses data that shrank by more than one part in twenty or
+lost a country, since a dump cut short builds without an error. Publishing a
+smaller archive on purpose is done by hand, with the workflow's "Publish even if
+the data shrank" box ticked.
 
 ## Working offline