From e42dd298f62089b411760d4bb1d7274c0fd36bf8 Mon Sep 17 00:00:00 2001 From: Pranavkumar Patel Date: Sun, 4 Oct 2026 12:01:19 -0400 Subject: [PATCH] Rebuild the landmark data every week, and put it on the site GeoNames rewrites its dump every day, and the landmark archive, the name index and the app's place-name packs were only rebuilt when somebody ran the Landmarks workflow. It now also runs on Mondays at 05:43 UTC, two hours after the day's dump is written. A scheduled run has no inputs, so the level and slice defaults are given again where they are used, now through the environment rather than pasted into the script. Nobody reads a scheduled run before it publishes, so a new step does: it measures what was built against what is published, the landmarks, their shards, the packed names and the countries with a pack, and refuses to publish a fall of more than one part in twenty or a country with no pack. A dump cut short builds without an error, and this is what would catch it. A smaller archive on purpose is published by hand, with the new "Publish even if the data shrank" box ticked. Tried against the published data: a normal week passes, a tenth fewer landmarks and one lost country are refused, a fall of 4.9% passes, and a repository with nothing published yet publishes. The site fetches the archive from the release, so a new one changed nothing in the tree and never deployed. The Pages workflow now follows every Landmarks run and builds after one that published, so the site and the packs move together. The workflow README, web/README.md and PLAY.md say when it runs. --- .github/workflows/README.md | 6 ++- .github/workflows/landmarks.yml | 87 ++++++++++++++++++++++++++++++--- .github/workflows/pages.yml | 10 ++++ android/PLAY.md | 2 +- web/README.md | 18 ++++--- 5 files changed, 105 insertions(+), 18 deletions(-) diff --git a/.github/workflows/README.md b/.github/workflows/README.md index 110094f..00ab96c 100644 --- a/.github/workflows/README.md +++ b/.github/workflows/README.md @@ -6,9 +6,11 @@ | `release-python.yml` | a `v*` tag | Builds and publishes the Python package to PyPI | | `release-npm.yml` | a `v*` tag | Builds and publishes the TypeScript package to npm | | `release-nuget.yml` | a `v*` tag | Builds and publishes the C# package to NuGet | +| `landmarks.yml` | Mondays at 05:43 UTC, and by hand | Builds the landmark archive, the name index and the app's place-name packs from the day's GeoNames dump, and publishes them as the `landmarks` and `place-packs` releases, unless they shrank against what is published | +| `pages.yml` | a push to main that touches the site, a Landmarks run that published, and by hand | Builds the website with the published landmark data and deploys it to GitHub Pages | -Nothing publishes on a push to a branch. A release happens because a version -tag was pushed, and for no other reason. +No package publishes on a push to a branch. A package release happens because a +version tag was pushed, and for no other reason. ## What CI proves that a single port's tests cannot diff --git a/.github/workflows/landmarks.yml b/.github/workflows/landmarks.yml index 5c77b7d..e8fb68a 100644 --- a/.github/workflows/landmarks.yml +++ b/.github/workflows/landmarks.yml @@ -20,11 +20,20 @@ # say which references are near enough -- and outside that range recovery does # not fail, it returns somewhere else entirely. # -# It is a job of its own, run by hand, for two reasons. It downloads 400 MB and -# spends five minutes on it, which is not a thing to do on every push to a -# stylesheet. And the data changing is a decision rather than a side effect: a -# new archive here does not reach the site until the next Pages run, which is -# the right way round for something a reader is asked to trust. +# It is a job of its own because it downloads 400 MB and spends five minutes on +# it, which is not a thing to do on every push to a stylesheet. It runs once a +# week, since GeoNames rewrites its dump every day and a week of edits is worth +# having, and by hand whenever the data is wanted sooner. A run that publishes +# starts the Pages workflow, so the site and the packs move together. +# +# Nobody reads a scheduled run before it publishes, so the run reads itself: +# the new data is measured against what is published and refused if it shrank +# (see "Compare with what is published"). Shrinking on purpose stays possible, +# by hand, as a decision rather than a side effect. +# +# GitHub stops the schedule of a public repository after 60 days with no +# activity in it, and says so by email first. Turn it back on under Actions, +# Landmarks. # # Actions are pinned to a commit rather than a tag: a tag can be moved to point # at different code, a commit cannot. The release itself is made with `gh`, @@ -43,6 +52,15 @@ on: description: 'Passes over the dump; more passes, less memory' required: false default: '8' + shrink: + description: 'Publish even if the data shrank' + type: boolean + required: false + default: false + schedule: + # Mondays, two hours after GeoNames writes the day's dump (about 03:35 UTC), + # and clear of the hour, when scheduled runs queue longest. + - cron: '43 5 * * 1' # Enough to write the release, and nothing else. permissions: @@ -88,14 +106,18 @@ jobs: echo "rows: $(wc -l < allCountries.txt)" # Slices keep the peak under about 2 GB. Without them this needs more - # than 4 and is one runner change away from failing. + # than 4 and is one runner change away from failing. A scheduled run has + # no inputs, so the defaults are said twice, here and above. - name: Build the shards + env: + LEVEL: ${{ inputs.level || '4' }} + SLICES: ${{ inputs.slices || '8' }} run: | node scripts/build-landmarks.mjs \ --geonames ../geonames \ --out landmarks \ - --level '${{ inputs.level }}' \ - --slices '${{ inputs.slices }}' | tee ../report.txt + --level "$LEVEL" \ + --slices "$SLICES" | tee ../report.txt working-directory: web # The other direction. The archive answers "what is near this point"; this @@ -132,6 +154,55 @@ jobs: tar -czf names.tar.gz -C web/names . ls -l landmarks.tar.gz names.tar.gz + # A dump cut short, or a change in its format, builds without an error + # and would publish a world with most of its places missing, to the site + # and to every phone that keeps a pack. So the new data is measured + # against what is published now: the landmarks, their shards, the names + # packed for the app, and every country with a pack. A week of GeoNames + # edits moves these by thousands in millions; a fall of one in twenty is + # not an edit, and nor is a country with no pack. Failing here publishes + # nothing, and the release stays as it was. + - name: Compare with what is published + if: ${{ !inputs.shrink }} + env: + GH_TOKEN: ${{ github.token }} + run: | + mkdir before + if ! gh release download place-packs --pattern packs.json --dir before; then + echo 'Nothing published yet, so nothing to compare with.' >> "$GITHUB_STEP_SUMMARY" + exit 0 + fi + gh release download landmarks --pattern landmarks.tar.gz --output - \ + | tar -xzO ./manifest.json > before/manifest.json + node -e ' + const fs = require("fs"); + const read = (file) => JSON.parse(fs.readFileSync(file, "utf8")); + const lines = (packs) => Object.values(packs.countries).reduce((sum, c) => sum + c.lines, 0); + const was = { archive: read("before/manifest.json"), packs: read("before/packs.json") }; + const now = { archive: read("web/landmarks/manifest.json"), packs: read("web/packs/packs.json") }; + const measured = [ + ["landmarks", was.archive.landmarks, now.archive.landmarks], + ["shards", was.archive.shards, now.archive.shards], + ["packed names", lines(was.packs), lines(now.packs)], + ["country packs", Object.keys(was.packs.countries).length, Object.keys(now.packs.countries).length], + ]; + const summary = ["| | published | built | change |", "| --- | ---: | ---: | ---: |"]; + const refused = []; + for (const [what, before, after] of measured) { + const change = ((after - before) / before) * 100; + summary.push(`| ${what} | ${before.toLocaleString("en")} | ${after.toLocaleString("en")} | ${change.toFixed(2)}% |`); + if (after < before * 0.95) refused.push(`${what} fell from ${before} to ${after}`); + } + const gone = Object.keys(was.packs.countries).filter((cc) => !(cc in now.packs.countries)); + if (gone.length) refused.push(`no pack any more for ${gone.join(", ")}`); + fs.appendFileSync(process.env.GITHUB_STEP_SUMMARY, summary.join("\n") + "\n\n"); + console.log(summary.join("\n")); + for (const reason of refused) { + console.log(`::error::${reason}. Nothing was published. If the data really did shrink, run this by hand with "Publish even if the data shrank".`); + } + if (refused.length) process.exit(1); + ' + # One moving tag rather than a dated one, so the Pages build has a fixed # name to ask for. Asking for `latest` would hand it whichever release is # newest, and the ports publish releases here too. diff --git a/.github/workflows/pages.yml b/.github/workflows/pages.yml index 94474eb..68cfea1 100644 --- a/.github/workflows/pages.yml +++ b/.github/workflows/pages.yml @@ -53,6 +53,13 @@ on: - 'typescript/README.md' - '.github/workflows/pages.yml' workflow_dispatch: + # The landmark archive and the name index are fetched from a release, so a + # new one changes nothing in this tree and no push would bring it here. The + # Landmarks workflow rebuilds them weekly; this follows each run, and builds + # only after one that published (see `if:` on the build job). + workflow_run: + workflows: [Landmarks] + types: [completed] # Read the tree, mint the token Pages needs to accept a deployment, and nothing # else. The build job below drops even the read permission it does not use. @@ -71,6 +78,9 @@ jobs: build: name: Build + # A Landmarks run that failed or was refused published nothing, and the + # site already has what is published. + if: github.event_name != 'workflow_run' || github.event.workflow_run.conclusion == 'success' runs-on: ubuntu-latest permissions: contents: read diff --git a/android/PLAY.md b/android/PLAY.md index b6c9c84..ba6addf 100644 --- a/android/PLAY.md +++ b/android/PLAY.md @@ -46,7 +46,7 @@ changes that page in the same pull request; it changes the already apart, the phone from 1, the watch from 1001 and the car from 2001, and each is raised with every upload of that bundle. - [ ] **The place-name packs** are live as the `place-packs` release. They are - published by the Landmarks workflow, run by hand. + published by the Landmarks workflow, every Monday and by hand. - [ ] **Store listing, graphics and screenshots**, as [below](#store-listing). The icon, the feature graphic and the screenshots, phone, 7-inch and 10-inch tablet, desktop, watch and Android Automotive OS, are made. The Android XR diff --git a/web/README.md b/web/README.md index 93571a0..076e552 100644 --- a/web/README.md +++ b/web/README.md @@ -322,13 +322,17 @@ integration that serves the files in development and copies them into the output at the end of a build. `landmarks/` is generated and **not committed**. It is built by -[`.github/workflows/landmarks.yml`](../.github/workflows/landmarks.yml), run by -hand, and published as a release asset under the `landmarks` tag. The Pages -build downloads that asset; if there is none it warns and carries on, and the -reference list says it could not be loaded. - -Rebuilding the data is therefore a decision rather than a side effect: a new -archive does not reach the site until the next Pages run. +[`.github/workflows/landmarks.yml`](../.github/workflows/landmarks.yml), every +Monday and by hand, and published as a release asset under the `landmarks` tag. +The Pages build downloads that asset; if there is none it warns and carries on, +and the reference list says it could not be loaded. + +A run that publishes starts the Pages workflow, so a new archive reaches the site +the same morning. Before publishing, the run measures what it built against what +is published, and refuses data that shrank by more than one part in twenty or +lost a country, since a dump cut short builds without an error. Publishing a +smaller archive on purpose is done by hand, with the workflow's "Publish even if +the data shrank" box ticked. ## Working offline