diff --git a/.github/workflows/README.md b/.github/workflows/README.md index 110094f..00ab96c 100644 --- a/.github/workflows/README.md +++ b/.github/workflows/README.md @@ -6,9 +6,11 @@ | `release-python.yml` | a `v*` tag | Builds and publishes the Python package to PyPI | | `release-npm.yml` | a `v*` tag | Builds and publishes the TypeScript package to npm | | `release-nuget.yml` | a `v*` tag | Builds and publishes the C# package to NuGet | +| `landmarks.yml` | Mondays at 05:43 UTC, and by hand | Builds the landmark archive, the name index and the app's place-name packs from the day's GeoNames dump, and publishes them as the `landmarks` and `place-packs` releases, unless they shrank against what is published | +| `pages.yml` | a push to main that touches the site, a Landmarks run that published, and by hand | Builds the website with the published landmark data and deploys it to GitHub Pages | -Nothing publishes on a push to a branch. A release happens because a version -tag was pushed, and for no other reason. +No package publishes on a push to a branch. A package release happens because a +version tag was pushed, and for no other reason. ## What CI proves that a single port's tests cannot diff --git a/.github/workflows/landmarks.yml b/.github/workflows/landmarks.yml index 5c77b7d..e8fb68a 100644 --- a/.github/workflows/landmarks.yml +++ b/.github/workflows/landmarks.yml @@ -20,11 +20,20 @@ # say which references are near enough -- and outside that range recovery does # not fail, it returns somewhere else entirely. # -# It is a job of its own, run by hand, for two reasons. It downloads 400 MB and -# spends five minutes on it, which is not a thing to do on every push to a -# stylesheet. And the data changing is a decision rather than a side effect: a -# new archive here does not reach the site until the next Pages run, which is -# the right way round for something a reader is asked to trust. +# It is a job of its own because it downloads 400 MB and spends five minutes on +# it, which is not a thing to do on every push to a stylesheet. It runs once a +# week, since GeoNames rewrites its dump every day and a week of edits is worth +# having, and by hand whenever the data is wanted sooner. A run that publishes +# starts the Pages workflow, so the site and the packs move together. +# +# Nobody reads a scheduled run before it publishes, so the run reads itself: +# the new data is measured against what is published and refused if it shrank +# (see "Compare with what is published"). Shrinking on purpose stays possible, +# by hand, as a decision rather than a side effect. +# +# GitHub stops the schedule of a public repository after 60 days with no +# activity in it, and says so by email first. Turn it back on under Actions, +# Landmarks. # # Actions are pinned to a commit rather than a tag: a tag can be moved to point # at different code, a commit cannot. The release itself is made with `gh`, @@ -43,6 +52,15 @@ on: description: 'Passes over the dump; more passes, less memory' required: false default: '8' + shrink: + description: 'Publish even if the data shrank' + type: boolean + required: false + default: false + schedule: + # Mondays, two hours after GeoNames writes the day's dump (about 03:35 UTC), + # and clear of the hour, when scheduled runs queue longest. + - cron: '43 5 * * 1' # Enough to write the release, and nothing else. permissions: @@ -88,14 +106,18 @@ jobs: echo "rows: $(wc -l < allCountries.txt)" # Slices keep the peak under about 2 GB. Without them this needs more - # than 4 and is one runner change away from failing. + # than 4 and is one runner change away from failing. A scheduled run has + # no inputs, so the defaults are said twice, here and above. - name: Build the shards + env: + LEVEL: ${{ inputs.level || '4' }} + SLICES: ${{ inputs.slices || '8' }} run: | node scripts/build-landmarks.mjs \ --geonames ../geonames \ --out landmarks \ - --level '${{ inputs.level }}' \ - --slices '${{ inputs.slices }}' | tee ../report.txt + --level "$LEVEL" \ + --slices "$SLICES" | tee ../report.txt working-directory: web # The other direction. The archive answers "what is near this point"; this @@ -132,6 +154,55 @@ jobs: tar -czf names.tar.gz -C web/names . ls -l landmarks.tar.gz names.tar.gz + # A dump cut short, or a change in its format, builds without an error + # and would publish a world with most of its places missing, to the site + # and to every phone that keeps a pack. So the new data is measured + # against what is published now: the landmarks, their shards, the names + # packed for the app, and every country with a pack. A week of GeoNames + # edits moves these by thousands in millions; a fall of one in twenty is + # not an edit, and nor is a country with no pack. Failing here publishes + # nothing, and the release stays as it was. + - name: Compare with what is published + if: ${{ !inputs.shrink }} + env: + GH_TOKEN: ${{ github.token }} + run: | + mkdir before + if ! gh release download place-packs --pattern packs.json --dir before; then + echo 'Nothing published yet, so nothing to compare with.' >> "$GITHUB_STEP_SUMMARY" + exit 0 + fi + gh release download landmarks --pattern landmarks.tar.gz --output - \ + | tar -xzO ./manifest.json > before/manifest.json + node -e ' + const fs = require("fs"); + const read = (file) => JSON.parse(fs.readFileSync(file, "utf8")); + const lines = (packs) => Object.values(packs.countries).reduce((sum, c) => sum + c.lines, 0); + const was = { archive: read("before/manifest.json"), packs: read("before/packs.json") }; + const now = { archive: read("web/landmarks/manifest.json"), packs: read("web/packs/packs.json") }; + const measured = [ + ["landmarks", was.archive.landmarks, now.archive.landmarks], + ["shards", was.archive.shards, now.archive.shards], + ["packed names", lines(was.packs), lines(now.packs)], + ["country packs", Object.keys(was.packs.countries).length, Object.keys(now.packs.countries).length], + ]; + const summary = ["| | published | built | change |", "| --- | ---: | ---: | ---: |"]; + const refused = []; + for (const [what, before, after] of measured) { + const change = ((after - before) / before) * 100; + summary.push(`| ${what} | ${before.toLocaleString("en")} | ${after.toLocaleString("en")} | ${change.toFixed(2)}% |`); + if (after < before * 0.95) refused.push(`${what} fell from ${before} to ${after}`); + } + const gone = Object.keys(was.packs.countries).filter((cc) => !(cc in now.packs.countries)); + if (gone.length) refused.push(`no pack any more for ${gone.join(", ")}`); + fs.appendFileSync(process.env.GITHUB_STEP_SUMMARY, summary.join("\n") + "\n\n"); + console.log(summary.join("\n")); + for (const reason of refused) { + console.log(`::error::${reason}. Nothing was published. If the data really did shrink, run this by hand with "Publish even if the data shrank".`); + } + if (refused.length) process.exit(1); + ' + # One moving tag rather than a dated one, so the Pages build has a fixed # name to ask for. Asking for `latest` would hand it whichever release is # newest, and the ports publish releases here too. diff --git a/.github/workflows/pages.yml b/.github/workflows/pages.yml index 94474eb..68cfea1 100644 --- a/.github/workflows/pages.yml +++ b/.github/workflows/pages.yml @@ -53,6 +53,13 @@ on: - 'typescript/README.md' - '.github/workflows/pages.yml' workflow_dispatch: + # The landmark archive and the name index are fetched from a release, so a + # new one changes nothing in this tree and no push would bring it here. The + # Landmarks workflow rebuilds them weekly; this follows each run, and builds + # only after one that published (see `if:` on the build job). + workflow_run: + workflows: [Landmarks] + types: [completed] # Read the tree, mint the token Pages needs to accept a deployment, and nothing # else. The build job below drops even the read permission it does not use. @@ -71,6 +78,9 @@ jobs: build: name: Build + # A Landmarks run that failed or was refused published nothing, and the + # site already has what is published. + if: github.event_name != 'workflow_run' || github.event.workflow_run.conclusion == 'success' runs-on: ubuntu-latest permissions: contents: read diff --git a/android/PLAY.md b/android/PLAY.md index b6c9c84..ba6addf 100644 --- a/android/PLAY.md +++ b/android/PLAY.md @@ -46,7 +46,7 @@ changes that page in the same pull request; it changes the already apart, the phone from 1, the watch from 1001 and the car from 2001, and each is raised with every upload of that bundle. - [ ] **The place-name packs** are live as the `place-packs` release. They are - published by the Landmarks workflow, run by hand. + published by the Landmarks workflow, every Monday and by hand. - [ ] **Store listing, graphics and screenshots**, as [below](#store-listing). The icon, the feature graphic and the screenshots, phone, 7-inch and 10-inch tablet, desktop, watch and Android Automotive OS, are made. The Android XR diff --git a/web/README.md b/web/README.md index 93571a0..076e552 100644 --- a/web/README.md +++ b/web/README.md @@ -322,13 +322,17 @@ integration that serves the files in development and copies them into the output at the end of a build. `landmarks/` is generated and **not committed**. It is built by -[`.github/workflows/landmarks.yml`](../.github/workflows/landmarks.yml), run by -hand, and published as a release asset under the `landmarks` tag. The Pages -build downloads that asset; if there is none it warns and carries on, and the -reference list says it could not be loaded. - -Rebuilding the data is therefore a decision rather than a side effect: a new -archive does not reach the site until the next Pages run. +[`.github/workflows/landmarks.yml`](../.github/workflows/landmarks.yml), every +Monday and by hand, and published as a release asset under the `landmarks` tag. +The Pages build downloads that asset; if there is none it warns and carries on, +and the reference list says it could not be loaded. + +A run that publishes starts the Pages workflow, so a new archive reaches the site +the same morning. Before publishing, the run measures what it built against what +is published, and refuses data that shrank by more than one part in twenty or +lost a country, since a dump cut short builds without an error. Publishing a +smaller archive on purpose is done by hand, with the workflow's "Publish even if +the data shrank" box ticked. ## Working offline