Repository navigation
Rebuild the landmark data every week, and put it on the site - #108
Merged
Merged
Conversation
GeoNames rewrites its dump every day, and the landmark archive, the name index and the app's place-name packs were only rebuilt when somebody ran the Landmarks workflow. It now also runs on Mondays at 05:43 UTC, two hours after the day's dump is written. A scheduled run has no inputs, so the level and slice defaults are given again where they are used, now through the environment rather than pasted into the script. Nobody reads a scheduled run before it publishes, so a new step does: it measures what was built against what is published, the landmarks, their shards, the packed names and the countries with a pack, and refuses to publish a fall of more than one part in twenty or a country with no pack. A dump cut short builds without an error, and this is what would catch it. A smaller archive on purpose is published by hand, with the new "Publish even if the data shrank" box ticked. Tried against the published data: a normal week passes, a tenth fewer landmarks and one lost country are refused, a fall of 4.9% passes, and a repository with nothing published yet publishes. The site fetches the archive from the release, so a new one changed nothing in the tree and never deployed. The Pages workflow now follows every Landmarks run and builds after one that published, so the site and the packs move together. The workflow README, web/README.md and PLAY.md say when it runs.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
The landmark archive, the name index and the app's place-name packs were rebuilt only when somebody ran the Landmarks workflow by hand, though GeoNames rewrites its dump every day.
Landmarks workflow
43 5 * * 1). GeoNames writes the day's dump at about 03:35 UTC. It still runs by hand too.levelandslicesfall back to 4 and 8 where they are used. They now go throughenvrather than into the script.manifest.json(fromlandmarks.tar.gz) andpacks.json. It refuses to publish if the landmarks, the shards or the packed names fall by more than 5%, or if a country loses its pack. A dump cut short builds without an error, so this is what would catch it. A failed check publishes nothing. If the data really did shrink, run it by hand with the new Publish even if the data shrank box ticked.Pages workflow
workflow_runtrigger on Landmarks and builds only when that run succeeded, so the site and the packs move together.Checked
ghagainst the published data (6,556,364 landmarks, 82,251 shards, 7,361,276 packed names, 251 countries). A normal week passes. 10% fewer landmarks fails. Losing one country (LI) fails. A 4.9% drop in shards passes. With nothing published yet, it publishes. The summary table renders.audit/deploys.py,audit/publishes.pyand their tests pass. Both workflows parse.Not covered here: GitHub turns off a public repository's schedule after 60 days with no activity, and sends an email first. The workflow header says where to turn it back on.
Docs: the workflow README table,
web/README.mdandandroid/PLAY.mdnow say when it runs.