data: add the Americas — 55 territories, 95 datasets, 16,195 features - #12
Merged
Conversation
Country outlines from Natural Earth 10m (public domain); first-level and municipal divisions from geoBoundaries gbOpen under permissive licences only; Chile from IDE Chile. 162 MB, largest file 14.2 MB, everything inside the CDN ceiling. The pipeline from the previous PR was reused unchanged in shape, but four things had to be fixed once it met 57 countries instead of one. Silent data loss ---------------- The municipal split filtered out features whose parent could not be resolved. Switching from a centroid test to largest-overlap cut Argentina's unmatched count from 13 to 8, but the remainder are not a bug in the join: geoBoundaries publishes 23 ADM1 units for Argentina and the country has 24 — the Autonomous City of Buenos Aires is missing — so its comunas overlap no province at all. Those now land in an `unassigned` part instead of vanishing, and the split is asserted to hold every input feature. Resolution measured against the wrong file ------------------------------------------ The size budget was applied to the whole-country file, which is optional. Brazil was on its way to 400 m tolerance because its 5,570 municipalities do not fit in one file — while every one of its 28 per-state files fits at 100 m with room to spare. Split levels now simplify at the standard tolerance and the budget is checked per part; the combined file is dropped when it does not fit. Licence claims that were not true --------------------------------- The catalog labelled every country CC BY 4.0, including territories whose only dataset is a public-domain Natural Earth outline. Licence varies per level, not per country: Brazil's outline is public domain, its states CC BY 2.5, its municipalities CC BY 3.0 IGO. Each dataset now records its own licence, upstream provider and vintage; the country-level value is derived, becoming "mixed" with an explicit list where levels differ. validate_data.py checks every dataset. Toolchain --------- mapshaper is pinned in package.json and installed locally. `npx -y` re-resolves the package on each of the several hundred invocations a continent needs, and concurrent runs collided on the npm cache — that was the intermittent Brazil failure. make_previews.mjs invokes the entry script with node directly, since execFileSync cannot spawn the .cmd shim on Windows. Upstream quirks, documented rather than smoothed over ----------------------------------------------------- - geoBoundaries' admUnitCount disagrees with the files it serves: Suriname reports 62 ADM2 units and ships 58, Honduras 299 and ships 298. Verified that source and built counts match exactly in every case; the loss is upstream. - Ecuador and Peru each ship an ADM1 whose shapeISO contains an asterisk, which is illegal in a Windows filename and broke the split with ENOENT. Split keys are now sanitised. - Jamaica (827 ADM2 against 14 parishes) and Saint Lucia (547 against 10 quarters) look mis-tiered and were left out rather than published wrong. Fifteen countries have no first-level divisions because theirs are ODbL or CC-BY-SA. That gap is the cost of the permissive-only rule and is named explicitly on the roadmap. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Two CI failures on the Americas PR. Manifests reordered between Windows and Linux --------------------------------------------- Sorting Path objects is not portable: WindowsPath compares case-insensitively, PosixPath does not. With a lowercase `unassigned.geojson` sitting among uppercase `US-XX.geojson` parts, the two platforms produced different orderings and the manifest-freshness check failed on a purely cosmetic difference. Sorting now uses the filename string everywhere, in both build_manifest.py and gen_catalog.py. apt update failing the docs build --------------------------------- The runner image carries third-party apt sources that intermittently 403 — azure-cli did today — which failed the whole update even though every package needed comes from Ubuntu main. Tolerate the update failure; a genuinely missing package still fails loudly at install. Also documents an upstream defect found while investigating: geoBoundaries codes South Dakota as `SU-SD` rather than `US-SD`, so its 66 counties are grouped correctly but under an incorrectly named file. Recorded in the USA manifest and left unaltered rather than silently rewritten. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Country outlines from Natural Earth 10m (public domain); first-level and municipal divisions from geoBoundaries
gbOpenunder permissive licences only; Chile from IDE Chile.162 MB of data, largest file 14.2 MB, everything inside the CDN ceiling. The catalog now has 55 country pages grouped by UN M49 sub-region.
The pipeline from #11 was reused unchanged in shape — but four things had to be fixed once it met 57 countries instead of one.
1. Silent data loss
The municipal split filtered out features whose parent could not be resolved. Switching from a centroid test to
largest-overlapcut Argentina's unmatched count from 13 to 8 — but the remainder are not a bug in the join:Those now land in an
unassignedpart instead of vanishing, and the split asserts it holds every input feature. 12 features across the continent are in this state.2. Resolution measured against the wrong file
The size budget was applied to the whole-country file, which is optional. Brazil was on its way to a 400 m tolerance because its 5,570 municipalities do not fit in one file — while every one of its 28 per-state files fits at 100 m with room to spare.
Split levels now simplify at the standard tolerance, the budget is checked per part, and the combined file is dropped when it does not fit.
3. Licence claims that were not true
The catalog labelled every country
CC-BY-4.0, including territories whose only dataset is a public-domain Natural Earth outline.Licence varies per level, not per country. Brazil alone: outline public domain, states CC BY 2.5, municipalities CC BY 3.0 IGO. Each dataset now records its own licence, upstream provider and vintage; the country-level value is derived, becoming
mixedwith an explicit list where levels differ.validate_data.pychecks every dataset, not just the country.Resulting distribution: 33 territories pure public domain, 22 mixed.
4. Toolchain
mapshaper is pinned in
package.jsonand installed locally.npx -yre-resolves the package on each of the several hundred invocations a continent needs, and concurrent runs collide on the npm cache — that was the intermittent Brazil failure.make_previews.mjsinvokes the entry script withnodedirectly, sinceexecFileSynccannot spawn the.cmdshim on Windows.Upstream quirks — documented, not smoothed over
admUnitCountdisagrees with the files geoBoundaries serves. Suriname reports 62 ADM2 units and ships 58; Honduras 299 and ships 298. I verified source and built counts match exactly in every case — the loss is upstream metadata, not this pipeline.shapeISOcontains an asterisk (EC-D*,PE-LMA*), illegal in a Windows filename, which broke the split with ENOENT halfway through writing. Split keys are now sanitised.Coverage gaps
Fifteen countries have no first-level divisions here — ODbL for Colombia, Costa Rica, Cuba, Guatemala, Guyana, Honduras, Haiti, Nicaragua, Panama, Suriname, Trinidad and Tobago, Uruguay and Saint Vincent and the Grenadines; CC-BY-SA for Grenada and Greenland. Several still have a permissive municipal tier, so they appear with an ADM2 and no ADM1 above it.
Bonaire/Sint Eustatius and Saba and Bouvet Island ship nothing at all.
This is the cost of the permissive-only rule and it is named explicitly on the roadmap. Gaps close by finding a national SDI or HDX release, not by relaxing the rule.
Verification
mkdocs build --strictclean;validate_data.py0 errors, 0 warnings across all 55site/is 18.3 MB with no file over 5 MB — the CI leak check was changed from a total-size cap to per-file, since the site legitimately grows with each continent while a leaked full-resolution dataset is always far larger than any preview🤖 Generated with Claude Code