Skip to content

data: add the Americas — 55 territories, 95 datasets, 16,195 features - #12

Merged
andresgmg merged 2 commits into
mainfrom
data/americas
Aug 11, 2026
Merged

data: add the Americas — 55 territories, 95 datasets, 16,195 features#12
andresgmg merged 2 commits into
mainfrom
data/americas

Conversation

@andresgmg

Copy link
Copy Markdown
Owner

Country outlines from Natural Earth 10m (public domain); first-level and municipal divisions from geoBoundaries gbOpen under permissive licences only; Chile from IDE Chile.

162 MB of data, largest file 14.2 MB, everything inside the CDN ceiling. The catalog now has 55 country pages grouped by UN M49 sub-region.

The pipeline from #11 was reused unchanged in shape — but four things had to be fixed once it met 57 countries instead of one.

1. Silent data loss

The municipal split filtered out features whose parent could not be resolved. Switching from a centroid test to largest-overlap cut Argentina's unmatched count from 13 to 8 — but the remainder are not a bug in the join:

geoBoundaries publishes 23 ADM1 units for Argentina. The country has 24. The Autonomous City of Buenos Aires is missing, so its comunas overlap no province at all.

Those now land in an unassigned part instead of vanishing, and the split asserts it holds every input feature. 12 features across the continent are in this state.

2. Resolution measured against the wrong file

The size budget was applied to the whole-country file, which is optional. Brazil was on its way to a 400 m tolerance because its 5,570 municipalities do not fit in one file — while every one of its 28 per-state files fits at 100 m with room to spare.

Split levels now simplify at the standard tolerance, the budget is checked per part, and the combined file is dropped when it does not fit.

3. Licence claims that were not true

The catalog labelled every country CC-BY-4.0, including territories whose only dataset is a public-domain Natural Earth outline.

Licence varies per level, not per country. Brazil alone: outline public domain, states CC BY 2.5, municipalities CC BY 3.0 IGO. Each dataset now records its own licence, upstream provider and vintage; the country-level value is derived, becoming mixed with an explicit list where levels differ. validate_data.py checks every dataset, not just the country.

Resulting distribution: 33 territories pure public domain, 22 mixed.

4. Toolchain

mapshaper is pinned in package.json and installed locally. npx -y re-resolves the package on each of the several hundred invocations a continent needs, and concurrent runs collide on the npm cache — that was the intermittent Brazil failure. make_previews.mjs invokes the entry script with node directly, since execFileSync cannot spawn the .cmd shim on Windows.

Upstream quirks — documented, not smoothed over

  • admUnitCount disagrees with the files geoBoundaries serves. Suriname reports 62 ADM2 units and ships 58; Honduras 299 and ships 298. I verified source and built counts match exactly in every case — the loss is upstream metadata, not this pipeline.
  • Ecuador and Peru each ship an ADM1 whose shapeISO contains an asterisk (EC-D*, PE-LMA*), illegal in a Windows filename, which broke the split with ENOENT halfway through writing. Split keys are now sanitised.
  • Jamaica (827 ADM2 against 14 parishes) and Saint Lucia (547 against 10 quarters) look mis-tiered and were left out rather than published wrong.
  • Brazil's ADM1 is a 2006 vintage. Shown on its catalog page rather than hidden.

Coverage gaps

Fifteen countries have no first-level divisions here — ODbL for Colombia, Costa Rica, Cuba, Guatemala, Guyana, Honduras, Haiti, Nicaragua, Panama, Suriname, Trinidad and Tobago, Uruguay and Saint Vincent and the Grenadines; CC-BY-SA for Grenada and Greenland. Several still have a permissive municipal tier, so they appear with an ADM2 and no ADM1 above it.

Bonaire/Sint Eustatius and Saba and Bouvet Island ship nothing at all.

This is the cost of the permissive-only rule and it is named explicitly on the roadmap. Gaps close by finding a national SDI or HDX release, not by relaxing the rule.

Verification

  • mkdocs build --strict clean; validate_data.py 0 errors, 0 warnings across all 55
  • Split parts sum exactly to their level total for every country
  • Source vs built feature counts identical for every dataset checked
  • Maps verified in-browser on Brazil: 4 / 30 / 5,573 SVG paths for 1 / 27 / 5,570 units
  • site/ is 18.3 MB with no file over 5 MB — the CI leak check was changed from a total-size cap to per-file, since the site legitimately grows with each continent while a leaked full-resolution dataset is always far larger than any preview

🤖 Generated with Claude Code

andresgmg and others added 2 commits August 11, 2026 12:13
Country outlines from Natural Earth 10m (public domain); first-level and
municipal divisions from geoBoundaries gbOpen under permissive licences only;
Chile from IDE Chile. 162 MB, largest file 14.2 MB, everything inside the CDN
ceiling.

The pipeline from the previous PR was reused unchanged in shape, but four
things had to be fixed once it met 57 countries instead of one.

Silent data loss
----------------
The municipal split filtered out features whose parent could not be resolved.
Switching from a centroid test to largest-overlap cut Argentina's unmatched
count from 13 to 8, but the remainder are not a bug in the join: geoBoundaries
publishes 23 ADM1 units for Argentina and the country has 24 — the Autonomous
City of Buenos Aires is missing — so its comunas overlap no province at all.
Those now land in an `unassigned` part instead of vanishing, and the split is
asserted to hold every input feature.

Resolution measured against the wrong file
------------------------------------------
The size budget was applied to the whole-country file, which is optional. Brazil
was on its way to 400 m tolerance because its 5,570 municipalities do not fit in
one file — while every one of its 28 per-state files fits at 100 m with room to
spare. Split levels now simplify at the standard tolerance and the budget is
checked per part; the combined file is dropped when it does not fit.

Licence claims that were not true
---------------------------------
The catalog labelled every country CC BY 4.0, including territories whose only
dataset is a public-domain Natural Earth outline. Licence varies per level, not
per country: Brazil's outline is public domain, its states CC BY 2.5, its
municipalities CC BY 3.0 IGO. Each dataset now records its own licence, upstream
provider and vintage; the country-level value is derived, becoming "mixed" with
an explicit list where levels differ. validate_data.py checks every dataset.

Toolchain
---------
mapshaper is pinned in package.json and installed locally. `npx -y` re-resolves
the package on each of the several hundred invocations a continent needs, and
concurrent runs collided on the npm cache — that was the intermittent Brazil
failure. make_previews.mjs invokes the entry script with node directly, since
execFileSync cannot spawn the .cmd shim on Windows.

Upstream quirks, documented rather than smoothed over
-----------------------------------------------------
- geoBoundaries' admUnitCount disagrees with the files it serves: Suriname
  reports 62 ADM2 units and ships 58, Honduras 299 and ships 298. Verified that
  source and built counts match exactly in every case; the loss is upstream.
- Ecuador and Peru each ship an ADM1 whose shapeISO contains an asterisk, which
  is illegal in a Windows filename and broke the split with ENOENT. Split keys
  are now sanitised.
- Jamaica (827 ADM2 against 14 parishes) and Saint Lucia (547 against 10
  quarters) look mis-tiered and were left out rather than published wrong.

Fifteen countries have no first-level divisions because theirs are ODbL or
CC-BY-SA. That gap is the cost of the permissive-only rule and is named
explicitly on the roadmap.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Two CI failures on the Americas PR.

Manifests reordered between Windows and Linux
---------------------------------------------
Sorting Path objects is not portable: WindowsPath compares case-insensitively,
PosixPath does not. With a lowercase `unassigned.geojson` sitting among
uppercase `US-XX.geojson` parts, the two platforms produced different orderings
and the manifest-freshness check failed on a purely cosmetic difference.
Sorting now uses the filename string everywhere, in both build_manifest.py and
gen_catalog.py.

apt update failing the docs build
---------------------------------
The runner image carries third-party apt sources that intermittently 403 —
azure-cli did today — which failed the whole update even though every package
needed comes from Ubuntu main. Tolerate the update failure; a genuinely missing
package still fails loudly at install.

Also documents an upstream defect found while investigating: geoBoundaries
codes South Dakota as `SU-SD` rather than `US-SD`, so its 66 counties are
grouped correctly but under an incorrectly named file. Recorded in the USA
manifest and left unaltered rather than silently rewritten.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
@andresgmg
andresgmg merged commit e4ffbe3 into main Aug 11, 2026
3 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant