data: add ingestion pipeline and Chile from IDE Chile DPA 2023

[andresgmg]

First real data under the conventions from #1, plus the pipeline the remaining continents will reuse. The catalog no longer says "not populated yet".

This is the tooling half of the Americas work. The continent's other 56 territories land in a follow-up PR that is data-only, so it can be reviewed for coverage and licensing rather than code.

Chile — 16 regions, 56 provinces, 345 communes

Sourced from IDE Chile's División Política Administrativa 2023 (CC BY), not geoBoundaries.

geoBoundaries was evaluated and rejected for Chile specifically: its ADM1 and ADM3 are fine, but its ADM2 is OpenStreetMap under ODbL and its communes are a 2020 vintage. DPA 2023 gives all three tiers from one source in one vintage, with CUT codes already stored as zero-padded strings — and it is the first openly licensed source for the provincial tier, which this repository never had.

345 communes rather than the official 346 is expected, not a defect. The Antártica commune (12202) is absent because the DPA package excludes Chile's Antarctic claim; the data stops at 56.5°S. geoBoundaries omits it for the same reason. This is documented in the manifest and on the roadmap as won't-fix.

The municipal level is split by region into 16 files (ADM3/CL-RM.geojson …), plus a whole-country file because it fits under the CDN ceiling. CI verifies the parts sum to exactly 345 — that is how a split that loses or duplicates a municipality gets caught.

Resolution is a distance, not a percentage

Everything is simplified to a 100 m ground tolerance, recorded per dataset in the manifest.

The first implementation bisected on a simplification percentage until the output fit a size budget. That was wrong twice over:

• It maximised every file up to the ceiling. Chile ADM1 landed at 18.4 MB for 16 polygons; at that rate the Americas would have approached 800 MB.
• A percentage keeps a fixed share of each source's vertices, so the resulting resolution depends on how densely each country happened to be digitised — neighbouring countries would ship at different fidelities.

A ground distance gives the whole repository one consistent real-world resolution. Calibrated against Chile's fjords, one of the worst coastlines on Earth: 49.7 MB at 10 m, 10.0 MB at 50 m, 4.6 MB at 100 m, 1.6 MB at 250 m.

Licensing correction

docs/contributing/sources.md was factually wrong and is corrected here.

geoBoundaries' gbOpen is a container of heterogeneous per-file licences, not a uniformly CC BY 4.0 dataset. Measured across its 129 Americas entries, 43 (33%) are copyleft — 35 ODbL, 8 CC-BY-SA, almost all OpenStreetMap-derived. Chile is a neat illustration: ADM1 and ADM3 are CC BY 3.0 IGO, ADM2 is ODbL.

Copyleft is excluded and validate-data.yml enforces the allow-list. That costs real coverage — thirteen Americas countries have no ADM1 this project can redistribute. Those gaps are named on the roadmap and get closed by finding permissive sources, not by relaxing the rule.

Preview maps now actually render

mkdocs.yml referenced geojson-map.js only inside a comment: the file did not exist and there was no extra_javascript key, so every <div class="geojson-map"> rendered as an empty 420 px box.

Leaflet is vendored locally and the previews are served from the docs site, not jsDelivr. Pointing at the CDN would mean the maps 404 under mkdocs serve and throughout PR review, since the data is not on the default branch until after the merge.

gen_catalog.py fixes

• Split levels render as one section per level with a parts table, not sixteen sections.
• Fixed a latent TypeError: sorted() on entries whose third element was a manifest dict would fall through to comparing dicts. Harmless at one country, a real hazard at fifty-seven.
• Unguarded ds['features'] / ds['path'] / ds['level'] no longer take down the whole docs build.

Verification

• mkdocs build --strict clean, including anchor validation
• scripts/validate_data.py — 0 errors, 0 warnings; negative-tested by setting the licence to ODbL-1.0, which fails with exit 1 as intended
• build_manifest.py run twice produces a byte-identical manifest with curated fields intact
• 16 ADM3 part files sum to exactly 345 features
• Maps verified in-browser: 19 / 59 / 348 SVG paths for 16 / 56 / 345 units, tiles loading, no console errors
• site/ is 6.2 MB — the 24 MB data tree did not leak into docs_dir
• fetch_sources.py --continent americas --dry-run enumerates 40 accepted and 23 rejected datasets; the dry run caught a bug in my own allow-list, where Canada's "Open Government Canada 2.0" was being rejected as unrecognised

Not in this PR

Chile's legacy root files are untouched and remain deprecated, per the window promised in about/versioning.md. CHL_ADM0 arrives with the Natural Earth pass in the Americas PR.

🤖 Generated with Claude Code