release: v1.14.0 — ACLED pagination walks cursors before offset dies 27 Oct

docs/ADRs/027_harvest_count_verification.md

@@ -77,7 +77,38 @@ The UCDP API has a fixed ~1000-event inconsistency between TotalCount and actual
 - Dual threshold added in v1.2.7.
 - `harvest_ucdp.py` reverted from `page_size=50000` to default `page_size=1000` in v1.2.6.
 - All three UCDP harvesters (annual, candidate, dot9) share `fetch_paginated()` and inherit these protections.
-- ACLED `fetch_paginated()` in `datafactory_harvester/sources/acled.py` does not implement TotalCount verification — the ACLED API does not expose a total count in its response envelope. Pagination terminates when a page returns fewer results than `page_size`. This "short page" heuristic detects complete pagination but cannot detect server-side truncation within a page. Downstream `validate_events()` catches schema violations but not silent count shortfalls.
+- ACLED `fetch_paginated()` in `datafactory_harvester/sources/acled.py` does not implement TotalCount verification — the ACLED API does not expose a total count in its response envelope. Downstream `validate_events()` catches schema violations but not silent count shortfalls.
+
+### Amendment 2026-10-10 — ACLED's detection signal changed (#487)
+
+The paragraph above used to end: *"Pagination terminates when a page returns fewer results than
+`page_size`. This 'short page' heuristic detects complete pagination but cannot detect server-side
+truncation within a page."* **That heuristic no longer exists.**
+
+ACLED deprecated offset pagination on 2026-10-27, and `fetch_paginated()` now walks cursors and
+terminates only on `next_cursor is null`. A short page is not evidence of anything under cursor
+pagination, so the break that keyed on page length was deleted — keeping it would have truncated
+harvests silently.
+
+**The replacement is strictly better on this ADR's own terms.** `next_cursor` is a *server-declared*
+continuation token, where the short-page rule was a *client-side inference*. The series boundary is
+now stated by ACLED rather than guessed by us, and two conditions that previously could not be
+distinguished from a clean finish now raise (ADR-011): a response missing `next_cursor`, and a
+response with no events while the cursor is still live.
+
+**What has NOT changed, and is the part that still matters here:** the envelope still reports
+`count: null` and `total_count: null` — re-confirmed against a live cursor response on 2026-10-10,
+on every page of an 8-page walk. So ACLED still has no server-declared total, this ADR's
+`len(fetched) >= TotalCount * (1 - tolerance)` rule still cannot be applied to it, and the exemption
+stands. Register entry C-153 is amended rather than closed for exactly this reason.
+
+That leaves a gap this ADR should name explicitly, because it is the gap that produced this ADR in
+the first place: nothing compares an ACLED harvest's event count against the previous harvest of the
+same year. `n_events` is written to the ingestion ledger and never read back. A zero-event year
+fails loud via `validate_events()`; **any non-zero shortfall is recorded as `outcome: "success"`
+with a new, short content digest.** Tracked as #488 with register entry C-359. Parity between the
+two pagination methods was measured before the deadline and the recording retained, since the
+comparison can never be repeated — `reports/investigation_acled_cursor/findings.md`.
 
 ---
 

docs/sources/acled.md

@@ -17,7 +17,7 @@
 | Temporal coverage | 1997– |
 | Temporal granularity | Daily events |
 | Update cadence | Weekly |
-| Access method | REST API, OAuth2 password grant (ADR-026) |
+| Access method | REST API, OAuth2 password grant (ADR-026), cursor pagination (`&cursor=X`) |
 | Authentication | OAuth2 (username + API key → bearer token). Credential sharing prohibited by EULA. |
 | Features produced | `acled_battles`, `acled_explosions`, `acled_violence_against_civilians`, `acled_protests`, `acled_riots`, `acled_strategic_developments`, `acled_fatalities`, `acled_count` |
 | Grid layers | Harvest → Consolidation → Viewpoint → Compilation → Assembly |
@@ -34,15 +34,44 @@ ACLED is the second conflict data source in the data factory, complementing UCDP
 
 **Harvest → Consolidation → Viewpoint → Compilation → Assembly.**
 
-- **Harvest:** Fetches events from the ACLED API using OAuth2 authentication. Year-by-year pagination adopted May 2026 (courtesy protocol — contact Katayoun before large re-fetches). ~4.2 GB total across all years.
+- **Harvest:** Fetches events from the ACLED API using OAuth2 authentication. Year-by-year harvesting adopted May 2026 (courtesy protocol — contact Katayoun before large re-fetches). Within a year, `fetch_paginated()` in `acled.py` walks the API with **cursor pagination** (see below). ~4.2 GB total across all years.
 - **Consolidation:** Merges weekly snapshots into a version-aware event store. Simpler than UCDP — single stream, no candidate/annual distinction.
 - **Viewpoint:** Applies event type filtering and aggregation. No survivorship rules needed (single stream). No temporal distribution needed (single-day events).
 - **Compilation:** Places events onto the PRIO-GRID (latlon → pgid) and aggregates per cell-month using declared strategies.
 - **Assembly:** Combined with UCDP, GHS-POP, GHS-BUILT-S, PRIO-GRID static, and GAUL admin into the final grid.
 
+## Pagination
+
+ACLED deprecated offset (`&page=N`) pagination on **2026-10-27** and we migrated to cursors
+before that date (#487). The contract, confirmed against a live response on 2026-10-10:
+
+| Thing | Value |
+|---|---|
+| Parameter | `&cursor=X`, starting at `&cursor=0` |
+| Continuation token | `next_cursor`, a **top-level key in the JSON body** |
+| Type | integer (observed values are monotonically increasing, consistent with row ids) |
+| End of series | `next_cursor` is `null` — **the only authoritative terminator** |
+| Precedence | if both `cursor` and `page` are sent, cursor wins and `page` is ignored *silently* |
+| CSV | under `_format=csv` the value moves to an `X-Next-Cursor` response header; we use `_format=json` |
+| Unsupported | `export_type=monadic` must keep `&page=X`. We never request monadic exports |
+
+**A short page is not the end of the series.** Under offset pagination it was, and
+`fetch_paginated()` used to stop on `len(results) < page_size`. That inference is invalid under
+cursor pagination and the break was deleted; keeping it is how the migration would have silently
+truncated a harvest. Two conditions now raise rather than guess (ADR-011): a response with no
+`next_cursor` key, and a response with no events while the cursor is still live.
+
+Because the cursor is an integer, `0` is both the starting value and falsy — so termination tests
+`is None` and never falsiness.
+
+Parity between the two methods was measured before the deadline and the recording is kept, because
+the comparison cannot be repeated: see `reports/investigation_acled_cursor/findings.md` and
+`tests/test_acled_cursor_parity_fixture.py`.
+
 ## Known limitations
 
 - **Authentication required.** Each user must have their own ACLED credentials. OAuth2 tokens expire and must be refreshed.
+- **No server-declared total.** The response reports `count: null` and `total_count: null`, re-confirmed under cursor pagination on 2026-10-10, so there is no figure to verify a harvest against (C-153). This is why ACLED is exempt from [ADR-027](../ADRs/027_harvest_count_verification.md)'s count-tolerance rule, and why a *partially* short harvest is currently recorded as a success (#488). A zero-event year does fail loud.
 - **Rate limiting.** The API has rate limits. The harvester respects these, but large historical re-fetches should be coordinated with ACLED.
 - **Taxonomy differences.** ACLED's event taxonomy does not map 1:1 to UCDP's. The 8 features in the compiled grid are ACLED-native categories, not harmonised with UCDP.
 - **Historical revisions.** ACLED may revise historical events in weekly updates. The consolidation layer handles this via append-only storage with version tracking.

pyproject.toml

@@ -1,6 +1,6 @@
 [project]
 name = "views-datafactory"
-version = "1.13.0"
+version = "1.14.0"
 description = "Data factory for the VIEWS conflict forecasting platform — provenance, harvesting, consolidation, viewpoint construction, compilation, and assembly."
 authors = [
     { name = "Simon Polichinel von der Maase", email = "simmaa@prio.org" },

reports/investigation_acled_cursor/README.md

@@ -0,0 +1,55 @@
+# Investigation: ACLED offset → cursor pagination
+
+**Issue:** #487 (migration), #488 (the short-harvest gap it surfaced)
+**Branch:** `feat/acled-cursor-pagination`
+**Risk register:** C-153 (Tier 3 — no server-declared total; this change replaces the heuristic it names), C-359
+**Started:** 2026-10-10
+**Investigators:** Simon Polichinel von der Maase, Claude Code
+**Status:** Complete — parity confirmed 2026-10-10. H1/H2/H3/H5/H6 resolved, H4 negative. See [`findings.md`](findings.md).
+
+---
+
+## Problem
+
+ACLED will remove offset (`&page=N`) pagination on **27 October 2026**. Our harvester uses it.
+
+The migration itself is small — one loop in `fetch_paginated()` in
+`src/datafactory_harvester/sources/acled.py`. What makes it worth pre-registering is the shape of
+the failure it risks. The change deletes the loop's termination condition and replaces it with a
+different one, and a termination bug in a harvester does not crash: it returns *fewer events* and
+records them as a success. Nothing in this repository compares a harvest's event count against the
+previous harvest of the same year (#488), so a truncated harvest would reach the assembled grid
+silently.
+
+ADR-027 exists because that already happened once, on UCDP: `page_size=50000` made the API return
+335,918 of 384,918 events while the harvest reported success
+(`reports/post_mortems/2026-04-25_stale_zarr_store.md`).
+
+So "the tests pass" is not sufficient evidence that the migration is correct. The claim that needs
+proving is a parity claim — that a cursor walk and an offset walk over the same slice return the
+same data — and it can only be proven **before 27 October**, while both methods still work.
+
+## Contents
+
+- [`pre_analysis_plan.md`](pre_analysis_plan.md) — the hypotheses, the falsification criteria and
+  the pre-committed decision table, registered before any implementation code
+- [`findings.md`](findings.md) — the answer, written after the measurement
+- [`parity.json`](parity.json) — the headline result: 7,609 events, all countries, identical both
+  ways
+- [`parity_fixture_slice.json`](parity_fixture_slice.json) — the narrow run (84 events, Somalia)
+  that produced the committed replay fixtures. Separate because an ACLED event is ~1.4 KB, so the
+  headline slice would have been a 10 MB fixture per walk — good evidence, bad fixture.
+
+## Methodology Note
+
+This investigation uses pre-registration. The pre-analysis plan must be committed before any
+implementation code is written, so that "parity held" is a result rather than a rationalisation.
+
+Two things follow from that, and both are recorded in the plan rather than assumed:
+
+1. **The decision table commits to both outcomes before either is observed** — including what we do
+   if parity fails, which is the branch nobody plans for.
+2. **The measurement must prove it can detect a difference before its null result is believed.** A
+   parity script that reports "identical" without a detection drill is indistinguishable from one
+   that compares a walk against itself. The drill is a precondition in the script, not an optional
+   flag — the same construction as `scripts/verify_area_majority_projection.py`.