ACLED deprecates offset pagination on 27 Oct 2026 — migrate fetch_paginated to cursors #487

What ACLED told us

"We've rolled out a new cursor-based pagination method and will deprecate the current offset method on 27 October 2026."

Our harvester walks ACLED by &page=N. After 27 October that parameter stops working.

Neither of these is ACLED's own page — both relay the notice:

• ACLED API pagination change mapme-initiative/mapme.biodiversity#492 (opened 30 Sep 2026)
• Update for ACLED API change forecastingresearch/forecastbench#315 (opened 29 Sep 2026)

Dates

The failure shape is why this matters more than the date suggests. A missed migration does not
crash on the 27th. It surfaces on 21 November as a short or failed harvest — quiet, a month late,
and in the dataset rather than the logs. This repo has been burned by exactly that before:
ADR-027 exists because page_size=50000 made the UCDP API silently return 335,918 of 384,918
events while the harvest reported success.

The API contract

• &cursor=X, starting &cursor=0.
• next_cursor comes back in the JSON body. (_format=csv returns it as an X-Next-Cursor
  header instead — we use _format=json, so the body is our path.)
• next_cursor: null means the walk is complete.
• If both cursor and page are sent, cursor wins and page is ignored silently.
• Not supported with export_type=monadic; monadic exports must keep &page=X. We never set
  export_type and never request CSV, so we are on the supported path.

One caveat worth carrying: a third-party PR (kwyjad/Pythia#949) notes that the field's name and
top-level placement rest on the notice rather than on an observed response. Confirming it against
live is therefore the first step of the work, not an assumption inside it.

Where the change lands

fetch_paginated() in src/datafactory_harvester/sources/acled.py is the single site. It is in no
__all__ and is not exported by src/datafactory_harvester/sources/__init__.py, whose public
surface is fetch_source(name, **kwargs). Production callers: two, both in this repo —
_fetch_single_year() in the same module, and scripts/harvest_acled.py.

There is a separate, unrelated fetch_paginated() in ucdp_annual.py used by the UCDP sources.
Out of scope: this deprecation is ACLED's.

The load-bearing deletion is the termination logic. Today the loop stops on an empty page or on a
page shorter than page_size. Under cursor pagination the only authoritative terminator is
next_cursor is None — a short page is not the end of the series, and treating it as one is
precisely how this migration would silently truncate a harvest.

This collides with C-153 and ADR-027 — read before touching anything

C-153 (Tier 3, open) says of ACLED: "The harvester terminates on empty/short pages (correct
behavior for complete pagination) but cannot detect if the API silently caps results within a
page." This migration deletes the heuristic C-153 names as the detection signal. ADR-027
records the same thing — ACLED's absence of a server-declared total, accepted as a limitation with
the short-page heuristic as the only available signal.

Both statements become false. Both need amending, not duplicating.

The compensating good news: next_cursor is a server-declared continuation signal, which is
strictly more information than inferring the end of a series from page length. On that axis the
migration improves C-153's risk rather than worsening it.

Also worth checking during the live confirmation: the committed fixture
tests/fixtures/acled_sample.json has count: null and total_count: null, which is the entire
basis of C-153 and of ADR-027's ACLED exemption. If the cursor API populates either, C-153 can be
closed properly and ADR-027's tolerance rule becomes applicable to ACLED.

Approach

One branch replaces the mechanism outright; the old loop survives only in a throwaway verification
script. Not expand-then-contract, for three reasons:

1. No consumer needs a migration window. Parallel Change exists to give other people's code
   time to move. Nothing outside this repo calls the pagination loop.
2. Rollback is already tag-based. The server deploys by checking out a git tag (ADR-022), so
   reverting to v1.13.0 is three documented commands. The previous tag already is the second
   path.
3. The fallback expires. A flag defaulting to offset, left in place past 27 October, reads
   valid and returns a short harvest.

Acceptance criteria

• next_cursor confirmed as a top-level key against a live response; raw envelope recorded
• count / total_count checked for values (decides whether C-153 closes or is amended)
• fetch_paginated() walks cursors; page gone from the request params entirely
• Termination only on next_cursor is None — including when the final page is full
• Missing next_cursor key raises, naming the field (fail loud, ADR-011)
• An empty page with a live cursor raises rather than terminating or looping
• AcledConfig unchanged — no field added or removed
• The two tests that are the only coverage of mid-walk token refresh and 401 recovery are
  re-authored as genuine multi-page cursor walks, not quietly reduced to single-page
• Offset and cursor envelopes recorded as fixtures, and a parity test over them that still
  runs after 27 October, when the live comparison is no longer possible
• Parity proven live before 27 October: identical event sets and identical content_digest
• The parity script carries a detection drill proving it can see a difference
• Six hand-applied mutants on the termination condition, each caught
• C-153 and ADR-027 amended
• docs/sources/acled.md updated (cite by symbol name — test_docs_citations.py forbids
  line-number citations under docs/)

Out of scope, tracked separately

Nothing compares a harvest's n_events against the previous harvest of the same year —
n_events is write-only, and compare_snapshots' n_removed is computed and discarded. A
zero-event year fails loud; any non-zero shortfall is recorded as outcome: "success".
Pre-existing, equally true of the offset path, and it touches shared provenance code. Tracked as
#488. What this change owns is narrower and explicit: the loop cannot silently truncate.