ACLED pagination walks cursors, not pages — offset dies 27 Oct (#487)

[Polichinel]

Closes #487 (by hand — Closes does not fire on PRs into development).

ACLED removed offset (&page=N) pagination on 27 October 2026. This migrates
fetch_paginated() to cursors and records the parity evidence while that evidence is still
possible to collect.

Why the failure shape drove the approach

A missed migration does not crash on the 27th. It surfaces on 21 November as a short harvest —
quiet, a month late, and in the dataset rather than the logs. ADR-027 exists because that already
happened to UCDP: page_size=50000 returned 335,918 of 384,918 events and the harvest reported
success.

So this is not treated as a twelve-line diff. It is treated as a change to a termination condition
in a monthly cron that nobody watches.

Six commits, in a deliberate order

18029fe — pre-registration, before any implementation code. Six hypotheses with falsification
criteria, and a §4 decision table committing to both outcomes including the branch nobody plans
for: if the two walks disagree, stop and ship neither. Committing this first is what makes "parity
held" a result rather than a rationalisation.

13ddda1 — the implementation. Two terminators deleted, and the deletion is the point:

if not results: break
if len(results) < config.page_size: break

Under offset pagination a short page genuinely meant the end of the series. Under cursor pagination
it means nothing, and only next_cursor is authoritative. Two conditions now raise rather than
guess (ADR-011): a response missing next_cursor, and a response with no events while the cursor
is still live.

eea3068 — the parity evidence, the docs, the register.

47089fd, acba3fb, 8a0c6fd — two review rounds. /code-review medium found four real
defects; /review-diff found three more, one of them in the detection drill itself. See
"What the reviews changed" below.

What the live probe changed about the design

next_cursor is an int (observed 1156286), not a string. So 0 is both the starting cursor
and falsy, and termination must test is None — if not next_cursor would end a walk on a
legitimate cursor. That became mutant M8, which the planned six mutants would not have probed.

The measurement

7,609 events, 8 requests each way, identical on all four oracles, content digest
ae4a6e815168e95f both times. Oracles: event count, event_id_cnty set, full-record equality over
the sorted lists, and the content_digest the ingestion ledger records.

The digest is the load-bearing one because validate_events() sorts before hashing — so a
different page ordering could not produce a false mismatch — and because it is the exact value
_year_is_cached() compares against. Full-record equality runs alongside it because the digest
covers only event_id_cnty and fatalities.

The detection drill is a precondition, not a flag

main() aborts if it fails. Six drills: identical input reports parity; a dropped event trips
all four oracles; a changed field trips records_equal even where counts and id sets agree;
and the two walk implementations actually diverge on input designed to separate them.

Verified by sabotage — forcing "parity": True, and separately deleting the short-page break from
the frozen offset walk. Both caught, exit 1.

This matters because the existing 22 ACLED mocks cannot substitute: they serve an ordered
side_effect list without inspecting the request, so they cannot tell a correct walk from one that
asks for the wrong page and is handed the right answer anyway.

H4 came back negative, and it is recorded that way

No page arrived short while the cursor was live. Every non-terminal page was exactly full; the final
page was short and carried a null cursor — so the deleted break would have agreed on this
slice.

That is in findings.md as a negative result rather than omitted. The deletion is structurally
right but not vindicated by this measurement. The honest basis for believing it is mutant M1
being caught, not H4 being positive.

H5 held, which decided whether C-153 closes

count and total_count are still null on every page under cursor pagination. So ACLED still
has no server-declared total, ADR-027's len(fetched) >= TotalCount * (1 - tolerance) rule still
cannot be applied to it, and C-153 is amended rather than closed. Had either field carried a
value it would have closed outright — predicting them null and checking anyway is why that is now
known rather than assumed.

Tests

17 failed on the bare implementation. Eight were fixed by one defaulted argument —
_make_acled_response(events, next_cursor=None), so a single-response test now describes a
complete walk.

Two needed care rather than patching: test_mid_pagination_token_refresh and
test_401_mid_pagination_refreshes_token are the only coverage of the token-refresh and
401-recovery branches inside pagination, and both would have gone green as single-page walks while
testing nothing. They carry live cursors instead.

New: full-final-page termination (the old code could not have passed it), short-page-with-live-cursor
continuation, next_cursor == 0 is live, cursor threading asserted on request count (an
event-count assertion cannot catch cursor = 0), both raise paths, and empty-with-null returning
[].

tests/test_acled_cursor_parity_fixture.py replays the recorded envelopes through the real
fetch_paginated — 31 event fields, 9 envelope keys, the actual opaque cursors in the actual order
— and carries its own truncation drill, because every other assertion in it would also pass against
a loop that stopped early.

All 12 termination mutants caught, baseline green, git diff clean afterwards. Two of them
exist only because a finding suggested them rather than the plan: M8 (terminate on falsiness
rather than is None, which the int cursor made a live hazard) and M12 (drop the non-scalar
cursor guard).

Fixtures: a separate, narrow slice

--write-fixtures is off by default. The headline walk recorded 9.9 MB per side against a repo
whose largest existing fixture is 9.4 KB. Trimming events would have broken the digest oracle;
shrinking the headline slice would have weakened the evidence. So the two artifacts get two slices,
and the fixture run is Somalia / 84 events / 113 KB — still a genuine two-page walk with a full
non-terminal page and a short terminal one.

Logged as a deviation in the pre-analysis plan rather than quietly re-scoped.

These fixtures cannot be regenerated after 27 October by anyone.

Register

• C-153 amended. It named the short-page heuristic as ACLED's only detection signal — which
  this change deletes. Reading the register before touching the code is what turned two silent
  falsehoods into two amendments. ADR-027's Implementation Notes got the same treatment.
• C-359 added. n_events is written to the ingestion ledger and never read back, so a non-zero
  shortfall is recorded as outcome: "success". Deliberately not fixed here — it touches shared
  provenance code and would confound two changes in one release. Tracked as A short ACLED harvest is recorded as success — n_events is written to the ledger and never read #488.

Scope, stated rather than implied

This change owns one sentence: the pagination loop cannot silently truncate. The
ledger-level count check is #488.

Verification

Check	Result
ruff check .	clean
mypy src/ (strict)	clean, 89 files
docs/validate_docs.sh	passed
test_docs_citations.py	passed (symbol-name citations only)
All 14 register-reading test files	41 passed, never sampled
Full suite, unpiped exit code	2 failures, both pre-existing and unrelated — test_falsification_shining_codex.py asserts a filename views-models retired in their 3f2d98ec. Skipped in CI (sibling repo absent). Tracked as #489
Detection drill	6/6; each verified by sabotage
Live parity	exit 0, digests identical
Mutation drill	12/12 caught

What the reviews changed

Two rounds, seven findings, all fixed. Four are worth naming here because they change claims made
above or in findings.md.

The walk had no termination bound (/code-review, medium). Both old terminators were gone,
page was only a counter, and production passes no max_pages — only the --proof probe does. A
cursor that stopped advancing, whether echoed by a bug or reset by a restatement, would have
re-requested the same full page forever: unbounded duplicates, one request per page_delay, against
a rate-limited API, in an unattended monthly cron. Under offset pagination page += 1 guaranteed
forward progress; under cursor pagination only the server can.

The review also caught the inconsistency that makes this embarrassing rather than merely missed:
the parity script had given its own walk a 40-request cap, with a comment explaining this exact
risk. The guard existed in the instrument and not in the thing being measured. Now a seen-cursor
set, which also catches a longer cycle (A→B→A).

The parity script could report PARITY on two truncated walks (/code-review, low). Both stopped
silently at the cap and the comparison then ran on two prefixes, which agree on all four data
oracles. On a slice larger than 40 × limit it would have logged "PARITY: both walks returned
identical data" and exited 0 — the single verdict §4 of the pre-analysis plan treats as licence to
ship. That is the error class this entire investigation exists to prevent, committed by its own
instrument. It never touched the recorded measurements (both ran to natural completion in 8 and 2
requests), but it could have.

A non-scalar next_cursor crashed with a bare TypeError (/review-diff, warning). The
progress guard tracks cursors in a set, so a list or dict cursor hit
TypeError: unhashable type: 'list' out of the membership test — verified empirically. Every other
envelope anomaly in this loop names the field and says what to check; this one handed an unattended
cron a stack trace with nothing pointing at ACLED.

And the drill had a hole the refactor had just opened (found by sabotage, not by review).
Moving the completion check into _compare — itself the fix for a two-owners-of-the-verdict
finding — left nothing verifying that _compare honours the flag. Hard-coding
walks_ran_to_completion: True passed the entire six-part drill cleanly. Drill 6 closes it.
That is the second time in this branch that sabotaging a check caught something that would
otherwise have shipped.

One unrelated guard tripped, and the fix is precedented

test_falsification_ghsbuilts_coverage_parity F1 requires GHS-BUILT-S to hold ≥70% of its siblings'
test-function count. The 12 tests added here moved the denominator 245 → 257 and the ratio 73% →
69%. GHS-BUILT-S did not regress — it is 178 functions before and after.

F2 was lowered 70% → 65% on 2026-06-29 for exactly this mechanism ("any new assertion in the
'other' files tripped the test"). F1 now follows, with the trigger recorded in its docstring.
Adding GHS-BUILT-S tests to restore a ratio would be gaming the metric rather than improving
anything. Both F1 and F2 also carried failure messages claiming 70% while enforcing a different
number.

Worth stating without acting on it: that guard penalises adding tests anywhere in its sibling set.
It is a relative measure doing a job that wants an absolute floor. Not this branch's business to
redesign.

Not in this PR

Release and deploy. The server runs a tag, so cursor code reaches production only via
1.13.0 → 1.14.0 and a deploy. Recommended before the 21 October cron, so the October run
exercises it while offset still works at ACLED and v1.13.0 is a live revert target. Separately
gated.

🤖 Generated with Claude Code

https://claude.ai/code/session_01EfeuULbPTqrx4qtNki5dy9