Repository navigation
release: v1.14.0 — ACLED pagination walks cursors before offset dies 27 Oct - #492
Merged
Merged
Conversation
chore: sync main back into development after v1.13.0
…session work (#486) Written 2026-09-02/08, guard-passing, and never committed — the session moved on to other work and these sat in the tree for five weeks. No content is new; this is the backlog reaching git. C-353 RE-TIERED 3 -> 2. The entry set its own promotion rule — "rises to Tier 2 the moment a consumer aggregates on gaul1 or gaul2" — and then argued Tier 3 from "the conflict aggregations run on gaul0". That second claim was about THIS repository. The consumer is views-faoapi, which exposes second-level subset and HDI-map endpoints and treats gaul2 as a first-class aggregation level. The promotion condition was already met the day the entry was written; one grep would have settled it. Eight of the nine wrong labels sit at exactly that level. Caught by the fao02 session preparing FAO Pre-Release Note 07, not here. It had inherited the false claim from this entry and corrected its own document too. The original Tier 3 reasoning is struck rather than deleted, so the correction stays legible. C-356 (Tier 3) — UCDP where_prec >= 4 events are geocoded whole into one cell. Measured against the served store: cell 148759 (Mekelle, Tigray) holds 121,915 in its peak month, 273,035 across the war window, and 96.1% of all state-based fatalities in its 5x5 neighbourhood, against a panel 99.99th percentile of 255. Systemic rather than one bad record — cell 150919 is the 1999-2000 Eritrea-Ethiopia border war by the same mechanism. ADR-049 shipped the mirror strategy opt-in with parity as the default, exactly as recommended; production uses parity, so this is live. The effective default is spread across three files that must stay aligned and nothing asserts they do (#483). C-357 (Tier 4) — the same parity constraint gave UCDP a swappable temporal strategy seam and ACLED a hardcoded one-liner. Correct today (ACLED events are single-day), and the asymmetry outlives the reason. C-358 (Tier 4) — this register is public, cites sibling repositories constantly, and nothing screens them. Second instance of the mechanism C-328 recorded: a C-353 amendment published views-faoapi line numbers here and in a public issue; views-faoapi is private. Corrected the same day. The entry carries a visibility lookup for all 32 views-platform repositories (15 private) so the check is a lookup rather than a judgement. Header: 358 IDs, 46 open (0/4/14/22 by tier). All 14 register guards pass; validate_docs passes. Claude-Session: https://claude.ai/code/session_01EfeuULbPTqrx4qtNki5dy9 Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
…#490) * docs: pre-register the ACLED cursor parity measurement before writing it (#487) ACLED removes offset pagination on 27 October 2026. The migration is one loop, but it deletes that loop's termination condition and replaces it with another, and a termination bug in a harvester returns fewer events rather than failing. Nothing reads n_events back to notice (#488), so it would become the cached truth on the following run. ADR-027 exists because that already happened once. So the parity claim needs a measurement, and the measurement can only be taken before the 27th while both methods still work. This commit lands first so that "parity held" is a result and not a rationalisation: - six hypotheses, including two with no prior (H4 short pages mid-walk, H5 whether the cursor envelope populates count/total_count, which would close C-153 outright rather than amend it) - a decision table committing to both outcomes, including the branch nobody plans for: if the two walks disagree, stop and ship neither - the detection drill as a precondition, not a flag. The existing ACLED tests mock an ordered list of responses without inspecting the request, so they cannot tell a correct walk from one handed the right answer by accident. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01EfeuULbPTqrx4qtNki5dy9 * feat: ACLED pagination walks cursors, not pages — offset dies 27 Oct (#487) Confirmed against live ACLED first: next_cursor IS a top-level body key, and it is an INT (observed 1148657), not a string. So 0 is both the starting cursor and a falsy value, and termination must test `is None` — `if not next_cursor` would end a walk on a legitimate cursor. Two terminators deleted, and the deletion is the whole point: if not results: break if len(results) < config.page_size: break Under offset pagination a short page genuinely did mean the end of the series. Under cursor pagination it means nothing at all, and only next_cursor is authoritative. Keeping either break is how this migration would have silently truncated a harvest — which nothing downstream would notice, because n_events is written to the ledger and never read (#488). Two new raises, both fail-loud per ADR-011, because ACLED reports count/total_count as null (C-153) so next_cursor is the only completeness signal the response carries: - envelope with no next_cursor key — unrecoverable, names the field and lists the keys it did get, same shape as the missing 'access_token' guard - no events while the cursor is still live — refuses both to continue (unbounded walk against a rate-limited API) and to stop (the silent truncation). H6 of the pre-analysis plan predicts this never fires on a real walk; the live run tests that. AcledConfig is untouched — no field added or removed. page_size is already the `limit`, and _fetch_single_year rebuilds the per-year config by re-listing all eleven fields, so a new one would have been silently dropped there. max_pages keeps its name: ACLED's own notice still says "page" under cursor pagination, and renaming would churn one script and ten call sites for nothing. Tests: 17 failed on the bare implementation. The fix for 8 of them was one defaulted argument — _make_acled_response(events, next_cursor=None) — so a single-response test now describes a complete walk. The rest needed real work, and two needed care: test_mid_pagination_token_refresh and test_401_mid_pagination_refreshes_token are the ONLY coverage of the token-refresh and 401-recovery branches inside pagination, and both would have gone green as single-page walks while testing nothing. They carry live cursors instead. Six new tests, each aimed at a mutant: - full final page + null cursor terminates (the old code could not have passed this) - short page + live cursor continues - next_cursor == 0 is live, not terminal - cursor threads through three pages, asserting REQUEST COUNT, so `cursor = 0` in place of `cursor = next_cursor` cannot pass - empty page + live cursor raises; missing next_cursor key raises - no events + null cursor returns [] without raising Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01EfeuULbPTqrx4qtNki5dy9 * test: record the cursor/offset parity before the comparison stops existing (#487) Measured: 7609 events, 8 requests each way, identical on all four oracles, content digest ae4a6e815168e95f both times. After 2026-10-27 this comparison cannot be made by anyone, so the recording is the deliverable, not the number. scripts/verify_acled_pagination_parity.py holds both walks, with the offset loop as a frozen local copy rather than an import — the same construction and the same reason as verify_area_majority_projection.py: importing production would mean a later change there silently moves the baseline the measurement was taken against. The detection drill is a precondition in main(), not a flag. Four drills: identical input reports parity; a dropped event trips all four oracles; a changed FIELD trips records_equal even where counts and id sets agree; and the two walk implementations actually diverge on input designed to separate them. Verified by sabotage — forcing "parity": True, and separately deleting the short-page break from the frozen offset walk — both caught, exit 1. Without it, "the digests match" is indistinguishable from hashing the same list twice, and the existing 22 ACLED mocks cannot substitute: they serve an ordered list without inspecting the request, so they cannot tell a correct walk from one handed the right answer by accident. Fixtures are a SEPARATE narrow slice (Somalia, 84 events, 113 KB) and --write-fixtures is off by default. The headline walk recorded 9.9 MB per side against a repo whose largest fixture is 9.4 KB. Trimming events would have broken the digest oracle and shrinking the headline slice would have weakened the evidence, so the two artifacts get two slices. Logged as a deviation in the pre-analysis plan rather than quietly re-scoped. test_acled_cursor_parity_fixture.py replays the real envelopes through the real fetch_paginated — 31 event fields, 9 envelope keys, the actual opaque cursors in the actual order. It carries its own truncation drill, because every other assertion in it would also pass against a loop that stopped early. Hypotheses resolved. H1 parity holds. H2 next_cursor is top-level and an INT (so 0 is falsy and `is None` is mandatory). H3 null exactly once, last. H5 count/total_count STILL null on every page — so C-153 is amended, not closed, and ADR-027's tolerance rule still cannot apply to ACLED. H6 no empty page with a live cursor. H4 came back NEGATIVE: no page arrived short while the cursor was live, so the deleted break would have agreed on this slice. Recorded as a negative result rather than omitted. The deletion is structurally right but not vindicated by this measurement — the honest basis for it is mutant M1 being caught, not H4. All 8 termination mutants caught (6 planned, plus M7 dropping the empty-page raise and M8 terminating on falsiness, which the int cursor made a live hazard the planned six did not probe). Register: C-153 amended — it named the short-page heuristic as ACLED's only detection signal, which this change deletes. C-359 added for the gap the planning surfaced: n_events is written to the ledger and never read, so a non-zero shortfall is recorded as success (#488). Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01EfeuULbPTqrx4qtNki5dy9 * fix: bound the cursor walk, and stop the parity script reporting false parity (#487) Four findings from /code-review medium on #490. Three were right about real defects; the fourth was right about where an entry was filed. **The walk had no termination bound left (medium).** Both old terminators are gone and `page` is now only a counter, so the exits are `next_cursor is None`, the two raises, and `max_pages` — which production never passes. `_fetch_single_year` calls fetch_paginated with no cap; only the --proof probe sets one. If ACLED ever echoed a cursor back, or reset it after a restatement, the loop re-requested the same full page forever: unbounded duplicates, one request per page_delay, against a rate-limited API, in an unattended monthly cron. Under offset pagination `page += 1` guaranteed forward progress; under cursor pagination only the server can. The review also caught the inconsistency that makes this embarrassing rather than merely missed: the parity script gave its own walk MAX_REQUESTS = 40 with a comment explaining exactly this risk. Production had nothing. Guard is a seen-set rather than `next_cursor != cursor`, because it also catches a longer cycle (A→B→A) and 80 integers costs nothing. Seeded with {0} so a response echoing the starting cursor is caught too — and that case is why test_zero_cursor_is_not_treated_as_terminal is gone. It asserted that a walk continues after next_cursor=0, which this change reclassifies as a stalled walk. Replaced by test_echoed_zero_cursor_raises_not_terminates, which is a better test: one scenario now separates three behaviours — correct code raises, `if not next_cursor: break` returns silently, and no guard loops forever. M8 is still caught. **The empty-page raise can abort a harvest on a benign boundary (low).** A slice holding an exact multiple of `limit` yields a final full page followed by an empty one. If ACLED echoes a live cursor there, the year fails. Odds are ~1-in-page_size per year-slice, so non-trivial across a 28-year re-fetch, and never observed — H6 was confirmed on one 8-page and one 2-page walk. Kept the raise rather than taking the suggested retry-once: fail-loud is the governing principle and the consequence is a failed harvest, not a wrong one. But the message now names the exact-multiple boundary as the likely cause and says what to check, so whoever hits it does not start by suspecting ACLED. **The parity script could report PARITY on two truncated walks (low).** Both walks stopped silently at MAX_REQUESTS and _compare then compared two prefixes, which agree on all four oracles. On a slice larger than 40×limit the script logged "PARITY: both walks returned identical data" and exited 0 — the one verdict the pre-analysis plan reads as licence to ship. That is the error class this entire investigation is about, committed by its own instrument. Walks now return hit_cap, the report records it, a capped run forces a non-parity verdict and exits 1 as INCONCLUSIVE, and drill 5 asserts the flag fires — verified by sabotage, like the other four. **C-359 was filed under "Deferred by Design" (low).** Moved into the Tier 3 section where the header counts it and where every other Tier 3 entry lives. My error: I picked an insertion anchor without checking which section it was in. Mutants: 8 → 11, all caught. M9 drops the progress guard, M10 never records a cursor as seen, M11 seeds the set empty instead of with 0. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01EfeuULbPTqrx4qtNki5dy9 * docs(investigation): fold the review findings into findings.md, re-stamp both measurements (#487) Two of this document's claims had gone stale and one of its artifacts could not be traced to the code that made it. Section 3.2 said a restatement-reset cursor was an open question. It still is — but the worst version of it now fails loud instead of looping, so the section says what was closed and what was not. Section 3.1 gains the one trigger for the empty-page raise that is both benign and foreseeable: a slice holding an exact multiple of limit. Roughly half a percent odds across a 28-year re-fetch, presenting as a failed year rather than a wrong one. New section 5 records all five review findings, including the two worth keeping visible rather than quietly fixing: The parity script could have reported PARITY on two truncated walks — truncation mistaken for completion, in the instrument built to detect truncation mistaken for completion. It never affected the recorded results (both walks ended naturally in 8 and 2 requests) but it could have on a larger slice, and parity.json now states walks_ran_to_completion rather than leaving it inferable. And parity.json's git_sha named 13ddda1, which does not contain the script — it landed in eea3068. For a file whose whole purpose is evidence that cannot be re-collected, that stamp was close to useless. Both measurements re-run at 47089fd. The re-run reproduced BYTE-IDENTICALLY: 7609 events, 8 requests each way, digest ae4a6e815168e95f, and the fixture recording's md5 is unchanged — which also confirms ACLED has not restated this slice. That corroboration was not planned; it fell out of fixing the stamp. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01EfeuULbPTqrx4qtNki5dy9 * fix: close three holes /review-diff found, including one in the drill itself (#487) Iteration 2 of the ritual. Two warnings, three suggestions, no criticals. All taken. **A non-scalar next_cursor crashed with a bare TypeError.** The progress guard tracks cursors in a set, so a list or dict cursor hit `TypeError: unhashable type: 'list'` out of the membership test — verified empirically. Every other envelope anomaly in this loop names the field and says what to check; this one gave an unattended cron a stack trace with nothing pointing at ACLED. A type change is the same class of provider drift as the missing key already guarded two lines up, so it now gets the same named ValueError. New mutant M12. **max_pages now beats a stalled cursor.** The guard ran first, so a capped walk raised on a cursor anomaly it was about to stop for anyway — verified: max_pages=1 with an echoed cursor raised instead of returning its page. Only `harvest_acled.py --proof` passes a cap, so this was `--proof` reporting FAIL for a credential that plainly works. The cron passes no cap, so the ordering change cannot weaken it. **The parity verdict had two owners.** `_compare` returned `parity: all(checks.values())` and then main() overrode it to False and added a key to `checks` after that aggregate was taken. No live bug, but a check added inside `_compare` later would have sat outside main()'s override logic. Completion is now an oracle passed INTO `_compare` rather than a correction applied after it. **And that refactor opened a hole the drill did not see.** Drill 5 proves the WALKS set hit_cap; nothing proved `_compare` acts on it. Hard-coding `walks_ran_to_completion: True` inside `_compare` passed the whole drill cleanly — found by sabotage, which is the only reason it was found at all. Drill 6 closes it: identical data with walks_completed=False must not report parity. Re-sabotaged, now caught. Also: F1's stale "Three drills" docstring, and a note that `_fake_fetch` ignores params so no drill can catch a malformed request — stated so "drill passed" is not later read as broader than it is. **Unrelated guard tripped, and the fix is precedented.** test_falsification_ghsbuilts_coverage_parity F1 requires GHS-BUILT-S to hold >=70% of its siblings' test-function count. The 12 tests added here moved the denominator 245 -> 257 and the ratio 73% -> 69%. GHS-BUILT-S did not regress — it is 178 functions before and after. F2 was lowered 70% -> 65% on 2026-06-29 for exactly this reason; F1 now follows, with the trigger recorded in its docstring. Adding GHS-BUILT-S tests to restore a ratio would be gaming the metric. Both F1 and F2 also had failure messages claiming 70% while enforcing a different number. Mutants: 11 -> 12, all caught. Drills: 5 -> 6, each verified by sabotage. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01EfeuULbPTqrx4qtNki5dy9 --------- Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
…dies 27 Oct (#487) (#491) The first release whose point is a deadline rather than a feature. ACLED deprecates offset pagination on 27 October 2026; the harvester walked their API by `&page=N`. In the wheel (#490): - fetch_paginated() in acled.py walks `&cursor=X` and terminates only on a server-declared `next_cursor is None` - the empty-page and short-page breaks are deleted — under cursor pagination a short page is not the end of the series, and treating it as one is how this migration would have truncated a harvest silently - the walk is bounded by a non-advancing-cursor guard, so a stalled cursor raises instead of looping against a rate-limited API in an unattended cron - AcledConfig is unchanged: no field added or removed Why now and not later: the failure shape. A missed migration does not crash on the 27th — it surfaces on the 21 November cron as a short harvest, quiet, a month late, and in the dataset rather than the logs. Deploying before the 21 October run means the new code runs in production while offset still works at ACLED, so `git checkout v1.13.0` remains a live revert target. Parity was measured while both methods still worked: 7,609 events, 8 requests each way, identical on count, event_id_cnty sets, full records and content_digest (ae4a6e815168e95f), reproduced byte-identically at a second commit. Real envelopes from both walks are committed as fixtures — after 27 October the offset side can never be recorded again. Not fixed here: a short harvest is still recorded as `outcome: "success"` because nothing reads `n_events` back (C-359, #488). C-153 is amended, not closed — ACLED still returns count/total_count as null. Claude-Session: https://claude.ai/code/session_01EfeuULbPTqrx4qtNki5dy9 Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
Contributor
Author
|
Correction: this PR's description cited The notebook is written in #493, together with two things found while writing it: a wider measurement (98,636 events over a full quarter at production's own |
This was referenced Oct 12, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Promotion of
developmentontomainfor v1.14.0. Merge as a merge commit — never squash (a squash rewrites the release SHAs and permanently breaks the ancestry the back-merge maintains; it cost us a near-miss at v1.10.0, seepublishing_to_pypi.md).Why this release has a date on it
ACLED deprecates offset pagination on 27 October 2026. The server runs a released tag (ADR-022), so nothing on
developmentis live — without this release the production harvester keeps sending&page=N.The failure shape is the reason for the urgency. A missed migration does not crash on the 27th. It surfaces on the 21 November cron as a short harvest — quiet, a month late, and in the dataset rather than the logs. ADR-027 exists because that exact shape already happened once:
page_size=50000made the UCDP API silently return 335,918 of 384,918 events while the harvest reported success.0 0 21 * *) — last run where offset still worksDeploying before the 21st means the October run exercises cursor code while offset still works at ACLED, keeping
v1.14.0revertible tov1.13.0. Deploy later and the first production run of the new code happens when the old code no longer works at all — rollback stops being an option exactly when it would be needed.What is in the wheel
fetch_paginated()inacled.pywalks&cursor=Xand terminates only on a server-declarednext_cursor is None. Two inferential terminators were deleted — the empty-page break and the short-page break — because under cursor pagination a short page is not the end of the series, and treating it as one is precisely how this migration would have truncated a harvest silently. The walk is bounded by a non-advancing-cursor guard, so a stalled cursor raises rather than looping against a rate-limited API in an unattended cron.AcledConfigis unchanged: no field added or removed.Evidence
Parity measured live while both methods still worked: 7,609 events, 8 requests each way, identical on event count,
event_id_cntysets, full-record equality andcontent_digest(ae4a6e815168e95f), reproduced byte-identically at a second commit. The measurement was pre-registered before any implementation code existed. Twelve mutants on the termination condition, all caught; six detection drills, each verified by sabotage.Real envelopes from both walks are committed as fixtures — after 27 October the offset side can never be recorded again, so the fixture replay is the only thing that can still make the parity claim.
reports/investigation_acled_cursor/·notebooks/acled_cursor_parity.ipynbOne hypothesis came back negative and is recorded as such: no page arrived short while the cursor was live, so the deleted short-page break would have agreed on the measured slice. The deletion is structurally right but not vindicated by the data — the honest basis is mutant M1 being caught.
Also carried in this promotion
Not fixed here, deliberately
A harvest that returns fewer events than it should is still recorded as
outcome: "success"—n_eventsgoes to the ledger and nothing reads it back (C-359, #488). C-153 is amended, not closed: ACLED still returnscount/total_countas null, so ADR-027's tolerance rule remains inapplicable to ACLED. The loop can no longer truncate silently; the ledger still cannot notice if something else does. Fixing that touches shared provenance code and was not worth doing under a deadline.🤖 Generated with Claude Code
https://claude.ai/code/session_01EfeuULbPTqrx4qtNki5dy9