Repository navigation
chore: bump to 1.14.0 — ACLED pagination walks cursors before offset dies 27 Oct (#487) - #491
Merged
Merged
Conversation
…dies 27 Oct (#487) The first release whose point is a deadline rather than a feature. ACLED deprecates offset pagination on 27 October 2026; the harvester walked their API by `&page=N`. In the wheel (#490): - fetch_paginated() in acled.py walks `&cursor=X` and terminates only on a server-declared `next_cursor is None` - the empty-page and short-page breaks are deleted — under cursor pagination a short page is not the end of the series, and treating it as one is how this migration would have truncated a harvest silently - the walk is bounded by a non-advancing-cursor guard, so a stalled cursor raises instead of looping against a rate-limited API in an unattended cron - AcledConfig is unchanged: no field added or removed Why now and not later: the failure shape. A missed migration does not crash on the 27th — it surfaces on the 21 November cron as a short harvest, quiet, a month late, and in the dataset rather than the logs. Deploying before the 21 October run means the new code runs in production while offset still works at ACLED, so `git checkout v1.13.0` remains a live revert target. Parity was measured while both methods still worked: 7,609 events, 8 requests each way, identical on count, event_id_cnty sets, full records and content_digest (ae4a6e815168e95f), reproduced byte-identically at a second commit. Real envelopes from both walks are committed as fixtures — after 27 October the offset side can never be recorded again. Not fixed here: a short harvest is still recorded as `outcome: "success"` because nothing reads `n_events` back (C-359, #488). C-153 is amended, not closed — ACLED still returns count/total_count as null. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01EfeuULbPTqrx4qtNki5dy9
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Version bump only:
1.13.0→1.14.0, plus the product-plan addendum. Nosrc/change — the behaviour change itself landed in #490.Why this release exists
ACLED deprecates offset pagination on 27 October 2026. The server runs a released tag (ADR-022), so nothing merged into
developmentis live. Without a release, the migration sits in the repo and the production harvester keeps sending&page=N.The failure shape is why this is dated rather than routine. A missed migration does not crash on the 27th. It surfaces on the 21 November cron as a short harvest — quiet, a month late, and in the dataset rather than the logs.
0 0 21 * *) — last run where offset still worksDeploying before the 21st means the October run exercises cursor code while offset still works at ACLED —
git checkout v1.13.0stays a live revert target. Wait until November and the first production run of the new code happens on a date when the old code no longer works at all, so rollback stops being an option exactly when it would be needed.What is in the wheel
fetch_paginated()inacled.pywalks&cursor=Xand terminates only on a server-declarednext_cursor is None. The empty-page and short-page breaks are gone — under cursor pagination a short page is not the end of the series. The walk is bounded by a non-advancing-cursor guard, so a stalled cursor raises rather than looping against a rate-limited API.AcledConfigis unchanged.Evidence
Parity measured while both methods still worked: 7,609 events, 8 requests each way, identical on event count,
event_id_cntysets, full-record equality andcontent_digest(ae4a6e815168e95f), reproduced byte-identically at a second commit. Pre-registered before any implementation code. Real envelopes from both walks are committed as fixtures — after 27 October the offset side can never be recorded again, so the fixture replay is the only thing that can still make the claim.reports/investigation_acled_cursor/·notebooks/acled_cursor_parity.ipynbNot fixed here
A harvest that returns fewer events than it should is still recorded as
outcome: "success"—n_eventsgoes to the ledger and nothing reads it back (C-359, #488). C-153 is amended, not closed: ACLED still returnscount/total_countas null. The loop can no longer truncate silently; the ledger still cannot notice if something else does.🤖 Generated with Claude Code
https://claude.ai/code/session_01EfeuULbPTqrx4qtNki5dy9