From a532b5887f742a950d194712b7cb9b4f9e996596 Mon Sep 17 00:00:00 2001 From: Polichinl Date: Mon, 12 Oct 2026 04:01:40 +0200 Subject: [PATCH 1/2] docs: the parity notebook that was asked for and never written, plus a wider measurement (#487) Two gaps, found at the release gate when the operator asked whether the full ritual had actually run. THE NOTEBOOK NEVER EXISTED. It was in the approved plan and was the most explicit thing requested -- "a human can look at the Jupyter notebook, go through it and say, okay, at this point in time, we could prove parity" -- and I cited it as done in the PR description, in the comment closing #487, in product_development_plan12.md and in a status relayed for an outgoing mail. No .ipynb had ever existed in any commit on any branch. Written now, committed with outputs, reading only committed artifacts so it still renders after 27 October when half the comparison becomes impossible to run. It recomputes the content digest from the raw recorded envelopes rather than reprinting the reported one, so the reader is not asked to trust the script that produced the report. CHECK 9 WAS UNSOUND, NOT MERELY SKIPPED. The plan's verification table called it "the one that would actually catch a botched migration": re-harvest a year and compare against the pre-migration ledger digest. The only such entry is 2025, harvested 2026-05-07. ACLED restates history continuously, so a re-harvest a year later would differ for reasons unrelated to pagination -- and a match would have been read as vindication it could not support. The flaw is in the oracle, so the check is withdrawn rather than deferred (findings.md 5.1). Its replacement is the instrument already built, widened: both walks in the same session over a full quarter, at limit=5000 -- production's actual page_size, which no previous measurement had used, and the exact parameter the deleted len(results) < page_size break compared against. 98,636 events, 20 requests per walk, all five oracles, digest af9fb35c03ac830f, neither walk capped. 13x the headline run. AND ONE THING THE NOTEBOOK ITSELF SURFACED. Printing the page structure made visible that the final page was short in all three slices -- 3,636 of 5,000, 609 of 1,000, 34 of 50. So a full final page, the one configuration where the deleted terminator is unambiguously wrong, never occurred live. The old code would have terminated correctly on everything measured. Recorded as findings 2.0, alongside the negative H4 result it mirrors: the case for this change rests on the contract and the mutation drill, not on an observed failure. Co-Authored-By: Claude Opus 5 (1M context) Claude-Session: https://claude.ai/code/session_01EfeuULbPTqrx4qtNki5dy9 --- notebooks/acled_cursor_parity.ipynb | 625 ++++++++++++++++++ reports/investigation_acled_cursor/README.md | 12 +- .../investigation_acled_cursor/findings.md | 84 ++- .../parity_quarter.json | 37 ++ .../pre_analysis_plan.md | 1 + 5 files changed, 755 insertions(+), 4 deletions(-) create mode 100644 notebooks/acled_cursor_parity.ipynb create mode 100644 reports/investigation_acled_cursor/parity_quarter.json diff --git a/notebooks/acled_cursor_parity.ipynb b/notebooks/acled_cursor_parity.ipynb new file mode 100644 index 0000000..308163b --- /dev/null +++ b/notebooks/acled_cursor_parity.ipynb @@ -0,0 +1,625 @@ +{ + "cells": [ + { + "cell_type": "markdown", + "id": "bbb341ba", + "metadata": {}, + "source": [ + "# Can we prove ACLED's cursor pagination returns the same data as offset?\n", + "\n", + "**Yes — and this notebook is the audit trail, not the proof.** The proof is\n", + "the committed JSON it reads. Nothing here contacts ACLED.\n", + "\n", + "> **Evidence as of 2026-10-12.** This notebook is a *record of a measurement\n", + "> taken at a point in time*, not a check that re-runs. It cannot be re-run\n", + "> against the live API after **27 October 2026**, because on that date the\n", + "> offset method ceases to exist and one half of the comparison becomes\n", + "> impossible to obtain. That is the whole reason it was taken early.\n", + "\n", + "**How to read it.** Work down. Every number printed below is read from a file\n", + "in `reports/investigation_acled_cursor/` or `tests/fixtures/acled_cursor/`,\n", + "all of which are committed. Where this notebook computes something itself, it\n", + "says so, and it recomputes rather than reprints — section 5 re-derives the\n", + "content digest from the raw recorded responses, so you are not asked to trust\n", + "the script that produced the report.\n", + "\n", + "**What would make this notebook a lie.** If the two walks had each silently\n", + "stopped early at the same page, they would agree on every data oracle while\n", + "both being wrong. That failure mode is real — it was found in this\n", + "investigation's own instrument during review — so a fifth oracle,\n", + "`walks_ran_to_completion`, is reported alongside the four data ones, and a\n", + "capped walk forces a non-parity verdict. Check it in section 3." + ] + }, + { + "cell_type": "markdown", + "id": "ca2e28ef", + "metadata": {}, + "source": [ + "## 1. Why this measurement exists\n", + "\n", + "ACLED's own notice, relayed verbatim in two independent third-party trackers:\n", + "\n", + "> \"We've rolled out a new cursor-based pagination method and will deprecate\n", + "> the current offset method on **27 October 2026**.\"\n", + "\n", + "- [mapme.biodiversity#492](https://github.com/mapme-initiative/mapme.biodiversity/issues/492)\n", + "- [forecastbench#315](https://github.com/forecastingresearch/forecastbench/issues/315)\n", + "\n", + "Neither is ACLED's own page, which is why the field name and placement were\n", + "confirmed against a live response before any code was written (section 2).\n", + "\n", + "Our harvester walked ACLED by `&page=N`. After 27 October that parameter\n", + "stops working.\n", + "\n", + "| Date | What |\n", + "|---|---|\n", + "| 29–30 Sep 2026 | The notice appears in downstream trackers |\n", + "| **21 Oct 2026** | Monthly pipeline cron — last run where offset still works |\n", + "| **27 Oct 2026** | Offset removed by ACLED |\n", + "| **21 Nov 2026** | First run that **must** be on cursor |\n", + "\n", + "**The failure shape is why this was urgent rather than routine.** A missed\n", + "migration does not crash on the 27th. It surfaces on 21 November as a short\n", + "harvest — quiet, a month late, and in the dataset rather than the logs. This\n", + "repository has already been burned by exactly that shape: ADR-027 exists\n", + "because `page_size=50000` made the UCDP API silently return 335,918 of\n", + "384,918 events while the harvest reported success." + ] + }, + { + "cell_type": "code", + "execution_count": 1, + "id": "995e52fd", + "metadata": { + "execution": { + "iopub.execute_input": "2026-10-12T01:43:52.286355Z", + "iopub.status.busy": "2026-10-12T01:43:52.286076Z", + "iopub.status.idle": "2026-10-12T01:43:52.293552Z", + "shell.execute_reply": "2026-10-12T01:43:52.292737Z" + } + }, + "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": [ + "quarter 2024-01-01|2024-03-31 limit=5000 country=all measured 2026-10-12T01:35:08Z at 70164d3\n", + "week 2024-03-01|2024-03-07 limit=1000 country=all measured 2026-10-10T23:11:30Z at 47089fd\n", + "fixture 2024-03-01|2024-03-07 limit=50 country=Somalia measured 2026-10-10T23:13:15Z at 47089fd\n" + ] + } + ], + "source": [ + "\"\"\"Load the committed evidence. No network access anywhere in this notebook.\"\"\"\n", + "\n", + "import hashlib\n", + "import json\n", + "from pathlib import Path\n", + "\n", + "REPORTS = Path(\"..\") / \"reports\" / \"investigation_acled_cursor\"\n", + "FIXTURES = Path(\"..\") / \"tests\" / \"fixtures\" / \"acled_cursor\"\n", + "\n", + "# Three slices were measured. They answer different questions, which is why\n", + "# all three are kept rather than only the largest.\n", + "SLICES = {\n", + " \"quarter\": \"parity_quarter.json\",\n", + " \"week\": \"parity.json\",\n", + " \"fixture\": \"parity_fixture_slice.json\",\n", + "}\n", + "\n", + "reports = {\n", + " name: json.loads((REPORTS / fname).read_text())\n", + " for name, fname in SLICES.items()\n", + "}\n", + "\n", + "for name, rep in reports.items():\n", + " sl = rep[\"slice\"]\n", + " print(f\"{name:8s} {sl['event_date']} limit={sl['limit']:<5d} \"\n", + " f\"country={sl['country']:<8s} \"\n", + " f\"measured {rep['measured_at_utc'][:19]}Z at {rep['git_sha']}\")" + ] + }, + { + "cell_type": "markdown", + "id": "2300cc6c", + "metadata": {}, + "source": [ + "## 2. What was confirmed against the live API first\n", + "\n", + "Three things were established with one authenticated request **before** any\n", + "implementation code was written, because the plan depended on all three and\n", + "the public notice is not an API contract:\n", + "\n", + "| Question | Answer, observed |\n", + "|---|---|\n", + "| Is `next_cursor` a top-level key in the JSON body? | Yes |\n", + "| What type is it? | **`int`** — e.g. `1148657`, monotonically increasing |\n", + "| Do `count` / `total_count` carry values now? | **No, still `null`** |\n", + "\n", + "**The type mattered more than it looks.** The walk starts at `cursor=0`, and\n", + "`0` is both the legitimate starting value and falsy in Python. Terminating on\n", + "`if not next_cursor` would therefore have been a live bug that passed every\n", + "happy-path test. Termination is on `next_cursor is None`, and a mutant that\n", + "swaps it for a falsiness check is in the mutation set specifically because\n", + "this observation suggested it.\n", + "\n", + "The `count`/`total_count` answer is why risk-register entry C-153 was\n", + "**amended rather than closed**: there is still no server-declared total to\n", + "validate a harvest against, so ADR-027's tolerance rule remains inapplicable\n", + "to ACLED." + ] + }, + { + "cell_type": "markdown", + "id": "5fb92c45", + "metadata": {}, + "source": [ + "## 3. The two walks, side by side\n", + "\n", + "Each slice was fetched **twice in the same session, minutes apart** — once by\n", + "`&page=N` using a frozen copy of the pre-migration production loop, once by\n", + "`&cursor=X`. Same session matters: ACLED restates history continuously, so\n", + "two fetches separated by months would differ for reasons that have nothing to\n", + "do with pagination." + ] + }, + { + "cell_type": "code", + "execution_count": 2, + "id": "2b4b2b60", + "metadata": { + "execution": { + "iopub.execute_input": "2026-10-12T01:43:52.296156Z", + "iopub.status.busy": "2026-10-12T01:43:52.295912Z", + "iopub.status.idle": "2026-10-12T01:43:52.301375Z", + "shell.execute_reply": "2026-10-12T01:43:52.300321Z" + } + }, + "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": [ + "slice events (offset) events (cursor) reqs reqs completed\n", + "------------------------------------------------------------------\n", + "quarter 98,636 98,636 20 20 True\n", + "week 7,609 7,609 8 8 True\n", + "fixture 84 84 2 2 True\n" + ] + } + ], + "source": [ + "hdr = (f\"{'slice':9s}{'events (offset)':>17s}{'events (cursor)':>17s}\"\n", + " f\"{'reqs':>6s}{'reqs':>6s}{'completed':>11s}\")\n", + "print(hdr)\n", + "print(\"-\" * len(hdr))\n", + "\n", + "for name, rep in reports.items():\n", + " completed = rep[\"checks\"][\"walks_ran_to_completion\"]\n", + " capped = rep[\"offset_hit_request_cap\"] or rep[\"cursor_hit_request_cap\"]\n", + " print(f\"{name:9s}{rep['offset_n_events']:>17,d}\"\n", + " f\"{rep['cursor_n_events']:>17,d}\"\n", + " f\"{rep['offset_requests']:>6d}{rep['cursor_requests']:>6d}\"\n", + " f\"{str(completed and not capped):>11s}\")" + ] + }, + { + "cell_type": "markdown", + "id": "74b1f198", + "metadata": {}, + "source": [ + "### The five oracles\n", + "\n", + "Parity requires **every one** of these to hold. Four are about the data; the\n", + "fifth is about the measurement itself.\n", + "\n", + "| Oracle | What it would catch |\n", + "|---|---|\n", + "| `count_equal` | A dropped or duplicated page |\n", + "| `id_sets_equal` | Different events, same count |\n", + "| `records_equal` | Same events, a differing field value |\n", + "| `digests_equal` | The ledger would record a different value |\n", + "| `walks_ran_to_completion` | **Both walks truncating at the request cap** |\n", + "\n", + "The digest is load-bearing because it is the exact value the ingestion ledger\n", + "records and `_year_is_cached()` compares against — so agreement here means\n", + "agreement on the thing the pipeline actually persists. It is also computed\n", + "over *sorted* records, which is why a different page ordering between the two\n", + "methods could not produce a false mismatch.\n", + "\n", + "Full-record equality is asserted alongside it because the digest covers only\n", + "`event_id_cnty` and `fatalities`, and would not have seen a divergence in,\n", + "say, `latitude`." + ] + }, + { + "cell_type": "code", + "execution_count": 3, + "id": "b06f950c", + "metadata": { + "execution": { + "iopub.execute_input": "2026-10-12T01:43:52.303655Z", + "iopub.status.busy": "2026-10-12T01:43:52.303427Z", + "iopub.status.idle": "2026-10-12T01:43:52.308905Z", + "shell.execute_reply": "2026-10-12T01:43:52.308206Z" + } + }, + "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": [ + "quarter count_equal=PASS id_sets_equal=PASS records_equal=PASS digests_equal=PASS walks_ran_to_completion=PASS\n", + " digest offset=af9fb35c03ac830f cursor=af9fb35c03ac830f\n", + " only_in_offset=[] only_in_cursor=[]\n", + " PARITY: True\n", + "\n", + "week count_equal=PASS id_sets_equal=PASS records_equal=PASS digests_equal=PASS walks_ran_to_completion=PASS\n", + " digest offset=ae4a6e815168e95f cursor=ae4a6e815168e95f\n", + " only_in_offset=[] only_in_cursor=[]\n", + " PARITY: True\n", + "\n", + "fixture count_equal=PASS id_sets_equal=PASS records_equal=PASS digests_equal=PASS walks_ran_to_completion=PASS\n", + " digest offset=5b73a9160f8a8256 cursor=5b73a9160f8a8256\n", + " only_in_offset=[] only_in_cursor=[]\n", + " PARITY: True\n", + "\n" + ] + } + ], + "source": [ + "oracles = [\"count_equal\", \"id_sets_equal\", \"records_equal\",\n", + " \"digests_equal\", \"walks_ran_to_completion\"]\n", + "\n", + "for name, rep in reports.items():\n", + " results = \" \".join(\n", + " f\"{o}={'PASS' if rep['checks'][o] else 'FAIL'}\" for o in oracles\n", + " )\n", + " print(f\"{name:9s} {results}\")\n", + " print(f\"{'':9s} digest offset={rep['offset_content_digest']} \"\n", + " f\"cursor={rep['cursor_content_digest']}\")\n", + " print(f\"{'':9s} only_in_offset={rep['only_in_offset']} \"\n", + " f\"only_in_cursor={rep['only_in_cursor']}\")\n", + " print(f\"{'':9s} PARITY: {rep['parity']}\\n\")" + ] + }, + { + "cell_type": "markdown", + "id": "e1561d03", + "metadata": {}, + "source": [ + "## 4. The widest slice ran at the page size production actually uses\n", + "\n", + "The first measurement used `limit=1000`. Production harvests with\n", + "`page_size=5000` (`AcledConfig`), so for a while no measurement had exercised\n", + "the page size the pipeline really runs.\n", + "\n", + "That is not a cosmetic gap. `limit` is the exact parameter the **deleted**\n", + "`len(results) < config.page_size` terminator compared against, so changing it\n", + "changes the arithmetic of the condition this migration removed." + ] + }, + { + "cell_type": "code", + "execution_count": 4, + "id": "bdcb8f19", + "metadata": { + "execution": { + "iopub.execute_input": "2026-10-12T01:43:52.311224Z", + "iopub.status.busy": "2026-10-12T01:43:52.311036Z", + "iopub.status.idle": "2026-10-12T01:43:52.315818Z", + "shell.execute_reply": "2026-10-12T01:43:52.314934Z" + } + }, + "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": [ + "week : 7,609 events limit=1000 8 requests\n", + "quarter : 98,636 events limit=5000 20 requests\n", + "\n", + "the quarter slice carries 13.0x the events of the week slice\n", + "and is the only one measured at production's page_size (5000)\n" + ] + } + ], + "source": [ + "q = reports[\"quarter\"]\n", + "w = reports[\"week\"]\n", + "ratio = q[\"cursor_n_events\"] / w[\"cursor_n_events\"]\n", + "\n", + "print(f\"week : {w['cursor_n_events']:>7,d} events \"\n", + " f\"limit={w['slice']['limit']:<5d} {w['cursor_requests']:>3d} requests\")\n", + "print(f\"quarter : {q['cursor_n_events']:>7,d} events \"\n", + " f\"limit={q['slice']['limit']:<5d} {q['cursor_requests']:>3d} requests\")\n", + "print(f\"\\nthe quarter slice carries {ratio:.1f}x the events of the week slice\")\n", + "print(f\"and is the only one measured at production's page_size \"\n", + " f\"({q['slice']['limit']})\")" + ] + }, + { + "cell_type": "markdown", + "id": "0929643d", + "metadata": {}, + "source": [ + "## 5. Recomputing the digest from the raw recorded responses\n", + "\n", + "Everything above is read from a report file. This section does not trust those\n", + "reports: it loads the **raw HTTP response envelopes** recorded during the\n", + "narrow run and re-derives the content digest independently, using the same\n", + "definition as `event_validation.py` — sort the `(event_id_cnty, fatalities)`\n", + "tuples, serialise with sorted keys, SHA-256, keep 16 hex characters.\n", + "\n", + "If this recomputation disagrees with the recorded digest, the report is wrong.\n", + "\n", + "**These envelopes are the irreplaceable artifact.** After 27 October the\n", + "offset side of this recording can never be produced again by anyone." + ] + }, + { + "cell_type": "code", + "execution_count": 5, + "id": "1c22f20e", + "metadata": { + "execution": { + "iopub.execute_input": "2026-10-12T01:43:52.318505Z", + "iopub.status.busy": "2026-10-12T01:43:52.318230Z", + "iopub.status.idle": "2026-10-12T01:43:52.327738Z", + "shell.execute_reply": "2026-10-12T01:43:52.326616Z" + } + }, + "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": [ + "offset walk: 2 recorded response(s)\n", + " page 1: 50 events next_cursor=\n", + " page 2: 34 events next_cursor=\n", + "cursor walk: 2 recorded response(s)\n", + " page 1: 50 events next_cursor=1992089\n", + " page 2: 34 events next_cursor=None\n" + ] + } + ], + "source": [ + "DIGEST_FIELDS = (\"event_id_cnty\", \"fatalities\") # acled.py\n", + "\n", + "\n", + "def content_digest(events: list[dict]) -> str:\n", + " \"\"\"Re-derivation of validate_events(...).content_digest.\"\"\"\n", + " rows = sorted(\n", + " tuple(ev.get(f, 0) for f in DIGEST_FIELDS) for ev in events\n", + " )\n", + " payload = json.dumps(rows, sort_keys=True).encode()\n", + " return hashlib.sha256(payload).hexdigest()[:16]\n", + "\n", + "\n", + "walks = {\n", + " name: json.loads((FIXTURES / f\"{name}_walk.json\").read_text())\n", + " for name in (\"offset\", \"cursor\")\n", + "}\n", + "\n", + "for name, pages in walks.items():\n", + " print(f\"{name} walk: {len(pages)} recorded response(s)\")\n", + " for i, page in enumerate(pages, start=1):\n", + " cursor = page.get(\"next_cursor\", \"\")\n", + " print(f\" page {i}: {len(page['data']):>3d} events \"\n", + " f\"next_cursor={cursor}\")" + ] + }, + { + "cell_type": "code", + "execution_count": 6, + "id": "5e7efed2", + "metadata": { + "execution": { + "iopub.execute_input": "2026-10-12T01:43:52.329962Z", + "iopub.status.busy": "2026-10-12T01:43:52.329705Z", + "iopub.status.idle": "2026-10-12T01:43:52.337666Z", + "shell.execute_reply": "2026-10-12T01:43:52.336913Z" + } + }, + "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": [ + "walk events recomputed digest matches report\n", + "------------------------------------------------------\n", + "offset 84 5b73a9160f8a8256 True\n", + "cursor 84 5b73a9160f8a8256 True\n", + "\n", + "report recorded : 5b73a9160f8a8256\n", + "symmetric difference : (empty)\n", + "full records equal : True\n" + ] + } + ], + "source": [ + "recorded = reports[\"fixture\"][\"cursor_content_digest\"]\n", + "\n", + "print(f\"{'walk':8s}{'events':>8s} {'recomputed digest':20s}\"\n", + " f\"{'matches report':>16s}\")\n", + "print(\"-\" * 54)\n", + "for name, pages in walks.items():\n", + " events = [ev for page in pages for ev in page[\"data\"]]\n", + " digest = content_digest(events)\n", + " print(f\"{name:8s}{len(events):>8d} {digest:20s}\"\n", + " f\"{str(digest == recorded):>16s}\")\n", + "\n", + "offset_events = [ev for p in walks[\"offset\"] for ev in p[\"data\"]]\n", + "cursor_events = [ev for p in walks[\"cursor\"] for ev in p[\"data\"]]\n", + "offset_ids = {ev[\"event_id_cnty\"] for ev in offset_events}\n", + "cursor_ids = {ev[\"event_id_cnty\"] for ev in cursor_events}\n", + "\n", + "def by_id(events: list[dict]) -> list[dict]:\n", + " return sorted(events, key=lambda e: e[\"event_id_cnty\"])\n", + "\n", + "\n", + "print(f\"\\nreport recorded : {recorded}\")\n", + "print(f\"symmetric difference : {offset_ids ^ cursor_ids or '(empty)'}\")\n", + "print(f\"full records equal : \"\n", + " f\"{by_id(offset_events) == by_id(cursor_events)}\")" + ] + }, + { + "cell_type": "markdown", + "id": "98f65ba6", + "metadata": {}, + "source": [ + "Note what the page structure shows about termination — and note carefully\n", + "what it does **not** show.\n", + "\n", + "The final page carries `next_cursor: null`, and the walk stops there because\n", + "the server said so, not because the page looked short. Under cursor pagination\n", + "that is the only authoritative terminator: a short page mid-series is not the\n", + "end, and the old code's `len(results) < limit` inference would have stopped\n", + "early on one.\n", + "\n", + "**But the final page was short in every slice measured** — 34 of 50 here,\n", + "609 of 1,000 in the week slice, 3,636 of 5,000 in the quarter. So the case\n", + "where the old code is *unambiguously* wrong, a final page that is exactly\n", + "full, never occurred live. The old code would have terminated correctly on\n", + "all three of these slices.\n", + "\n", + "That case is covered only by `test_full_final_page_with_null_cursor_terminates`,\n", + "which constructs it synthetically. It is a real limitation of this evidence\n", + "and it is the same limitation as the negative H4 result in section 6: the\n", + "measurement confirms the two methods agree, and does *not* demonstrate the\n", + "new terminator succeeding where the old one would have failed." + ] + }, + { + "cell_type": "markdown", + "id": "a560b8e8", + "metadata": {}, + "source": [ + "## 6. What this notebook does **not** establish\n", + "\n", + "Stated plainly, because an evidence document that only lists its strengths is\n", + "advocacy.\n", + "\n", + "1. **One hypothesis came back negative.** No page ever arrived short *while\n", + " the cursor was still live*, across any slice. So the deleted short-page\n", + " break would have agreed with the new code on everything measured here. The\n", + " deletion is structurally correct — a short page genuinely is not the end of\n", + " a cursor series — but it is **not vindicated by this data**. The honest\n", + " basis for it is that restoring the break is caught by the mutation drill,\n", + " not that the measurement caught it.\n", + "\n", + "2. **The decisive case was never observed live.** The final page was short\n", + " in all three slices, so the one configuration where the deleted terminator\n", + " is unambiguously wrong -- a final page containing exactly `limit` events,\n", + " followed by `next_cursor: null` -- was never produced by ACLED during the\n", + " measurement. It is covered by a synthetic unit test only. This is the same\n", + " gap as point 1, seen from the other side.\n", + "\n", + "3. **Parity on three slices is not parity on all history.** The slices are\n", + " 2024, all countries, no event-type filter. Nothing here measures behaviour\n", + " across an ACLED restatement, or at the year boundaries the production\n", + " harvester actually walks.\n", + "\n", + "4. **A short harvest would still be recorded as a success.** `n_events` goes\n", + " into the ingestion ledger and nothing reads it back. This migration makes\n", + " the *loop* unable to truncate silently; it does not make the *ledger* able\n", + " to notice if something else does. Tracked as C-359 and issue #488, and\n", + " deliberately not fixed under deadline.\n", + "\n", + "5. **This notebook is not a test.** CI does not execute it. The claim it\n", + " renders is enforced by `tests/test_acled_cursor_parity_fixture.py`, which\n", + " replays the same recorded envelopes through the real `fetch_paginated()`\n", + " and asserts the event count, page count and digest. The notebook is the\n", + " human-readable argument *about* that test, not a substitute for it." + ] + }, + { + "cell_type": "markdown", + "id": "31965aca", + "metadata": {}, + "source": [ + "## 7. Verdict" + ] + }, + { + "cell_type": "code", + "execution_count": 7, + "id": "b2c4b0b0", + "metadata": { + "execution": { + "iopub.execute_input": "2026-10-12T01:43:52.340376Z", + "iopub.status.busy": "2026-10-12T01:43:52.340129Z", + "iopub.status.idle": "2026-10-12T01:43:52.346114Z", + "shell.execute_reply": "2026-10-12T01:43:52.345170Z" + } + }, + "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": [ + "ACLED cursor-vs-offset pagination parity\n", + "================================================\n", + "slices measured : 3\n", + "events compared in total : 106,329\n", + "every oracle held : True\n", + "no walk hit its cap : True\n", + "\n", + "At the timestamps and against the slices recorded above, ACLED's\n", + "cursor pagination returned exactly the same events as its offset\n", + "pagination -- identical counts, identical event IDs, identical\n", + "field values, and an identical content digest, on walks that each\n", + "ran to the server's own declared end of series.\n" + ] + } + ], + "source": [ + "all_parity = all(rep[\"parity\"] for rep in reports.values())\n", + "total_events = sum(rep[\"cursor_n_events\"] for rep in reports.values())\n", + "no_caps = all(\n", + " not (rep[\"offset_hit_request_cap\"] or rep[\"cursor_hit_request_cap\"])\n", + " for rep in reports.values()\n", + ")\n", + "\n", + "print(\"ACLED cursor-vs-offset pagination parity\")\n", + "print(\"=\" * 48)\n", + "print(f\"slices measured : {len(reports)}\")\n", + "print(f\"events compared in total : {total_events:,d}\")\n", + "print(f\"every oracle held : {all_parity}\")\n", + "print(f\"no walk hit its cap : {no_caps}\")\n", + "print()\n", + "if all_parity and no_caps:\n", + " print(\"At the timestamps and against the slices recorded above, ACLED's\")\n", + " print(\"cursor pagination returned exactly the same events as its offset\")\n", + " print(\"pagination -- identical counts, identical event IDs, identical\")\n", + " print(\"field values, and an identical content digest, on walks that each\")\n", + " print(\"ran to the server's own declared end of series.\")\n", + "else:\n", + " print(\"NOT PARITY -- do not read this notebook as supporting the change.\")" + ] + } + ], + "metadata": { + "kernelspec": { + "display_name": "Python 3", + "language": "python", + "name": "python3" + }, + "language_info": { + "codemirror_mode": { + "name": "ipython", + "version": 3 + }, + "file_extension": ".py", + "mimetype": "text/x-python", + "name": "python", + "nbconvert_exporter": "python", + "pygments_lexer": "ipython3", + "version": "3.13.7" + } + }, + "nbformat": 4, + "nbformat_minor": 5 +} diff --git a/reports/investigation_acled_cursor/README.md b/reports/investigation_acled_cursor/README.md index 5902aee..8b97662 100644 --- a/reports/investigation_acled_cursor/README.md +++ b/reports/investigation_acled_cursor/README.md @@ -34,11 +34,19 @@ same data — and it can only be proven **before 27 October**, while both method - [`pre_analysis_plan.md`](pre_analysis_plan.md) — the hypotheses, the falsification criteria and the pre-committed decision table, registered before any implementation code - [`findings.md`](findings.md) — the answer, written after the measurement -- [`parity.json`](parity.json) — the headline result: 7,609 events, all countries, identical both - ways +- [`parity_quarter.json`](parity_quarter.json) — the headline result: 98,636 events, a full + quarter, all countries, at production's own `page_size=5000`, identical both ways +- [`parity.json`](parity.json) — the first run: 7,609 events, one week, `limit=1000`, identical + both ways. Superseded as the headline but kept: it is a second independent measurement, taken + two days earlier at a different page size - [`parity_fixture_slice.json`](parity_fixture_slice.json) — the narrow run (84 events, Somalia) that produced the committed replay fixtures. Separate because an ACLED event is ~1.4 KB, so the headline slice would have been a 10 MB fixture per walk — good evidence, bad fixture. +- [`../../notebooks/acled_cursor_parity.ipynb`](../../notebooks/acled_cursor_parity.ipynb) — the + human-readable argument, committed with outputs. Reads only the files above and the recorded + envelopes; makes no network request, so it still renders after 27 October. It recomputes the + content digest from the raw envelopes rather than reprinting the reported one, and section 6 + lists what the evidence does **not** establish. ## Methodology Note diff --git a/reports/investigation_acled_cursor/findings.md b/reports/investigation_acled_cursor/findings.md index 5e111ea..437b430 100644 --- a/reports/investigation_acled_cursor/findings.md +++ b/reports/investigation_acled_cursor/findings.md @@ -53,7 +53,7 @@ It is now confirmed against a real response. ### 1.2 The two walks returned identical data -H1 confirmed. Four independent oracles, all agreeing: +H1 confirmed, on two slices measured two days apart. Four independent oracles, all agreeing: | Oracle | Result | |---|---| @@ -65,6 +65,31 @@ H1 confirmed. Four independent oracles, all agreeing: Slice: all countries, `2024-03-01|2024-03-07`, `limit=1000`, no event-type filter. Recorded in `parity.json` with the UTC timestamp and git sha. +**Re-measured wider, at production's real page size (2026-10-12).** The run above used +`limit=1000`; production harvests with `page_size=5000` (`AcledConfig`). That gap meant the +headline measurement had never exercised the page size the pipeline actually uses. Repeated over a +full quarter at the production limit, recorded in `parity_quarter.json`: + +| Oracle | Result | +|---|---| +| Event count | 98,636 = 98,636 | +| `event_id_cnty` set | identical, symmetric difference empty | +| Full-record equality over sorted lists | equal | +| `content_digest` | `af9fb35c03ac830f` = `af9fb35c03ac830f` | +| Both walks ran to completion (neither hit the request cap) | true | + +Slice: all countries, `2024-01-01|2024-03-31`, `limit=5000`, no event-type filter. 20 requests per +walk, at `git_sha` `70164d3` — the commit that becomes `v1.14.0`. Nineteen pages of exactly 5,000 +events, then a terminal page of 3,636 with `next_cursor: null`. + +That is **13x the events** of the headline run and the first measurement at the page size +production uses. It does not merely add rows: `limit` is the parameter the deleted +`len(results) < config.page_size` break compared against, so changing it changes the arithmetic of +the terminator this migration removed. Both slices still agree on every oracle. + +Why a second same-session comparison rather than the end-to-end ledger check that was planned — +see §5.1. + The digest is the load-bearing oracle because it is the value the ingestion ledger records and `_year_is_cached()` compares against, and because `validate_events()` sorts before hashing — so a different page ordering between the walks could not have produced a false mismatch. Full-record @@ -139,6 +164,28 @@ asserting the **request count** and the **sequence of cursors sent** — ## 2. What we THINK we know (inferred from evidence) +### 2.0 The decisive case never occurred live + +Noticed 2026-10-12 while writing the notebook, which prints the page structure +cell-by-cell and made it visible. + +**The final page was short in all three measured slices** — 3,636 of `limit=5000` (quarter), +609 of 1,000 (week), 34 of 50 (fixture). None was exactly full. + +That matters because a full final page is the one configuration where the deleted +`len(results) < config.page_size` break is *unambiguously* wrong: it would have accepted a full +final page as "more to come", asked for one more page, and then had to decide what an empty +response meant. A short final page, by contrast, is a case the old code handled correctly. + +So the live measurement demonstrates that the two methods **agree**; it does not demonstrate the +new terminator succeeding where the old one fails. That case exists only in +`test_full_final_page_with_null_cursor_terminates`, which constructs it synthetically. + +This is the same gap as the negative H4 result below, approached from the other side, and the two +should be read together: nothing ACLED returned during this investigation would have broken the +old code. The case for the change rests on the contract (`next_cursor` is the server's own +declaration, page length is an inference) and on the mutation drill, not on an observed failure. + ### 2.1 The old short-page heuristic would probably have worked on this slice too H4 came back **negative**: no page returned fewer than `limit` events while the cursor was still @@ -252,7 +299,40 @@ pagination loop cannot silently truncate**, which the mutation drill demonstrate --- -## 5. What the code review changed, and one thing it found in this document's own instrument +## 5. What the reviews changed, and one thing they found in this document's own instrument + +### 5.1 The end-to-end check the plan called the most valuable one was unsound + +Raised by the operator at the release gate, 2026-10-12, before publishing: had the full ritual +actually run? The code review had (two rounds, seven findings, all fixed). What had **not** run was +the approved plan's Verification check 9: + +> Live end-to-end: `harvest_acled.py` for one year against a cleared snapshot → ledger `n_events` +> **and** `content_digest` match the pre-migration ledger entry for that year. + +annotated in that plan as *"the one that would actually catch a botched migration."* + +**On re-reading, that check cannot do the job claimed for it.** The only pre-migration per-year +entry in `provenance/acled/ingestion_ledger.jsonl` is `version: "2025_2025"`, harvested +2026-05-07: 411,089 events, digest `21f9337c4f795b84`. **ACLED restates history** — it backfills +and corrects past events continuously. A re-harvest of 2025 in October 2026 would differ from the +May figure for reasons that have nothing to do with pagination, and the comparison could not +separate "the pagination is broken" from "the source was revised, as it always is". A mismatch +would be uninterpretable, and — worse — a *match* would have been read as vindication it could not +support. The flaw is in the oracle, not the execution: no run of check 9 would have been +informative. + +The sound instrument for this question is the one already built: both walks in the **same session, +minutes apart**, where restatement cannot differentially affect them. So the remedy was to widen +*that* rather than to run check 9 — see the second table in §1.2. The planned check is withdrawn, +not deferred. + +This is also the honest reason the widening happened at all: it was not in the plan. It exists +because the plan's strongest stated check turned out to be the weakest, and that was noticed at the +gate rather than after the deploy. + +### 5.2 What the code review changed + `/code-review medium` on PR #490 raised five findings. Four were real; all are fixed. Recorded here rather than only in the PR, because two of them change claims made above. diff --git a/reports/investigation_acled_cursor/parity_quarter.json b/reports/investigation_acled_cursor/parity_quarter.json new file mode 100644 index 0000000..fa49b3d --- /dev/null +++ b/reports/investigation_acled_cursor/parity_quarter.json @@ -0,0 +1,37 @@ +{ + "question": "Do ACLED's cursor and offset pagination return the same events?", + "pre_analysis_plan": "reports/investigation_acled_cursor/pre_analysis_plan.md", + "method": "Harvest one slice twice against live ACLED -- once by &page=N terminating on a short or empty page (frozen copy of the pre-#487 production loop), once by &cursor=X terminating only on next_cursor is None. Compare on four independent oracles: event count, event_id_cnty set, full-record equality over the sorted lists, and the content_digest the ingestion ledger records. A detection drill runs first as a precondition.", + "measured_at_utc": "2026-10-12T01:35:08.828796+00:00", + "git_sha": "70164d3", + "slice": { + "event_date": "2024-01-01|2024-03-31", + "limit": 5000, + "country": "all", + "event_types": "all (no event_type filter)" + }, + "offset_requests": 20, + "cursor_requests": 20, + "max_requests": 40, + "offset_hit_request_cap": false, + "cursor_hit_request_cap": false, + "h4_short_pages_with_live_cursor": [], + "h5_count_total_count": { + "count": null, + "total_count": null + }, + "offset_n_events": 98636, + "cursor_n_events": 98636, + "offset_content_digest": "af9fb35c03ac830f", + "cursor_content_digest": "af9fb35c03ac830f", + "only_in_offset": [], + "only_in_cursor": [], + "checks": { + "count_equal": true, + "id_sets_equal": true, + "records_equal": true, + "digests_equal": true, + "walks_ran_to_completion": true + }, + "parity": true +} diff --git a/reports/investigation_acled_cursor/pre_analysis_plan.md b/reports/investigation_acled_cursor/pre_analysis_plan.md index e7ecd16..dbcbfde 100644 --- a/reports/investigation_acled_cursor/pre_analysis_plan.md +++ b/reports/investigation_acled_cursor/pre_analysis_plan.md @@ -252,3 +252,4 @@ tests do not actually constrain the thing this change is about. |---|---|---| | 2026-10-10 | **The measurement was split across two slices**, not one as §6 Step 4 specified. A headline run (all countries, `limit=1000`, 7,609 events) answers H1/H3/H4/H6; a narrow run (Somalia, `limit=50`, 84 events) produced the committed fixtures. | §6 Step 5 required the raw envelopes to be committed as fixtures, and Step 4's slice sizing was chosen for walk depth without costing out the artifact. An ACLED event is ~1.4 KB, so the headline walk recorded 9.9 MB per side — against a repository whose largest existing fixture is 9.4 KB. Trimming the envelopes would have broken the digest oracle, and shrinking the headline slice would have weakened the evidence. The two artifacts have different jobs, so they get different slices and both are committed. `--write-fixtures` is now off by default so this cannot be stumbled into. | | 2026-10-10 | **A ninth and tenth mutant were added** to §6 Step 7's six: M7 (drop the empty-page-with-live-cursor raise) and M8 (terminate on falsiness rather than `is None`). | M8 was prompted by a finding, not by the plan: `next_cursor` turned out to be an `int`, so `0` is a legitimate cursor *and* falsy. `if not next_cursor` is therefore a live hazard that the registered six did not probe. Adding mutants strengthens the drill and cannot weaken it, so this was taken as within the spirit of Step 7. | +| 2026-10-12 | **A third slice was measured after the implementation shipped** — all countries, `2024-01-01|2024-03-31`, `limit=5000`, 98,636 events, 20 requests per walk, recorded in `parity_quarter.json`. Post-hoc by construction, since the plan was closed. | Two reasons, and the second is the real one. (1) Both earlier slices used `limit=1000` or `50`; production harvests at `page_size=5000`, so no measurement had ever run at the page size the pipeline uses — and `limit` is the exact parameter the deleted `len(results) < page_size` break compared against. (2) The plan's Verification check 9, annotated as *"the one that would actually catch a botched migration"*, was found at the release gate to rest on an invalid oracle: ACLED restates history, so the pre-migration ledger digest for 2025 cannot be compared against a re-harvest a year later. Check 9 is **withdrawn, not deferred** — see findings.md §5.1. This slice is its replacement, and it is reported as a post-hoc confirmation rather than a registered prediction. Parity held on all five oracles; H4 remained negative here too. | From 9d05555d1d7c8ede0841e0eacfa2ac7af4af9eb1 Mon Sep 17 00:00:00 2001 From: Polichinl Date: Mon, 12 Oct 2026 04:01:48 +0200 Subject: [PATCH 2/2] test: follow views-models' config rename instead of asserting the old name (#489) Two assertions in the shining_codex falsification audit required config_deployment.py in views-models/models/shining_codex/configs/. That file was renamed to config_maturity.py upstream, so both failed locally while CI stayed green -- CI has no views-models checkout, so the class skips there entirely. A test that only fails on the one machine that can actually evaluate it is worse than no test. What F1 asserts is that the config ModelPathManager demands at init is present, not what it is called, so the assertion follows the file. The failure message now names the rename and asks the next reader to check whether it was renamed again rather than deleted, which is the distinction that matters and the one that cost time this time. Full suite is exit 0 with zero failures after this. Closes #489 Co-Authored-By: Claude Opus 5 (1M context) Claude-Session: https://claude.ai/code/session_01EfeuULbPTqrx4qtNki5dy9 --- tests/test_falsification_shining_codex.py | 24 +++++++++++++++-------- 1 file changed, 16 insertions(+), 8 deletions(-) diff --git a/tests/test_falsification_shining_codex.py b/tests/test_falsification_shining_codex.py index 337f463..4549f7d 100644 --- a/tests/test_falsification_shining_codex.py +++ b/tests/test_falsification_shining_codex.py @@ -3,8 +3,12 @@ Claim: "I can execute (load data, train and evaluate) the new model shining_codex now." -F1 (resolved): ModelPathManager requires config_deployment.py and -config_sweep.py — both now exist. +F1 (resolved): ModelPathManager requires a maturity/deployment config +and config_sweep.py — both exist. The first was named +config_deployment.py when this audit was written and was renamed to +config_maturity.py in views-models (#489); the assertion follows the +file, since what F1 tests is that the config ModelPathManager demands +is present, not what it is called. F2 (resolved): Feature set renames ged_sb_best → lr_ged_sb (matching novel_heuristics column contract). @@ -28,12 +32,16 @@ reason=SKIP_REASON, ) class TestF1ConfigFiles: - """F1 (resolved): ModelPathManager requires config_deployment.py - and config_sweep.py — both are checked at init with validate=True.""" + """F1 (resolved): ModelPathManager requires the maturity config and + config_sweep.py — both are checked at init with validate=True.""" - def test_config_deployment_exists(self) -> None: - path = _SHINING_CODEX / "configs" / "config_deployment.py" - assert path.exists() + def test_config_maturity_exists(self) -> None: + path = _SHINING_CODEX / "configs" / "config_maturity.py" + assert path.exists(), ( + "views-models renamed config_deployment.py to " + "config_maturity.py; if this fails, check whether it was " + "renamed again rather than deleted (#489)" + ) def test_config_sweep_exists(self) -> None: path = _SHINING_CODEX / "configs" / "config_sweep.py" @@ -41,7 +49,7 @@ def test_config_sweep_exists(self) -> None: def test_all_required_configs_exist(self) -> None: required = [ - "config_deployment.py", + "config_maturity.py", "config_hyperparameters.py", "config_meta.py", "config_partitions.py",