Repository navigation
Jurisprudence extraction errata, label-based paragraph ids, PDF footnotes - #2
Merged
Merged
Conversation
… footnote numbers
Root causes of the defects in vclt-interpreter docs/UNHRDB_ERRATA.md:
- clean_extract._clean_page_text dropped whole PyMuPDF blocks whose top
edge sat in the header/footer bands, i.e. whole paragraphs that start
near the top or bottom of a page (Toonen 5.1, 8.2, end of 8.6; Estrella
1.3, 1.6, 9.2). Jurisprudence now keeps margin blocks that look like body
text (keep_body_in_margins); GC/SP callers are unchanged.
- ocr_jurisprudence read Tesseract TSV with csv's default quoting: a word
starting with '"' swallowed the following rows and their line breaks
(Coeriel 2.2 holding 2.3-4.1, raw coordinates in 2.2 and 7.3). Parse with
QUOTE_NONE; ingestion rebuilds leaked rows into lines, so the existing
OCR output is repaired without re-OCR.
- A "safe correction" rewrote every "authors" to "author's"; removed, and
the unambiguous forms are undone at ingestion.
- PDF parser: "article\n17." after an unfinished sentence, "8~5", "-158-"
page numbers, headings ending in ":", endnotes and "-*-" notes, numbered
and joint opinion headings, an article list read as paragraph "2,".
- DOCX: decimal IDs with a closing period ("2.6.") were not paragraph
starts; joint opinions shared the previous opinion's namespace.
- Footnotes: [[fn:N]] and n used the Word w:id, which counts title-page
"*" notes and continuationNotice records; now the printed number.
- Outcome: classified from the operative paragraph(s) of the majority text
instead of the catalogue title and a 22,000-character tail; unresolved
rather than guessed. adoptionDate is no longer the catalogue's submission
date; documents.json gains adoptionDateIso / submittedDateIso.
Tests: tests/python/test_jur_errata.py on the errata documents' official
texts. Old vs new on the sources still on disk: 971 DOCX, continuity issues
264 -> 212, none worse; 103 PDF, 117 -> 29, none worse.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Read-only scan of docs/jur (what build_db.py loads into the API). Extends vclt-interpreter/stage_b/check_continuity.py with part restarts, merged and truncated paragraphs, TSV residue, page numbers, glued headings and endnotes, opinion namespaces, and an optional spot-check against the live API. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…aph numbers
Found by comparing old and new extraction on every source still on disk
while building the paragraph redirect map:
- An opinion appended under "APPENDIX" with no heading of its own was
dropped once endnotes stopped being glued on (CCPR/C/38/D/275/1988).
- "CCPR/C/50/D/428/1990 / Annex / English / Page 4" at the top of OCR pages
read as a heading and closed the open paragraph; the rest of it was lost.
- Text after a heading whose number OCR garbled ("10.% The Human Rights
Committee ...") was dropped; it is kept as a generated paragraph and named
from its neighbours when they leave one choice ("§.2" after 5.1 is 5.2).
- All-caps titles in opinion and appendix sections are not paragraphs.
On the 1,244 local sources the only text the new extractor drops is
endnotes, TSV residue and front matter.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Jurisprudence paragraph ids were positional ({docId}-{idx:04d}): recovering
one missing paragraph renumbered every later one and broke saved citations.
They are now {docId}:{label} (ccpr-c-50-d-488-1992:8.2, ...:OP1-U3); a
repeated label gets ~2, ~3 and unlabelled records u1, u2. GC and SP ids are
unchanged.
build_jurisprudence_shards.py reads the previous build before overwriting
it, aligns its paragraphs to the new ones by text (jur_paragraph_ids.align)
and keeps a cumulative, chained map in docs/jur/paragraph-redirects.json:
every id ever published -> its current id, or null when the old record was
not a paragraph of the decision. Each paragraph carries its legacyIds.
Matches the text cannot settle are written to a review queue
(--review-out); reviewed answers come back with --decisions. A document
whose rebuild lost most of its text is reported, not redirected.
Frontend: one resolver (resolveParagraphId) for ?p= links, the reader,
jumps and the workspace; it uses shard legacyIds, then /api/paragraph.
Bookmarks, notes and pins move to the new id once it is known; nothing is
dropped when resolution fails.
Migration run on the current corpus: 152,731 redirects, all to the same
text, none to review. Re-extraction of the 1,244 documents with local
sources keeps 99.2% of ids; 281 redirected, 47 removed, 84 to review.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…graph "... legal aid system. ~92-" read as text that stops mid-sentence, so the "14." opening the next page was taken for a wrapped reference and glued to 12.2 (CCPR/C/39/D/250/1987). Page numbers OCR'd with "~" are noise, and a trailing page number does not count when deciding whether text ended. Jev's review of the redirect queue surfaced it. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
scripts/jev_redirect_review.py sends the review queue of build_jurisprudence_shards.py to Jev (TypeSafe), reusing the travaux project's client and key: per old paragraph, which candidate holds its passage, or none. Picks at p >= 0.9 (the travaux threshold) go to the decisions file the next build reads with --decisions; the rest stay for a person with Jev's suggestion and probabilities. Extraction regressions are not asked. Candidates now carry head and tail excerpts, since a passage merged into another paragraph usually sits at its end. Simulated re-ingestion of the 1,244 documents with local sources: 65 questions, 23 accepted, all checked by hand (5 corrected the provisional match); 42 left for review. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
PDF and OCR'd decisions had no footnotes: clean_extract cut the notes off
each page, the superscript references were deleted as stray digits, and the
endnotes of old compilations were glued to the last paragraph or dropped.
They now come out as on the DOCX route, [{n, text}] with [[fn:n]] markers:
- Modern PDFs: superscript references become markers; the notes are read
below the footnote rule (or, in older Word exports with body-size notes,
from the first line that opens with a small non-superscript number),
page by page, a note continuing onto the next page included. Numbers must
run on or restart at 1, so a note line that starts with a figure stays
part of its note.
- Old compilations and scans: "Notes" / "Note N" / "-*-" endnote blocks and
"a/" notes at the foot of a page are parsed; "a/", "(1)" and bare
"word.1" references are linked, the k-th reference to a mark taking the
k-th note with it (marks restart per page). OCR-garbled marks on their own
line ("!I", "BI", "£/") are numbered by place and linked where the text
repeats them. Lettered notes are numbered 1, 2, ... and keep `mark`.
- A note nothing legible refers to is kept on the decision's last
paragraph with `anchored: false`; title-page "*" notes are skipped, as on
the DOCX route.
Found on the way and fixed: endnotes swallowed the opinion that followed
them ("APPENDIX I", or a page opening in prose); "[Original: Spanish]"
above an English translation made both routes drop annexed opinions (11 of
966 local DOCX, CCPR/C/144/D/2982/2017); short heading underlines were not
footnote rules.
Checked: footnotes from the PDF equal those from the DOCX (paragraph,
number, text) for CCPR/C/139/D/2929/2017 (12), CCPR/C/131/D/3163/2018 (33)
and CCPR/C/82/D/903/1999 (15). On the 286 local PDF/OCR sources: 770 notes
in 117 documents, 85% linked to the citing sentence, no dangling markers;
label continuity unchanged (none worse); only front matter, headers and
title-page notes leave the text.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
PDF notes of old UN compilations are lettered ("a/"); markers and the
popover now show the printed `mark` instead of the running number, which
stays the marker key. Notes whose reference OCR garbled travel on the
decision's last paragraph as `anchored: false`; the reader and the dossier
list them under the paragraph ("Notes not marked in the text"), each
opening the usual popover, whose head says so. The print list uses the
mark too. Numbered footnotes render as before.
Checked in the browser on CCPR/C/18/D/74/1980 rebuilt from its PDF (c/ in
para. 1.6; a/ and b/ unanchored under para. 11) and on CRPD/C/GC/6.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
compact_document and lite_document copy a whitelist of fields, and alternativeIds (added to jurisprudence_info.json with the Narymbaev merge, 266f030) was not on it, so any rebuild broke links to the retired stub ccpr-c-133-d-2904-2016-2907-2016. Caught by Playwright R22 run against a rebuilt docs/jur. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Fixes the jurisprudence defects listed in vclt-interpreter
docs/UNHRDB_ERRATA.mdat their source in the extraction pipeline. The ids of jurisprudence paragraphs become label-based, with a redirect map. PDF footnotes are recovered. The frontend learns to read both.This does not change the published data. The pipeline fixes take effect at the next re-ingestion, which first needs the raw sources downloaded again (
ohchr_jurisprudence/rawis empty). Merging ships the frontend (docs/assets) and the tooling. The frontend works with today's data and with the new format.Extraction (
ingest_jurisprudence.py,clean_extract.py,ocr_jurisprudence.py)unresolvedinstead of a guess.adoptionDate.adoptionDateIsoandsubmittedDateIsoare derived on every build.nuse the printed number, not Word's internal footnote id. Title-page*notes had shifted Sudalenko by 3, and Lula by 4 and then 7.Paragraph ids (
jur_paragraph_ids.py,build_jurisprudence_shards.py){docId}:{label}(ccpr-c-50-d-488-1992:8.2). A repeated label gets~2; unlabelled records getu1,u2. General Comment and Special Procedures ids are unchanged.docs/jur/paragraph-redirects.json, and every paragraph lists itslegacyIds. Matches it can't decide go to a review queue, and reviewed answers come back with--decisions.scripts/jev_redirect_review.pysends that queue to Jev, using the travaux client. Answers at p ≥ 0.9 are accepted; on the simulation that was 23 of 65, all checked by hand.alternativeIds, which broke links to the retired Narymbaev id.Frontend (
docs/assets/app.js,app.css)?p=links, the reader, jumps and the workspace. It uses the shardlegacyIds, then/api/paragraph. Bookmarks, notes and pins move to the new id, and none is dropped when resolution fails.a/).Tests and checks
tests/python: 63 tests on the official texts of the errata documents.tests/paragraph-ids.unit.mjs(12) andtests/footnote-marks.unit.mjs(4).test.skipF9), both on today'sdocs/jurand on a rebuilt one with label ids. The pre-push hook ran it again.clipboard-read, and WebKit blocks clipboard reads in headless mode.scripts/audit_jur_quality.pyrepeats the corpus scan (1,566 of 4,333 documents flagged today).Before re-ingesting
catalog.jsonland the manifest.ingest_jurisprudence.py.build_jurisprudence_shards.py→jev_redirect_review.py→ rebuild with--decisions→scripts/audit_jur_quality.py.🤖 Generated with Claude Code