Skip to content

feat: add real documents that carry mathematics - #5

Open
tobocop2 wants to merge 2 commits into
xberg-io:mainfrom
tobocop2:feat/math-corpus
Open

feat: add real documents that carry mathematics#5
tobocop2 wants to merge 2 commits into
xberg-io:mainfrom
tobocop2:feat/math-corpus

Conversation

@tobocop2

@tobocop2 tobocop2 commented Aug 13, 2026

Copy link
Copy Markdown
Member

Closes #4.

What this adds

Every supported format that can hold mathematics is now represented by a real published document. The set answers a question the hand-written fixtures could not: whether extraction works on files people actually publish.

It found nine defects. Three returned no formulas at all on real files while their fixtures passed.

defect document that found it
A ZIP-based office format identified from a 4096-byte header a 27-slide deck whose ppt/presentation.xml sits at byte 107,190
An EPUB href is a URL, a ZIP entry name is not an open textbook with %3A in its file names
An ODT list item's frames were never read the OASIS OpenDocument specification, 0 of its 68 formulas
A JATS formula inside a <p> was never captured JOSS and eLife articles
An embedded formula object carries its own DOCTYPE a LibreOffice math guide
The AsciiMath parser panics on a multi-byte character a sensor specification using
DocBook ships as .xml 8 Khronos reference pages
PowerPoint writes equations as a14:m an equation deck
An HWP equation inside a sentence a Korean 수식 document

Two are bugs in xberg that cost text and images as well as formulas, filed as xberg-io/xberg#1426 and #1427.

Licensing

136 documents are permissively licensed and are vendored. 12 carry ShareAlike, GPL or GFDL terms, and the redistribution policy in LICENSES.md keeps copyleft content out of the repository, so those are recorded as reference only: source and licence, without the bytes. That follows the treatment hiertext already has.

Four documents state a dual licence, CC-BY OR GPL-3.0-or-later, and are vendored under the CC-BY arm.

MATH_PROVENANCE.md lists every document with its source, licence and notation. Each licence was read from the source repository, the document itself, or the publisher's stated terms.

What is in this commit, and what is not

81 text documents are committed here. The other 55 vendored documents are binary formats, which .gitignore keeps out of git by design, so their bytes reach the corpus through scripts/publish_corpus.py and corpus.lock.json. They are ready to publish and are not part of this diff.

corpus.lock.json is untouched: the publisher writes it, so hand-editing it would point the manifest at objects the bucket does not hold yet.

Getting the binaries in

A gitignored file cannot travel in a pull request, so scripts/fetch_math_binaries.py downloads all 55 from the sources in MATH_PROVENANCE.md and writes each to the path it belongs at:

python3 scripts/fetch_math_binaries.py
python3 scripts/publish_corpus.py --bucket xberg-test-documents

scripts/math-binaries.json carries each source URL with the sha256 and size of the file this corpus was validated against. A download whose digest does not match is written to <path>.mismatch and reported, so a source that changed upstream cannot quietly replace a good file. Files already present and matching are skipped, so the script is safe to re-run.

No ground truth

These documents ship without ground_truth/ files, and MATH_PROVENANCE.md says why. They bring no upstream ground truth, and deriving it by running the extractor would record today's output as the definition of correct, so a later regression would match the golden file and pass.

Four of them do carry 1,046 LaTeX strings written by their own authors, in MathML annotation-tex. That is real formula-level ground truth, independent of any extractor. README.md places targets of that kind in ground_truth/structured/, which the repository does not yet contain, so this change does not invent the format.

Needs a decision

Five LibreOffice documents are vendored, and their upstream metadata disagrees with itself. GitHub's licence API reports GPL-3.0 for LibreOffice/core, while the project states MPL-2.0 and LGPL-3.0-or-later, and this repository already vendors LibreOffice/core as MPL-2.0. They are treated as permissive on that precedent. Say the word and they move to reference.

Every supported format that can hold mathematics is now represented by a
real published document. The set was assembled to answer a question the
hand-written fixtures could not: whether extraction works on files people
actually publish. It found nine defects, three of which returned no
formulas at all on real files while their fixtures passed.

136 documents are permissively licensed and are vendored. 81 of those are
text and land in this commit; the rest are binary formats, so their bytes
belong in the bucket and reach the corpus through publish_corpus.py.

12 carry ShareAlike, GPL or GFDL terms. The redistribution policy in
LICENSES.md keeps copyleft content out of the repository, so they are
recorded as reference only, with their source and licence and without
their bytes.

MATH_PROVENANCE.md lists every document with its source, licence and
notation. The licence of each was read from the source repository, the
document itself, or the publisher's stated terms.
@tobocop2
tobocop2 requested a review from Goldziher as a code owner August 13, 2026 22:19
@tobocop2
tobocop2 marked this pull request as draft August 14, 2026 02:49
The binary formats that carry mathematics are gitignored, like every
other corpus binary, so they cannot travel in a pull request. Without a
way to obtain them, publish_corpus.py has nothing in the working tree to
scan and the bucket never receives them.

scripts/fetch_math_binaries.py downloads all 55 from the sources recorded
in MATH_PROVENANCE.md and writes each to the repository path it belongs
at. scripts/math-binaries.json carries the source URL, the sha256 of the
file this corpus was validated against, and the size.

A download whose digest does not match is written to <path>.mismatch and
reported, so a source that changed upstream cannot quietly replace a good
file. Files already present and matching are skipped, so the script is
safe to re-run.
@tobocop2
tobocop2 marked this pull request as ready for review August 14, 2026 02:59
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Add 148 verified real documents that carry mathematics

1 participant