feat: add real documents that carry mathematics - #5
Open
tobocop2 wants to merge 2 commits into
Open
Conversation
Every supported format that can hold mathematics is now represented by a real published document. The set was assembled to answer a question the hand-written fixtures could not: whether extraction works on files people actually publish. It found nine defects, three of which returned no formulas at all on real files while their fixtures passed. 136 documents are permissively licensed and are vendored. 81 of those are text and land in this commit; the rest are binary formats, so their bytes belong in the bucket and reach the corpus through publish_corpus.py. 12 carry ShareAlike, GPL or GFDL terms. The redistribution policy in LICENSES.md keeps copyleft content out of the repository, so they are recorded as reference only, with their source and licence and without their bytes. MATH_PROVENANCE.md lists every document with its source, licence and notation. The licence of each was read from the source repository, the document itself, or the publisher's stated terms.
tobocop2
marked this pull request as draft
August 14, 2026 02:49
tobocop2
force-pushed
the
feat/math-corpus
branch
from
August 14, 2026 02:50
71a7a23 to
c1ad081
Compare
The binary formats that carry mathematics are gitignored, like every other corpus binary, so they cannot travel in a pull request. Without a way to obtain them, publish_corpus.py has nothing in the working tree to scan and the bucket never receives them. scripts/fetch_math_binaries.py downloads all 55 from the sources recorded in MATH_PROVENANCE.md and writes each to the repository path it belongs at. scripts/math-binaries.json carries the source URL, the sha256 of the file this corpus was validated against, and the size. A download whose digest does not match is written to <path>.mismatch and reported, so a source that changed upstream cannot quietly replace a good file. Files already present and matching are skipped, so the script is safe to re-run.
tobocop2
force-pushed
the
feat/math-corpus
branch
from
August 14, 2026 02:58
c1ad081 to
225a378
Compare
tobocop2
marked this pull request as ready for review
August 14, 2026 02:59
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Closes #4.
What this adds
Every supported format that can hold mathematics is now represented by a real published document. The set answers a question the hand-written fixtures could not: whether extraction works on files people actually publish.
It found nine defects. Three returned no formulas at all on real files while their fixtures passed.
ppt/presentation.xmlsits at byte 107,190%3Ain its file names<p>was never captured≤.xmla14:mTwo are bugs in xberg that cost text and images as well as formulas, filed as xberg-io/xberg#1426 and #1427.
Licensing
136 documents are permissively licensed and are vendored. 12 carry ShareAlike, GPL or GFDL terms, and the redistribution policy in
LICENSES.mdkeeps copyleft content out of the repository, so those are recorded as reference only: source and licence, without the bytes. That follows the treatmenthiertextalready has.Four documents state a dual licence,
CC-BY OR GPL-3.0-or-later, and are vendored under the CC-BY arm.MATH_PROVENANCE.mdlists every document with its source, licence and notation. Each licence was read from the source repository, the document itself, or the publisher's stated terms.What is in this commit, and what is not
81 text documents are committed here. The other 55 vendored documents are binary formats, which
.gitignorekeeps out of git by design, so their bytes reach the corpus throughscripts/publish_corpus.pyandcorpus.lock.json. They are ready to publish and are not part of this diff.corpus.lock.jsonis untouched: the publisher writes it, so hand-editing it would point the manifest at objects the bucket does not hold yet.Getting the binaries in
A gitignored file cannot travel in a pull request, so
scripts/fetch_math_binaries.pydownloads all 55 from the sources inMATH_PROVENANCE.mdand writes each to the path it belongs at:scripts/math-binaries.jsoncarries each source URL with the sha256 and size of the file this corpus was validated against. A download whose digest does not match is written to<path>.mismatchand reported, so a source that changed upstream cannot quietly replace a good file. Files already present and matching are skipped, so the script is safe to re-run.No ground truth
These documents ship without
ground_truth/files, andMATH_PROVENANCE.mdsays why. They bring no upstream ground truth, and deriving it by running the extractor would record today's output as the definition of correct, so a later regression would match the golden file and pass.Four of them do carry 1,046 LaTeX strings written by their own authors, in MathML
annotation-tex. That is real formula-level ground truth, independent of any extractor.README.mdplaces targets of that kind inground_truth/structured/, which the repository does not yet contain, so this change does not invent the format.Needs a decision
Five LibreOffice documents are vendored, and their upstream metadata disagrees with itself. GitHub's licence API reports GPL-3.0 for
LibreOffice/core, while the project states MPL-2.0 and LGPL-3.0-or-later, and this repository already vendorsLibreOffice/coreas MPL-2.0. They are treated as permissive on that precedent. Say the word and they move to reference.