feat(diagrams): diagram fixture corpus, ground truth and repo-wide README - #2
Merged
Conversation
Adds the corpus side of diagram node/edge recovery (xberg-io/xberg#579). Fixtures and DOT ground truth are from #1 by @MannXo; this reworks them and closes three gaps. Filed under diagrams/<format>/ so the set stays inspectable as a set, with diagrams/manifest.json as its index: per fixture, what produced it, why it is in the set, and what the correct answer is. The manifest also carries the page and bbox slots the issue asks for and lets one document hold several graphs, neither of which a bare .dot file can express. Three fixes to the ground truth: graphviz_network draws an undirected graph, and its ground truth restated it as a digraph with directed edges. Undirectedness is the one property that fixture exists to test, so it is now a graph with -- edges, and the manifest records `directed` explicitly. dot -Tsvg writes the graph it just laid out back into its own output: one <title> per node holding the node id, one per edge holding src->dst. That is the complete edge list, verbatim, in the file, and xberg's SVG extractor already reads <title>. A recogniser scores 4/4 nodes and 4/4 edges on every Graphviz fixture without touching a coordinate, so as they stood these fixtures could not measure geometry recovery at all. Each now ships twice, as emitted and with the element titles stripped, and the manifest says which is which. Reading the titles is correct behaviour when they are there -- it is just a different capability, and the two have to be measured apart. An empty .dot meant both "not a diagram" and "not annotated yet". The manifest now carries `negative` and a reason; the empty files stay as the on-disk expectation. Adds a third negative that was already shipping: images/5_level_paging_..._2017.svg is, despite its name, an inferno flame graph -- 77 nested rects, a CSS block and a script, no connectors anywhere.
The README described this as the "xberg PDF→Markdown benchmark corpus" and documented only that slice. The repo holds about 1,400 documents across 105 extensions and 1,465 ground-truth files, most of which the README never mentioned, so the answer to "how do I add a fixture" was not written down anywhere. Documents repo-wide: the GCS bucket mechanism and why it replaced Git LFS, which extensions are git-tracked versus bucket-managed and where that list is enforced, the fetch/add/publish workflow including that publishing needs bucket write access and must happen before the refreshed lock file is pushed, the directory map, the ground-truth conventions, the vendor versus reference licence classes, and what CI actually checks. The bucket extension list is generated from scripts/corpus-patterns.txt rather than retyped, so it cannot drift.
This was referenced Aug 9, 2026
…iz titles A geometry-only fixture is meant to force recovery from shapes and strokes, but `strip_svg_titles.py` only removed `<title>` from node and edge groups, so the fixtures it produced still stated their own answer three ways over: graphviz repeats the whole edge list in XML comments, mermaid encodes both endpoints in `id="L_start_auth_0"` and again in `data-id`, and plantuml in `id="Read config-to-Open input"` plus a comment. Cluster titles survived too, which told a recogniser outright which rectangles are containers rather than nodes -- the judgement the cluster fixtures exist to test. Rather than a list of the dialects seen so far, two producer-agnostic rules: drop element titles including clusters', and drop every id nothing in the document refers to, plus all comments. An id no `url(#...)`, `href="#..."` or stylesheet selector points at cannot affect rendering, so removing it changes no pixels -- each stripped fixture still rasterises byte-identically. What stays is whatever gives an element's type without naming its endpoints, because knowing a stroke is some connector leaves the whole task intact. Renamed, since "titles" no longer describes what it does.
…positives The set was four Graphviz drawings, so it measured one tool's idioms and little else. This adds the three axes that were missing. Producers, because a recogniser tuned on `dot -Tsvg` does not survive real files: Mermaid (a 4.4 KB CSS block, HTML labels in `<foreignObject>`, edge labels on an opaque background box that looks exactly like a small node), PlantUML, and LibreOffice Draw (glued connectors that stop short of the outline, so endpoints match only by proximity). Features that break naive recovery hardest: cluster and swimlane containers that are not nodes, orthogonal elbow routing, self-loops, two edges crossing in mid-drawing that share no endpoint, record labels with ports, stroke-only nodes with CJK/Hebrew/Arabic labels, icon nodes with no outline at all, and 128 nodes for the profiling the review asked for. False positives, which nothing covered: a table drawn with ruling lines -- the same closed regions joined by straight strokes, and whatever rejects it must leave table detection working -- a pie chart whose leader lines look exactly like elbow connectors into labelled nodes, and a printed form. Plus `mixed_page.svg`, a page holding prose, a table and one figure, so the answer cannot be decided per file. `libreoffice_connectors.fodg` is also the corpus's first Class A fixture: it names its endpoints with `draw:start-shape`/`draw:end-shape`, and it renders to the Class B SVG beside it, so one ground truth measures both the stated graph and the inferred one. Two committed checks keep the answer key honest. `test_diagram_manifest.py` runs in CI without a renderer and asserts, among other things, that every geometry variant is byte-for-byte what stripping its parent produces -- the only check that does not depend on knowing how a producer encodes its answer. `check_diagram_ground_truth.py` re-derives each graph with `dot -Tplain` and diffs it against the hand-written ground truth; it found that `dir=back` is recorded as drawn rather than as declared, which is correct, and confirmed all ten Graphviz fixtures agree.
… labels Running the new fixtures through the scorer on xberg#1410 turned up two faults in the ground truth rather than in the code. `graphviz_record` scored 0/3 nodes, and the recovery was right: it found one node per record and joined the fields with a newline, exactly as `org_chart.dot` already keys a two-line box. The `" | "` separator invented here was the outlier, so it is gone. `graphviz_large` reported one more edge than it draws, because a prose comment mentioning `N_i -> N_i+16` was read as an edge -- consumers parse these files line by line and do not all strip `//`. Reworded, and a test now keeps DOT syntax out of every ground-truth comment, since relying on each reader to strip them is the more fragile half of the bargain.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Related
Corpus side of xberg-io/xberg#579 / xberg-io/xberg#1410. Supersedes #1 — the fixtures and DOT
ground truth there are @MannXo's work, reworked here rather than reviewed twice. Credited in
ATTRIBUTIONS.md.Why
diagram_dot_ground_truth.rson #1410 readsdiagrams/*.svgandground_truth/dot/*.dot. Neitherexisted in this submodule, so every case hit the skip branch and both tests passed without scoring
anything. The corpus side and the code side each assumed the other had landed.
With this branch, that test scores 7 fixtures at 100% node and edge recall instead of skipping all
of them.
What is here
diagrams/<format>/withdiagrams/manifest.jsonas the index — per fixture: what produced it, whyit is in the set, the expected node and edge counts, whether it is a negative, and the page/bbox
slots #579 asks for. A bare
.dotcannot express any of that, nor a document holding severalgraphs.
Three fixes to the ground truth as it stood:
graphviz_networkwas undirected ground truth written as a digraph. The source isgraph { gw -- s1; ... }. Undirectedness is the one property that fixture exists to test, so it isnow
graphwith--, and the manifest recordsdirectedexplicitly. This immediately exposedthat undirected graphs are unrepresentable end to end — see the finding below.
The Graphviz fixtures stated their own answer.
dot -Tsvgwrites one<title>per node holdingthe node id and one per edge holding
src->dst, so the complete edge list is in the file as text —and xberg's SVG extractor already reads
<title>. A recogniser could score 4/4 and 4/4 on all fourfixtures without inspecting a coordinate. Each now ships twice, as emitted and with element titles
stripped (
scripts/strip_svg_titles.py), so title-reading and geometry inference are measuredapart. Reading titles when they are there is correct behaviour; it is just a different capability.
An empty
.dotmeant both "not a diagram" and "not annotated yet". The manifest now carriesnegativeplus a reason. Adds a third negative that was already shipping:images/5_level_paging_..._2017.svgis, despite its name, an inferno flame graph — 77 nestedrects, a CSS block, a script, and not one connector.
README
Rewritten repo-wide. It described this as the "PDF→Markdown benchmark corpus" and documented only
that slice, so "how do I add a fixture" was written down nowhere. Now covers the GCS bucket and why
it replaced Git LFS, git-tracked vs bucket-managed and where that is enforced, the
fetch/add/publish workflow, the directory map, ground-truth conventions, licence classes, and what
CI checks. The bucket extension list is generated from
scripts/corpus-patterns.txtso it cannotdrift.
Verification
No binaries added, so
corpus.lock.jsonis untouched and no publish step is needed.Against #1410 with the submodule pointed here:
and the geometry-only variants score identically — 4/4, 4/4, 3/3 — which proves the recogniser
infers from geometry rather than reading the metadata the file leaks.
Two things #1410 needs to change
diagrams/svg/, so theCASEStable needs updating.graphviz_networknow reports 0/0 edges: the test's DOT parser only understands->, so a--ground truth parses as zero edges and scores a vacuous 100%. Undirected graphs are currentlyunmeasurable. The manifest carries
directed: falsefor the scorer to key on.Checklist