Skip to content

Publish an interactive yaml-test-suite conformance matrix#68

Draft
jskoiz wants to merge 6 commits into
mainfrom
claude/saneyaml-parser-strategy-kyza3a
Draft

Publish an interactive yaml-test-suite conformance matrix#68
jskoiz wants to merge 6 commits into
mainfrom
claude/saneyaml-parser-strategy-kyza3a

Conversation

@jskoiz

@jskoiz jskoiz commented Jun 12, 2026

Copy link
Copy Markdown
Owner

Summary

Adds a public, interactive conformance-matrix web page to the hosted guide: all 402 curated yaml-test-suite cases run against saneyaml, serde_yaml, yaml-rust2, and saphyr, with per-case accept/reject results and per-library scores.

  • examples/conformance_matrix_html.rs — generator reusing the acceptance methodology of conformance_compare (tree/Value load per library, serde_yaml via its document iterator, third-party calls guarded with catch_unwind). Renders a self-contained HTML page (no external assets, vanilla JS filtering by case id/name/feature, expected outcome, and disagreements), with library versions read from Cargo.lock and case links pinned to the upstream data-branch commit from SOURCE.toml.
  • docs/conformance/index.html — committed generated page, so the existing mdBook Pages workflow serves it at /saneyaml/conformance/index.html without building dev-dependencies. mdBook 0.5.3 copies it through verbatim (verified locally).
  • docs/conformance.md — reference page with the headline scores and regeneration commands, linked from SUMMARY.md, the docs index, the crate README, and added to the package include list.

Headline numbers (from the generator run)

library spec accept/reject (400 neutral cases) duplicate-key policy (2 cases)
saneyaml 400/400 (100%) 2/2
serde_yaml 0.9.34 333/400 (83.25%) 2/2
yaml-rust2 0.11.0 400/400 (100%) 2/2
saphyr 0.0.6 400/400 (100%) 0/2

The 2 tree-error cases are scored separately from the neutral spec axis, matching the framing in conformance_compare.

Also in this branch

  • contrib/oss-fuzz/ — ready-to-submit google/oss-fuzz project definition (project.yaml, Dockerfile, build.sh) building all 11 cargo-fuzz targets with debug assertions kept on and shipping each checked-in corpus as the seed corpus, plus submission/verification instructions.
  • examples/syq.rs — a minimal yq-style CLI over the lossless ConfigEditor API (get/set/push/rename/rm by JSON Pointer, stdin or file, optional in-place), now referenced from the editing guide as the runnable demo of comment-preserving edits.
  • CLAUDE.md — repository guidance for AI assistants.

Verification

  • cargo run --locked --example conformance_matrix_html regenerates the committed page byte-identically
  • cargo clippy --locked on both new examples with -D warnings, cargo fmt --all --check, cargo package --locked --allow-dirty --list, and cargo test --locked --test conformance_dashboard all pass
  • mdbook build (0.5.3, the CI-pinned version) succeeds and includes the matrix page in the book output
  • syq exercised end-to-end on a docker-compose-style file: inline comments and anchors survive set/push/rename/rm

Add examples/conformance_matrix_html.rs, which feeds all 402 curated
yaml-test-suite cases to saneyaml, serde_yaml, yaml-rust2, and saphyr
using the same acceptance methodology as conformance_compare, and
renders a self-contained, filterable HTML matrix with per-library
scores. The generated page is committed at docs/conformance/index.html
so the mdBook Pages deploy serves it without building dev-dependencies.

A new docs/conformance.md reference page records the headline scores
(saneyaml 400/400 spec cases plus 2/2 duplicate-key policy cases) and
the regeneration commands, and is linked from the guide summary, the
docs index, the crate README, and the published package file list.
@jskoiz jskoiz force-pushed the claude/saneyaml-parser-strategy-kyza3a branch from 5311114 to 64f9092 Compare June 12, 2026 06:40
jskoiz added 3 commits June 12, 2026 07:33
Stage a ready-to-submit google/oss-fuzz project definition under
contrib/oss-fuzz: project.yaml, a base-builder-rust Dockerfile, and a
build.sh that builds all cargo-fuzz targets with debug assertions kept
on (the targets assert parser invariants) and packages each target's
checked-in corpus as its seed corpus. The README documents local
verification with the oss-fuzz helper and the submission steps.
Add examples/syq.rs, a small command-line YAML editor over the lossless
ConfigEditor API: get/set/push/rename/rm addressed by RFC 6901 JSON
Pointers, reading from a file or stdin, with optional in-place rewrite.
Edits preserve comments, anchors, ordering, and untouched bytes, which
is the capability the editing guide now points at for a runnable demo.
@jskoiz

jskoiz commented Jun 12, 2026

Copy link
Copy Markdown
Owner Author

Review (2026-06-12, while in draft). CI is fully green (mdBook build included). Two concrete fixes worth making before un-drafting, both in examples/conformance_matrix_html.rs:

  1. Unescaped interpolation in the score cards (~line 297): the per-library card writes name = library.name and version = library.version into the HTML without escape_html(), unlike the case data which is escaped everywhere else. Versions come from Cargo.lock, so practical risk is low, but the generator should escape every interpolated string uniformly — a hostile/corrupted lockfile shouldn't be able to inject markup into the published page.
  2. Unguarded slice of the commit hash (~line 428): &source.data_branch_commit[..12] panics if SOURCE.toml ever carries a hash shorter than 12 chars (or with a multi-byte char in range). source.data_branch_commit.get(..12).unwrap_or(&source.data_branch_commit) or an explicit length check with a clear error would be sturdier.

Things I checked that look good:

  • Reproducibility of the committed docs/conformance/index.html: iteration is over BTreeMap/sorted case arrays, no timestamps or environment-dependent values — byte-stable regeneration claim holds up from the code.
  • Packaging: the generated index.html is correctly not in the Cargo.toml include list; only docs/conformance.md ships in the crate.
  • contrib/oss-fuzz/: referenced fuzz targets all exist under fuzz/fuzz_targets/; project.yaml/build.sh look ready to submit.
  • examples/syq.rs: pointer_tokens() implements RFC 6901 correctly (~0/~1 decoding, leading-/ validation), and the in-place write path is read → edit → finish → write with no partial-write hazard.

jskoiz added 2 commits June 12, 2026 19:53
Escape library names and Cargo.lock-derived versions like every other
interpolated string in the page, and replace the panicking [..12] slice
of the pinned suite commit with a length-guarded prefix. Generated
output is unchanged for the current inputs.
Document the road to 1.0 as verifiable gates in docs/ROADMAP.md (MSRV
and deprecation policy, what 1.0 will and will not promise) and link it
from the guide, docs index, README status section, and the published
package list.

Gate every PR with cargo-semver-checks against the published release
and keep the wasm32-unknown-unknown library build green via two new CI
jobs; the wasm check passes today without code changes.

Give the conformance matrix a prefers-color-scheme dark theme by moving
all page colors to CSS variables; the regenerated page is otherwise
unchanged.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant