Skip to content

feat: add --prune/-p to validate-kgx - #211

Merged
SkyeAv merged 3 commits into
mainfrom
add-prune
Oct 5, 2026
Merged

SkyeAv merged 3 commits into
mainfrom
add-prune

Conversation

@SkyeAv

@SkyeAv SkyeAv commented Oct 5, 2026

Copy link
Copy Markdown
Owner

Adds a --prune/-p flag to validate-kgx that rewrites the edges NDJSON in place, keeping only edges that validate strictly, so a finished build can be repaired into a graph that downstream KGX consumers can ingest cleanly instead of just scoring it.

Prune Core

  • biolink.prune_kgx_edges(edges_path, limit): one streaming pass classifies every edge against validate_kgx's own strict bar (validate_record dispatching on the record's own category), so a prune can never disagree with what a plain validation run reports. Real defects (non-coercible values, extras the model will never declare, rejected predicates), deliberate pending carryovers (synonym/xref/relation/provided_by, ...), and lines that are not JSON at all are all removed (malformed lines counted separately).
  • Atomic, lossless rewrite: kept lines stream into a sibling .{name}.tmp (opened with newline="" so kept lines stay byte-identical) and are swapped in with os.replace only after every line was written; a clean file is not rewritten at all (rewritten false, bytes untouched), a missing edges file is reported without ever being pruned, and an unwritable destination raises with the original file intact.
  • _validate_file extraction: validate_kgx's per-file loop is factored into a private helper with an unchanged report shape, so the agent consumer at agent.py:2035 and the docs examples see identical output.
  • Report shape: before (the original file's section), after (equal to a plain validate_kgx run over the rewritten file), and dropped (the removed records' problem histogram and examples), plus pruned/kept/malformed/rewritten counts and strict ok/ok_excluding_pending verdicts over the post-prune state.

CLI Flag

  • validate_kgx_command gains prune: without the flag the output is byte-identical to before; with it the command prints the nodes section, the edges section, edges: pruned N/M non-compliant records plus the dropped records' problems/examples, and the post-prune edges line, then gates the exit on the post-prune strict state (green exactly when the nodes are also clean).
  • Short alias: -p binds through the parser beside --prune (pinned by a parse_args test).

Design

  • Strict bar on purpose: --prune removes pending carryovers too. Pending is a scoring forgiveness, not compliance, and the requirement for the final graph is zero compliance failures; on a real graph the carryovers are most of the edges (the DAKP example loses roughly 800085 of 2000085), and the flag trades them for compliance. This is stated in the help text, docs/cli.md, and the CHANGELOG so it cannot surprise silently.
  • Nodes are never pruned: dropping one would strand its references; cascading that into edges is a modelling decision, not a validation repair.
  • Deferred: plain validate-kgx (no --prune) still crashes with a raw JSONDecodeError traceback on a malformed line; only the prune path handles them gracefully. A friendly report there is a follow-up.

Docs

  • docs/cli.md: new --prune/-p flag-table row and a "Repairing a graph with --prune" section with example output.
  • CHANGELOG.md: new ## Unreleased entry (release chore versions it, matching the feat: add a parquet source and reader #204 pattern).

Testing

All testing ran on the 80-core remote box wenceslaus (the laptop OOMs on the suite), using the repo venv against the pinned biolink-model 4.4.5:

  • PYTHONUTF8=1 PYTHONPATH=src .venv/bin/python -m pytest -> 1687 passed, 52 skipped (full suite incl. 13 new prune tests and the e2e proof)
  • .venv/bin/python -m ruff check . -> All checks passed!; ruff format --check . -> 101 files already formatted
  • pyright --pythonpath .venv/bin/python -> 0 errors, 0 warnings, 0 informations
  • mkdocs build --strict -> built in 1.61s
  • End-to-end proof on a real built graph (real rs.build_fullmap_db redb, real six-stage build_pipeline): injected 2 defects + 1 pending carryover + 1 malformed line into the emitted edges file, ran the command -> edges: pruned 4/5 non-compliant records, KGX output is Biolink-compliant., exit 0; plain validate_kgx over the pruned pair -> 1/1 valid (0 failures), ok true, and every surviving line byte-identical to the original build.

Caveats:

  • The test host has LANG=en_US, so PYTHONUTF8=1 is required or five pre-existing fixture-decode tests fail on UTF-8 em dashes; those failures are environmental and unrelated to this change.
  • cargo is absent on the test host, so the Rust CI jobs could not run there; no file under rust/ changed (git diff --name-only origin/main...HEAD), so those jobs are unaffected.

@coderabbitai

coderabbitai Bot commented Oct 5, 2026

Copy link
Copy Markdown

Important

  • 🔍 Trigger review

This repository does not receive automatic reviews because it has fewer than 10 stars.

⚙️ Run configuration
  • Configuration used: defaults
  • Review profile: CHILL
  • Plan: Advanced
  • Run ID: 82f12371-0ab7-4d9e-b637-41226e3517c1
  • Autopilot · Keep fixing CodeRabbit findings and required CI, and resolving merge conflicts

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

Comment @coderabbitai help to get the list of available commands.

@SkyeAv
SkyeAv merged commit 5194e7d into main Oct 5, 2026
7 of 9 checks passed
SkyeAv added a commit that referenced this pull request Oct 5, 2026
Cut 19.6.0 and bump the package version in pyproject.toml,
CITATION.cff, and the uv.lock self-package entry.

Minor: validate-kgx can now repair a graph in place with the new
--prune (-p) flag, which rewrites the edges NDJSON keeping only
strictly-valid edges via the new public biolink.prune_kgx_edges;
the rewrite is atomic, byte-identical for kept lines, a no-op on a
clean file, and never touches nodes (#211).

Changelog:
- Versioned the Unreleased section as 19.6.0 and added the #211 PR
  link to the --prune entry.

Docs: none needed here; #211 already updated docs/cli.md.

Testing:
- uv run pytest -q -> 1687 passed, 52 skipped
- cargo test --manifest-path rust/Cargo.toml -> 156 passed, 0 failed
- cargo clippy --all-targets -- -D warnings -> clean
- uv run ruff check . / ruff format --check . / pyright / uv lock
  --check / cargo fmt --check -> all pass
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant