Performance Improvements for Large Bundle Validation

[jordanpadams]

💡 Description

Addresses a cluster of performance and memory bottlenecks that cause validate to fail or run unacceptably slowly on large PDS4 bundles (23,000+ products) and table products with millions of records. Each sub-issue targets a specific hot spot identified through profiling and user-reported failures.

Sub-Issues

NASA-PDS/validate

• validate does not complete validation of large bundles (23000+ products) due to OutOfMemoryError in InMemoryRegistrar #1571 — OOM failure on large bundles: unbounded HashMap growth and O(n) getChildTargets() scan in InMemoryRegistrar. Fixed by adding a parent→children index and AtomicInteger count caches. PR Fix OutOfMemoryError when validating large bundles (23,000+ products) #1594 in review.
• As a PDS data engineer, I want schematron transformers to be cached rather than recompiled for every label so that bundle validation is significantly faster #1565 ✅ — Schematron Transformer recompiled from scratch for every label. Fixed by caching the compiled Templates object; estimated 30–50% reduction in validation time for large bundles.
• As a PDS developer, I want inline regex patterns in FieldValueValidator pre-compiled as static constants so that per-field Pattern compilation overhead is eliminated #1570 — Inline regex patterns in FieldValueValidator.checkFormat() recompiled via String.matches() on every field of every record. Fix: promote to static final Pattern constants.

NASA-PDS/pds4-jparser

• As a PDS data engineer, I want RawTableReader to use buffered bulk I/O so that large table products with millions of records can be validated without extreme performance degradation pds4-jparser#197 — RawTableReader.readNextLine() reads one byte at a time via ByteWiseFileAccessor, resulting in hundreds of millions of individual ByteBuffer.get() calls for large tables. Fix: 8 KB internal read buffer with bulk readBytes().
• RawTableReader readNextLine() treats 0xFF data bytes as EOF due to byte-to-int sign-extension pds4-jparser#200 — Related correctness bug discovered during Change behaviour for printing of "dots" to show progress #197 work: 0xFF data bytes sign-extend to int −1 and are silently treated as EOF, truncating table reads.

Acceptance Criteria

• Large bundles (23,000+ products) complete validation without OutOfMemoryError
• Schematron XSLT compiled once per unique schematron, not once per label
• No Pattern object allocated more than once per unique pattern string during field validation
• Table products with millions of records validate using buffered I/O, not byte-at-a-time reads
• 0xFF data bytes pass through RawTableReader correctly and are not treated as EOF