Add transactional derived-artifact materialization

[iskandr]

Closes #90.

Third PR in the proposed 1.17.0 stack: depends on #98 (which depends on #97). Merge those in order, retarget this PR to master, and rerun checks before landing. Nothing is merged or published by this PR.

Changes

• Add materialize and inspect_materialization for caller-built, versioned multi-output artifacts. The caller supplies raw URL/local sources, an opaque transform version/options identity, required outputs and a scientific builder.
• Reuse existing transport, POSIX writer locks, safe regular-file checks, progress and generation-store initialization. Keep a distinct ownership marker; never adopt bundles or populated legacy directories, even with force.
• Keep private completed inputs and resumable HTTP partials across failures. Copy local inputs without changing/deleting originals. Retry a failed/killed build without downloading a completed matching source again.
• Validate the exact output inventory and trusted expectations before publishing the entire output tree plus dependency receipt through an immutable generation and atomic pointer.
• Record source identities, redacted origins, observed raw/output hashes and sizes, acquisition-time trust, transform identity and accepted resumable transport validators. Do not equate observed consistency or ETags with trusted verification.
• Require explicit refresh on changed source/transform/output identity. Keep previous output and returned snapshot paths usable through failed or successful refreshes.
• Remove only owned inputs for the completed transaction after successful publication/recovery by default; offer explicit source retention. Cache hits remain offline, quiet and read-only.
• Enable progress by default for this new API. Keep existing fetch_and_transform behavior compatible.
• Document builder contracts, retained/peak disk use, recovery and cleanup limits. Add a tiny offline reference-DNA example to CI. No biological parser, PyEnsembl or MHCflurry code changes.

Validation

• Full local suite: 934 tests pass, 94% overall coverage, 97% materialization-module coverage.
• 89 materialization regressions cover raw gzip expectations, two-output atomicity, real loopback HTTP resume (trusted hash and strong-ETag modes), interrupted/killed builders, completed-input retry, identity changes, failed final output, both publication boundaries, private-state corruption, links/FIFOs, concurrent same/unrelated artifacts, read-only offline hits, large receipts, input retention, default progress, local copy callbacks and callback interruption.
• Ruff and diff checks pass; offline examples pass.
• Source distribution/wheel build and Twine checks pass; isolated Python 3.14 wheel smoke verifies gzip-derived pair publication, trusted receipts, owned-input cleanup and offline fast reuse.

Deliberate limits

Installation initially retains the existing resumable-bundle POSIX/local-filesystem constraint; #89 handles cross-platform bundle work separately. Builders are trusted application code and own scientific validation/decompression progress. Signed URL changes are dependency changes until caller-declared identity policy is added separately (#92). Old generations and abandoned source/staging variants are not implicitly pruned (#91).

[coveralls]

Coverage Report for CI Build 37022732426

Coverage increased (+0.4%) to 93.877%

Details

• Coverage increased (+0.4%) from the base build.
• Patch coverage: 14 uncovered changes across 2 files (394 of 408 lines covered, 96.57%).
• No coverage regressions found.

Uncovered Changes

File	Changed	Covered	%
datacache/materialization.py	395	382	96.71%
datacache/bundles.py	6	5	83.33%
Total (5 files)	408	394	96.57%

Coverage Regressions

No coverage regressions found.

---

Coverage Stats

Relevant Lines:	3005
Covered Lines:	2821
Line Coverage:	93.88%
Coverage Strength:	6.56 hits per line

---

💛 - Coveralls