Skip to content

Add transactional archive directory installs - #86

Merged
iskandr merged 3 commits into
masterfrom
feature/archive-directory-install
Oct 1, 2026
Merged

iskandr merged 3 commits into
masterfrom
feature/archive-directory-install

Conversation

@iskandr

@iskandr iskandr commented Sep 30, 2026 •

Copy link
Copy Markdown
Contributor

MHCflurry's tar downloads need the entire extracted tree and source receipt to become available together. An interrupted extraction into the final directory can leave partial data that looks installed. This adds a shared archive installer that validates in private staging, then publishes a complete immutable generation and an atomic current pointer.

  • install_archive / inspect_archive support full tar trees, ordered split parts, local predownloaded parts, and trusted aggregate or per-part checksums. Historical sources explicitly opt into observed-receipt verification.
  • VersionedArchiveRegistry supplies concrete versions and defaults, progress/retry forwarding, offline paths, inspection, and source/hash/time status. A layout callback and transactional extra_files support MHCflurry's release layout and DOWNLOAD_INFO.csv.
  • Tar paths normalize standard leading ./ components, then reject traversal, links, special files, reserved names, and duplicate/case-folded collisions. Header scanning stops immediately when member or expansion limits are exceeded; FIFO sources fail without blocking.
  • Large tree manifests support the same sizes when written, inspected, and recovered. The written receipt and complete tree are validated before generation publication. Reuse compares the complete consumer metadata definition, including removed entries.
  • Cross-platform writer locks serialize installation. Failed refreshes preserve old generations; interrupted first pointer publication can recover offline. Existing foreign/legacy directories are never silently claimed.
  • VersionedFileRegistry.status() also exposes recorded URLs and observed SHA-256, and the integration guide documents reusable fixed-file, archive-tree, and multi-file APIs for downstream libraries.

Validation: ./test.sh passes all 807 tests with 93% overall coverage; lint and diff checks pass. Regression tests cover standard tar paths and collisions, a 12,000-file manifest with offline recovery, early resource-limit rejection, FIFO input, metadata removal/changes, and invalid receipt rejection before publication. All CI test jobs pass on Linux Python 3.9–3.14, macOS, and focused Windows archive tests. The non-required Coveralls comparison reports lower overall coverage than the base branch after adding the archive module.

Closes #85

@coveralls

Copy link
Copy Markdown

Coverage Report for CI Build 36768096745

Coverage decreased (-2.4%) to 93.114%

Details

  • Coverage decreased (-2.4%) from the base build.
  • Patch coverage: 91 uncovered changes across 1 file (543 of 634 lines covered, 85.65%).
  • No coverage regressions found.

Uncovered Changes

File Changed Covered %
datacache/archives.py 632 541 85.6%
Total (3 files) 634 543 85.65%

Coverage Regressions

No coverage regressions found.


Coverage Stats

Coverage Status
Relevant Lines: 2585
Covered Lines: 2407
Line Coverage: 93.11%
Coverage Strength: 5.58 hits per line

💛 - Coveralls

@coveralls

coveralls commented Sep 30, 2026 •

Copy link
Copy Markdown

Coverage Report for CI Build 36808213563

Coverage decreased (-2.4%) to 93.174%

Details

  • Coverage decreased (-2.4%) from the base build.
  • Patch coverage: 90 uncovered changes across 1 file (552 of 642 lines covered, 85.98%).
  • 1 coverage regression across 1 file.

Uncovered Changes

File Changed Covered %
datacache/archives.py 638 548 85.89%
Total (4 files) 642 552 85.98%

Coverage Regressions

1 previously-covered line in 1 file lost coverage.

File Lines Losing Coverage Coverage
datacache/_filesystem.py 1 95.24%

Coverage Stats

Coverage Status
Relevant Lines: 2593
Covered Lines: 2416
Line Coverage: 93.17%
Coverage Strength: 7.44 hits per line

💛 - Coveralls

@iskandr
iskandr merged commit 153c19a into master Oct 1, 2026
20 of 22 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Support safe whole-archive directory installation for existing consumer layouts

2 participants