Add transactional archive directory installs

.github/workflows/tests.yml

@@ -57,3 +57,19 @@ jobs:
         # error. Don't let that flake fail the whole test run.
         continue-on-error: true
         uses: coverallsapp/github-action@v2.2.3
+
+  archive-windows:
+    runs-on: windows-latest
+    steps:
+      - name: Checkout repository
+        uses: actions/checkout@v4
+      - name: Set up Python 3.12
+        uses: actions/setup-python@v5
+        with:
+          python-version: "3.12"
+      - name: Install package and test dependencies
+        run: |
+          python -m pip install --upgrade pip
+          python -m pip install -e ".[test]"
+      - name: Test cross-platform archive publication
+        run: python -m pytest -q tests/test_archives.py

CHANGELOG.md

@@ -1,5 +1,32 @@
 # Changelog
 
+## 1.16.0
+
+- Add `install_archive` and `inspect_archive` for safely installing a complete
+  tar archive tree as one immutable generation. Ordered split archives are
+  concatenated byte-for-byte; URL and local-path sources share the same API.
+  Extraction rejects traversal, links, special files, path collisions and
+  reserved metadata names, and ignores archive ownership and permission bits.
+  Standard `.` / `./` tar paths are normalized safely; resource limits stop
+  header scanning at the first violation, and local FIFOs cannot block a writer.
+- Add `VersionedArchiveRegistry`, a catalogue facade with pinned defaults,
+  explicit versions, reusable status rows, fast local path resolution, consumer
+  layout callbacks, local predownloaded-part overrides, and the same progress,
+  retry, verification and recovery behavior as `install_archive`.
+- Include the recorded URL and observed SHA-256 in `VersionedFileRegistry`
+  status rows so fixed-file consumers can report the same download provenance.
+- Publish consumer metadata such as MHCflurry's `DOWNLOAD_INFO.csv` inside the
+  same transaction with `extra_files`. Archive receipts record the ordered
+  source identities, assembled and per-part observed hashes, and every extracted
+  file. Trusted expectations remain distinct from observed consistency checks.
+  Reuse checks the complete consumer metadata inventory, including removals.
+  Large manifests are read without the small control-record size cap and
+  validated in staging before the generation is published.
+- Serialize archive writers with a cross-platform file lock, retain immutable
+  generations across refreshes, and recover a complete generation after an
+  interrupted pointer publication. Legacy or otherwise foreign directories are
+  never claimed, including with `force=True` (#85).
+
 ## 1.15.0
 
 - Add `VersionedFileRegistry` for established single-file caches with fixed

README.md

@@ -5,8 +5,9 @@
 
 Download, verify, transform, and cache datasets for Python applications,
 including OpenVax libraries such as pyensembl. DataCache provides streaming
-downloads, gzip/ZIP decompression, reusable local paths, offline inspection,
-and SQLite caches built from pandas DataFrames.
+downloads, gzip/ZIP decompression, transactional tar-tree installation,
+reusable local paths, offline inspection, and SQLite caches built from pandas
+DataFrames.
 
 ## Install
 
@@ -162,8 +163,10 @@ lists every public signature, default, return value, exception, and example.
 | Task | API | Result |
 | --- | --- | --- |
 | Install or reuse a versioned dataset | `VersionedDatasetRegistry`, `install_bundle(...)` | Mapping of asset names to snapshot paths |
+| Install versioned archive trees | `VersionedArchiveRegistry`, `install_archive(...)` | Immutable extracted-generation `Path` |
 | Reuse an established fixed-path versioned file cache | `VersionedFileRegistry` | One Path and a legacy-compatible root receipt |
 | Inspect a complete dataset generation | `inspect_bundle(...)` | `BundleInspection` |
+| Inspect an installed archive tree | `inspect_archive(...)` | `ArchiveInspection` |
 | Discard retained partial download bytes | `discard_partial(destination)` | Installed file unchanged |
 | Download or reuse one file | `fetch_file(...)`, `Cache.fetch(...)` | Local path string |
 | Compute a path without filesystem access | `expected_path(...)`, `Cache.local_path(...)` | Path string |
@@ -186,6 +189,8 @@ FASTA files with `fetch_file`, then parse them in the consuming library.
 
 - [Versioned datasets and bundles](docs/bundles.md): pinned versions, atomic installation,
   offline recovery, path lifetime, and downstream adoption.
+- [Archive directory installation](docs/archives.md): safe tar extraction, split archives,
+  transactional consumer receipts, and legacy-directory migration.
 - [API reference](https://github.com/openvax/datacache/blob/master/docs/api.md):
   all public functions, `Cache` methods, inspection results, and exceptions.
 - [Downloads and cache inspection](https://github.com/openvax/datacache/blob/master/docs/downloads.md): destinations, naming,
@@ -202,7 +207,8 @@ FASTA files with `fetch_file`, then parse them in the consuming library.
 
 ## Guarantees and limits
 
-Downloads are staged privately and published atomically after validation.
+Downloads and complete archive trees are staged privately and published
+atomically after validation.
 New files respect the process umask; replacements preserve existing access
 permissions. This includes pyensembl's private download helpers.
 

datacache/__init__.py

@@ -33,6 +33,8 @@
 from .cache import Cache
 from .resume import discard_partial
 from .bundles import BundleInspection, VersionedDatasetRegistry, inspect_bundle, install_bundle
+from .archives import (
+    ArchiveInspection, VersionedArchiveRegistry, inspect_archive, install_archive)
 from .file_registry import VersionedFileRegistry
 from .version import __version__
 
@@ -41,10 +43,14 @@
     'fetch_file',
     'discard_partial',
     'BundleInspection',
+    'ArchiveInspection',
+    'VersionedArchiveRegistry',
     'VersionedDatasetRegistry',
     'VersionedFileRegistry',
     'inspect_bundle',
     'install_bundle',
+    'inspect_archive',
+    'install_archive',
     'expected_path',
     'file_exists',
     'resolve_path',

datacache/_filesystem.py

@@ -58,9 +58,10 @@ def file_lock(path, *, private=False):
 
 
 def read_json(path, limit=1024 * 1024):
+    """Read a regular JSON file; ``limit=None`` permits growing tree manifests."""
     with os.fdopen(open_regular(path), 'rb') as handle:
-        data = handle.read(limit + 1)
-    if len(data) > limit:
+        data = handle.read() if limit is None else handle.read(limit + 1)
+    if limit is not None and len(data) > limit:
         raise ValueError('JSON record is too large')
     return json.loads(data)