Support ETag-validated resume and raw downloads (1.14.0)

CHANGELOG.md

@@ -1,5 +1,17 @@
 # Changelog
 
+## 1.14.0
+
+- Allow `resume=True` with `expected_size` alone when the server supplies a
+  strong ETag. Resume uses `If-Range` and validates the returned range and
+  validator; servers without a strong ETag still require `expected_sha256`.
+  Complete size-only partials restart instead of being published without a
+  fresh response. Size-only validation does not verify content checksums (#80).
+- Add `raw=True` to `fetch_file` and `Cache.fetch` to disable archive
+  decompression and HTML conversion at arbitrary output names. It composes
+  with integrity checks, atomic replacement, progress, provenance, and resume;
+  existing suffix inference is unchanged when raw mode is omitted (#81).
+
 ## 1.13.0
 
 - Add integrity-pinned resumable raw HTTP downloads with `resume=True`, private

README.md

@@ -147,8 +147,11 @@ read-only offline inspection, and keeps returned paths usable during refreshes.
 The [bundle guide](docs/bundles.md) covers shared OpenVax roots, generated indices,
 and migration from downstream registries.
 
-For large raw HTTP files, pass `resume=True` with `expected_sha256` and
-`expected_size` to `fetch_file` or `Cache.fetch`. Interrupted transfers retain
+For large raw HTTP files, pass `resume=True` and `expected_size` to `fetch_file`
+or `Cache.fetch`. Supply `expected_sha256` when available; otherwise the server
+must provide a strong ETag for safe range continuation. Size-only validation
+does not verify a content checksum. Pass `raw=True` to preserve gzip, ZIP, or
+HTML payloads at arbitrary output names without conversion. Interrupted transfers retain
 private partials and continue on retry. See [resumable downloads](docs/downloads.md#resumable-http-downloads).
 
 ## Choose the right API

datacache/cache.py

@@ -125,7 +125,8 @@ def fetch(
             show_progress=False,
             record_provenance=False,
             allow_empty=False,
-            resume=False):
+            resume=False,
+            raw=False):
         """
         Return the local path to the downloaded copy of a given URL.
         Don't download the file again if it's already present,
@@ -138,6 +139,9 @@ def fetch(
         record_provenance=True records where the file came from, for
         inspect_file, as described for fetch_file.
         allow_empty=True accepts an empty file, which is otherwise rejected.
+        raw=True disables decompression and HTML conversion even for an explicit
+        filename. It cannot be combined with decompress=True. Resume options
+        have the same size, hash, and strong-ETag requirements as fetch_file.
 
         `use_wget_if_available` is deprecated and ignored (datacache always uses
         its streaming Python downloader now); passing it emits a warning.
@@ -162,7 +166,8 @@ def fetch(
             show_progress=show_progress,
             record_provenance=record_provenance,
             allow_empty=allow_empty,
-            resume=resume)
+            resume=resume,
+            raw=raw)
 
         self._local_paths[key] = path
         return path

datacache/download.py

@@ -533,7 +533,8 @@ def fetch_file(
         show_progress=False,
         record_provenance=False,
         allow_empty=False,
-        resume=False):
+        resume=False,
+        raw=False):
     """
     Download a remote file and store it locally in a cache directory. Don't
     download it again if it's already present (unless `force` is True.)
@@ -553,6 +554,12 @@ def fetch_file(
         under a distinct cache key. An explicit filename or destination lacking
         the source's .zip/.gz suffix still implies decompression for compatibility.
 
+    raw : bool, optional
+        Disable archive decompression and HTML-to-CSV conversion regardless of
+        the output name. Incompatible with decompress=True. Defaults to False,
+        preserving legacy output-name inference. Integrity expectations then
+        describe the unchanged payload, after HTTP transfer decoding.
+
     subdir : str, optional
         Application name selecting a platform cache directory, "datacache" by
         default. It is not nested inside the default cache. Ignored when
@@ -619,8 +626,12 @@ def fetch_file(
 
     resume : bool, optional
         Keep private partials and resume raw HTTP(S) transfers, default False.
-        Requires both expected_sha256 and expected_size, and a POSIX local
-        filesystem. Does not support decompression or HTML conversion. Retries
+        Requires expected_size and a POSIX local filesystem. If expected_sha256
+        is omitted, the server must supply a strong ETag on every accepted
+        response; it is sent as If-Range when resuming. ETags prevent mixing
+        representations, but do not verify a trusted checksum. Cache hits with
+        size alone check only their byte count. Does not support decompression
+        or HTML conversion; raw=True permits arbitrary output names. Retries
         report cumulative bytes, resetting only when the server requires a
         fresh transfer. discard_partial(destination) explicitly drops partials.
         Interrupted downloads retain at most expected_size partial bytes.
@@ -654,6 +665,10 @@ def fetch_file(
     """
     if not isinstance(resume, bool):
         raise ValueError("resume must be a boolean")
+    if not isinstance(raw, bool):
+        raise ValueError("raw must be a boolean")
+    if raw and decompress:
+        raise ValueError("raw=True cannot be combined with decompress=True")
     if resume:
         from .resume import validate_resume
         validate_resume(download_url, expected_sha256, expected_size)
@@ -683,11 +698,14 @@ def fetch_file(
         destination=destination, cache_root=cache_root)
     source_suffix = _source_suffix(download_url)
     output_suffix = os.path.splitext(full_path)[1].lower()
-    archive_decompression = bool(decompress or (explicit_output and output_suffix != source_suffix))
+    archive_decompression = not raw and bool(
+        decompress or (explicit_output and output_suffix != source_suffix))
+    html_conversion = (not raw and explicit_output and source_suffix in (".htm", ".html")
+                       and output_suffix == ".csv")
     if resume and (decompress or
                    (source_suffix in (".gz", ".zip") and archive_decompression) or
-                   (explicit_output and source_suffix in (".htm", ".html") and output_suffix == ".csv")):
-        raise ValueError("resume=True supports raw downloads only; retain the archive suffix")
+                   html_conversion):
+        raise ValueError("resume=True supports raw downloads only; use raw=True or retain the archive suffix")
     if not force:
         try:
             validate_file(full_path, expected_sha256, expected_size)
@@ -728,7 +746,7 @@ def fetch_file(
         chunk_size=chunk_size,
         progress_callback=progress_callback,
         decompress=archive_decompression,
-        convert_html=explicit_output,
+        convert_html=html_conversion,
         explicit_output=explicit_output,
         expected_sha256=expected_sha256,
         expected_size=expected_size,

datacache/resume.py

@@ -1,4 +1,4 @@
-"""Integrity-pinned raw HTTP downloads with private, persistent partials."""
+"""Hash- or strong-ETag-validated HTTP downloads with private partials."""
 
 from contextlib import closing
 import hashlib
@@ -20,8 +20,8 @@
 
 def validate_resume(download_url, expected_sha256, expected_size):
     _validate_expectations(expected_sha256, expected_size)
-    if expected_sha256 is None or expected_size is None:
-        raise ValueError('resume=True requires expected_sha256 and expected_size')
+    if expected_size is None:
+        raise ValueError('resume=True requires expected_size')
     if urlsplit(download_url).scheme.lower() not in ('http', 'https'):
         raise ValueError('resume=True supports only raw HTTP/HTTPS downloads')
     if os.name != 'posix':
@@ -61,9 +61,15 @@ def discard_partial(destination):
             (directory / name).unlink(missing_ok=True)
 
 
+def _strong_etag(value):
+    # RFC 9110 entity-tag syntax, without the weak W/ prefix. Embedded quotes,
+    # whitespace, and control characters cannot authorize an If-Range request.
+    return isinstance(value, str) and re.fullmatch(r'"[\x21\x23-\x7e\x80-\xff]*"', value) is not None
+
+
 def _validator(headers):
     etag = headers.get('ETag', '')
-    if etag.startswith('"') and etag.endswith('"'):
+    if _strong_etag(etag):
         return {'header': 'ETag', 'value': etag}
     # Last-Modified is not always strong, but changes still invalidate a partial.
     modified = headers.get('Last-Modified')
@@ -80,7 +86,8 @@ def download_resumable(download_url, destination, *, expected_sha256, expected_s
     directory = _prepare_directory(destination)
     partial = directory / 'partial'
     metadata_path = directory / 'metadata.json'
-    identity = {'sha256': expected_sha256.lower(), 'size': expected_size,
+    identity = {'sha256': expected_sha256.lower() if expected_sha256 is not None else None,
+                'size': expected_size,
                 'url_hash': hashlib.sha256(download_url.encode()).hexdigest()}
     with file_lock(directory / 'lock', private=True):
         # Another installer may have completed while we waited for its lock.
@@ -100,10 +107,14 @@ def download_resumable(download_url, destination, *, expected_sha256, expected_s
             # Unlink, never truncate an unknown path planted in the state dir.
             partial.unlink(missing_ok=True)
         validator = metadata.get('validator')
-        if validator is not None and not (
+        valid_validator = (
                 isinstance(validator, dict) and validator.get('header') in ('ETag', 'Last-Modified')
                 and isinstance(validator.get('value'), str)
-                and '\r' not in validator['value'] and '\n' not in validator['value']):
+                and '\r' not in validator['value'] and '\n' not in validator['value']
+                and (validator['header'] != 'ETag' or _strong_etag(validator['value'])))
+        strong_validator = (valid_validator and validator['header'] == 'ETag')
+        if ((validator is not None and not valid_validator)
+                or (expected_sha256 is None and not strong_validator)):
             metadata['validator'] = None
             partial.unlink(missing_ok=True)
         write_json(metadata_path, metadata)
@@ -122,16 +133,20 @@ def report():
                     offset = output.tell()
                     if offset >= expected_size:
                         output.flush()
-                        try:
-                            validate_file(partial, expected_sha256, expected_size)
-                        except FileValidationError:
-                            output.seek(0)
-                            output.truncate()
-                            offset = 0
-                            metadata['validator'] = None
-                        else:
-                            report()
-                            break
+                        if expected_sha256 is not None:
+                            try:
+                                validate_file(partial, expected_sha256, expected_size)
+                            except FileValidationError:
+                                pass
+                            else:
+                                report()
+                                break
+                        # Size alone cannot validate a completed persistent
+                        # partial, including an empty one. Request fresh bytes.
+                        output.seek(0)
+                        output.truncate()
+                        offset = 0
+                        metadata['validator'] = None
                     callback_failed = False
                     try:
                         # One protocol restart per attempt, independent of retries.
@@ -175,6 +190,10 @@ def report():
                                     offset = 0
                                 else:
                                     raise FileValidationError(destination, 'expected HTTP 200 or 206')
+                                if expected_sha256 is None and (
+                                        current_validator is None or current_validator['header'] != 'ETag'):
+                                    raise FileValidationError(
+                                        destination, 'resume without expected_sha256 requires a strong ETag')
                                 metadata['validator'] = current_validator
                                 write_json(metadata_path, metadata)
                                 for chunk in response.iter_content(chunk_size=chunk_size):

datacache/version.py

@@ -1 +1 @@
-__version__ = '1.13.0'
+__version__ = '1.14.0'

docs/api.md

@@ -1,7 +1,7 @@
 # Public API reference
 
 This reference covers every name exported in `datacache.__all__` and every
-public `Cache` method in DataCache 1.13.0. Import these names from `datacache`.
+public `Cache` method in DataCache 1.14.0. Import these names from `datacache`.
 Signatures below show all defaults; arguments after `*` are keyword-only.
 Method signatures omit `self` and are called on a `Cache` instance.
 
@@ -60,7 +60,8 @@ fetch_file(
     timeout=None, use_wget_if_available=None, chunk_size=1048576,
     progress_callback=None, *, destination=None, cache_root=None, expected_sha256=None,
     expected_size=None, max_retries=2, retry_backoff=1.0, retry_max_delay=30.0,
-    show_progress=False, record_provenance=False, allow_empty=False, resume=False
+    show_progress=False, record_provenance=False, allow_empty=False, resume=False,
+    raw=False
 )
 ```
 
@@ -75,6 +76,7 @@ replacement leaves the previous file intact.
 | `download_url` | Source URL string: HTTP, HTTPS, FTP, or `file://`. |
 | `filename` | Optional string cache key. Omitted names are inferred from the full URL. Names are normalized by [build_local_filename](#build_local_filename), not treated as exact paths. |
 | `decompress` | Request gzip/ZIP contents. With inferred filenames, `False` retains the archive and `True` selects a separate decompressed key. An explicit output lacking the source's compression suffix also implies decompression. |
+| `raw` | Boolean disabling archive decompression and HTML conversion regardless of the output name. Default `False` preserves legacy inference; cannot be combined with `decompress=True`. Expectations describe the unchanged payload after HTTP transfer decoding. |
 | `subdir` | Application name selecting a platform cache directory; `None` selects `datacache`. See [get_data_dir](#get_data_dir). |
 | `force` | Download even when a cached file exists. Does not automatically broaden its permissions. |
 | `timeout` | Per-attempt timeout in seconds; `None` supplies no timeout. HTTP also accepts a Requests `(connect, read)` tuple. Timeouts limit connection/read inactivity, not overall elapsed time. |
@@ -89,7 +91,7 @@ replacement leaves the previous file intact.
 | `retry_backoff` | Initial retry delay in seconds, doubled for subsequent retries. Must be finite and non-negative. |
 | `retry_max_delay` | Maximum retry delay in seconds, finite and non-negative. A server `Retry-After` exceeding this limit stops retries. |
 | `show_progress` | Boolean enabling optional tqdm download, decompression, and hash bars. Requires `datacache[progress]` when a bar is needed. Cache hits are quiet. |
-| `resume` | Boolean retaining private partials for raw integrity-pinned HTTP transfers. Requires SHA-256, size and a POSIX local filesystem. See [resumable downloads](downloads.md#resumable-http-downloads). |
+| `resume` | Boolean retaining private partials for raw HTTP transfers. Requires size and a POSIX local filesystem. Without an expected SHA-256, every accepted response must supply a strong ETag. Size-only cache hits check only byte count. See [resumable downloads](downloads.md#resumable-http-downloads). |
 | `record_provenance` | Boolean; after publishing a download, also write a hidden `.<name>.datacache.json` record of the source URL (without user name, password, query string, or fragment; the path is kept as is, so avoid recording URLs with secrets in their path), the fetch time, the size, and the SHA-256 when `expected_sha256` verified it. [inspect_file](#inspect_file) reports these offline. The record has the file's permissions. Default `False`: a caller that downloads to a temporary name and then moves the file would leave the record behind. Cache hits never write one, any new download removes a previous record first, and a failure to write one never fails the download. |
 | `allow_empty` | Boolean accepting an empty installed file; default `False`. A complete but empty response, such as a withdrawn upstream record, is otherwise never published: HTTP retries it as transient, then raises `FileValidationError`. An empty cached file is likewise an invalid hit. `expected_size=0` also allows an empty file. |
 
@@ -867,7 +869,8 @@ Cache.fetch(
     url, filename=None, decompress=False, force=False, timeout=None,
     use_wget_if_available=None, *, chunk_size=1048576, progress_callback=None,
     expected_sha256=None, expected_size=None, max_retries=2, retry_backoff=1.0,
-    retry_max_delay=30.0, show_progress=False, record_provenance=False, allow_empty=False, resume=False
+    retry_max_delay=30.0, show_progress=False, record_provenance=False, allow_empty=False,
+    resume=False, raw=False
 )
 ```