Streamable Archive

Streamable Archive is a Python command-line tool for transferring large directories with chunked compression, HTTP Range requests, resumable downloads, and immediate extraction.

Instead of downloading a complete archive before extracting it, Streamable Archive processes one independently compressed chunk at a time:

Download one compressed chunk
            ↓
Verify the chunk
            ↓
Extract immediately
            ↓
Write to the target file
            ↓
Continue with the next chunk

The receiving device does not need to store a complete .raw archive. It still needs enough free space for the final extracted files.

The project name is Streamable Archive. The Python package and CLI command remain streamable_archive for compatibility.

Features

• Stream a directory into a .raw data file and a .rawinfo metadata file;
• Split source data into independently compressed chunks;
• Use zlib compression from the Python standard library;
• Verify every compressed chunk with SHA-256;
• Automatically derive the .raw URL from a .rawinfo URL;
• Download compressed chunks with HTTP Range requests;
• Download and decompress multiple chunks concurrently with a bounded in-flight window;
• Extract and write each chunk immediately after downloading it;
• Retry failed chunks with exponential backoff;
• Resume interrupted transfers at chunk granularity;
• Estimate required target disk space before downloading;
• Validate paths, chunk ranges, metadata sizes, and checksums;
• Restore empty files, Unicode names, nested directories, and large files;
• Run without additional runtime dependencies;
• Provide unit tests and HTTP end-to-end tests.

Current limitations

The current version does not provide:

• A graphical user interface;
• An embedded HTTP file server;
• Upload integration for cloud drives or object storage;
• Encryption or password protection;
• Symbolic links, hard links, or other special filesystem objects;
• Complete restoration of operating-system-specific permissions and ownership metadata.

How it works

Streamable Archive does not compress one complete stream and then cut the compressed bytes arbitrarily. Instead, it splits the original data first and compresses every chunk independently:

Original file
[Raw chunk 0][Raw chunk 1][Raw chunk 2]...
       ↓           ↓           ↓
Compressed chunk 0  Compressed chunk 1  Compressed chunk 2
       └──────────────┬──────────────┘
                      ↓
                 project.raw

The .rawinfo file stores the directory structure, file sizes, file offsets, compressed offsets, compressed lengths, and checksums.

The default 1 MiB chunk size refers to the uncompressed source data size, not the HTTP request size. The actual Range request size depends on the compression ratio of each chunk.

Requirements

• Python 3.12 or newer;
• An HTTP server that supports Range requests;
• Enough free space on the target volume for all extracted files;
• No additional runtime dependencies for the packer or downloader.

Install the project in editable mode:

python -m pip install -e .

Install test dependencies:

python -m pip install -e ".[test]"

If the streamable_archive executable is not available in your PATH, use the module entry point:

python -m streamable_archive --help

Quick start

1. Create an archive

python -m streamable_archive pack <source_dir> -o <output_prefix>

Example:

python -m streamable_archive pack \
  "/data/project" \
  -o "/share/project"

The command creates:

<output_prefix>.raw
<output_prefix>.rawinfo

Default packer settings:

• Raw chunk size: 1 MiB;
• Compression algorithm: zlib;
• Compression level: 6.

Use custom chunk and compression settings:

python -m streamable_archive pack \
  "/data/project" \
  -o "/share/project" \
  --chunk-size 4194304 \
  --compression-level 6

--chunk-size is the uncompressed size of each source chunk. Larger chunks reduce the number of HTTP requests, but increase the amount of data that must be retried after a failure.

View packer options:

python -m streamable_archive pack --help

2. Serve the archive files

Streamable Archive does not provide an embedded HTTP server. Use an existing HTTP file server and place the two generated files in the same directory:

project.raw
project.rawinfo

The downloader only needs the .rawinfo URL:

http://192.168.1.10/share/project.rawinfo

It automatically derives the data URL:

http://192.168.1.10/share/project.raw

The server must support HTTP Range requests. Check it with curl:

curl -i \
  -H "Range: bytes=0-99" \
  http://192.168.1.10/share/project.raw

A successful response should include:

HTTP/1.1 206 Partial Content
Content-Range: bytes 0-99/...

If the server does not support Range requests, the downloader does not silently download a multi-gigabyte .raw file in full.

3. Download and extract while downloading

python -m streamable_archive fetch <rawinfo_url> -o <target_dir>

Example:

python3 -m streamable_archive fetch \
  "http://192.168.1.10/share/project.rawinfo" \
  -o "/data/restored/project"

The downloader will:

1. Download and validate .rawinfo;
2. Derive the .raw URL;
3. Validate metadata and available target disk space;
4. Request each compressed chunk with HTTP Range;
5. Verify the compressed chunk checksum;
6. Extract the chunk and write it at the correct file offset;
7. Persist progress;
8. Continue until all chunks are complete.

View downloader options:

python -m streamable_archive fetch --help

Downloader options

--retries <n>             Maximum retries per chunk, default: 5
--timeout <seconds>       Timeout for one HTTP request, default: 30
--resume                  Resume interrupted work, enabled by default
--no-resume               Ignore existing progress for the current task
--download-workers <n>    Concurrent download/decompression workers, default: 4
--max-in-flight <n>       Bounded completed chunks waiting for the single writer
--sync-every <n>          Force disk sync after every n written chunks, default: 32
--delete-existing         Delete all existing target contents without prompting

Example:

python -m streamable_archive fetch \
  "http://192.168.1.10/share/project.rawinfo" \
  -o "/data/restored/project" \
  --retries 5 \
  --timeout 60 \
  --download-workers 4 \
  --max-in-flight 8 \
  --resume

If the target directory is not empty, fetch asks whether to delete all existing contents. Answer N (the default) to keep the contents and continue/resume, or answer Y to clear the directory before starting. Use --delete-existing to skip the prompt and clear it in scripts.

Progress is stored at:

<target_dir>/.streamable-archive-progress.json

The progress file is removed automatically after the task completes successfully.

Disk space

Streaming extraction avoids storing a complete compressed .raw file on the receiving device. It does not remove the need for space for the final extracted directory.

For example:

Original directory: 100 GiB
.raw archive:       60 GiB
Extracted directory:100 GiB

The target volume must have enough space for the final 100 GiB directory, plus the controlled processing buffer for the current chunk.

Before the first Range request, the downloader checks available space using total_uncompressed_size from .rawinfo. If there is not enough space, it stops and reports the estimated requirement and available space.

File format

The packer creates two files:

<name>.raw
<name>.rawinfo

.raw contains a sequential concatenation of independent compressed chunks:

[CompressedChunk0][CompressedChunk1][CompressedChunk2]...

.rawinfo is UTF-8 JSON and includes, among other fields:

• format and format_version;
• chunk_size;
• Compression algorithm and parameters;
• Checksum algorithm;
• File and directory entries;
• Total uncompressed and compressed sizes;
• file_offset for every chunk;
• raw_offset for every chunk;
• compressed_size for every chunk;
• uncompressed_size for every chunk;
• SHA-256 checksum for every compressed chunk.

Safety behavior

The downloader rejects:

• Absolute paths;
• .. path traversal;
• File and directory path conflicts;
• Overlapping or out-of-range chunks;
• Inconsistent metadata totals;
• Compressed data with an invalid checksum;
• Data marked complete before writing has finished.

The first version does not execute programs or scripts from an archive and does not restore symbolic links or other special filesystem objects.

Testing

Run the complete test suite:

python -m pytest -q

Run the Python compilation check:

python -m compileall -q streamable_archive

The test suite covers:

• Metadata serialization and validation;
• CLI help and invalid arguments;
• Small files and chunk boundaries;
• Large-file chunking;
• Independent zlib chunks;
• Local restoration;
• HTTP Range and 206 Partial Content behavior;
• Chunk retries;
• Resumable downloads;
• Avoiding repeated requests for completed chunks;
• Path safety;
• Insufficient disk space;
• HTTP end-to-end restoration.

Large-directory validation

The current implementation has completed an end-to-end test with a large directory. The test included archive creation, HTTP Range downloads, immediate extraction, and per-file SHA-256 verification after restoration.

Results:

Source files:              503
Source directories:         95
Uncompressed size:   2,535,408,429 bytes
Independent chunks:       2903
Compressed size:     1,931,778,646 bytes
Restored files:             503
Missing files:                0
Extra files:                  0
Content mismatches:           0

Manual verification

After creating an archive, place .raw and .rawinfo in the same directory on an HTTP server. On another device with Python 3.12 or newer, install the project and run:

python3 -m pip install -e .
python3 -m streamable_archive fetch \
  http://<source-host>/share/project.rawinfo \
  -o /data/project

Depending on the local Python installation, python can be used instead of python3.

Recommended checks:

• Python environment and dependency installation;
• Network connectivity between the source and target devices;
• HTTP Range support;
• Writing to the target volume;
• Unicode file names;
• Restarting after an interruption;
• File-by-file comparison between the source and restored directories.

Project documentation

• Product requirements
• Implementation roadmap
• Development and validation status
• 中文文档

Project name and package name

The public project name is Streamable Archive.

The Python distribution and CLI command remain streamable_archive:

python -m streamable_archive --help

This keeps the existing package interface stable while providing a clearer project name for discovery and documentation.

License

This project is released under the BSD Zero Clause License (0BSD). It permits:

• Commercial use;
• Private use;
• Modification;
• Copying;
• Distribution;
• Sublicensing.

The license does not require preserving a copyright notice or license notice. See LICENSE for the full text.