netdisk-stream-uploader

StreamPack — A streaming netdisk uploader with volume-based compression.

中文文档

netdisk-stream-uploader is the project name (repository / PyPI package); StreamPack is the product name — and the application name registered on the Baidu Open Platform, which is why the upload path is fixed to /apps/StreamPack.

Compress multiple files and folders into a ZIP archive, split it into volumes, and upload each volume to Baidu Netdisk as it is produced — without ever storing the whole archive in memory or on disk.

Background

Uploading a large dataset to a netdisk usually follows this pattern:

Compress everything into a single large .zip file (taking gigabytes locally).
Upload it.
Delete the .zip after the upload completes.

This requires N GB of free local disk space, where N is the size of the compressed archive. For datasets on the order of 100 GB, most machines cannot provide that space on their system drive.

netdisk-stream-uploader uses a different approach:

Streaming compression — ZIP bytes are generated on the fly and never buffered as a whole.
Volume splitting — the byte stream is cut into volumes of a configurable size (1 GB by default).
Upload-then-delete — each volume is uploaded as soon as it is ready, then removed locally after a successful upload.
Backpressure control — the number of in-flight volumes is bounded, capping peak disk usage.
Resumable — task state is persisted in SQLite; an interrupted upload resumes from the last successful volume.

Peak local disk usage is approximately the size of a single volume (1 GB by default), independent of the total source size.

Features

Streaming compression, powered by stream-zip
Volume-based upload, each volume well under the Baidu API single-file limit
Resumable uploads, with SQLite task tracking and automatic skipping of completed volumes
Source fingerprint check, confirming nothing changed before a resume
Self-describing archive: the first ZIP entry is __streampack__.json (task parameters + full file list)
Automatic retries with exponential backoff (3 attempts by default)
Concurrent uploads, with a configurable worker count
Backpressure control, an optional cap on in-flight volumes
Resource-aware scheduling, choosing temp directories by free space and enforcing a minimum free-disk threshold
Orphan cleanup, removing leftover temp files from crashed sessions on the next start
Custom output extension, the archive is always ZIP but can be named .zi123p or similar
Optional AES-256 encryption (WinZip AES); the password is never stored, not even in the archive metadata
OAuth 2.0 (Authorization Code, OOB), no cookie hacks and no password exposure

Installation

pip install -r requirements.txt

Python 3.10 or newer is required.

Why you need your own app credentials

This is a local client that runs no server of its own: files go straight from your machine to your netdisk, with no third party in between. Two consequences follow:

Nobody can see your data — no relay, no logs, no backend;
The author cannot hold credentials for you — that would mean operating your account under someone else's identity, with every upload counted against their quota, their app review, and their record of violations.

So each user registers their own app once. It is free; the only cost is a one-time identity verification (about 10 minutes).

Do not use an AppKey / SecretKey handed to you by someone else. It puts your files under their app quota, and lets them switch it off whenever they like.

First run

python main.py upload \
  --source /path/to/files \
  --files "a.mp4,b.mp4" \
  --remote /apps/StreamPack \
  --name my_archive \
  --volume-size 1073741824

On the first run you will be prompted for your Baidu AppKey and SecretKey (from the Baidu Netdisk Open Platform). They are stored locally in ~/.netdisk_uploader/credentials.json and never leave your machine.

You will then be asked to open an authorization URL in your browser and paste the resulting code back into the terminal. The token is stored in ~/.netdisk_uploader/token.json.

Usage

Upload

# Upload specific files
python main.py upload -s /data -f "a.mp4,b.mp4" -r /apps/StreamPack -n backup

# Upload everything under --source (use ".")
python main.py upload -s /data -f "." -r /apps/StreamPack -n backup

# Custom volume size (1 MB, useful for testing)
python main.py upload -s /data -f "." -r /apps/StreamPack -n backup -v 1048576

# Enable backpressure (default cap = workers × 2)
python main.py upload -s /data -f "." -r /apps/StreamPack -n backup --backpressure

# Custom backpressure cap
python main.py upload -s /data -f "." -r /apps/StreamPack -n backup --backpressure=20

# Custom output extension
python main.py upload -s /data -f "." -r /apps/StreamPack -n backup --ext .zi123p

# Encrypted archive (prefer the environment variable over the command line)
export STREAMPACK_PASSWORD='your-pass'
python main.py upload -s /data -f "." -r /apps/StreamPack -n backup --password "$STREAMPACK_PASSWORD"

Offline rehearsal

Switch the backend to local and volumes are written to a local directory instead of the netdisk — no network, no quota, so you can check the split before committing to an upload.

python main.py upload -s /data -f "." -r /backup -n backup -v 1048576 --provider local

Output lands in ./_rehearsal/backup/ (change it with config set local_rehearsal_dir). The same works for rehearsing a resume offline: Ctrl+C halfway, then resume <task_id> --provider local.

Resume an interrupted task

# List all tasks
python main.py list

# Show task details
python main.py status <task_id>

# Resume by ID
python main.py resume <task_id>

# Or: re-run the exact same command — task_id is deterministic
python main.py upload -s /data -f "a.mp4,b.mp4" -r /apps/StreamPack -n backup

If the process is killed (window closed, power loss), the task stays in running. The next command of any kind resets it to interrupted and prints how many tasks were reset; resume continues from there.

Before resuming, the source files are fingerprinted. If anything was added, removed or modified, the differences are listed and the resume aborts rather than continuing with a misaligned stream:

python main.py resume <task_id>                 # default full: fingerprint + per-volume MD5
python main.py resume <task_id> --verify=stat   # fingerprint only
python main.py resume <task_id> --verify=off    # skip the check (only if you know nothing changed)

Find which volume holds a file

The self-contained layout (--layout self-contained) records the volume-to-file mapping in the local database, so this works offline:

python main.py find c.pdf              # across all tasks
python main.py find photos --task <id> # within one task

找到 1 处：
  src/docs/c.pdf
    任务 8f92e68213c3c048（demo） 第 2 卷
    /apps/StreamPack/demo.part0002.zip

In the GUI: Settings → Find file… (Ctrl+F).

The continuous layout keeps no such mapping — a volume holds byte fragments, not whole files — so finding nothing there is expected.

Graphical interface

pip install -r requirements-gui.txt   # PySide6-Essentials only, no QtWebEngine
python main.py --gui

The window shows the task list with progress, per-volume details, resume, delete, and a live log pane.

New task: Settings → New upload task… (Ctrl+N), or just drag files/folders into the window. The dialog takes a name, your app name (the path is derived from it), volume size, an optional extension and encryption password; pick the local backend to rehearse offline.

Settings: Settings → Settings… changes the UI language (auto/zh/en), theme (auto/light/dark), app name and subfolder, and per-volume retries, plus "open config file" / "open data folder".

Everything the UI does ends up in the same CLI: a new task launches streampack.exe upload …, resume launches resume <id>, so behaviour is identical to the command line.

Delete task records

python main.py delete <task_id> --yes

Only local records are deleted; uploaded files are untouched.

Global options

These apply to every subcommand and go before it:

python main.py --debug list              # DEBUG-level logging
python main.py --quiet list              # WARNING and above only
python main.py --log-file run.log list   # also write to a log file
python main.py --lang en list            # switch language (zh / en / auto)
python main.py --no-proxy list           # ignore the system proxy for this run
python main.py --version                 # print the version
python main.py --gui                     # graphical interface

--debug takes precedence over --quiet.

How it works

[source files] --stream--> [ZIP bytes] --split--> [volume 1] --upload--> [Baidu]
                                                   [volume 2] --upload--> [Baidu]
                                                   ...
                                                   [volume N] --upload--> [Baidu]

Each volume is deleted after a successful upload. Task state (which volumes succeeded) is stored in SQLite at ~/.netdisk_uploader/tasks.db.

A failed volume is retried with exponential backoff: 3 attempts by default, waiting 2s then 4s. Only after all attempts fail is the volume marked failed; the remaining volumes continue normally.

Configuration

Configuration resolves in this order (highest first):

environment variables STREAMPACK_<FIELD NAME>
the user config file ~/.netdisk_uploader/config.json
code defaults in src/config/settings.py

There is no config.ini — the file is JSON at %USERPROFILE%\.netdisk_uploader\config.json (override with STREAMPACK_CONFIG). It only stores options you changed explicitly, so:

never changed anything → the file does not exist, and that is normal
changed something → only those keys appear in it

python main.py config path prints the location; Settings → Open config file opens it.

Manage it with the config command — no source edits needed:

python main.py config list                      # every option and where its value comes from
python main.py config get max_upload_retries    # read one
python main.py config set max_upload_retries 5  # change one (written to the config file)
python main.py config unset max_upload_retries  # back to the default
python main.py config path                      # where the config file lives

The file only stores options you changed explicitly; everything else keeps following the code defaults, so new defaults take effect after an upgrade.

Options:

Option	Default	Meaning
`preferred_temp_dirs`	`D:/netdisk_temp`, `E:/netdisk_temp`, `C:/netdisk_temp`	Where volumes are stored during upload
`min_free_disk_bytes`	2 GB	Refuse to use a disk with less free space
`max_parallel_workers`	0 (= CPU cores − 1)	Concurrent uploads
`max_pending_volumes`	None	Backpressure cap (None = disabled; the CLI flag wins)
`max_upload_retries`	3	Max attempts per volume, including the first
`retry_backoff_seconds`	2.0	Delay before the first retry; doubles each time
`provider`	`baidu`	Storage backend: `baidu` or `local`
`local_rehearsal_dir`	`./_rehearsal`	Root directory for `provider=local`
`app_name`	`StreamPack`	Baidu app name; determines the writable folder `/apps/<app name>`
`remote_subdir`	empty	Optional subfolder under the app folder

Any option can also be overridden per-session with an environment variable:

export STREAMPACK_MAX_UPLOAD_RETRIES=1     # Linux / macOS
set STREAMPACK_MAX_UPLOAD_RETRIES=1        # Windows

Language

Both the command line and the GUI ship in Chinese and English. Three ways to switch:

python main.py --lang en --gui          # one run
python main.py config set language en   # persist it
export STREAMPACK_LANG=en               # environment variable

Priority: --lang > STREAMPACK_LANG > the language config option > system locale. language defaults to auto.

The source strings are the Chinese text in the code; the English catalogue lives in locale/en/LC_MESSAGES/streampack.po. After editing it:

python scripts/build_mo.py     # .po → .mo, pure stdlib, no gettext toolchain needed

Adding a language: copy locale/en to locale/<lang>, translate the .po, rebuild, then add the code to SUPPORTED in src/i18n.py.

Packaging

pip install pyinstaller
pwsh -File scripts/build_release.ps1

Output:

Path	Purpose
`dist/StreamPack/`	Runnable as-is, no installation
`dist/StreamPack-<version>-win64.zip`	Portable package, unzip and run
`dist/StreamPack-Setup-<version>.exe`	Installer, the recommended option for end users

The folder holds two entry points: streampack.exe (command line, keeps the console so logs are visible) and streampack-gui.exe (GUI, no console window). Both share the dependencies in _internal/.

Ship only the installer and the portable package — a bare exe cannot run without its _internal/ folder.

Pushing a v* tag makes GitHub Actions run the tests, build both artifacts and create a release (.github/workflows/build.yml) — nothing needs to be installed locally.

Installing

Double-click StreamPack-Setup-<version>.exe. It installs into %LOCALAPPDATA%\Programs\StreamPack — no administrator rights needed — and creates a Start Menu shortcut plus an entry under Settings → Apps.

Silent install (for scripts):

.\StreamPack-Setup-0.7.0.exe --silent --result-file install.json
# {"ok": true, "version": "0.7.0", "install_dir": "...", "copied": 7}

Options: --dir <path> to change the location, --no-shortcut to skip the shortcut. On failure the exit code is 1 and the result file carries an error field.

Running the download

Extract it first — double-clicking an exe inside the archive will not find the sibling _internal folder
streampack-gui.exe opens the GUI; streampack.exe is the command line
On first run, enter your Baidu AppKey / SecretKey, then authorize in the browser and paste the code

About "Unknown publisher": the build is not code-signed, so SmartScreen warns once — click "More info → Run anyway".

Local self-signing (optional)

If you only use it on your own machine, signing once makes the warning go away. After installing, open Tools → Trust and sign locally (this machine only)…. The confirmation lists exactly what will happen:

create or reuse a local root CN=Hungry-M for the current Windows user
issue a project child certificate CN=StreamPack
add the root to "Trusted Root Certification Authorities" and the child to "Trusted Publishers"
sign the program, then reopen it

A running exe cannot sign itself (the file is locked), so the app hands its PID and path to trust-sign.ps1 and exits; the helper waits for the file to unlock, signs, and relaunches.

This is local-only: distributing the signed build does not help anyone else — their machines do not trust your certificate and will report an invalid one. To verify a download instead, compare with SHA256SUMS.txt in the release:

Get-FileHash .\StreamPack-v0.7.0-win64.zip -Algorithm SHA256

To remove the trust again:

powershell -ExecutionPolicy Bypass -File trust-sign.ps1 -RemoveCert

It removes the child first and only deletes the root when no other project's child certificate is left.

If you would rather not run an exe at all, the source works the same way:

pip install -r requirements.txt -r requirements-gui.txt
python main.py --gui

Uninstalling

Installed version: Settings → Apps → StreamPack, or run uninstall.cmd in the install directory. Credentials and the task database are kept by default; add -PurgeData to remove them too.

Portable version: no installer, so no uninstaller. It only leaves things in two places:

Delete the extracted StreamPack folder
Delete %USERPROFILE%\.netdisk_uploader\ — credentials, token, task database and config all live there

The folder ships with cleanup.cmd: double-clicking it only previews what would be removed; pass a flag to act:

cleanup.cmd -Yes

It only deletes files it can prove are ours (by checking for credentials.json / token.json / tasks.db / config.json), never scans by folder name, never kills processes, and never touches certificates — those need trust-sign.ps1 -RemoveCert.

Doing it by hand is just as fine:

Remove-Item "$env:USERPROFILE\.netdisk_uploader" -Recurse -Force

Files already uploaded to the netdisk are not deleted; remove those in the netdisk itself.

FAQ

invalid_client during authorization

Almost always a mangled SecretKey. The common cause is an invisible control character pasted in from the web console (e.g. \u0016), or a stray space or newline. Copy it again from the console; the stored string is visible in token.json.

Upload returns 404 or 403

The target directory does not exist or is not writable. The current console no longer has a separate "app directory" field: writable paths are fixed to /apps/{app name}, and the app name must be ASCII — a Chinese name breaks the path.

If you are using someone else's open-source tool with your own app, the default StreamPack is the author's app name and will not match yours. Set it once:

python main.py config set app_name your-app-name
# uploads to /apps/your-app-name; add a subfolder if you want another level
python main.py config set remote_subdir backups

invalid_grant when exchanging the code

Authorization codes are single-use and expire in about 10 minutes. Reopen the authorization URL and get a fresh one.

The installer says Could not create temporary directory

The installer is a onefile build: it unpacks itself into the temp directory first. This error means the temp directory is not writable, not that the package is corrupt. Check in order:

Are you launching from a restricted context (sandbox, restricted token, some managed endpoints)? Try double-clicking from Explorer instead.
Does %TEMP% exist and is it writable?
```
$env:TEMP; Test-Path $env:TEMP
```

Point the temp variables at a writable directory and retry (the package itself is unchanged):

$env:TEMP = "$env:LOCALAPPDATA\Temp"; $env:TMP = $env:TEMP
.\StreamPack-Setup-0.8.0.exe

Do not use --runtime-tmpdir "." to unpack next to the exe: that requires starting from a writable directory and leaves _MEI… leftovers beside the installer.

ProxyError / Unable to connect to proxy

The program uses the system proxy (not proxy environment variables). This error normally means the proxy client is not running while the system proxy setting is still enabled. Two options:

# 1) start your proxy client, then retry
# 2) bypass the proxy for this run only (changes nothing system-wide)
python main.py --no-proxy upload -s ... -f ... -n ...

--no-proxy affects this invocation only; it does not touch the registry or environment variables.

Upload timeouts, some volumes failing

Run resume first — completed volumes are skipped:

python main.py resume <task_id>

If it still fails, in order:

Flaky system proxy — retry with --no-proxy (in practice the same task went through in one pass):
```
python main.py --no-proxy resume <task_id>
```
Slow uplink — the default concurrency is CPU cores − 1 (7 here), too many for a slow link:
```
python main.py config set max_parallel_workers 2
```
Coarse retry granularity — smaller volumes mean less to re-upload per failure:
```
python main.py upload ... --volume-size 2M
```

Failure logs include how long each attempt took (耗时 11.0s): failing after a few seconds usually means the connection was cut (proxy/firewall); no progress for a long time means the uplink is saturated.

Do I need to re-authorize often

Normally no. The access token expires in about 30 days and is refreshed automatically with the refresh token; you are only asked to authorize again when the refresh itself fails.

How much local disk do I need

Roughly volume size × (1 + in-flight volumes). Backpressure is off by default, so the peak can approach the whole archive; use --backpressure to keep it tight.

Can I interrupt it

Yes. Ctrl+C stops scheduling further volumes and removes the ones not yet uploaded; completed volumes stay recorded, and resume continues from there.

How do I put the volumes back together

With the default --layout continuous, each volume is a slice of one continuous ZIP byte stream — concatenate them in order:

cat arch.part*.zip > arch.zip

copy /b arch.part0001.zip+arch.part0002.zip+arch.part0003.zip arch.zip

The result opens directly. Its first entry is __streampack__.json, recording the task parameters and the full file list, so you can tell where the volumes came from.

I want every volume to open on its own

Use --layout self-contained:

python main.py upload -s D:\data -f . -n backup --volume-size 1G --layout self-contained

Each volume is an independent ZIP — pick any one, open it with 7-Zip, done. No concatenation. Three costs:

Constraint	Detail
A file cannot exceed the volume	A 12 GB video will not fit a 1 GB volume; the check happens before uploading, listing the offending files, instead of failing halfway
Packing is by uncompressed size	with 0.5% headroom, so two 1 GB files do not fit a 2 GB volume
Resume granularity is the whole volume	a failed volume is re-uploaded entirely (usually a group of files)

Keep folders together:

python main.py upload ... --layout self-contained --pack-by folder

--pack-by folder groups by top-level folder and keeps a folder in one volume when it fits, splitting inside the folder only when necessary — the volume size cap is unchanged; it just makes "unzip one volume, get one whole folder" more likely. The default --pack-by size simply fills volumes.

Index file: after the task completes, a <name>.index.json is uploaded too, mapping each volume to the files it holds:

{"layout": "self-contained", "pack_by": "folder",
 "volumes": [{"part": 1, "remote_path": ".../backup.part0001.zip",
              "files": ["photos/a.jpg", "photos/b.jpg"]}]}

Use it to find which volume holds a given file. A failed index upload does not affect the volumes.

The two layouts produce different task IDs, so they never collide; resume follows the layout recorded when the task was created.

How do I open an encrypted archive

Use a tool that supports WinZip AES — 7-Zip, WinRAR, Keka. Windows Explorer's built-in ZIP support does not handle AES; a "cannot open" error there does not mean the archive is corrupt.

The password is stored nowhere (the database only keeps a PBKDF2 verifier for comparison), so losing it is unrecoverable. To resume an encrypted task you must supply the same password — --password or STREAMPACK_PASSWORD. Without it you are prompted; with a wrong one the resume is rejected immediately instead of wasting a full compression pass.

Requirements

Python 3.10+
A Baidu Netdisk Open Platform application (free to register)

Roadmap

RAR is out of scope: the compression algorithm is proprietary, the vendor only open-sources extraction and forbids using it to create archives, and no Python writer exists.

Contributing

Issues and pull requests are welcome.

Adding another netdisk: see docs/storage-provider.md, or open an issue with the netdisk support template
Setup, commit conventions and release flow: see CONTRIBUTING.md
Translations: edit locale/*/LC_MESSAGES/*.po, then run python scripts/build_mo.py

License

Apache License 2.0 — see LICENSE. Bundled third-party components and their licenses are listed in THIRD_PARTY_NOTICES.md.

Security policy: SECURITY.md. Change history: CHANGELOG.md.

Acknowledgements

stream-zip — streaming ZIP generation without buffering the whole archive
Baidu Netdisk Open Platform

Navigation Menu

Repository navigation

About

Repository files navigation

netdisk-stream-uploader

Background

Features

Installation

Why you need your own app credentials

First run

Usage

Upload

Offline rehearsal

Resume an interrupted task

Find which volume holds a file

Graphical interface

Delete task records

Global options

How it works

Configuration

Language

Packaging

Installing

Running the download

Local self-signing (optional)

Uninstalling

FAQ

Requirements

Roadmap

Contributing

License

Acknowledgements

About

Releases

Packages

Used by

Contributors

Languages

Name		Name	Last commit message	Last commit date
.github		.github
assets		assets
docs		docs
locale/en/LC_MESSAGES		locale/en/LC_MESSAGES
packaging		packaging
scripts		scripts
src		src
tests		tests
.gitattributes		.gitattributes
.gitignore		.gitignore
CHANGELOG.md		CHANGELOG.md
CONTRIBUTING.md		CONTRIBUTING.md
LICENSE		LICENSE
README.md		README.md
README.zh.md		README.zh.md
SECURITY.md		SECURITY.md
THIRD_PARTY_NOTICES.md		THIRD_PARTY_NOTICES.md
gui-debug.spec		gui-debug.spec
gui-debug2.spec		gui-debug2.spec
gui_main.py		gui_main.py
main.py		main.py
pytest.ini		pytest.ini
requirements-dev.txt		requirements-dev.txt
requirements-gui.txt		requirements-gui.txt
requirements.txt		requirements.txt

Latest commit

History

Folders and files

Repository files navigation

netdisk-stream-uploader

Background

Features

Installation

Why you need your own app credentials

First run

Usage

Upload

Offline rehearsal

Resume an interrupted task

Find which volume holds a file

Graphical interface

Delete task records

Global options

How it works

Configuration

Language

Packaging

Installing

Running the download

Local self-signing (optional)

Uninstalling

FAQ

Requirements

Roadmap

Contributing

License

Acknowledgements

About

Topics

Resources

Contributing

Security policy

Stars

Watchers

Forks

Releases

Packages

Used by

Contributors

Languages