You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Multi-cloud migrations, scheduled backups, and CI/CD deployments need a “make dest look like source” primitive instead of manual put loops.
Post-transfer validation (hash / size comparison) is table stakes for preventing silent corruption; competitors like rclone expose sync + check as core commands.
Having both commands lets users follow a reliable “sync → check → promote” workflow inside one CLI.
Feature Requirements
storify sync <source> <dest>
Walk source and destination simultaneously, skip identical files (size+mtime or hash), upload/overwrite changed files, and optionally delete extra destination files (--delete=never|during|after).
Optional knobs: --dry-run, --concurrency N, --include/--exclude pattern, --track-renames, --create-empty-src-dirs.
Compare source/dest without modifying either side. Report matches, differences, and missing entries; support --one-way, --download (byte-for-byte when hashes are unavailable), and report files (--differ, --missing-on-src, etc.).
CLI wiring: add sync/check modules under src/cli, parse flags, and reuse the existing Context + Operator setup.
Dual traversal: build a comparer that iterates source/dest listings (via OpenDAL Lister), producing diff records (Match, SrcOnly, DstOnly) to feed sync or check pipelines.
Sync executor: for Match entries, decide whether to copy; queue deletes based on --delete mode; reuse storage::operations::{get, put, delete} for actual transfers.
Check reporter: implement a lightweight reporter (stdout or file) emitting = / + / - / * markers or JSON; optionally fall back to streaming comparison when hashes are missing.
Testing: extend tests/behavior with fixtures covering added/modified/deleted/missing cases on the local fs backend and at least one S3-compatible backend.
Success Criteria
storify sync can mirror a directory tree, respecting delete modes and skipping unchanged files.
storify check surfaces differences with clear exit codes; sync + check are documented in README/CLI help.
Behaviour tests prove that new commands work end-to-end for local + remote backends.
Why It Matters
putloops.sync+checkas core commands.Feature Requirements
storify sync <source> <dest>--delete=never|during|after).--dry-run,--concurrency N,--include/--exclude pattern,--track-renames,--create-empty-src-dirs.storify check <source> <dest>--one-way,--download(byte-for-byte when hashes are unavailable), and report files (--differ,--missing-on-src, etc.).Implementation Sketch
sync/checkmodules undersrc/cli, parse flags, and reuse the existingContext+Operatorsetup.Lister), producing diff records (Match,SrcOnly,DstOnly) to feed sync or check pipelines.Matchentries, decide whether to copy; queue deletes based on--deletemode; reusestorage::operations::{get, put, delete}for actual transfers.= / + / - / *markers or JSON; optionally fall back to streaming comparison when hashes are missing.tests/behaviorwith fixtures covering added/modified/deleted/missing cases on the localfsbackend and at least one S3-compatible backend.Success Criteria
storify synccan mirror a directory tree, respecting delete modes and skipping unchanged files.storify checksurfaces differences with clear exit codes;sync+checkare documented in README/CLI help.