Summary
Improve the reliability of SFTP uploads and downloads over unstable networks through two complementary features: atomic writes (to prevent partially-written destination files) and resumable transfers (to recover interrupted transfers without re-uploading already-transferred data).
Motivation
On slow or flaky connections hussh transfers can be interrupted mid-stream, leaving a corrupt or truncated file at the destination with the same name as the intended output. There is also no way to resume a large upload; the entire file must be retransmitted from scratch.
Proposed API
Atomic Writes
Add an atomic flag (default: False) to sftp_write. When True, the file is uploaded to a <name>.hussh.tmp path and atomically renamed to the final path only on success. The temp file is removed if the transfer fails.
# Destination file is only created/replaced once the upload completes fully
conn.sftp_write("/local/app.tar.gz", "/remote/app.tar.gz", atomic=True)
Resumable Transfers
Add a resume flag (default: False) to sftp_write and sftp_read. When True, hussh checks the size of the existing destination file and uses SFTP seek to append only the missing bytes.
# If the remote file is already 500 MB, start uploading from byte 500 MB
conn.sftp_write("/local/disk.img", "/remote/disk.img", resume=True)
# If the local file is already 200 MB, continue downloading from byte 200 MB
conn.sftp_read("/remote/backup.tar", "/local/backup.tar", resume=True)
Implementation Notes
Atomic writes
- Derive the temp path:
"{destination}.hussh.tmp".
- Upload to the temp path using the normal SFTP write path.
- On success, call SFTP
rename(temp, destination).
- On any error, call SFTP
unlink(temp) (best-effort) and re-raise.
Resumable transfers
- Upload (
sftp_write): sftp.stat(remote_path) → get st_size. Open the remote file with APPEND flag, seek(st_size). Seek the local file to the same offset before reading.
- Download (
sftp_read): stat(local_path) → get local size. Open the local file in append mode. Open the remote file and seek(local_size).
- If the existing destination is larger than or equal to the source, treat as complete (no-op).
- Both options should compose:
atomic=True, resume=True is a valid (if unusual) combination; atomicity takes precedence for the final rename.
Acceptance Criteria
Summary
Improve the reliability of SFTP uploads and downloads over unstable networks through two complementary features: atomic writes (to prevent partially-written destination files) and resumable transfers (to recover interrupted transfers without re-uploading already-transferred data).
Motivation
On slow or flaky connections hussh transfers can be interrupted mid-stream, leaving a corrupt or truncated file at the destination with the same name as the intended output. There is also no way to resume a large upload; the entire file must be retransmitted from scratch.
Proposed API
Atomic Writes
Add an
atomicflag (default:False) tosftp_write. WhenTrue, the file is uploaded to a<name>.hussh.tmppath and atomically renamed to the final path only on success. The temp file is removed if the transfer fails.Resumable Transfers
Add a
resumeflag (default:False) tosftp_writeandsftp_read. WhenTrue, hussh checks the size of the existing destination file and uses SFTPseekto append only the missing bytes.Implementation Notes
Atomic writes
"{destination}.hussh.tmp".rename(temp, destination).unlink(temp)(best-effort) and re-raise.Resumable transfers
sftp_write):sftp.stat(remote_path)→ getst_size. Open the remote file withAPPENDflag,seek(st_size). Seek the local file to the same offset before reading.sftp_read):stat(local_path)→ get local size. Open the local file in append mode. Open the remote file andseek(local_size).atomic=True, resume=Trueis a valid (if unusual) combination; atomicity takes precedence for the final rename.Acceptance Criteria
atomic=Trueonsftp_write(sync and async)resume=Trueonsftp_writeandsftp_read(sync and async)rename)