Skip to content

feat: Atomic Writes and Resumable Transfers for SFTP #86

@JacobCallahan

Description

@JacobCallahan

Summary

Improve the reliability of SFTP uploads and downloads over unstable networks through two complementary features: atomic writes (to prevent partially-written destination files) and resumable transfers (to recover interrupted transfers without re-uploading already-transferred data).

Motivation

On slow or flaky connections hussh transfers can be interrupted mid-stream, leaving a corrupt or truncated file at the destination with the same name as the intended output. There is also no way to resume a large upload; the entire file must be retransmitted from scratch.

Proposed API

Atomic Writes

Add an atomic flag (default: False) to sftp_write. When True, the file is uploaded to a <name>.hussh.tmp path and atomically renamed to the final path only on success. The temp file is removed if the transfer fails.

# Destination file is only created/replaced once the upload completes fully
conn.sftp_write("/local/app.tar.gz", "/remote/app.tar.gz", atomic=True)

Resumable Transfers

Add a resume flag (default: False) to sftp_write and sftp_read. When True, hussh checks the size of the existing destination file and uses SFTP seek to append only the missing bytes.

# If the remote file is already 500 MB, start uploading from byte 500 MB
conn.sftp_write("/local/disk.img", "/remote/disk.img", resume=True)

# If the local file is already 200 MB, continue downloading from byte 200 MB
conn.sftp_read("/remote/backup.tar", "/local/backup.tar", resume=True)

Implementation Notes

Atomic writes

  1. Derive the temp path: "{destination}.hussh.tmp".
  2. Upload to the temp path using the normal SFTP write path.
  3. On success, call SFTP rename(temp, destination).
  4. On any error, call SFTP unlink(temp) (best-effort) and re-raise.

Resumable transfers

  • Upload (sftp_write): sftp.stat(remote_path) → get st_size. Open the remote file with APPEND flag, seek(st_size). Seek the local file to the same offset before reading.
  • Download (sftp_read): stat(local_path) → get local size. Open the local file in append mode. Open the remote file and seek(local_size).
  • If the existing destination is larger than or equal to the source, treat as complete (no-op).
  • Both options should compose: atomic=True, resume=True is a valid (if unusual) combination; atomicity takes precedence for the final rename.

Acceptance Criteria

  • atomic=True on sftp_write (sync and async)
  • resume=True on sftp_write and sftp_read (sync and async)
  • Interrupted upload leaves no file (atomic) or a partial file smaller than source (resumable)
  • Integration tests that simulate interruption and verify correct behaviour
  • Documentation updated with examples and caveats (e.g. server must support rename)

Metadata

Metadata

Assignees

No one assigned

    Labels

    enhancementNew feature or request

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions