Skip to content

feat: add local vs GP comparison, dynamic threads, smoothed ETA, and rate-limit retries#59

Open
nileshgarva wants to merge 1 commit into
xob0t:mainfrom
nileshgarva:feature/compare-and-dynamics
Open

feat: add local vs GP comparison, dynamic threads, smoothed ETA, and rate-limit retries#59
nileshgarva wants to merge 1 commit into
xob0t:mainfrom
nileshgarva:feature/compare-and-dynamics

Conversation

@nileshgarva

@nileshgarva nileshgarva commented Jun 9, 2026

Copy link
Copy Markdown

Summary

This PR introduces several performance and quality-of-life enhancements to the upload workflow, including automated comparison against Google Photos, dynamic thread scaling, smoothed ETA estimation, and robust network retry policies.


Key Features Implemented

1. Compare Local vs Google Photos (Duplicate Skipping)

  • Automated Indexing: On drag-and-drop, the app automatically checks files against Google Photos before queueing the upload.
  • Metadata Cache: Saves Google Photos filenames in a lightweight cache file (CompareFileList.csv under config directory) for sub-second startup times.
  • Differential Uploads: Automatically filters the queue to only upload missing files and logs them in missing_files.txt. Successfully uploaded files are appended to the CSV cache.
  • Protobuf Parser: Built raw varint and protobuf handlers for GET_LIB_STATE and GET_LIB_PAGE_INIT calls directly in Go, ensuring no external tools or runtime python dependencies are required.

2. Real-Time Dynamic Thread Scaling

  • Live Worker Adjustments: Added a Settings gear panel with an interactive range slider inside the active upload panel.
  • Dynamic Goroutine Scaling:
    • Increasing threads spawns new workers to read from the upload channel instantly.
    • Decreasing threads flags excess workers to exit cleanly immediately after completing their current file.

3. Smoothed ETA Estimation

  • Jitter Prevention: Calculates remaining time using an Exponential Moving Average (EMA) algorithm ($\alpha = 0.1$) updated once per second.
  • Steady Countdown: Prevents countdown jumps caused by sudden throughput drops or progress jumps when files finish.

4. Rate-Limit (429) & Network Retry Backoff

  • Context Plumbing: Context cancellation is wired through all comparison and token generation requests, letting the user abort active network calls and retries immediately when clicking "Cancel".
  • Auto-Backoff: Implemented exponential backoff with random jitter for robust retries when encountering rate limits or 5xx server responses.

5. Windows Launch Optimization

  • Built with -ldflags "-H windowsgui" flags, disabling the Command Prompt console popup on Windows to open the GUI window directly.

Verification

  • Frontend built successfully (npm run build).
  • Backend compiled and packaged correctly on Windows (go build).
  • Verified duplicate checking, file logging, dynamic workers addition/subtraction, and cancellation propagation.
image

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants