shared-drive directory transfers do not scale to large trees

[shepilov]

Problem

POST /sharings/drives/move performs directory copy-like operations synchronously and does not scale to very large trees, such as folders containing 200,000 files.

This affects:

• Directory copies
• Cross-instance directory moves
• Cross-stack directory moves
• Personal drive to or from shared drive transfers
• Shared drive to shared drive transfers

A same-instance move that only changes directory metadata is not affected.

Current behavior

The current implementation:

1. Walks the complete source tree.
2. Materializes every directory and file in memory.
3. Creates directories sequentially.
4. Transfers files sequentially.
5. Performs several remote requests per file:
   • Read source metadata
   • Download source content
   • Read destination parent
   • Upload destination content
6. Resolves effective access repeatedly through the shared-drive routes.
7. Deletes the source only after the complete copy succeeds.

In the shared-drive-to-shared-drive loop, response bodies are also closed with defer inside the file loop. This can retain many bodies and connections until the complete request finishes.

For a 200,000-file tree, this can produce:

• Memory usage proportional to the complete tree
• Hundreds of thousands of effective-access resolutions
• Several HTTP round trips per file
• Excessive CouchDB and remote-stack load
• Connection exhaustion
• Request and reverse-proxy timeouts
• No progress reporting
• No cancellation or resume mechanism
• A partially created destination after failure
• Duplicate conflict-suffixed files when the caller retries

PR #4926 makes remote authorization correct by routing operations through shared-drive endpoints, but this adds authorization overhead to the existing per-file algorithm. The scalability problem should be handled separately rather than adding a larger preflight permission matrix to that PR.

Proposed direction

Move large directory transfers to an asynchronous, server-side job:

• Return a job identifier without keeping the client request open.
• Traverse the source incrementally using pagination instead of materializing the complete tree.
• Process files with bounded concurrency.
• Close download bodies immediately after each transfer.
• Avoid resolving the same source or destination folder access for every file.
• Authorize the source and destination when creating the job and revalidate before destructive source deletion.
• Store checkpoints so a failed job can resume without creating duplicate conflict-suffixed files.
• Delete the source only after every destination item has been confirmed.
• Expose progress and terminal failure information.
• Define explicitly how nested sharing roots are handled when their containing directory is moved or copied.

Acceptance criteria

• A synthetic directory containing 200,000 files can be transferred without memory usage growing proportionally to the entire tree.
• Open HTTP bodies and connections remain bounded throughout the operation.
• Transfer concurrency is bounded and configurable.
• The initiating HTTP request does not remain open for the full transfer.
• Job status reports total, processed, failed, and remaining items.
• Retrying or resuming a failed transfer does not create duplicates.
• Source deletion occurs only after the destination is complete.
• Effective access is enforced on both the source and destination.
• Tests cover personal-to-drive, drive-to-personal, and drive-to-drive flows.
• The load-test scenario is added to or coordinated with Load-test shared drives #4867.

Related

• Use Effective Access In Backend Move Operations #4844
• Load-test shared drives #4867
• feat: Enforce effective member access on shared drive moves #4926