Repository navigation
feat: 大文件分片级断点续传((b) 方案:抽 upload_parts + 会话落 SQLite) - #7
Merged
Merged
Conversation
探针阶段二已证明服务端认几小时前的 uniqueId/batchNo。这一版按 (b) 落地:
Wopan 只懂协议,会话生命周期在 job 层。
分层
- Wopan.upload_parts(ses, info, fh, start, on_part):从第 start 片接到末片,
upload() 与续传共用同一个循环,避免两条路径悄悄分叉
- engine.upload_resumable():建/复用/判废会话、按当前 token 重加密 fileInfo、
每 32 片刷一次 done;不认识 sqlite,SessionStore 由 runner/main 注入
- store.SessionStore:upload_sessions 单表,与 DB 同库另开连接
会话快照存整份 {uid, batch_no, info_plain, size, total, done, token_fp}
- 只存 uniqueId 续不上:服务端还比 batchNo 与加密后的 fileInfo
- 存 info_plain 而非密文:密文 key 由 access_token 派生,token 一换就解不开
- token_fp 不符 → 保守重开会话(对应探针里「token 绑定未知」那条未验证分支)
判废与收尾
- 键 = (dir_id, relpath, size, mtime_ns),文件一改自动失效
- TTL 24h,懒清理(prune 在每次 sync/push 开头跑一次)
- verify 分流:哈希/体积/未找到 → 删会话(已证伪);复核时列目录抛异常 →
保留会话(transient,下次直接续,不用重传 1.2GiB)
并发
- tpbackup.db 是 rollback journal(非 WAL),写锁全库:不做多线程并发写,
每 32 片由 job 侧串行刷一次,锁只罩一次 UPSERT
开关
- 默认只对 ≥16MiB 的文件启用(TP_RESUME_MIN_SIZE 可调,TP_RESUME=0 关闭);
小文件本来一片,存会话纯亏,也把 blast radius 压在这个阈值以上
验证
- 29 项单测全绿(新增 12 项:分片偏移/短读/会话往返/TTL/三件套沿用/判废)
- 真机回归(容器内 /tmp 跑新代码,未动线上):小文件 sync 2 个 + push 1 个
复核通过;24MB 文件人为断在第 3 片 → done=2 落盘 → 续传只补第 3 片,
uniqueId 沿用,云端 sha256 与本地逐字一致
顺带修:part_count 的 part 默认值改为调用时取 PART(原写法定义时绑定,
改 PART 不生效,测试与将来做可配置分片都会踩)。
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
探针阶段二已验证:服务端认 2h6m 前的 uniqueId/batchNo,且只补剩余片就能拼出完整文件。这一版按审定的 (b) 落地。
分层
Wopan.upload_parts(ses, info, fh, start, on_part):从第 start 片接到末片。upload()与续传共用这一个循环,避免两条路径分叉。engine.upload_resumable():建/复用/判废会话、按当前 token 重加密 fileInfo、每 32 片刷 done。不认识 sqlite。store.SessionStore:upload_sessions单表,与 DB 同库另开连接,由 runner/main 注入。会话快照存整份
{uid, batch_no, info_plain, size, total, done, token_fp, mtime_ns}。只存 uniqueId 续不上——服务端还比 batchNo 与加密后的 fileInfo;存 info_plain 而非密文,是因为密文 key 由 access_token 派生,token 一换就解不开。token_fp 不符时按保守策略重开会话。判废与收尾
并发
tpbackup.db是 rollback journal(非 WAL),写锁全库。所以不做多线程并发写:每 32 片由 job 侧串行刷一次,锁只罩一次 UPSERT。开关
默认只对 ≥16MiB 启用(
TP_RESUME_MIN_SIZE可调,TP_RESUME=0关闭)。小文件本来一片,存会话纯亏;这条阈值也是 blast radius 的边界。验证
顺带修
part_count的 part 默认值改成调用时取PART。原写法part=PART在定义时绑定,改PART不生效(写测试时先踩了这一下)。未覆盖
token 轮换后旧会话服务端是否仍接受——探针没测到这一分支,本实现取保守策略(重开),未做真机验证。废弃会话的云端残片何时 GC 也没验,设计里记一笔。