[fix][fs] Fix warmup query hanging for 10 minutes after warmup has finished - #198
Merged
Merged
Conversation
The query loop only stopped when the client reported total == 0. The client keeps a finished task's status readable for kFinishedStatusTtl (10 minutes) and reports the completed snapshot for that whole window, so total only drops to 0 long after the warmup itself is done. This stalled the progress bar at 100% for a full 10 minutes on every `dingo fs warmup add` without --daemon. Break once every block is accounted for (finished + errors >= total), which is the real completion signal, and keep the total == 0 case for when the client has already dropped the status. Update the progress bar before breaking so it ends at 100%. Also correct the result format in a comment and an error message: the client emits [total/finished/errors], not [finished/total/errors] (the parsing was already using the correct order).
jackblack369
approved these changes
Aug 14, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem
dingo fs warmup add <path>(without--daemon) leaves the progress bar stuck at 100% for exactly 10 minutes before the command returns.Observed on a live cluster: the warmup itself finished in ~20ms, but the CLI kept polling for 10 more minutes.
Root cause
The two sides disagree on what "finished" means.
Client —
WarmupManager::FinishTaskkeeps a completed task's snapshot infinished_status_forkFinishedStatusTtl = 10 minutes, soGetWarmupTaskStatuskeeps returning5/5/0for that whole window and only returns0/0/0once the TTL expires.CLI — the poll loop in
runQuerytreatedtotal == 0as the only completion signal, so it had to wait out the client's TTL.This is a regression from dingofs
8e728fdd7("[feat][client] Make warmup scheduling nonblocking"). Before that commit a finished task was dropped fromwarmup_tasks_immediately and the status went to0/0/0right away, which is what thetotal == 0check implicitly relied on. Retaining the finished snapshot is intentional on the client side (otherwisewarmup querycannot distinguish "not started" from "just finished"), so the fix belongs here.Fix
Break once every block is accounted for —
finished + errors >= total— which is the actual completion signal. Thetotal == 0case is kept for when the client has already dropped the status.bar.Set64runs before the new break so the bar ends at 100% rather than at a partial value.Also corrects a comment and an error message that described the payload as
[finished/total/errors]; the client emits[total/finished/errors]and the parsing already used that order.Testing
go buildandgo vetpass forcli/command/fs/warmup/;gofmtclean.go build ./...fails in this repo, but identically on a cleanmain(58undefined:errors inproto/dingofs/proto/{cache,web}andinternal/rpc/mds.go— the generated proto code is out of sync with the Go code). I could not produce adingobinary to re-run the scenario against a live mount. Please build in a working environment and confirmdingo fs warmup addnow returns in seconds instead of 10 minutes before merging.