Report endpoint restore failures#6558
Conversation
|
Note This is an automated comment that will be appended during run. Note All workloads for linux-x86_64-relwithdebinfo have completed. Tip Planned checks for linux-x86_64-relwithdebinfo. 🟢 linux-x86_64-relwithdebinfo target: cloud/tasks/,cloud/storage/ (test time: 263s): all tests PASSED for commit 42d53f0.
🟢 linux-x86_64-relwithdebinfo target: cloud/disk_manager/ (test time: 285s): all tests PASSED for commit 42d53f0.
🔴 linux-x86_64-relwithdebinfo target: cloud/blockstore/ (test time: 1433s): some tests FAILED for commit 42d53f0.
🔴 linux-x86_64-relwithdebinfo target: cloud/blockstore/ (test time: 90s): some tests FAILED for commit 42d53f0.
🔴 linux-x86_64-relwithdebinfo target: cloud/blockstore/ (test time: 92s): some tests FAILED for commit 42d53f0.
🟢 linux-x86_64-relwithdebinfo target: cloud/filestore/ (test time: 5148s): all tests PASSED for commit 42d53f0.
|
| Y_UNUSED(ctx); | ||
| Y_UNUSED(request); | ||
|
|
||
| if (AtomicGet(RestoringStage) == Failed) { |
There was a problem hiding this comment.
May be it will be better to extend TListEndpointsResponse with fiels representing RestoringStage(something like EndpointsStorageState) value instead of failing request with E_FAIL? If the request could not be completed because of some underlying subsystem failure, E_INVALID_STATE seems to be more appropriate error code.
|
Note This is an automated comment that will be appended during run. Note All workloads for linux-x86_64-relwithdebinfo have completed. Tip Planned checks for linux-x86_64-relwithdebinfo. 🟢 linux-x86_64-relwithdebinfo target: cloud/tasks/,cloud/storage/ (test time: 260s): all tests PASSED for commit 3031e32.
🟢 linux-x86_64-relwithdebinfo target: cloud/disk_manager/ (test time: 263s): all tests PASSED for commit 3031e32.
🟢 linux-x86_64-relwithdebinfo target: cloud/blockstore/ (test time: 1448s): all tests PASSED for commit 3031e32.
🟢 linux-x86_64-relwithdebinfo target: cloud/filestore/ (test time: 4924s): all tests PASSED for commit 3031e32.
|
What changed
TEndpointManagerwith an atomic counter.E_FAILfromListEndpointsafter failed restore, including the number of failed endpoints.ShouldRestoreEndpointWithNbdDeviceto verify the newListEndpointserror behavior.Why
Previously restore errors were reported to logs/counters, but the final restore stage still became
Completed. As a result, clients listing endpoints could observeEndpointsWereRestored=trueeven when some endpoints failed to restore.Validation
git diff --check./ya make -tt cloud/blockstore/libs/endpoints/ut -F TEndpointManagerTest::ShouldRestoreEndpointWithNbdDevice./ya make -tt cloud/blockstore/libs/endpoints/ut