Export snapshots through disk-manager-admin#6567
Open
BarkovBG wants to merge 5 commits into
Open
Conversation
Reads all chunks of a ready snapshot (or image) directly from the dataplane storage (chunk map in YDB, chunk data in S3 or YDB), verifies checksums and assembles them into a local raw image file. Zero chunks are skipped, so the output is sparse where possible; incremental snapshots need no special handling since their chunk map is complete.
BarkovBG
force-pushed
the
users/barkovbg/exporter-reuse-snapshot-source
branch
from
July 21, 2026 15:26
2675f1a to
4d2419e
Compare
BarkovBG
changed the title
The dataplane package and the snapshot exporter kept two copies of the 4 MiB chunk size that had to stay equal. ChunkSize now lives next to the Chunk type in dataplane/common; the dataplane package aliases it and the exporter uses it directly.
Formatting a chunk with fmt reflects over the whole multi-megabyte Data slice. Test mock argument diffs format chunk arguments on every mocked ReadChunk call, which makes timing-sensitive exporter tests flaky.
Chunk map streaming, entry caching and zero chunk detection are delegated to snapshot.NewSnapshotSource — the same source implementation that dataplane transfer tasks use. The exporter itself is an ordered pipeline: the scheduler emits chunks in stream order, filling chunk map gaps with zero chunks, a bounded set of readers fills data chunks concurrently, and the writer streams everything to the destination strictly in order. Read-ahead is enforced by the source inflight queue, so the chunk map does not have to reside in memory. Single-process export is sufficient, so the --partition and --partition-count flags are removed and the command exports the whole snapshot. The encryption warning now matches the command help: exported data of an encrypted disk is the stored ciphertext.
BarkovBG
force-pushed
the
users/barkovbg/exporter-reuse-snapshot-source
branch
from
July 21, 2026 16:15
5a78278 to
88224e5
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Adds raw snapshot export to
disk-manager-admin snapshots export(alsoavailable as
snapshot export). The command streams raw snapshot bytes tostdout, writes logs to stderr; parallelism of chunk reads is controlled by
--read-workers.disk-manager-admin \ --server-config /etc/disk-manager/server-config.txt \ snapshots export --id <snapshot-or-image-id> > image.rawStructure
[Disk Manager] Add disk-manager-export-snapshot tool— the originalstandalone export tool.
Export snapshots through disk-manager-admin(authored by @EvgeniyKozev) —the export feature squashed from the
codex/export-snapshot-stdoutbranch.Share chunk size constant through dataplane/common— the dataplane packageand the exporter kept two copies of the 4 MiB chunk size that had to stay
equal;
ChunkSizenow lives next to theChunktype.Add Chunk.String to keep chunk formatting cheap— formatting a chunk withfmt reflects over the whole 4 MiB Data slice; test mock argument diffs
format chunk arguments on every mocked ReadChunk call, which made
timing-sensitive exporter tests flaky and slow.
Rework snapshot exporter on top of snapshotSource— chunk map streaming,entry caching and zero chunk detection are delegated to
snapshot.NewSnapshotSource, the same source implementation that dataplanetransfer tasks use. The exporter is an ordered pipeline: the scheduler emits
chunks in stream order (filling chunk map gaps with zero chunks), a bounded
set of readers fills data chunks concurrently, and the writer streams
everything strictly in order. Read-ahead is enforced by the source inflight
queue, so the chunk map does not have to reside in memory. Single-process
export is sufficient, so partitioned export is removed.
Testing
holes, partial last chunk), concurrency (parallel reads, ordered output,
bounded read-ahead), error paths and chunk map validation.
dataplane/commonunit tests pass;disk-manager-adminanddisk-managerbuild.
snapshot_export_test(YDB and S3, checksum verification of theexported stream) runs as a LARGE test.