Support durable directories and combined restore for gVisor - #1379
Support durable directories and combined restore for gVisor#1379nybidari wants to merge 1 commit into
Conversation
There was a problem hiding this comment.
This is being deleted in #1376, please use the substrate proto ActorTemplate instead.
substrate/pkg/proto/ateapipb/ateapi.proto
Line 707 in 07ebc56
There was a problem hiding this comment.
Thanks, updated the changes.
f1de00d to
f1b7546
Compare
| // "-debug", | ||
| // "-debug-log", ateompath.RunscDebugLogDir(r.actorUID, containerName) + "/", | ||
| // "-debug-to-user-log", | ||
| // "-log-packets", | ||
| // "-strace", |
There was a problem hiding this comment.
nit: we should write these to the debug mode for the ateom, that can be a follow-up but generally prefer to do that over leave commented lines :-)
There was a problem hiding this comment.
Removed the commented debug flags now, will have a follow up PR to wire these flags to ateom's debug mode.
| } | ||
| case ateompb.SnapshotScope_SNAPSHOT_SCOPE_FULL: | ||
| // Checkpoint pause container (root of the sandbox) | ||
| if err := rcmd.cmdCheckpoint(ctx, "pause", checkpointPath); err != nil { |
There was a problem hiding this comment.
Should this be pause -> tar -> checkpoint instead of checkpoint -> tar? Or IOW is this the right failure handling for just resuming if the disk is transiently full or something?
| tarErr := tarDurableVolumes(ctx, ateompath.DurableDirVolumeMountsDir(req.GetActorUid()), checkpointPath) | ||
| if err := rcmd.cmdResume(ctx, "pause"); err != nil { | ||
| return nil, fmt.Errorf("while resuming pause container: %w", err) | ||
| } |
There was a problem hiding this comment.
claude surfaced:
| tarErr := tarDurableVolumes(ctx, ateompath.DurableDirVolumeMountsDir(req.GetActorUid()), checkpointPath) | |
| if err := rcmd.cmdResume(ctx, "pause"); err != nil { | |
| return nil, fmt.Errorf("while resuming pause container: %w", err) | |
| } | |
| tarErr := tarDurableVolumes(ctx, ateompath.DurableDirVolumeMountsDir(req.GetActorUid()), checkpointPath) | |
| // Undoing our own pause must not depend on the caller's context: | |
| // tarutil does not check ctx, so a deadline expiring mid-tar would | |
| // fail the resume instantly and leave the sandbox paused forever. | |
| resumeCtx, cancelResume := context.WithTimeout(context.WithoutCancel(ctx), resumeTimeout) | |
| defer cancelResume() | |
| if err := rcmd.cmdResume(resumeCtx, "pause"); err != nil { | |
| return nil, fmt.Errorf("while resuming pause container: %w", err) | |
| } |
I think that's right, or some variation on it, the alternative is to make tarutil context aware, but that seems like a bigger lift for this PR.
There was a problem hiding this comment.
Thanks, updated the changes.
- Use direct host bind mounts for durable directories in gVisor containers instead of internal overlay filestores. - Move tarutil into internal/tarutil so it can be shared across ateom packages. - Archive durable directory bind mounts into durable-dir.tar on full and data checkpoints (replacing runsc fscheckpoint). - On restore, unpack durable-dir.tar into the host bind mount before starting/restoring containers. - Support SNAPSHOT_SCOPE_DATA_ON_GOLDEN in ateom-gvisor, combining golden guest state with latest actor durable data. - Allow atelet to narrow full gVisor captures to durable-dir.tar for data snapshots. - Enable combined restore demo tests for gVisor with proto ActorTemplate.
3 participants
Fixes #451