CLI deployment recovery hardening - #423
Merged
Merged
Conversation
Preserve configured retry behavior unless operators explicitly override it, and make interrupted provisioning easier to resume. Surface actionable infrastructure and state-lock errors while preventing incomplete variants from being treated as ready.
mkultraWasHere
requested
a lite review from Copilot
and removed request for
Copilot
August 10, 2026 23:14
There was a problem hiding this comment.
Pull request overview
Hardens CLI deployment recovery and failure handling.
Changes:
- Adds playbook-specific provisioning resume hints and explicit zero-retry semantics.
- Validates variant completion and lab-config materialization.
- Adds safely quoted Terragrunt state-lock recovery guidance.
Reviewed changes
Copilot reviewed 20 out of 20 changed files in this pull request and generated 2 comments.
Show a summary per file
| File | Description |
|---|---|
docs/mkdocs/docs/provisioning.md |
Documents zero retries. |
docs/mkdocs/docs/cli-reference.md |
Documents resume and retry flags. |
docs/cli.md |
Documents variant markers. |
cli/internal/variant/generator.go |
Adds completion certification and write failures. |
cli/internal/variant/generator_test.go |
Tests marker behavior. |
cli/internal/terragrunt/runner.go |
Adds lock recovery hints. |
cli/internal/terragrunt/runner_test.go |
Tests lock parsing and quoting. |
cli/internal/config/config.go |
Classifies missing and orphaned configs. |
cli/internal/config/config_test.go |
Tests missing-config classification. |
cli/internal/ansible/retry.go |
Distinguishes explicit retry overrides. |
cli/internal/ansible/retry_test.go |
Tests zero-retry execution. |
cli/cmd/up.go |
Adds playbook-level resume support. |
cli/cmd/up_test.go |
Tests resume and flag forwarding. |
cli/cmd/provision.go |
Adds structured failures and marker checks. |
cli/cmd/provision_test.go |
Tests retries, failures, and variants. |
cli/cmd/lab_reset.go |
Applies new retry semantics. |
cli/cmd/infra_cmd.go |
Hardens config materialization. |
cli/cmd/infra_cmd_test.go |
Tests materialization failures. |
cli/cmd/extension.go |
Applies retry overrides to extensions. |
ad/GOAD-variant-1/.dreadgoad-variant-complete |
Certifies the existing variant. |
💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.
Ensure variant completion markers certify every required validation and output. Preserve custom playbook selections in generated provisioning resume commands.
There was a problem hiding this comment.
Pull request overview
Copilot reviewed 20 out of 20 changed files in this pull request and generated 1 comment.
Suppressed comments (1)
cli/internal/variant/generator.go:280
- Removing only the marker does not make an existing target safe to certify.
variant generatecan run against an existing directory, and each run creates new random mappings; when a transformed filename changes, the old generated file is never removed bycopyAndTransform. The run can then validate the new files and write a completion marker over a directory containing stale outputs from the previous variant. Generate into a fresh staging directory and replace the target only after validation, or reject/clean an existing target before removing its marker.
markerPath := filepath.Join(g.TargetPath, CompletionMarkerName)
if err := os.Remove(markerPath); err != nil && !errors.Is(err, os.ErrNotExist) {
return fmt.Errorf("remove stale completion marker: %w", err)
}
Keep resume hints scoped to the original execution options and prevent variant regeneration from mixing randomized outputs in an existing target.
Contributor
Author
|
Fixed in 8e56060 — variant generation now exclusively creates a fresh target and rejects any existing target without modifying it, preventing stale randomized outputs from being mixed and recertified. |
Use the switch form required by go-critic and record pflag as a direct test dependency after go mod tidy.
mkultraWasHere
added a commit
that referenced
this pull request
Aug 11, 2026
Brings in 32 commits, including #423 CLI deployment recovery hardening, which independently implemented several findings from the stand-up audit: --from-playbook resume, pointer-based retry overrides so --max-retries 0 means zero, a structural flag-forwarding test for the provision command, and a doctor failure message that no longer advertises --skip-doctor. Conflicts and how they were settled: up.go -- main rewrote the resume path around provisionFailure and upResumeOptions; kept all of it, and re-applied the Azure module fix (d4d23a5) on top, since main's runUpInfraApply still omitted --with-bastion/--with-controller and would have reintroduced the break. up_test.go -- both sides added the file. No function names collide, so this is the union: main's 11 provision/resume tests plus the 6 infra and Azure-module tests, on a merged import block. provider.go, aws/ec2.go, aws/provider.go, azure/provider.go -- both sides added fields to Instance. Kept Account/Group and main's Tags. main extracted discoverInstances, so the Account population moved into that helper rather than staying in the old inline body. .gitignore -- kept the console build artifacts and the ADCS zip entries; main tracks the template sources but not the generated archives. #418 gave Kali an AWS path, so --with-kali is no longer Azure-only and the up flag help drops that qualifier. Co-Authored-By: Claude <noreply@anthropic.com>
mkultraWasHere
added a commit
that referenced
this pull request
Aug 11, 2026
Account comes from the enclosing Reservation rather than the instance, so it is the one field a refactor of this loop can silently drop -- nothing else reads OwnerId. The existing test builds a reservation with no OwnerId and asserts only Name and Tags, so removing the assignment failed nothing. That nearly mattered: #423 extracted the loop into appendDiscoveredInstances, and merging it meant relocating the assignment by hand. Account feeds `dreadgoad status` and from there the web app's cloud-account display, where an empty value reads as a discovery failure rather than a missing field. Co-Authored-By: Claude <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
The deployment workflow now fails earlier and resumes more precisely after interrupted runs.
Added
dreadgoad up --from provision --from-playbook <playbook>resumes provisioning at the playbook that failed.force-unlockrecovery command.Changed
Breaking
--max-retries 0now runs once without retrying; omit the flag to retain configured retry defaults.Fixed
dreadgoad upresume hints preserve the failed playbook suffix, host limit, infra selectors, and explicit retry overrides.Notes