Skip to content

CLI deployment recovery hardening - #423

Merged
mkultraWasHere merged 4 commits into
mainfrom
dg/azure-cli-fix
Aug 11, 2026
Merged

CLI deployment recovery hardening#423
mkultraWasHere merged 4 commits into
mainfrom
dg/azure-cli-fix

Conversation

@mkultraWasHere

@mkultraWasHere mkultraWasHere commented Aug 10, 2026

Copy link
Copy Markdown
Contributor

The deployment workflow now fails earlier and resumes more precisely after interrupted runs.

Added

  • dreadgoad up --from provision --from-playbook <playbook> resumes provisioning at the playbook that failed.
  • Terragrunt state-lock failures include a safely quoted force-unlock recovery command.

Changed

  • CLI documentation now covers precise provisioning resumes and explicit zero-retry behavior.

Breaking

  • --max-retries 0 now runs once without retrying; omit the flag to retain configured retry defaults.
  • Variant generation now requires an absent target, and provisioning rejects unmarked targets; move or remove an existing target before regenerating.
  • Malformed or orphaned lab-config overlays now stop infrastructure materialization; fix or remove the overlay before retrying.

Fixed

  • dreadgoad up resume hints preserve the failed playbook suffix, host limit, infra selectors, and explicit retry overrides.
  • Variant generation no longer certifies targets after file-write or validation failures, or mixes stale files from an earlier randomized run.
  • Lab-config materialization creates missing destination directories and reports actionable errors.

Notes

  • AWS-only preflight and reconnect behavior, inventory permissions, Proxmox secret transport, and full cancellation threading remain follow-up audit items.

Preserve configured retry behavior unless operators explicitly override it, and make interrupted provisioning easier to resume. Surface actionable infrastructure and state-lock errors while preventing incomplete variants from being treated as ready.
@dreadnode-renovate-bot dreadnode-renovate-bot Bot added area/ad-labs Changes made to AD lab definitions area/docs Changes made to documentation labels Aug 10, 2026
@mkultraWasHere
mkultraWasHere requested a lite review from Copilot and removed request for Copilot August 10, 2026 23:14

Copilot AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Hardens CLI deployment recovery and failure handling.

Changes:

  • Adds playbook-specific provisioning resume hints and explicit zero-retry semantics.
  • Validates variant completion and lab-config materialization.
  • Adds safely quoted Terragrunt state-lock recovery guidance.

Reviewed changes

Copilot reviewed 20 out of 20 changed files in this pull request and generated 2 comments.

Show a summary per file
File Description
docs/mkdocs/docs/provisioning.md Documents zero retries.
docs/mkdocs/docs/cli-reference.md Documents resume and retry flags.
docs/cli.md Documents variant markers.
cli/internal/variant/generator.go Adds completion certification and write failures.
cli/internal/variant/generator_test.go Tests marker behavior.
cli/internal/terragrunt/runner.go Adds lock recovery hints.
cli/internal/terragrunt/runner_test.go Tests lock parsing and quoting.
cli/internal/config/config.go Classifies missing and orphaned configs.
cli/internal/config/config_test.go Tests missing-config classification.
cli/internal/ansible/retry.go Distinguishes explicit retry overrides.
cli/internal/ansible/retry_test.go Tests zero-retry execution.
cli/cmd/up.go Adds playbook-level resume support.
cli/cmd/up_test.go Tests resume and flag forwarding.
cli/cmd/provision.go Adds structured failures and marker checks.
cli/cmd/provision_test.go Tests retries, failures, and variants.
cli/cmd/lab_reset.go Applies new retry semantics.
cli/cmd/infra_cmd.go Hardens config materialization.
cli/cmd/infra_cmd_test.go Tests materialization failures.
cli/cmd/extension.go Applies retry overrides to extensions.
ad/GOAD-variant-1/.dreadgoad-variant-complete Certifies the existing variant.

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

Comment thread cli/internal/variant/generator.go Outdated
Comment thread cli/cmd/up.go Outdated
Ensure variant completion markers certify every required validation and output. Preserve custom playbook selections in generated provisioning resume commands.

Copilot AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Copilot reviewed 20 out of 20 changed files in this pull request and generated 1 comment.

Suppressed comments (1)

cli/internal/variant/generator.go:280

  • Removing only the marker does not make an existing target safe to certify. variant generate can run against an existing directory, and each run creates new random mappings; when a transformed filename changes, the old generated file is never removed by copyAndTransform. The run can then validate the new files and write a completion marker over a directory containing stale outputs from the previous variant. Generate into a fresh staging directory and replace the target only after validation, or reject/clean an existing target before removing its marker.
	markerPath := filepath.Join(g.TargetPath, CompletionMarkerName)
	if err := os.Remove(markerPath); err != nil && !errors.Is(err, os.ErrNotExist) {
		return fmt.Errorf("remove stale completion marker: %w", err)
	}

Comment thread cli/cmd/up.go Outdated
Keep resume hints scoped to the original execution options and prevent variant regeneration from mixing randomized outputs in an existing target.
@mkultraWasHere

Copy link
Copy Markdown
Contributor Author

Fixed in 8e56060 — variant generation now exclusively creates a fresh target and rejects any existing target without modifying it, preventing stale randomized outputs from being mixed and recertified.

Use the switch form required by go-critic and record pflag as a direct test dependency after go mod tidy.
@mkultraWasHere
mkultraWasHere added this pull request to the merge queue Aug 11, 2026
Merged via the queue into main with commit 00cfc0b Aug 11, 2026
9 checks passed
@mkultraWasHere
mkultraWasHere deleted the dg/azure-cli-fix branch August 11, 2026 00:36
mkultraWasHere added a commit that referenced this pull request Aug 11, 2026
Brings in 32 commits, including #423 CLI deployment recovery hardening,
which independently implemented several findings from the stand-up audit:
--from-playbook resume, pointer-based retry overrides so --max-retries 0
means zero, a structural flag-forwarding test for the provision command,
and a doctor failure message that no longer advertises --skip-doctor.

Conflicts and how they were settled:

up.go -- main rewrote the resume path around provisionFailure and
upResumeOptions; kept all of it, and re-applied the Azure module fix
(d4d23a5) on top, since main's runUpInfraApply still omitted
--with-bastion/--with-controller and would have reintroduced the break.

up_test.go -- both sides added the file. No function names collide, so
this is the union: main's 11 provision/resume tests plus the 6 infra and
Azure-module tests, on a merged import block.

provider.go, aws/ec2.go, aws/provider.go, azure/provider.go -- both
sides added fields to Instance. Kept Account/Group and main's Tags. main
extracted discoverInstances, so the Account population moved into that
helper rather than staying in the old inline body.

.gitignore -- kept the console build artifacts and the ADCS zip entries;
main tracks the template sources but not the generated archives.

#418 gave Kali an AWS path, so --with-kali is no longer Azure-only and
the up flag help drops that qualifier.

Co-Authored-By: Claude <noreply@anthropic.com>
mkultraWasHere added a commit that referenced this pull request Aug 11, 2026
Account comes from the enclosing Reservation rather than the instance, so
it is the one field a refactor of this loop can silently drop -- nothing
else reads OwnerId. The existing test builds a reservation with no
OwnerId and asserts only Name and Tags, so removing the assignment failed
nothing.

That nearly mattered: #423 extracted the loop into
appendDiscoveredInstances, and merging it meant relocating the assignment
by hand. Account feeds `dreadgoad status` and from there the web app's
cloud-account display, where an empty value reads as a discovery failure
rather than a missing field.

Co-Authored-By: Claude <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

area/ad-labs Changes made to AD lab definitions area/docs Changes made to documentation

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants