feat(cloudformation): provision stack set instances as real stacks - #2534
Merged
Merged
Conversation
Stack instances were stubs: CreateStackInstances answered a random OperationId and deployed nothing, DescribeStackInstance always said CURRENT, and the operation APIs reported SUCCEEDED for any id. - Typed stack set store with instances and operations; legacy extras records migrate on load - Create/Update/DeleteStackInstances drive CreateStack/UpdateStack/ DeleteStack in each target account and region, with per-instance parameter overrides, failure tolerance, region order, account gate Lambda, and StopStackSetOperation - UpdateStackSet redeploys all or the targeted instances; the rest go OUTDATED - Operations recorded per target for DescribeStackSetOperation, ListStackSetOperations and ListStackSetOperationResults - SERVICE_MANAGED: OU resolution (nested, filter types, AccountsUrl), trusted access check, CallAs=DELEGATED_ADMIN - ImportStacksToStackSet, CreateStackSet from StackId, ListStackSetAutoDeploymentTargets - DetectStackSetDrift checks instance resources against backing services; ListStackInstanceResourceDrifts reports them - Modeled errors: StackSetNotFound, StackInstanceNotFound, OperationNotFound, OperationInProgress, OperationIdAlreadyExists, StackSetNotEmpty, NameAlreadyExists, StaleRequest, InvalidOperation Closes #2515
- Claim each target RUNNING before it deploys so a stop that lands during a deployment cancels the remaining targets instead of settling STOPPED while they keep deploying - CallAs=DELEGATED_ADMIN only reaches service-managed stack sets - Reject a NextToken past the end of the list
…drift start - Update/DeleteStackInstances reach instances deployed through OUs nested below (or above) the requested OUs - DetectStackSetDrift re-checks for a running operation or reused OperationId before recording its operation
- An update deploys an instance that never got a stack instead of failing it; suspended accounts are re-checked and skipped again - A region listed twice deploys once - TemplateURL, AccountsUrl and StackIdsUrl are read from the caller's account, not the management account a delegated admin acts on - Switching a stack set to SELF_MANAGED drops its AutoDeployment
- Operations deploy in a detached task on the server and the call returns the OperationId at once, so a client that drops its request cannot strand the operation RUNNING - Operations interrupted by a restart settle when state loads - A late outcome from a settled operation no longer writes over instances a newer operation owns - e2e and conformance tests poll operations to completion
…orded A read landing before the background deployment task started saw a RUNNING operation with no results and settled it SUCCEEDED, leaving every target undeployed. Results are now seeded PENDING in the same locked step that records the operation.
…ettles - Instances an operation will deploy are recorded OUTDATED/PENDING when the operation is recorded, so they are listable while it runs; stopping it cancels the ones that never started - Restore folds finished stacks in before settling interrupted operations, so a stack that completed before a restart is not reported as failed
…ary, restart states - AutoDeployment.Enabled=false alone no longer inherits retention - GetTemplateSummary honors CallAs=DELEGATED_ADMIN for stack sets - Instances whose targets never started settle CANCELLED on restart, matching their results
…ops fail - Suspension skips deployments only; DeleteStackInstances still removes an instance in a suspended account, so its stack set can be emptied - Suspension is only considered for service-managed stack sets - An operation interrupted by a restart settles FAILED (or STOPPED) with a reason, regardless of failure tolerance
…hecks The provisioner records a role's physical id as its ARN, but IAM keys roles by name, so drift detection reported every CloudFormation-created role as deleted.
…e cases - Targets skipped, cancelled or gated still take on the requested parameter overrides, so a later redeploy uses them - DetectStackSetDrift reports a reused OperationId as the declared InvalidOperationException - A drift check that could check no instance reports NOT_CHECKED
A stack provisioning in the background (custom resources) returned RUNNING and never counted against the failure tolerance, so every remaining target deployed even after it failed. The operation now waits for such a stack to settle before moving on.
…ERABLE on failed delete - The instance points at its stack while the stack provisions in the background, so a restart during the wait does not orphan it - A stack that cannot be deleted leaves its instance INOPERABLE
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Closes #2515. StackSet instances were stubs:
CreateStackInstancesreturned a random OperationId and deployed nothing,DescribeStackInstancealways answeredCURRENT,ListStackInstanceswas always empty, andDescribeStackSetOperationreportedSUCCEEDEDfor any id.Stack instances are now real stacks, provisioned into their target account and region:
stack_sets.rs) holding the stack set definition, its instances and its operations. Records persisted by older builds in the genericextrasstore migrate on load.CreateStack/UpdateStack/DeleteStackper account x region (StackSet-<name>-<uuid>), with per-instanceParameterOverrides(UsePreviousValue, revert-by-omission),RetainStacks, region order, per-region failure tolerance, theAWSCloudFormationStackSetAccountGateLambda, andStopStackSetOperation.DescribeStackSetOperation. Instances showOUTDATED/PENDINGwhile they wait. Stacks that provision asynchronously settle on the next read; operations cut short by a restart settleFAILEDon load.OUTDATED).AccountFilterType,AccountsUrl, trusted-access check, suspended accounts skipped,CallAs=DELEGATED_ADMINscoped to service-managed stack sets.CreateStackSetfromStackId,ListStackSetAutoDeploymentTargets.ListStackInstanceResourceDriftsreports them. The existence check shared withDetectStackDriftis factored intoresource_exists.StackSetNotFound,StackInstanceNotFound,OperationNotFound,OperationInProgress,OperationIdAlreadyExists,StackSetNotEmpty,NameAlreadyExists,StaleRequest,InvalidOperation,StackNotFound,LimitExceeded.DELETEDand describable by id.Test plan
cargo nextest run -p fakecloud-cloudformation --lib: 408 passed, including 30 new stack set tests (multi-account/region provisioning, overrides, partial updates, tolerance, stop mid-deployment, async settle, restart recovery, OUs/filters/suspension, delegated admin scope and URLs, import, drift, account gate)cloudformation_stack_sets.rs(AWS SDK against the real binary): queues created per region in the default and a member account, parameter override reachesVisibilityTimeout, update/delete/not-empty/not-foundcargo nextest run -p fakecloud-conformance --test cloudformation: stack set lifecycle rewritten as a real sequenceconformance run --services cloudformation: 3434/3434 variants passcargo clippy --workspace --all-targets -- -D warnings,cargo fmtSurface
website/content/docs/services/cloudformation.mdstack sets section rewritten; drift section corrected (it detects deleted resources, it does not always reportIN_SYNC); gotcha added for unnamed resources colliding across same-account stacks.Summary by cubic
Closes #2515. StackSet instances now provision and manage real CloudFormation stacks in each target account and region instead of returning placeholder operation IDs or always reporting
CURRENT. Mutating calls return before deployment completes, so clients must poll the operation status.StackSet behavior
extrasrecords.UpdateStackSetcan redeploy selected instances while leaving othersOUTDATED;DeleteStackInstancescan retain stacks, leaving an undeletable stackINOPERABLE, andDeleteStackSetrejects non-empty sets.AccountsUrl, suspended accounts, trusted access, and delegated administrators.StackId, auto-deployment target listing, and caller-account URL resolution.NOT_CHECKEDwhen no instance can be checked, and rejects reused drift operation IDs withInvalidOperationException.DELETED, and adds conformance, end-to-end, and documentation coverage.Written for commit f2996cb. Summary will update on new commits.