feat(templatize): add multi-stamp parallel execution for entrypoint runs#5697
Merged
Merged
Conversation
Collaborator
Author
|
/hold |
Contributor
There was a problem hiding this comment.
Pull request overview
This PR updates tooling/templatize to support running stamped service-group graphs concurrently within a single entrypoint invocation, while isolating per-stamp execution state (outputs/config/pipelines) to prevent cross-stamp collisions.
Changes:
- Introduces stamp-layered step outputs via
ExecutionState.RecordOutputandExecutionState.GetOutputs(stamp)and updates step runners (ARM/ARMStack/Helm/Shell) to resolve inputs against the correct stamp view. - Switches entrypoint graph generation to
graph.ForStampedEntrypoint(...)and adds astampConfigExecutorwrapper to apply per-stamp configuration during execution. - Adds multi-stamp unit tests and updates CLI options/helpers + Makefile wiring to drive stamp-count-based runs/cleanup.
Reviewed changes
Copilot reviewed 15 out of 16 changed files in this pull request and generated 1 comment.
Show a summary per file
| File | Description |
|---|---|
| tooling/templatize/pkg/pipeline/shell.go | Shell step input resolution now reads outputs via GetOutputs(id.Stamp) for stamp isolation. |
| tooling/templatize/pkg/pipeline/run.go | Core execution updated for stamped graphs, layered outputs, per-stamp config executor, and stamp-aware graph lookup. |
| tooling/templatize/pkg/pipeline/run_test.go | Adds unit tests covering multi-stamp execution, output isolation, subscription resolution, and error propagation. |
| tooling/templatize/pkg/pipeline/helm.go | Helm runner now resolves replacements/inputs using stamp-scoped outputs. |
| tooling/templatize/pkg/pipeline/arm.go | ARM runner now passes stamp context through and resolves inputs using stamp-scoped outputs. |
| tooling/templatize/pkg/pipeline/arm_stack.go | ARMStack input resolution switched to GetOutputs(id.Stamp); stack name logic remains stamp-influenced via options.Stamp. |
| tooling/templatize/internal/well_formed_test.go | Updates well-formed validation iteration to match new stamped step keying in the graph. |
| tooling/templatize/go.mod | Bumps ARO-Tools config / pipelines versions needed for stamped graph support. |
| tooling/templatize/go.sum | Updates checksums for new ARO-Tools dependency versions. |
| tooling/templatize/cmd/rolloutoptions.go | Persists ConfigResolver in completed rollout options to enable per-stamp config resolution. |
| tooling/templatize/cmd/entrypoint/run/options.go | Entry-point run now precomputes stamped graph inputs (stamp configs/pipelines) and passes them into pipeline execution. |
| tooling/templatize/cmd/entrypoint/entrypointutils/stamps.go | New helpers for stamp list derivation, per-stamp config resolution, and stamped graph assembly. |
| tooling/templatize/cmd/entrypoint/entrypointutils/options.go | Adds --stamp-count-config-ref and updates regional RG precheck extraction to span all stamp configs. |
| tooling/templatize/cmd/entrypoint/cleanup/options.go | Cleanup now builds stamped graph + uses all stamp configs to discover RGs for deletion. |
| Makefile | Updates local-run and cleanup invocations to use --stamp-count-config-ref=mgmt.stamps.count. |
| dev-infrastructure/templates/mgmt-cluster.bicep | Makes module deployment name stamp-unique via uniqueString(resourceGroup().name) to avoid cross-stamp ARM conflicts. |
geoberle
force-pushed
the
templatize-stamping
branch
from
7d0be4a to
407a03f
Compare
geoberle
force-pushed
the
templatize-stamping
branch
from
407a03f to
e03c3df
Compare
geoberle
force-pushed
the
templatize-stamping
branch
from
e03c3df to
6c2440e
Compare
geoberle
force-pushed
the
templatize-stamping
branch
from
6c2440e to
b519c4b
Compare
geoberle
force-pushed
the
templatize-stamping
branch
from
b519c4b to
a5c75d9
Compare
geoberle
force-pushed
the
templatize-stamping
branch
from
a5c75d9 to
7660073
Compare
geoberle
force-pushed
the
templatize-stamping
branch
from
7660073 to
dd52a09
Compare
geoberle
force-pushed
the
templatize-stamping
branch
from
dd52a09 to
9579113
Compare
geoberle
force-pushed
the
templatize-stamping
branch
from
03db15d to
1497e1d
Compare
geoberle
force-pushed
the
templatize-stamping
branch
from
1497e1d to
1afa8c0
Compare
geoberle
force-pushed
the
templatize-stamping
branch
from
1afa8c0 to
a414923
Compare
geoberle
force-pushed
the
templatize-stamping
branch
from
a414923 to
44171c4
Compare
geoberle
force-pushed
the
templatize-stamping
branch
from
44171c4 to
426131f
Compare
geoberle
force-pushed
the
templatize-stamping
branch
from
426131f to
23b0630
Compare
ARO HCP deploys management clusters as "stamps" within Azure regions. Previously templatize handled one stamp per invocation. This change makes the execution engine natively iterate stamps in parallel. - ExecutionState gains per-stamp output isolation (StampOutputs). GetOutputs(stamp) returns a merged snapshot (stamp layer over base). All step runners (ARM, ARM stack, Helm, shell) use stamp-aware input resolution. - New --stamp-count-config-ref flag reads stamp count from config (e.g. mgmt.stamps.count). Works with both --entrypoint and --service-group modes. BuildStampList/ResolveStampConfigs in entrypointutils/stamps.go serve run and cleanup subcommands. - RunStampedPipeline added for service-group mode. Uses graph.ForStampedPipeline (new in ARO-Tools) to build a stamp-aware execution graph for a single service, then wraps the executor with stampConfigExecutor for per-stamp config injection. - Cleanup uses the same stamped graph constructors (ForStampedPipeline, ForStampedEntrypoints) to discover resource groups, matching the execution path. Iterates graph ResourceGroups in deterministic order.
Contributor
|
/lgtm |
|
[APPROVALNOTIFIER] This PR is APPROVED This pull-request has been approved by: geoberle, stevekuznetsov The full list of commands accepted by this bot can be found here. The pull request process is described here DetailsNeeds approval from an approver in each of these files:
Approvers can indicate their approval by writing |
Collaborator
Author
|
/test image-updater-images |
Collaborator
Author
|
/test images |
Collaborator
Author
|
/retest |
Collaborator
Author
|
/hold cancel |
Contributor
Collaborator
Author
|
this is infra setup only. failed e2e are irrelevant /override ci/prow/e2e-parallel |
|
@geoberle: Overrode contexts on behalf of geoberle: ci/prow/e2e-parallel DetailsIn response to this:
Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository. |
Collaborator
Author
|
/test images |
Collaborator
Author
|
/override ci/prow/e2e-parallel |
|
@geoberle: Overrode contexts on behalf of geoberle: ci/prow/e2e-parallel DetailsIn response to this:
Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
ARO HCP deploys management clusters as "stamps" within Azure regions. Previously templatize handled one stamp per invocation. This change makes the execution engine natively iterate stamps in parallel.
Execution state isolation: ExecutionState gains per-stamp output isolation (StampOutputs). GetOutputs(stamp) returns a merged snapshot (stamp layer over base). All step runners (ARM, ARM stack, Helm, shell) use stamp-aware input resolution.
Stamp resolution: New
--stamp-count-config-refflag reads stamp count from config (e.g.mgmt.stamps.count). Works with both--entrypointand--service-groupmodes.BuildStampList/ResolveStampConfigsinentrypointutils/stamps.goserve run and cleanup subcommands.RunStampedPipeline: Added for service-group mode. Uses
graph.ForStampedPipeline(new in ARO-Tools) to build a stamp-aware execution graph for a single service, then wraps the executor withstampConfigExecutorfor per-stamp config injection.Cleanup: Uses the same stamped graph constructors (
ForStampedPipeline,ForStampedEntrypoints) to discover resource groups, matching the execution path. Iterates graph ResourceGroups in deterministic order.Why
Management clusters are deployed as stamps within a region. The tooling needs to handle multiple stamps per region natively — deploying each stamp's resources with stamp-specific configuration while maintaining correct output isolation between stamps.
Testing
ExecutionStatestamp output isolation,GetOutputsmerging,stampConfigExecutorper-stamp config swappingForStampedPipeline(unstamped service, stamped multi-stamp expansion, children stripped, step ordering preserved)PR Checklist