fix(e2e): restrict VM SKU fallbacks to RP-allowlisted instance types#6015
Open
roivaz wants to merge 1 commit into
Open
fix(e2e): restrict VM SKU fallbacks to RP-allowlisted instance types#6015roivaz wants to merge 1 commit into
roivaz wants to merge 1 commit into
Conversation
The e2e VM-size selectors picked SKUs based only on Azure availability and
capabilities, with no awareness of the ARO-HCP RP instance-type allowlist
(cluster-service cloud-resource-constraints-config). When a region/subscription
made the preferred candidates unavailable, selection could fall through to a
SKU that Azure advertises but the RP rejects, failing node pool creation with:
RESPONSE 400 InvalidRequestContent:
Machine type 'Standard_D8lds_v6' is not supported.
Only the plain "s" and AMD "as" D-series variants are enabled in the RP
allowlist; the local-disk "ds"/"lds"/"ads" variants are not. This restricts
the Preferred lists and NamePattern fallback regexes of the worker selectors
to allowlisted families:
- default-worker: D8ds_v5/D8lds_v6 -> D8s_v5/D8s_v6 (D8as_v5 kept)
- small-worker: D4ds_v5/D4lds_v6 -> D4s_v5/D4s_v6 (D4as_v5 kept)
- ephemeral-osdisk-worker: Dsv3 is the only allowlisted family with
cache-based ephemeral OS disk support, so candidates are the Dsv3 family
(D8s_v3 preferred, D16s_v3/D32s_v3 as availability fallback)
The NamePattern regexes are tightened from `[^p]*s_v[3456]` to
`(?:a)?s_v[3456]` so the deterministic fallback path also cannot pick a
non-allowlisted local-disk SKU. Jumpbox is unchanged (plain Azure VM, not
subject to RP instance-type constraints).
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
|
[APPROVALNOTIFIER] This PR is APPROVED This pull-request has been approved by: roivaz The full list of commands accepted by this bot can be found here. The pull request process is described here DetailsNeeds approval from an approver in each of these files:
Approvers can indicate their approval by writing |
Contributor
There was a problem hiding this comment.
Pull request overview
This PR updates the E2E VM SKU selection logic so worker node pool VM sizes (both preferred choices and deterministic regex-based fallbacks) stay within ARO-HCP RP allowlisted instance types, preventing node pool creation failures when Azure-advertised-but-RP-rejected SKUs are selected.
Changes:
- Tighten default and small worker SKU preferences to RP-allowlisted D-series variants.
- Narrow fallback regex patterns to prevent selecting non-allowlisted local-disk variants (
ds/lds/ads) and ARM64pvariants. - Restrict ephemeral OS disk worker selection to allowlisted Dsv3 sizes that support ephemeral OS disks via cache.
| Preferred: []string{DefaultWorkerVMSize, "Standard_D8ds_v5", "Standard_D8as_v5", "Standard_D8lds_v6"}, | ||
| NamePattern: regexp.MustCompile(`^Standard_D[4-8][^p]*s_v[3456]$`), | ||
| Preferred: []string{DefaultWorkerVMSize, "Standard_D8s_v5", "Standard_D8as_v5", "Standard_D8s_v6"}, | ||
| NamePattern: regexp.MustCompile(`^Standard_D[4-8](?:a)?s_v[3456]$`), |
| Preferred: []string{SmallWorkerVMSize, "Standard_D4ds_v5", "Standard_D4as_v5", "Standard_D4lds_v6"}, | ||
| NamePattern: regexp.MustCompile(`^Standard_D[2-4][^p]*s_v[3456]$`), | ||
| Preferred: []string{SmallWorkerVMSize, "Standard_D4s_v5", "Standard_D4as_v5", "Standard_D4s_v6"}, | ||
| NamePattern: regexp.MustCompile(`^Standard_D[2-4](?:a)?s_v[3456]$`), |
Comment on lines
+425
to
+426
| Preferred: []string{DefaultWorkerVMSize, "Standard_D8s_v5", "Standard_D8as_v5", "Standard_D8s_v6"}, | ||
| NamePattern: regexp.MustCompile(`^Standard_D[4-8](?:a)?s_v[3456]$`), |
|
@roivaz: The following test failed, say
Full PR test history. Your PR dashboard. DetailsInstructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository. I understand the commands that are listed here. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Why
The e2e VM-size selectors in
test/util/framework/vm_sku_selection.gochoose SKUs based only on Azure availability and capabilities (vCPUs, ephemeral OS disk support, etc.), with no awareness of the ARO-HCP RP's instance-type allowlist (cluster-servicecloud-resource-constraints-config).When a region/subscription makes the preferred candidates unavailable (as happened in a PROD
uksouthrehearsal), selection falls through to a SKU that Azure advertises but the RP rejects, failing node pool creation:Only the plain
sand AMDasD-series variants are enabled in the RP allowlist; the local-diskds/lds/adsvariants are not.What
Restrict the
Preferredlists andNamePatternfallback regexes of the worker selectors to allowlisted families:default-workerD8ds_v5,D8lds_v6D8s_v5,D8s_v6(D8as_v5kept)small-workerD4ds_v5,D4lds_v6D4s_v5,D4s_v6(D4as_v5kept)ephemeral-osdisk-workerD8ds_v5,D8ads_v5,D8lds_v6D16s_v3,D32s_v3(Dsv3is the only allowlisted family with cache-based ephemeral OS disk)The
NamePatternregexes are tightened from[^p]*s_v[3456]to(?:a)?s_v[3456]so the deterministic fallback path also cannot pick a non-allowlisted local-disk SKU. Historical primary preferences (Standard_D8s_v3/Standard_D4s_v3) are unchanged, so behaviour only diverges when those are genuinely unusable.Jumpbox is left unchanged — it is a plain Azure VM and not subject to RP instance-type constraints.
Testing
cd test && go test ./util/framework/...→ okds/lds/ads/p) and accepts the enableds/asfamilies.