Aro 26724 resource aware e2e#6029
Conversation
…ARO-HCP into ARO-26724-resource-aware-e2e
|
[APPROVALNOTIFIER] This PR is NOT APPROVED This pull-request has been approved by: mgahagan73 The full list of commands accepted by this bot can be found here. DetailsNeeds approval from an approver in each of these files:Approvers can indicate their approval by writing |
There was a problem hiding this comment.
Pull request overview
This PR updates the ARO-HCP E2E test harness to support resource-aware scheduling of managed-identity “containers”, aiming to reduce managed identity starvation by declaring per-test demand and enforcing it both in CI verification and at runtime.
Changes:
- Introduce a new
labels.MIContainers(N)label and apply it across E2E specs to declare managed identity container demand. - Update the E2E test runner to (1) derive MI pool capacity from
LEASED_MSI_CONTAINERS, (2) enforce MIContainers labeling, and (3) schedule tests using openshift-tests-extension resource pools. - Add a
verify-mi-containerspre-merge check and bumpgithub.com/openshift-eng/openshift-tests-extensionto a newer revision.
Reviewed changes
Copilot reviewed 50 out of 53 changed files in this pull request and generated 1 comment.
Show a summary per file
| File | Description |
|---|---|
| tooling/helmtest/internal/config_test.go | Adds unit tests for digest placeholder replacement helpers. |
| tooling/hcpctl/go.mod | Bumps openshift-tests-extension dependency version. |
| tooling/hcpctl/go.sum | Updates dependency checksums for the bump. |
| test/util/labels/labels.go | Replaces MI demand labels with MIContainers(N) label generator. |
| test/go.mod | Bumps openshift-tests-extension dependency version for the test module. |
| test/go.sum | Updates dependency checksums for the bump in the test module. |
| test/e2e/simple_negative_cases.go | Adds MIContainers labeling to an E2E spec. |
| test/e2e/shoebox_logs.go | Adds MIContainers labeling to an E2E spec. |
| test/e2e/oidc_issuer_workload_identity.go | Adds MIContainers labeling to an E2E spec. |
| test/e2e/nodepool_version_upgrade.go | Adds MIContainers labeling to multiple DescribeTable specs. |
| test/e2e/nodepool_update_nodes.go | Adds MIContainers labeling to an E2E spec. |
| test/e2e/nodepool_labels_taints.go | Adds MIContainers labeling to an E2E spec. |
| test/e2e/nodepool_ephemeral_osdisk.go | Adds MIContainers labeling to an E2E spec. |
| test/e2e/nodepool_autoscaling.go | Adds MIContainers labeling to multiple E2E specs. |
| test/e2e/kusto_logs_present.go | Adds MIContainers labeling to an E2E spec. |
| test/e2e/image_registry_policy.go | Labels these engineering tests as MIContainers(0). |
| test/e2e/image_registry_cluster_create.go | Adds MIContainers labeling to an E2E spec. |
| test/e2e/idms_lifecycle.go | Adds MIContainers labeling to an E2E spec. |
| test/e2e/gpu_nodepools_create_delete.go | Adds MIContainers labeling to an E2E spec. |
| test/e2e/fleet_registration.go | Labels this engineering test as MIContainers(0). |
| test/e2e/external_auth_list_and_verify.go | Adds MIContainers labeling to an E2E spec. |
| test/e2e/external_auth_create.go | Adds MIContainers labeling to an E2E spec. |
| test/e2e/exporter_metrics.go | Labels this engineering test as MIContainers(0). |
| test/e2e/control_plane_minor_upgrade.go | Adds MIContainers labeling to a DescribeTable spec. |
| test/e2e/control_plane_automated_z_stream_upgrade.go | Adds MIContainers labeling to a DescribeTable spec. |
| test/e2e/complete_cluster_create_multiversion.go | Adds MIContainers labeling to a DescribeTable spec. |
| test/e2e/clusters_sharing_resgroup.go | Replaces old MI demand label with MIContainers(3). |
| test/e2e/cluster_versions.go | Labels this spec as MIContainers(0). |
| test/e2e/cluster_version_backlevel.go | Adds MIContainers labeling to an E2E spec. |
| test/e2e/cluster_update.go | Adds MIContainers labeling to multiple It specs. |
| test/e2e/cluster_tls_endpoints.go | Adds MIContainers labeling to an E2E spec. |
| test/e2e/cluster_pullsecret.go | Adds MIContainers labeling to an E2E spec. |
| test/e2e/cluster_nsg_subnet_reuse.go | Replaces old MI demand label with MIContainers(2). |
| test/e2e/cluster_fips_mode.go | Adds MIContainers labeling to multiple It specs. |
| test/e2e/cluster_delete_cx_rg.go | Adds MIContainers labeling to an E2E spec. |
| test/e2e/cluster_delayed_role_assignments.go | Adds MIContainers labeling to an E2E spec. |
| test/e2e/cluster_create_v20260630preview.go | Adds MIContainers labeling to an E2E spec. |
| test/e2e/cluster_create_private_kv.go | Adds MIContainers labeling to an E2E spec. |
| test/e2e/cluster_create_nodepool_osdisk.go | Adds MIContainers labeling to an E2E spec. |
| test/e2e/cluster_create_complex_cilium_kv.go | Adds MIContainers labeling to an E2E spec. |
| test/e2e/cluster_create_cni_cilium.go | Adds MIContainers labeling to an E2E spec. |
| test/e2e/cluster_autoscaling.go | Adds MIContainers labeling to an E2E spec. |
| test/e2e/cluster_authorized_cidrs_connectivity.go | Adds MIContainers labeling to an E2E spec. |
| test/e2e/arm64_nodepool.go | Adds MIContainers labeling to an E2E spec. |
| test/e2e/admin_credential_lifecycle.go | Adds MIContainers labeling to an E2E spec. |
| test/e2e/admin_api.go | Adds MIContainers labeling to multiple It specs. |
| test/cmd/aro-hcp-tests/main.go | Implements MIContainers label parsing + resource pool scheduling/enforcement. |
| sessiongate/go.mod | Bumps openshift-tests-extension dependency version. |
| sessiongate/go.sum | Updates dependency checksums for the bump. |
| Makefile | Adds verify-mi-containers to the repo verify target. |
| hack/verify-mi-containers/main.go | Adds a static verifier for MIContainers labeling vs AssignIdentityContainers usage. |
| hack/verify-mi-containers/go.mod | Adds a standalone Go module for the verifier. |
| go.work | Includes the new verifier module in the workspace. |
| if len(violations) > 0 { | ||
| fmt.Println("ERROR: The following test specs have MIContainers label issues.") | ||
| fmt.Println(" Every It() and DescribeTable() must include labels.MIContainers(N)") | ||
| fmt.Println(" where N matches the count passed to AssignIdentityContainers.") | ||
| fmt.Println() | ||
| fmt.Println(" Examples:") | ||
| fmt.Println(` It("should create cluster", labels.MIContainers(1), func(ctx context.Context) {`) | ||
| fmt.Println(` DescribeTable("upgrades", labels.MIContainers(1), func(ctx context.Context, ...) {`) | ||
| fmt.Println() | ||
| for _, v := range violations { | ||
| fmt.Println(" " + v) | ||
| } | ||
| fmt.Println() | ||
| fmt.Println("Fix the labels listed above. Use labels.MIContainers(0) for tests") | ||
| fmt.Println("that do not call AssignIdentityContainers.") | ||
| os.Exit(1) | ||
| } |
|
@mgahagan73: The following tests failed, say
Full PR test history. Your PR dashboard. DetailsInstructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository. I understand the commands that are listed here. |
https://redhat.atlassian.net/browse/ARO-26724
What
Updates to latest OTE to get resource pools support, Adds resource aware scheduler to ARO HCP e2e tests using managed identities, updates test labels, adds pre-merge check to confirm tests are properly labeled.
Why
Long term solution to managed identity starvation test issues.
Testing
Dev and INT CI testing for now during this Draft phase.
Special notes for your reviewer
PR Checklist
If E2E tests are included:
demonstrate that the test is able to detect a defect/error and fail with
proper error message and logs which communicates nature of the problem.