Skip to content

Aro 26724 resource aware e2e#6029

Draft
mgahagan73 wants to merge 3 commits into
Azure:mainfrom
mgahagan73:ARO-26724-resource-aware-e2e
Draft

Aro 26724 resource aware e2e#6029
mgahagan73 wants to merge 3 commits into
Azure:mainfrom
mgahagan73:ARO-26724-resource-aware-e2e

Conversation

@mgahagan73

Copy link
Copy Markdown
Collaborator

https://redhat.atlassian.net/browse/ARO-26724

What

Updates to latest OTE to get resource pools support, Adds resource aware scheduler to ARO HCP e2e tests using managed identities, updates test labels, adds pre-merge check to confirm tests are properly labeled.

Why

Long term solution to managed identity starvation test issues.

Testing

Dev and INT CI testing for now during this Draft phase.

Special notes for your reviewer

PR Checklist

  • PR is scoped to a single task (no mixed concerns)
  • Title follows Conventional Commits format
  • Summary explains the "Why" behind the change
  • Linked to relevant ticket/issue
  • Screenshots included (if graph/UI/metrics changes)
  • Self-reviewed the diff
  • CI/CD checks are passing (ignore Tide)
  • Draft PR used for WIP (if applicable)
  • Commit history is clean (rebased/squashed)
  • Tricky code blocks are commented
  • Specific reviewers tagged
  • All comment threads resolved before merge

If E2E tests are included:

  • E2E tests follow Principles of Good E2E Test Case Design
  • If new E2E use case is covered (via a new test or new check/verifier),
    demonstrate that the test is able to detect a defect/error and fail with
    proper error message and logs which communicates nature of the problem.

Copilot AI review requested due to automatic review settings July 11, 2026 02:52
@openshift-ci

openshift-ci Bot commented Jul 11, 2026

Copy link
Copy Markdown

[APPROVALNOTIFIER] This PR is NOT APPROVED

This pull-request has been approved by: mgahagan73
Once this PR has been reviewed and has the lgtm label, please assign roivaz for approval. For more information see the Code Review Process.

The full list of commands accepted by this bot can be found here.

Details Needs approval from an approver in each of these files:

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

This PR updates the ARO-HCP E2E test harness to support resource-aware scheduling of managed-identity “containers”, aiming to reduce managed identity starvation by declaring per-test demand and enforcing it both in CI verification and at runtime.

Changes:

  • Introduce a new labels.MIContainers(N) label and apply it across E2E specs to declare managed identity container demand.
  • Update the E2E test runner to (1) derive MI pool capacity from LEASED_MSI_CONTAINERS, (2) enforce MIContainers labeling, and (3) schedule tests using openshift-tests-extension resource pools.
  • Add a verify-mi-containers pre-merge check and bump github.com/openshift-eng/openshift-tests-extension to a newer revision.

Reviewed changes

Copilot reviewed 50 out of 53 changed files in this pull request and generated 1 comment.

Show a summary per file
File Description
tooling/helmtest/internal/config_test.go Adds unit tests for digest placeholder replacement helpers.
tooling/hcpctl/go.mod Bumps openshift-tests-extension dependency version.
tooling/hcpctl/go.sum Updates dependency checksums for the bump.
test/util/labels/labels.go Replaces MI demand labels with MIContainers(N) label generator.
test/go.mod Bumps openshift-tests-extension dependency version for the test module.
test/go.sum Updates dependency checksums for the bump in the test module.
test/e2e/simple_negative_cases.go Adds MIContainers labeling to an E2E spec.
test/e2e/shoebox_logs.go Adds MIContainers labeling to an E2E spec.
test/e2e/oidc_issuer_workload_identity.go Adds MIContainers labeling to an E2E spec.
test/e2e/nodepool_version_upgrade.go Adds MIContainers labeling to multiple DescribeTable specs.
test/e2e/nodepool_update_nodes.go Adds MIContainers labeling to an E2E spec.
test/e2e/nodepool_labels_taints.go Adds MIContainers labeling to an E2E spec.
test/e2e/nodepool_ephemeral_osdisk.go Adds MIContainers labeling to an E2E spec.
test/e2e/nodepool_autoscaling.go Adds MIContainers labeling to multiple E2E specs.
test/e2e/kusto_logs_present.go Adds MIContainers labeling to an E2E spec.
test/e2e/image_registry_policy.go Labels these engineering tests as MIContainers(0).
test/e2e/image_registry_cluster_create.go Adds MIContainers labeling to an E2E spec.
test/e2e/idms_lifecycle.go Adds MIContainers labeling to an E2E spec.
test/e2e/gpu_nodepools_create_delete.go Adds MIContainers labeling to an E2E spec.
test/e2e/fleet_registration.go Labels this engineering test as MIContainers(0).
test/e2e/external_auth_list_and_verify.go Adds MIContainers labeling to an E2E spec.
test/e2e/external_auth_create.go Adds MIContainers labeling to an E2E spec.
test/e2e/exporter_metrics.go Labels this engineering test as MIContainers(0).
test/e2e/control_plane_minor_upgrade.go Adds MIContainers labeling to a DescribeTable spec.
test/e2e/control_plane_automated_z_stream_upgrade.go Adds MIContainers labeling to a DescribeTable spec.
test/e2e/complete_cluster_create_multiversion.go Adds MIContainers labeling to a DescribeTable spec.
test/e2e/clusters_sharing_resgroup.go Replaces old MI demand label with MIContainers(3).
test/e2e/cluster_versions.go Labels this spec as MIContainers(0).
test/e2e/cluster_version_backlevel.go Adds MIContainers labeling to an E2E spec.
test/e2e/cluster_update.go Adds MIContainers labeling to multiple It specs.
test/e2e/cluster_tls_endpoints.go Adds MIContainers labeling to an E2E spec.
test/e2e/cluster_pullsecret.go Adds MIContainers labeling to an E2E spec.
test/e2e/cluster_nsg_subnet_reuse.go Replaces old MI demand label with MIContainers(2).
test/e2e/cluster_fips_mode.go Adds MIContainers labeling to multiple It specs.
test/e2e/cluster_delete_cx_rg.go Adds MIContainers labeling to an E2E spec.
test/e2e/cluster_delayed_role_assignments.go Adds MIContainers labeling to an E2E spec.
test/e2e/cluster_create_v20260630preview.go Adds MIContainers labeling to an E2E spec.
test/e2e/cluster_create_private_kv.go Adds MIContainers labeling to an E2E spec.
test/e2e/cluster_create_nodepool_osdisk.go Adds MIContainers labeling to an E2E spec.
test/e2e/cluster_create_complex_cilium_kv.go Adds MIContainers labeling to an E2E spec.
test/e2e/cluster_create_cni_cilium.go Adds MIContainers labeling to an E2E spec.
test/e2e/cluster_autoscaling.go Adds MIContainers labeling to an E2E spec.
test/e2e/cluster_authorized_cidrs_connectivity.go Adds MIContainers labeling to an E2E spec.
test/e2e/arm64_nodepool.go Adds MIContainers labeling to an E2E spec.
test/e2e/admin_credential_lifecycle.go Adds MIContainers labeling to an E2E spec.
test/e2e/admin_api.go Adds MIContainers labeling to multiple It specs.
test/cmd/aro-hcp-tests/main.go Implements MIContainers label parsing + resource pool scheduling/enforcement.
sessiongate/go.mod Bumps openshift-tests-extension dependency version.
sessiongate/go.sum Updates dependency checksums for the bump.
Makefile Adds verify-mi-containers to the repo verify target.
hack/verify-mi-containers/main.go Adds a static verifier for MIContainers labeling vs AssignIdentityContainers usage.
hack/verify-mi-containers/go.mod Adds a standalone Go module for the verifier.
go.work Includes the new verifier module in the workspace.

Comment on lines +58 to +74
if len(violations) > 0 {
fmt.Println("ERROR: The following test specs have MIContainers label issues.")
fmt.Println(" Every It() and DescribeTable() must include labels.MIContainers(N)")
fmt.Println(" where N matches the count passed to AssignIdentityContainers.")
fmt.Println()
fmt.Println(" Examples:")
fmt.Println(` It("should create cluster", labels.MIContainers(1), func(ctx context.Context) {`)
fmt.Println(` DescribeTable("upgrades", labels.MIContainers(1), func(ctx context.Context, ...) {`)
fmt.Println()
for _, v := range violations {
fmt.Println(" " + v)
}
fmt.Println()
fmt.Println("Fix the labels listed above. Use labels.MIContainers(0) for tests")
fmt.Println("that do not call AssignIdentityContainers.")
os.Exit(1)
}
@openshift-ci

openshift-ci Bot commented Jul 11, 2026

Copy link
Copy Markdown

@mgahagan73: The following tests failed, say /retest to rerun all failed tests or /retest-required to rerun all mandatory failed tests:

Test name Commit Details Required Rerun command
ci/prow/verify 9538412 link true /test verify
ci/prow/e2e-parallel 9538412 link true /test e2e-parallel
ci/prow/images 9538412 link true /test images

Full PR test history. Your PR dashboard.

Details

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository. I understand the commands that are listed here.

@mgahagan73 mgahagan73 marked this pull request as draft July 11, 2026 03:20
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants