OTA-1870: OLS-service: add cluster-updates periodic evaluation job

[fao89]

Depends-On: openshift/lightspeed-service#2908

Add a new periodic CI job to evaluate OLS cluster update prompts. This job runs daily at 14:10 UTC and tests OLS responses across 5 OpenShift cluster upgrade scenarios:

• Precheck analysis (Update Service health, cluster history, available updates)
• Targeted upgrade path validation
• Cluster health assessment at latest version
• Real-time upgrade progress monitoring
• Upgrade failure diagnosis and remediation

The test uses GPT-4o-mini model and validates evaluation artifacts with 0% error rate requirement. It follows the same pattern as ols-evaluation-periodics but focuses specifically on cluster upgrade workflow AI responses.

Co-Authored-By: Claude Sonnet 4.5 noreply@anthropic.com

Summary by CodeRabbit

Added a new daily periodic Prow job to the OpenShift CI configuration for the openshift-lightspeed-service component (ci-operator/config/openshift/lightspeed-service/openshift-lightspeed-service-main__4.21.yaml): ols-cluster-updates-periodics. The job runs at 14:10 UTC (cron: 10 14 * * *) and provisions an AWS FIPS cluster claim (amd64, us-east-1, variant: fips, version 4.21).

In that workflow, the job executes the cluster-updates test using the latest CLI image, runs tests/scripts/test-cluster-updates.sh with the required OLS provider/model configuration, and mounts the needed provider credential secrets (bam, azureopenai, openai, watsonx, vertex, plus an offline token). It also configures container CPU (100m) and wires the Lightspeed service dependency via env: OLS_IMAGE (using lightspeed-service-api).

Practically, this expands CI coverage for validating OLS cluster upgrade/workflow AI responses across the cluster update lifecycle—precheck analysis, targeted upgrade path validation, health checks at the latest version, live upgrade progress monitoring, and upgrade failure diagnosis/remediation—using GPT-4o-mini with strict evaluation requirements (including a 0% error rate for evaluation artifacts).

[openshift-ci-robot]

@fao89: This pull request references OTA-1870 which is a valid jira issue.

Warning: The referenced jira issue has an invalid target version for the target branch this PR targets: expected the story to target the "5.0.0" version, but no target version was set.

Details

In response to this:

Depends-On: openshift/lightspeed-service#2908

Add a new periodic CI job to evaluate OLS cluster update prompts. This job runs daily at 14:10 UTC and tests OLS responses across 5 OpenShift cluster upgrade scenarios:

• Precheck analysis (Update Service health, cluster history, available updates)
• Targeted upgrade path validation
• Cluster health assessment at latest version
• Real-time upgrade progress monitoring
• Upgrade failure diagnosis and remediation

The test uses GPT-4o-mini model and validates evaluation artifacts with 0% error rate requirement. It follows the same pattern as ols-evaluation-periodics but focuses specifically on cluster upgrade workflow AI responses.

Co-Authored-By: Claude Sonnet 4.5 noreply@anthropic.com

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the openshift-eng/jira-lifecycle-plugin repository.

[coderabbitai]

No actionable comments were generated in the recent review. 🎉

ℹ️ Recent review info

⚙️ Run configuration

Configuration used: Repository YAML (base), Central YAML (inherited)

Review profile: CHILL

Plan: Enterprise

Run ID: 658a799c-351b-44fe-a583-cac8ccf5c0ba

📥 Commits

Reviewing files that changed from the base of the PR and between 4345940 and 13bca73.

⛔ Files ignored due to path filters (1)

• ci-operator/jobs/openshift/lightspeed-service/openshift-lightspeed-service-main-periodics.yaml is excluded by !ci-operator/jobs/**

📒 Files selected for processing (1)

• ci-operator/config/openshift/lightspeed-service/openshift-lightspeed-service-main__4.21.yaml

🚧 Files skipped from review as they are similar to previous changes (1)

• ci-operator/config/openshift/lightspeed-service/openshift-lightspeed-service-main__4.21.yaml

---

Walkthrough

Adds a new periodic CI test entry for lightspeed-service 4.21 with an AWS cluster claim, generic-claim workflow, cluster-updates script execution, secret mounts, CPU requests, and an OLS_IMAGE dependency.

Changes

Periodic Test Configuration

Layer / File(s)	Summary
New periodic test entry ci-operator/config/openshift/lightspeed-service/openshift-lightspeed-service-main__4.21.yaml	Adds ols-cluster-updates-periodics with cron scheduling, AWS cluster_claim, generic-claim workflow, test-cluster-updates.sh, provider/model environment exports, secret mounts, CPU requests, and OLS_IMAGE dependency wiring.

Estimated code review effort: 2 (Simple) | ~10 minutes

🚥 Pre-merge checks | ✅ 5 | ❌ 10

❌ Failed checks (10 inconclusive)

Check name	Status	Explanation	Resolution
Stable And Deterministic Test Names	❓ Inconclusive	Repository clone failed, so this custom check could not run with code access.	Retry the review run. If this persists, inspect pre-merge custom-check logs for infrastructure or agent runtime failures.
Test Structure And Quality	❓ Inconclusive	Repository clone failed, so this custom check could not run with code access.	Retry the review run. If this persists, inspect pre-merge custom-check logs for infrastructure or agent runtime failures.
Microshift Test Compatibility	❓ Inconclusive	Repository clone failed, so this custom check could not run with code access.	Retry the review run. If this persists, inspect pre-merge custom-check logs for infrastructure or agent runtime failures.
Single Node Openshift (Sno) Test Compatibility	❓ Inconclusive	Repository clone failed, so this custom check could not run with code access.	Retry the review run. If this persists, inspect pre-merge custom-check logs for infrastructure or agent runtime failures.
Topology-Aware Scheduling Compatibility	❓ Inconclusive	Repository clone failed, so this custom check could not run with code access.	Retry the review run. If this persists, inspect pre-merge custom-check logs for infrastructure or agent runtime failures.
Ote Binary Stdout Contract	❓ Inconclusive	Repository clone failed, so this custom check could not run with code access.	Retry the review run. If this persists, inspect pre-merge custom-check logs for infrastructure or agent runtime failures.
Ipv6 And Disconnected Network Test Compatibility	❓ Inconclusive	Repository clone failed, so this custom check could not run with code access.	Retry the review run. If this persists, inspect pre-merge custom-check logs for infrastructure or agent runtime failures.
No-Weak-Crypto	❓ Inconclusive	Repository clone failed, so this custom check could not run with code access.	Retry the review run. If this persists, inspect pre-merge custom-check logs for infrastructure or agent runtime failures.
Container-Privileges	❓ Inconclusive	Repository clone failed, so this custom check could not run with code access.	Retry the review run. If this persists, inspect pre-merge custom-check logs for infrastructure or agent runtime failures.
No-Sensitive-Data-In-Logs	❓ Inconclusive	Repository clone failed, so this custom check could not run with code access.	Retry the review run. If this persists, inspect pre-merge custom-check logs for infrastructure or agent runtime failures.

✅ Passed checks (5 passed)

Check name	Status	Explanation
Description Check	✅ Passed	Check skipped - CodeRabbit’s high-level summary is enabled.
Title check	✅ Passed	The title accurately and concisely describes the added cluster-updates periodic evaluation job.
Docstring Coverage	✅ Passed	No functions found in the changed files to evaluate docstring coverage. Skipping docstring coverage check.
Linked Issues check	✅ Passed	Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check	✅ Passed	Check skipped because no linked issues were found for this pull request.

✨ Finishing Touches

🧪 Generate unit tests (beta)

• Create PR with unit tests

---

_{Comment @coderabbitai help to get the list of available commands.}

[openshift-ci]

[APPROVALNOTIFIER] This PR is NOT APPROVED

This pull-request has been approved by: fao89
Once this PR has been reviewed and has the lgtm label, please assign raptorsun for approval. For more information see the Code Review Process.

The full list of commands accepted by this bot can be found here.

Details

Needs approval from an approver in each of these files:

• ci-operator/config/openshift/lightspeed-service/OWNERS
• ci-operator/jobs/openshift/lightspeed-service/OWNERS

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

[openshift-merge-bot]

[REHEARSALNOTIFIER]
@fao89: the pj-rehearse plugin accommodates running rehearsal tests for the changes in this PR. Expand 'Interacting with pj-rehearse' for usage details. The following rehearsable tests have been affected by this change:

Test name	Repo	Type	Reason
periodic-ci-openshift-lightspeed-service-main-4.21-ols-cluster-updates-periodics	N/A	periodic	Periodic changed

Interacting with pj-rehearse

Comment: /pj-rehearse to run up to 5 rehearsals
Comment: /pj-rehearse skip to opt-out of rehearsals
Comment: /pj-rehearse {test-name}, with each test separated by a space, to run one or more specific rehearsals
Comment: /pj-rehearse more to run up to 10 rehearsals
Comment: /pj-rehearse max to run up to 25 rehearsals
Comment: /pj-rehearse auto-ack to run up to 5 rehearsals, and add the rehearsals-ack label on success
Comment: /pj-rehearse list to get an up-to-date list of affected jobs
Comment: /pj-rehearse abort to abort all active rehearsals
Comment: /pj-rehearse network-access-allowed to allow rehearsals of tests that have the restrict_network_access field set to false. This must be executed by an openshift org member who is not the PR author

Once you are satisfied with the results of the rehearsals, comment: /pj-rehearse ack to unblock merge. When the rehearsals-ack label is present on your PR, merge will no longer be blocked by rehearsals.
If you would like the rehearsals-ack label removed, comment: /pj-rehearse reject to re-block merging.

[fao89]

/pj-rehearse periodic-ci-openshift-lightspeed-service-main-4.21-ols-cluster-updates-periodics

[openshift-merge-bot]

@fao89: now processing your pj-rehearse request. Please allow up to 10 minutes for jobs to trigger or cancel.

[openshift-ci]

@fao89: The following test failed, say /retest to rerun all failed tests or /retest-required to rerun all mandatory failed tests:

Test name	Commit	Details	Required	Rerun command
ci/rehearse/periodic-ci-openshift-lightspeed-service-main-4.21-ols-cluster-updates-periodics	53463b8	link	unknown	/pj-rehearse periodic-ci-openshift-lightspeed-service-main-4.21-ols-cluster-updates-periodics

Full PR test history. Your PR dashboard.

Details

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository. I understand the commands that are listed here.