OCPMCP-108: weekly mcpchecker eval periodic for openshift-mcp-server#81947
OCPMCP-108: weekly mcpchecker eval periodic for openshift-mcp-server#81947cajieh wants to merge 1 commit into
Conversation
|
@cajieh: This pull request references OCPMCP-108 which is a valid jira issue. Warning: The referenced jira issue has an invalid target version for the target branch this PR targets: expected the epic to target the "5.0.0" version, but no target version was set. DetailsIn response to this:
Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the openshift-eng/jira-lifecycle-plugin repository. |
openshift-ci-robot
added
the
jira/valid-reference
|
Note Reviews pausedIt looks like this branch is under active development. To avoid overwhelming you with review comments due to an influx of new commits, CodeRabbit has automatically paused this review. You can configure this behavior by changing the Use the following commands to manage reviews:
Use the checkboxes below for quick actions:
WalkthroughAdds a weekly CI job that runs the MCP server with ChangesMCP checker evaluation
Estimated code review effort: 3 (Moderate) | ~20 minutes Sequence Diagram(s)sequenceDiagram
participant periodic-mcpchecker-eval
participant MCPServer
participant mcpchecker
participant ARTIFACT_DIR
periodic-mcpchecker-eval->>MCPServer: make run-server TOOLSETS=core,config
periodic-mcpchecker-eval->>mcpchecker: check eval-core.yaml with suite=core
mcpchecker-->>periodic-mcpchecker-eval: JSON evaluation results
periodic-mcpchecker-eval->>ARTIFACT_DIR: Copy JSON and write JUnit XML
periodic-mcpchecker-eval->>MCPServer: Stop server on exit
Suggested reviewers: 🚥 Pre-merge checks | ✅ 14 | ❌ 1❌ Failed checks (1 warning)
✅ Passed checks (14 passed)
✨ Finishing Touches🧪 Generate unit tests (beta)
Comment |
|
[APPROVALNOTIFIER] This PR is NOT APPROVED This pull-request has been approved by: cajieh The full list of commands accepted by this bot can be found here. DetailsNeeds approval from an approver in each of these files:
Approvers can indicate their approval by writing |
![coderabbitai[bot]](https://avatars.githubusercontent.com/in/347564?s=60&v=4){kind=small}
There was a problem hiding this comment.
🧹 Nitpick comments (1)
ci-operator/config/openshift/openshift-mcp-server/openshift-openshift-mcp-server-main.yaml (1)
18-18: 🔒 Security & Privacy | 🔵 Trivial | ⚡ Quick winGo patch version is behind latest security release.
Go 1.26.3 go1.26.3 (released 2026-05-07) includes security fixes to the go command, the pack tool, and the html/template, net, net/http, net/http/httputil, net/mail, and syscall packages, but newer patches exist: go1.26.4 (released 2026-06-02) includes security fixes to the crypto/x509, mime, and net/textproto packages... go1.26.5 (released 2026-07-07) includes security fixes to the crypto/tls and os packages. Consider bumping to 1.26.5 to pick up the latest security fixes.
🔧 Proposed fix
- RUN curl -fsSL https://go.dev/dl/go1.26.3.linux-amd64.tar.gz | tar -C /usr/local -xzf - \ + RUN curl -fsSL https://go.dev/dl/go1.26.5.linux-amd64.tar.gz | tar -C /usr/local -xzf - \🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@ci-operator/config/openshift/openshift-mcp-server/openshift-openshift-mcp-server-main.yaml` at line 18, Update the Go toolchain download URL in the Docker build RUN command to use Go 1.26.5 instead of 1.26.3, keeping the existing Linux amd64 archive extraction flow unchanged.
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Nitpick comments:
In
`@ci-operator/config/openshift/openshift-mcp-server/openshift-openshift-mcp-server-main.yaml`:
- Line 18: Update the Go toolchain download URL in the Docker build RUN command
to use Go 1.26.5 instead of 1.26.3, keeping the existing Linux amd64 archive
extraction flow unchanged.
ℹ️ Review info
⚙️ Run configuration
Configuration used: Repository YAML (base), Central YAML (inherited)
Review profile: CHILL
Plan: Enterprise
Run ID: 94040479-07ab-40ea-a1df-9508cd873ac9
⛔ Files ignored due to path filters (1)
ci-operator/jobs/openshift/openshift-mcp-server/openshift-openshift-mcp-server-main-periodics.yamlis excluded by!ci-operator/jobs/**
📒 Files selected for processing (1)
ci-operator/config/openshift/openshift-mcp-server/openshift-openshift-mcp-server-main.yaml
3320f6e to
abd97a3
Compare
|
/retest |
ffdeba9 to
4b4d1b8
Compare
|
/pj-rehearse periodic-ci-openshift-openshift-mcp-server-main-periodic-mcpchecker-eval |
|
@cajieh: now processing your pj-rehearse request. Please allow up to 10 minutes for jobs to trigger or cancel. |
4b4d1b8 to
fdc3fda
Compare
|
/pj-rehearse periodic-ci-openshift-openshift-mcp-server-main-periodic-mcpchecker-eval |
1 similar comment
|
/pj-rehearse periodic-ci-openshift-openshift-mcp-server-main-periodic-mcpchecker-eval |
|
@cajieh: now processing your pj-rehearse request. Please allow up to 10 minutes for jobs to trigger or cancel. |
fdc3fda to
4b19743
Compare
|
/pj-rehearse periodic-ci-openshift-openshift-mcp-server-main-periodic-mcpchecker-eval |
|
@cajieh: now processing your pj-rehearse request. Please allow up to 10 minutes for jobs to trigger or cancel. |
|
@cajieh: The following test failed, say
Full PR test history. Your PR dashboard. DetailsInstructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository. I understand the commands that are listed here. |
|
/pj-rehearse periodic-ci-openshift-openshift-mcp-server-main-periodic-mcpchecker-eval |
|
@cajieh: now processing your pj-rehearse request. Please allow up to 10 minutes for jobs to trigger or cancel. |
2 participants
Summary
periodic-mcpchecker-eval, Mon 09:00 UTC) that runs mcpchecker core-suite evals againstopenshift/openshift-mcp-serveron a live OpenShift cluster (ipi-aws/openshift-org-aws), with an 80% verify gate and JSON + JUnit artifacts under${ARTIFACT_DIR}for Spyglass / Sippy (OCPMCP-108 / OCPMCP-308.1).evals/core-eval-testing/builtin-openai/eval-core.yaml(judge via sharedagent.yamlllmJudge.ref; no deprecatedJUDGE_*env vars).ocp-mcpcredentials fromtest-credentialsat/var/run/ocp-mcpand exportsOPENAI_API_KEYfromopenai-token(Vault:selfservice/ocp-mcp/openai-credentials).Test plan
ci-operator-config/generated-config/yamllintpass on this PRopenai-tokenis synced into theocp-mcpCI secret (or update mount accordingly)/pj-rehearse periodic-ci-openshift-openshift-mcp-server-main-periodic-mcpchecker-evalafter OpenAI token is available in CI${ARTIFACT_DIR}/mcpchecker-out.jsonand${ARTIFACT_DIR}/junit_mcpchecker.xmlCo-authored by Claude and Cursor.
Summary by CodeRabbit
Adds a weekly mcpchecker core-suite evaluation job for
openshift/openshift-mcp-server, running in Kind with nested Podman. It uses dedicated evaluation tooling, LLM-judge credentials, an 80% verification gate, and publishes JSON/JUnit artifacts through OpenShift CI.