ci: enable API compliance and condenser test labels - #4204
Conversation
Co-authored-by: openhands <openhands@all-hands.dev>
enyst
added
api-compliance-test
OpenHands AI
Python API breakage checks — ✅ PASSEDResult: ✅ PASSED |
REST API breakage checks (OpenAPI) — ✅ PASSEDResult: ✅ PASSED |
🧪 Integration Tests ResultsOverall Success Rate: 88.9% 📁 Detailed Logs & ArtifactsClick the links below to access detailed agent/LLM logs showing the complete reasoning process for each model. On the GitHub Actions page, scroll down to the 'Artifacts' section to download the logs.
📊 Summary
📋 Detailed Resultslitellm_proxy_anthropic_claude_sonnet_4_6
litellm_proxy_openai_gpt_5.5
Skipped Tests:
Failed Tests:
|
Co-authored-by: openhands <openhands@all-hands.dev>
|
Integration Tests Triggered
Results will be posted here when complete. |
API Compliance Test ReportRun: Results Matrix✅ accepted ❌ rejected
Summary
Full API responses available in workflow artifacts |
🧪 Integration Tests ResultsOverall Success Rate: 100.0% 📊 Summary
📋 Detailed Resultslitellm_proxy_openai_gpt_5.5
Skipped Tests:
|
…ecial-integration-test-labels # Conflicts: # .github/workflows/integration-runner.yml
|
🤖 OpenHands is reviewing this PR. Head commit: This comment was posted by an AI agent (OpenHands). |
all-hands-bot
left a comment
all-hands-bot
left a comment
There was a problem hiding this comment.
This review was created by an AI agent (OpenHands) on behalf of the repository maintainers.
Overview
This PR restores label-triggered CI routing for condenser-test and api-compliance-test inside the existing integration-runner.yml, after the dedicated workflow files were removed in #3974. Routing the labels into the shared workflow (rather than reviving duplicate files) is the simpler, more maintainable approach and keeps a single place for the trigger/comment/consolidation logic.
The change is well-scoped: condenser tests reuse the existing matrix-driven run-integration-tests/consolidate-results path with a narrower default model set, and API compliance gets a standalone job since its a##_* runner is not part of run_infer.py's [tbc]*.py glob. The post-dispatch-comment fixes (adding pull-requests: write and --repo "$GITHUB_REPOSITORY") are correct.
Correctness
condenser-test->run_infer.py --test-type condenser: thec##_*files are classified ascondenserintests/integration/run_infer.py, and the label branches insetup-matrix,run-integration-tests, andconsolidate-resultsare all consistent. Default model set (gpt-5.5,claude-sonnet-4-6) is reasonable.api-compliance-test-> standalonerun-api-compliance-testsjob invokingtests/integration/api_compliance/run_compliance.py --output-dir compliance-results. Confirmed the script accepts--output-dirand writescompliance_report.md. Correctly excluded fromsetup-matrix(no matrix),run-integration-tests, andconsolidate-results(produces notest-results-*artifacts).- The
run_api_compliancedispatch input is a boolean (not atest_typevalue), and the job condition gates oninputs.run_api_compliance-- so dispatchingtest_type=allno longer implicitly runs compliance. This is a deliberate separation and is fine. - Report-posting
ifcorrectly fires for bothpull_request_targetandworkflow_dispatchwith a non-emptyissue_number.
Security
- The compliance job follows the same
pull_request_targetpattern asrun-integration-tests: label-gated by maintainers,persist-credentials: false, checks out the fork PR head, and only exposesLLM_API_KEY_EVAL+github.token. No new credential surface is introduced. The compliance job correctly scopes permissions tocontents: read,issues: write,pull-requests: write(noid-token: write, which the integration job needs for OIDC but compliance does not). Consistent and low risk.
Minor / non-blocking notes
- The compliance report comment is posted via
gh issue comment ... --body-file "$REPORT"directly on the raw report, whereas the condenser/integration path pipes the report throughsanitize_openhands_mentionsfirst. Since the compliance report is generated from raw LLM API responses to malformed-history patterns (not arbitrary PR content), the risk of@OpenHandsself-mention loops is negligible -- but for consistency you may want to apply the same sanitization helper. - The compliance comment uses
gh issue commentwhileconsolidate-results's label path usesgh pr comment. Both work against PR numbers; just an inconsistency to be aware of.
These are not blockers.
Risk assessment
Low risk. This is a CI-only change restoring existing label behavior in a single workflow file. Live runs on the PR branch (linked in the description) exercised both suites end-to-end, including report comments. No code paths that ship to users are affected.
No material issues found; the change is safe to merge.
…er labels Temporary reviewer evidence (.pr/, local-excluded, force-added): - api-compliance-test: live run against the eval proxy rejected a malformed tool_use history (HTTP 400 -> LLMMalformedConversationHistoryError), 3.2s. - condenser-test: --test-type condenser correctly selects the c01..c05 suite. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
|
📁 PR Artifacts Notice This PR contains a |
|
🤖 OpenHands is reviewing this PR. Head commit: This comment was posted by an AI agent (OpenHands). |
all-hands-bot
left a comment
all-hands-bot
left a comment
There was a problem hiding this comment.
This review was created by an AI agent (OpenHands) on behalf of the repository maintainers.
Overview
This PR restores label-triggered CI routing for condenser-test and api-compliance-test inside the existing integration-runner.yml, after PR #3974 removed the dedicated workflow files that previously recognized those labels. Routing the labels into the shared workflow (instead of reviving two duplicated files) is the simpler, more maintainable approach and keeps a single source of truth for trigger/comment/consolidation logic.
The change is well-scoped and the routing is consistent across all gating points:
condenser-test-> the existing matrix-drivenrun-integration-tests/consolidate-resultspath with--test-type condenserand a narrower two-model default (gpt-5.5,claude-sonnet-4-6).api-compliance-test-> a standalonerun-api-compliance-testsjob invokingtests/integration/api_compliance/run_compliance.py --output-dir compliance-results, correctly excluded fromsetup-matrix/run-integration-tests/consolidate-resultssince it produces notest-results-*artifacts.
Correctness
c##_*files are classified ascondenserintests/integration/run_infer.py(test_type = "condenser" forc-prefixed files), and--test-type condenseris a valid argparse choice. The label branch added insetup-matrix,run-integration-tests, andconsolidate-resultsare all consistent.- The compliance runner reads
LLM_API_KEY/LLM_BASE_URLfrom env (tests/integration/api_compliance/base.py), which the job sets toLLM_API_KEY_EVALand the eval proxy -- correct. run_api_complianceis a separate boolean dispatch input, sotest_type=allno longer implicitly runs compliance. This is a deliberate, sensible separation.- The report-posting
ifcorrectly fires for bothpull_request_targetandworkflow_dispatchwith a non-emptyissue_number; thefind ... -print -quit+if [ -n "$REPORT" ]guard means a failed run with no report file silently skips posting rather than erroring. - Patch 2 fixes (
pull-requests: writepermission +--repo "$GITHUB_REPOSITORY") are correct --GITHUB_REPOSITORYis a default Actions env var, so no explicitenv:declaration is needed.
Security
The compliance job follows the same pull_request_target pattern already established by run-integration-tests: maintainer-gated by label, persist-credentials: false, checks out the head PR ref, and only exposes LLM_API_KEY_EVAL + github.token. Permissions are scoped to contents: read, issues: write, pull-requests: write -- correctly omitting id-token: write, which the integration job needs for OIDC but compliance does not. No new credential surface is introduced; the checkout-head-under-pull_request_target consideration is pre-existing and accepted for this maintainer-gated workflow, not a regression from this PR.
Minor / non-blocking notes
- The compliance report is posted raw via
gh issue comment --body-file "$REPORT", whereas the condenser/integration path pipes the report throughsanitize_openhands_mentionsfirst. The compliance report is generated from LLM API error messages and pattern descriptions rather than arbitrary PR content, so the@OpenHandsself-mention risk is low -- but for consistency you may want to apply the same sanitization helper (inline comment below). - The compliance comment uses
gh issue commentwhileconsolidate-results's label path usesgh pr comment. Both post to the same issue-comment endpoint and work against PR numbers, so this is functionally fine -- just an inconsistency to be aware of. - The
.pr/evidence files are auto-removed on approval for same-repo PRs (this PR is not cross-repository), so no manual cleanup is needed before merge.
These are not blockers.
Risk assessment
Low risk. This is a CI-only change restoring existing label behavior in a single workflow file. No user-shipping code paths are affected. Live runs linked in the PR description exercised both suites end-to-end against the eval proxy, including report comments on both label and dispatch paths.
No material issues found; the change is safe to merge.
Co-authored-by: openhands <openhands@all-hands.dev>
3 participants
HUMAN:
Run the test suites for condenser and LLM API format, each on label. They're not run automatically; setting them on label allows us to use the label when suitable.
AGENT:
Why
The
api-compliance-testandcondenser-testlabels still exist, and their test runners still live in the repository, but PR #3974 removed the only workflows that recognized those labels. Applying either label now creates no test job.This restores the missing routes in the existing integration workflow instead of reviving two duplicated workflow files.
Summary
condenser-testto the existingc##_*runner and use a two-model default matrix.api-compliance-testto an independent job for thea##_*compliance runner.REST API contract changes
Compared with base OpenAPI
2b38718ad34afor public/api/**paths.Issue Number
N/A. Follow-up requested in #4203.
How to Test
Static validation:
Live runs against the PR branch:
99cf968d:c01skipped as expected andc04_token_condenserdid not trigger condensation.9c445c26fixes both.9c445c26:c01skipped as expected.Video/Screenshots
N/A — CI workflow change. Live workflow runs and generated reports are linked above.
Type
Notes
The existing labels are applied to this PR as requested. Their label-triggered jobs skip against the old
mainworkflow becausepull_request_targetevaluates the base branch definition. Branch-scoped manual dispatch validates the same runner paths before merge; label-trigger behavior becomes active after merge.This PR was created by an AI agent (OpenHands) on behalf of the user.
Agent Server images for this PR
• GHCR package: https://github.com/OpenHands/agent-sdk/pkgs/container/agent-server
Variants & Base Images
eclipse-temurin:17-jdknikolaik/python-nodejs:python3.13-nodejs22-slimgolang:1.21-bookwormPull (multi-arch manifest)
# Each variant is a multi-arch manifest supporting both amd64 and arm64 docker pull ghcr.io/openhands/agent-server:0cb037c-pythonRun
All tags pushed for this build
About Multi-Architecture Support
0cb037c-python) is a multi-arch manifest supporting both amd64 and arm640cb037c-python-amd64) are also available if needed