fix(ci): bind verification to exact source heads - #88
Conversation
📝 WalkthroughWalkthroughCI의 모든 Changes정확한 소스 커밋 CI
Estimated code review effort: 3 (Moderate) | ~20 minutes Possibly related issues
🚥 Pre-merge checks | ✅ 5✅ Passed checks (5 passed)
✨ Finishing Touches📝 Generate docstrings
🧪 Generate unit tests (beta)
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
|
@coderabbitai review Please review exact current source head |
🧩 Analysis chain🏁 Script executed: #!/bin/bash
set -euo pipefail
head="e8e9d6ca82738eea2de9efb03ab78da068a225f7"
base="bf2cc2e140dc3ff4a56c3203f80f41bb9fed5d10"
echo '== Repository and commit identities =='
git remote -v
git cat-file -e "${head}^{commit}"
git cat-file -e "${base}^{commit}"
printf 'HEAD checkout: '; git rev-parse HEAD
printf 'Requested head: '; git rev-parse "$head"
printf 'Requested base: '; git rev-parse "$base"
echo '== Changed files: requested base...head =='
git diff --name-status "$base" "$head"
echo '== Focused CI workflow diff =='
git diff --unified=80 "$base" "$head" -- .github/workflows/ci.yml
echo '== Current CI workflow with line numbers =='
git show "${head}:.github/workflows/ci.yml" | nl -ba
echo '== All checkout and verification lines at requested head =='
git show "${head}:.github/workflows/ci.yml" |
rg -n -C 5 'actions/checkout|git rev-parse HEAD|github\.event\.pull_request\.head\.sha|persist-credentials'
echo '== PR `#69` metadata and changed files =='
gh pr view 69 --repo ContextualWisdomLab/pg-llm-batch \
--json number,state,isDraft,headRefOid,baseRefOid,title,files \
--jq '{number,state,isDraft,headRefOid,baseRefOid,title,files:[.files[].path]}'
echo '== PR `#69` workflow-related patch, if available =='
gh pr diff 69 --repo ContextualWisdomLab/pg-llm-batch -- .github/workflows/ci.yml 2>&1 || trueLength of output: 6697 🏁 Script executed: #!/bin/bash
set -euo pipefail
head="e8e9d6ca82738eea2de9efb03ab78da068a225f7"
base="bf2cc2e140dc3ff4a56c3203f80f41bb9fed5d10"
repo="ContextualWisdomLab/pg-llm-batch"
echo '== Current CI workflow, line-numbered =='
git show "${head}:.github/workflows/ci.yml" | awk '{printf "%4d %s\n", NR, $0}'
echo '== Checkout and verification contexts =='
git show "${head}:.github/workflows/ci.yml" |
awk '
/name: Checkout|uses: actions\/checkout|ref:.*github\.event\.pull_request\.head\.sha|persist-credentials|name: Verify exact source head|git rev-parse HEAD/ {
printf "%4d %s\n", NR, $0
}'
echo '== Workflow contract test changes =='
git diff --unified=100 "$base" "$head" -- tests/test_workflow_contracts.py
echo '== Changelog and exact-source evidence document changes =='
git diff --unified=30 "$base" "$head" -- CHANGELOG.md docs/doctoring/exact-source-ci-evidence.md
echo '== Draft PR `#69` metadata and changed-file scope =='
gh api "repos/${repo}/pulls/69" \
--jq '{number,state,draft,head_sha:.head.sha,base_sha:.base.sha,title,changed_files}'
gh api --paginate "repos/${repo}/pulls/69/files?per_page=100" \
--jq '.[] | {filename,status,sha}'
echo '== Draft PR `#69` CI workflow patch =='
gh api "repos/${repo}/pulls/69/files?per_page=100" \
--jq '.[] | select(.filename == ".github/workflows/ci.yml") | .patch // "No ci.yml patch"'
echo '== Requested-head check runs =='
gh api "repos/${repo}/commits/${head}/check-runs?per_page=100" \
--jq '.check_runs[] | {name,status,conclusion,head_sha,details_url,app:.app.slug}'
echo '== Requested-head commit statuses =='
gh api "repos/${repo}/commits/${head}/status" \
--jq '{sha,state,statuses:[.statuses[] | {context,state,target_url,creator:.creator.login}]}'Length of output: 30149
I found no code issues in the requested scope.
The completed CI, Security Scan, and SAST checks are attached to the requested source head.
|
![coderabbitai[bot]](https://avatars.githubusercontent.com/in/347564?s=60&v=4){kind=small}
There was a problem hiding this comment.
Actionable comments posted: 1
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In `@tests/test_workflow_contracts.py`:
- Around line 44-55: 수정 대상은
test_ci_checks_out_and_verifies_the_exact_source_head_in_every_job입니다. 전역 문자열 개수
비교를 제거하고 YAML의 jobs.*.steps를 순회해 각 actions/checkout 단계별로 with.ref가
exact_source_expression과 일치하는지 검증하세요. 각 checkout 바로 다음 단계가 name: Verify exact
source head를 사용하고, run 명령이 동일한 git rev-parse HEAD 비교식을 포함하는지도 함께 검증하며, 모든
checkout이 계약을 만족해야 테스트가 통과하도록 하세요.
🪄 Autofix
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: Organization UI
Review profile: CHILL
Plan: Pro Plus
Run ID: bf917e1e-8827-40c8-bda0-f94e0b6a2dd3
📒 Files selected for processing (4)
.github/workflows/ci.ymlCHANGELOG.mddocs/doctoring/exact-source-ci-evidence.mdtests/test_workflow_contracts.py
|
@opencode-agent @cwl-noema-review review Review exact current head Verify that every |
![opencode-agent[bot]](https://avatars.githubusercontent.com/in/1549082?s=60&v=4){kind=small}
There was a problem hiding this comment.
Pull request overview
OpenCode cannot approve yet because required coverage evidence did not pass.
Review outcome
1. HIGH .github/workflows/opencode-review.yml:1 - Coverage evidence did not prove required test/docstring evidence
-
Problem: The required coverage-evidence job result was
failure, so OpenCode cannot establish approval sufficiency for this head. -
Root cause: Automated approval is only valid when the same-head coverage-evidence job proves supported repository test suites passed and configured docstring gates passed or were advisory, or reports not applicable because no supported source files or package manifests exist. Missing, failed, skipped, unavailable, or unsupported-tooling test evidence is a blocker.
-
Fix: Install or configure the repository test/docstring evidence tooling when source files or package manifests exist, rerun the current-head coverage-evidence job, and approve only after it reports
successwith required evidence or explicit no-source not-applicable evidence. -
Regression test: Keep the approval branch checking
needs.coverage-evidence.result == successbefore posting APPROVE, and publish REQUEST_CHANGES when coverage-evidence blocker states such as cancelled, skipped, failed, unsupported-tooling, or below-100 evidence are present. -
Result: REQUEST_CHANGES
-
Reason: coverage-evidence result was
failure, so required test/docstring evidence was not proven for current head9bd4d26c6f34a4bdba567beff5609423bc159cc6. -
Head SHA:
9bd4d26c6f34a4bdba567beff5609423bc159cc6 -
Workflow run: 31315169176
-
Workflow attempt: 1
Coverage evidence
Coverage evidence job did not run or did not publish coverage evidence.
Changed-File Evidence Map
flowchart LR
PR["PR changed files"] --> Evidence["OpenCode bounded evidence"]
Evidence --> S1["Workflow: ci.yml"]
S1 --> I1["GitHub Actions review job"]
I1 --> R1["Review risk: Workflow: ci.yml"]
R1 --> V1["actionlint plus required checks"]
Evidence --> S2["Changed file: CHANGELOG.md"]
S2 --> I2["repository behavior"]
I2 --> R2["Review risk: Changed file: CHANGELOG.md"]
R2 --> V2["required checks"]
Evidence --> S3["Docs: exact-source-ci-evidence.md"]
S3 --> I3["operator or user guidance"]
I3 --> R3["Review risk: Docs: exact-source-ci-evidence.md"]
R3 --> V3["docs review"]
Evidence --> S4["Test: test_workflow_contracts.py"]
S4 --> I4["regression suite"]
I4 --> R4["Review risk: Test: test_workflow_contracts.py"]
R4 --> V4["targeted test run"]
OpenCode Review Overview
Pull request overviewOpenCode cannot approve yet because required coverage evidence did not pass. Review outcome1. HIGH .github/workflows/opencode-review.yml:1 - Coverage evidence did not prove required test/docstring evidence
Coverage evidenceCoverage evidence job did not run or did not publish coverage evidence. Changed-File Evidence Mapflowchart LR
PR["PR changed files"] --> Evidence["OpenCode bounded evidence"]
Evidence --> S1["Workflow: ci.yml"]
S1 --> I1["GitHub Actions review job"]
I1 --> R1["Review risk: Workflow: ci.yml"]
R1 --> V1["actionlint plus required checks"]
Evidence --> S2["Changed file: CHANGELOG.md"]
S2 --> I2["repository behavior"]
I2 --> R2["Review risk: Changed file: CHANGELOG.md"]
R2 --> V2["required checks"]
Evidence --> S3["Docs: exact-source-ci-evidence.md"]
S3 --> I3["operator or user guidance"]
I3 --> R3["Review risk: Docs: exact-source-ci-evidence.md"]
R3 --> V3["docs review"]
Evidence --> S4["Test: test_workflow_contracts.py"]
S4 --> I4["regression suite"]
I4 --> R4["Review risk: Test: test_workflow_contracts.py"]
R4 --> V4["targeted test run"]
|
There was a problem hiding this comment.
Pull request overview
OpenCode cannot approve yet because required coverage evidence did not pass.
Review outcome
1. HIGH .github/workflows/opencode-review.yml:1 - Coverage evidence did not prove required test/docstring evidence
-
Problem: The required coverage-evidence job result was
failure, so OpenCode cannot establish approval sufficiency for this head. -
Root cause: Automated approval is only valid when the same-head coverage-evidence job proves supported repository test suites passed and configured docstring gates passed or were advisory, or reports not applicable because no supported source files or package manifests exist. Missing, failed, skipped, unavailable, or unsupported-tooling test evidence is a blocker.
-
Fix: Install or configure the repository test/docstring evidence tooling when source files or package manifests exist, rerun the current-head coverage-evidence job, and approve only after it reports
successwith required evidence or explicit no-source not-applicable evidence. -
Regression test: Keep the approval branch checking
needs.coverage-evidence.result == successbefore posting APPROVE, and publish REQUEST_CHANGES when coverage-evidence blocker states such as cancelled, skipped, failed, unsupported-tooling, or below-100 evidence are present. -
Result: REQUEST_CHANGES
-
Reason: coverage-evidence result was
failure, so required test/docstring evidence was not proven for current head9bd4d26c6f34a4bdba567beff5609423bc159cc6. -
Head SHA:
9bd4d26c6f34a4bdba567beff5609423bc159cc6 -
Workflow run: 31317015216
-
Workflow attempt: 1
Coverage evidence
Coverage evidence job did not run or did not publish coverage evidence.
Changed-File Evidence Map
flowchart LR
PR["PR changed files"] --> Evidence["OpenCode bounded evidence"]
Evidence --> S1["Workflow: ci.yml"]
S1 --> I1["GitHub Actions review job"]
I1 --> R1["Review risk: Workflow: ci.yml"]
R1 --> V1["actionlint plus required checks"]
Evidence --> S2["Changed file: CHANGELOG.md"]
S2 --> I2["repository behavior"]
I2 --> R2["Review risk: Changed file: CHANGELOG.md"]
R2 --> V2["required checks"]
Evidence --> S3["Docs: exact-source-ci-evidence.md"]
S3 --> I3["operator or user guidance"]
I3 --> R3["Review risk: Docs: exact-source-ci-evidence.md"]
R3 --> V3["docs review"]
Evidence --> S4["Test: test_workflow_contracts.py"]
S4 --> I4["regression suite"]
I4 --> R4["Review risk: Test: test_workflow_contracts.py"]
R4 --> V4["targeted test run"]
|
@coderabbitai review |
|
|
@coderabbitai review |
|
|
@coderabbitai review |
✅ Action performedReview finished.
|
There was a problem hiding this comment.
Actionable comments posted: 1
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In `@tests/test_workflow_contracts.py`:
- Around line 100-109: The workflow contract checks in _workflow_job_steps must
validate parsed or strictly field-matched YAML rather than substring presence.
For each checkout step, inspect the actual uses field and with.ref value; for
the following step, inspect the actual name and run fields, excluding comments
and unrelated values while preserving the existing exact-source requirements.
🪄 Autofix
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: Organization UI
Review profile: CHILL
Plan: Pro Plus
Run ID: d31385fe-216e-42fc-b50b-0198d312117a
📒 Files selected for processing (1)
tests/test_workflow_contracts.py
|
Tip For best results, initiate chat on the files or code changes. 🧩 Analysis chain🏁 Script executed: #!/bin/bash
set -euo pipefail
head="ab78e4744a0709edfe9a45d28ec29b80a05771d5"
base="bf2cc2e140dc3ff4a56c3203f80f41bb9fed5d10"
echo "== Commit identity =="
git cat-file -e "${head}^{commit}"
git cat-file -e "${base}^{commit}"
printf 'requested head: '; git rev-parse "$head"
printf 'requested base: '; git rev-parse "$base"
echo "== Changed files =="
git diff --name-status "$base" "$head"
echo "== Candidate-file outlines =="
ast-grep outline .github/workflows/ci.yml --items all || true
ast-grep outline tests/test_workflow_contracts.py --items all --type function
echo "== Focused diff =="
git diff --unified=100 "$base" "$head" -- \
.github/workflows/ci.yml \
tests/test_workflow_contracts.py \
docs/doctoring/exact-source-ci-evidence.md
echo "== Requested-head workflow =="
git show "${head}:.github/workflows/ci.yml" | nl -ba
echo "== Requested-head exact-source contract contexts =="
git show "${head}:tests/test_workflow_contracts.py" |
nl -ba |
sed -n '/def _workflow_job_steps/,/^def /p; /def test_ci_checks_out_and_verifies_the_exact_source_head_in_every_job/,/^def /p; /def test_ci_workflow_contract_rejects_decoy/,/^def /p'
echo "== Parsed checkout and immediate-next-step fields =="
git show "${head}:.github/workflows/ci.yml" > /tmp/ci.yml
python3 - <<'PY'
from pathlib import Path
import yaml
workflow = yaml.safe_load(Path("/tmp/ci.yml").read_text())
expected = "${{ github.event.pull_request.head.sha || github.sha }}"
for job_name, job in workflow["jobs"].items():
steps = job.get("steps", [])
for index, step in enumerate(steps):
if step.get("uses", "").startswith("actions/checkout@"):
following = steps[index + 1] if index + 1 < len(steps) else {}
print({
"job": job_name,
"ref": step.get("with", {}).get("ref"),
"persist-credentials": step.get("with", {}).get("persist-credentials"),
"next_name": following.get("name"),
"next_run": following.get("run"),
"expected": expected,
})
PY
echo "== Exact-source documentation clauses =="
git show "${head}:docs/doctoring/exact-source-ci-evidence.md" |
nl -ba |
rg -n -C 2 'persist-credentials|checkout|verification|rollback|evidence|source head|source-head'Length of output: 21590
Add this assertion in the checkout loop: assert _step_nested_field(step, "with", "persist-credentials") == "false"Add a decoy assertion for The workflow itself correctly binds each checkout to You are interacting with an AI system. |
Pull request was converted to draft
Dismissed as stale infrastructure-only review evidence. This exact-head REQUEST_CHANGES recorded a central coverage-evidence failure, not a pg-llm-batch source finding. The read-only central review path has materially changed since that run, and its corresponding central failure review is now dismissed. Live pg-llm-batch policy currently requires zero approving reviews. This dismissal does not approve #88, does not transfer old evidence, and does not waive exact-source CI/security/merge gates; fresh final evidence remains required before merge.
There was a problem hiding this comment.
Actionable comments posted: 1
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In `@tests/test_workflow_contracts.py`:
- Line 181: Extend the checkout-step assertions in the workflow contract test to
require _step_nested_field(step, "with", "persist-credentials") to equal "false"
for every checkout, alongside the existing with.ref check. Add a decoy
regression case proving values in env or comments do not satisfy this contract,
so only the checkout with.persist-credentials setting is accepted.
🪄 Autofix
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: Organization UI
Review profile: CHILL
Plan: Pro Plus
Run ID: 10d54754-4828-43dc-b3d9-e9e6e4d54c2f
📒 Files selected for processing (1)
tests/test_workflow_contracts.py
Dismissed as predecessor-head infrastructure-only evidence: this review targeted 9bd4d26 and was blocked by the former central coverage-evidence path, not a current pg-llm-batch source finding. The PR is now at a1c224c with fresh repository CI/Security/SAST success and resolved current threads. Dismissal is not approval and does not waive required current-head workflows.
There was a problem hiding this comment.
Pull request overview
OpenCode cannot approve yet because required coverage evidence did not pass.
Review outcome
1. HIGH .github/workflows/opencode-review.yml:1 - Coverage evidence did not prove required test/docstring evidence
-
Problem: The required coverage-evidence job result was
failure, so OpenCode cannot establish approval sufficiency for this head. -
Root cause: Automated approval is only valid when the same-head coverage-evidence job proves supported repository test suites passed and configured docstring gates passed or were advisory, or reports not applicable because no supported source files or package manifests exist. Missing, failed, skipped, unavailable, or unsupported-tooling test evidence is a blocker.
-
Fix: Install or configure the repository test/docstring evidence tooling when source files or package manifests exist, rerun the current-head coverage-evidence job, and approve only after it reports
successwith required evidence or explicit no-source not-applicable evidence. -
Regression test: Keep the approval branch checking
needs.coverage-evidence.result == successbefore posting APPROVE, and publish REQUEST_CHANGES when coverage-evidence blocker states such as cancelled, skipped, failed, unsupported-tooling, or below-100 evidence are present. -
Result: REQUEST_CHANGES
-
Reason: coverage-evidence result was
failure, so required test/docstring evidence was not proven for current heada1c224c519a595341026392c989b023eeaa148a9. -
Head SHA:
a1c224c519a595341026392c989b023eeaa148a9 -
Workflow run: 31528245568
-
Workflow attempt: 1
Coverage evidence
Coverage evidence job did not run or did not publish coverage evidence.
Changed-File Evidence Map
flowchart LR
PR["PR changed files"] --> Evidence["OpenCode bounded evidence"]
Evidence --> S1["Workflow: ci.yml"]
S1 --> I1["GitHub Actions review job"]
I1 --> R1["Review risk: Workflow: ci.yml"]
R1 --> V1["actionlint plus required checks"]
Evidence --> S2["Changed file: CHANGELOG.md"]
S2 --> I2["repository behavior"]
I2 --> R2["Review risk: Changed file: CHANGELOG.md"]
R2 --> V2["required checks"]
Evidence --> S3["Docs: exact-source-ci-evidence.md"]
S3 --> I3["operator or user guidance"]
I3 --> R3["Review risk: Docs: exact-source-ci-evidence.md"]
R3 --> V3["docs review"]
Evidence --> S4["Test: test_workflow_contracts.py"]
S4 --> I4["regression suite"]
I4 --> R4["Review risk: Test: test_workflow_contracts.py"]
R4 --> V4["targeted test run"]
1 participant
RCA
Protected
mainCI checks out GitHub's syntheticpull_requestmerge ref by default, while repository acceptance needs independently attributable evidence from the exact pull-request source head. #88 is the bounded repository-local repair; scheduler hardening remains separately owned by #69.Bounded implementation
actions/checkoutin.github/workflows/ci.ymlbindswith.refto${{ github.event.pull_request.head.sha || github.sha }};with.persist-credentials: false;Verify exact source headstep comparesgit rev-parse HEADwith the same expected SHA;github.shafallback;uses,with.ref,with.persist-credentials,name, andrun.Test-first and review repair
The original exact-source contract was test-first. CodeRabbit later found real assurance weaknesses in the contract harness: global string counting, substring field matching, and omission of the checkout credential-isolation invariant. Each was repaired narrowly. Current source
a1c224c519a595341026392c989b023eeaa148a9enforces both exactwith.refandwith.persist-credentials: false; all currently returned inline review threads are resolved.Three predecessor-head OpenCode
CHANGES_REQUESTEDsubmissions are dismissed as stale infrastructure-only evidence. Dismissal is not approval and waives no current-head gate.Current exact-source proof
a1c224c519a595341026392c989b023eeaa148a9.main:bf2cc2e140dc3ff4a56c3203f80f41bb9fed5d10.31521712803: completed / success. Python 3.10/3.12/3.14, 351 non-integration tests with 3 deselected, exact 100% owned production statement/branch coverage, 100% public docstrings, Ruff, lock freshness, package builds, Compose validation, and both container builds passed. CI jobs checked out this exact contributor head and passed the immediategit rev-parse HEADidentity check with checkout credentials disabled.31521712785: completed / success.31521712819: completed / success.31521709437: completed / success.31521711331: completed / success.31521711307: completed / success.31521711298: completed / success.31521711297:required-workflow-bootstrap,coverage-source-tree, andcoverage-evidenceare completed / success; the final requiredopencode-reviewjob remains queued on GitHub-hostedubuntu-latestwith no runner assigned at the latest refetch. Queued evidence is not success.The final OpenCode queue is a current control-plane/capacity wait, not evidence of a pg-llm-batch source defect. Do not retrigger, cancel, weaken, or replace it while the exact source remains unchanged.
Live review-policy classification
Fresh organization ruleset inspection reports
required_approving_review_count: 0,require_code_owner_review: false,require_last_push_approval: false, andrequired_review_thread_resolution: true. ProtectedAGENTS.mdseparately keeps code-owner review disabled/on hold for the solo-maintainer state. Independent approval is therefore not a current repository-policy prerequisite. No COMMENTED/status/model/author evidence is treated as approval, and no review result substitutes for required quality/security/source-identity workflow evidence.Read-only central dependency
ContextualWisdomLab/.github#790remains read-only from this loop. Its historical review state does not establish a current pg-llm-batch source finding and does not add an approval requirement beyond live leaf policy. No leaf workaround or central mutation is permitted here.Merge boundary
Do not merge until the unchanged current head has terminal-success for every live required workflow, including the currently queued final Required OpenCode
opencode-reviewjob, zero valid unresolved findings, and unchanged mergeability/ruleset state after a final exact-head/live-base refetch. Pending, queued, cancelled, skipped-required, absent, neutral-required, stale-head, predecessor, status-only, synthetic-only, author-only, infrastructure-only, or failed evidence is not acceptance.