[kelos-self-update] Prompt Tuning: reviewers can approve a superseded PR head #1412

🤖 Kelos Self-Update Agent @gjkim42

Latest verdict

• Status: VALID
• Checked at: 2026-10-26T06:03:18Z
• Evidence: The default branch is now 68257b1, from #1841. No commit since 3040ff1 (2026-10-15) touches self-development/. The gap remains: kelos-reviewer and the PR path of kelos-api-reviewer still only review git diff origin/main...HEAD. Neither the regular sticky template nor the API sticky template has a reviewed-commit field. A search of self-development/ and internal/examples/ finds no headRefOid or "reviewed commit". New case: an API PR merged on a head neither reviewer saw. On #1841 (kind/api), the Kelos: kelos-api-reviewer Check Run (21:43:35Z to 21:48:55Z) and the Kelos: kelos-reviewer Check Run (21:48:23Z to 21:55:01Z) on 2026-10-25 both ran on 37aaa96. They posted the API APPROVE sticky (last updated 21:48:31Z) and the regular APPROVE sticky (21:54:38Z). At 22:07:03Z the branch was force-pushed to e7e6184, which has the same parent 84b993d. That push changed 6 files (+99/−8). It added symlink resolution to --skill directories in internal/cli/skill.go and new kubelet projected-path byte limits to validateSkillFilePath in internal/controller/job_builder.go. No Kelos Check Run exists on e7e6184. The PR merged at 22:33:29Z as 68257b1 with both stickies still showing APPROVE and naming no commit. The predicted case on Integrate native Session goals and loops across agents #1840 has happened: the APPROVE sticky, posted on dbf066e at 2026-10-24T11:20:01Z, is still in place after force-pushes to 360da14 (2026-10-25T22:29:08Z) and 47ee6ec (22:58:54Z). No Kelos Check Run exists on either new head, yet the comment still reads as a current approval. The earlier cases #1839, #1836, #1828, and #1829 still apply. So does #1823, still open at 2ee8f9d with APPROVE stickies from before its 2026-10-16 force-push. No overlap: #1249 is still at 5cee024. Its reviewer-file hunks add only checklist items and no head binding. Open PRs #1840, #1843, #1844, and #1831 touch nothing under self-development/, internal/examples/, or .github/workflows/. Issue and PR searches for headRefOid and "reviewed commit" find no duplicate beyond this slot. Configuration still aligned: the base-agent.yaml agentsMD is byte-identical (4,667 bytes) to gjkim42/kanon-repo instructions/AGENTS.md, which last changed at 27c4781. The kanon-repo HEAD is still b30d6f4, and the skills source is still gjkim42/kanon-repo. All 12 root spawners and tasks/fake-strategist-task.yaml list base-agent first. The newest merged generated-by-kelos PRs are image bumps from 2026-10-25: #1833, #1835, #1837, #1838, and #1842. Weaker alternatives unchanged: the kelos/needs-input leftovers are only noise. They are kelos-squash-commits step 7, the kelos-api-reviewer closing line, and README line 224 and line 366. #969 and #544 still cover the drift in tasks/fake-strategist-task.yaml.

Area: Prompt Tuning

Summary

Kelos reviewer prompts can publish or keep an APPROVE sticky comment for a commit that is no longer the PR head. The comments do not name the reviewed commit. The agents do not verify that their checkout matches the live PR head before analysis, and they do not recheck the head right before publishing.

The gap continues after publication. Maintainers, pick-up Sessions, and kelos-squash-commits can all push after a verdict is published. Because the sticky comment names no commit, nobody can tell from the comment alone whether the verdict still covers the current head.

I chose this candidate after reviewing the root self-development spawner prompts, the 20 most recent merged PRs labeled generated-by-kelos, the 20 most recent generated issues, and current open generated issues for overlap. It has the highest impact of the prompt-tuning candidates found.

Current prompt gap

• kelos-reviewer.yaml fetches main and reviews origin/main...HEAD on the branch Kelos checked out. It never reads headRefOid, verifies the checkout against it, or rechecks it before publication.
• kelos-api-reviewer.yaml has the same gap on its PR path. Its issue-review path does not need commit binding.
• The regular sticky template and the API sticky template contain a verdict and scope but no reviewed SHA.
• The reporting.checks Check Run does not bind the verdict. The webhook handler attaches it to the head SHA it resolves when the command arrives, and it reports only task success or failure. The review verdict lives only in the sticky comment.

The reviewers need their own guard. A maintainer, a pick-up Session, or another agent can change the head while a review is running or after it has been published. The Session prompts deliberately leave follow-up steps to the agent and the maintainer (PR #1641). The sticky comment is therefore the only place that can show which commit a verdict covers.

Evidence from agent activity

The head changed between a review request and publication

On PR #1602, a maintainer posted /kelos review at 07:55:00 UTC on 2026-08-08. The PR was force-pushed to final head f448565 at 07:59:52 UTC. The sticky approval, first created on August 6, was updated at 08:02:06 UTC on August 8, before merge. Its body names no reviewed SHA. These timestamps show the branch changed between the request and publication, but they do not show which commit was analyzed. This is evidence of missing traceability, not proof of an outdated review.

A regular verdict went stale after later pushes

On PR #1569, the regular reviewer published its last sticky verdict at 10:29:40 UTC on 2026-08-08. The branch was force-pushed at 10:29:58 and again at 10:40:18, ending at head 0906431. Only the API reviewer ran after that, and its prompt deliberately leaves general correctness to the regular reviewer. The PR merged without a current regular verdict.

A surviving approval described a different diff

On PR #1496, the reviewer published an APPROVE describing a serial Session StatefulSet restart change. The branch was then force-pushed and merged without another regular review. The reviewed commit 8e56ed8 changed only .github/workflows/deploy-dev.yaml. The final head 51b1344 changed only internal/sessionruntime/codex.go and internal/sessionruntime/server_test.go. The approval that remained described a diff sharing no files with what merged.

Proposed minimal prompt changes

1. In kelos-reviewer and on the PR path of kelos-api-reviewer, read the PR's live headRefOid before analysis. Fetch and check out that exact commit, and confirm that git rev-parse HEAD matches it.
2. Read headRefOid again right before creating or updating the sticky comment. If it changed, do not publish the old verdict. Either review the complete new diff, or publish a clearly non-approving stale-run status that records both SHAs and asks for a fresh run.
3. Add **Reviewed commit**: <full SHA> to both PR sticky-review templates. Leave issue-only API review comments unchanged.

Out of scope

• Review-handoff steps in kelos-workers and kelos-pr-responder. PR #1641 deliberately reduced these Session prompts to the target URL and left workflow steps to the shared agent configuration. Adding a handoff checklist would reverse that decision.
• Automatic re-review from kelos-squash-commits. It already hands the PR back to the maintainer after the force-push by posting Squash complete.. Once each sticky comment records its reviewed SHA, the maintainer can see when a verdict predates the squash and post /kelos review again.
• Issue-only API reviews. They have no PR head to bind.

Related but not duplicate

• Closed issue #752 proposed re-triggering polling sources on push. These reviewers are webhook spawners triggered by commands, and a single in-flight review still needs commit binding.
• Closed issue #962 proposed deduplicating and debouncing webhook events. It does not cover binding a verdict to a commit.
• Open issues #1089 and #1203 concern routing reviewer commands to PRs rather than plain issues. They do not address a PR head changing during or after review.
• Targeted issue and PR searches for headRefOid, reviewed commits, stale reviewer heads, and superseded heads found no duplicate beyond this slot. No open PR implements the proposed binding.

Acceptance criteria

• Both PR reviewers (kelos-reviewer and the PR path of kelos-api-reviewer) verify that their checkout matches the live PR head before analysis, and include the full reviewed SHA in their sticky comments.
• Both reviewers recheck the live PR head right before publication. If the head changed, no APPROVE or REQUEST CHANGES verdict is presented as current: the agent either reviews the new head or publishes a non-approving stale-run status.
• Focused tests in internal/examples/self_development_test.go pin the commit-binding instructions and the reviewed-commit field in both PR reviewers.
• The kelos-workers and kelos-pr-responder Session prompts, kelos-squash-commits, and issue-only API review behavior are unchanged.