Repository navigation
CI: project agent — silence-by-default bumps, gh shim enforcement, no orchestration issues, weekly cadence - #497
Merged
Merged
Conversation
… orchestration issues, weekly cadence Daily bumps carried zero new information (#330 got 19 in 6 weeks, 9 after a human replied with a linked PR) and every run spawned a throwaway orchestration issue (33/month). Silence is now the default outcome, comment rules are enforced mechanically by a gh shim rather than relying on prompt compliance, findings post to one long-lived log issue only when something happened, and the cadence drops to weekly. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem
The project agent workflow (
.github/workflows/project.yml,.github/prompts/project.md) was extremely spammy:**Project assessment bump (date)** Still relevant.comment on every run. Deprecate dead pre-event-sourcing tables (rename first, drop later) + delete dead code referencing them #330 got 19 such bumps in 6 weeks, 9 of them after the maintainer had already commented with a linked PR. Historically ReminderService uses invalid Task casts that fail at runtime #28/NotificationService uses invalid Task cast that fails at runtime #29 got 28 bumps each.--bodyinstead of--body-file), so prompt-only enforcement wasn't reliable — this needed a mechanical backstop.Changes
Prompt rewrite (
.github/prompts/project.md): deleted the "bump if exists" rule and its template. Rule 2 (search before creating) now defaults to doing nothing when a matching open issue exists — open already means "still relevant". New rule 3: a comment on an existing finding is allowed only to close it as resolved, or to add evidence that actually changed — never to restate what's already there — with hard exclusions (never comment if a human commented, a PR references it, or the last agent comment is <14 days old). All references to the orchestration issue are gone; the agent's final action is now writing its summary to$AGENT_SUMMARY_FILE, and it's told$AGENT_LOG_ISSUEexists and must never be touched.Mechanical enforcement (
scripts/agent-shim/gh): a bash shim placed first onPATHfor the agent step. It resolves the realgh, and refuses (REFUSED by project-agent gh shim: ..., exit 1)gh issue commentwhen the issue is the log issue, has a human comment, is referenced by a PR (via the timeline API), or was bot-commented on withinAGENT_COMMENT_THROTTLE_DAYS(14d); refusesgh issue closeon the log issue. Everything else passes through. Every call appends one line to$AGENT_ACTIONS_LOG(ALLOWED/REFUSED/PASSTHROUGH) — this log is the evidence the shim was actually live inside the codex sandbox.scripts/agent-shim/test.shexercises all 7 rules against a fakegh(19/19 assertions pass, no network). Both scripts are shellcheck-clean.No more per-run orchestration issue (
.github/workflows/project.yml): removed thecleanupjob and per-run issue creation. Added "Ensure project agent log issue" (finds-or-creates one long-lived, pinned, openProject agent logissue) and a PATH/self-test step that fails the run ifghdoesn't resolve to the shim. Post-agent steps (if: always()) assert the agent wrote a summary file and that the actions log is non-empty (empty = shim never reached inside the sandbox = hard failure, never a silent "clean day"), publish both to the step summary, and post to the log issue only when the actions log contains at least oneALLOWEDline. Clean runs post nothing anywhere.Weekly cadence: cron
37 4 * * 1, workflow renamedWeekly Project Assessment. Updatedworkflow-canary.yml's stall threshold (7d2h) andagent-retry.yml's workflow-name match. Grepped the tracked tree for remaining "Daily Project Assessment" references — none left outside these files.Verification
actionlintclean on all three touched workflows;shellcheckclean on both shim scripts;scripts/agent-shim/test.sh19/19 pass;scripts/check-comment-ratio.sh --cachedpasses (comment blocks capped at 2 lines everywhere, including the shim scripts).gh resolves to the shim: /home/runner/work/bots/bots/scripts/agent-shim/gh✅agent_posted=true,shim_reached=true(actions log non-empty) ✅gh issue comment/create/closecalls on it. The onlyghcall made all run wasgh issue list --state all --label project ...(a search), which the shim logged asPASSTHROUGH issue listand let through — proving the shim was live inside the sandbox.ALLOWEDline, "Post to log issue when actions were taken" correctly posted nothing:No ALLOWED actions this run — nothing to post.REFUSEDpath wasn't mechanically exercised live (nothing was attempted to refuse), but it is covered by the 19/19 offlinetest.shassertions (human-comment, PR-reference, and throttle cases all assertREFUSED).Follow-ups (not in this PR)
product.yml/product.mdhave the identical per-run orchestration-issue pattern; deliberately out of scope here.ghshim is reusable —product.ymlcould adopt it the same way if its own bump/comment behavior ever needs the same mechanical backstop.🤖 Generated with Claude Code