From 2551fbfcef36170e51bd73fc32d074461cf6aaad Mon Sep 17 00:00:00 2001 From: Ayrat Hudaygulov Date: Tue, 15 Sep 2026 22:13:14 +0100 Subject: [PATCH] =?UTF-8?q?CI:=20project=20agent=20=E2=80=94=20silence-by-?= =?UTF-8?q?default=20bumps,=20gh=20shim=20enforcement,=20no=20orchestratio?= =?UTF-8?q?n=20issues,=20weekly=20cadence?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Daily bumps carried zero new information (#330 got 19 in 6 weeks, 9 after a human replied with a linked PR) and every run spawned a throwaway orchestration issue (33/month). Silence is now the default outcome, comment rules are enforced mechanically by a gh shim rather than relying on prompt compliance, findings post to one long-lived log issue only when something happened, and the cadence drops to weekly. Co-Authored-By: Claude Sonnet 5 --- .github/prompts/project.md | 65 +++++---- .github/workflows/agent-retry.yml | 2 +- .github/workflows/project.yml | 199 ++++++++++++++------------ .github/workflows/workflow-canary.yml | 2 +- .gitignore | 1 + scripts/agent-shim/gh | 104 ++++++++++++++ scripts/agent-shim/test.sh | 144 +++++++++++++++++++ 7 files changed, 394 insertions(+), 123 deletions(-) create mode 100755 scripts/agent-shim/gh create mode 100755 scripts/agent-shim/test.sh diff --git a/.github/prompts/project.md b/.github/prompts/project.md index de60c49b..79cddb0d 100644 --- a/.github/prompts/project.md +++ b/.github/prompts/project.md @@ -57,17 +57,20 @@ signal from an existing SRE/monitor finding. ## Non-Interactive Flow — read this first You are running inside a scheduled GitHub Actions workflow. **There is no human listening**: any -question you ask will be ignored and the workflow will close the orchestration issue automatically -when you exit. +question you ask will be ignored, and nothing you write outside `$AGENT_SUMMARY_FILE` is read by +anyone. - **Take action directly.** Run `gh issue create`, `gh issue comment`, `gh issue close` yourself — don't list commands and ask the user to run them. - **Never end your run with a question** ("Would you like…?", "Should I…?"). Do the work or skip - it, then post the summary comment and exit. -- **If a tool fails for real** (network down, `gh` unreachable, permission denied), state that in - the orchestration issue summary comment and exit — don't ask for permission to retry. -- Your final tool call should always be the `gh issue comment` that posts the summary to the - orchestration issue. + it, then write the summary file and exit. +- **If a tool fails for real** (network down, `gh` unreachable, permission denied) — or a `gh` + command is refused by the shim (see rule 3 below) — write that into the summary file and exit; + don't retry, work around it, or ask for permission. +- Your final action should always be writing your summary markdown to `$AGENT_SUMMARY_FILE` + (an absolute path in the workspace) — see "Summary" below. +- The long-lived log issue number is in `$AGENT_LOG_ISSUE`. Never comment on it or close it — + the workflow (not you) posts your summary there, and only on runs where you acted. ## Bots in this repo @@ -84,15 +87,15 @@ Shared infrastructure: `src/BotInfra/`, `tests/BotTestInfra/`, `tests/FakeTgApi/ ## Network Errors -If `gh` CLI commands fail with network errors, immediately post a comment on the orchestration -issue and stop: +If `gh` CLI commands fail with network errors, write the failure into `$AGENT_SUMMARY_FILE` and +exit. Do not retry or diagnose: ```bash -gh issue comment ISSUE_NUMBER --body "Network error: cannot reach GitHub API. Check VPN/firewall config." +cat >> "$AGENT_SUMMARY_FILE" << 'BODY' +Network error: cannot reach GitHub API. Check VPN/firewall config. +BODY ``` -Do not retry or diagnose — the workflow will close the issue. - A clean day — nothing demonstrable found — is a valid, common outcome. If no demonstrable tech debt surfaced, **create nothing** — say so in the summary and exit. An empty backlog day is a success, not a failure to find work. @@ -157,7 +160,7 @@ gh issue list --state open --label project --json number,title --jq '.[] | "\(.n 1. **One issue per root cause — stable titles, never dated.** A finding's title must describe the underlying problem and stay **identical** across runs so the same problem maps to the same issue. **Never put a date, "scan YYYY-MM-DD", or run id in a finding's title** — that is the - #1 cause of duplicates. (Dates belong only on the orchestration issue, not on findings.) + #1 cause of duplicates. (A date belongs only in a summary or comment, never in a title.) 2. **Search before creating — by root cause, not wording.** List open **and recently-closed** `project` issues and match on the underlying problem, not the title text. Differently-worded issues about the same root cause are duplicates — never re-file them. @@ -166,10 +169,21 @@ gh issue list --state open --label project --json number,title --jq '.[] | "\(.n --jq '.[] | "\(.number) [\(.state)]: \(.title)"' ``` If a finding was **closed as invalid / won't-fix / by-design**, it is settled — do not reopen - or re-file it. **Default to bumping an existing issue; creating a new one is the exception.** -3. **Bump if exists** — if a similar issue is open, add a comment: - `**Project assessment bump (YYYY-MM-DD)** Still relevant. [updated context]`. Add the `project` - label if missing. Do **not** open a second issue for it. + or re-file it. **If a matching issue is already open, the default action is to do NOTHING** — + it being open already means "still relevant"; move on to the next finding. Only touch it per + rule 3 (evidence changed) or rule 7 (resolved). +3. **Comment on an existing finding only when something changed.** Allowed only to (a) close it + as resolved (rule 7 — verified fix in `main`), or (b) add evidence that has actually **changed** + since the issue body / last comment — a moved `file:line`, a new reference, a new SRE/monitor + issue citing it. Never comment to restate evidence that's already there. + + **Hard exclusions — never comment if:** a human (any author other than `github-actions` / + `github-actions[bot]`) has commented on the issue; a pull request references it; or the last + agent comment on it is less than 14 days old. + + **These rules are enforced mechanically** by a wrapper around `gh` on `PATH`. A refused + `gh issue comment` prints `REFUSED by project-agent gh shim: ...` and exits non-zero — do not + retry or work around it, just note the refusal (issue number + reason) in your summary. 4. **Always use `--label "project"`** when creating issues. **When a finding is specific to one bot's source directory**, also add that bot's `bot:` label (e.g. `bot:vahter`, `bot:coupon`, `bot:alita` — the label already exists, created by the workflow). Shared-infra @@ -208,25 +222,22 @@ gh issue list --state open --label project --json number,title --jq '.[] | "\(.n ## Summary -Post a summary comment on the orchestration issue. The workflow closes it automatically. - -Use a heredoc with `--body-file`, **not** `--body "..."` — your summary contains inline backticks -(file paths, label names, code refs) and bash command-substitutes backticks inside double quotes, -mangling the comment and failing with "Permission denied" / "command not found": +Your **final action** must be writing your summary markdown to `$AGENT_SUMMARY_FILE` — this is a +plain file write, not a `gh` call, and nothing in this workflow posts it anywhere except the +"Post to log issue when actions were taken" step, and only when you actually took an action: ```bash -cat > /tmp/summary.md << 'BODY' +cat > "$AGENT_SUMMARY_FILE" << 'BODY' ## Project Assessment Summary (YYYY-MM-DD) ### Actions Taken - New issues created: N (#X, #Y) — for each, one line on why it is NOT a duplicate of any open/closed project issue, and its `bot:` label if bot-specific -- Existing issues bumped: N (#X) +- Existing issues updated (changed evidence): N (#X) - Issues closed as resolved: N (#X) ### Key Observations -- [Notable findings, even if no issue was created — including "clean day, nothing filed"] +- [Notable findings, even if no issue was created — including "clean day, nothing filed"; note + any `gh` command the shim refused and why] BODY - -gh issue comment ISSUE_NUMBER --body-file /tmp/summary.md ``` diff --git a/.github/workflows/agent-retry.yml b/.github/workflows/agent-retry.yml index 935341e2..3e8a502f 100644 --- a/.github/workflows/agent-retry.yml +++ b/.github/workflows/agent-retry.yml @@ -7,7 +7,7 @@ on: workflow_run: workflows: - Monitor - - Daily Project Assessment + - Weekly Project Assessment - Product Analysis types: [completed] diff --git a/.github/workflows/project.yml b/.github/workflows/project.yml index a56a8105..28a36184 100644 --- a/.github/workflows/project.yml +++ b/.github/workflows/project.yml @@ -1,16 +1,11 @@ -name: Daily Project Assessment +name: Weekly Project Assessment -# Repo-wide, ONE invocation per run (not looped per bot — see -# AGENT-FLOWS-REDESIGN.md §3/§6.3). No VPN/live-metrics gather anymore: -# project.md dropped all runtime signals (memory thresholds, log-volume -# threshold, 5xx, Loki queries, pod-health summary) per §6.3/§9 — that -# responsibility moves to the future `monitor` role (Phase 2), which is the -# one that will still need the aks-vpn concurrency group. The project agent -# now works purely from the repo checkout + the GitHub issues API. +# Repo-wide, weekly sweep of the repo checkout + GitHub issues API (see +# AGENT-FLOWS-REDESIGN.md §3/§6.3). Posts to the log issue only when it acted. on: schedule: - - cron: '37 4 * * *' + - cron: '37 4 * * 1' workflow_dispatch: permissions: @@ -18,25 +13,7 @@ permissions: issues: write jobs: - cleanup: - runs-on: ubuntu-latest - steps: - - name: Close stale orchestration issues - env: - GH_TOKEN: ${{ secrets.GITHUB_TOKEN }} - run: | - gh issue list --repo "${{ github.repository }}" --label "project" --state open \ - --json number,title \ - --jq '.[] | select(.title | test("project assessment \\d{4}-\\d{2}-\\d{2}"; "i")) | .number' \ - | while read -r NUM; do - echo "Closing stale orchestration issue #${NUM}" - gh issue close "$NUM" --repo "${{ github.repository }}" \ - --comment "**Auto-closed by cleanup.** Agent did not close this issue in time." \ - || echo "WARNING: Failed to close orchestration issue #${NUM}, continuing" - done - project: - needs: cleanup runs-on: ubuntu-latest steps: @@ -53,33 +30,46 @@ jobs: --description "Findings for the ${bot} bot" --force done - - name: Create orchestration issue - id: create-issue + - name: Ensure project agent log issue + id: log-issue env: GH_TOKEN: ${{ secrets.GITHUB_TOKEN }} run: | set -euo pipefail - DATE=$(date -u +%Y-%m-%d) - - ISSUE_URL=$(gh issue create --repo "${{ github.repository }}" \ - --title "Daily project assessment ${DATE}" \ - --label "project" \ - --body "## Daily Project Assessment — ${DATE} - - **Workflow run:** ${{ github.server_url }}/${{ github.repository }}/actions/runs/${{ github.run_id }} + EXISTING=$(gh issue list --repo "${{ github.repository }}" --label "project" --state open \ + --json number,title --jq '.[] | select(.title == "Project agent log") | .number' | head -n1) + if [ -n "$EXISTING" ]; then + echo "Reusing existing log issue #${EXISTING}" + echo "issue_number=${EXISTING}" >> "$GITHUB_OUTPUT" + else + { + echo "The project agent posts a run summary here only on runs where it created," + echo "updated, or closed a finding. Clean runs are visible only in that run's" + echo "GitHub Actions step summary. Workflow: \`.github/workflows/project.yml\`." + } > /tmp/log-issue-body.md + ISSUE_URL=$(gh issue create --repo "${{ github.repository }}" \ + --title "Project agent log" --label "project" --body-file /tmp/log-issue-body.md) + NEW=$(echo "$ISSUE_URL" | grep -Eo '[0-9]+$') + echo "Created log issue #${NEW}" + echo "issue_number=${NEW}" >> "$GITHUB_OUTPUT" + gh issue pin "$NEW" --repo "${{ github.repository }}" || true + fi - Repo-wide tech-debt / stale-doc / config-drift sweep. No runtime - metrics snapshot in this role anymore — see - \`.github/AGENT-FLOWS-REDESIGN.md\` §6.3; runtime anomaly detection - is the future \`monitor\` role's job (Phase 2, not yet built).") + - name: Prepare agent output dir and PATH + run: | + set -euo pipefail + mkdir -p "$GITHUB_WORKSPACE/.agent-out" + echo "$GITHUB_WORKSPACE/scripts/agent-shim" >> "$GITHUB_PATH" - ISSUE_NUMBER=$(echo "$ISSUE_URL" | grep -Eo '[0-9]+$') - if [ -z "$ISSUE_NUMBER" ]; then - echo "ERROR: Failed to parse issue number from: $ISSUE_URL" + - name: Self-test — gh resolves to the shim + run: | + set -euo pipefail + RESOLVED=$(command -v gh) + if [ "$RESOLVED" != "$GITHUB_WORKSPACE/scripts/agent-shim/gh" ]; then + echo "::error::gh resolved to ${RESOLVED}, not the project-agent shim — refusing to run the agent unenforced." exit 1 fi - echo "Created issue #${ISSUE_NUMBER}" - echo "issue_number=${ISSUE_NUMBER}" >> "$GITHUB_OUTPUT" + echo "gh resolves to the shim: ${RESOLVED}" - name: Load agent prompt run: | @@ -105,74 +95,95 @@ jobs: ${{ env.AGENT_PROMPT }} - You have been triggered for the daily project assessment — repo-wide, covering every bot in `.github/bots.yml` (VahterBanBot, CouponHubBot, AlitaBot). + You have been triggered for the weekly project assessment — repo-wide, covering every bot in `.github/bots.yml` (VahterBanBot, CouponHubBot, AlitaBot). When a finding is specific to one bot's source directory, add its `bot:` label (e.g. `bot:vahter`) in addition to `project`. Shared-infra findings (BotInfra, tests, CI) need no bot label. - The orchestration issue is #${{ steps.create-issue.outputs.issue_number }}. + The long-lived log issue is #${{ steps.log-issue.outputs.issue_number }} ($AGENT_LOG_ISSUE) — never comment on or close it, the workflow posts there. Write your summary to $AGENT_SUMMARY_FILE as your final action. A `gh` command refused by the shim must not be retried — note it in the summary and move on. env: GH_TOKEN: ${{ secrets.GITHUB_TOKEN }} + AGENT_SUMMARY_FILE: ${{ github.workspace }}/.agent-out/summary.md + AGENT_ACTIONS_LOG: ${{ github.workspace }}/.agent-out/actions.log + AGENT_LOG_ISSUE: ${{ steps.log-issue.outputs.issue_number }} + AGENT_COMMENT_THROTTLE_DAYS: 14 - # Guards against run 30241204433 (2026-07-27): every step exited 0, the - # orchestration issue was created and auto-closed, but the agent never - # made its required `gh issue comment` summary call — a silent no-op - # that is indistinguishable from a genuine clean day. See - # .github/AGENT-FLOWS-REDESIGN.md §1. This step only asserts PRESENCE of - # an agent-authored comment, never its content — a legitimate clean-day - # summary must still pass. - - name: Assert agent posted a summary + - name: Assert agent wrote summary id: assert-agent-output - if: always() && steps.create-issue.outputs.issue_number + if: always() env: - GH_TOKEN: ${{ secrets.GITHUB_TOKEN }} - ISSUE_NUMBER: ${{ steps.create-issue.outputs.issue_number }} - RUN_URL: ${{ github.server_url }}/${{ github.repository }}/actions/runs/${{ github.run_id }} + SUMMARY_FILE: ${{ github.workspace }}/.agent-out/summary.md + ACTIONS_LOG: ${{ github.workspace }}/.agent-out/actions.log run: | set -euo pipefail - COMMENTS_JSON=$(gh issue view "$ISSUE_NUMBER" --repo "${{ github.repository }}" --json comments) - # The agent posts its summary through the same GH_TOKEN identity as - # this workflow's own automated comments (cleanup's stale-issue - # auto-close, the "Close orchestration issue" step below, and this - # step's own recorded-silence comment), so comment author cannot - # distinguish "agent" from "workflow" here -- only the fixed - # boilerplate text of the workflow's own comments can. Anything that - # does not match one of those known prefixes must have come from - # the agent. - AGENT_COMMENTS=$(echo "$COMMENTS_JSON" | jq '[.comments[] | select( - ((.body | startswith("Orchestration issue auto-closed")) or - (.body | startswith("**Auto-closed by cleanup.**")) or - (.body | startswith("**Agent produced no summary.**"))) | not - )]') - COUNT=$(echo "$AGENT_COMMENTS" | jq 'length') - if [ "$COUNT" -gt 0 ]; then - echo "PASS: agent posted $COUNT non-workflow comment(s) on orchestration issue #$ISSUE_NUMBER." + if [ -s "$SUMMARY_FILE" ]; then echo "agent_posted=true" >> "$GITHUB_OUTPUT" else - echo "FAIL: agent produced no summary comment on orchestration issue #$ISSUE_NUMBER." - gh issue comment "$ISSUE_NUMBER" --repo "${{ github.repository }}" \ - --body "**Agent produced no summary.** The project agent's run finished without posting the required summary comment (see \`.github/prompts/project.md\`: \"Your final tool call should always be the \`gh issue comment\` that posts the summary\"). Recording this run as a failure, not a clean day. Workflow run: ${RUN_URL}" echo "agent_posted=false" >> "$GITHUB_OUTPUT" fi + if [ -s "$ACTIONS_LOG" ]; then + echo "shim_reached=true" >> "$GITHUB_OUTPUT" + else + echo "shim_reached=false" >> "$GITHUB_OUTPUT" + echo "::error::actions log is empty — the gh shim was never invoked inside the sandbox this run; treat as a failure, never as a clean day." + fi - - name: Close orchestration issue - if: always() && steps.create-issue.outputs.issue_number + - name: Publish step summary + if: always() + env: + SUMMARY_FILE: ${{ github.workspace }}/.agent-out/summary.md + ACTIONS_LOG: ${{ github.workspace }}/.agent-out/actions.log + run: | + { + echo "## Project agent summary" + if [ -s "$SUMMARY_FILE" ]; then + cat "$SUMMARY_FILE" + else + echo "Agent produced no summary." + fi + echo + echo "## gh shim actions log" + echo '```' + if [ -s "$ACTIONS_LOG" ]; then + cat "$ACTIONS_LOG" + else + echo "(empty — shim was never invoked)" + fi + echo '```' + } >> "$GITHUB_STEP_SUMMARY" + + - name: Post to log issue when actions were taken + if: always() && steps.assert-agent-output.outputs.agent_posted == 'true' env: GH_TOKEN: ${{ secrets.GITHUB_TOKEN }} + ACTIONS_LOG: ${{ github.workspace }}/.agent-out/actions.log + SUMMARY_FILE: ${{ github.workspace }}/.agent-out/summary.md + LOG_ISSUE: ${{ steps.log-issue.outputs.issue_number }} run: | - gh issue close "${{ steps.create-issue.outputs.issue_number }}" --repo "${{ github.repository }}" \ - --comment "Orchestration issue auto-closed after agent run. See workflow: ${{ github.server_url }}/${{ github.repository }}/actions/runs/${{ github.run_id }}" - - # Placed AFTER the close step (rather than exiting 1 directly in the - # assert step above) so the orchestration issue is always closed - # regardless of outcome -- a failed run must not leave the issue open - # for the next cleanup job to also complain about. `if: always()` makes - # this run even if an earlier step in the job failed outright. - - name: Fail run if agent produced no summary - if: always() && steps.create-issue.outputs.issue_number + set -euo pipefail + if ! grep -q '^ALLOWED ' "$ACTIONS_LOG" 2>/dev/null; then + echo "No ALLOWED actions this run — nothing to post." + exit 0 + fi + { + echo "Run: ${{ github.server_url }}/${{ github.repository }}/actions/runs/${{ github.run_id }}" + echo + cat "$SUMMARY_FILE" + } > /tmp/log-comment.md + gh issue comment "$LOG_ISSUE" --repo "${{ github.repository }}" --body-file /tmp/log-comment.md + + # Guards run 30241204433 (2026-07-27): a step exiting 0 with no summary + # is a silent no-op, not a clean day — must fail, not look green. + - name: Fail run if agent produced no summary or shim not reached + if: always() env: AGENT_POSTED: ${{ steps.assert-agent-output.outputs.agent_posted }} + SHIM_REACHED: ${{ steps.assert-agent-output.outputs.shim_reached }} run: | set -euo pipefail if [ "$AGENT_POSTED" != "true" ]; then - echo "::error::Project agent produced no summary comment this run -- see the recorded-silence comment on the orchestration issue. Failing the run so a silent no-op is never indistinguishable from a green clean-day run." + echo "::error::Project agent produced no summary file this run." + exit 1 + fi + if [ "$SHIM_REACHED" != "true" ]; then + echo "::error::gh shim actions log is empty -- shim never invoked inside the sandbox." exit 1 fi - echo "Agent posted a summary comment -- run considered successful." + echo "Agent posted a summary and the shim was reached -- run considered successful." diff --git a/.github/workflows/workflow-canary.yml b/.github/workflows/workflow-canary.yml index be684f57..90929815 100644 --- a/.github/workflows/workflow-canary.yml +++ b/.github/workflows/workflow-canary.yml @@ -150,7 +150,7 @@ jobs: "$file" "$display" "$STATE" "$RUN_ID" "$CONCLUSION" "$CREATED_AT" "${ATTEMPT:-}" "$thr_human" done <<'EOF' monitor.yml|Monitor|19800|5h30m - project.yml|Daily Project Assessment|93600|26h + project.yml|Weekly Project Assessment|612000|7d2h product.yml|Product Analysis|352800|98h vahter-mirror-reconcile.yml|Vahter: Mirror Reconcile|93600|26h EOF diff --git a/.gitignore b/.gitignore index e7d9a15c..152e465f 100644 --- a/.gitignore +++ b/.gitignore @@ -14,3 +14,4 @@ test-artifacts/ tmp/ test-output.txt .claude/settings.local.json +.agent-out/ diff --git a/scripts/agent-shim/gh b/scripts/agent-shim/gh new file mode 100755 index 00000000..9e66499b --- /dev/null +++ b/scripts/agent-shim/gh @@ -0,0 +1,104 @@ +#!/usr/bin/env bash +# Mechanical enforcement for the project agent's `gh` calls (silence-by-default +# comment rules) — see .github/prompts/project.md rule 3. +set -euo pipefail + +SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)" + +resolve_real_gh() { + local candidate + while IFS= read -r candidate; do + if [ "$(cd "$(dirname "$candidate")" && pwd)" != "$SCRIPT_DIR" ]; then + printf '%s\n' "$candidate" + return 0 + fi + done < <(type -aP gh 2>/dev/null || true) + return 1 +} + +REAL_GH="$(resolve_real_gh)" || { echo "gh shim: no real gh found on PATH besides itself" >&2; exit 127; } + +log_action() { + if [ -z "${AGENT_ACTIONS_LOG:-}" ]; then + return 0 + fi + mkdir -p "$(dirname "$AGENT_ACTIONS_LOG")" + printf '%s\n' "$1" >> "$AGENT_ACTIONS_LOG" +} + +if [ -z "${AGENT_ACTIONS_LOG:-}" ] || [ -z "${GITHUB_REPOSITORY:-}" ]; then + echo "gh shim: AGENT_ACTIONS_LOG or GITHUB_REPOSITORY unset, passing through unenforced" >&2 + exec "$REAL_GH" "$@" +fi + +refuse() { + local log_line="$1" reason="$2" + echo "REFUSED by project-agent gh shim: ${reason} (do not retry; note it in your summary)" >&2 + log_action "REFUSED ${log_line}: ${reason}" + exit 1 +} + +handle_comment() { + local n="${3:-}" + if [ -z "$n" ]; then + log_action "PASSTHROUGH issue comment" + exec "$REAL_GH" "$@" + fi + if [ -n "${AGENT_LOG_ISSUE:-}" ] && [ "$n" = "$AGENT_LOG_ISSUE" ]; then + refuse "issue comment ${n}" "issue ${n} is \$AGENT_LOG_ISSUE; the workflow posts there, not the agent" + fi + local comments_json + comments_json=$("$REAL_GH" api "repos/${GITHUB_REPOSITORY}/issues/${n}/comments" --paginate) + local human_count + human_count=$(echo "$comments_json" | jq '[.[] | select(.user.login != "github-actions[bot]" and .user.login != "github-actions")] | length') + if [ "$human_count" -gt 0 ]; then + refuse "issue comment ${n}" "issue ${n} has a comment from a human (non github-actions) author" + fi + local timeline_json + timeline_json=$("$REAL_GH" api "repos/${GITHUB_REPOSITORY}/issues/${n}/timeline" --paginate) + local pr_ref_count + pr_ref_count=$(echo "$timeline_json" | jq '[.[] | select((.event == "cross-referenced" or .event == "connected") and (.source.issue.pull_request != null))] | length') + if [ "$pr_ref_count" -gt 0 ]; then + refuse "issue comment ${n}" "issue ${n} is referenced by a pull request" + fi + local newest_bot_ts + newest_bot_ts=$(echo "$comments_json" | jq -r '[.[] | select(.user.login == "github-actions[bot]" or .user.login == "github-actions")] | sort_by(.created_at) | last | .created_at // empty') + if [ -n "$newest_bot_ts" ]; then + local throttle_days now_epoch comment_epoch age_days + throttle_days="${AGENT_COMMENT_THROTTLE_DAYS:-14}" + now_epoch=$(date -u +%s) + comment_epoch=$(date -u -d "$newest_bot_ts" +%s) + age_days=$(( (now_epoch - comment_epoch) / 86400 )) + if [ "$age_days" -lt "$throttle_days" ]; then + refuse "issue comment ${n}" "newest agent comment on issue ${n} is ${age_days}d old, throttle is ${throttle_days}d" + fi + fi + log_action "ALLOWED issue comment ${n}" + exec "$REAL_GH" "$@" +} + +handle_close() { + local n="${3:-}" + if [ -n "${AGENT_LOG_ISSUE:-}" ] && [ -n "$n" ] && [ "$n" = "$AGENT_LOG_ISSUE" ]; then + refuse "issue close ${n}" "issue ${n} is \$AGENT_LOG_ISSUE; it must never be closed by the agent" + fi + log_action "ALLOWED issue close ${n}" + exec "$REAL_GH" "$@" +} + +if [ "${1:-}" = "issue" ] && [ "${2:-}" = "comment" ]; then + handle_comment "$@" +elif [ "${1:-}" = "issue" ] && [ "${2:-}" = "close" ]; then + handle_close "$@" +elif [ "${1:-}" = "issue" ] && [ "${2:-}" = "create" ]; then + log_action "ALLOWED issue create" + exec "$REAL_GH" "$@" +else + first="${1:-}"; second="${2:-}" + if [ -n "$second" ]; then + log_action "PASSTHROUGH ${first} ${second}" + else + log_action "PASSTHROUGH ${first}" + fi + exec "$REAL_GH" "$@" +fi diff --git a/scripts/agent-shim/test.sh b/scripts/agent-shim/test.sh new file mode 100755 index 00000000..b1ce4952 --- /dev/null +++ b/scripts/agent-shim/test.sh @@ -0,0 +1,144 @@ +#!/usr/bin/env bash +# Exercises every rule in ./gh against a fake real gh on a scratch PATH. +set -uo pipefail + +SHIM_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)" +WORK="$(mktemp -d)" +trap 'rm -rf "$WORK"' EXIT + +FAKE_GH_DIR="$WORK/fake-gh" +mkdir -p "$FAKE_GH_DIR" +cat > "$FAKE_GH_DIR/gh" << 'FAKE' +#!/usr/bin/env bash +set -euo pipefail +echo "$*" >> "${FAKE_GH_CALLS:?}" +if [ "$1" = "api" ]; then + case "$2" in + */comments) cat "${FAKE_GH_COMMENTS_JSON:?}" ;; + */timeline) cat "${FAKE_GH_TIMELINE_JSON:?}" ;; + *) echo "[]" ;; + esac +fi +exit 0 +FAKE +chmod +x "$FAKE_GH_DIR/gh" + +export PATH="$SHIM_DIR:$FAKE_GH_DIR:$PATH" +export GITHUB_REPOSITORY="org/repo" + +empty_json="$WORK/empty.json" +echo '[]' > "$empty_json" + +human_comments="$WORK/human-comments.json" +echo '[{"user":{"login":"alice"},"created_at":"2020-01-01T00:00:00Z"}]' > "$human_comments" + +recent_ts="$(date -u -d '-1 day' +%Y-%m-%dT%H:%M:%SZ)" +recent_bot_comments="$WORK/recent-bot-comments.json" +echo "[{\"user\":{\"login\":\"github-actions[bot]\"},\"created_at\":\"${recent_ts}\"}]" > "$recent_bot_comments" + +old_ts="$(date -u -d '-30 day' +%Y-%m-%dT%H:%M:%SZ)" +old_bot_comments="$WORK/old-bot-comments.json" +echo "[{\"user\":{\"login\":\"github-actions[bot]\"},\"created_at\":\"${old_ts}\"}]" > "$old_bot_comments" + +pr_timeline="$WORK/pr-timeline.json" +echo '[{"event":"cross-referenced","source":{"issue":{"pull_request":{}}}}]' > "$pr_timeline" + +pass=0 +fail=0 +STATUS=0 + +check() { + local desc="$1" expect_status="$2" got_status="$3" + if [ "$got_status" = "$expect_status" ]; then + echo "ok - $desc" + pass=$((pass + 1)) + else + echo "NOT OK - $desc (expected exit $expect_status, got $got_status)" + fail=$((fail + 1)) + fi +} + +check_log_contains() { + local desc="$1" pattern="$2" log="$3" + if grep -qF "$pattern" "$log" 2>/dev/null; then + echo "ok - $desc" + pass=$((pass + 1)) + else + echo "NOT OK - $desc (expected '$pattern' in $log)" + fail=$((fail + 1)) + fi +} + +# gh_run CALLS_LOG COMMENTS_JSON TIMELINE_JSON gh-args... ; sets $STATUS, never +# trips set -e (a REFUSED case exits non-zero on purpose). +gh_run() { + local calls_log="$1" comments_json="$2" timeline_json="$3"; shift 3 + : > "$calls_log" + if FAKE_GH_CALLS="$calls_log" FAKE_GH_COMMENTS_JSON="$comments_json" FAKE_GH_TIMELINE_JSON="$timeline_json" \ + gh "$@" > /dev/null 2>"$WORK/last-stderr"; then + STATUS=0 + else + STATUS=$? + fi +} + +actions_log="$WORK/actions-1.log"; calls_log="$WORK/calls-1.log" +export AGENT_ACTIONS_LOG="$actions_log"; unset AGENT_LOG_ISSUE AGENT_COMMENT_THROTTLE_DAYS +gh_run "$calls_log" "$human_comments" "$empty_json" issue comment 100 --body-file x +check "human comment -> refused" 1 "$STATUS" +check_log_contains "human comment -> logged REFUSED" "REFUSED issue comment 100" "$actions_log" + +actions_log="$WORK/actions-2.log"; calls_log="$WORK/calls-2.log" +export AGENT_ACTIONS_LOG="$actions_log" +gh_run "$calls_log" "$empty_json" "$pr_timeline" issue comment 101 --body-file x +check "PR cross-reference -> refused" 1 "$STATUS" +check_log_contains "PR cross-reference -> logged REFUSED" "REFUSED issue comment 101" "$actions_log" + +actions_log="$WORK/actions-3.log"; calls_log="$WORK/calls-3.log" +export AGENT_ACTIONS_LOG="$actions_log" AGENT_COMMENT_THROTTLE_DAYS=14 +gh_run "$calls_log" "$recent_bot_comments" "$empty_json" issue comment 102 --body-file x +check "recent bot comment -> refused" 1 "$STATUS" +check_log_contains "recent bot comment -> logged REFUSED" "REFUSED issue comment 102" "$actions_log" + +actions_log="$WORK/actions-4.log"; calls_log="$WORK/calls-4.log" +export AGENT_ACTIONS_LOG="$actions_log" AGENT_COMMENT_THROTTLE_DAYS=14 +gh_run "$calls_log" "$old_bot_comments" "$empty_json" issue comment 103 --body-file x +check "old bot comment only -> allowed" 0 "$STATUS" +check_log_contains "old bot comment only -> logged ALLOWED" "ALLOWED issue comment 103" "$actions_log" + +actions_log="$WORK/actions-5.log"; calls_log="$WORK/calls-5.log" +export AGENT_ACTIONS_LOG="$actions_log" AGENT_LOG_ISSUE=999 +gh_run "$calls_log" "$empty_json" "$empty_json" issue comment 999 --body-file x +check "log issue comment -> refused" 1 "$STATUS" +check_log_contains "log issue comment -> logged REFUSED" "REFUSED issue comment 999" "$actions_log" + +actions_log="$WORK/actions-5b.log"; calls_log="$WORK/calls-5b.log" +export AGENT_ACTIONS_LOG="$actions_log" AGENT_LOG_ISSUE=999 +gh_run "$calls_log" "$empty_json" "$empty_json" issue close 999 +check "log issue close -> refused" 1 "$STATUS" +check_log_contains "log issue close -> logged REFUSED" "REFUSED issue close 999" "$actions_log" + +actions_log="$WORK/actions-6.log"; calls_log="$WORK/calls-6.log" +export AGENT_ACTIONS_LOG="$actions_log"; unset AGENT_LOG_ISSUE +gh_run "$calls_log" "$empty_json" "$empty_json" issue create --title t --body-file x +check "issue create -> allowed" 0 "$STATUS" +check_log_contains "issue create -> logged ALLOWED" "ALLOWED issue create" "$actions_log" + +actions_log="$WORK/actions-6b.log"; calls_log="$WORK/calls-6b.log" +export AGENT_ACTIONS_LOG="$actions_log" +gh_run "$calls_log" "$empty_json" "$empty_json" issue close 200 +check "issue close (non-log-issue) -> allowed" 0 "$STATUS" +check_log_contains "issue close -> logged ALLOWED" "ALLOWED issue close 200" "$actions_log" + +actions_log="$WORK/actions-7.log"; calls_log="$WORK/calls-7.log" +export AGENT_ACTIONS_LOG="$actions_log" +gh_run "$calls_log" "$empty_json" "$empty_json" issue list --label project +check "unrelated command -> passthrough exits ok" 0 "$STATUS" +check_log_contains "unrelated command -> logged PASSTHROUGH" "PASSTHROUGH issue list" "$actions_log" +check_log_contains "unrelated command -> reached fake gh" "issue list --label project" "$calls_log" + +echo +echo "test.sh: ${pass} passed, ${fail} failed" +if [ "$fail" -gt 0 ]; then + exit 1 +fi