Repository navigation
ci: open an issue when :testing stops publishing - #1132
Conversation
|
[APPROVALNOTIFIER] This PR is APPROVED This pull-request has been approved by: hanthor The full list of commands accepted by this bot can be found here. Needs approval from an approver in each of these files:
Approvers can indicate their approval by writing |
bce9d6c
ℹ️ Sentinel notice — informational; author is trusted (owner), merge is not blockedHive flagged this PR (author @hanthor, head
Hive did not add the |
There was a problem hiding this comment.
One verified problem with the freshness signal; otherwise the mechanism reads correctly.
created is the last build-input commit time, not the promotion time (scripts/testing-freshness.py:51-57)
The script treats the image config's Created as "when :testing was published". It is not. Every image build runs podman build --source-date-epoch <epoch> where the epoch is git log -1 --format=%ct -- Containerfile packages system_files scripts config .gitmodules docs/agent (Justfile:469-471, same in build-ghcr at Justfile:509-514, via scripts/podman-epoch-args.sh). Podman ≥ 5.5 writes that epoch into the config's created, which is what skopeo inspect reports.
Checked against the live tag: ghcr.io/projectbluefin/utah:testing config created is 2026-10-30T10:14:07Z; the promotion that produced it ran at 10:49 (run 41043543296), and the org.opencontainers.image.created label says 10-29 21:09. All three differ because none is "when the tag moved".
Consequence: the check fires on a quiet weekend. A Friday-evening commit to build inputs, then daily green post-testing-e2e promotions over Saturday/Sunday, still leaves created at Friday — by Sunday morning every flavor is >36 h "stale" and a priority/critical-urgent issue opens while nothing is wrong. The reverse also holds: a promotion that re-tags an image built from a fresh commit after a long freeze reads "fresh" even if the gate had been broken for days before.
Suggested fix: measure what the issue is actually about — the time of the last post-testing-e2e run that reached promote-to-testing — via gh run list --workflow post-testing-e2e.yml --status success (the job already has actions: read). The registry created can stay as a secondary column in the table, but should not be the staleness decision. tests/test_testing_freshness.py exercises stale() with synthetic datetimes, so it would not catch this; a test asserting the signal source would.
Smaller note
open_issue() searches in:body "utah:testing-freshness" and then filters by the HTML-comment marker. If GitHub's search index does not match text inside HTML comments, the search returns nothing and a new issue is opened every 6 h. I could not verify either way from here — flagging it as an open question, not a defect; searching on the title prefix would avoid depending on it.
Verified fine
- New workflow:
permissions: {}at top, job scoped tocontents: read,issues: write,actions: read;persist-credentials: false; pinned checkout SHA.ubuntu-26.04is already used bybuild.ymlandimage-baselines.yml. scripts/flavors.py imagesreturns[{"image": "utah"}, …]for all four flavors, which is whatimages()reads.- Labels
kind/bug,area/ci,priority/critical-urgentall exist in the repo, sogh issue create --labelwill not fail. - RFC 3339 nanosecond trimming in
parse_createdis correct and tested.
Since this adds a scheduled workflow with issues: write, a repository approver should confirm the automation is wanted — the author is one, so I expect that is settled.
Confidence: 3/5 (needs attention) — 1 high finding
— hive: agent=reviewer backend=copilot model=claude-fable-5.1 copilot=1.0.88
|
@hanthor this PR is back in your court ( Review asks:
Reply here or push a new head to this branch (same sign-off); CI and the review swarm re-run automatically and the PR returns to the normal path. Queued in your relay fix queue (3 waiting): fixes are handed out before new work as soon as you have capacity. |
…created date Review on #1132: the image config's created is SOURCE_DATE_EPOCH (last build-input commit), so a quiet weekend of green promotions would read as stale and a promotion after a long freeze as fresh. The check now finds the newest post-testing-e2e run whose promote-to-testing job ran the 'Move :testing to the tested digests' step, and treats none in the last 60 runs as stale. Assisted-by: Claude Opus via omp
…created date Review on #1132: the image config's created is SOURCE_DATE_EPOCH (last build-input commit), so a quiet weekend of green promotions would read as stale and a promotion after a long freeze as fresh. The check now finds the newest post-testing-e2e run whose promote-to-testing job ran the 'Move :testing to the tested digests' step, and treats none in the last 60 runs as stale. Assisted-by: Claude Opus via omp
c788e1e to
f82398f
Compare
:testing advances only when post-testing-e2e promotes it. A broken gate therefore freezes the tag while every build stays green: from 2026-10-30 to 11-06 a missing provenance attestation stopped every promotion, and nothing said so. testing-freshness.yml runs every six hours. scripts/testing-freshness.py reads each flavor's :testing creation time from the registry (skopeo inspect). When any flavor is older than 36 hours (or its tag is unreadable), it opens one issue labelled kind/bug, area/ci and priority/critical-urgent for the hive to pick up. The issue body carries a per-flavor table and the latest post-testing-e2e run with its failed jobs. Later runs refresh the same issue, found by a body marker, and close it once every flavor is fresh again. A dry run against the live registry flags all four flavors as stale (created 2026-10-30, 175 h ago) and names the latest e2e run's failed jobs. Assisted-by: Claude Opus via omp Signed-off-by: James Reilly <jreilly1821@gmail.com>
…created date Review on #1132: the image config's created is SOURCE_DATE_EPOCH (last build-input commit), so a quiet weekend of green promotions would read as stale and a promotion after a long freeze as fresh. The check now finds the newest post-testing-e2e run whose promote-to-testing job ran the 'Move :testing to the tested digests' step, and treats none in the last 60 runs as stale. Assisted-by: Claude Opus via omp Signed-off-by: James Reilly <jreilly1821@gmail.com>
…thout search The alarm still judges :testing by the completion time of the "Move :testing to the tested digests" step in post-testing-e2e's promote-to-testing job, the only writer of the tag. Nothing on the image can carry that time: the move is skopeo copy --preserve-digests, and the config's created is SOURCE_DATE_EPOCH. - Scan runs started within the threshold plus a day of slack, paginated, instead of the newest 60: a busy queue can no longer push a recent promotion out of view, and a freeze costs a bounded number of API calls. - Find the open alarm issue through the REST issues listing (label area/ci plus the body marker). gh issue list --label/--search goes through issue search, whose index lag or HTML-comment handling could open duplicates. - Run issue edit/close/create through a text helper: gh issue edit prints a URL, which the JSON helper would have failed to parse. - Tests pin the job and step names to post-testing-e2e.yml, cover a run that promoted but failed a later job, a failed promote job, and the issue lookup. Assisted-by: Claude Opus via GitHub Copilot Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Signed-off-by: James Reilly <jreilly1821@gmail.com>
f82398f to
16a950d
Compare
|
New head Freshness signal is now the promotion time. The script no longer uses the image's
The script scans runs started within the threshold plus 24 h of slack, with pagination, instead of a fixed count. The concurrency group allows one pending run behind one running run, and recent runs finished within 2.5 h of being created. If no run in the window moved the tag, that counts as stale, never fresh. A new test checks that the job and step names still exist in Open question about search and HTML comments: the script no longer depends on search. It finds its issue through the REST issues listing ( Checks run:
|
There was a problem hiding this comment.
Read the diff against the tree at 16a950d before the body. The mechanism in this head is sound; the PR body describes the previous one.
PR body is out of date. It says the script "reads each flavor's :testing creation time from the registry (skopeo inspect)" with a per-flavor table and "6 tests". The code on this head does none of that: scripts/testing-freshness.py:70-88 takes the completed_at of the "Move :testing to the tested digests" step in the newest completed post-testing-e2e run whose promote-to-testing job succeeded, and tests/test_testing_freshness.py has 12 tests (all pass locally). The new approach is the right one (the earlier review explained why created is SOURCE_DATE_EPOCH), and docs/skills/ci-workflows/references/iso-e2e.md already describes it correctly — only the body lags. Worth refreshing before merge since the squash message will carry it.
Verified against the tree:
post-testing-e2e.yml:373definespromote-to-testing,:418the step name,:429theskopeo copy --all --preserve-digests … :testingwrite, matchingPROMOTE_JOB/MOVE_STEP;SignalSourceTestspins all three so a rename fails CI.tests/run_suite.pydiscovers everytests/test_*.py, sojust testinbuild.ymlruns the new file without wiring.- Labels
kind/bug,area/ci,priority/critical-urgentexist, sogh issue create --labelwill not fail.open_issue()goes through the REST listing withselect(.pull_request == null)and the body marker, not search — the open question from the earlier review is resolved. is_stale(None, …)→True: no promotion in the 36 h + 24 h window reads as stale, never fresh.- Workflow: top-level
permissions: {}, job scoped tocontents: read,issues: write,actions: read,persist-credentials: false, checkout pinned by SHA,ubuntu-26.04already used by 13 other workflows here.
Trust note for the record: this adds a scheduled workflow with issues: write on GITHUB_TOKEN. The author is an OWNERS approver and has already /approved, so I take that decision as made.
Confidence: 5/5 (safe)
— hive: agent=reviewer backend=copilot model=claude-fable-5.1 copilot=1.0.88
|
Needs a decision, not a merge:
Head: verdict requires_human @f8814f9, 20/20 checks green, soak 30m0s elapsed. What to do: a trusted merger approves this head ( Clear the cause and the hive re-evaluates on its next pass; the stage moves to Updated 2026-11-07 16:35 UTC. |
|
/hive decision approve Owner approval for the workflow-file fence. The new workflow has |
|
@hanthor approved head |
|
@hanthor approved head |
There was a problem hiding this comment.
HUMAN DECISION NEEDED — @hanthor confirm a maintainer accepts a scheduled workflow holding issues: write on the default token (unchanged since last review).
Summary: Re-review of the alarm that opens an issue when the :testing image stops being promoted. Since the last look the branch only picked up merges from main (dependency digest bumps, a docs note and a new ujust recipe); nothing in the alarm itself changed. The one open question is still whether the scheduled job may hold permission to create and close issues; the PR body says a maintainer approved that scope, and a human should confirm it.
Delta reviewed: f8814f9 → bce9d6c (7 commits, all Merge branch 'main'; PR files .github/workflows/testing-freshness.yml, scripts/testing-freshness.py, tests/test_testing_freshness.py, docs/skills/ci-workflows/references/iso-e2e.md unchanged).
security
- Still present:
.github/workflows/testing-freshness.yml:30grantsissues: writeongithub.tokenin a cron job (:11,:38). Top-level ispermissions: {}(:19), checkout usespersist-credentials: false(:35), no secrets beyondgithub.token. PR body states this scope is maintainer-approved; confirm and this becomes approve.
Delta cross-check: main's post-testing-e2e.yml changes are testsuite digest bumps only; promote-to-testing job (post-testing-e2e.yml:373) and step "Move :testing to the tested digests" (:418) still match scripts/testing-freshness.py:44-45.
No findings from: correctness, intent-alignment, style, docs-currency.
Confidence: 3/5 (needs attention) — 1 medium finding; 1 perspective requires a human decision
— hive: agent=reviewer backend=copilot model=claude-fable-5.1 copilot=1.0.88
|
@hanthor approved head |
There was a problem hiding this comment.
No correctness blocker found: scripts/testing-freshness.py:71-89 checks the actual successful move step, not the overall run conclusion or image creation date. The existing caller gates that step on main and the promotion ratchet, so skipped promotions do not count as fresh.
Final trust review remains with @castrojo: .github/workflows/testing-freshness.yml:28-40 adds a scheduled issues: write job that executes the checked-out script and can create, edit, and close its marked alarm issue. The PR body states this scope is already maintainer-approved; this is a final implementation review of that approved scope, not a request to redesign it. The sentinel notice and new workflow write permission warrant human sign-off before merge.
Confidence: 3/5 (needs attention) — 1 perspective requires a human decision
— hive: agent=reviewer backend=copilot model=gpt-6.1-sol copilot=1.0.88
|
@hanthor approved head |
|
Merged by hive contributor-lane auto-merge: CI green for ≥30m, reviewer verdict not required, @hanthor approved this head as the owner decision ( |
Owner policy: "if the :stable tag is over a week old and :testing passes, we promote it". execute-release.yml now checks daily at 06:00 UTC instead of every Monday. A scheduled run releases only when :stable last moved more than seven days ago (or a flavor has no :stable yet) and :testing points at a different digest for at least one flavor of `scripts/flavors.py releases`; otherwise it posts a ::notice saying whether age or nothing-new held it back. Promotion pushes and manual dispatch are unchanged, and every release still goes through the e2e-evidence gate and the release gate. :stable's age is the completion of the reusable release's "Promote digests to target tags" step in the newest execute-release run that succeeded at it, read in batched GraphQL queries (checkType ALL so a rerun cannot hide an earlier retag). Not the image config created (SOURCE_DATE_EPOCH, the #1132 pitfall) and not the testing-YYYYMMDD version label, which dates the build that was later released, per flavor and to the day. The step retags every flavor or fails, so one time dates them all and a partial failure keeps the oldest age. Registry reads use skopeo inspect --no-tags: without it skopeo lists every tag, over 30 s per image here. Assisted-by: Claude Opus via GitHub Copilot Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Signed-off-by: James Reilly <jreilly1821@gmail.com>
:testingmoves only when post-testing-e2e promotes it. A broken gate therefore freezes the tag while every build stays green: from 2026-10-30 to 11-06 a missing provenance attestation stopped every promotion, and nothing said so.What this adds
testing-freshness.yml: runs every six hours withpermissions: {}at the top level. The job gets onlycontents: read,actions: readandissues: write.scripts/testing-freshness.py: finds the newest post-testing-e2e run whosepromote-to-testingjob actually ran "Move :testing to the tested digests".kind/bug,area/ciandpriority/critical-urgentfor the hive. The issue shows when the tag last moved and the newest run's failed jobs.createdfield. That is SOURCE_DATE_EPOCH (the last build-input commit), not the promotion time.Verification
tests/test_testing_freshness.pycovers finding the last real promotion (a green run whose move step was skipped doesn't count), in-progress runs, no promotion in the window (stale), the threshold edges and the week-long freeze.just checkand pre-commit pass.Permission
issues: writeon a scheduledgithub.tokenjob is the maintainer-approved scope: the job's entire purpose is to file and close this one issue. It pushes nothing and uses no secrets.