Repository navigation
🌱 ci: reduce PR critical path without dropping checks - #16026
Conversation
Signed-off-by: Andrew Anderson <andy@clubanderson.com>
|
Changelog: this PR changes code but carries no changelog entry If it is user-visible — a feature, a fix an operator would notice, a This is a reminder, not a gate; it never blocks a merge. |
Open PR overlap checkBase branch: Conflicts are checked with every PR merged onto the current tip of No conflicting open PRs or same-file overlaps were found. |
7a83946
ℹ️ Sentinel notice — informational; author is trusted (owner), merge is not blockedHive flagged this PR (author @clubanderson, head
Hive did not add the |
|
Measurement update: the first real candidate run confirms the Podman cleanup contract fell from 132–147s to 1s, with all assertions retained. The protected build-and-test aggregate is green after all three lanes (Go 108s, contracts 115s, runtime 153s); original serial job execution was 456–506s in the initial sample. Remaining required test/security checks are still running, so this is not yet a claim of a five-minute PR. A second measured bottleneck is the coverage job post-step: setup-go archives shared Go caches for 28s after scoring despite this job only listing packages and consuming profiles. A follow-up will disable that redundant cache save, regenerate the readonly profile and run fresh full CI. Baseline evidence includes seven comparable full 23-shard, successful v5 PR pipelines; affected-only runs are excluded from the comparable distribution. |
Signed-off-by: Andrew Anderson <andy@clubanderson.com>
|
First full after run: all required checks passed, but the complete required pipeline took 385s (6m25s), so it did not meet the five-minute target. Causal critical path: scope 3s → package discovery 13s → dashboard shard 238s → coverage 50s, plus 81s queue/scheduling. Required runner work was 85.25 minutes versus the comparable baseline median 85.87; the earlier serial build bottleneck is removed without a runner-minute explosion. Follow-up now pushed: coverage no longer archives shared Go caches; gosec keeps its exact directory set but bounds package-loading fanout to two workers instead of host NumCPU. The pinned gosec source confirms its default uses runtime.NumCPU rather than the container quota; runner config requests 1.5 CPUs/bursts to six on 32-CPU nodes. The first after log scans all 203 baseline directories, with no omissions, plus one newly introduced package; duplicate Crux scans are gone. Fresh final-candidate CI is now the next measurement. |
Signed-off-by: Andrew Anderson <andy@clubanderson.com>
There was a problem hiding this comment.
Split looks sound; one semantic change to the required check is worth a look before merge.
Verified at 7d7d48b:
v2-ci.yml: every step of the oldbuild-and-testis still present acrossbuild-go(:123-242),contract-tests(:244-893) andruntime-tests(:895-1299); the step lists are contiguous slices of the original. Grepped the two new lanes forgo,setup-go,cmd/hive,go.mod— nothing, so dropping the Go toolchain from them is safe. TheEnsure PyYAMLstep in the new lanes is a copy of the one already inbuild-go:138-148, not a new install path.- The protected name survives as the aggregator (
:1301-1318);create-issue-on-failure:1325andnightly-release:1601stillneeds: build-and-test. On this head the three lanes took ~2–2.5 min each against 20-min timeouts, and the aggregator ran and passed. integration-technical-v2-ci.ymlmirrors the same split.
Finding (medium) — v2-ci.yml:1314-1317: the aggregator requires SCOPE_RESULT == success. Today the lanes' if: at :126 deliberately treats required-context-scope.result != 'success' as "run the full lane anyway" — the fallback for when the scope script's gh api paging (:75-105, set -euo pipefail) hiccups. With this change, that path now runs all three lanes to green and then fails the required check anyway, because scope is not success. Before this PR a scope failure was self-healing; after it, a transient API error on the scope job blocks merge until re-run. If fail-closed is intended, the docs paragraph in development.md ("fails if any lane fails, is cancelled, or unexpectedly skips") should say so; otherwise drop $SCOPE_RESULT from the loop and let skipped lanes be the only scope-related failure.
Open question (not verified from this repo): the gosec ./...-only scan assumes gosec's own directory walk picks up src/crux despite its separate go.mod. If it doesn't, Crux is no longer scanned; a one-line check of the gosec log's package count on this run would settle it.
— hive: agent=reviewer-2 backend=copilot model=claude-fable-5.1 copilot=1.0.88
|
Second complete measured candidate: 416s (6m56s), all required checks green, not five minutes. The scanner improvement is real: pinned gosec cache hit (26 MB), installation skipped, identical 204 unique directories scanned with zero omissions, scan 105s / job 114s versus first candidate scan ~300s / job 350s. Coverage is now 13s instead of 50s. The remaining required critical path is test scheduling: 281s execution plus 135s queue/scheduling, versus 27s median scheduling in the comparable baseline. The shared runner snapshot had 131 busy of 136 online runners; independent REST quota is also exhausted by shared account traffic (measurements continue through GraphQL). A third surgical change now pushed starts package discovery alongside scope; all shards and required aggregates still require both successful verdicts. This removes an unnecessary runner-queue wave without adding shards or omitting work. Fresh full CI will establish whether the current shared capacity can actually meet 300s. |
32fe205
ℹ️ Sentinel notice — informational; author is trusted (owner), merge is not blockedHive flagged this PR (author @clubanderson, head
Hive did not add the |
There was a problem hiding this comment.
Decision needed: @hanthor should decide whether the protected build-and-test check may become a three-lane aggregate (build-go / contract-tests / runtime-tests) that now also fails whenever required-context-scope itself fails, where it previously still ran the tests.
This edits six .github/workflows/*.yml files, so per hive policy I am routing it to a second approver rather than confirming it myself. What I verified at 32fe205:
Checks out
- The three new lanes are the original
build-and-teststep list split at two existing boundaries (v2-ci.yml:244,:895); no step was dropped or reordered. The protected check name is preserved by the aggregate atv2-ci.yml:1301-1319, which hard-fails unless all fourneedsresults aresuccess. - No step under
contract-testsusesgo:bin/go-wrapper.test.sh:25mocks it.runtime-testscarries its ownactions/setup-node(v2-ci.yml:914). TheEnsure PyYAMLblock (including thecurl … get-pip.py | python3fallback) is a verbatim copy of the step already inbuild-go(v2-ci.yml:138-147), not new. v2-tests.yml:list-packagesnever readneeds.required-context-scope.*(no references in :1106-1180), and every shard and both aggregates still listrequired-context-scopeinneeds(:283,:559,:853,:1421,:1641,:1892,:2037), so the dependency removal does not loosen gating.bin/test_hive_podman_cleanup.sh:243-268: the Python scan preserves the shell semantics —git ls-files -z,is_file()≙[[ -f ]], byte-literal containment ≙grep -qF, same exemption set — andsys.exit("FAIL: …")underset -euo pipefail(:9) is equivalent to thefail()helper at:36-39(stderr + exit 1).
One behaviour change the body does not call out (info)
- Old
build-and-testhadif: always() && (scope.result != 'success' || run-ci == 'true'), so a failed/transientrequired-context-scopestill ran the tests and the check could go green. The new aggregate requiresSCOPE_RESULT == success(v2-ci.yml:1314-1318), so a scope failure now fails the required check outright. Stricter (fail-closed), probably intended, but it is a semantic change to the merge gate worth an explicit yes from a maintainer.
I have not approved or merged anything.
— hive: agent=reviewer backend=copilot model=claude-fable-5.1 copilot=1.0.88
|
Hive status — updated in place; edits do not notify. Full history is in the Activity Monitor. clubanderson this PR is held at Why:
Next: the hive does not push to your branch. Push a new head (or reply here); CI and the review swarm re-run automatically and the PR returns to the normal path. Hive filed review findings that do not block this PR as backlog items:
Cap: 10 backlog item(s) per PR per day. Review findings from Read the workflow diffs against v5 and the aggregate test. The restructuring looks correct to me; the one red check is self-inflicted, not a defect.
Suggestion, not a defect: |
clubanderson
left a comment
There was a problem hiding this comment.
Post-publication review: the protected build-and-test aggregate fails closed on any non-success lane; scope remains a prerequisite for every test shard and both required test/coverage verdicts; discovery alone is parallelized. The tracked-file scan retains every literal pattern/exemption and binary/symlink coverage, and read failures now fail closed. Gosec retains its pinned version and complete directory set; cache misses rebuild the tool, and bounded workers change scheduling, not rules. Readonly integration profiles remain generated with the same permission/runner boundaries. Race/shuffle flags, all 23 shards, coverage floors, test membership and release/image behavior are unchanged. No significant correctness issue identified; the independent initial code-review agent also reported none.
The conservative first final-candidate measurement includes the earlier required push gate: 300s exactly (PR-workflow-only 297s). Earlier 385/416s PR-only measurements are 388/418s on that same inclusive origin definition. An exact-SHA technical rerun is in progress to avoid claiming reliability from one borderline result. ARM64/overlayfs image jobs are non-required but remain monitored before merge; this is not approval to bypass pending/broken CI.
Signed-off-by: Andrew Anderson <andy@clubanderson.com>
Signed-off-by: Andrew Anderson <andy@clubanderson.com>
clubanderson
left a comment
There was a problem hiding this comment.
Reviewed exact head c916e1a, including the native-runner change. All three image jobs retain their same-repository PR guard: fork PRs do not gain self-hosted execution, registry credentials or image publishing. Public PR ARM64 builds use standard free ubuntu-24.04-arm ephemeral VMs (already used by the Podman ARM64 lane); explicit HIVE_ARM64_RUNNER overrides retain priority. Private repositories and release/push events retain the original runner selection and publishing logic. QEMU remains on non-native hosts; platforms, permissions, no-cache freshness defense, SUID/CA/smoke assertions and release semantics remain intact. Added structural regression coverage runs in CI. Required scopes/aggregates still fail closed; test selections, race/shuffle flags, floors and shard count remain unchanged. Current-head full CI must finish before any merge. Earlier 260s is technical-only warm rerun, not a second full gate; latest completed full required-gate event was 320s, not under five minutes.
c916e1a
ℹ️ Sentinel notice — informational; author is trusted (owner), merge is not blockedHive flagged this PR (author @clubanderson, head
Hive did not add the |
clubanderson
left a comment
There was a problem hiding this comment.
Final exact-source measurement for c916e1a: all eleven required contexts passed in 340s (297 execution +43 scheduling); all 25 contemporaneous workflows settled by 361s (315 execution +46 scheduling/finalization), including native images, Podman and overlayfs. One superseded policy cancellation has a successful replacement; no failed technical work excluded. Native image workflow is 336s/16.50 runner-minutes versus matched baseline 1302–1315s/33.13–33.73. Native/fork/isolation/public-only/release behavior review remains applicable to this unchanged head. This does NOT meet five-minute acceptance. More importantly, eight actual merged PRs show median open-to-merge 41m49.5s, latest-head-to-merge 28m14s and post-required-green-to-merge 17m29s. All eight merged despite Tide pending; two merged before optional images completed. The real remaining diagnostic is review/rework/merge admission, not a blanket claim that Tide or all optional images block merging. PR remains unmerged; no automatic approvals or policy/protection changes. Full timestamps, per-PR evidence, caveats and bounded next decision are in the updated description.
There was a problem hiding this comment.
Decision needed: @hanthor should decide whether the protected build-and-test context may become a fail-closed aggregate of three lanes and whether public-PR ARM64 Docker jobs may move to GitHub-hosted ubuntu-24.04-arm.
Routing to a maintainer rather than judging alone: this edits six .github/workflows/* files, changes what the protected build-and-test check means, and changes runner selection. Those are trust-surface decisions. The author is an OWNERS approver, so this is not a question of standing — it is that a second approver should sign off on CI-gate semantics.
What I verified in the tree at c916e1a (not from the body):
v2-ci.yml:123renames the old job tobuild-go;:244contract-tests(61 steps) and:895runtime-tests(37 steps) carry the remaining steps;:1301re-createsbuild-and-testas aneeds:aggregate whose single step exits 1 unless scope, go, contract and runtime results are allsuccess.if: always() && !cancelled()means a failed lane does produce a redbuild-and-test, not a skipped one. Fail-closed as claimed.- Toolchain coverage of the split:
contract-testshas nosetup-go, but the only scripts in it that mentiongoarebin/go-wrapper.test.sh(uses a mock viaHIVE_GO_REAL,bin/go-wrapper.test.sh:20-40) andbin/test_hive_podman_setup.sh:679(asserts no host go).runtime-testskeepssetup-node. Nothing in the two Go-less lanes shells out to the real toolchain. persist-credentials: ${{ !inputs.integration_candidate }}is preserved on every new checkout; no newpermissions:blocks; thecurl | python3pip bootstrap is copied verbatim from the pre-existing step (1 occurrence on v5, now 3).docker.yml:282/687/896:ubuntu-24.04-armis selected only whenevent_name == 'pull_request' && repository.private == falseandHIVE_ARM64_RUNNERis unset; push/release paths unchanged. QEMU setup is gated onrunner.arch != 'ARM64'.v2-tests.yml:1109:list-packageslosesneeds: required-context-scope; I confirmed the shard and aggregate jobs stillneeds:scope, so membership is unaffected.
Open question I could not verify (not a defect): go-security-analysis.yml:271 drops ./crux/... on the claim that gosec walks nested modules itself. src/crux/go.mod is a separate module. The body says logs show 204 directories still scanned; a maintainer should confirm that count includes src/crux/** before this merges, since a silent drop of the Crux scan would not fail CI.
I have not approved, and cannot — the decision above belongs to a human approver.
— hive: agent=reviewer-2 backend=copilot model=claude-fable-5.1 copilot=1.0.88
|
Resolved the concrete scan-membership question on exact c916e1a: actual gosec logs include 42 paths matching /crux, including src/crux itself and src/crux/agent, api, channel, config, etc., within the same 204 unique directories. This is measured membership, not reliance on go list nested-module behavior. Evidence job: https://github.com/hivecommons/hive/actions/runs/41200939682/job/124536485036 . The remaining reviewer request is a genuine maintainer decision: @hanthor / @Danathar, please confirm the fail-closed protected aggregate and public-PR-only free native ARM64 runner selection. Existing gates/permissions/release paths remain intact. The author will not substitute self-approval or admin override for that explicit decision. Five-minute end-to-end target is still NOT met; 340s required / 361s all workflows is partial progress. The experimental hold is not a permanent demand for five minutes, but the explicit needs-decision review must be resolved before shipping. |
There was a problem hiding this comment.
Decision needed: @clubanderson should decide whether to lift hold on this 8-workflow CI change now that the ca4290a test repair is verified, or keep it parked until the conflicts-with-open-pr overlap is resolved.
Delta review of ca4290a only (previous head 918ce74 already carries a hive verdict; nothing below re-litigates it).
The one-line fix is correct and load-bearing. .github/scripts/test-generate-integration-ci.py:51 retargets the checkout mutation from build-and-test to build-go. At this head build-and-test in .github/workflows/v2-ci.yml is an aggregate (needs: [required-context-scope, build-go, contract-tests, runtime-tests]) with no actions/checkout step, so the old next(...) raised StopIteration — a test error, not the ValueError the test asserts. build-go does check out, and it is an active job the generator's provenance loop at .github/scripts/generate-integration-ci.py:137-140 actually scans (active job {name} overrides the tested subject). Verified on a clone at ca4290a: suite passes (2 tests OK); with the added line removed it fails with StopIteration.
Red native-queue-policy is not this diff's fault. Job 124573705547 fails at .github/scripts/native-queue-policy.cjs:62 with #16026: head/base moved, draft, closed, or held — the PR carries the hold label. It will clear when hold is lifted.
Routing, not a defect. This PR edits 8 files under .github/workflows/ (runner selection now prefers ubuntu-24.04-arm for public PR image builds, gates reshaped around run-ci, gosec concurrency bounded). That is a trust surface, so I am not calling it clean on my own authority; the question above is the one a maintainer needs to answer. I have not changed any labels.
— hive: agent=reviewer-2 backend=copilot model=claude-fable-5.1 copilot=1.0.88
|
Exact-source timing update for ca4290a: all eleven protected required contexts passed in 265s (4m25s), from the contemporaneous feature-push gate at 15:11:46Z through the final required check at 15:16:11Z. All 23 test shards executed. The comparable baseline required-gate median is 561s (9m21s), n=7. Required execution ancestry consumed 71.47 runner-minutes, versus baseline median 85.92. This is required-check timing, not PR-to-merge timing. Required-context workflow suites finished their optional work after 353s; all 23 applicable source-event workflows settled after 365s (6m05s). The all-workflow result is not unqualified green: native-queue-policy fails because of our experimental hold. The previously reported 340s/361s remains a prior candidate measurement, not this head. The scope-error full-lane fallback now preserves baseline behavior, and the generator checkout-provenance repair passes. Independent human review of the native public-PR ARM runner and workflow split remains outstanding; requested reviewers are hanthor and Danathar. No deployment or verified five-minute PR-to-merge outcome is claimed. |
ca4290a
ℹ️ Sentinel notice — informational; author is trusted (owner), merge is not blockedHive flagged this PR (author @clubanderson, head
Hive did not add the |
|
Resuming this PR at the repository owner’s explicit November 2 request. A dedicated implementation agent is integrating current v5, preserving scope-error full-lane fallback and all required checks, and refreshing CI evidence. Native ARM64 runner selection and workflow trust review will be evaluated against the current diff; this note does not substitute for a required maintainer review or claim a new timing result. |
Signed-off-by: Andy Anderson <andy@clubanderson.com>
|
November 2 source refresh: merged current v5 Gate clarification from actual sources: v5 branch protection has eleven required contexts and |
There was a problem hiding this comment.
Decision needed: @clubanderson should decide whether #16026 or the already-reviewed #16045 owns the gosec Scan line (they conflict on the same line and #16045's new test rejects #16026's form), and sign off on the workflow/runner changes as the workflows owner.
Conclusion: the CI restructuring reads correct against the v5 tree, but this PR edits nine .github/workflows/* files and conflicts with #16045 on the same line, so a maintainer has to choose which lands — I am routing rather than judging it clean.
Conflict with #16045 (verified, medium). Both PRs rewrite the Scan step in go-security-analysis.yml, integration-candidate.yml and integration-technical-go-security-analysis.yml:
- #16045 →
gosec -concurrency 4 -exclude-generated ./... ./crux/...and addstest_gosec_concurrency_fits_the_runner_cpu_limitin.github/scripts/test-native-queue-workflows.py, which requires the regex\bgosec\b.*\s-concurrency[ =](\d+)\bto match the Scanrun:. - #16026 →
"${{ runner.temp }}/hive-ci-tools/gosec" -concurrency "${HIVE_CI_GOSEC_CONCURRENCY:-2}" -exclude-generated ./...(.github/workflows/go-security-analysis.ymlhunk @ 264–271).
-concurrency "${HIVE_CI_GOSEC_CONCURRENCY:-2}" does not match (\d+) after -concurrency , so whichever merges second either fails that test or has to rewrite it. These are not duplicates — different concurrency (4 vs env-default 2), and only #16026 caches the binary and drops ./crux/... — so both cannot merge as written.
Open question (not verified, info). Dropping ./crux/... from the gosec args rests on gosec's own directory walk crossing into the nested src/crux module. I did not run gosec to confirm the crux packages still appear in the scan; worth checking the package count in the first green run before relying on it.
What I checked and found correct:
- The
build-and-test→build-go/contract-tests/runtime-testssplit keeps every step: at merge-basev2-ci.yml:135–873the steps moved intocontract-tests(lines 251–873) invoke nogo/node/npm, so the missingsetup-go/setup-nodein that job is fine;setup-nodestays withruntime-tests. - The new
build-and-testaggregate usesalways() && !cancelled()plus an explicit!= successcheck per lane, so a skipped or cancelled lane cannot produce a green required context.src/deploy/test_ci_build_lane_aggregate.shexercises exactly that. list-packagesinv2-tests.ymllosingneeds: required-context-scope: at merge-base (v2-tests.yml:1108–1420) the job referencesneeds.required-context-scopeonly in its ownif:and computesaffected-pkgs.txtitself (Select affected packages, :1209), so nothing it reads disappears. Shards still gate on scope.docker.yml:matrix.self_hostedis declared (:275,:279) but not referenced by any step, so moving public-PR arm64 builds toubuntu-24.04-armdoesn't break a conditional, andrunner.arch != 'ARM64'correctly skips QEMU there.
Flagging for a maintainer; I have not approved or changed anything.
— hive: agent=reviewer-2 backend=copilot model=claude-fable-5.1 copilot=1.0.88
Signed-off-by: Andy Anderson <andy@clubanderson.com>
|
Dependency reconciliation: merged newly landed #16045 / v5 |
There was a problem hiding this comment.
Read the workflow diffs at 1a34e4d against v5 and checked the two claims most likely to drop coverage. Both hold.
- gosec
./...without./crux/...(.github/workflows/go-security-analysis.yml, Scan step):src/go.worksays a bare./...is root-module only for the go tool, so I compared the gosec logs rather than trust the comment. Base v5 run 41465174915 and this PR's run 41460836751 both reportFiles : 1785; the base log lists eachsrc/crux/<pkg>Import directorytwice (82 lines), this PR's once (41). gosec walks the filesystem itself, so Crux is still scanned and the double scan is gone — the comment is accurate. build-and-testcheck name preserved (.github/workflows/v2-ci.yml, new aggregate job): itneedsbuild-go,contract-tests,runtime-tests, runs underalways() && !cancelled()with the same scope/release-gate guard as the lanes, and fails unless all three results aresuccess— so a skipped or failed lane cannot turn the protected check green.src/deploy/test_ci_build_lane_aggregate.shcovers that.docker.ymlruns-on: the newubuntu-24.04-armbranch only applies whenvars.HIVE_ARM64_RUNNERis unset and the event is apull_requeston a public repo; push/release selection is unchanged, andsetup-qemuis skipped only whenrunner.arch == 'ARM64'. No permission blocks change in any workflow.- The
get-pip.pyfallback in the newcontract-tests/runtime-testsjobs is the same text already inv2-ci.yml's existing jobs, not a new pattern.
The red native-queue-policy check is assertEligible rejecting the hold label (native-queue-policy.cjs:64), not a defect in the diff; it will clear when the hold is lifted. Looks correct to me.
— hive: agent=reviewer-2 backend=copilot model=claude-fable-5.1 copilot=1.0.88
|
/approve Owner-requested, agent-assisted decision under @clubanderson's explicit delegation for this batch; this is not a separate independent human review. Reviewed the current diff and surrounding source at the exact commit below. Inspected head: Scoped workflow-owner direction: accept the public-PR-only native hosted ARM64 fallback (explicit override/private/push/release boundaries preserved), the three-lane fail-closed build-and-test aggregate with the original scope-error full-lane fallback, and cached pinned gosec with explicit concurrency 4 plus the single recursive scan. Inspected canonical current diff and generated-mirror deltas; the #16045 ownership conflict is reconciled at this head, not a reason to demand a duplicate patch. All current technical checks are green; native-queue-policy is red because the intentional hold remains. I am NOT clearing that hold or merging over another active coordinator. Existing coordinator: reconcile this owner direction with the already-requested eligible review and explicitly release the coordinated hold only when your current-head work is complete. There is no universal enforced second-human GitHub approval count in the protection we read, but the normal Prow review labels and genuine requested review remain. Historical 265 seconds is not a measurement of this head, the live v5 baseline was not re-verified, and no new five-minute PR-to-merge result is claimed. This PR is authored by the stored account. I am not submitting an impossible native self-approval or pretending this is a separate human review. |
|
[APPROVALNOTIFIER] This PR is APPROVED This pull-request has been approved by: clubanderson The full list of commands accepted by this bot can be found here. The pull request process is described here DetailsNeeds approval from an approver in each of these files:
Approvers can indicate their approval by writing |
|
Coordinated hold disposition at exact No LGTM or human GitHub review is invented: eligible Prow reviewer LGTM remains outstanding, and the requested review path must be satisfied before normal-queue merge. Native policy must refresh after hold removal. No CI/protection/permission change, bypass, or new timing claim. |
|
Final coordinator receipt for The PR is still OPEN, not merged. Actual Prow Tide now reports exactly: "Not mergeable. Needs lgtm label." Approval is recorded, hold/overlap are absent, and the original workflow/native-runner owner-choice question is resolved. The concrete remaining action is a genuine LGTM/current-head review from an eligible reviewer under OWNERS (existing review requests remain); it is not another owner trust decision or a requirement for one exclusive named reviewer. I have not fabricated self-LGTM, GitHub approval, or bypassed this remaining normal review gate. No new timing claim. |
|
Needs a decision, not a merge: the PR edits CI workflow / action files: awaiting owner approval — a hive owner / trusted merger's GitHub Approve review on the current head (or a dashboard Head: verdict approved @1a34e4d, 76/76 checks green, DCO yes, soak 0s of 10m0s. How to release it:
Clear the cause and the hive re-evaluates on its next pass; the stage moves to |
|
/hive decision approve #16026: green head, reviewer question answered with the reversible default — proceed as implemented Rationale: no owner rule holds this back; a merge is reversible and keeps work moving. Hive decided: Approved for merge. Decided by the hive at @clubanderson's request as the owner's delegate under the owner-delegate policy (rule |
There was a problem hiding this comment.
Approved by @clubanderson for Hive auto-merge on green CI.
|
@hivecommons-hive[bot]: changing LGTM is restricted to collaborators DetailsIn response to this:
Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository. |
|
Hive queued this PR but could not set the |
|
Thank you for your contribution! Your PR has been merged. We'd love to hear how your experience was: share feedback |
Summary
Acceptance clarification: the real goal is elapsed time until a PR actually merges, not merely a short technical workflow. This PR has NOT demonstrated five-minute PR-to-merge or all-CI completion. It is partial progress and may ship once its legitimate maintainer decision and checks are satisfied; five minutes is not an artificial merge prerequisite. The measurements below separate required gates, every applicable workflow, and the actual merge funnel.
Reduce CI turnaround without dropping tests or checks:
build-and-testname as a fail-closed aggregate.ubuntu-24.04-armVMs for public PR Docker ARM64 jobs, already used by the Podman ARM64 lane. Explicit ARM overrides win; private repositories and release/push events retain previous runner selection. QEMU remains available for non-native hosts; no paid infrastructure or permissions change.Stable measurement definition and correction
Required wall is the interval from the earliest contemporaneous workflow creation that emits a configured required context, including feature-push
gate, through the last required check completing. It includes initial queues and inter-job scheduling, all required aggregates, coverage, security, DCO and signoff. It is not execution-only and is not all-CI completion.An earlier description incorrectly used a PR-event-only boundary. It reported baseline 547s, range 480–947s, 85.87 runner-minutes, and candidate 297s. Including the earlier required push gate consistently corrects those to 561s, 482–949s, 85.92 runner-minutes, and 300s. The earlier numbers are superseded, not alternative success claims.
7a839467d7d48b32fe205, full fresh event32fe205, warm technical rerun only6b79e3f, full fresh eventc916e1a, full fresh eventBaseline selection: seven successful comparable full 23-shard v5 technical pipelines; affected-only and cancelled/failed principal pipelines excluded. Range 482–949s; nearest-rank empirical p90 949s is the maximum of only seven observations, not a robust SLO estimate. Component medians need not sum to median wall. Runner-minutes sum actual gate-ancestry job durations, not billed rounding, and exclude optional image work.
Exact boundaries and baseline run evidence
All timestamps below are UTC on 2026-10-31.
78c30d99498753c034aff1e84c0a483e75fe64557da25079e7a839467d7d48b32fe205, fresh32fe205, technical rerun6b79e3f, freshc916e1a, freshThe rerun repeats all three technical workflows, including all 23 shards, but retains earlier successful nontechnical DCO/signoff/verifier/gate checks. Its boundary is the recorded rerun request, not the original workflow creation, and it is not a second full required-gate event. The gosec binary cache is warm; per-job Go compilation remains isolated. No after p90 or guaranteed five-minute SLO is asserted.
The prior native full event had a cold gosec tool-cache miss (36s installation, 79s scan), not a warm technical-only rerun. Actual scan logs still show 204 unique directories. Comparisons across fresh events are observations on an evolving v5 base, not an isolated causal experiment:
c916e1atested synthetic merge6584f7eb01bdaf9e659d29717fe8246f90b5813a, whose first parent is95eafcdfa61ad235655987156e65406723895e97, later than the feature branch's originald8e1d41. PR RESTbase.shalags and is not sufficient merge-source evidence.Exact-head all-applicable-workflow result
The all-CI ≤300s goal failed: 361s (6m01s) on
c916e1a, from 12:24:06 to 12:30:07 UTC. The last job finished at 12:30:06; one additional second is workflow finalization. This includes all 25 contemporaneous push/PR/PR-target workflows, optional native image builds, Podman, overlayfs and advisory patch coverage. All technical workflows succeeded; one superseded Native queue policy cancellation has a successful replacement and is disclosed in the evidence. No technical failure or ongoing image job was excluded.GitHub documents standard ARM64 runners as free and unlimited for public repositories. This public repository already uses the exact runner in its Podman ARM64 lane. No image/security/smoke work was disabled or moved, and no paid infrastructure or protection change was made.
Real PR-to-merge funnel: the larger remaining delay
Eight recently created merged v5 PRs, unfiltered by rework/holds, show:
This eight-PR funnel is a broader cohort than the seven full-pipeline CI baseline; its 628s must not silently replace the stable 561s CI baseline.
Observed blockers versus hypotheses
hivecommons-hive[bot]and two throughclubanderson; none shows native merge-queue timeline events. Tide's pending label message is not proof Tide delayed these merges.Bounded next decision / no merge claim
There is no after PR-to-merge measurement because this optimization PR remains open. Its iterative benchmark and deliberate operator hold/testing/rework are not represented as normal automated CI latency. The original five-minute-only hold condition is superseded: shipping this partial improvement is authorized once its genuine exact-head maintainer decision is resolved and checks pass. Subsequent evidence-only PR-body edits may retrigger policy checks; the source-event measurement above stays frozen and explicit.
The dominant observed user-experience residual is review/rework/admission after technical green, not just the remaining 61s all-CI overrun. The next bounded action is read-only correlation of exact-head review-verdict, webhook delivery, stage/merge-sweep and pacing audit timestamps for the cited eight PRs, then a targeted event-delivery/admission proposal preserving all reviews, holds, exact-head proofs and protections. Changing approval/merge policy needs an explicit decision; this PR does not do it. Separately, a bounded existing-shard balance/compilation investigation could address the 272s dashboard slice without adding a shard explosion. No five-minute success or safe end-to-end merge improvement is claimed.
Validation and review
1a34e4d7bca5acda79b5370d059fc463036f911eDCO-merges v5 through 🐛 fix(ci): bound gosec concurrency so Scan stops losing its runner #16045 (1e83f1f94), preserving explicit gosec concurrency 4, pinned cache, complete single recursive scan, and the structural guard (now also covering the composed candidate). Canonical mirrors were regenerated; independent precommit review found no defect. All 23 test shards and all applicable technical checks passed. The October 31 timing evidence below remains historical; no new timing or PR-to-merge result is asserted. Exact-head independent review at 15:04:27 UTC confirms aggregate, runner boundary and measured Crux scan coverage. Owner direction at 15:11:07 UTC accepts those scoped trust changes. The coordinator released the experimental hold after verifying current-head work complete. Eligible Prow LGTM/review and refreshed native-policy evidence remain required; no review or bypass is fabricated.required_pull_request_reviews: null, and no repository rulesets. CODEOWNERS lists @clubanderson, @hanthor and @Danathar for workflow paths.src/pkg/policies/defaults/reviewer-queue.md:395-411requires human disposition of trust-surface changes, not exclusive authority for a named reviewer or a universal second-person GitHub approval. The parent/operator supplied an exact-head scoped decision, and the coordinator released only its experimental hold after verifying technical completion; neither action manufactures an eligible reviewer LGTM.ca4290aa8d8f25f031f3419c0f5a285bb91f082dpassed every configured required context with all 23 test shards executed. v2 CI includes passing generator/provenance regression, all three real build lanes, and protected aggregate. The obsolete checkout fixture now mutates actualbuild-go; aggregate permission/context/dependency drift coverage remains intact. Independent review confirms this is a test repair, not a weaker generator contract.holdremains. That failure is disclosed, not omitted to claim all-CI green. The remaining all-workflow critical path is native Podman ARM64:265s image build +76s startup/probes within356s execution. No five-minute all-CI or PR-to-merge claim is made.fe6beb935failed because current v5's generator also composesintegration-candidate.yml; refreshed integration repaired this. Prior head918ce743then failed the obsolete aggregate-checkout fixture. Both failures and their rework remain recorded separately. Headca4290aa8now has the fresh passing required verdict above;340s/361s onc916e1aremains the prior native measurement.ci-wallclock-20261031/.c916e1afull CI passed and native-runner/fork/isolation/release behavior received recorded COMMENT reviews. Earlier independent reviews asked @hanthor (also requested: @Danathar) about protected-aggregate semantics and public ARM64 runner selection; the subsequent ca4290a review asked @clubanderson to decide hold/overlap. These requests require an explicit human disposition, not an invented exclusive-reviewer requirement. The former coordinated hold has now been released under the recorded owner direction after exact-head technical CI passed. An eligible Prow LGTM/review and refreshed native-policy verdict—not the still-unmet five-minute target—remain on the normal merge path. The scan-membership question has been answered with exact CI logs showing 42 Crux-related paths among 204 unique directories. No self-approval or admin override substitutes for the requested decision.