Skip to content

ci: call the shared acceptance-spec workflows for specs and spec-first proof - #2243

Merged
buggtb merged 4 commits into
developmentfrom
ci/use-shared-acceptance-spec
Oct 12, 2026
Merged

buggtb merged 4 commits into
developmentfrom
ci/use-shared-acceptance-spec

Conversation

@buggtb

@buggtb buggtb commented Oct 12, 2026 •

Copy link
Copy Markdown
Contributor

What

acceptance.yml stops carrying its own copies of the spec runner workflow and the spec-first proof. It becomes two thin callers of the reusable workflows in spiculedata/acceptance-spec (run-specs.yml, base-check.yml), pinned to the full SHA 65c4575aa9f7b7f64874c90a106db7f8456663e5, in mode: enforce. What stays here is what is Saiku's: the Maven build of the launcher with the GitHub Packages token, the loopback FoodMart demo start, the REST paths, the admin/admin demo login.

Also: deletes acceptance-base-check.mjs and its test (the shared copy has the same cases), adds acceptance-caller.test.mjs (guards the pin, enforce mode, check names, trust model), and updates docs/acceptance-specs.md, docs/ci-overview.md, acceptance/README.md, AGENTS.md, docs/review-rubric.md, docs/ci-feedback.md (logic lives upstream at a pinned SHA; "Bumping acceptance-spec"; "What is still local").

Deliberately NOT migrated: the ci / acceptance convention job and the local acceptance-check/runner/spec scripts. See gap 1 and 2 below.

Why

One copy of the logic instead of two, with Saiku's enforced behaviour unchanged. Where the shared workflows could not reproduce something Saiku enforces, I did not ship a regression: that part stays local and the missing capability is a separate upstream PR.

Compatibility analysis

Check names (a called job reads <caller job name> / <called job name>; every caller job here is named acceptance):

Check name Before Now
ci / acceptance convention ci.yml job, in the ci rollup unchanged (still local)
acceptance / detect changed specs acceptance.yml same name (from base-check)
acceptance / specs on base acceptance.yml same name (skipped when no spec changed, as before)
acceptance / spec fails on base acceptance.yml same name (skipped when no spec changed, as before)
acceptance / specs acceptance.yml; the hive label watcher reads it (EXTRA_CHECKS in quality-queue/label-watch.sh, exact check_name) kept by an alias job (name: acceptance / specs, needs: run-specs, red unless run-specs succeeded). The called job is named acceptance specs upstream, so it appears as the NEW acceptance / acceptance specs; no caller job id can produce acceptance / specs
acceptance / acceptance specs n/a NEW (the real run-specs job)
acceptance / specs on head n/a NEW (base-check's own head run)

Nothing outside this repository was edited. Names that are new are additive; no existing name changed or vanished.

Behaviour Reproduced?
Enforcing mode (red job on red spec / unproven base) Yes: mode: enforce. Unchanged: neither is in ci or a ruleset (only ci is required), so they report red but do not block, as before.
Convention gate (cutoff from vars.ACCEPTANCE_ENFORCE_FROM_ISSUE, acceptance-waived, inside ci) GAP 1, not migrated. Upstream's built-in waivable list is broader: `(^
Session login, base path Yes: session-path: /rest/saiku/session, base-path: /rest/saiku/api. Specs 818 and 2193 (session login, api) pass on head and base.
Credentials Names change to ACCEPTANCE_USER / ACCEPTANCE_PASSWORD, passed as literals admin / admin under secrets: (a reusable workflow can only take them that way). Side effect: GitHub masks the word "admin" in these logs. No existing spec uses {{env.SAIKU_*}} or SAIKU_BASE_URL (grepped), so none breaks; the docs now say not to.
Maven build, token, fork not-run Yes: BUILD_TOKEN: ${{ secrets.GH_PACKAGES_TOKEN }}, skip-without-build-token: true, same settings.xml and Maven command (written from $GITHUB_ACTOR/$BUILD_TOKEN inside build-command). Difference: on a fork, upstream skips the base build even when a cached base jar exists (before, a cache hit could still run api specs on a fork; head is not-run there anyway, so the verdict is not-run either way).
Base JAR cache keyed on base SHA GAP 3 (upstream bug). At the pinned SHA the cache path is base/./acceptance-launcher.jar, which actions/cache rejects ("Relative pathing '.' and '..' is not allowed"); nothing is saved, so a re-push rebuilds the base (~9 min, parallel with head, so about the same wall clock, more runner minutes). Fix: spiculedata/acceptance-spec#4. base-cache-paths is already set here and starts working on the next re-pin. Proven on a throwaway PR pointed at the fix: second push Cache hit for: acceptance-base-v1-..., Build the BASE app skipped, specs on base 43 s (vs 10 m 3 s cold).
Concurrency Same workflow-level group acceptance-specs-<PR>, cancel-in-progress.
Timeouts Same: 25 min build/run jobs (input), 5 min detect and verdict (upstream), alias 5 min.
Node / JDK node-version: "24", JDK 22 Temurin with Maven cache, as before.
Launcher start Same flags (SAIKU_DEMO=true, SAIKU_TELEMETRY=off, -XX:+ExitOnOutOfMemoryError, loopback :18080), detached by upstream's script; readiness 300 s on /rest/saiku/info (upstream accepts any 2xx, before exactly 200). The fat JAR is copied to acceptance-launcher.jar so start and cache need no glob.
ci-feedback.json integration GAP 4, dropped, not enforced. The CI run no longer writes the acceptance key into an artifact ci-feedback.json. Nothing reads it: ci-feedback.yml builds its own document from the API and never downloads PR artifacts (workflow header), and a grep of this repo and the hive tooling found no consumer. The local runner still has --feedback/attach and spec acceptance/2172 still asserts it. Results JSON is still uploaded; artifact names now carry the run id instead of the PR number.
Manual workflow_dispatch with target: deployed GAP 5, dropped, not enforced. Upstream refuses any event but pull_request. The ACCEPTANCE_BASE_URL variable is now unused.
Trust model pull_request only, contents: read, the only real secret is GH_PACKAGES_TOKEN; all actions SHA-pinned; scripts fetched at exactly the pinned SHA and verified (log: acceptance-spec scripts: spiculedata/acceptance-spec @ 65c4575...). Guard test added.
Cost A PR that touches no spec: same as before plus a few-second alias job. A PR that adds or changes a spec: the head launcher is built twice (acceptance specs and base-check's specs on head), 3 Maven builds instead of 2. Upstream's opt-in share-build avoids it but serialises base after head; not enabled.
Scripts and tests Deleted acceptance-base-check.mjs + test: every logic test case exists in upstream's acceptance-base-check.test.mjs (same names), the workflow-wiring tests are replaced by acceptance-caller.test.mjs. Kept acceptance-check.mjs, acceptance-runner.mjs, acceptance-spec.mjs + tests: the gate needs the first and last (gap 1), and acceptance/2172/check.mjs imports all three and asserts local behaviour including ci-feedback.

Verification

All on GitHub, SHA b989e320 and earlier commits of this PR; throwaway PRs #2245/#2246 are closed and their branches deleted, never merged.

  • This PR (no spec change): acceptance / acceptance specs pass (2m21s, all 9 specs pass incl. api specs with session login), acceptance / specs pass, acceptance / detect changed specs pass, specs on head / specs on base / spec fails on base skipped (as before), ci / acceptance convention pass, ci / flake-policy tests pass, ci pass. Every check that passed on development still exists under the same name and passes.
  • Convention still bites (THROWAWAY enforcement proof (red): do not merge #2245, body Closes #999990, no spec): ci / acceptance convention failure: acceptance convention: acceptance/999990/spec.json MISSING / this PR closes #999990 but adds no acceptance/999990/spec.json; ci failure.
  • Spec already green on base is rejected (THROWAWAY enforcement proof (red): do not merge #2245, adds acceptance/999991, session-login api spec that works on base): acceptance / specs on base success, specs on head success, acceptance / spec fails on base failure: acceptance/999991/spec.json: passes on the base branch: it proves nothing about this change....
  • Red on base, green on head is accepted (THROWAWAY enforcement proof (green): do not merge #2246, command spec checking a file only the PR adds): acceptance / spec fails on base success: red on base, green on head; ci / acceptance convention success (it ships its spec).
  • Cache fix proof: see gap 3 (throwaway pinned to the upstream fix).
  • Local: node --test .github/scripts/*.test.mjs 414 pass; python3 -m unittest pass; acceptance/2172 passes with the local scripts.

Risk

Low for enforcement: nothing that gated a merge changed (ci rollup and the convention job untouched). Medium for naming: acceptance / specs now exists only via the alias job; if the alias is removed the hive watcher silently goes blind to acceptance (it reads a missing check as pending). Known cost: extra head build on spec-changing PRs, and the base cache is off until the re-pin. Pin bumps are a one-line, two-place change (docs: "Bumping acceptance-spec").

Unverified

  • Fork PRs (no GH_PACKAGES_TOKEN): the not-run path was not exercised on GitHub; only reasoned from upstream's skip-without-build-token and its fixture tests.
  • spec-already-green label excusing a base-green spec was not exercised end to end (script cases are tested upstream).
  • Upstream PRs Building queries with dimensions with multiple hierarchies #3 and Connections to non-default MySQL ports fails - Upgrade MySQL Connector/J #4 are not merged; this PR does not depend on them. After they merge: re-pin, then move the convention job to convention.yml with Saiku's waivable-paths list and confirm the base cache hit.
  • hive/RUNBOOK.md line 165 still says acceptance / specs runs against demo.saiku.bi; it was already stale and is outside this repo (not edited).

🤖 Generated with Claude Code

…first proof

acceptance.yml becomes two thin callers of spiculedata/acceptance-spec
(run-specs.yml and base-check.yml) pinned to a full commit SHA, in
mode: enforce. Saiku keeps only what is its own: the Maven build of the
launcher with the GitHub Packages token, the loopback FoodMart demo start,
the REST paths and the admin/admin demo login. The caller jobs are named
`acceptance` so the called jobs keep the names `acceptance / detect changed
specs`, `acceptance / specs on base` and `acceptance / spec fails on base`;
a small `specs` job republishes the result as `acceptance / specs`, which
the hive label watcher keys on.

Removes the duplicated acceptance-base-check.mjs and its test (the shared
copy carries the same cases) and adds acceptance-caller.test.mjs to guard
the pin, the enforcing mode, the check names and the trust model. The
convention gate (`ci / acceptance convention`) and the local
acceptance-check/runner/spec scripts stay: see docs/acceptance-specs.md.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
Signed-off-by: Tom Barber <tom@spicule.co.uk>
@github-actions github-actions Bot added risk/high Change risk tier: high (see docs/risk-tiers.md) ci Created by Hive for agent-filed issue provenance and removed risk/high Change risk tier: high (see docs/risk-tiers.md) labels Oct 12, 2026
@github-actions

Copy link
Copy Markdown

Semantic model diff

No semantic model changed in this pull request, so there was nothing to diff.
saiku#1434 · reproduce locally with saiku model diff

…t a pinned SHA

acceptance-specs.md gains "What is still local" (the convention gate and its
scripts, and why) and "Bumping acceptance-spec"; ci-overview.md, acceptance/
README.md, AGENTS.md, review-rubric.md and ci-feedback.md point at them and
list the check names, the CI credential names (ACCEPTANCE_USER/PASSWORD) and
the dropped manual deployed run.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
Signed-off-by: Tom Barber <tom@spicule.co.uk>
@github-actions

github-actions Bot commented Oct 12, 2026 •

Copy link
Copy Markdown

CI feedback

passing · b989e32
Generated 2026-10-12T04:05:51.573Z. Machine-readable copy: ci-feedback.json (schema v1, see docs/ci-feedback.md) - artifact on this run.

Nothing to fix: every check in this commit concluded successfully.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
Signed-off-by: Tom Barber <tom@spicule.co.uk>
Upstream builds base/./acceptance-launcher.jar, which actions/cache rejects
(spiculedata/acceptance-spec#4 fixes it). The setting stays so the cache starts
working on the next re-pin.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
Signed-off-by: Tom Barber <tom@spicule.co.uk>
@buggtb buggtb changed the title ci: use the shared acceptance-spec workflows for specs and spec-first proof ci: call the shared acceptance-spec workflows for specs and spec-first proof Oct 12, 2026
@buggtb
buggtb marked this pull request as ready for review October 12, 2026 04:06
@buggtb
buggtb added this pull request to the merge queue Oct 12, 2026
Merged via the queue into development with commit 4066b9b Oct 12, 2026
23 checks passed
@github-actions

Copy link
Copy Markdown

PR acceptance metric

Acceptance rate: 89.8% — 447 merged / 498 decided PRs (2 still open, excluded from the denominator).

Metric Value
Acceptance rate 89.8%
Accepted / rejected / open 447 / 51 / 2
Median time to first review (merged) 0.5 h
Median time to merge (merged) 0.9 h
Median time open (decided) 1.2 h
Mean review rounds on merged PRs 1.0
Authors with at least one merged PR 6
Author Merged Rejected Acceptance
buggtb 195 4 98.0%
juanresendiz813 141 1 99.3%
app/dependabot 102 41 71.3%
app/spicule-hive 6 1 85.7%
oiacub1 2 0 100.0%
Prithvi1994 1 0 100.0%
laborima 0 3 0.0%
spicule-hive-bot 0 1 0.0%

Generated by scripts/pr-metrics.mjs — see docs/metrics.md.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

agent-config ci Created by Hive for agent-filed issue provenance docs risk/high Change risk tier: high (see docs/risk-tiers.md)

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant