Skip to content

refactor: single owner for the factory gate catalog — Justfile check vs validate.yml vs rebuild-rpms.yml prepare - #127

Closed
hivecommons-hive[bot] wants to merge 1 commit into
mainfrom
arch/gate-catalog-owner
Closed

hivecommons-hive[bot] wants to merge 1 commit into
mainfrom
arch/gate-catalog-owner

Conversation

@hivecommons-hive

Copy link
Copy Markdown
Contributor

Refactor

Refs #126 — this PR is Part A only. Part B (.github/workflows/validate.yml)
is deliberately left open, because nothing this agent can push may contain it;
see "What is still waiting" below.

just check is documented as "Everything CI gates on, minus the builds", but
nothing held it to that. The gate catalog was restated three times with
divergent contents:

Where Gates
Justfile check factory_contract.py, validate.py
.github/workflows/validate.yml factory_contract.py, validate.py, pytest tests
.github/workflows/rebuild-rpms.yml prepare check_workflow_quoting.py, runtime_contract.py … --check, unittest discover -s tests

tools/check_workflow_quoting.py — the linter that keeps a build script
embedded in a workflow from closing its own quote — and the runtime-contract
check were absent from the documented local gate entirely, so just check
green and CI green were different claims, in both directions.

Exact ground this PR claims

  • Justfile — check now runs all five gates, and the two orphans get
    their own named recipes (workflow-quoting, runtime-contract) so each is
    individually runnable. test joins the check chain.
  • tests/test_gate_catalog.py — new file. Recognises a gate
    structurally, not from a hand-written list: a workflow run: block is a
    gate step when every logical command in it begins with python3 and contains
    no $. That signature separates the fixed-argument validators, run purely
    for their exit status, from the pipeline tools, which are always
    parameterised by a package name or capture their output into a shell
    variable. It then asserts that set equals what just check reaches, in
    both directions
    — a workflow gate missing from just check fails, and so
    does a just check gate no workflow enforces. It also asserts the
    runtime-contract gate keeps --check, without which the tool dumps the
    resolved package list instead of reporting a verdict.

Nothing else is touched: no tool behavior, no workflow, no config, no recipe,
no package.

This follows the pattern the repository already uses for generated artifacts —
tests/test_render_packit_config.py gating .packit.yaml,
tools/publish_gate.py::assert_gate_enforced gating the publish rule — applied
to the gate catalog itself, which had no owner.

Verification

The recogniser resolves to exactly the five real gates across all 10 workflows,
with no false positives — source_pipeline.py, dist_bump.py,
import_rawhide.py, packit_workflow.py, scan_rawhide_state.py and
recalculate_hummingbird_gaps.py are all correctly excluded:

tools/check_workflow_quoting.py                                  <- rebuild-rpms.yml
tools/factory_contract.py                                        <- validate.yml
tools/runtime_contract.py config/bluefin-packages.toml config/runtime-contract.toml --check
                                                                 <- rebuild-rpms.yml
tools/validate.py                                                <- validate.yml
unit-tests                                                       <- rebuild-rpms.yml, validate.yml
  • python3 -m pytest tests/test_gate_catalog.py -q → 4 passed, 10 subtests passed
  • python3 -m unittest discover -s tests (the runner rebuild-rpms.yml prepare uses) collects it too
  • just --list and just --dry-run check parse and expand to the five gates in order
  • All five gates pass on this branch: checked 10 workflows: no build script contains a quote that would close it; validated runtime contract: 72 packages; factory contract satisfied; validated 191 source RPMs
  • Mutation checks — the gate is load-bearing, not tautological:
    • dropping workflow-quoting from check → SUBFAILED(gate='tools/check_workflow_quoting.py')
    • dropping --check from the runtime-contract recipe → 3 failures
    • adding a just check gate no workflow runs → SUBFAILED(gate='tools/publish_gate.py')
    • a fourth test fails if the recogniser ever matches fewer than four gates, so the other assertions cannot go vacuously green

Pre-existing red, untouched

main is currently red on three tests — test_package_inventory,
test_render_packit_config, test_source_inventory — because
packages/firefox/ was deleted in 0312fbe while the counts still say 192.
That is PR #113's ground and is not touched here. This branch adds no new
failure: the same three fail on main and on this branch, and
test_gate_catalog.py passes on both.

What is still waiting

The coverage hole described in #126 — check_workflow_quoting.py runs only in
rebuild-rpms.yml's prepare job, whose paths: filter excludes 8 of the 10
workflows, so a PR editing import-rawhide-package.yml or
packit-srpm-pilot.yml runs the shell-quoting linter zero times — is not
fixed by this PR
. Fixing it requires adding jobs to
.github/workflows/validate.yml, and this agent's App push token is minted at
the contributor tier, which carries no workflows permission; GitHub rejects
server-side any push whose diff touches .github/workflows/**. That is a token
ceiling, not a judgement about the change. #126 carries the exact replacement
YAML so applying it is mechanical, and needs a human or an ISSUES_PRS_MERGE
agent.

tests/test_gate_catalog.py accepts that follow-up change as-is: both commands
are already in the recognised gate set, and running them from a second workflow
only adds to the set of workflows mapped to the same gate identity.

Disjointness

Checked against every open PR on this repository. #103 claims
docs/architecture.md counts and tests/test_architecture_counts.py; #119
claims the rebuild-rpms.yml matrix resolver and tools/rebuild_matrix.py;
#113 claims the firefox removal and the four count assertions; #107 claims
tools/import_bluefin_rawhide.py coverage; #124 claims renovate.json. None
touches Justfile or tests/test_gate_catalog.py.


Filed by architect agent (ACMM L5 — hold-gated mode). Hold-gated: human review required.

— hive: agent=architect backend=copilot model=claude-opus-5

`just check` is documented as "Everything CI gates on, minus the builds", but
nothing held it to that. The gate catalog was restated three times with
divergent contents:

* Justfile `check` ran tools/factory_contract.py and tools/validate.py;
* .github/workflows/validate.yml ran those two plus the unit suite;
* the `prepare` job of .github/workflows/rebuild-rpms.yml ran
  tools/check_workflow_quoting.py, tools/runtime_contract.py --check and the
  unit suite under a different runner.

So tools/check_workflow_quoting.py -- the linter that keeps an embedded build
script from closing its own quote -- and the runtime-contract check were absent
from the documented local gate entirely, and a contributor running `just check`
got a different verdict from CI in both directions.

Justfile: `check` now runs all five gates, and each gate gets its own named
recipe so it is individually runnable.

tests/test_gate_catalog.py: new. Recognises a gate structurally rather than
from a hand-written list -- a workflow `run:` block is a gate step when every
logical command in it begins with `python3` and contains no `$`, which
separates the fixed-argument validators from the package-parameterised pipeline
tools -- and asserts the set matches what `just check` reaches, in both
directions. Also asserts the runtime-contract gate keeps `--check`, without
which the tool dumps the resolved package list instead of reporting a verdict.

This does not change which workflows run which gates; that part needs a
workflow-file edit this agent's push token cannot make.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Signed-off-by: sec-check[bot] <sec-check[bot]@users.noreply.github.com>
@hivecommons-hive hivecommons-hive Bot added the hold Work is intentionally paused. label Sep 17, 2026
@hivecommons-hive

Copy link
Copy Markdown
Contributor Author

Important

Held for human review by the hive's ACMM level gate.

This PR was opened by the "architect" agent while Hive policy required a human checkpoint for that agent. Non-outreach agents are held at ACMM L3–L5; the outreach agent is always held because it publishes project-facing communication.

Hive will automatically remove the hold label once current policy no longer requires a level hold for "architect". If this is an outreach PR, a human must review it and remove the label.

@hanthor hanthor left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The mechanism here is genuinely good, and better than your own description of it. I want to lead with that, because my objection is to the thesis, not to the code.

The structural recogniser is the right idea. Deriving "what is a gate" from the shape of the command — every logical command in the run: block starts with python3 and contains no $ — rather than from a hand-written constant is what keeps this from becoming the fourth stale list. I mutation-tested it rather than taking the PR body's word, and it holds up under pressure, including in a case you did not claim:

M1  drop workflow-quoting from `just check`
    -> SUBFAILED(gate='tools/check_workflow_quoting.py')   1 failed, 4 passed, 8 subtests
M2  drop --check from the runtime-contract recipe
    -> 3 failed, 3 passed  (both directions plus test_the_runtime_contract_gate_runs_in_checking_mode)
M3  add a `just check` gate no workflow runs
    -> SUBFAILED(gate='tools/publish_gate.py')             1 failed, 4 passed, 10 subtests
M6  delete `python3 tools/validate.py` from validate.yml only
    -> SUBFAILED(gate='tools/validate.py')                 1 failed, 4 passed, 8 subtests

M6 is mine, not yours, and it is the one that convinced me. If someone quietly drops a gate from CI while leaving it in the Justfile, the reverse-direction test catches it. That is the failure mode that actually matters and it is covered.

I also probed the $ escape hatch, which was my main worry — a gate rewritten as python3 tools/check_workflow_quoting.py $GITHUB_WORKSPACE disappears from the recognised set entirely:

M5  workflow gate parameterised with $
    -> 3 failed: SUBFAILED(check_workflow_quoting.py), SUBFAILED(runtime_contract.py ... --check),
       FAILED test_the_workflows_still_contain_recognisable_gates

It fails loudly, because the Justfile still lists the gate and now no workflow owns it. Good. Worth knowing that all three prepare gates share one run: block, so a single $ anywhere in that block drops all three at once — the blast radius is larger than one gate, but it is still noisy rather than silent.

And the Justfile parses and expands correctly:

$ just --dry-run check
python3 tools/factory_contract.py
python3 tools/validate.py
python3 tools/check_workflow_quoting.py
python3 tools/runtime_contract.py config/bluefin-packages.toml config/runtime-contract.toml --check
python3 -m pytest tests -q

All five gates pass on the branch, and python3 -m pytest tests -q goes 178 → 182 passed with 27 → 37 subtests against the merge-base I measured at 0312fbe. CI on merge commit 604f91b reports 3 failed, 182 passed, matching exactly. The three failures are the pre-existing firefox count breakage, present on the base too. mergeable_state is unstable for that reason alone.

Now the objection.

The title says "single owner for the factory gate catalog". The Justfile is not the single owner, and this PR does not make it one. Your table lists three sources of the catalog. I grepped for every remaining one and found three more that the PR leaves untouched and unguarded:

$ grep -rn "factory_contract\|check_workflow_quoting\|runtime_contract\|pytest tests" \
    .pre-commit-config.yaml docs/architecture.md README.md AGENTS.md docs/contributing.md

.pre-commit-config.yaml:32:        entry: python3 tools/factory_contract.py
.pre-commit-config.yaml:38:        entry: python3 tools/check_workflow_quoting.py
docs/architecture.md:139: | Factory onboarding contract | `tools/factory_contract.py` | ...
docs/architecture.md:140: | Package factory configuration | `tools/validate.py` | ...
docs/architecture.md:141: | Unit tests | `pytest tests` | ...
docs/architecture.md:143: `just check` runs the first two, `just test` the third, ...
README.md:131: just check   # factory contract + package configuration
AGENTS.md:30:  just check           # factory contract + configuration validation
docs/contributing.md:54: just check   # factory onboarding contract + package configuration

.pre-commit-config.yaml is a fourth independent restatement, and it diverges the same way the others did: it runs factory_contract.py and check_workflow_quoting.py, and omits validate.py, the runtime contract, and the tests. That is precisely the shape of drift this PR exists to eliminate, in a file the PR body does not mention at all. test_gate_catalog.py only reads .github/workflows/*.yml and the Justfile, so nothing notices.

docs/architecture.md:137-141 is a fifth — a hand-written gate table that names three of the five gates and does not mention check_workflow_quoting.py or runtime_contract.py. Again, exactly the divergence the PR is about, left in place.

And this PR actively makes docs/architecture.md:143 false. That line reads:

just check runs the first two, just test the third

After this change, just check runs all five including the test suite. The statement was true before the PR and is wrong after it, and nothing fails. tools/factory_contract.py checks that relative documentation links resolve, not that documented behaviour matches. For a PR whose entire thesis is that a restated gate catalog goes stale unless something owns it, shipping a new stale restatement in the same commit is the part I cannot wave through. The same applies more mildly to the # factory contract + package configuration comments in README.md, AGENTS.md and docs/contributing.md, which are now incomplete descriptions of what just check does.

There is also a real behavioural consequence you should decide on deliberately rather than inherit. Adding test to the check chain flips the local gate from green to red today:

$ cd wt/B17-base && just check; echo $?
0
$ cd wt/B17-127 && just check; echo $?
1

AGENTS.md:36 says "Run just check and just test before every commit. CI runs the same commands, so a green local run is the gate, not a second opinion." From this commit until #113 lands, no local run is green and that instruction cannot be followed. I think folding test into check is the right end state — it is what makes the Justfile a truthful catalog — but it should land after #113, or the sequencing should be called out.

One more thing, for the record rather than as a request. A consistency gate is not a preservation gate, and the anti-vacuity floor of 4 does not close that. I deleted check_workflow_quoting from both the Justfile and rebuild-rpms.yml in one go:

M4  gate deleted from BOTH Justfile and workflow
    -> 4 passed, 8 subtests passed

Green. Four gates remain, the two sides agree, nothing objects. That is inherent to what this test is — it enforces agreement, not existence — and I am not asking you to change it. But the docstring's "the gate catalog itself, which had no owner" oversells it: what now has an owner is the agreement between two of the five catalogs.

What I would like before approving:

  1. Update docs/architecture.md:143 and the gate table above it so they describe the five gates and what just check actually runs. This is the one I consider blocking — the PR introduces the error.
  2. Either bring .pre-commit-config.yaml into the recognised set, or say plainly in the PR body that it remains an unguarded fourth restatement, and retitle away from "single owner" to something like "make just check and the workflow gates agree", which is what the code honestly does and does well.
  3. Note the just check red-until-#113 sequencing, or reorder behind it.

None of this is a defect in test_gate_catalog.py, which I am happy with. It is a scope-versus-claim mismatch, and on a PR that is specifically about catalogs telling the truth, the claim has to be true too.


Generated by Claude Code

@hivecommons-hive hivecommons-hive Bot changed the title [architect] refactor: single owner for the factory gate catalog — Justfile check vs validate.yml vs rebuild-rpms.yml prepare refactor: single owner for the factory gate catalog — Justfile check vs validate.yml vs rebuild-rpms.yml prepare Sep 17, 2026

@hanthor hanthor left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The idea — one owner for the gate catalog, checked structurally — is good, and the Justfile half is fine. The structural gate detector is what fails: merged onto today's main it reports two false positives and fails its own test twice:

FAIL: test_just_check_runs_every_gate_ci_runs (gate='tools/rebuild_matrix.py')
  rebuild-rpms.yml runs 'tools/rebuild_matrix.py' but 'just check' does not
FAIL: test_just_check_runs_every_gate_ci_runs (gate='tools/source_pipeline.py --output work/sources --report-dir work/reports')
  bump-gnome-sources.yml runs 'tools/source_pipeline.py --output work/sources --report-dir work/reports' but 'just check' does not

Neither is a gate. rebuild_matrix.py is the planner — it writes the stage matrices to $GITHUB_OUTPUT (via the environment, not $ in the command line, which is why the "no $" heuristic misses it), and source_pipeline.py in the bump workflow re-downloads every rewritten tarball; both are pipeline steps that just check should never run. The signature "every command starts with python3 and contains no $" is not enough to separate a validator from a tool, and the next python3 tools/… step added to a workflow will trip it again.

Two ways out, either is fine: (a) mark gates explicitly — a # gate comment on the workflow step, or a small allowlist of tool basenames in the test, so the catalog is declared rather than inferred; or (b) restrict the detector to validate.yml and the Validate factory policy and tests step of prepare, which is where the gates actually live. With main having moved since this opened (#70, #133), please rebase as part of it.


Generated by Claude Code

@hanthor

hanthor commented Sep 18, 2026

Copy link
Copy Markdown
Member

Closing rather than waiting on the detector rework (two false positives on main, see review). The Justfile half — check running the quoting and runtime-contract gates — is worth carrying on its own and I'll fold it into the next CI change.


Generated by Claude Code

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

agent/scanner Filed or owned by the scanner agent. hold Work is intentionally paused.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant