feat(ci): CT-04 CodeQL with honest ruby/python/actions matrix (TIN-3398)

[Jess Sullivan (Jesssullivan)]

CT-04 — CodeQL with an honest language matrix

Wave 1 "Enforcement spine" milestone (2026-08-06). Linear project: ci-templates — Strict Governance & Org Interface Contract · Linear issue: TIN-3398. Closes #110.

Revised 2026-08-06 after opus-verify FLAG (see PR comment for the full defect list this addresses). Summary of what changed and why is in the "Review-fix changelog" section near the bottom — read that first if you reviewed the original version of this PR.

Completion metric (verbatim, from #110)

gh api repos/tinyland-inc/ci-templates/code-scanning/analyses --jq '[.[].language]|unique' contains ruby, python, actions; and gh api 'repos/tinyland-inc/ci-templates/code-scanning/alerts?state=open&severity=high' returns []; and gh api 'repos/tinyland-inc/ci-templates/code-scanning/alerts?state=open&severity=critical' returns [].

Honest metric status: 0/3 clauses provable pre-merge, all 3 structurally satisfiable

None of the three clauses can be true yet — no CodeQL run has ever executed against this repo, so code-scanning/analyses is empty and there is nothing for the alert-severity queries to return but [] (a vacuous, not yet meaningful, pass). This is not a deferral of scope; it's an inherent property of a code-scanning completion metric: it can only be evaluated after the workflow lands on main and runs once. This PR's job is to make the workflow structurally correct and its first run trustworthy, not to fake a pre-merge alert query.

• Clause 1 (languages): .github/workflows/codeql.yml adds a matrixed analyze job over [ruby, python, actions] — the actual code surface in this repo (scripts/*.rb, scripts/*.py, and the workflow/action YAML itself). All three use build-mode: none and queries: security-extended. Triggers: pull_request (branches: main), push (branches: main), and a weekly schedule. After merge, the first push:main run populates all three languages into code-scanning/analyses, satisfying clause 1.
• Clauses 2/3 (zero open high/critical alerts): unprovable until that first run exists. OPERATOR STEP (exact commands — syntax-verified live against tinyland-inc/ci-templates itself right now: code-scanning/analyses returns [null] and both alert-count queries return 0, confirming no CodeQL has ever run and the calls authenticate/parse correctly; re-run after this PR merges and the first push:main CodeQL run completes, typically visible within ~10 minutes post-merge):

  env -u GH_TOKEN -u GITHUB_TOKEN gh api repos/tinyland-inc/ci-templates/code-scanning/analyses --jq '[.[].language] | unique'
  env -u GH_TOKEN -u GITHUB_TOKEN gh api 'repos/tinyland-inc/ci-templates/code-scanning/alerts?state=open&severity=high' --jq 'length'
  env -u GH_TOKEN -u GITHUB_TOKEN gh api 'repos/tinyland-inc/ci-templates/code-scanning/alerts?state=open&severity=critical' --jq 'length'

  If either alert count is nonzero, each alert must be fixed or dismissed-with-reason (gh api -X PATCH repos/tinyland-inc/ci-templates/code-scanning/alerts/{number} -f state=dismissed -f dismissed_reason=<reason>) before the metric is truly met. This PR cannot pre-verify that outcome.

Scope note (per #110's own acceptance criterion #4 — "state plainly what CodeQL does not cover")

CodeQL has no shell analyzer. This repo's largest realized risk class is inline run: shell with ${{ }} expression interpolation (see PR #99 / TIN-3270 cited in #110) and scripts/*.sh. Neither is covered by this PR. The workflow file states this explicitly at the bottom (# What CodeQL does NOT cover here...):

• Inline run: blocks embedded in workflow/action YAML: not extracted or analyzed. Tracked as a real, un-solved gap.
• scripts/*.sh: not covered. A shellcheck job was in the original version of this PR but was dropped during review-fix — see "Review-fix changelog" below. Follow-up is tracked, not silently dropped.
• Vendored JSON (schemas/*.json, tinyland.repo.json): out of scope by construction, not code in any CodeQL-supported sense.
• Transitive dependency CVEs: out of scope for CT-04 — already covered by GitHub-native Dependabot (security_and_analysis.dependabot_security_updates=enabled, confirmed live).

Runner-class decision (explicit, does not inherit CT-01's tinyland-nix)

runs-on: ubuntu-latest. None of ruby/python/actions need the Nix devShell or Bazel cache tinyland-nix exists to provide (build-mode: none — nothing to build); the CodeQL bundle download is a hosted-runner-native path; and the ARC pool is under documented capacity pressure, so a scan job with zero cluster-native requirement should not consume shared tinyland-nix slots. Rationale is recorded in-file as a comment so it isn't silently assumed to carry over from CT-01.

Dependency note

#110 lists this as depending on CT-01 (#107, in-flight at the time of this PR). This workflow does not structurally depend on validate.yml existing — it's a standalone trigger surface — so it's implementable and provable independently. Sequencing onto branch protection (CT-02, #108) still correctly waits on this PR's check name landing, per CT-02's own completion metric.

Review-fix changelog (2026-08-06, addressing opus-verify FLAG)

The original version of this PR (975e25c) was flagged with 4 defects + minors. All are fixed in 068af20:

1. D1 (blocker) — shellcheck job was red on arrival. shellcheck scripts/*.sh exits 1 on unmodified main today (3× pre-existing SC2016 info findings in scripts/cache-attachment-contract*.sh; shellcheck's default severity floor treats info as fatal). Per the reviewer's own recommended options, the shellcheck job is dropped entirely — CodeQL is this PR's actual deliverable, and landing an enforcement job that fails on its own first run (including gating this PR's own pull_request trigger) is worse than shipping CodeQL alone with the gap stated honestly. Shell coverage is deferred to its own follow-up PR that also wires it into the Justfile check chain (doctrine: Justfile is the single validation entrypoint), rather than apt-get install-ing an unpinned shellcheck directly on a hosted runner outside that entrypoint (which was D4 in the same review, resolved for free by dropping the job).
2. D2 — floating action tags. All four third-party action refs are now pinned to full commit SHAs per .greptile/rules.md:7:
   • actions/checkout@v4 → actions/checkout@d23441a48e516b6c34aea4fa41551a30e30af803 # v6.1.0 — also fixes a two-major regression (@v4 was the only @v4 checkout in the repo; every other workflow is on @v6/v6.1.0's pinned SHA). Verified live: git ls-remote --tags https://github.com/actions/checkout.git → refs/tags/v6.1.0 = d23441a48e516b6c34aea4fa41551a30e30af803, byte-identical to the pin already used in spoke-ci-restricted.yml / spoke-lane-env-restricted.yml.
   • github/codeql-action/init@v3 and github/codeql-action/analyze@v3 → both pinned to c4dd10e44af883a891fe31ced449bcb4a6728b9b # v3.37.6. Verified live: git ls-remote --tags https://github.com/github/codeql-action.git → refs/tags/v3^{} (dereferenced annotated tag) = c4dd10e44af883a891fe31ced449bcb4a6728b9b; confirmed via gh api repos/github/codeql-action/tags that this commit is tagged v3.37.6.
3. D3 — no timeout-minutes. Added timeout-minutes: 20 to the analyze job (.greptile/rules.md: "Use timeout-minutes on all jobs"). The shellcheck job's timeout is moot — the job no longer exists.
4. Minor — concurrency.cancel-in-progress could cancel a push:main run. Narrowed to cancel-in-progress: ${{ github.event_name == 'pull_request' }} — redundant PR re-runs still get cancelled, but a rapid second push to main can no longer clobber the scan the zero-open-alerts metric clauses read from.
5. Minor — missing CHANGELOG entry. Added an [Unreleased] entry per golden rule 4 (missing from the original commit).
6. Minor items resolved as a byproduct of dropping the shellcheck job (no longer applicable): missing shell: bash on inline run: blocks, missing job-level permissions on the shellcheck job, fail-open empty-glob exit 0 branch.

Static proofs run locally (no workflow dispatches, per lane discipline)

Re-run on the fixed tree (068af20), in a clean isolated worktree:

• ruby -e 'require "yaml"; YAML.load_file(...)' over all .github/**/*.{yml,yaml} — parses clean (just yaml-parse).
• actionlint .github/workflows/codeql.yml — 0 errors.
• ruby scripts/lint-runs-on.rb --root . — 40 runs-on checked, 0 FAIL, 22 WARN across 11 workflow files (unchanged from the pre-existing baseline; dropping the shellcheck job removed one previously-checked literal, net zero new WARN/FAIL).
• ruby scripts/restricted-workflow-contract.rb — the only failure is the pre-existing, unrelated spoke-lane-env legacy-bytes drift (TIN-489/docs(tin-489): stop teaching Blahaj-dispatch PR-env routing — owner overlay is the producer #122, tracked and already fixed on branch fix/spoke-lane-env-pin-drift-20260806 / draft PR fix(ci): re-cut spoke-lane-env legacy byte pin after TIN-489 docs drift #129) — confirmed by diffing this branch's output against a clean origin/main checkout: identical single failure, nothing new introduced by this PR.
• Full just check (PATH adjusted so python3 resolves an interpreter with jsonschema installed — the nix devShell isn't buildable on this darwin sandbox due to a remote-builder machine mismatch unrelated to the repo; no repo files changed for that workaround): every target through restricted-workflow-contract-check passes; that target fails for the pre-existing reason above; every target after it (endpoint-free-check, ci-cached-endpoint-free-check, cache-backed-optin-contract-check, cache-contract-selftest, secrets-scan-dir, lint-runs-on-selftest, lint-runs-on-check) was run individually and passes clean.

No AI attribution in this commit/PR.

[linear-code]

TIN-3398

[Jess Sullivan (Jesssullivan)]

Verification pass (read-only) — FLAG, 4 defects

Verified against #110's COMPLETION METRIC and acceptance criteria verbatim, on a clean detached worktree of origin/ct-04-codeql-20260806 (975e25c) — i.e. not the shared scratchpad tree, so no other lane's untracked files contaminate the results.

What holds up

• ruby -e YAML.load_file over .github/**/*.{yml,yaml} — clean. yq — clean. actionlint .github/workflows/codeql.yml — 0 errors. Reproduced.
• ruby scripts/lint-runs-on.rb --root . on the clean worktree: 41 runs-on checked, 0 FAIL, 22 WARN across 11 workflow files. (The PR body's "42 / 12 files" was measured on the shared tree with CT-03's untracked secrets-scan.yml present. Conclusion unchanged: ubuntu-latest in codeql.yml adds no WARN and no FAIL.)
• Matrix [ruby, python, actions] + security-extended + pull_request/push:main/weekly schedule matches the deliverable. Criterion 4 (in-file inventory of what CodeQL does not cover) is genuinely satisfied. Runner-class rationale is recorded in-file, as the issue asked.
• Honest framing of the two unprovable metric clauses (open&severity=high|critical → []) is correct: they need a post-merge default-branch run.

D1 (blocker) — the shellcheck job is red on arrival

shellcheck scripts/*.sh fails on the repo as it stands today, before this PR changes anything:

$ shellcheck scripts/*.sh ; echo $?     # on origin/main, unmodified
scripts/cache-attachment-contract-selftest.sh:98:  SC2016 (info)
scripts/cache-attachment-contract.sh:130:          SC2016 (info)
scripts/cache-attachment-contract.sh:135:          SC2016 (info)
1

shellcheck's default severity floor is style, so info-level findings are fatal to the job. The workflow passes no --severity, adds no # shellcheck disable=SC2016 at the three sites, and does not fix them. The shellcheck job — the required substitute coverage for acceptance criterion 2 — therefore fails its very first run, including on this PR's own pull_request trigger. --severity=warning exits 0 today, if that is the intended floor; but it must be an explicit, justified choice, not left implicit.

Note this is also a hole in the claimed static-proof set: shellcheck is already in this repo's flake.nix devShell (line 27), so the job's own command was runnable locally and was not run. Everything that was run (yq/actionlint/ruby-YAML/lint-runs-on) only exercises the file's syntax, never its semantics.

D2 — floating action tags violate this repo's own pinning doctrine

Four unpinned third-party refs: actions/checkout@v4 (×2), github/codeql-action/init@v3, github/codeql-action/analyze@v3.

• .greptile/rules.md:7 — "Pin ALL third-party actions to full commit SHA (e.g. actions/checkout@a5ac7e5…), never floating tags."
• AGENTS.md Golden rule 1 — "…while third-party Actions use full 40-character commit SHAs."

Separately, @v4 is the only actions/checkout@v4 in the repo: all ten pre-existing workflows use @v6 or the SHA-pinned d23441a48e516b6c34aea4fa41551a30e30af803 # v6.1.0. That is a two-major regression against the repo's own floor, and with Dependabot live on Actions here (#96, #91) it will generate an immediate bump PR. For a workflow whose entire purpose is supply-chain security posture, unpinned github/codeql-action/* is the sharpest edge of the finding.

D3 — no timeout-minutes on either job

.greptile/rules.md — "Use timeout-minutes on all jobs to prevent runaway billing." Nine of the ten pre-existing workflows comply. Neither analyze nor shellcheck sets it.

D4 — shell coverage bypasses the Justfile entrypoint

AGENTS.md:36 ("Run just check before pushing") and the Justfile header ("Use just <recipe> locally and nix develop --command just <recipe> in CI") make the Justfile the single validation entrypoint; just check already aggregates 15 targets. This PR instead sudo apt-get installs an unpinned shellcheck on a hosted runner and shells out directly. Consequences: the check is invisible to a local just check; it is not reproducible through the devShell that already ships shellcheck; and it adds a second CI surface next to CT-01 (#128), whose validate lane runs exactly just check. The doctrine-conformant shape is a shellcheck recipe added to the check chain, executed by CT-01's lane.

Minor (non-blocking)

• Inline run: blocks omit explicit shell: bash (.greptile/rules.md, "Shell in Workflows").
• The shellcheck job has no job-level permissions block (inherits top-level contents: read, so no privilege issue — style only).
• Empty-glob branch exit 0s: if scripts/*.sh ever moves, the enforcement job goes silently green. Fail-open in a governance lane.
• concurrency.cancel-in-progress: true is keyed on github.ref and so also applies to push: main; a rapid second push to main can cancel the default-branch analysis that metric clause 1 depends on. Consider exempting the default branch.

Cross-lane note (not this PR's defect)

ruby scripts/restricted-workflow-contract.rb already fails on origin/main: spoke-lane-env: legacy workflow bytes changed (8e7e444f…); default-off proof invalid. Reproduced on a clean origin/main worktree, so it pre-dates this branch — but it means CT-01's just check lane (#128) will also be red on arrival until that baseline is resolved. Flagging for the Wave-1 sequencing, not for CT-04.

Verdict

FLAG. D1 is the blocker — the acceptance-criterion-2 job does not pass. D2 is the doctrine violation most at odds with the issue's intent. D3/D4 are conformance. The CodeQL matrix itself, and the honesty of the scope inventory, are sound and should survive the fix.

[Jess Sullivan (Jesssullivan)]

Fixed in 068af20 (see the review-fix changelog section of the updated PR body for full detail):

• D1 (blocker): shellcheck job dropped entirely per the recommended option (CodeQL is this PR's deliverable; job was red on arrival against unmodified main, unrelated to CodeQL). Deferred to its own follow-up PR wired into just check.
• D2: all four floating action refs pinned to full commit SHAs (checkout -> v6.1.0's d23441a4..., codeql-action/init+analyze -> v3.37.6's c4dd10e4...), both resolved live via git ls-remote --tags.
• D3: timeout-minutes: 20 added to the analyze job (the shellcheck job's timeout is moot since the job is gone).
• Minor: concurrency.cancel-in-progress narrowed to pull_request only so a push:main run can't be cancelled by a rapid second push; CHANGELOG [Unreleased] entry added.

just check re-verified clean through every target except the pre-existing spoke-lane-env pin-drift failure (tracked separately, draft PR #129) — confirmed identical on a clean origin/main worktree, nothing new introduced by this PR. Still draft per lane discipline.

[Jess Sullivan (Jesssullivan)]

Re-verification of the review-fix commit (068af20) — FLAG (1 substantive + 1 minor)

Read-only re-run of every reproduction step from the original FLAG (comment), in a clean throwaway clone of origin/ct-04-codeql-20260806 @ 068af206331280df230541ffbf43d98e67eaf63c with an origin/main worktree as the control.

Original defects: all four resolved, confirmed

Defect	Status	Evidence
D1 shellcheck red-on-arrival	RESOLVED	yq '.jobs | keys' → ["analyze"]. The job is gone, not disabled. grep -c 'run:' → 4, all inside comment prose; yq '.jobs.analyze.steps[]' shows 3 steps, all uses:, zero run:. Nothing in this workflow can execute shellcheck.
D2 floating action tags	RESOLVED	All 3 uses: are 40-char SHAs. Verified live: git ls-remote --tags actions/checkout → refs/tags/v6.1.0 = d23441a48e516b6c34aea4fa41551a30e30af803 (byte-identical to the pin already in spoke-ci-restricted.yml:146); git ls-remote --tags github/codeql-action → refs/tags/v3.37.6^{} = c4dd10e44af883a891fe31ced449bcb4a6728b9b. The v4→v6 regression is fixed.
D3 no timeout-minutes	RESOLVED	timeout-minutes: 20 on analyze, the only remaining job.
D4 shell coverage bypasses Justfile	RESOLVED (by removal)	No apt-get, no direct shellcheck invocation, no second CI surface.
MINORs (shell: bash, shellcheck job permissions, fail-open empty glob)	MOOT	No run: steps remain.
MINOR cancel-in-progress could cancel push:main	RESOLVED	cancel-in-progress: ${{ github.event_name == 'pull_request' }}; actionlint accepts the expression.

No regressions introduced

• just yaml-parse, json-parse, internal-refs-check, js-bazel-runner-contract-check, flywheel-reapi-proof-contract-check, endpoint-free-check, ci-cached-endpoint-free-check, cache-backed-optin-contract-check, cache-contract-selftest, manifest-validate-selftest, secrets-scan-dir, lint-runs-on-selftest, lint-runs-on-check — all rc=0 on this branch.
• just restricted-workflow-contract-check — rc=1 with the byte-identical message on this branch and on a clean origin/main worktree (spoke-lane-env: legacy workflow bytes changed (8e7e444f…)). Pre-existing, not this PR's. (repo-manifest-validate fails only because the nix devShell won't build on this darwin sandbox — environmental.)
• lint-runs-on.rb --root .: main = 39 checked / 0 FAIL / 22 WARN / 10 files; this branch = 40 / 0 / 22 / 11. Exactly +1 runs-on, no new WARN or FAIL — the author's number reproduces from the control.
• actionlint .github/workflows/codeql.yml → 0 findings. (Repo-wide actionlint has pre-existing findings in js-bazel-package.yml / spoke-ci.yml, untouched here.)
• Commit 068af20 is GPG-signed (%G? = G); PR still DRAFT; no AI attribution anywhere in the diff or commit message.

New: the first run will actually run

Checked because D1 was exactly this class of defect: repos/tinyland-inc/ci-templates is visibility: public (CodeQL is free, no GHAS blocker) and code-scanning/default-setup is state: not-configured (advanced setup will not be rejected for conflicting with default setup). The two operator-step commands in the PR body reproduce verbatim from here: analyses → [null], alerts?state=open&severity=high → 0. dependabot_security_updates: enabled also confirmed.

---

D5 (substantive) — Closes #110 will auto-close an issue with an explicitly unmet acceptance criterion, and the "tracked" follow-up does not exist

Issue #110 acceptance criterion 2 is verbatim: "Shellcheck runs in CI over every scripts/*.sh", and its Deliverables section says "Shell coverage substitute, in the same issue: shellcheck over scripts/*.sh… PR #95 already ran shellcheck manually — promote that to CI." Dropping the job is the right call for this PR — but the PR body still opens with "Closes #110.", so merging silently closes CT-04 with AC2 unfulfilled.

Compounding it, three separate places assert the deferral is tracked, with no citation:

• PR body: "Follow-up is tracked, not silently dropped."
• PR body: "Inline run: blocks … Tracked as a real, un-solved gap."
• codeql.yml:23-26: "shellcheck enforcement is deferred to its own follow-up PR…"

There is no such tracker. gh issue list --repo tinyland-inc/ci-templates --state all --search shellcheck returns only #110 and an unrelated closed #9; the 30 open issues contain no shellcheck/run:-extraction item, and no TIN id is cited for one.

Fix (either is fine, both are cheap):

• change Closes #110 → Refs #110 and leave CT-04 open until AC2 lands; or
• file the follow-up issue now (CT-04a: shellcheck over scripts/*.sh wired into the Justfile check chain, plus the three SC2016 sites it currently trips on) and cite its number in the PR body, the codeql.yml header comment, and the "What CodeQL does not cover" footer — then Closes #110 is defensible only if CT-04 - CodeQL with an honest language matrix (net-new, no site.scaffold precedent) #110's AC2 is formally re-scoped in the issue body.

D6 (minor) — security-events: write lacks the justification comment the same rules file requires

.greptile/rules.md:10: "Use permissions blocks at the job level. Default to read-only; justify any write permissions in comments." The analyze job (codeql.yml:72-75) grants security-events: write and actions: read with no inline rationale. This PR's whole revision was a conformance pass against that file (D2 = line 7, D3 = "Use timeout-minutes on all jobs"), so the one remaining clause from it should not be left open. One comment line — "security-events: write is required by codeql-action/analyze to upload SARIF; actions: read for workflow metadata on private-repo runs" — clears it.

[Jess Sullivan (Jesssullivan)]

Re-cutting, same reason as CT-01.

codeql.yml:70 pins runs-on: ubuntu-latest. The 2026-08-19 operator ruling (TIN-3914, 1f07c4a) retired GitHub-hosted runners from the estate with no opt-out input, and just no-hosted-runners-check scans every .github/**/*.yml label-aware and case-insensitive, so this file cannot land on main as written.

The premise holds: .github/workflows/codeql.yml is absent from main and this repo has no code scanning at all. The ruby/python/actions matrix is the honest one for what is actually in the tree, and the review fixes from 068af20 (dropping the red-on-arrival shellcheck, pinning actions, adding a timeout) and the job-level permissions rationale from 35c1423 should all come back.

The re-cut's actual work is the runner-class decision -- where the CodeQL bundle downloads and the analysis runs on a GF cache-fronted capability class. That is the PR, not a detail of it.