Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion .github/workflows/ci-minimal.yml
Original file line number Diff line number Diff line change
Expand Up @@ -34,7 +34,7 @@ jobs:
GATE_EVENT: ${{ github.event_name }}
run: |
if [ "$GATE_EVENT" = "pull_request" ]; then
uvx --from git+https://github.com/zackees/ci.yml@1047fa448f90ea1c9c2abc66441c6b399e58bc29 ci-lint local-gate verify --repo .
uvx --from git+https://github.com/zackees/ci.yml@653196986fcc5d7469d74e8d4836b547729ee8e3 ci-lint local-gate verify --repo .
else
echo "Local proof verification is required on pull requests"
fi
Expand Down
18 changes: 12 additions & 6 deletions .github/workflows/dylint.yml
Original file line number Diff line number Diff line change
Expand Up @@ -42,10 +42,12 @@ jobs:
name: Dylint policy
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v6
- name: Checkout source
uses: actions/checkout@v6
with:
ref: ${{ inputs.ref }}
- uses: astral-sh/setup-uv@v3
- name: Set up uv
uses: astral-sh/setup-uv@v3
- name: Validate Dylint policy files
run: |
uv run --no-project python ci/check_dylint_allowlists.py
Expand All @@ -59,10 +61,12 @@ jobs:
run:
shell: bash
steps:
- uses: actions/checkout@v6
- name: Checkout source
uses: actions/checkout@v6
with:
ref: ${{ inputs.ref }}
- uses: astral-sh/setup-uv@v3
- name: Set up uv
uses: astral-sh/setup-uv@v3
- name: Validate Dylint allowlist paths
run: uv run --no-project python ci/check_dylint_allowlists.py
- name: Enforce shrink-only .fbuild allowlist
Expand All @@ -73,7 +77,8 @@ jobs:
uv run --no-project python ci/check_fbuild_path_baseline.py --base FETCH_HEAD
- name: Validate platform-boundary ledgers
run: uv run --no-project python ci/enforce_platform_boundary.py --print-totals
- uses: zackees/setup-soldr@v0
- name: Setup soldr
uses: zackees/setup-soldr@v0
with:
# ci-tests applies the prescribed bounded resource contract for the
# workspace validation domain. Dylint deliberately remains on its
Expand Down Expand Up @@ -257,7 +262,8 @@ jobs:
needs: [policy, dylint]
runs-on: ubuntu-latest
steps:
- env:
- name: Require successful Dylint jobs
env:
POLICY: ${{ needs.policy.result }}
FULL_DYLINT: ${{ needs.dylint.result }}
run: |
Expand Down
8 changes: 7 additions & 1 deletion ci-attestations.yml
Original file line number Diff line number Diff line change
@@ -1,10 +1,16 @@
# Both reusable Ubuntu jobs must pass. This does not prove boards, full mode,
# Windows/macOS execution or cross-target Dylint coverage.
# Windows/macOS execution or cross-target Dylint coverage. Ordinary-PR Dylint is a separate native lane.
version: 1
gates:
general/all/ubuntu-ci-guards: {lane: linux-minimal, fidelity: native}
rust/x86_64-unknown-linux-gnu/workspace-clippy: {lane: linux-minimal, fidelity: native}
rust/x86_64-unknown-linux-gnu/workspace-test: {lane: linux-minimal, fidelity: native}
rust/x86_64-unknown-linux-gnu/python-facade-test: {lane: linux-minimal, fidelity: native}
general/all/dylint-policy: {lane: dylint, fidelity: native}
rust/x86_64-unknown-linux-gnu/dylint-library-check: {lane: dylint, fidelity: native}
rust/x86_64-unknown-linux-gnu/workspace-dylint: {lane: dylint, fidelity: native}
jobs:
ci-minimal.yml:linux: [general/all/ubuntu-ci-guards, rust/x86_64-unknown-linux-gnu/workspace-clippy, rust/x86_64-unknown-linux-gnu/workspace-test, rust/x86_64-unknown-linux-gnu/python-facade-test]
dylint.yml:policy: [general/all/dylint-policy]
dylint.yml:dylint: [rust/x86_64-unknown-linux-gnu/dylint-library-check, rust/x86_64-unknown-linux-gnu/workspace-dylint]
dylint.yml:gate: [general/all/dylint-policy, rust/x86_64-unknown-linux-gnu/dylint-library-check, rust/x86_64-unknown-linux-gnu/workspace-dylint]
14 changes: 9 additions & 5 deletions ci/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -2,9 +2,10 @@

Python scripts for CI, packaging, and development tooling. All invoked via `uv run`.

`local_gate.py` runs both required Ubuntu jobs through bosn's pinned act2
engine, then replays the entire existing `dylint.yml` workflow with the ordinary
unlabelled PR event. It checks clean source identity, native Linux x64 execution,
`local_gate.py` has separate `--lane linux-minimal` and `--lane dylint` entrypoints.
The first runs both required Ubuntu jobs through bosn's pinned act2 engine;
the second replays the entire existing `dylint.yml` workflow with the ordinary
unlabelled PR event. The default runs both. It checks clean source identity, native Linux x64 execution,
all three Dylint job verdicts and the completed policy, library and workspace
steps. The wrapper uses released bosn 0.1.13 for the stock runner tools.
`local-gate.toml` enforces PR attestations. The local gate uses the existing
Expand All @@ -14,8 +15,11 @@ a substitute for the ordinary PR selection. Library UI fixtures retain their
existing source-change condition. Shared replay coverage binds every Ubuntu action and command to its literal
execution name, including the verifier prerequisite. Its event decision executes
on dispatch; the attestation check remains required on PRs. The original Ubuntu
terminal report is forwarded unchanged to the pinned shared checker; Dylint
retains its additional private command checks and still runs before stamping. Board builds, extended/full
and Dylint terminal reports are each forwarded unchanged to the pinned shared
checker. Every concrete job, prerequisite, action and validation command is
declared. The matching exact-hit guarded library cache save may be skipped;
validation steps still need executed-success evidence. Dylint retains its
additional private command checks, and both lanes must pass before stamping. Board builds, extended/full
mode and other native hosts retain their remote coverage. The migration is
tracked in [#1635](https://github.com/FastLED/fbuild/issues/1635), with Dylint
parity and complete cold/warm measurements in [#1643](https://github.com/FastLED/fbuild/issues/1643).
Expand Down
14 changes: 11 additions & 3 deletions ci/local_dylint_gate.py
Original file line number Diff line number Diff line change
Expand Up @@ -2,6 +2,8 @@

from __future__ import annotations

import json
import os
import subprocess
from dataclasses import dataclass
from pathlib import Path
Expand All @@ -12,6 +14,7 @@
RunSelection,
bosn_command,
document,
output,
verify_run,
wire_string,
)
Expand Down Expand Up @@ -133,6 +136,11 @@ def run_dylint(workspace: Path, sha: str) -> None:
run_id = wire_string(submitted, "run")
print(f"bosn ordinary-PR Dylint run: {run_id}", flush=True)
subprocess.run(bosn_command("ci", "wait", run_id), cwd=workspace, check=True)
verify_dylint(
document(bosn_command("ci", "show", run_id, "--json")), workspace, sha
)
report = output(bosn_command("ci", "show", run_id, "--json"))
raw: JsonValue = json.loads(report)
if not isinstance(raw, dict):
raise ValueError("bosn returned a non-object Dylint report")
verify_dylint(raw, workspace, sha)
report_path = os.environ.get("CI_LINT_GATE_REPLAY_REPORT")
if report_path:
Path(report_path).write_text(report, encoding="utf-8")
82 changes: 46 additions & 36 deletions ci/local_gate.py
Original file line number Diff line number Diff line change
@@ -1,11 +1,12 @@
"""Run both required Ubuntu jobs through bosn's pinned act2 engine.
"""Replay the selected Ubuntu or ordinary-PR Dylint workflow through act2.

This proof covers native Linux x64 only. Board builds, extended/full mode and
the other native hosts retain their remote validation.
"""

from __future__ import annotations

import argparse
import json
import os
import platform
Expand Down Expand Up @@ -187,7 +188,45 @@ def document(argv: list[str]) -> dict[str, JsonValue]:
return raw


def run_ubuntu(sha: str) -> None:
submitted = document(
bosn_command(
"ci",
"run",
"--workspace",
str(ROOT),
"--workflow",
WORKFLOW,
"--job",
"linux",
"--event",
"workflow_dispatch",
"--mode",
"minimal",
"--sha",
sha,
"--timeout-secs",
"7200",
"--json",
)
)
run_id = wire_string(submitted, "run")
print(f"bosn local gate run: {run_id}", flush=True)
subprocess.run(bosn_command("ci", "wait", run_id), cwd=ROOT, check=True)
report = output(bosn_command("ci", "show", run_id, "--json"))
raw: JsonValue = json.loads(report)
if not isinstance(raw, dict):
raise ValueError("bosn returned a non-object terminal report")
verify_run(RunProof.from_json(raw), ROOT, sha)
report_path = os.environ.get("CI_LINT_GATE_REPLAY_REPORT")
if report_path:
Path(report_path).write_text(report, encoding="utf-8")


def main() -> None:
parser = argparse.ArgumentParser(description=__doc__)
parser.add_argument("--lane", choices=("linux-minimal", "dylint"))
lane = parser.parse_args().lane
if (
output(["git", "symbolic-ref", "refs/remotes/origin/HEAD"]).strip()
!= "refs/remotes/origin/main"
Expand All @@ -203,7 +242,7 @@ def main() -> None:
"--with",
"pyyaml==6.0.2",
"--with",
"ci-lint @ git+https://github.com/zackees/ci.yml@1047fa448f90ea1c9c2abc66441c6b399e58bc29",
"ci-lint @ git+https://github.com/zackees/ci.yml@653196986fcc5d7469d74e8d4836b547729ee8e3",
"python",
"-m",
"unittest",
Expand Down Expand Up @@ -238,48 +277,19 @@ def main() -> None:
):
raise ValueError("Linux x64 tests require a native Linux x64 host and daemon")
sha = output(["git", "rev-parse", "HEAD"]).strip()
submitted = document(
bosn_command(
"ci",
"run",
"--workspace",
str(ROOT),
"--workflow",
WORKFLOW,
"--job",
"linux",
"--event",
"workflow_dispatch",
"--mode",
"minimal",
"--sha",
sha,
"--timeout-secs",
"7200",
"--json",
)
)
run_id = wire_string(submitted, "run")
print(f"bosn local gate run: {run_id}", flush=True)
subprocess.run(bosn_command("ci", "wait", run_id), cwd=ROOT, check=True)
report = output(bosn_command("ci", "show", run_id, "--json"))
raw: JsonValue = json.loads(report)
if not isinstance(raw, dict):
raise ValueError("bosn returned a non-object terminal report")
verify_run(RunProof.from_json(raw), ROOT, sha)
report_path = os.environ.get("CI_LINT_GATE_REPLAY_REPORT")
if report_path:
Path(report_path).write_text(report, encoding="utf-8")
if lane in (None, "linux-minimal"):
run_ubuntu(sha)
from ci.local_dylint_gate import run_dylint

run_dylint(ROOT, sha)
if lane in (None, "dylint"):
run_dylint(ROOT, sha)
if (
output(["git", "rev-parse", "HEAD"]).strip() != sha
or output(["git", "status", "--porcelain", "--untracked-files=normal"]).strip()
):
raise ValueError("worktree changed while the local gate ran")
print(
f"Passed both required Ubuntu jobs and the ordinary-PR Dylint workflow: {sha}",
f"Passed the selected native Linux workflow coverage: {sha}",
flush=True,
)

Expand Down
2 changes: 1 addition & 1 deletion ci/render_workflows.py
Original file line number Diff line number Diff line change
Expand Up @@ -392,7 +392,7 @@ def render_local_gate_verify() -> str:
GATE_EVENT: ${{ github.event_name }}
run: |
if [ "$GATE_EVENT" = "pull_request" ]; then
uvx --from git+https://github.com/zackees/ci.yml@1047fa448f90ea1c9c2abc66441c6b399e58bc29 ci-lint local-gate verify --repo .
uvx --from git+https://github.com/zackees/ci.yml@653196986fcc5d7469d74e8d4836b547729ee8e3 ci-lint local-gate verify --repo .
else
echo "Local proof verification is required on pull requests"
fi
Expand Down
12 changes: 11 additions & 1 deletion ci/test_shared_replay.py
Original file line number Diff line number Diff line change
Expand Up @@ -26,14 +26,24 @@ def test_repository_declares_complete_reusable_ubuntu_graph(self) -> None:
assert config is not None
self.assertEqual(check_replay_static(config, ROOT), [])
self.assertEqual(
{job.source_job for job in config.jobs},
{job.source_job for job in config.jobs if "linux-minimal" in job.lanes},
{
"ci-minimal.yml:verify",
"check-ubuntu.yml:check",
"check-ubuntu.yml:python-facade-tests",
},
)

def test_dylint_lane_declares_the_whole_pr_workflow(self):
raw = tomllib.loads((ROOT / "local-gate.toml").read_text())
selections = raw["gate"]["replay"]["selections"]
selected = [item for item in selections if item["lane"] == "dylint"]
self.assertEqual(len(selected), 1)
self.assertEqual(selected[0]["workflow"], "dylint.yml")
self.assertTrue(selected[0]["all-jobs"])
self.assertNotIn("job", selected[0])
self.assertEqual(selected[0]["event"], "pull_request")


if __name__ == "__main__":
unittest.main()
32 changes: 31 additions & 1 deletion local-gate.toml
Original file line number Diff line number Diff line change
Expand Up @@ -6,7 +6,7 @@ verify = "ci-minimal.yml:verify"
mode = "enforce"

[gate.lanes.linux-minimal]
run = ["uv", "run", "--no-project", "--python", "3.12", "python", "-m", "ci.local_gate"]
run = ["uv", "run", "--no-project", "--python", "3.12", "python", "-m", "ci.local_gate", "--lane", "linux-minimal"]
tools = ["bosn", "uv"]

# Shared proof binds every concrete Ubuntu job and its verifier prerequisite.
Expand Down Expand Up @@ -38,3 +38,33 @@ lane = "linux-minimal"
job = "linux"
event = "workflow_dispatch"
inputs = {}

[gate.lanes.dylint]
run = ["uv", "run", "--no-project", "--python", "3.12", "python", "-m", "ci.local_gate", "--lane", "dylint"]
tools = ["bosn", "uv"]

[[gate.replay.jobs]]
source-job = "dylint.yml:dylint"
key = "Dylint/Dylint Full (Linux builder, all OS targets)"
steps = ["Checkout source", "Set up uv", "Validate Dylint allowlist paths", "Enforce shrink-only .fbuild allowlist", "Validate platform-boundary ledgers", "Setup soldr", "Repair registry sources missing packaged files", "Prepare prebuilt Dylint tools and driver", "Install rustfmt for the Dylint toolchain", "Check Dylint library formatting", "Restore compiled Dylint libraries", "Test Dylint libraries", "Reuse cached Dylint library tree", "Run dylint over workspace", "Save compiled Dylint libraries"]
lanes = ["dylint"]
cache-save-steps = ["Save compiled Dylint libraries"]

[[gate.replay.jobs]]
source-job = "dylint.yml:gate"
key = "Dylint/Dylint"
steps = ["Require successful Dylint jobs"]
lanes = ["dylint"]

[[gate.replay.jobs]]
source-job = "dylint.yml:policy"
key = "Dylint/Dylint policy"
steps = ["Checkout source", "Set up uv", "Validate Dylint policy files"]
lanes = ["dylint"]

[[gate.replay.selections]]
lane = "dylint"
workflow = "dylint.yml"
all-jobs = true
event = "pull_request"
inputs = {}
Loading