Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
14 changes: 13 additions & 1 deletion .github/workflows/ci.yml
Original file line number Diff line number Diff line change
Expand Up @@ -88,7 +88,7 @@ jobs:
- name: Check Zig formatting
run: >-
zig fmt --check
build.zig build.zig.zon src
build.zig build.zig.zon src benchmark
test/acceptance.zig test/dogfood.zig test/readme.zig test/readme

- name: Run ReleaseSafe unit, acceptance, dogfood, and consumer tests
Expand All @@ -97,6 +97,18 @@ jobs:
- name: Build and validate ReleaseSafe guide and API docs
run: zig build docs -Doptimize=ReleaseSafe

- name: Measure extraction and cached rule performance
run: zig build benchmark-check

- name: Upload performance evidence
if: always() && hashFiles('zig-out/benchmark/results.json') != ''
uses: actions/upload-artifact@043fb46d1a93c77aae656e7c1c64a875d1fc6a0a # v7.0.1
with:
name: performance-results
path: zig-out/benchmark/results.json
if-no-files-found: error
retention-days: 30

- name: Upload validated documentation site
uses: actions/upload-artifact@043fb46d1a93c77aae656e7c1c64a875d1fc6a0a # v7.0.1
with:
Expand Down
10 changes: 9 additions & 1 deletion CONTRIBUTING.md
Original file line number Diff line number Diff line change
Expand Up @@ -27,6 +27,7 @@ the repository documentation records decisions that outlive an individual ticket
python -B scripts/check_ci.py
zig build test -Doptimize=ReleaseSafe
zig build docs -Doptimize=ReleaseSafe
zig build benchmark-check
```

5. Open a pull request that links the issue, explains Zig-specific trade-offs, and records the exact
Expand Down Expand Up @@ -65,11 +66,18 @@ README/license root commit is the one unavoidable exception to the pull-request

CI runs Debug tests on current GitHub-hosted Linux, Windows, and macOS runners. A Linux quality job
checks formatting, reruns the complete suite in `ReleaseSafe`, builds the documentation site and Zig
API docs, and uploads the validated artifact. The complete test command includes unit tests,
API docs, runs the deterministic extraction benchmark, and uploads both validated artifacts. The
complete test command includes unit tests,
acceptance fixtures, executable dogfood rules, and the standalone README consumer.

Zig 0.16.0 is installed by `scripts/install_zig.py` from exact official archive URLs. The script
checks both the published byte size and SHA-256 before extraction. External workflow actions are
pinned to immutable commits with their reviewed release tag in a comment. CI deliberately uses no
dependency or build cache; adding one requires a key containing the Zig version and all relevant ZON
hash/lock inputs.

Performance work starts with evidence. `zig build benchmark` generates the versioned fixture and
writes `zig-out/benchmark/results.json`; `zig build benchmark-check` also enforces the broad budgets
recorded in `benchmark/budgets.json`. Update those budgets only from an uncached hosted Linux CI run,
and preserve substantial headroom for shared-runner variance. Correctness, deterministic graph
output, cache behavior, and zero live tracked allocations remain hard gates in both modes.
27 changes: 27 additions & 0 deletions benchmark/README.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,27 @@
# Extraction benchmark

`zig build benchmark` generates a deterministic Zig project under `.zig-cache/benchmark`, runs the
public extraction pipeline stage by stage in `ReleaseSafe`, exercises one cold and twenty cached
fluent checks, and writes machine-readable evidence to `zig-out/benchmark/results.json`.

The fixture has 240 source files in eight nested folders. Every file imports twelve peers and the
standard library, producing a dense but deterministic graph. Generation is deliberately outside
the measurements. The benchmark times enumeration, source loading, tokenize/parse,
resolution/classification, normalization, projection, the first rule check, and cached rule checks
separately. A tracking allocator records allocation calls, bytes requested, and peak live bytes for
each stage. Graph counts, a stable digest, projection cardinality, rule results, cache behavior, and
zero live tracked bytes are correctness gates even in report-only mode.

`zig build benchmark-check` additionally enforces `budgets.json`. Budgets are intentionally broad
hosted-CI regression tripwires, not microbenchmark promises or laptop-derived product guarantees.
The evidence block records the hosted run used to choose them. Update it only from an uncached
`ubuntu-latest` quality run, retaining enough headroom for normal shared-runner variance.

The current runtime ceilings are rounded above 25 times the recorded green Linux baseline, with
100–200 ms floors for filesystem stages too small for a useful ratio. The total ceiling is 20
seconds. The 64 MiB caller-allocator ceiling is more than eleven times the recorded peak. These are
coarse order-of-magnitude regression guards; changing fixture scale requires a new evidence run and
budget review rather than silently widening them.

The generated fixture is versioned in its directory name. Change that version whenever its shape or
contents change, so stale files cannot silently influence a new benchmark definition.
36 changes: 36 additions & 0 deletions benchmark/budgets.json
Original file line number Diff line number Diff line change
@@ -0,0 +1,36 @@
{
"schema_version": 1,
"evidence": {
"status": "hosted-ci-green",
"runner": "ubuntu-latest",
"run_url": "https://github.com/LukasNiessen/ArchUnitZig/actions/runs/32723254284",
"observed_commit": "8fcd3476c5db9828a9bdbde2afdb6f19a3983abb",
"baseline": {
"total_duration_ns": 784640735,
"peak_live_bytes": 5966099,
"stages": {
"enumeration_ns": 480079,
"source_loading_ns": 3344149,
"tokenize_parse_ns": 37220524,
"resolution_classification_ns": 151594513,
"normalization_ns": 36955236,
"projection_ns": 53994820,
"first_check_ns": 183165402,
"cached_checks_ns": 317886012,
"cached_check_per_iteration_ns": 15894301
}
}
},
"max_total_ns": 20000000000,
"max_peak_live_bytes": 67108864,
"stages": {
"enumeration_ns": 100000000,
"source_loading_ns": 200000000,
"tokenize_parse_ns": 1000000000,
"resolution_classification_ns": 4000000000,
"normalization_ns": 1000000000,
"projection_ns": 1500000000,
"first_check_ns": 5000000000,
"cached_check_per_iteration_ns": 500000000
}
}
46 changes: 46 additions & 0 deletions benchmark/contracts.zig
Original file line number Diff line number Diff line change
@@ -0,0 +1,46 @@
const std = @import("std");

test {
_ = @import("tracking_allocator.zig");
}

test "budgets cover every measured stage with positive values" {
var parsed = try std.json.parseFromSlice(
std.json.Value,
std.testing.allocator,
@embedFile("budgets.json"),
.{},
);
defer parsed.deinit();
const root = parsed.value.object;
try std.testing.expectEqual(@as(i64, 1), root.get("schema_version").?.integer);
try std.testing.expect(root.get("max_total_ns").?.integer > 0);
try std.testing.expect(root.get("max_peak_live_bytes").?.integer > 0);
const evidence = root.get("evidence").?.object;
try std.testing.expectEqualStrings("hosted-ci-green", evidence.get("status").?.string);
try std.testing.expect(std.mem.startsWith(
u8,
evidence.get("run_url").?.string,
"https://github.com/LukasNiessen/ArchUnitZig/actions/runs/",
));
const baseline = evidence.get("baseline").?.object;
try std.testing.expect(root.get("max_total_ns").?.integer >= baseline.get("total_duration_ns").?.integer * 25);
try std.testing.expect(root.get("max_peak_live_bytes").?.integer >= baseline.get("peak_live_bytes").?.integer * 8);
const stages = root.get("stages").?.object;
const baseline_stages = baseline.get("stages").?.object;
const expected = [_][]const u8{
"enumeration_ns",
"source_loading_ns",
"tokenize_parse_ns",
"resolution_classification_ns",
"normalization_ns",
"projection_ns",
"first_check_ns",
"cached_check_per_iteration_ns",
};
try std.testing.expectEqual(expected.len, stages.count());
for (expected) |name| {
try std.testing.expect(stages.get(name).?.integer > 0);
try std.testing.expect(stages.get(name).?.integer >= baseline_stages.get(name).?.integer * 25);
}
}
Loading