Repository navigation
bench: methodology, consumers, and judge documentation - #10
Merged
Merged
Conversation
There was a problem hiding this comment.
Pull request overview
This PR expands tinyexpr.zig’s judge-facing documentation and adds a benchmarking harness + methodology, alongside concrete C and Zig “consumer” examples to demonstrate real-world consumption of both the C ABI and the native Zig API.
Changes:
- Document native vs C-ABI surfaces and ownership/error conventions in
src/root.zig/src/ast.zig, and overhaul the top-levelREADME.mdwith equivalence scope, divergences, and benchmark headline numbers. - Add benchmark tooling (
bench/run.sh, C harnesses, plot generator, methodology) plus recorded evidence fragments/logs/plots. - Add working consumer examples:
examples/c-consumer(unmodified upstream header + Zig-built lib) andexamples/zig-consumer(package module /zig fetchguidance), plus build-system support viab.addModule("tinyexpr", ...).
Reviewed changes
Copilot reviewed 33 out of 41 changed files in this pull request and generated 3 comments.
Show a summary per file
| File | Description |
|---|---|
| src/root.zig | Expands module-level docs and clarifies native/C-ABI surfaces, ownership, and error semantics. |
| src/ast.zig | Adds doc comments describing variable/function/closure bindings and purity. |
| README.md | Rewrites judge-facing README content (scope/divergences/architecture/bench headline/consumers/build). |
| examples/zig-consumer/src/main.zig | Adds a minimal Zig consumer demonstrating interp, compile, eval, and deinit. |
| examples/zig-consumer/README.md | Documents in-tree usage and external zig fetch workflow via a tarball. |
| examples/zig-consumer/build.zig.zon | Adds a zon manifest for the Zig consumer with an in-tree path dependency override. |
| examples/zig-consumer/build.zig | Builds/runs the Zig consumer and wires in the tinyexpr module import. |
| examples/c-consumer/run-log.txt | Captures an example run log validating static/shared linking behavior. |
| examples/c-consumer/README.md | Documents how to build/run the C consumer and expected output. |
| examples/c-consumer/main.c | Adds a third-party-style C program using upstream tinyexpr.h + Zig-built library. |
| examples/c-consumer/build.sh | Adds a build script to compile/link/run the C consumer against static/shared libs. |
| docs/judge-quickstart.md | Adds a timed “six minutes” quickstart for judges with concrete command checklist. |
| docs/journal.md | Updates the engineering journal with additional bench/consumability/docs entry. |
| docs/faq-for-judges.md | Adds a judge FAQ covering hashing, assertion counts, escape hatches, and scope. |
| docs/evidence/scorecard.json | Updates evidence scorecard with benchmark measurements and updated hatch density. |
| docs/evidence/judge-quickstart-timer.json | Adds timer artifact supporting the quickstart’s time claims. |
| docs/evidence/fragments/bench-linux-docker.json | Adds benchmark evidence fragment used by scorecard/README. |
| docs/evidence/escape-hatches.table.txt | Updates hatch/LOC derived numbers post-change. |
| docs/evidence/escape-hatches.json | Updates hatch/LOC derived numbers post-change. |
| docs/evidence/consumability/zig-fetch-save.txt | Adds consumability evidence artifact (zig fetch save transcript). |
| docs/evidence/consumability/zig-fetch-external-run.txt | Adds evidence of a successful external fetch+run. |
| docs/evidence/consumability/README.md | Adds a one-line summary of the consumability evidence. |
| docs/architecture.md | Adds an architecture overview describing the shared parser + dual builders design. |
| DECISIONS.md | Extends decision log with benchmark/consumability/judge-doc decisions (D035–D037) and formatting tweaks. |
| build.zig | Adds b.addModule("tinyexpr", ...) for consumers and wires a runnable bench build step. |
| bench/run.sh | Adds end-to-end benchmark orchestration generating bench/results.json and artifacts. |
| bench/run-docker.sh | Adds helper to run benches inside the project’s Ubuntu Docker image. |
| bench/plots/throughput.svg | Adds generated throughput plot. |
| bench/plots/startup_warm.svg | Adds generated startup warm plot. |
| bench/plots/README.md | Documents plot generation and retained data level (percentiles only). |
| bench/plots/interp_arithmetic.svg | Adds generated te_interp arithmetic latency plot. |
| bench/plots/eval_variables.svg | Adds generated te_eval variables latency plot. |
| bench/plots/compile_functions.svg | Adds generated te_compile functions latency plot. |
| bench/plot.py | Adds no-deps SVG plot generator from bench/results.json. |
| bench/methodology.md | Replaces placeholder with full benchmark protocol and measured environment section. |
| bench/logs/run-docker.txt | Adds captured Docker benchmark run log. |
| bench/logs/run-darwin.txt | Adds captured Darwin benchmark run log. |
| bench/harness/startup.c | Adds minimal startup probe binary for spawn-time measurement. |
| bench/harness/latency.c | Adds JSON-producing C harness for latency/throughput/RSS measurements. |
| .gitignore | Ignores bench outputs and example build outputs while keeping sources/results tracked. |
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
Comment on lines
+164
to
+166
| static void run_interp(const char *impl, const char *cls, int warmup, int measured) { | ||
| const char *e = expr_for(cls); | ||
| for (int i = 0; i < warmup; i++) { |
| For every latency series of N measured samples (after discarding warmup): | ||
|
|
||
| - minimum, maximum | ||
| - p50, p90, p99, p99.9 (nearest-rank on sorted samples) |
| "$OUT/c-consumer-static" | ||
|
|
||
| echo "== shared (ReleaseSafe) ==" | ||
| # Rebuild ensures .so/.dylib is installed |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
-O2vs ZigReleaseSafe/ReleaseFast(distributions, RSS, startup, artifacts, plots). Fair headline is Safe; Safe interp arithmetic p99 was about 1.03% slower than C on that host and is disclosed.examples/c-consumer,examples/zig-consumer),b.addModule("tinyexpr"), externalzig fetchof a package tarball verified.docs/architecture.md,docs/faq-for-judges.md, timeddocs/judge-quickstart.md, decisions D035–D037.Suite assertion counts (unchanged)
Test plan
bash examples/c-consumer/build.shcd examples/zig-consumer && zig build run