Skip to content

bench: methodology, consumers, and judge documentation - #10

Merged
kavyabhand merged 4 commits into
mainfrom
bench/consumability-docs
Aug 2, 2026
Merged

kavyabhand merged 4 commits into
mainfrom
bench/consumability-docs

Conversation

@kavyabhand

Copy link
Copy Markdown
Member

Summary

  • Benchmark protocol and Linux Docker measurements for C -O2 vs Zig ReleaseSafe / ReleaseFast (distributions, RSS, startup, artifacts, plots). Fair headline is Safe; Safe interp arithmetic p99 was about 1.03% slower than C on that host and is disclosed.
  • Working C and Zig consumers (examples/c-consumer, examples/zig-consumer), b.addModule("tinyexpr"), external zig fetch of a package tarball verified.
  • Judge-facing README rewrite, docs/architecture.md, docs/faq-for-judges.md, timed docs/judge-quickstart.md, decisions D035–D037.

Suite assertion counts (unchanged)

Config Required
default 4930/4930
pow 4924/4924
natlog 4934/4934
pow_natlog 4928/4928

Test plan

  • CI green
  • README claims match scorecard (includes bench fragment)
  • bash examples/c-consumer/build.sh
  • cd examples/zig-consumer && zig build run

Copilot AI review requested due to automatic review settings August 2, 2026 10:53

Copilot AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

This PR expands tinyexpr.zig’s judge-facing documentation and adds a benchmarking harness + methodology, alongside concrete C and Zig “consumer” examples to demonstrate real-world consumption of both the C ABI and the native Zig API.

Changes:

  • Document native vs C-ABI surfaces and ownership/error conventions in src/root.zig / src/ast.zig, and overhaul the top-level README.md with equivalence scope, divergences, and benchmark headline numbers.
  • Add benchmark tooling (bench/run.sh, C harnesses, plot generator, methodology) plus recorded evidence fragments/logs/plots.
  • Add working consumer examples: examples/c-consumer (unmodified upstream header + Zig-built lib) and examples/zig-consumer (package module / zig fetch guidance), plus build-system support via b.addModule("tinyexpr", ...).

Reviewed changes

Copilot reviewed 33 out of 41 changed files in this pull request and generated 3 comments.

Show a summary per file
File Description
src/root.zig Expands module-level docs and clarifies native/C-ABI surfaces, ownership, and error semantics.
src/ast.zig Adds doc comments describing variable/function/closure bindings and purity.
README.md Rewrites judge-facing README content (scope/divergences/architecture/bench headline/consumers/build).
examples/zig-consumer/src/main.zig Adds a minimal Zig consumer demonstrating interp, compile, eval, and deinit.
examples/zig-consumer/README.md Documents in-tree usage and external zig fetch workflow via a tarball.
examples/zig-consumer/build.zig.zon Adds a zon manifest for the Zig consumer with an in-tree path dependency override.
examples/zig-consumer/build.zig Builds/runs the Zig consumer and wires in the tinyexpr module import.
examples/c-consumer/run-log.txt Captures an example run log validating static/shared linking behavior.
examples/c-consumer/README.md Documents how to build/run the C consumer and expected output.
examples/c-consumer/main.c Adds a third-party-style C program using upstream tinyexpr.h + Zig-built library.
examples/c-consumer/build.sh Adds a build script to compile/link/run the C consumer against static/shared libs.
docs/judge-quickstart.md Adds a timed “six minutes” quickstart for judges with concrete command checklist.
docs/journal.md Updates the engineering journal with additional bench/consumability/docs entry.
docs/faq-for-judges.md Adds a judge FAQ covering hashing, assertion counts, escape hatches, and scope.
docs/evidence/scorecard.json Updates evidence scorecard with benchmark measurements and updated hatch density.
docs/evidence/judge-quickstart-timer.json Adds timer artifact supporting the quickstart’s time claims.
docs/evidence/fragments/bench-linux-docker.json Adds benchmark evidence fragment used by scorecard/README.
docs/evidence/escape-hatches.table.txt Updates hatch/LOC derived numbers post-change.
docs/evidence/escape-hatches.json Updates hatch/LOC derived numbers post-change.
docs/evidence/consumability/zig-fetch-save.txt Adds consumability evidence artifact (zig fetch save transcript).
docs/evidence/consumability/zig-fetch-external-run.txt Adds evidence of a successful external fetch+run.
docs/evidence/consumability/README.md Adds a one-line summary of the consumability evidence.
docs/architecture.md Adds an architecture overview describing the shared parser + dual builders design.
DECISIONS.md Extends decision log with benchmark/consumability/judge-doc decisions (D035–D037) and formatting tweaks.
build.zig Adds b.addModule("tinyexpr", ...) for consumers and wires a runnable bench build step.
bench/run.sh Adds end-to-end benchmark orchestration generating bench/results.json and artifacts.
bench/run-docker.sh Adds helper to run benches inside the project’s Ubuntu Docker image.
bench/plots/throughput.svg Adds generated throughput plot.
bench/plots/startup_warm.svg Adds generated startup warm plot.
bench/plots/README.md Documents plot generation and retained data level (percentiles only).
bench/plots/interp_arithmetic.svg Adds generated te_interp arithmetic latency plot.
bench/plots/eval_variables.svg Adds generated te_eval variables latency plot.
bench/plots/compile_functions.svg Adds generated te_compile functions latency plot.
bench/plot.py Adds no-deps SVG plot generator from bench/results.json.
bench/methodology.md Replaces placeholder with full benchmark protocol and measured environment section.
bench/logs/run-docker.txt Adds captured Docker benchmark run log.
bench/logs/run-darwin.txt Adds captured Darwin benchmark run log.
bench/harness/startup.c Adds minimal startup probe binary for spawn-time measurement.
bench/harness/latency.c Adds JSON-producing C harness for latency/throughput/RSS measurements.
.gitignore Ignores bench outputs and example build outputs while keeping sources/results tracked.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

Comment thread bench/harness/latency.c
Comment on lines +164 to +166
static void run_interp(const char *impl, const char *cls, int warmup, int measured) {
const char *e = expr_for(cls);
for (int i = 0; i < warmup; i++) {
Comment thread bench/methodology.md
For every latency series of N measured samples (after discarding warmup):

- minimum, maximum
- p50, p90, p99, p99.9 (nearest-rank on sorted samples)
"$OUT/c-consumer-static"

echo "== shared (ReleaseSafe) =="
# Rebuild ensures .so/.dylib is installed
@kavyabhand
kavyabhand merged commit e1c8ab6 into main Aug 2, 2026
43 checks passed
@kavyabhand
kavyabhand deleted the bench/consumability-docs branch August 2, 2026 10:57
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants