Skip to content

docs: one-command reproduction, and what orchestrate check guarantees #13

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
Immanuel-Fowler wants to merge 2 commits into from
Closed

docs: one-command reproduction, and what orchestrate check guarantees #13

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
2 commits
Select commit Hold shift + click to select a range
afbc07f
docs: one-command reproduction, and what check guarantees
Immanuel-Fowler Sep 21, 2026
8b89757
docs(benchmarks): sample tables from one run_all.py run
Immanuel-Fowler Sep 21, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
10 changes: 10 additions & 0 deletions CHANGELOG.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -80,6 +80,16 @@ own changelog in [editors/vscode/CHANGELOG.md](editors/vscode/CHANGELOG.md).
says so. `benchmarks/diagnostics_coverage.py` writes the same classification as CSV,
JSON, and Markdown.

- **One command regenerates every table.** `python3 benchmarks/run_all.py` builds the
release compiler and runs the boundary ladder, the tick cost, and the diagnostics
coverage in order, writing every CSV, JSON, and Markdown table under
`benchmarks/results/` and joining the reports into `results.md`. `benchmarks/README.md`
documents it and the three scripts.
- **What `orchestrate check` guarantees is written down**, in one paragraph of
`docs/language-reference.md` before "Debugging Generated Code": what a check does and
finds, and what it does not — no codegen, no Cargo, library-only rules unenforced,
module serverlet bodies unwalked, foreign sources unchecked.

### Fixed
- **Sixteen wrong programs are told so by `orchestrate check` instead of rustc.** The
corpus found them: a call into a module or a foreign function with the wrong argument
Expand Down
190 changes: 120 additions & 70 deletions benchmarks/README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -3,16 +3,38 @@
Every table below is generated by a script from a run's machine-readable output. Do not
retype numbers: rerun the script and paste the Markdown it prints.

## One command

```sh
python3 benchmarks/run_all.py
```

builds `target/release/orchestrate` and runs the three scripts below in order, writing
every table under `benchmarks/results/` (gitignored) and joining the three Markdown
reports into `results.md`. `--no-typescript` skips the TypeScript rung of the ladder,
`--skip <name>` leaves a step out, `--out DIR` writes elsewhere, and `ORCHESTRATE=<path>`
uses a compiler you already built. Run it on an idle machine: two of the three are timing
benchmarks. Python 3.10+ is required.

| Script | Table | Output |
|---|---|---|
| `benchmarks/landline_latency/run.py` | the boundary ladder: payload, call-rate, concurrent-callers, and state sweeps | `boundary_ladder.{csv,json,md}`, `boundary_ladder_samples.csv` |
| `benchmarks/tick_cost/run.py` | the library-mode tick cost | `tick_cost.{csv,json,md}` |
| `benchmarks/diagnostics_coverage.py` | how many invalid programs `orchestrate check` rejects | `diagnostics_coverage.{csv,json,md}` |

Each script can also run alone:

```sh
cargo build --release
ORCHESTRATE=target/release/orchestrate python3 benchmarks/landline_latency/run.py
ORCHESTRATE=target/release/orchestrate python3 benchmarks/tick_cost/run.py
ORCHESTRATE=target/release/orchestrate python3 benchmarks/diagnostics_coverage.py
```

Each script writes CSV, JSON, and Markdown under `benchmarks/results/` (gitignored), and
the JSON carries an environment header: CPU, OS, architecture, `orchestrate`, `rustc`,
`cargo`, Python, the TypeScript toolchain and backend, the wasmtime version, the sample
counts, the commit SHA, and whether the tree was dirty. Python 3.10+ is required.
Each writes CSV, JSON, and Markdown under `benchmarks/results/`, and the JSON carries an
environment header: CPU, OS, architecture, `orchestrate`, `rustc`, `cargo`, Python, the
TypeScript toolchain and backend, the wasmtime version, the sample counts, the commit
SHA, and whether the tree was dirty.

Neither benchmark runs in CI, because shared runners are too noisy for timing. CI only
checks that `landline_latency/main.orch` still typechecks.
Expand Down Expand Up @@ -68,83 +90,83 @@ Reading the results:
### Sample results


One run on Apple M2 (macOS-15.3.2-arm64-arm-64bit, arm64), 2026-09-21T03:08:59+00:00, commit `417e0b591ada` (dirty tree). rustc 1.98.1 (48a229cea 2026-09-01); Python 3.12.7; wasmtime 37.0.3; TypeScript rung included. Round trips in microseconds unless the unit says otherwise; the state rungs are nanoseconds per statement. Your numbers will differ.
One run on Apple M2 (macOS-15.3.2-arm64-arm-64bit, arm64), 2026-09-21T03:54:37+00:00, commit `afbc07fb4be4`. rustc 1.98.1 (48a229cea 2026-09-01); Python 3.12.7; wasmtime 37.0.3; TypeScript rung included. Round trips in microseconds unless the unit says otherwise; the state rungs are nanoseconds per statement. Your numbers will differ.

#### Payload sweep

Calls back to back, one caller.

| Kind | Payload | Samples | p50 | p90 | p99 | max |
|---|---|---:|---:|---:|---:|---:|
| in-process | int | 2000 | 11 | 15 | 24 | 128 |
| in-process | string-1KB | 2000 | 8 | 11 | 13 | 20 |
| in-process | int[1000] | 2000 | 7 | 9 | 12 | 40 |
| sandbox | int | 2000 | 8 | 11 | 14 | 36 |
| sandbox | string-1KB | 2000 | 10 | 13 | 15 | 22 |
| sandbox | int[1000] | 2000 | 13 | 16 | 20 | 27 |
| secret | int | 2000 | 17 | 21 | 31 | 44 |
| secret | string-1KB | 2000 | 18 | 22 | 33 | 43 |
| secret | int[1000] | 2000 | 24 | 30 | 39 | 85 |
| python | int | 2000 | 25 | 27 | 39 | 101 |
| python | string-1KB | 2000 | 27 | 29 | 40 | 57 |
| python | int[1000] | 2000 | 557 | 578 | 639 | 781 |
| typescript | int | 2000 | 46 | 54 | 70 | 645 |
| typescript | string-1KB | 2000 | 48 | 75 | 159 | 1101 |
| typescript | int[1000] | 2000 | 314 | 941 | 1254 | 1626 |
| in-process | int | 2000 | 14 | 20 | 31 | 136 |
| in-process | string-1KB | 2000 | 8 | 12 | 16 | 41 |
| in-process | int[1000] | 2000 | 7 | 10 | 14 | 29 |
| sandbox | int | 2000 | 7 | 15 | 31 | 43 |
| sandbox | string-1KB | 2000 | 9 | 21 | 40 | 271 |
| sandbox | int[1000] | 2000 | 13 | 18 | 45 | 140 |
| secret | int | 2000 | 17 | 22 | 34 | 77 |
| secret | string-1KB | 2000 | 17 | 22 | 34 | 96 |
| secret | int[1000] | 2000 | 25 | 31 | 45 | 118 |
| python | int | 2000 | 24 | 32 | 49 | 186 |
| python | string-1KB | 2000 | 25 | 29 | 44 | 131 |
| python | int[1000] | 2000 | 568 | 596 | 677 | 1114 |
| typescript | int | 2000 | 45 | 54 | 69 | 649 |
| typescript | string-1KB | 2000 | 47 | 57 | 73 | 535 |
| typescript | int[1000] | 2000 | 314 | 1024 | 1393 | 2050 |

#### Call-rate sweep

The `int` payload. At 100 Hz and 1 kHz the caller sleeps between calls; at 10 kHz it spins.

| Kind | Rate | Samples | p50 | p90 | p99 | max |
|---|---|---:|---:|---:|---:|---:|
| in-process | unpaced | 2000 | 11 | 15 | 24 | 128 |
| in-process | 100hz | 300 | 37 | 58 | 76 | 90 |
| in-process | 1khz | 1000 | 30 | 38 | 63 | 377 |
| in-process | 10khz | 2000 | 7 | 10 | 19 | 77 |
| sandbox | unpaced | 2000 | 8 | 11 | 14 | 36 |
| sandbox | 100hz | 300 | 38 | 59 | 93 | 147 |
| sandbox | 1khz | 1000 | 31 | 40 | 57 | 140 |
| sandbox | 10khz | 2000 | 7 | 11 | 29 | 256 |
| secret | unpaced | 2000 | 17 | 21 | 31 | 44 |
| secret | 100hz | 300 | 70 | 108 | 145 | 575 |
| secret | 1khz | 1000 | 56 | 76 | 103 | 286 |
| secret | 10khz | 2000 | 15 | 24 | 38 | 91 |
| python | unpaced | 2000 | 25 | 27 | 39 | 101 |
| python | 100hz | 300 | 78 | 165 | 225 | 303 |
| python | 1khz | 1000 | 57 | 113 | 165 | 417 |
| python | 10khz | 2000 | 21 | 26 | 38 | 139 |
| typescript | unpaced | 2000 | 46 | 54 | 70 | 645 |
| typescript | 100hz | 300 | 126 | 231 | 481 | 737 |
| typescript | 1khz | 1000 | 106 | 189 | 276 | 495 |
| typescript | 10khz | 2000 | 42 | 61 | 88 | 380 |
| in-process | unpaced | 2000 | 14 | 20 | 31 | 136 |
| in-process | 100hz | 300 | 33 | 48 | 113 | 379 |
| in-process | 1khz | 1000 | 29 | 41 | 58 | 119 |
| in-process | 10khz | 2000 | 7 | 11 | 42 | 416 |
| sandbox | unpaced | 2000 | 7 | 15 | 31 | 43 |
| sandbox | 100hz | 300 | 34 | 50 | 66 | 244 |
| sandbox | 1khz | 1000 | 31 | 42 | 54 | 565 |
| sandbox | 10khz | 2000 | 7 | 11 | 19 | 33 |
| secret | unpaced | 2000 | 17 | 22 | 34 | 77 |
| secret | 100hz | 300 | 60 | 92 | 128 | 152 |
| secret | 1khz | 1000 | 50 | 73 | 104 | 1028 |
| secret | 10khz | 2000 | 15 | 26 | 36 | 84 |
| python | unpaced | 2000 | 24 | 32 | 49 | 186 |
| python | 100hz | 300 | 85 | 141 | 241 | 629 |
| python | 1khz | 1000 | 59 | 104 | 150 | 641 |
| python | 10khz | 2000 | 19 | 28 | 38 | 50 |
| typescript | unpaced | 2000 | 45 | 54 | 69 | 649 |
| typescript | 100hz | 300 | 108 | 321 | 2631 | 4782 |
| typescript | 1khz | 1000 | 92 | 202 | 457 | 17651 |
| typescript | 10khz | 2000 | 41 | 50 | 67 | 331 |

#### Concurrent callers

The `int` payload with 1, 2, 4, or 8 callers sharing one serverlet; each sample is one caller's round trip.

| Kind | Rate | Samples | p50 | p90 | p99 | max |
|---|---|---:|---:|---:|---:|---:|
| in-process | callers-1 | 2000 | 11 | 15 | 24 | 128 |
| in-process | callers-2 | 2000 | 9 | 13 | 16 | 36 |
| in-process | callers-4 | 4000 | 9 | 15 | 23 | 42 |
| in-process | callers-8 | 8000 | 6 | 14 | 20 | 26 |
| sandbox | callers-1 | 2000 | 8 | 11 | 14 | 36 |
| sandbox | callers-2 | 2000 | 9 | 14 | 15 | 16 |
| sandbox | callers-4 | 4000 | 10 | 15 | 19 | 86 |
| sandbox | callers-8 | 8000 | 6 | 14 | 22 | 39 |
| secret | callers-1 | 2000 | 17 | 21 | 31 | 44 |
| secret | callers-2 | 2000 | 21 | 26 | 36 | 57 |
| secret | callers-4 | 4000 | 79 | 152 | 341 | 800 |
| secret | callers-8 | 8000 | 97 | 261 | 366 | 1483 |
| python | callers-1 | 2000 | 25 | 27 | 39 | 101 |
| python | callers-2 | 2000 | 31 | 36 | 47 | 63 |
| python | callers-4 | 4000 | 64 | 71 | 85 | 173 |
| python | callers-8 | 8000 | 130 | 143 | 177 | 279 |
| typescript | callers-1 | 2000 | 46 | 54 | 70 | 645 |
| typescript | callers-2 | 2000 | 62 | 78 | 100 | 580 |
| typescript | callers-4 | 4000 | 133 | 156 | 191 | 716 |
| typescript | callers-8 | 8000 | 264 | 288 | 350 | 1138 |
| in-process | callers-1 | 2000 | 14 | 20 | 31 | 136 |
| in-process | callers-2 | 2000 | 9 | 14 | 20 | 34 |
| in-process | callers-4 | 4000 | 10 | 15 | 21 | 31 |
| in-process | callers-8 | 8000 | 6 | 14 | 21 | 42 |
| sandbox | callers-1 | 2000 | 7 | 15 | 31 | 43 |
| sandbox | callers-2 | 2000 | 9 | 14 | 17 | 26 |
| sandbox | callers-4 | 4000 | 11 | 16 | 19 | 36 |
| sandbox | callers-8 | 8000 | 7 | 15 | 23 | 34 |
| secret | callers-1 | 2000 | 17 | 22 | 34 | 77 |
| secret | callers-2 | 2000 | 20 | 26 | 35 | 63 |
| secret | callers-4 | 4000 | 43 | 51 | 65 | 97 |
| secret | callers-8 | 8000 | 87 | 99 | 111 | 157 |
| python | callers-1 | 2000 | 24 | 32 | 49 | 186 |
| python | callers-2 | 2000 | 31 | 35 | 46 | 64 |
| python | callers-4 | 4000 | 64 | 73 | 83 | 247 |
| python | callers-8 | 8000 | 129 | 142 | 174 | 4003 |
| typescript | callers-1 | 2000 | 45 | 54 | 69 | 649 |
| typescript | callers-2 | 2000 | 70 | 149 | 2291 | 16812 |
| typescript | callers-4 | 4000 | 125 | 150 | 185 | 679 |
| typescript | callers-8 | 8000 | 249 | 281 | 377 | 1116 |

#### State rungs

Expand All @@ -153,7 +175,7 @@ Nanoseconds per statement, each sample a batch of 1,000 statements; `loop-only`
| Kind | Samples | p50 | p90 | p99 | max |
|---|---:|---:|---:|---:|---:|
| local-let | 2000 | 0 | 0 | 1 | 1 |
| shared-let | 2000 | 9 | 10 | 11 | 25 |
| shared-let | 2000 | 9 | 10 | 10 | 11 |
| loop-only | 2000 | 0 | 0 | 1 | 1 |

## tick_cost — library mode
Expand All @@ -171,18 +193,46 @@ a run, so medians overlap between cases within a few nanoseconds.
### Sample results


One run on Apple M2 (macOS-15.3.2-arm64-arm-64bit, arm64), 2026-09-21T03:17:52+00:00, commit `6d0686246e13` (dirty tree). rustc 1.98.1 (48a229cea 2026-09-01). Nanoseconds per `tick_sync` on a current-thread runtime: the median of 21 rounds of 200,000 ticks, after 5 warm-up rounds. Your numbers will differ.
One run on Apple M2 (macOS-15.3.2-arm64-arm-64bit, arm64), 2026-09-21T03:55:18+00:00, commit `afbc07fb4be4`. rustc 1.98.1 (48a229cea 2026-09-01). Nanoseconds per `tick_sync` on a current-thread runtime: the median of 21 rounds of 200,000 ticks, after 5 warm-up rounds. Your numbers will differ.

| Case | ns per tick (median) | fastest round | p90 round |
|---|---:|---:|---:|
| host_method_alone | 3.97 | 1.95 | 4.67 |
| empty | 7.87 | 6.90 | 9.13 |
| one_call | 7.47 | 7.11 | 7.68 |
| five_calls | 14.26 | 12.90 | 14.57 |
| instance_let | 6.41 | 6.24 | 7.56 |
| shared_let | 11.91 | 10.94 | 12.43 |
| event | 213.90 | 209.93 | 226.42 |
| host_method_alone | 1.97 | 1.96 | 1.99 |
| empty | 6.16 | 6.06 | 6.35 |
| one_call | 7.27 | 7.08 | 7.43 |
| five_calls | 14.28 | 12.59 | 14.46 |
| instance_let | 6.28 | 6.24 | 6.39 |
| shared_let | 11.68 | 10.86 | 11.93 |
| event | 211.34 | 196.99 | 226.94 |

Derived from the fastest rounds, which move least between runs: an empty tick costs 6.90 ns; each host call in a tick costs 1.45 ns (five calls minus one, over four), against 1.95 ns for the trait method called through its vtable alone. The empty tick and the one-call tick are within noise of each other at this resolution.
Derived from the fastest rounds, which move least between runs: an empty tick costs 6.06 ns; each host call in a tick costs 1.38 ns (five calls minus one, over four), against 1.96 ns for the trait method called through its vtable alone. The empty tick and the one-call tick are within noise of each other at this resolution.

Medians overlap between cases within a few nanoseconds: the OS moves the thread between core types and clock states during a run, which is why the fastest round is reported beside them. Run on an idle machine, and compare cases within one run.

## diagnostics_coverage — what `orchestrate check` rejects

Runs every program in `tests/error_cases/diagnostics/`, each deliberately invalid,
through `orchestrate check` and then `orchestrate build`, and classifies where it was
rejected: by `check`, by the build before Cargo runs, through Cargo without a rustc error
code, by rustc against generated code (a leak, held to `KNOWN_LEAKS.txt` by the test
`diagnostics_never_leak_rustc`), or not at all. The same classification is asserted by
`cargo test`; the script writes it as a table with the counts the paper states.

### Sample results


84 deliberately invalid programs in `tests/error_cases/diagnostics/`, run on 2026-09-21T03:55:25+00:00 at commit `afbc07fb4be4` with OrchestrateLang compiler v0.14.0.

| Rejected by | Programs | Meaning |
|---|---:|---|
| `orchestrate check` | 76 | the typechecker's own error, before any code is generated |
| build, before Cargo | 6 | codegen or the driver, still the compiler's own error |
| Cargo, no rustc code | 0 | a `compile_error!` the generator planted, reported through Cargo |
| rustc | 2 | leaked: a rustc error against generated code |
| nothing | 0 | wrongly accepted |

**76 of 84 invalid programs are rejected by `orchestrate check`.**
82 of 84 are rejected as the compiler's own error before Cargo runs; 0 more are rejected through Cargo without a rustc error code; 2 leak a rustc error; 0 are wrongly accepted.

The per-case rows are in `benchmarks/results/diagnostics_coverage.md` after a run.

69 changes: 69 additions & 0 deletions benchmarks/run_all.py
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,69 @@
#!/usr/bin/env python3
"""Regenerate every table the paper uses, in one command.

Usage: python3 benchmarks/run_all.py [--out DIR] [--no-typescript] [--skip NAME ...]
Set ORCHESTRATE to the compiler binary; the default builds and uses target/release/orchestrate.

Runs, in order:
benchmarks/landline_latency/run.py the boundary ladder (payload, call-rate, callers, state)
benchmarks/tick_cost/run.py the library-mode tick cost
benchmarks/diagnostics_coverage.py how many invalid programs `orchestrate check` rejects

Each writes CSV, JSON, and Markdown under --out (default benchmarks/results/), with an
environment header. This script then writes results.md there, the three Markdown
reports in one file, and prints where everything went. Run it on an idle machine: the
first two are timing benchmarks.
"""
import argparse
import os
import pathlib
import subprocess
import sys

REPO = pathlib.Path(__file__).resolve().parent.parent
STEPS = [
("boundary_ladder", "benchmarks/landline_latency/run.py"),
("tick_cost", "benchmarks/tick_cost/run.py"),
("diagnostics_coverage", "benchmarks/diagnostics_coverage.py"),
]


def main():
parser = argparse.ArgumentParser(description=__doc__, formatter_class=argparse.RawDescriptionHelpFormatter)
parser.add_argument("--out", default=str(REPO / "benchmarks" / "results"))
parser.add_argument("--no-typescript", action="store_true", help="skip the TypeScript rung of the ladder")
parser.add_argument("--skip", nargs="*", default=[], choices=[name for name, _ in STEPS], help="steps to leave out")
args = parser.parse_args()

orchestrate = os.environ.get("ORCHESTRATE")
if not orchestrate:
built = subprocess.run(["cargo", "build", "--release", "--quiet"], cwd=REPO)
if built.returncode != 0:
sys.exit("cargo build --release failed")
orchestrate = str(REPO / "target" / "release" / "orchestrate")
env = dict(os.environ, ORCHESTRATE=orchestrate)
out = pathlib.Path(args.out)
out.mkdir(parents=True, exist_ok=True)

for name, script in STEPS:
if name in args.skip:
print(f"skipping {name}")
continue
command = [sys.executable, str(REPO / script), "--out", str(out)]
if name == "boundary_ladder" and args.no_typescript:
command.append("--no-typescript")
print(f"== {name}: {' '.join(command[1:])}", flush=True)
result = subprocess.run(command, cwd=REPO, env=env)
if result.returncode != 0:
sys.exit(f"{name} failed")

reports = [out / f"{name}.md" for name, _ in STEPS if (out / f"{name}.md").exists()]
combined = "\n\n".join(p.read_text().rstrip() for p in reports) + "\n"
(out / "results.md").write_text(combined)
print(f"\nall tables regenerated under {out}:")
for p in sorted(out.iterdir()):
print(f" {p.name}")


if __name__ == "__main__":
main()
27 changes: 27 additions & 0 deletions docs/language-reference.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -824,6 +824,33 @@ orchestrate build --lib <file.orch> -o <dir> [--target <triple>] [--rust-version
| `build --lib … --dependency '<name> = <spec>'` | Adds a Cargo dependency to the generated crate; repeatable. A relative `path` is resolved against the working directory |
| `build --lib … --dependencies <file.toml>` | Adds every dependency a TOML fragment declares (a `[dependencies]` header is optional); paths resolve the same way |

### What `check` guarantees

`orchestrate check` parses the entry file, resolves every `use module` directory and
`load` sub-file, reads the sidecars of `load_foreign` declarations so that calls into
foreign functions are typed — including reading a `.wasm` module's own export table and
refusing a sidecar that names an export the module lacks or types it differently — and
runs the typechecker over the entry file. A program that passes is free of the errors the
typechecker knows how to find: unknown names, mismatched types and arities in calls,
handler calls a serverlet does not declare, `trigger` arguments that do not match the
event, struct literals with missing or unknown fields, non-exhaustive matches, a `fn` that
waits, a `task` that names top-level state, a handler type that cannot cross a secret,
landline, or sandboxed boundary, and a sandboxed handler that calls a host function it
was not granted. A program that fails is reported in OrchestrateLang terms, with a line
and a column, before any code is generated.

It guarantees nothing beyond that. `check` generates no Rust and runs no Cargo, so the
checks that live in code generation are not made: it accepts a `host` block or an
`on_tick` outside a library build, it does not walk the handler bodies of a serverlet
declared inside an imported module, and it cannot see a Rust foreign function's body or
a landline's Python or TypeScript source, which `check-foreign` checks with each
language's own tools. A program `check` accepts may therefore still be refused by
`build`, by the compiler's own error or, in the cases listed in
`benchmarks/results/diagnostics_coverage.md` and `LIMITATIONS-notes.md`, by rustc against
generated code. The diagnostics corpus under `tests/error_cases/diagnostics/` measures
how much of the invalid-program space `check` covers, and `benchmarks/diagnostics_coverage.py`
prints the count.

### Debugging Generated Code

When a program fails to compile due to a Rust-level error, the compiler prints a translated, user-friendly error message. For more detail, set the `ORCH_SHOW_GENERATED` environment variable to `1` to dump the full cargo stderr and the generated Rust source:
Expand Down
Loading