From bee62e447d8ccea78db92be9a0948744bebefd4e Mon Sep 17 00:00:00 2001 From: onatozmenn Date: Wed, 12 Aug 2026 17:31:28 +0300 Subject: [PATCH] Record the contracts-free comparison --- ROADMAP.md | 8 ++- benchmarks/README.md | 6 ++ benchmarks/RESULTS.md | 10 ++-- benchmarks/STARLARK.md | 122 +++++++++++++++++++++++++++++++++++++++++ 4 files changed, 139 insertions(+), 7 deletions(-) create mode 100644 benchmarks/STARLARK.md diff --git a/ROADMAP.md b/ROADMAP.md index 481605f..dfd1433 100644 --- a/ROADMAP.md +++ b/ROADMAP.md @@ -314,9 +314,11 @@ from zero. Everything after it is multiplication; this one is addition. - [ ] Ship a second model family in the benchmark. Today only `OPENAI_API_KEY` is configured, so "does this hold across model families" is unmeasured, and the result is quotable only with that caveat attached. -- [ ] The comparison arm: the same six tasks in a language with no contracts. - `benchmarks/README.md` already names this as the thing that would - falsify the pitch. Running it is worth more than any announcement. +- [x] The comparison arm: the same six behaviours in contracts-free Starlark. + One no-tool run passed 6/6 where the five Deed MCP runs passed 5/6. This + does not establish an effect size, but it does falsify the claim that the + current six tasks show contracts helping. The exact answers and the next + benchmark requirement are in `benchmarks/STARLARK.md`. ### Stage 2 — Make the second repository not need us diff --git a/benchmarks/README.md b/benchmarks/README.md index 1e805f7..367ac17 100644 --- a/benchmarks/README.md +++ b/benchmarks/README.md @@ -105,6 +105,12 @@ with no contracts, then the contracts are decoration and the pitch is wrong. That is a real possible outcome and the reason to run this before believing the pitch, rather than after. +The first comparison is now recorded in [STARLARK.md](STARLARK.md). One +contracts-free Starlark run passed 6/6 where the five Deed MCP runs passed 5/6. +That does not prove contracts hurt, but it means these six tasks do not support +the stronger claim that contracts helped. The next benchmark has to make a +contract's value observable rather than translating routine functions. + ## What it has said so far [RESULTS.md](RESULTS.md) is the record: one model, one build, six tasks, five diff --git a/benchmarks/RESULTS.md b/benchmarks/RESULTS.md index d4fe671..c2ba8f4 100644 --- a/benchmarks/RESULTS.md +++ b/benchmarks/RESULTS.md @@ -136,10 +136,12 @@ one answer and one confirmation. - **One model family.** Everything above is `gpt-5.6`. Whether any of it holds elsewhere is unmeasured, and every number here should be read with that attached. -- **No comparison language.** The six tasks have not been run in a language - with no contracts. `README.md` names that as the thing that would falsify the - pitch, and until it is run, "contracts helped" is not something this file - establishes. +- **The comparison language won this first run.** The same behaviours were run + once in contracts-free Starlark with no tools, and all 6/6 passed. See + [STARLARK.md](STARLARK.md) for the generated answers and checks. One run is + not an effect size, and the translated prompts are simpler because they do + not teach effects, refinements or `Result`; it is enough to say this record + does not establish that contracts helped on these tasks. - **Six tasks is a floor.** They cover the language rather than a domain. An answer that checks is not an answer that is good. - **The control is from a different build.** See the warning above. diff --git a/benchmarks/STARLARK.md b/benchmarks/STARLARK.md new file mode 100644 index 0000000..0f586bf --- /dev/null +++ b/benchmarks/STARLARK.md @@ -0,0 +1,122 @@ +# The contracts-free comparison + +The six benchmark behaviours were translated to Starlark and handed to the +same `gpt-5.6-luna` model once on 2026-08-12. Starlark was chosen because it is +a Python-like language the model already knows, but its evaluator has no +imports, filesystem, network, process API or ambient host authority. Model code +was run by `go.starlark.net`, not by Python on the benchmark machine. + +This is the comparison arm [`README.md`](README.md) asked for: the same visible +behaviour and hidden edge cases in a language with no contracts, refinement +types, effect rows or generated properties. + +## Result + +| Arm | Runs | Answered | Passed hidden behaviour checks | +| --- | ---: | ---: | ---: | +| Deed with `deed mcp` | 5 | 30/30 | 25/30 | +| Starlark, no tools | 1 | 6/6 | **6/6** | + +The Starlark run took 17.0 seconds across all six calls. Every answer was a +single file and all six passed on the first attempt. + +This does **not** establish that contracts hurt. It does establish that this +benchmark does not yet show that contracts help: its behaviours are routine in +a familiar general-purpose language, while three Deed tasks spend part of their +prompt budget teaching effects, refinements and `Result`. The comparison is +also one run against five, so it is a direction for the next benchmark rather +than an effect-size estimate. + +## The generated answers + +```python +# twice.star +def twice(n): + return n + n +``` + +```python +# total.star +def total(numbers): + result = 0 + for number in numbers: + result += number + return result + +def largest(numbers, fallback): + if len(numbers) == 0: + return fallback + result = numbers[0] + for number in numbers[1:]: + if number > result: + result = number + return result +``` + +```python +# grade.star +Low = "low" +Middling = "middling" +High = "high" + +def grade(score): + if score < 40: + return Low + if score >= 80: + return High + return Middling + +def describe(mark): + if mark == Low: + return "low" + if mark == Middling: + return "middling" + return "high" +``` + +```python +# split_evenly.star +def split_evenly(amount, people): + if people <= 0: + return (False, "there is nobody to share with") + if amount < 0: + return (True, -((-amount) // people)) + return (True, amount // people) +``` + +```python +# audit.star +def collected(entries): + return ", ".join(entries) +``` + +```python +# stock.star +def take_one(count): + return count - 1 + +def restock(count, delivered): + return count + delivered +``` + +## What was checked + +The scorer exercised the same edge cases as the Deed task modules: negative +inputs to `twice`; empty and all-negative lists for `largest`; every grade +boundary; positive, zero and negative divisors; empty, singleton and multiple +audit entries; and restocking followed by taking one. + +The scorer was checked in both directions before the model run. The six hand +written references scored 6/6. Replacing `largest`'s first element with the +fallback reproduced the Deed benchmark's surviving bug and scored 5/6, naming +`largest([-5, -2], 0) = 0, want -2`. + +## What follows + +Do not respond by adding easier Deed tasks or by weakening the comparison. A +second benchmark should ask for work where contracts provide observable value: +a change to one module that breaks an unseen caller, a generated counterexample, +or a proof obligation that distinguishes two implementations which pass the +same examples. Until then, the honest claim is that `deed mcp` makes an unknown +language writable, not that contracts beat a familiar language on these six +functions.