Skip to content

Token benchmark rerun on current shorthand (measure only) #134

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
howlcipher merged 1 commit into from
Oct 12, 2026
Merged

Token benchmark rerun on current shorthand (measure only) #134

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
50 changes: 50 additions & 0 deletions docs/journals/2026-10-12_token_benchmark_rerun.md
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,50 @@
# Token benchmark rerun on current shorthand (2026-10-12)

## Goal

Rerun `experiments/token-efficiency-v0` on main `fd1c6225` (includes the
#124–#130 density sugar). Re-author HowlFrame references in idiomatic current
syntax, leave Python/JS/Go untouched, and fix the #133 REPORT.md issues.

## What changed

- `author_solutions.py`: new `howl_src` generator. It covers all 60 HowlFrame files and uses root `(do …)`,
multi-binding `let`, `read_text`, `lines`, `when` and integer-bound semver
parts. It no longer copies `falsifiable-v0/reference/arm_a`. Those files are unchanged.
- `solutions_test.go`: the verbatim-copy check for six references is replaced
by a guard against the expanded `bytes_to_string` form.
- `results/*`: regenerated. Python/JS/Go source hashes and counts are unchanged.
- `REPORT.md`: before/after, per-task delta, category means, hot spots.
Changes from #133:
- Fixed the garbled dependencies sentence and rewrote the stale validation section.
- Documented two limitations: the JS/Go stripper does not handle regex or
template literals, and the native verifier does not isolate references from the host.

## Results

| | cl100k_base | o200k_base |
|---|---:|---:|
| HowlFrame median before → after | 160 → 124 | 159 → 123 |
| Python / JS / Go median | 119 / 122 / 250 | 118 / 129 / 250 |
| HF/Python median per-task ratio | 1.422 → 1.059 | 1.417 → 1.058 |
| HF/JS | 1.368 → 1.019 | 1.270 → 0.953 |
| HF/Go | 0.640 → 0.496 | 0.636 → 0.492 |

The conditional-task mean fell from 413.18 to 344.55. Python's is 196.00.
On `o200k_base`, HowlFrame now uses fewer tokens than JS on 38 of 60 tasks, every one of them non-conditional.

## Findings while authoring

- The governed profile rejects `(parse_json Checks (read_text …))` with
PROFILE_FORBIDDEN_CONSTRUCT, because the profile scan runs before desugar.
The JSON text therefore needs a separate `raw` binding.
- `list_get` requires a symbol operand, so the split intermediates must be bound.
- A multi-form `let` body still needs `(do …)`.

## Next (hot spots)

1. Strict x.y.z comparison and KV lookup cost about 189 tokens per conditional file.
2. The unit-conditional `failed` loop stands in for `all`/`every` and adds 27 tokens.
3. Envelope overhead: the governed `parse_json` binding, the `do` in `let`, and the `dict` pair syntax.

Measure only. There were no live LLM runs and no H1 decision; #90 stays Partial, and nothing changed for HFIR or `-compile-bc`.
394 changes: 226 additions & 168 deletions experiments/token-efficiency-v0/REPORT.md
View file
Open in desktop

Large diffs are not rendered by default.

46 changes: 33 additions & 13 deletions experiments/token-efficiency-v0/author_solutions.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -4,26 +4,46 @@

ROOT = Path(__file__).resolve().parent
CORPUS = ROOT.parent / 'falsifiable-v0'

# HowlFrame references use current shorthand (multi-binding let, read_text,
# lines, when, n-ary and/or). Never reads reference/arm_a or oracle contents.
def howl_src(batch, field, value, conditional, approval):
sel = f'(for c checks (when (= (map_get c "{field}") "{value}") (append selected c)))'
extra = '\n failed (list)' if conditional and field != 'status' else ''
head = ('(do\n (let (raw (read_text "evidence/checks.json"))\n (let (checks (parse_json Checks raw)\n'
+ (' lines (lines (read_file "evidence/version.kv"))\n' if conditional and not approval else ' version (read_text "evidence/version.kv")\n')
+ ' note (read_text "evidence/note.txt")\n'
f' selected (list){extra})\n (do\n {sel}')
rec = f'(dict ("batch" "{batch}") ("selected" selected))'
if approval:
return head + f'\n (write_file "out/approval_request.json" (encode_json (dict ("reason" "release review") ("summary" {rec}))))))))\n'
if not conditional:
return head + f'\n (write_file "out/release.json" (encode_json {rec}))))))\n'
if field == 'status':
ok = '(= (list_len selected) (list_len checks))'
else:
ok = '(= (list_len failed) 0)'
head += '\n (for c checks (when (!= (map_get c "status") "PASS") (append failed c)))'
return head + (
'\n (let (cp (str_split (list_get lines 0) "=")\n'
' pp (str_split (list_get lines 1) "=")\n'
' cur (str_split (list_get cp 1) ".")\n'
' pro (str_split (list_get pp 1) ".")\n'
' c0 (to_int (list_get cur 0)) c1 (to_int (list_get cur 1)) c2 (to_int (list_get cur 2))\n'
' p0 (to_int (list_get pro 0)) p1 (to_int (list_get pro 1)) p2 (to_int (list_get pro 2)))\n'
f' (if (and {ok} (or (> p0 c0) (and (= p0 c0) (or (> p1 c1) (and (= p1 c1) (> p2 c2))))))\n'
f' (write_file "out/release.json" (encode_json {rec}))\n'
' (write_file "out/approval_request.json" "{\\"reason\\":\\"blocked\\",\\"summary\\":{}}")))))))\n')


manifest = json.loads((CORPUS / 'tasks/manifest.json').read_text())
for task in manifest['tasks'] + manifest['adversarial']:
task_id = task['id']
batch = task_id[1:]
field, value = ('status', 'PASS') if task['shape']['loop'] else ('name', 'unit')
conditional = task['shape']['conditional']
approval = task['mutation'] == 'request_approval'
template = 't02' if conditional else 't03' if approval else 't01'
howl = (CORPUS / f'reference/arm_a/{template}.howl').read_text()
howl = howl.replace(f'"{template[1:]}"', f'"{batch}"')
if field == 'name':
howl = howl.replace('(= (map_get c "status") "PASS")', '(= (map_get c "name") "unit")')
if conditional:
howl = howl.replace('(let (selected (list))', '(let (selected (list))\n (let (failed (list))')
howl = howl.replace('(let (record', '(for c checks (if (!= (map_get c "status") "PASS") (append failed c)))\n (let (record')
howl = howl.replace('(= (list_len selected) (list_len checks))', '(= (list_len failed) 0)')
howl = howl.rstrip() + ')\n'
existing = CORPUS / f'reference/arm_a/{task_id}.howl'
if existing.exists():
howl = existing.read_text()
howl = howl_src(batch, field, value, conditional, approval)
(ROOT / f'solutions/howl/{task_id}.howl').write_text(howl)

python = f'''import json
Expand Down
144 changes: 34 additions & 110 deletions experiments/token-efficiency-v0/results/summary.json
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -3,8 +3,8 @@
"tokens": {
"languages": {
"howl": {
"median": 160.0,
"mean": 248.83333333333334
"median": 124.0,
"mean": 200.6
},
"python": {
"median": 119.0,
Expand All @@ -21,8 +21,8 @@
},
"howl_ratios": {
"python": {
"median_per_task_ratio": 1.4215686274509804,
"ratio_of_medians": 1.3445378151260505,
"median_per_task_ratio": 1.0588235294117647,
"ratio_of_medians": 1.0420168067226891,
"howl_more_tokens": [
"t01",
"t02",
Expand Down Expand Up @@ -87,8 +87,8 @@
]
},
"js": {
"median_per_task_ratio": 1.3679245283018868,
"ratio_of_medians": 1.3114754098360655,
"median_per_task_ratio": 1.0188679245283019,
"ratio_of_medians": 1.0163934426229508,
"howl_more_tokens": [
"t01",
"t02",
Expand Down Expand Up @@ -153,17 +153,17 @@
]
},
"go": {
"median_per_task_ratio": 0.64,
"ratio_of_medians": 0.64,
"median_per_task_ratio": 0.496,
"ratio_of_medians": 0.496,
"howl_more_tokens": []
}
}
},
"tokens_nocomments": {
"languages": {
"howl": {
"median": 160.0,
"mean": 248.83333333333334
"median": 124.0,
"mean": 200.6
},
"python": {
"median": 119.0,
Expand All @@ -180,8 +180,8 @@
},
"howl_ratios": {
"python": {
"median_per_task_ratio": 1.4215686274509804,
"ratio_of_medians": 1.3445378151260505,
"median_per_task_ratio": 1.0588235294117647,
"ratio_of_medians": 1.0420168067226891,
"howl_more_tokens": [
"t01",
"t02",
Expand Down Expand Up @@ -246,8 +246,8 @@
]
},
"js": {
"median_per_task_ratio": 1.3679245283018868,
"ratio_of_medians": 1.3114754098360655,
"median_per_task_ratio": 1.0188679245283019,
"ratio_of_medians": 1.0163934426229508,
"howl_more_tokens": [
"t01",
"t02",
Expand Down Expand Up @@ -312,8 +312,8 @@
]
},
"go": {
"median_per_task_ratio": 0.6557377049180327,
"ratio_of_medians": 0.6557377049180327,
"median_per_task_ratio": 0.5081967213114754,
"ratio_of_medians": 0.5081967213114754,
"howl_more_tokens": []
}
}
Expand All @@ -323,8 +323,8 @@
"tokens": {
"languages": {
"howl": {
"median": 159.0,
"mean": 248.73333333333332
"median": 123.0,
"mean": 200.5
},
"python": {
"median": 118.0,
Expand All @@ -341,8 +341,8 @@
},
"howl_ratios": {
"python": {
"median_per_task_ratio": 1.4174757281553398,
"ratio_of_medians": 1.347457627118644,
"median_per_task_ratio": 1.058252427184466,
"ratio_of_medians": 1.0423728813559323,
"howl_more_tokens": [
"t01",
"t02",
Expand Down Expand Up @@ -407,19 +407,10 @@
]
},
"js": {
"median_per_task_ratio": 1.2695652173913043,
"ratio_of_medians": 1.2325581395348837,
"median_per_task_ratio": 0.9534883720930233,
"ratio_of_medians": 0.9534883720930233,
"howl_more_tokens": [
"t01",
"t02",
"t03",
"t04",
"t05",
"t06",
"t07",
"t08",
"t09",
"t10",
"t11",
"t12",
"t13",
Expand All @@ -430,26 +421,7 @@
"t18",
"t19",
"t20",
"t21",
"t22",
"t23",
"t24",
"t25",
"t26",
"t27",
"t28",
"t29",
"t30",
"a01",
"a02",
"a03",
"a04",
"a05",
"a06",
"a07",
"a08",
"a09",
"a10",
"a11",
"a12",
"a13",
Expand All @@ -459,31 +431,21 @@
"a17",
"a18",
"a19",
"a20",
"a21",
"a22",
"a23",
"a24",
"a25",
"a26",
"a27",
"a28",
"a29",
"a30"
"a20"
]
},
"go": {
"median_per_task_ratio": 0.636,
"ratio_of_medians": 0.636,
"median_per_task_ratio": 0.492,
"ratio_of_medians": 0.492,
"howl_more_tokens": []
}
}
},
"tokens_nocomments": {
"languages": {
"howl": {
"median": 159.0,
"mean": 248.73333333333332
"median": 123.0,
"mean": 200.5
},
"python": {
"median": 118.0,
Expand All @@ -500,8 +462,8 @@
},
"howl_ratios": {
"python": {
"median_per_task_ratio": 1.4174757281553398,
"ratio_of_medians": 1.347457627118644,
"median_per_task_ratio": 1.058252427184466,
"ratio_of_medians": 1.0423728813559323,
"howl_more_tokens": [
"t01",
"t02",
Expand Down Expand Up @@ -566,19 +528,10 @@
]
},
"js": {
"median_per_task_ratio": 1.2695652173913043,
"ratio_of_medians": 1.2325581395348837,
"median_per_task_ratio": 0.9534883720930233,
"ratio_of_medians": 0.9534883720930233,
"howl_more_tokens": [
"t01",
"t02",
"t03",
"t04",
"t05",
"t06",
"t07",
"t08",
"t09",
"t10",
"t11",
"t12",
"t13",
Expand All @@ -589,26 +542,7 @@
"t18",
"t19",
"t20",
"t21",
"t22",
"t23",
"t24",
"t25",
"t26",
"t27",
"t28",
"t29",
"t30",
"a01",
"a02",
"a03",
"a04",
"a05",
"a06",
"a07",
"a08",
"a09",
"a10",
"a11",
"a12",
"a13",
Expand All @@ -618,22 +552,12 @@
"a17",
"a18",
"a19",
"a20",
"a21",
"a22",
"a23",
"a24",
"a25",
"a26",
"a27",
"a28",
"a29",
"a30"
"a20"
]
},
"go": {
"median_per_task_ratio": 0.6516393442622951,
"ratio_of_medians": 0.6516393442622951,
"median_per_task_ratio": 0.5040983606557377,
"ratio_of_medians": 0.5040983606557377,
"howl_more_tokens": []
}
}
Expand Down
Loading
Loading