Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
5 changes: 5 additions & 0 deletions .gitignore
Original file line number Diff line number Diff line change
Expand Up @@ -24,3 +24,8 @@ templates/library-*.style.json
templates/landed.index.json
.decklet-host.json
.decklet-history/

# authoring benchmark: what score writes beside each candidate
benchmarks/authoring/**/deck.html
benchmarks/authoring/**/deck.finalize.json
benchmarks/authoring/**/verify-out/
1 change: 1 addition & 0 deletions CHANGELOG.md
Original file line number Diff line number Diff line change
@@ -1,6 +1,7 @@
# Changelog
## Unreleased

- **Agent authoring benchmark (#208).** `benchmarks/authoring/` scores whether an agent following SKILL.md reaches `FINALIZE PASS` on attempt 1 with the deck the prompt asked for. Five frozen prompts (a pitch, a QBR, a one-page document, a 4:5 carousel and a twelve-vendor landscape with its logo files in `inputs/`) each carry semantic keys: format, slide range, required entities and minimum chart, logo, donut or image counts. `benchmark.mjs check` validates the suite; `score <run-dir>` runs `finalize` on each frozen candidate and writes one row per case: pass, the failed stage, the semantic misses, the top three diagnostic codes, repairs and minutes from the runner's `run.json`. It never launches a model. Five hand-built references prove the wiring and are marked `evidence: false`. README.md gives the fair-run protocol: same commit and prompts, an installed decklet outside this repo, attempt 1 frozen. Gate: test/benchmark.test.mjs (the manifest, a broken manifest, semantic misses, stubbed and live scoring of every reference, a missing and a broken candidate, the CLI).
- **Update notice in the finalize receipt (#206).** `node bin/finalize.mjs` adds `update: {installed, available, noticeRequired, source}` to its receipt and prints one line when a newer decklet is on main. `lib/release.mjs` reads `version` from `package.json` on main with curl under a 4-second limit (git tags lag the releases), caches the answer for 24 hours under `$XDG_CACHE_HOME/decklet`, reports `unavailable` instead of failing when offline, and never installs anything; `DECKLET_NO_UPDATE_CHECK=1` turns it off. The API asks only when called with `update`, so tests and embedders make no request. The check never changes `ok`. SKILL.md Step 6 tells the agent to mention it once. Gate: test/release.test.mjs (newer, same and older versions, a failed or garbled fetch, the 24-hour cache, the off switch, and the receipt).
- **Coded verify diagnostics (#201, verify half).** `verify` now returns `diagnostics` in the shape `validate` has: the contract's codes first, then one per rendered problem with its `engine`, a `subject` of `{slide, el, id}` (or `{slide, master, id}`), evidence and fixes. Eleven `parity/` codes cover overflow (`{need, have}`, `w: <need>`), nowrap lines, line count, ink through text, a straddled container, text over text, an arrow in a fill, the canvas edge, occlusion and the counter's margin and drift, plus `parity/page-error`, `parity/crowding` (a warning), `ae/over-threshold` and `deck/network`. The codes are read from the problem strings the browser pass already writes, so the in-page measuring code is unchanged. `verify --json` prints them, `results.json` carries them (and is now written after the aggregate errors, so it holds those too), and the `finalize` receipt carries them on the verify stage. Gate: test/verify-codes.test.mjs (a static check that every problem string verify can push has a code, each code's subject and evidence, crowding, page errors, AE and the network, the receipt, and a live overflow read back through the API, `results.json` and `--json`).
- **Two windows no longer lose an edit in the store.** When window B was typing as window A saved, B saved its words on top of A's model, but A's older save could reach localStorage after B's. Both windows looked right and the store held A's model without B's words, so a reload lost them. A CI probe caught this on Firefox in 12 of 180 runs under load and 0 of 60 alone. A window now watches the `storage` event: when the store holds a model it already built on, it writes its newer model back, and it writes the log back when entries are missing. This was the Firefox two-window failure on `main`.
Expand Down
1 change: 1 addition & 0 deletions CONTRIBUTING.md
Original file line number Diff line number Diff line change
Expand Up @@ -38,6 +38,7 @@ Node 22 or newer. Nothing else to install for the engine or the CLI.
| HTML pages to model | `bin/import-html.mjs` |
| Worked examples | `examples/<name>/brief.md` + `model.json` |
| Layout library | `templates/` (then `npm run build:library`) |
| A SKILL.md change, and its measured effect | `SKILL.md`, then a run of `benchmarks/authoring/` (its README has the protocol) |

## License

Expand Down
37 changes: 37 additions & 0 deletions benchmarks/authoring/README.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,37 @@
# Authoring benchmark

Does an agent that follows `SKILL.md` reach `FINALIZE PASS` on attempt 1, with the deck the prompt asked for? Rerun it on every SKILL.md change and compare the numbers.

## What a case checks

`manifest.json` holds five cases: a pitch, a quarterly business review, a one-page document, a 4:5 carousel and a twelve-vendor competitive landscape. Each case has a frozen prompt in `prompts/` and semantic keys the scorer checks on the candidate's `model.json`:

| key | the candidate must |
|---|---|
| `format` | name this format |
| `density` | (in the prompt) build at this density |
| `slides` | have between `[min, max]` slides |
| `entities` | list every one of these companies in `deck.entities` |
| `counts` | carry at least this many `chart`, `logo`, `donut` or `img` rows |

A row passes only when `finalize` passes (validate `--strict`, create, verify `--strict`) and no semantic key misses. A deck that renders but drops a required vendor fails, and so does a faithful deck that `finalize` rejects.

## Fair-run protocol

1. Run every configuration on the same commit, prompt set, time limit and tool access. Record the agent and model names in `run.json`.
2. Run the agent from an installed copy of decklet (plugin or `npx skills add`), never from this repository, so it sees what users see and never sees these cases or the references.
3. Give it the prompt and, for a case with `inputs`, a copy of that folder in its working directory.
4. One complete agent invocation is attempt 1. Freeze its `model.json` (and `style.json`, if any) into `<run-dir>/<case>/` when it ends. Later attempts never replace it.
5. Write `run.json`: `{agent, model, cases: {<id>: {minutes, repairs}}}`, with repairs counted from the transcript (runs of finalize or verify after the first).
6. Score it and keep the result:

```bash
node benchmarks/authoring/benchmark.mjs check
node benchmarks/authoring/benchmark.mjs score <run-dir> --out benchmarks/authoring/results/<date>-<agent>-<model>.json
```

The script never launches a model; the runner owns models, credentials, timing and transcripts.

## References

`reference/` holds one hand-built candidate per case. They prove the scorer and the cases are wired (test/benchmark.test.mjs scores them), and `run.json` marks them `"evidence": false`. They are never benchmark results.
95 changes: 95 additions & 0 deletions benchmarks/authoring/benchmark.mjs
Original file line number Diff line number Diff line change
@@ -0,0 +1,95 @@
#!/usr/bin/env node
// decklet authoring benchmark — does an agent following SKILL.md reach FINALIZE PASS on attempt 1, with the deck the prompt asked for?
// node benchmarks/authoring/benchmark.mjs check [--manifest manifest.json]
// node benchmarks/authoring/benchmark.mjs score <run-dir> [--out result.json] [--no-webkit] [--no-verify]
// A run dir holds <case>/model.json (and an optional style.json) per case plus run.json {agent, model, cases: {id: {minutes, repairs}}}.
// This script never launches a model: the runner owns models, prompts, timing and transcripts. README.md has the protocol.
import fs from 'node:fs';
import path from 'node:path';
import {fileURLToPath} from 'node:url';
import {finalize} from '../../bin/finalize.mjs';
import {FORMATS} from '../../bin/validate.mjs';
import {DENSITY} from '../../lib/layouts.mjs';
import {isMain} from '../../lib/is-main.mjs';

const here = path.dirname(fileURLToPath(import.meta.url));
const COUNTS = ['chart', 'logo', 'donut', 'img'];
const read = f => JSON.parse(fs.readFileSync(f, 'utf8'));

export function check(manifest, dir = here) {
const problems = [], ids = new Set();
if (!Array.isArray(manifest.cases) || !manifest.cases.length) return {problems: ['cases must be a non-empty array']};
for (const c of manifest.cases) {
const id = c.id || '?', P = m => problems.push(`${id}: ${m}`);
if (!/^[a-z0-9-]+$/.test(c.id || '')) P('id must be a slug');
if (ids.has(c.id)) P('duplicate id'); ids.add(c.id);
if (!c.prompt || !fs.existsSync(path.join(dir, c.prompt)) || !fs.readFileSync(path.join(dir, c.prompt), 'utf8').trim()) P(`prompt ${c.prompt} not found`);
if (!FORMATS.includes(c.format)) P(`format "${c.format}" is not one of ${FORMATS.join('|')}`);
if (!DENSITY[c.density]) P(`density "${c.density}" is not one of ${Object.keys(DENSITY).join('|')}`);
if (!(Array.isArray(c.slides) && c.slides.length === 2 && c.slides.every(Number.isInteger) && c.slides[0] >= 1 && c.slides[0] <= c.slides[1])) P('slides must be [min, max]');
if (!Array.isArray(c.entities)) P('entities must be an array');
if (!c.counts || typeof c.counts !== 'object' || Array.isArray(c.counts)) P('counts must be an object');
else for (const k of Object.keys(c.counts)) if (!COUNTS.includes(k)) P(`counts.${k} is not one of ${COUNTS.join('|')}`);
if (c.inputs && !fs.existsSync(path.join(dir, c.inputs))) P(`inputs ${c.inputs} not found`);
}
return {problems};
}

const rowsOf = m => [...(m.slides || []).flatMap(s => s?.els || []), ...(m.master || [])];
const fillMedia = m => (m.slides || []).flatMap(s => Object.values(s?.fill || {})).filter(v => v && typeof v === 'object' && !Array.isArray(v));
const count = (m, kind) => {
const rows = rowsOf(m);
if (kind === 'chart') return rows.filter(r => r?.chart).length + (m.slides || []).filter(s => /^chart/.test(s?.template || '')).length;
if (kind === 'logo') return rows.filter(r => r?.logo != null).length + fillMedia(m).filter(v => v.logo != null).length;
if (kind === 'img') return rows.filter(r => r?.img).length + fillMedia(m).filter(v => v.img).length;
return rows.filter(r => r?.[kind] != null).length;
};

export function semanticMisses(c, m) {
const misses = [], n = (m.slides || []).length, fmt = m.format || 'slides';
if (fmt !== c.format) misses.push(`format ${fmt}, wanted ${c.format}`);
if (n < c.slides[0] || n > c.slides[1]) misses.push(`slides ${n}, wanted ${c.slides[0]}-${c.slides[1]}`);
const listed = new Set((m.entities || []).flat().map(e => String(e).toLowerCase()));
for (const e of c.entities) if (!listed.has(e.toLowerCase())) misses.push(`entities missing ${e}`);
for (const [k, want] of Object.entries(c.counts)) { const got = count(m, k); if (got < want) misses.push(`${k} ${got}, wanted at least ${want}`); }
return misses;
}

const topCodes = receipt => {
const n = new Map();
for (const s of receipt.stages) for (const d of s.diagnostics || []) n.set(d.code, (n.get(d.code) || 0) + 1);
return [...n].sort((a, b) => b[1] - a[1] || a[0].localeCompare(b[0])).slice(0, 3).map(([c]) => c);
};

export async function score(runDir, {manifestPath = path.join(here, 'manifest.json'), verifyFn = null, webkit = 'auto'} = {}) {
const manifest = read(manifestPath), runFile = path.join(runDir, 'run.json'), run = fs.existsSync(runFile) ? read(runFile) : {};
const rows = [];
for (const c of manifest.cases) {
const dir = path.join(runDir, c.id), modelFile = path.join(dir, 'model.json'), styleFile = path.join(dir, 'style.json');
const meta = {repairs: run.cases?.[c.id]?.repairs ?? null, minutes: run.cases?.[c.id]?.minutes ?? null};
if (!fs.existsSync(modelFile)) { rows.push({case: c.id, status: 'missing', pass: false, finalize: false, misses: ['no model.json'], topCodes: [], ...meta}); continue; }
const receipt = await finalize(modelFile, {style: fs.existsSync(styleFile) ? styleFile : null, out: path.join(dir, 'deck.html'), webkit, ...(verifyFn ? {verifyFn} : {}), log: () => {}});
const misses = semanticMisses(c, read(modelFile)), failed = receipt.stages.find(s => s.status !== 'pass');
rows.push({case: c.id, status: receipt.ok ? 'finalized' : 'failed', pass: receipt.ok && !misses.length, finalize: receipt.ok, ...(failed ? {failedStage: failed.stage} : {}), misses, topCodes: topCodes(receipt), ...meta});
}
return {benchmark: manifest.benchmark, version: manifest.version, agent: run.agent ?? null, model: run.model ?? null, evidence: run.evidence ?? run.agent !== 'reference',
engine: JSON.parse(fs.readFileSync(path.join(here, '../../package.json'), 'utf8')).version, scoredAt: new Date().toISOString(),
rows, summary: {passed: rows.filter(r => r.pass).length, total: rows.length}};
}

if (isMain(import.meta.url)) {
const [cmd, ...a] = process.argv.slice(2), flag = k => a.includes(k), opt = k => a.includes(k) ? a[a.indexOf(k) + 1] : null;
if (cmd === 'check') {
const mp = opt('--manifest') || path.join(here, 'manifest.json'), {problems} = check(read(mp), path.dirname(path.resolve(mp)));
for (const p of problems) console.error('ERROR ' + p);
console.log(problems.length ? `manifest has ${problems.length} problem(s)` : `manifest ok: ${read(mp).cases.length} cases`);
process.exit(problems.length ? 1 : 0);
}
const runDir = a.find(x => !x.startsWith('--') && x !== opt('--out'));
if (cmd !== 'score' || !runDir) { console.error('usage: node benchmarks/authoring/benchmark.mjs check [--manifest file] | score <run-dir> [--out file] [--no-webkit] [--no-verify]'); process.exit(2); }
const noVerify = async () => ({ok: true, errors: [], warnings: [], skipped: ['layout parity + AE: --no-verify'], parity: [], diagnostics: []});
const r = await score(runDir, {webkit: flag('--no-webkit') ? false : 'auto', ...(flag('--no-verify') ? {verifyFn: noVerify} : {})});
for (const x of r.rows) console.log(`${x.pass ? 'pass' : 'FAIL'} ${x.case.padEnd(10)} ${x.status.padEnd(9)} ${x.misses.join('; ') || '-'} ${x.topCodes.join(', ')}`);
console.log(`${r.summary.passed}/${r.summary.total} passed${r.evidence ? '' : ' (not evidence: reference candidates)'}`);
if (opt('--out')) fs.writeFileSync(opt('--out'), JSON.stringify(r, null, 1) + '\n');
}
1 change: 1 addition & 0 deletions benchmarks/authoring/inputs/landscape/bluefin.svg
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
1 change: 1 addition & 0 deletions benchmarks/authoring/inputs/landscape/corvid.svg
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
1 change: 1 addition & 0 deletions benchmarks/authoring/inputs/landscape/delta-loom.svg
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
1 change: 1 addition & 0 deletions benchmarks/authoring/inputs/landscape/emberly.svg
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
1 change: 1 addition & 0 deletions benchmarks/authoring/inputs/landscape/fathom.svg
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
1 change: 1 addition & 0 deletions benchmarks/authoring/inputs/landscape/gridline.svg
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
1 change: 1 addition & 0 deletions benchmarks/authoring/inputs/landscape/halcyon.svg
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
1 change: 1 addition & 0 deletions benchmarks/authoring/inputs/landscape/ironleaf.svg
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
1 change: 1 addition & 0 deletions benchmarks/authoring/inputs/landscape/juniper.svg
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
1 change: 1 addition & 0 deletions benchmarks/authoring/inputs/landscape/kestrel.svg
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
1 change: 1 addition & 0 deletions benchmarks/authoring/inputs/landscape/lumen.svg
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
1 change: 1 addition & 0 deletions benchmarks/authoring/inputs/landscape/northwind.svg
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
13 changes: 13 additions & 0 deletions benchmarks/authoring/manifest.json
Original file line number Diff line number Diff line change
@@ -0,0 +1,13 @@
{
"benchmark": "decklet-authoring",
"version": 1,
"cases": [
{"id": "pitch", "prompt": "prompts/pitch.md", "format": "slides", "density": "speaker", "slides": [8, 12], "entities": [], "counts": {"chart": 1}},
{"id": "qbr", "prompt": "prompts/qbr.md", "format": "slides", "density": "reading", "slides": [6, 10], "entities": [], "counts": {"chart": 1}},
{"id": "one-pager", "prompt": "prompts/one-pager.md", "format": "document-letter", "density": "reading", "slides": [1, 1], "entities": [], "counts": {"chart": 1}},
{"id": "carousel", "prompt": "prompts/carousel.md", "format": "carousel-4x5", "density": "speaker", "slides": [5, 8], "entities": [], "counts": {}},
{"id": "landscape", "prompt": "prompts/landscape.md", "format": "slides", "density": "speaker", "slides": [2, 4],
"entities": ["Northwind", "Bluefin", "Corvid", "Delta Loom", "Emberly", "Fathom", "Gridline", "Halcyon", "Ironleaf", "Juniper", "Kestrel", "Lumen"],
"counts": {"logo": 12}, "inputs": "inputs/landscape"}
]
}
9 changes: 9 additions & 0 deletions benchmarks/authoring/prompts/carousel.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,9 @@
Use the decklet skill to make a LinkedIn carousel of 5 to 8 cards on the `carousel-4x5` format at speaker density, saved as `deck.html` next to `model.json`.

Topic: three habits of fast engineering teams.
- Small changes: teams that merge under 200 lines ship twice as often.
- Fast review: a first review inside four hours halves cycle time.
- Green main: a main branch that stays green saves about a day a week.
- Close with a card asking the reader to pick one habit and measure it for a month.

Give every card one graphic. Finish with `finalize` passing.
5 changes: 5 additions & 0 deletions benchmarks/authoring/prompts/landscape.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,5 @@
Use the decklet skill to make a competitive landscape deck of 2 to 4 slides on the 16:9 `slides` format at speaker density, saved as `deck.html` next to `model.json`.

The market is workflow tools for operations teams. The twelve vendors are Northwind, Bluefin, Corvid, Delta Loom, Emberly, Fathom, Gridline, Halcyon, Ironleaf, Juniper, Kestrel and Lumen. Their logos are the SVG files in `inputs/landscape/`, one per vendor, named after it. Every vendor appears with its logo. List all twelve in `deck.entities`.

Group them however the deck reads best. Finish with `finalize` passing.
9 changes: 9 additions & 0 deletions benchmarks/authoring/prompts/one-pager.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,9 @@
Use the decklet skill to make a one-page document on the `document-letter` format at reading density, saved as `deck.html` next to `model.json`.

Subject: results of the onboarding checklist pilot.
- 40 engineers hired in August and September used a guided checklist.
- Median time to first shipped change fell from 9 working days to 4.
- Next step: extend the checklist to the support team in November and measure the same interval.
- Source: onboarding tracker.

Include the before/after numbers as a chart and cite the source. Finish with `finalize` passing.
9 changes: 9 additions & 0 deletions benchmarks/authoring/prompts/pitch.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,9 @@
Use the decklet skill to make a presented investor pitch deck, 8 to 12 slides on the 16:9 `slides` format at speaker density, saved as `deck.html` next to `model.json`.

Company: Tidewater, a scheduling app for loading docks at regional warehouses.
- Problem: trucks wait an average of 2.4 hours for a dock door; detention fees cost a mid-size warehouse about $310,000 a year.
- Product: drivers book a door slot from their phone; the warehouse sees every arrival on one board.
- Traction: 38 warehouses live, monthly recurring revenue up from $41k in January to $186k in September.
- Ask: $6M seed to reach 250 warehouses in 18 months.

Show the revenue growth as a chart. Finish with `finalize` passing.
Loading
Loading