Skip to content

distiller and promotion ladder: from a closed Claude run (transcript, PR diff, receipt) to a candidate protocol card, and tier promotion on consecutive green receipts #436

Description

@mecattaf

Why

Tom's words: "tool-and-protocol-distill from the exsititign claude runs". The corpus exists: 104 human sessions labelled dotfiles on both benchmark arms (/home/tom/today/review/2026-09-19/labels-.tsv), transcripts under /home/tom/.claude/projects/, 127 merged PRs. MEASURED on transcript eea0aa1c (PR 389): the protocol is fetch main, read by sed and grep, worktree on a named branch, edit, verify by nix build and diff, commit, rebase, ff-merge, switch, remove the worktree. Design: /home/tom/today/review/2026-09-19/ISSUE-ZERO-DISTILLATION.md section 4.

Outcome

  • protocols/tools/distill.py: deterministic pre-filter and packaging. Input: a session id (resolved through /home/tom/today/drain/index-.tsv), a PR number or commit range, and an optional receipt path. Rejects a run unless the diff touches at most three files in one module and the transcript contains at least one verification argv (nix build, nix eval, nix flake check, shellcheck, tests/). Emits the tool sequence, the diff, the PR body, and the verification argv list as one JSON packet.
  • A tally job (Opus at first; the claude-code adapter) that turns the packet into a candidate card and writes it under /home/tom/today/review//cards/.candidate.json (review gate). A candidate becomes a card only through a PR Tom merges into protocols/.
  • Validation: one replay by a tier under tally on a fresh worktree against a fixture issue, evaluator green on every verify entry, normalized diff equal to the source diff. The receipt id goes into tier.validated.
  • Promotion ladder as data in the card's history: opus to sonnet after one green replay; sonnet to halogen after three consecutive green Sonnet replays with zero card edits plus one Halogen dry run whose diff equals Sonnet's; halogen to atomic after three consecutive green Halogen runs on real issues under a 30k prompt; any red demotes one rung.
  • protocols/tools/promote.py: reads the receipts directory and, when the update-adopt receipts carry merged_prs, marks fleet-green on a card whose PR reached known-good on the worker.
  • When the v3 harvest output lands: a one-page moves-to-classes.tsv mapping each repeated move (three or more sessions) to a card class, and each LAW heuristic scoped to dotfiles to a card guard.

Acceptance

  • distill.py on session eea0aa1c-0d5a-4bdb-8ebb-4e2316ca6e1b and PR 389 produces a packet whose verification list contains the nix build and diff -r calls the transcript shows.
  • One candidate card from that packet is reviewed against the hand-written pin-bump and file-add cards and the differences are recorded in the candidate's history.
  • promote.py run on an empty receipts directory changes nothing and says so.

Refs #354, tally-ts-sdk #77, tally-ts-sdk #73, #403.

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    needs-tomtriage bucket 2026-09-20

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions