skills: cs329a-self-improving-agents — hand your agent the course, get a checkable disposition - #1
Merged
Merged
Conversation
…t a checkable disposition An Agent Skill (Claude Code plugin marketplace at the repo root, source "./"; also copy-pasteable into any harness that reads SKILL.md + references/). The agent reads ten ideas distilled from Stanford CS329A "Self-Improving AI Agents" against the repo it is pointed at and writes a disposition record: one verdict per idea (APPLIES / DOES NOT APPLY / ALREADY IN PLACE / NOT DECIDABLE), a quoted file:line or command output per verdict, bars pre-committed before any number is read, "declined, because X" accepted as complete. Ships our summaries and the papers, never the lectures (Stanford's). Every one of the 28 paper URLs and 9 lecture links was opened and matched before writing papers.md; test/skill-references.test.mjs pins the static shape and, under CHECK_LINKS=1 (its own CI job, so an arXiv outage reds one legible line), that every link still resolves. references/worked-example.md is the run on our own system: the idea we were most excited about came back NOT DECIDABLE (0 of 39 labels joinable), the most valuable ship was a log file, and adversarial verifiers refuted three of four builder receipts. Misses included on purpose. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01DGSjiRBHQmuF1BNw1W7dTF
… verdicts split from actions, 5/9 split into halves A fresh agent with no memory of the system ran the skill as first written, against the same repo the worked example describes, under a record-only mandate. It came back with three findings the sub-agent run had missed (the per-lens history had written zero rows and is years from its own bar; the gate-history file is gitignored; the persona planner and evaluator are the same model) and eight defects in the skill. Every one of the eight is fixed here and recorded in references/worked-example.md: - worked-example.md was an answer key for the target repo and the references table said to read it early. Now step 7, after the table is filled, read for form only; warning at the top of the file; asserted by the test. - "Expect NOT DECIDABLE on the idea you find most exciting" anchored the reader. Removed from the procedure. - Summary buckets (took/declined) did not match the verdict vocabulary (APPLIES/DOES NOT APPLY), so a record-only run had to report Took: 0 against APPLIES rows or fudge. Verdicts and actions are separate axes now. - No branch for a mandate that forbids all action. Step 0 reads the mandate; the template carries it as its first line. - Steps 2 and 3 could not both be obeyed (an inventory is numbers). Bars now govern only numbers a verdict turns on; inventory counts are exempt; a post-hoc bar is disclosed in the row. - Ideas 5 and 9 were two ideas each. Split into 5a/5b and 9a/9b — twelve rows. - Nothing told a second run how to treat a first. Added: re-verify, never inherit. - The "outward" boundary did not cover the record itself. Ruled internal. - Frontmatter promised a file path per verdict; DOES NOT APPLY supplies a command and its (empty) output instead. Reworded. Also: "~13 health gates" in the worked example was stale (26 in the manifest); README says twelve rows and names the cold run. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01DGSjiRBHQmuF1BNw1W7dTF
…as written, 1 adapted A second model family reviewed the revised skill for ways it would make an agent write a wrong, unverifiable, or misleading record. Every finding carried line numbers; none touched security. What changed: - Mandate vocabulary defined (read-only / record-only / small reversible / full ships). Read-only no longer writes the record into the repo; writing the record never counts as "acted on." - ALREADY IN PLACE split into (code) and (running), counted separately. - APPLIES vs NOT DECIDABLE precedence: a missing record alone is NOT DECIDABLE. - Negative-search evidence form: directory, exact command, exit code, "0 matches" line. New fifth verdict UNREADABLE, never rounded to DOES NOT APPLY. - Three inventory rows the ideas needed and the procedure lacked: candidate generation/selection, pre-execution review points, model routing per call. - Inventory-count exemption from bars no longer covers a count a verdict turns on. Post-hoc bars stay disclosed (not run-invalidating — the adapted finding) but are now counted in a summary line at the top. - ideas.md: own-trace preference labelled a Q&A remark with no paper; idea 1 routes self-ranking to ideas 2 AND 8; idea 2's ALREADY IN PLACE needs filtering, weighting, and disagreement. papers.md: LATS un-mapped from 10. - Tests now match their headers: exact frontmatter key set; worked-example.md first mentioned only after step 5 and only as the gated instruction; reference scan covers links and full filenames, both directions. - Public copy corrected: "cold" → no memory but contaminated; eight → ten; "every paper the course teaches" → the videos' set; "a quoted path per verdict" → file:line or scoped command with exit code and output, in both manifests and the README row. Recorded in references/worked-example.md as the third round. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01DGSjiRBHQmuF1BNw1W7dTF
…t a repo we do not own NousResearch/hermes-agent at 4209d371 (10,666 files, a self-described self-improving agent), shallow-cloned into a scratch directory, nothing written into it. The full disposition record is references/worked-example-2.md, in the template's shape. Twelve rows: APPLIES 4 (5b, 6, 8, 10) · DOES NOT APPLY 2 (1, 2) · ALREADY IN PLACE code 4 (5a, 7, 9a, 9b) · running 1 (4) · NOT DECIDABLE 1 (3) · UNREADABLE 0. Acted on: 0, by mandate. Bars written after their number: 1, disclosed. The take, if only one thing gets done there: the learning loop's judge is the generator by default — background_review.py forks the same model to decide what to save, and the auxiliary chain that runs the curator, /review and the goal judge resolves to the main model first — while a second family is plumbed and defaulted for MoA only. Two config defaults would move it. Also on record: a held-out-eval setting in an example config that no code reads; a learning-loop invariants doc referenced from three files that lives in a hub-installed skill, unreachable from the tree; CI on main mixed 78/6/14/2 over 100 runs in 31 hours, so the check carries signal. The negative-search form earned its place: a keyword sweep for judges read as "none," and writing the DOES NOT APPLY row — exact command, exit code, match list — forced a phrase-level re-run that found two. Ten notes on the skill from the run sit at the bottom of the file. None are applied here; several change the template, and that is a decision, not a patch. The record says "refuted by: nobody yet" and means it — no upstream issue was filed. SKILL.md step 7 and the references table now name the second file, gated after step 5 like the first; the template's sources list gains its line and a "before the run — say so" answer, which this run needed. README gets one sentence. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01CkTF1JryMkGt4Qx2A2vHEh
…ndings, 12 as written, 2 adapted A cross-family adversarial round against worked-example-2 returned fourteen findings (5 P0, 6 P1, 3 P2). Every one was re-verified against the tree at 4209d371 before folding. Two verdicts fell: - Idea 2, DOES NOT APPLY → NOT DECIDABLE. The inventory searched only Python, and the judge ensemble is markdown: research-paper-writing's three blind CoT judges ranking by Borda count, a 3-5 reviewer ensemble, a meta-reviewer. The record had accepted a markdown skill as real functionality for idea 7, so it could not consistently exclude this one. In a skills-based ecosystem, prompts are code; a Python-only glob inventories half the system. - Idea 9b, ALREADY IN PLACE → APPLIES. verification_evidence is scope bookkeeping — status comes from exit_code == 0, and nothing judges intermediate steps. Scope honesty is orthogonal to process reward. One claim asserted the opposite of the code: "saves every trajectory" — batch_runner saves ONLY successes and discards zero-reasoning samples, and the record's own not-checked line admitted the file was never read past its header. The claim came from the header. Also folded: 5a and 9a citations moved from grep-adjacent lines to the lines that do the thing (smart approval is real, reachable, and opt-in; the breaker is prompt-text escalation, not a block); idea 1 narrowed (the repo's own hyphenated "best-of-N" vocabulary broke the search); idea 3's fill-rate estimate struck; idea 6's regime claim rested on the one allowance that is a duration; idea 4's (running) attributed to CI alone; three zero-match receipts restated with directory and exit code; the disagree count restated under the reproducible command; the sources list gains the row for the record itself. The round is a section in the record, original wrong rows quoted — and the record's own most-likely-wrong nomination was wrong: it named 5a, which survived; 2 and 9b fell. Self-assessment of the weakest row is a same-model judgment. Idea 8, in miniature, inside the record about idea 8. New counts: APPLIES 5 · DOES NOT APPLY 1 · code 3 · running 1 · NOT DECIDABLE 2. Sum still 12. README sentence updated; the skill's where-it-stops bullet now tells this story. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01CkTF1JryMkGt4Qx2A2vHEh
…ed, 1 was praise
The ten notes at the bottom of worked-example-2, David-approved, now in the
skill itself:
- The answer keys move to references/answer-keys/ behind a door-sign README.
The prose gate ("step 7 only") failed on two runs in a row, because
loading the skill means reading its file layout; the structural gate makes
the failure require effort instead of preventing it. The gating test now
checks both halves: first-mention-after-step-5 for each key, AND that each
key physically lives in the quarantined folder.
- Step 2: the glob is part of the receipt; prompt/skill/workflow dirs are
searched with the same categories (markdown workflows are code — a judge
ensemble hid from one run exactly this way); inventory by search with
bounded reads, never a megabyte file whole; a config setting nothing
reads is evidence.
- Step 3: a number you meet while inventorying counts as read.
- Step 4: when the claim is about what code DOES, the receipt is the doing
line, not the docstring — a header asserted the opposite of the body, and
one audited run quoted it.
- Template: "(code, opt-in)" as row prose under ALREADY IN PLACE (adapted —
not a third counted tag); "Surfaces unreachable: N" beside UNREADABLE,
which counts ideas and buried a real unreachable surface at zero; time
spent includes setup and any refutation round; sources list paths updated.
- ideas.md idea 2: lone judges route to 3 and 8 explicitly, and a
markdown-defined ensemble counts. Idea 6 reworded (adapted — not split):
the verdict turns on whether abandonment and repetition are COUNTED
failure classes; the regime is context, and only wall-clock allowances
place a dispatch in one.
Not applied: the note praising the negative-search form — it is the reason
the idea-2 verdict got corrected mid-run, twice.
190 pass; plugin validates.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01CkTF1JryMkGt4Qx2A2vHEh
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What this adds
An Agent Skill,
skills/cs329a-self-improving-agents/, and the plugin/marketplace manifests that make this repo installable with:The skill hands an agent ten ideas distilled from Stanford CS329A "Self-Improving AI Agents" and asks it to decide, idea by idea, whether each names a gap in the repo it is pointed at — writing a disposition record with one verdict per row (APPLIES / DOES NOT APPLY / ALREADY IN PLACE / NOT DECIDABLE), a quoted
file:lineor command output per verdict, bars pre-committed before any deciding number is read, and "declined, because X" accepted as complete. It offers; the agent decides; the record is what a human reviews. Same shape asSELF-AUDIT.md, with a source instead of five questions.What it deliberately does not ship
The lecture transcripts. They are Stanford's.
references/lectures.mdlinks the nine videos with our own per-part summaries;references/papers.mdcarries the 28 papers with URLs that were each opened and matched to the title before the file was written (METR's paper has been retitled since v1; SWiRL's canonical title has no prefix — both noted).test/skill-references.test.mjspins the static shape always, and underCHECK_LINKS=1— a separate CI job, so an arXiv outage reds one legible line, never the unit suite — checks every link still resolves.The honest test, and what it changed
Before opening this PR, a fresh agent with no memory of the system was handed the skill as first written and pointed at the repo the worked example describes, under a record-only mandate. It found three things the earlier sub-agent run had missed (a per-lens history file that had written zero rows and is years from its own bar; a gate-history file living in a gitignored directory; a persona-testing planner and evaluator that are the same model), and eight defects in the skill — the worst being that
worked-example.mdfunctioned as an answer key. All eight are fixed in the second commit and recorded, with the run's own contamination caveat, inreferences/worked-example.md.Review ask
The README "Skills" section and the new table row are the public face — those are the lines to read. Everything else is the skill folder, the two JSON manifests (
claude plugin validatepasses), the test, one CI job, and a version bump to 0.4.0.🤖 Generated with Claude Code
https://claude.ai/code/session_01DGSjiRBHQmuF1BNw1W7dTF