Skip to content

fix(learn): canonical stateRoot resolution + scope-routed guideline synthesis - #33

Open
cmschuetz wants to merge 1 commit into
mainfrom
fix/learn-resolver-and-scope-routing
Open

fix(learn): canonical stateRoot resolution + scope-routed guideline synthesis#33
cmschuetz wants to merge 1 commit into
mainfrom
fix/learn-resolver-and-scope-routing

Conversation

@cmschuetz

Copy link
Copy Markdown
Owner

Why

Investigating "is /strapped:learn broken? feedback isn't being collected" surfaced that feedback was being collected (10 unsynthesized critiques across 5 runs) — the failure was a false negative: learn was the only strapped skill that never resolved stateRoot via state.mjs, so any mis-resolution glob'd to zero and reported "no critiques, stop" — indistinguishable from genuinely none.

A second, deeper issue: learn dumped every synthesized guideline into one CLAUDE.md, whose load scope is only "working in the plugin repo." So harness-behavior rules learned from runs never fired for runs against other target repos — the majority of intended use.

Heads-up (from the requester): this likely won't merge as-is — it's a reference point for the direction.

What

Resolver gap (the concrete bug)

  • New state.mjs runroot — slug-less { stateRoot, runRoot } resolution that hard-fails on an unresolvable/relative anchor, so an empty cross-run glob is unambiguously "no runs" and never a silently-wrong root. Documented in conventions.md; +3 CLI tests.
  • learn Step 1 now resolves via runroot, echoes the resolved root + match count, and treats a resolution error as a loud stop.

Scope routing (the design fix)

  • learn Steps 2–5 now classify each cluster harness-behavior vs repo-development and route it:

    • harness rules → most specific stage prompt (src/workflows/strapped-run/stages/*.ts, review-loop.ts) / SKILL.md step / conventions.md
    • repo-dev rules → the pertaining repo's CLAUDE.md (resolved from the run's repos map), not necessarily the plugin's.
  • The 4 generalizable guidelines synthesized this pass were relocated out of CLAUDE.md into their targeted prompts so they reach every run:

    Guideline New home
    Verify plan claims vs origin/main (fetch first) plan.ts planner
    Add a real test suite as the validation gate plan.ts planner
    Review design decisions vs the ask's intent review-loop.ts completeness lens
    No grep-guard prose regression tests implement.ts implementer
    Reconcile plan vs current main before implementing implement/SKILL.md pre-flight
  • CLAUDE.md keeps only genuinely repo-dev rules (new-repo/breaking-changes-fine, no-machinery-without-signal) + a pointer to where harness guidance lives.

  • Plugin 0.6.0 → 0.7.0.

Not addressed (flagged)

Critique writing remains a pure model-followed instruction with no hook/script enforcement — autonomous /strapped:run flows skip it silently. Left as a follow-up.

Test

bun run typecheck && bun run lint && bun test → 109 pass (+3). Generated strapped-run.js rebuilt and in sync; all 4 relocated snippets verified present.

…ynthesis

The learn skill was the only strapped skill that never resolved stateRoot via
state.mjs — it left resolution to improvisation, so any mis-resolution glob'd to
zero and reported "no critiques, stop", indistinguishable from genuinely-none.
It also dumped every synthesized guideline into one CLAUDE.md, whose scope is
only "developing the plugin repo" — so harness-behavior rules learned from runs
never fired for runs against other repos.

- Add `state.mjs runroot`: slug-less { stateRoot, runRoot } resolution that
  hard-fails on an unresolvable/relative anchor, so an empty cross-run glob is
  unambiguously "no runs". Documented in conventions.md, +3 CLI tests.
- learn Step 1 now resolves via `runroot`, echoes the root + match count, and
  treats a resolution error as a loud stop (never "no critiques").
- learn Steps 2-5 classify each cluster harness-behavior vs repo-development and
  route it: harness rules to the most specific stage prompt / SKILL / conventions;
  repo-dev rules to the *pertaining* repo's CLAUDE.md (resolved from the run's
  repos map), not necessarily the plugin's.
- Relocate the 4 generalizable guidelines synthesized this pass into their
  targeted prompts so they reach every run:
    * verify plan claims vs origin/main (fetch first)  -> plan.ts planner
    * add a real test suite as the gate                -> plan.ts planner
    * review design decisions vs the ask's intent      -> review-loop.ts lens a
    * no grep-guard prose regression tests             -> implement.ts implementer
    * reconcile plan vs current main before implement  -> implement/SKILL.md
  CLAUDE.md keeps only genuinely repo-dev rules (new-repo/breaking-changes,
  no-machinery-without-signal) + a pointer to where harness guidance lives.
- Bump plugin 0.6.0 -> 0.7.0.
cmschuetz added a commit that referenced this pull request Jul 28, 2026
Remove four genuine, meaning-preserving redundancies from the harness stage
prompts, each verified via the D1-D3 eval suite's A/B mode with no correctness
regression:

- planner (stages/plan.ts, steps 3-5): define "meaningful diff" once; state the
  cross-repo base rule + ordering-only consequence once (drop step 4's restatement)
- reviewer (review-loop.ts): collapse the AC-checklist paragraph's doubled
  equal-weight clause
- soundness lens b (review-loop.ts PLAN_LENSES.b): define "meaningful diff" once
  instead of repeating the exclusion phrase twice
- implementer (stages/implement.ts): fold the filler lead-in into the header and
  drop the doubled "commit" in the closing paragraph

The eval cases (planner/reviewer/implementer) now carry baseline->candidate
variants so `bun run eval --suite src/eval/suites/harness --ab` reproduces the
verification (exit 0: correctness held at parity across all cases). End-to-end
envelope cost/latency is below the harness's agentic execution-noise floor, so
the attributable win is the deterministic prompt-text reduction (-302 chars);
the eval's decisive role is certifying zero correctness regression. A written
PR-#33 verdict and full A/B numbers live in the run dir's optimizations-report.md.

Regenerates plugins/strapped/workflows/strapped-run.js; plugin 0.8.0 -> 0.8.1
(patch: prompt compaction, no behavior change).

Implements D4 of the evaluation-system run.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant