docs(routing): prefer enforcement over prose, and make harness-artefact materializable - #44
Merged
Merged
Conversation
The scope-escalation ladder ranked only the three prose destinations (skill-update / project-rule / user-memory), so a mechanically enforceable finding was pulled toward "add a paragraph to a skill" by the loudest rule in the skill. checkpoint and harness-artefact appeared only as a tiebreaker table. Make routing two-dimensional: axis 1 asks whether the rule can be enforced (mechanical gate > llm_reviews > prose), axis 2 applies the existing reach ladder to whatever stays prose. The enforcement ordering is not invented here — it cites agent-harness-skill's enforcement-mechanisms.md, which already ranks the ten instruments and states the CI/hook parity rule. Also name the axes' conflict explicitly (a gate reaches one repo, a skill reaches all of them) and resolve it: the prose beside a gate is the recipe for installing that gate elsewhere, not a restatement of the rule. Add linter/analyzer rules and branch protection to the harness-artefact artefact list; both were materializable but unlisted. Signed-off-by: Sebastian Mendel <info@sebastianmendel.de>
Destination 6 was a seven-line delegation stub: an artefact list plus "invokes agent-harness-skill bootstrap". patch-workflow.md covered only skill source repos, so a harness-artefact proposal had no target-repo selection, no rule against overwriting an existing hook config, and no handling for instruments that are an API call rather than a file. Five differences from the skill-repo case are now specified: target selection (worktree off the dirty session repo), verify-before-bootstrap against the AH-* checkpoints, CI/hook parity in the same PR, server-side rules emitted as a command for the user instead of a patch, and analyzer-config edits preferred where the analyzer already runs. Signed-off-by: Sebastian Mendel <info@sebastianmendel.de>
The only eval naming harness-artefact was a negative case (a transient tool error that must not become a gate). Nothing asserted that a mechanically preventable friction routes to a gate rather than to a prose rule, so the enforceability axis had no regression target. Covers the A17 shape where CI catches what a hook could have: extend the existing hook config, do not restate the rule in CLAUDE.md, and do not let the reach ladder claim the finding first. Signed-off-by: Sebastian Mendel <info@sebastianmendel.de>
The enforceability axis creates a case the "exactly one destination" contract could not express: a gate belongs in the repo the friction happened in, and the recipe for installing it belongs in the skill that owns the topic. Left unresolved, that either loses the propagation or splits into two proposals whose prose half can be approved while the gate half is rejected. A pair is one proposal, one approval, one slot against the ≤10 cap, and two reported rows (4a/4b) so a half-failed pair stays visible. Bounded at two parts; the propagation half carries installation, never a restatement of the rule. Also report server-side instruments as `manual` — they are an API call, not a patch, and retro never applies them. Signed-off-by: Sebastian Mendel <info@sebastianmendel.de>
|
CybotTM
changed the title
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem
classification-heuristic.mdanddestination-taxonomy.mdboth state the routing bias as a single ranked ladder:skill-update/new-skill›project-rule›user-memory. The two mechanical destinations —checkpoint(5) andharness-artefact(6) — are absent from that ladder. They appear only in a "choosing between adjacent destinations" tiebreaker table, which is then overridden three lines later by "bias upward in reach".Effect: a finding that a pre-commit hook, a CI job, or a linter rule could have caught gets routed to a paragraph in a skill, which the model can still ignore. Nothing in the skill states that a gate outranks a sentence.
Two further gaps surfaced while fixing that, and are fixed here as well: destination 6 had no materialization mechanics at all, and no eval asserted the mechanical routing.
Changes
1. Routing becomes two-dimensional (
ae6b0bf)checkpointmechanical:/harness-artefact) › LLM review (checkpointllm_reviews:) › prose (skill-update/project-rule/user-memory).The enforcement ordering is not invented here.
agent-harness-skill/references/enforcement-mechanisms.mdalready ranks the ten instruments server-side → convention-based and states the CI/hook parity rule ("when CI catches a mechanical issue that a hook could have caught, the absence of the hook is the bug"). This imports and cites it.Guard against over-correction: most B16–B18 reusable learnings are judgment-shaped and belong in tier 3. The test is whether a check could have failed on the friction as it actually occurred — not whether a regex could be written after the fact.
harness-artefactalso gains two artefact types that were materializable but unlisted: linter/static-analysis rules (ships where the analyzer already runs) and branch protection/rulesets.2. Materialization mechanics for destination 6 (
d7996d2)Destination 6 was a seven-line delegation stub — an artefact list plus "invokes
agent-harness-skillbootstrap".patch-workflow.mdcovered only skill source repos, so aharness-artefactproposal had no target-repo selection and no rule against clobbering an existing hook config. Five differences are now specified:AH-*checkpoint globs; extend the existing artefact, never overwrite itmanual, never apply silently3. Paired materialization (
6ef32c7)The axes genuinely conflict — a gate lands in one repo, a
skill-updatereaches every repo the skill touches. Resolved as a bounded exception to "every finding maps to exactly one destination":harness-artefact/checkpointskill-updateOne proposal, one approval, one slot against the ≤10 cap, two reported rows (
4a/4b) so a half-failed pair stays visible. Two parts maximum. The propagation half carries installation instructions, never a restatement of the rule the gate enforces — if the prose still makes sense with the gate deleted, it is a restatement and gets dropped.Splitting a pair into two proposals was the alternative and is rejected: it allows the prose half to be approved while the gate half is rejected, reproducing the exact failure this PR fixes.
4. Positive eval (
bdb8c52)evals/harness-artefact-over-prose.md— the A17 shape where CI catches in 90s what a hook catches in 2s. Asserts the gate routing, extending the hollow hook config rather than replacing it, and that the reach ladder does not claim the finding first. Previously the only eval namingharness-artefactwas a negative case.Verification
pre-commit run --files <changed>— markdownlint passed on all five filespython3 -m pytest tests/ -q— 76 passedvalidate-evals.py --evals-dir skills/retro/evals— 10 scenarios well-formed (was 9)"Paired materialization","Harness artefacts"and"Routing — enforceability first, then reach"citation resolves to an existing##headingSKILL.md is untouched — it sits at 491 words against the 500 cap, so the new material lives in the references the classify phase already reads.