v0.6.1 — prompt tightening (caveman measured & dropped)#47
Merged
Conversation
Opt-in, default-off output-token reduction, orthogonal to the engine trio. A stdlib terse-output preamble prepended to FREE-PROSE claude -p dispatches at the single chokepoint (hooks/_background_agent.py:dispatch), skipping schema-constrained ones via the output_schema it already has — so JSON verdicts/findings and the verifier/skeptic/gate prompts are never touched. Plus a deterministic tightening of the verbose non-verdict prompt constants. Honest + measurement-gated: caveman's headline ~65% does not transfer (Forge output is mostly already-terse schema-constrained JSON); if the measured free-prose saving is <10%, the toggle is dropped and only the static tightening ships. caveman-compress (model call + pip) and caveman-shrink (Node/MCP proxy) are out per REQ-NF-024. ADR-011. Ref: T-220
Add the v0.6.1 caveman toggle to OrchestrationConfig, mirroring session_reuse: - caveman_mode: bool = False added to _TOGGLES (strict `is True`, fail-soft). - caveman_level: str = "lite" coerced to lite|full; any other value (typo, unsupported ultra/wenyan, non-string) keeps the safe default. Inert this task — not yet wired into dispatch, so the engine and every dispatch stay byte-identical to v0.6.0 (REQ-NF-041). T-222 consumes the resolved level. 10 new config tests; full unit suite 1786 green; validate-plugin 0. Ref: T-220 Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
New pure-stdlib, never-raising module — the one caveman lever that ports under the no-pip rule (REQ-NF-024): - terse_preamble(level): lite|full instruction string; unknown ⇒ lite. - apply(prompt, *, enabled, has_schema, level): prepends the preamble, but is the IDENTITY when disabled OR has_schema (schema-constrained dispatches never altered, REQ-NF-041), and degrades to the original prompt on any internal error. Not yet wired into dispatch (T-222 does that) — adding the module alone changes no behavior. 14 new tests (lite/full/unknown coercion, identity matrix, exact composition, never-raises on raising-preamble and non-str prompt). Ref: T-221 Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Apply _caveman.apply to the prompt in _background_agent.dispatch, immediately before cmd is built (REQ-CM-002): - Level resolves from the threaded `caveman` arg, else the FORGE_CAVEMAN env var (off|lite|full), else off; an unsupported value resolves to off. - has_schema = output_schema is not None, so schema-constrained dispatches (verify verdicts, skeptics, gate, structured miners) are skipped structurally — their output is never altered. Thread the resolved level config→engine→dispatch, mirroring session_reuse: - run_workflow gains a `caveman` param, injected into node_kwargs and child sub-DAG run_kwargs ONLY when set — so off ⇒ the dispatch kwargs are byte-identical to v0.6.0 (REQ-NF-041). - _orchestrate.fan_out gains a `caveman` param threaded to run_workflow. The verifier always sets output_schema (auto-skipped); parallel_build's code-builders are deliberately NOT threaded (REQ-NF-041 code fidelity) — they honor only the global env switch if an operator opts in. Observer's free-text JSON poll is a documented known limit (ADR-011). 10 new tests (dispatch apply/skip/off/env/override/invalid; engine + fan_out thread-when-set / absent-when-off). Full unit suite 1810; integration 14; validate 0; full-pipeline 12/12 with the toggle off AND on. Ref: T-222 Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
… no toggle) Drop pure filler (pleasantries, redundant articles/clauses) from the verbose free-prose prompt constants — the reliable static win, independent of the caveman toggle (REQ-CM-004): - dreamer._CONSOLIDATION_PROMPT — drop "You have just completed…/Provide a single short paragraph"; keep the output spec (3-5 sentences, terse, no bullet lists). - autopilot._stage_prompt / _heal_prompt — "Run the work for X in this project" → "Run X"; drop "with blocking issues" / "and advances". - parallel_build._default_build_prompt — drop "the project's" / "for this task". The verify / skeptic / gate-verdict / observer prompts are LEFT UNCHANGED (clarity-/JSON-sensitive) — asserted by new shape tests. Semantics preserved on every tightened prompt (instructions + output format intact). 7 new shape tests (tightened forms + dropped-filler absence + verify/skeptic/ observer unchanged). Full unit suite 1817; validate 0; full-pipeline 12/12. Ref: T-223 Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
REQ-CM-005 gate: prove a >=10% free-prose output-token saving or ship only the
static tightening. A real before/after on the Dreamer consolidation prompt
(model haiku, N=5/arm, actual usage.output_tokens via the dispatch chokepoint)
measured mean -52.9% (OFF 621.2 -> ON 949.6) — no reduction; the preamble arm
was larger on average, median saving ~5%, both under 10%.
Forge's free-prose prompts are already length-bounded ("3-5 sentences, terse,
no bullet lists"), so a generic brevity instruction has no headroom — exactly the
prior the SRS recorded. Shipping a token-reduction toggle that measurably does
not reduce tokens would be dishonest; the gate is followed, not rationalized.
Decision: DROP the runtime toggle. This reverts:
- b49ef00 (T-220) caveman_mode/caveman_level config
- 02464d0 (T-221) hooks/_caveman.py core
- 52bc5d1 (T-222) dispatch chokepoint + engine wiring
and their tests. dispatch / run_workflow / fan_out / OrchestrationConfig return
byte-identical to v0.6.0.
KEPT: T-223 (550851e) static non-verdict prompt tightening — a real always-on
input-token win, independent of any toggle.
Recorded: build/06-evaluation/v0.6.1-caveman-measurement.md (raw data + honest
limitations) and decisions.md. Full unit suite 1783; validate 0; full-pipeline
12/12.
Ref: T-224
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Record the v0.6.1 outcome honestly: - ADR-011 (new): caveman mode built behind a default-off toggle, measured (mean -52.9% on the Dreamer free-prose prompt, <10% gate), runtime toggle rejected per REQ-CM-005; static prompt tightening shipped instead. caveman-compress/caveman-shrink rejected at research time (stdlib violation). - ROADMAP: v0.6.1 row (landing) + the caveman section updated to the measured/ dropped outcome (keeps the deferred-follow-up keywords). - progress.md: v0.6.1 current-state entry with the gate decision + commit chain. No code change (validate 0). references/orchestration-config.md + README need no edit — the dropped toggle was never advertised there. decisions.md entry landed in T-224. Docs assertion tests green. Ref: T-225 Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
…dropped) Bump 0.6.0 → 0.6.1 (plugin.json + marketplace.json). CHANGELOG [0.6.1]: - Changed: deterministic tightening of verbose non-verdict prompt constants (dreamer/autopilot/parallel_build) — a small always-on input-token win. - Rejected (measured, not shipped): the caveman runtime toggle. Built and fully tested, then dropped because the measured free-prose output-token saving was negative (mean -52.9%, <10% REQ-CM-005 gate). ADR-011 + eval note. - Unchanged: dispatch/engine/config byte-identical to v0.6.0; no new config. ROADMAP v0.6.1 row 🟡→🟢; progress.md LANDING→RELEASED. Banner/social evergreen. Pre-release green: unit 1783, validate 0, full-pipeline 12/12, integration 14. Ref: T-226 Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
v0.6.1 — caveman mode measured and rejected; static prompt tightening only
An honest, measurement-gated release with zero default behavior change.
What shipped
dreamer/autopilot/parallel_build) — a small always-on input-token win. verify/skeptic/gate/observer prompts left unchanged.What was measured and dropped
orchestration.caveman_mode,hooks/_caveman.py, dispatch-chokepoint + engine wiring) was built and fully tested, then dropped per the REQ-CM-005 gate: a real before/after on the Dreamer free-prose prompt (haiku, N=5/arm, actualusage.output_tokens) showed mean −52.9% — no saving, well under the ≥10% bar (Forge's free-prose prompts are already length-bounded). The measurement gate worked as designed.build/06-evaluation/v0.6.1-caveman-measurement.md.Invariants
_background_agent.dispatch,run_workflow,_orchestrate.fan_out,OrchestrationConfigare byte-identical to v0.6.0. No new config, noFORGE_CAVEMANenv var.T-220..T-226. SRS
build/01-srs/srs-v0.6.1.md, DAGbuild/04-plan/task-dag-v0.6.1.md.🤖 Generated with Claude Code