Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
95 changes: 95 additions & 0 deletions devlog/2026-07-23_quality-gate-rejection-prompt/DESIGN.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,95 @@
# DESIGN - Quality-gate rejection recovery guidance

- Task ID: `2026-07-23_quality-gate-rejection-prompt`

## 1. Goal

Make the correct compression-rejection recovery path discoverable by the model,
so a non-adaptive model (e.g. glm-5.2) does not deadlock when the blocking
pre-commit quality gate rejects an oversized-range compression.

## 2. Data flow (unchanged)

```
compress(args) → validateArgs → prepareSession → resolveRanges
→ evaluatePreCommitQuality ← BLOCKING gate (qualityGate.enabled)
passed → applyCompressionState → finalizeSession → commit
FAILED → buildQualityRejectionError → THROW
state.qualityGateRetryPending = true
(model sees error text, must retry)
```

This PR does NOT change the flow. It only changes the **content** the model sees
at three points: the tool schema, the thrown error text, and the system prompt.

## 3. Changes

### 3.1 Tool schema — discoverable escape hatch

`lib/compress/range.ts` + `lib/compress/message.ts`: add `.describe(...)` to
`acknowledgeRisk`. The describe states it bypasses the quality gate when the
model judges the information loss acceptable, is usable on the first attempt
(no prior rejection needed), and that for content which matters a dense
summary or splitting an oversized range is preferred. (Round 2 removed the
preemptive-ACK guard — see §6 of WORKLOG — so ACK is no longer gated behind a
prior rejection.)

### 3.2 Rejection message — recovery path sized to the failure

`lib/compress/quality-gate/rejection.ts` `buildQualityRejectionError`:
introduce a "large range" branch using `originalTokens` (already computed in
`computeStats`).

- **Large range** (`originalTokens > 50000`): lead with **SPLIT** guidance —
"break into 2-3 smaller contiguous ranges, compress each in the same batch
call". A single dense summary is impractical at this size.
- **Small range**: keep the existing "write a denser summary" lead.
- **Both paths** additionally mention `summaryMaxChars` (raise the cap) and
`acknowledgeRisk: true` (last resort). Keep the existing metrics block
(Range / Original / Summary / Ratio / Retention / Gate layer / rougeF1 /
top20Recall) and the appended `HOW_TO_COMPRESS_RULES`.

Threshold rationale (token-only, ratio intentionally excluded): the branch is
gated on **absolute token count only** (`originalTokens > 50000`), NOT on the
summary:tokens ratio. A small range with a terse summary (e.g. 1K tokens → 10
chars, ratio 400:1) is better served by "write a denser summary" advice, not
"split the range" — splitting an already-small range makes no sense. L1 floor is
`retentionPct = summaryChars / (originalTokens*4) >= 1%`, i.e.
`summaryChars >= originalTokens*0.04`. At 50K tokens the dense minimum is
already 2000 chars and grows linearly; by ~250K it exceeds 9000 chars —
impractical for one summary. 50K is a sensible "crossover" where splitting
becomes the better advice. (The `ratioNum` field in `computeStats` is computed
only for the displayed metrics string; it does not influence the branch.)

### 3.3 System prompt — set expectations up front

`lib/prompts/system.ts`: add a "COMPRESSION REJECTION HANDLING" section (placed
after the tools / philosophy block, before PERIODIC CONTEXT STATUS). It states:
(1) compress can be rejected when qualityGate is on, (2) the three recovery
paths in priority order (split → denser/longer → acknowledgeRisk), (3) an
explicit anti-loop rule: after two rejections of the same range, change
strategy rather than resubmit an identical summary.

## 4. Backward compatibility

- No persisted-state field added/removed.
- No exported signature change. `buildQualityRejectionError(plan, result)`
keeps the same signature; only its output text changes.
- No internal `dcp` tag touched.
- The blocking gate's pass/fail logic is untouched — only guidance text.

## 5. Testing

- Existing `tests/quality-gate-enforcement.test.ts` cases (1000-token small
ranges) stay green: header, range, "1000 tokens", `acknowledgeRisk`,
`HOW TO COMPRESS` all still present.
- New case: large range (>50K tokens) → message includes "split" advice.
- New case: small range → message keeps "denser" advice (regression guard).

## 6. Out of scope (follow-ups)

- Anti-deadlock counter: after N consecutive rejections of the same range,
auto-downgrade to non-blocking. Requires a `qualityGateConsecutiveRejects`
counter in `SessionState` → persistence migration. Separate PR.
- Making `qualityGate.enabled`'s blocking-vs-non-blocking behavior match the
README (which only documents the non-blocking post-commit gate). Doc PR.
76 changes: 76 additions & 0 deletions devlog/2026-07-23_quality-gate-rejection-prompt/REQ.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,76 @@
# REQ - Quality-gate rejection: tell the model the correct recovery path

- Task ID: `2026-07-23_quality-gate-rejection-prompt`
- Home Repo: `opencode-acp`
- Created: 2026-07-23
- Status: InProgress
- Priority: P1
- Owner: awork
- References: dog/opencode-acp#33

## 1. Background & Problem Statement

- **Context**: `qualityGate.enabled: true` activates a **blocking pre-commit**
quality gate (`evaluatePreCommitQuality` in `lib/compress/quality-gate/evaluate.ts`,
wired in `lib/compress/range.ts`/`message.ts`). A summary that fails the gate
throws `buildQualityRejectionError` and the compression does not commit.
- **Current behavior (symptom)**: In session `ses_07fa6ea18ffe...` (issue #33),
glm-5.2 tried to compress a ~243K-token range into an 8543-char summary
(0.88% retention, L1 floor is 1%). The gate rejected it 23 times. Each
rejection re-billed ~320K tokens and injected ~15KB of error text. The model
never used the `acknowledgeRisk` escape hatch (0/23) and never split the range.
Result: context stuck at ~320K, ~millions of tokens billed for zero progress.
- **Expected behavior**: After a rejection, the model must be able to discover a
*correct* recovery path. The three valid paths are: (1) split an oversized
range into smaller ranges, (2) write a denser / longer summary (raise
`summaryMaxChars`), (3) `acknowledgeRisk: true` as last resort.
- **Impact**: Deadlock for non-adaptive models → token blowup + unusable
compression when `qualityGate.enabled` is on.

## 2. Root Cause (prompt-chain audit)

Three gaps make the correct recovery path undiscoverable:

1. **Tool schema** (`lib/compress/range.ts:91`, `lib/compress/message.ts:66`):
`acknowledgeRisk: tool.schema.boolean().optional()` has **no `.describe()`**
(sibling fields `dangerous` and `summaryMaxChars` do). The model cannot learn
what the parameter does from the tool definition.
2. **Rejection message** (`lib/compress/quality-gate/rejection.ts:42-75`): tells
the reason (metrics) + mentions `acknowledgeRisk`, but the only recovery hint
is "rewrite a more complete summary" — wrong/impossible for a 243K-token
range (would need ≥9725 chars). Never advises **splitting** the range, never
mentions `summaryMaxChars`.
3. **System prompt** (`lib/prompts/system.ts`): completely silent on the fact
that compression can be rejected and how to recover. The model has no prior
expectation of rejection.

## 3. Constraints & Non-Goals

- **Constraints**:
- No `as any` / `@ts-ignore` (AGENTS.md).
- Backward compatible: no persisted-state format change, no internal `dcp`
tag rename (AGENTS.md §2.6).
- L1 gate contract unchanged — we only improve *guidance text*, not the gate
thresholds.
- Follow existing code style (double quotes, no semicolons, 4-space indent).
- **Non-Goals (this PR)**:
- Anti-deadlock auto-downgrade after N consecutive rejections (needs a state
counter → schema change). Tracked as a follow-up.
- Changing gate thresholds or the blocking-vs-non-blocking design.
- Touching the external `context-compress-algorithms` package.

## 4. Acceptance Criteria

- [ ] `acknowledgeRisk` has a `.describe()` in both `range.ts` and `message.ts`.
- [ ] `buildQualityRejectionError` gives **split-range** guidance when the range
is large (>50K tokens, absolute count — ratio intentionally excluded so a
small-but-terse range still gets denser-summary advice), and keeps
denser-summary guidance for small ranges. Both paths mention
`summaryMaxChars` + `acknowledgeRisk`.
- [ ] `system.ts` has a "COMPRESSION REJECTION HANDLING" section listing the
three recovery paths in priority order, and an explicit "do not loop"
rule.
- [ ] New test asserts large-range guidance mentions splitting.
- [ ] Existing tests in `tests/quality-gate-enforcement.test.ts` still pass.
- [ ] `npm run typecheck`, `npm run format:check`, `npm test`, `npm run build`
all pass.
121 changes: 121 additions & 0 deletions devlog/2026-07-23_quality-gate-rejection-prompt/WORKLOG.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,121 @@
# WORKLOG - Quality-gate rejection recovery guidance

- Task ID: `2026-07-23_quality-gate-rejection-prompt`
- Home Repo: `opencode-acp`
- Status: Done
- Updated: 2026-07-23 00:55

## 1. Summary

- **What was done**: Made the quality-gate rejection recovery path discoverable
by the model — added a `.describe()` to `acknowledgeRisk`, branched the
rejection message to advise SPLITTING oversized ranges (vs the old "rewrite
longer" hint that is impossible for 243K-token ranges), and added a
"COMPRESSION REJECTION HANDLING" section to the system prompt.
- **Why**: In dog/opencode-acp#33, glm-5.2 deadlocked — it tried to compress a
243K-token range into an 8543-char summary, the blocking pre-commit gate
rejected it 23×, and the model never discovered the correct recovery (split
/ summaryMaxChars / acknowledgeRisk). Each failure re-billed ~320K tokens.
- **Behavior / compatibility changes**: No. The gate's pass/fail logic, state
schema, persisted format, and exported signatures are unchanged. Only guidance
text (tool schema description, error message body, system prompt) changed.
- **Risk level**: Low.

## 2. Change Log

### Commits

| Commit | Description |
|--------|-------------|
| (this PR) | fix: guide models to the correct quality-gate rejection recovery path |

### Files

| File | Change |
|------|--------|
| `lib/compress/range.ts` | `acknowledgeRisk` gains a `.describe()` (was the only param without one) |
| `lib/compress/message.ts` | same `acknowledgeRisk` `.describe()` for message mode |
| `lib/compress/quality-gate/rejection.ts` | `buildQualityRejectionError` now branches: ranges >50K tokens get SPLIT guidance, smaller ranges get denser-summary guidance; both mention `summaryMaxChars` + `acknowledgeRisk` |
| `lib/prompts/system.ts` | new "COMPRESSION REJECTION HANDLING" section (3 recovery paths in priority order + anti-loop rule) |
| `tests/quality-gate-enforcement.test.ts` | +2 tests: large-range → split guidance; small range → denser-summary guidance |

## 3. Verification

- `npm run typecheck` — clean.
- `npm test` — 837 pass, 0 fail (includes the 2 new tests).
- `npm run build` — success; `dist/index.js` contains "SPLIT THE RANGE",
"COMPRESSION REJECTION HANDLING", and the new `acknowledgeRisk` describe.
- Existing rejection tests (`buildQualityRejectionError includes range, stats,
and acknowledgeRisk instructions`; ratio/retention computation) still pass.

## 4. Pre-existing issues noticed (NOT addressed here)

- `npm run format:check` flags 246 files on clean `github/master` — prettier
config drift, predates this PR, not enforced by CI (`ci.yml` runs only
typecheck + test). Reported separately; this PR keeps its diff to intended
changes only.
- README/AGENTS.md document the quality gate as "non-blocking (logger.warn
only)" but `qualityGate.enabled` also activates a blocking pre-commit gate.
Doc mismatch — follow-up.

## 5. Follow-ups

- README doc fix for the blocking-vs-non-blocking gate description.

## 6. Round 2 — make `acknowledgeRisk` usable anytime (per issue #33 feedback)

- **Trigger**: user analyzed the pre-restart session (dog/opencode-acp#33) and
found the model doing a "write-short → rejected → retry-with-ACK" dance 7× in
a row. User's reframe: the "ACK only after a rejection" rule MANUFACTURES the
deliberate-fail pattern, and over-blocking causes the model to avoid
compressing entirely (the actual root of the 330K problem) — worse than
occasional bad compressions.
- **Change**: removed the preemptive-`acknowledgeRisk` guard so ACK bypasses
the gate on the FIRST attempt, no prior rejection needed. The gate still
BLOCKS non-ACK calls (protection by default + explicit opt-out → the model is
never trapped into avoiding compression). Post-commit `evaluateBatchQuality`
still runs + warns → observability retained.
- **Removed**: `buildPreemptiveAcknowledgeError` (now dead code), its export,
and its test. The integration test "preemptive acknowledgeRisk ... is
rejected" was FLIPPED to assert it now succeeds.
- **Wording**: `acknowledgeRisk` `.describe()` and the system-prompt ACK bullet
no longer say "never preemptive"; they say "usable on the first attempt".
- **Verification**: `npm run typecheck` clean; `npm test` 836 pass / 0 fail;
`npm run build` ok; bundle verified (guard text gone, new describe present).
- **Dropped earlier idea**: "constrain ACK / refuse short-summary ACK /
rate-limit" — would worsen over-blocking; explicitly rejected by user.
- **Still out of scope**: making the gate fully advisory (warn-only). This PR
keeps it blocking for non-ACK calls; free-ACK is the middle ground.

## Iteration 3 — hardcoded-values cleanup (review follow-up)

- **Trigger**: review (dog/opencode-acp#34) flagged three hardcoded values that
drifted from config reality.
- **Fix 1 — `ratioNum` dead code** (`rejection.ts`): the PR added a
`ratioNum: number | null` field to `computeStats`'s return, but no caller
read it (`buildRecoveryGuidance` uses `originalTokens`; metrics use the
pre-formatted `ratio` string). Leftover from a dropped ratio-branching idea.
Removed the field + its computation; inlined `ratio` again.
- **Fix 2 — `12000` stale example** (`rejection.ts`, `system.ts`): the
escapeHatches string and system-prompt both hardcoded `12000` as a
`summaryMaxChars` example, but the real `maxSummaryLengthHard` default is
`20000` (`config.ts`), and the tool schema already interpolates the real
value. The example was both wrong and frozen (didn't follow config).
- `rejection.ts`: `buildQualityRejectionError` now takes
`maxSummaryLengthHard: number` as a 3rd param; `buildRecoveryGuidance`
receives it and interpolates `${maxSummaryLengthHard}`.
Callers `range.ts`/`message.ts` pass `ctx.config.compress.maxSummaryLengthHard`.
- `system.ts`: removed the number entirely (static prompt string can't
interpolate config; now says "see the tool schema for the current max").
- `tests/quality-gate-enforcement.test.ts`: all 4 call sites updated to pass
`20000`.
- **Fix 3 — `layer1MinRetentionPct` 5.0 vs 1.0** (`config.ts`): the runtime
default was `5.0` but `dcp.schema.json`, `README.md`, and the cc-alg
`DEFAULT_ROUGE_RECALL_V1_CONFIG` all say `1.0`. The v1.13.0 changelog
documents the floor as 1%. The `LARGE_RANGE_TOKENS = 50_000` threshold
rationale (crossover where the 1% floor demands a multi-thousand-char
summary) assumes 1%; at 5% the real crossover is ~12K tokens. Aligned
`config.ts` to `1.0`. Updated the `LARGE_RANGE_TOKENS` comment to name the
config key and note the proportional coupling.
- **Verification**: `npm run typecheck` clean; `npm test` 836 pass / 0 fail;
`npm run build` ok.
12 changes: 7 additions & 5 deletions lib/compress/message.ts
Original file line number Diff line number Diff line change
Expand Up @@ -22,7 +22,6 @@ import {
import { resolveKeepMarkers } from "./keep-markers"
import type { CompressMessageToolArgs } from "./types"
import {
buildPreemptiveAcknowledgeError,
buildQualityRejectionError,
evaluatePreCommitQuality,
} from "./quality-gate"
Expand Down Expand Up @@ -63,7 +62,12 @@ function buildSchema(maxSummaryLengthHard: number) {
.describe(
"Set to true ONLY when you are certain the most recent message(s) must be compressed. Required when a range includes the tail of the conversation.",
),
acknowledgeRisk: tool.schema.boolean().optional(),
acknowledgeRisk: tool.schema
.boolean()
.optional()
.describe(
'Set to true to bypass the quality gate when you judge the summary acceptable despite some information loss (e.g. the range is low-value, or you cannot make it denser). Usable on the first attempt. For content that matters, prefer writing a dense summary or splitting an oversized range instead.',
),
}
}

Expand Down Expand Up @@ -192,9 +196,6 @@ export function createCompressMessageTool(ctx: ToolContext): ReturnType<typeof t

const qualityGateRetryPendingBefore = ctx.state.qualityGateRetryPending

if (acknowledgeRisk && !ctx.state.qualityGateRetryPending) {
throw buildPreemptiveAcknowledgeError()
}
if (acknowledgeRisk) {
ctx.state.qualityGateRetryPending = false
} else {
Expand All @@ -219,6 +220,7 @@ export function createCompressMessageTool(ctx: ToolContext): ReturnType<typeof t
messageTokenById: plan.selection.messageTokenById,
},
result,
ctx.config.compress.maxSummaryLengthHard,
)
}
}
Expand Down
2 changes: 1 addition & 1 deletion lib/compress/quality-gate/index.ts
Original file line number Diff line number Diff line change
Expand Up @@ -14,6 +14,6 @@ export {
} from "./registry"

export { evaluateBlockQuality, evaluateBatchQuality, evaluatePreCommitQuality } from "./evaluate"
export { buildQualityRejectionError, buildPreemptiveAcknowledgeError } from "./rejection"
export { buildQualityRejectionError } from "./rejection"
export type { RejectionPlanInfo } from "./rejection"
export { ensureBuiltinGatesRegistered } from "./algorithms"
Loading
Loading