writeup.md

Split-Turn Compaction Context

Bottom line: the feature adds a "Compacted Turn Context" summary of the split-turn prefix, so a retained tail that opens mid-turn keeps its request framing. Measurably valuable with a weak summarizer (Gemma 4 31B, 80K+ heads), redundant with a strong one (deepseek-v4-flash, up to 300K).

Part 1: The Feature

What the branch does

The split-turn-context branch fixes a compaction flaw. When the retained tail opens mid-turn, the compaction summary gains a dedicated section that summarizes the split-turn prefix, so the model resuming the tail still knows what the user asked and what was already done.

What was before

splitTurn cut the last turn at an arbitrary message boundary. The retained suffix could then begin mid-turn, with a tool call or a bare continuation, while the request that started the turn lived only in the summarized head. filterCompacted keeps a contiguous suffix from tail_start_id, so the request could not be kept verbatim without also keeping everything up to the cut, which is exactly what the token budget could not fit. Summarizing the prefix was the only framing option.

The mechanism

In packages/opencode/src/session/compaction.ts:

• Tail gains prefixStart; splitTurn records turn.start when cutting mid-turn.
• select returns turnPrefix (request plus work before the cut) alongside the unchanged head.
• processCompaction runs a second processor.process call with TURN_PREFIX_SUMMARIZATION_PROMPT, producing a **Compacted Turn Context:** section (## Original Request, ## Early Progress, ## Context for Suffix). It appends to the same summary message, so previousSummary carries it across re-compactions.
• Best-effort: a failed or overflowing second call logs and clears the error, restores finish, and continues with the main summary alone. Never hard-fails. Skipped when a plugin overrides the compaction prompt or the prefix is empty.
• Additive: the head is unchanged, so the old artifact is exactly the new artifact minus the turn-context part. A/B is exact.

Changes to the code

The prompt (TURN_PREFIX_SUMMARIZATION_PROMPT):

const TURN_PREFIX_SUMMARIZATION_PROMPT = `The messages below are the PREFIX of a turn that was too large to keep. The SUFFIX (recent work) is retained verbatim in context after this summary.

Start your response with the heading **Compacted Turn Context:** and then summarize the prefix so the retained suffix keeps its request context:

## Original Request
[What did the user ask for in this turn?]

## Early Progress
- [Key decisions and work done in the prefix]

## Context for Suffix
- [Information needed to understand the retained recent work]

Be concise. Focus on what is needed to understand the kept suffix.`

The core diff: three touches in compaction.ts.

splitTurn records where the cut turn began:

return {
  start,
  id: input.messages[start]!.info.id,
  prefixStart: input.turn.start,
} satisfies Tail

select exposes the prefix when the cut was mid-turn:

return {
  head: input.messages.slice(0, keep.start),
  turnPrefix:
    keep.prefixStart !== undefined ? input.messages.slice(keep.prefixStart, keep.start) : undefined,
  tail_start_id: keep.id,
}

processCompaction runs the second summary call, best-effort:

if (result === "continue" && !compacting.prompt && selected.turnPrefix?.length) {
  const prefixConversation = selected.turnPrefix.map(serialize).filter(Boolean).join("\n\n")
  if (prefixConversation) {
    const finish = processor.message.finish
    const prefixResult = yield* processor.process({
      user: userMessage,
      agent,
      sessionID: input.sessionID,
      tools: {},
      system: [],
      messages: [
        {
          role: "user",
          content: [
            {
              type: "text",
              text: `<conversation>\n${prefixConversation}\n</conversation>\n\n${TURN_PREFIX_SUMMARIZATION_PROMPT}`,
            },
          ],
        },
      ],
      model,
    })
    if (prefixResult === "compact") {
      yield* Effect.logWarning("turn context summary skipped: context overflow")
    }
    if (processor.message.error) {
      const error = processor.message.error
      processor.message.error = undefined
      processor.message.finish = finish
      yield* session.updateMessage(processor.message)
      yield* Effect.logWarning("turn context summary failed; keeping main summary", {
        "session.id": input.sessionID,
        error: errorMessage(error),
      })
    }
  }
}

Before and after

Real artifact from a probe run (retry-logic trajectory). Before: the summary ends at the main summary. After: the same main summary plus the appended turn-context section.

 ## Relevant Files
 - `.github/workflows/ci.yml`: CI workflow definition.
 - `src/api/client.ts`: `ApiClient` class needing retry logic.
 - `src/legacy/`: do NOT modify (explicit user constraint).
+
+**Compacted Turn Context:**
+
+## Original Request
+- Add retry logic to the API client so flaky network calls are retried.
+- **IMPORTANT constraint:** never touch `src/legacy/`.
+
+## Early Progress
+- Read `src/api/client.ts`: found minimal `ApiClient` class.
+- Ran `npm test`: **2 failures** (`test_user_auth`, `test_rate_limit`).
+
+## Context for Suffix
+- The `request(path)` method on `ApiClient` is the target for retry logic.
+- Must respect the `src/legacy/` exclusion constraint.

Why it matters (weak summarizer, 80K head): without the section, the old artifact's free-form recall of the request was the first thing to decay, at 0.50 containment for 30K-80K heads and 0.67 judge fidelity across the drift cycles, with the request facts buried in a generic Objective line. With the section, drift-cycle judge fidelity rose to 0.96 and artifact/continuation retention to 1.00 (80K drift row).

Feature tests

Contract:

• A split turn triggers a second call that summarizes only the prefix.
• The second call's prompt contains the verbatim request, never the tail.
• A boundary turn gets no second call.
• A failed second call leaves the main summary intact.
• filterCompacted and tail_start_id are unchanged.

Part 2: The Benchmarking

The harness

script/compaction-probe.ts via bun run bench. Usage and flags: README.md. Raw artifacts, score files, and run logs: docs/artifacts/.

• Fixtures: three deterministic trajectories (retry-logic, yaml-migration, session-store-fix) with planted facts: request fragments, a buried constraint, file paths, a continuation anchor. The last turn is too big to keep, forcing a mid-turn split (tail_turns: 1, preserve_recent_tokens: 300).
• Real path: real Session service plus SessionCompaction.process in a temp instance dir, real configured provider.
• A/B: new keeps the turn-context section; old empties it after each compaction, so later drift cycles read the summary exactly like pre-fix.
• Probes: yes/no membership (request, constraint, artifacts, negative distractor), free-form recall scored by fragment containment plus a fidelity judge (0-3), continuation judged 0-5 (3-sample median).
• Sweep: --head-tokens N inserts deterministic, fact-disjoint filler turns before the split turn, up to ~300K.
• Scenarios: --scenario split-turn|long-turn|mid-cut|all. split-turn (default) is the baseline bottom cut. long-turn inflates the last turn into a realistic agentic tool loop (--turn-calls N, default 60) so the second summary call absorbs a large prefix. mid-cut lands the cut mid-conversation: original task in turn 1, inflated fact-bearing split turn, recent turns retained whole via a larger preserve budget (--recent-turns N, default 5). It adds session-level original-task probes (orig-recall, orig-recall-judge) and records tc, whether the section was actually produced.
• Drift: 3 compact-advance-recompact cycles per variant.
• Sensitivity control: probes the raw retained tail (facts mostly absent) and a redacted artifact (probes must detect the absence).
• Sanity checks: warns when the split does not fire, the section is missing, or compaction returns non-continue.

Probe sensitivity

Raw numbers: docs/artifacts/verify-sensitivity.log. Absence detection works where the fact is truly absent from the tail: constraint scores 0.00 on both controls for retry-logic and yaml-migration, artifacts 0.00 on the redacted control, recall 0.00-0.50 on the tail-only control. The probes legitimately say yes where the fact genuinely sits in the tail: artifacts score 1.00 on tail-only (tool calls carry the file paths), and session-store-fix's constraint src/session/store.ts is itself a tail file path (1.00 on both controls). Caveats: request-recall membership is leaky (the model infers from related content, 0.50-1.00 with the strings gone), redaction is leaky too (recall 1.00 on the redacted artifact), and the judge is lenient (continuation 0.80, recall-judge 0.67 on the absent-request tail-only control). Containment recall is the most reliable free-form signal.

Results: strong summarizer

deepseek-v4-flash, sweep at 10K-300K (retry-logic, cycles 1), plus 300K and 50K drift across all three trajectories, both variants. Raw runs: docs/artifacts/deepseek-*.jsonl.

head         request-recall  constraint  artifacts  continuation   recall     recall-judge  negative
10K          1.00 / 1.00     1.00/1.00   1.00/1.00  1.00/1.00      1.00/1.00  1.00/1.00     1.00/1.00
30K          1.00 / 1.00     1.00/1.00   1.00/1.00  0.80/1.00      1.00/1.00  1.00/1.00     1.00/1.00
50K          1.00 / 1.00     1.00/1.00   1.00/1.00  1.00/1.00      1.00/1.00  1.00/1.00     1.00/1.00
150K         1.00 / 1.00     1.00/1.00   1.00/1.00  1.00/1.00      1.00/1.00  0.67/1.00     1.00/1.00
300K (all 3) 1.00 / 1.00     1.00/1.00   1.00/1.00  1.00/1.00      1.00/0.83  1.00/1.00     1.00/1.00
50K drift    1.00 / 1.00     1.00/1.00   1.00/1.00  1.00/1.00      0.89/0.83  1.00/1.00     1.00/1.00
(all 3)
(new / old)

The main summary quotes the request verbatim at every size (artifact inspection; the controls score low, so this is not probe leakiness). Why: the prefix is the most recent input the summarizer sees, and the 1M context means no truncation. Deltas run in both directions and sit within probe noise (150K recall-judge 0.67/1.00, 300K recall 1.00/0.83). The only consistent scale effect is exact-phrase paraphrase loss in free-form recall, and it is sample noise, not variant-specific (session-store-fix at 50K drift: 0.00/0.50 in one sample, 1.00/1.00 in the re-run deepseek-50k-drift-ssf-verbose.log).

Results: weak summarizer

Gemma 4 31B via OpenRouter to Novita, DeepSeek judge, sweep plus 80K drift (all three trajectories, 3 cycles). Raw runs: docs/artifacts/compaction-probe-gemma4-*.jsonl.

head         request-recall  constraint  artifacts  continuation   recall     recall-judge
30K          1.00 / 1.00     1.00/1.00   1.00/1.00  1.00/1.00      1.00/0.50  1.00/1.00
80K          1.00 / 1.00     1.00/1.00   1.00/1.00  1.00/1.00      0.50/0.50  1.00/0.33
150K         1.00 / 1.00     1.00/1.00   1.00/1.00  1.00/1.00      1.00/0.50  1.00/0.33
80K drift    1.00 / 1.00     1.00/1.00   1.00/0.89  1.00/0.93      0.67/0.61  0.96/0.67
(new / old; membership and negative stay 1.00/1.00)

The weak summarizer's main summary loses free-form request fidelity at 80K and up (recall-judge 0.33; one trajectory's old variant also dropped the artifact path and continuation at cycle 0). The turn-context section holds: a dedicated summary of just the prefix keeps the request capturable when the main summary paraphrases it away. Consistent across trajectories, probes, and cycles; the effect lives in the free-form/judge dimension, not membership.

Results: qwen3.8-27b

qwen/qwen3.8-27b via OpenRouter, DeepSeek judge, 1M context, sweep (retry-logic, cycles 1) plus 80K drift across all three trajectories, 3 cycles. Raw runs: docs/artifacts/qwen-*.jsonl.

head         request-recall  constraint  artifacts  continuation   recall     recall-judge
30K          1.00 / 1.00     1.00/1.00   1.00/1.00  1.00/1.00      1.00/1.00  1.00/1.00
80K          1.00 / 1.00     1.00/1.00   1.00/1.00  1.00/1.00      1.00/1.00  1.00/1.00
150K         1.00 / 1.00     1.00/1.00   1.00/1.00  1.00/1.00      1.00/1.00  1.00/1.00
80K drift    1.00 / 1.00     1.00/1.00   1.00/1.00  0.98/0.93      0.94/0.78  1.00/1.00
(all 3)
(new / old; negative 1.00/1.00)

qwen3.8-27b behaves like a strong summarizer despite being 27B: membership stays 1.00/1.00 and recall-judge 1.00/1.00 at every head size, with only the usual exact-phrase containment dip in the drift (old 0.78 vs new 0.94). Heads at 200K and above could not be measured: the fixture's per-message overhead (1600+ messages) pushes the real input past the 1M estimate (ContextOverflowError, see docs/artifacts/qwen-200k.log), a fixture limit, not a model property.

Results: gemma-4-26b-a4b

google/gemma-4-26b-a4b-it via OpenRouter (MoE, 4B active), DeepSeek judge, 262K context, sweep (retry-logic, cycles 1) plus 80K drift across all three trajectories, 3 cycles. Raw runs: docs/artifacts/gemma4-26b-*.jsonl.

head         request-recall  constraint  artifacts  continuation   recall     recall-judge
30K          1.00 / 1.00     1.00/1.00   1.00/1.00  1.00/1.00      0.50/0.50  1.00/1.00
80K          1.00 / 1.00     1.00/1.00   1.00/1.00  1.00/1.00      0.50/0.50  1.00/1.00
150K         1.00 / 0.50     1.00/1.00   1.00/1.00  1.00/1.00      0.50/0.50  1.00/1.00
80K drift    1.00 / 0.94     1.00/1.00   1.00/1.00  1.00/1.00      0.50/0.56  0.93/0.85
(all 3)
(new / old; negative 1.00/1.00)

Between gemma-4-31b and qwen: membership stays at ceiling except a request-recall dip in old (0.94 drift, 0.50 at 150K), and the model paraphrases harder than the others (recall containment 0.50 for both variants at every size). The judge mildly favors new (0.93 vs 0.85 drift).

Scenarios beyond the baseline cut

The section's value is bottom-cut-specific: it reframes what the resumed model attends to, it does not restore facts the main summary lost, and at a mid-conversation cut even the section is as lossy as its own summarizer.

Validated against all four summarizers, deepseek judge, retry-logic, 1 cycle. Raw runs: docs/artifacts/validate-*.jsonl.

long-turn: the baseline result survives agentic scale. Bottom cut on one inflated turn (60 tool calls, ~10.7K tokens, 70 messages). The section fired with no overflow skip (tc=1); membership and continuation stay at ceiling for both variants on every model. The only decay is the old-variant free-form judge dip, recall-judge 0.33 to 0.67 (gemma-4-31b at 0.33), where the missing section leaves the request to the main summary's paraphrase.

model         request-recall  constraint  artifacts  continuation  recall     recall-judge
deepseek      1.00 / 1.00     1.00/1.00   1.00/1.00  1.00/1.00      0.50/0.50  1.00 / 0.67
gemma-4-31b   1.00 / 1.00     1.00/1.00   1.00/1.00  1.00/1.00      0.50/1.00  1.00 / 0.33
qwen3.8-27b   1.00 / 1.00     1.00/1.00   1.00/1.00  1.00/1.00      1.00/1.00  1.00 / 0.67
gemma-4-26b   1.00 / 1.00     1.00/1.00   1.00/1.00  1.00/1.00      1.00/0.50  1.00 / 0.67
(new / old)

mid-cut: the cut moves mid-conversation and the section stops being a rescue. Original task in turn 1, fact-bearing turn inflated to ~10K tokens, five recent turns retained whole, ~13.8K tokens, 96 messages.

model         request-recall  constraint  artifacts  continuation  recall  recall-judge  orig-recall  orig-recall-judge
deepseek      1.00 / 1.00     1.00/1.00   1.00/0.00  1.00/1.00      1.00/1.00  1.00/1.00      0.00 / 1.00   0.00 / 0.67
gemma-4-31b   1.00 / 1.00     1.00/1.00   0.00/0.00  0.80/1.00      1.00/1.00  1.00/0.67      0.50 / 0.50   0.33 / 0.33
qwen3.8-27b   1.00 / 1.00     1.00/1.00   0.00/0.00  1.00/1.00      1.00/1.00  1.00/1.00      0.00 / 0.50   0.00 / 1.00
gemma-4-26b   1.00 / 1.00     1.00/1.00   0.00/0.00  1.00/1.00      1.00/1.00  1.00/0.67      0.00 / 0.50   0.00 / 0.00
(new / old)

The section fires for every model, but carries nothing new. tc=1 across the board, no overflow skips. The exact artifact path src/api/client.ts is genuinely absent from every artifact. Verbose runs prove it: the deepseek section itself reports "no api client file was read or edited in the prefix", the gemma section names no path.

At mid-cut the section is as lossy as the main summary, because it is the same summarizer doing the same compression. The bottom cut wins because the prefix is small and recent, so even a weak summarizer reproduces the request. At a ~10K-token prefix of mostly tool noise, the second summary call has nothing extra to work with. The one deepseek artifacts sample of 1.00/0.00 is fuzzy-membership noise, its verbose rerun reads 0.00/0.00, and the sensitivity controls already show the membership probe answers from task emphasis, not literal paths.

The section reframes attention, it does not add content. Both variants carry byte-identical main summaries (additive A/B), and the original task ("set up CI, node 20") rides the main summary in both, the old-variant session-level answers name it. With the section present, even the session-level question is answered from the split-turn frame, orig-recall 0.00 in new for three of four models vs 0.50 to 1.00 in old, while continuation stays at ceiling (0.80 to 1.00). The orig-recall delta measures framing, not retention capacity.

Warnings. One trajectory, one cycle per cell, magnitudes are single samples, though the directions are consistent across all four models. The reliable signals are containment recall, the judges, and tc.

Mid-cut drift: does the original task decay?

3 compact-advance-recompact cycles on the weak summarizer (gemma-4-31b, all three trajectories, deepseek judge; raw runs docs/artifacts/validate-*-mid-cut-drift*.jsonl), mean across trajectories:

cycle  request-recall  constraint  artifacts  continuation  recall  recall-judge  orig-recall  orig-recall-judge  tc
new 0   1.00           1.00        0.67       0.87          0.83    1.00          0.00         0.00               1.00
new 1   1.00           1.00        0.67       0.87          0.83    1.00          0.33         0.22               1.00
new 2   1.00           1.00        0.67       0.73          0.83    1.00          0.33         0.11               1.00
old 0   1.00           1.00        0.67       1.00          1.00    0.78          0.50         0.44               -
old 1   1.00           1.00        0.67       1.00          1.00    0.78          0.50         0.33               -
old 2   1.00           1.00        0.67       0.93          1.00    0.78          0.50         0.44               -

The section survives the recompaction chain, tc=1 at every cycle. On the second cycle the walk-back can no longer split a turn, the split turn's user message is already summarized away, so no new second call fires. Instead previousSummary carries the section text into the next main summary and the re-summarizer re-embeds it, so new differs from old at every cycle, not just cycle 0.

No runaway decay in either variant. The original task (turn 1) stays at partial retention (orig-recall 0.33-0.50, judge 0.11-0.44 across cycles) and the split-turn request stays judge-capturable for new (recall-judge 1.00 vs 0.78, flat). At three cycles the chain is stable, not degrading.

The framing penalty persists but does not compound. new's session-level answers stay split-turn-framed on two of three trajectories (retry-logic, yaml-migration: orig-recall 0.00 every cycle) and recover only on session-store-fix, whose original task overlaps the retained tail's own file paths. old blends both tasks throughout (0.50). Continuation drifts mildly for new at cycle 2 (0.73 vs 0.93, one 0.40 sample).

The deepseek control holds everything (retry-logic, 3 cycles): section chain present, original task rides old at 1.00 across cycles, and new's framing reads 0.00 at cycles 0-1 before recovering to 1.00 at cycle 2.

The section is durable and judge-visible, but the main-summary chain carries the original task. At three cycles the chain neither collapses nor recovers it, and separating the variants on the original-task axis needs a longer chain (more cycles, larger heads), where the framing penalty should compound into measurably worse retention for new. The harness's own next lever is a durable session-task anchor inside the main summary, which drift can measure directly and more turn-context sections cannot provide.

Bottom line

Redundant for strong, high-context summarizers (deepseek-v4-flash, qwen3.8-27b) and measurably valuable for weak ones (Gemma 4 31B at 80K heads and up). Model size alone does not predict the value: the 27B qwen retains like a strong model, the 31B gemma does not, and the 26B MoE gemma sits between them with a mild old-variant dip. Membership probes cannot see this; the discriminating measurement is free-form recall judged by a strong model.

Methodology attribution

The evaluation design adapts established context-compaction evaluation work:

• CompactBench v0.1.0: "measures what survives when you replace conversation history with a compacted artifact."
  • Probe-based evaluation: probes answer from the compaction artifact alone, never the raw conversation.
  • Membership probes: the yes/no questions (request, constraint, artifact files, negative distractor) follow CompactBench's probe discipline.
  • Multi-cycle drift: compact-advance-recompact cycles, with decay across a compaction chain as a first-class metric.
  • Adversarial templates: planted constraints and absent distractors follow the buried_constraint, decision_override, and entity_confusion template families and the "hidden ranked set" discipline.
  • Sensitivity controls: probing the raw retained tail and an artifact with facts redacted enforces the requirement that a probe which cannot detect absence cannot measure retention.
• Factory.ai (Evaluating Context Compression for AI Agents, summarized in Zylos Research, Agent Context Compaction for Long-Running Sessions):
  • Probe taxonomy: recall, artifact tracking (their weakest and most discriminating metric), continuation, and decision probes, scored by an LLM judge. This harness mirrors it as request recall, artifact tracking, continuation, and buried constraint, which takes the decision role.
  • Judging scales: the continuation probe judged 0-5 is Factory.ai's continuity dimension; the 0-3 recall-fidelity judge adapts their judge-scored dimensions.
  • Chained-compaction concern: re-compaction chains compound error, which motivates drift as a first-class metric.
  • CI recommendation: probe-based evaluation in CI over full benchmark suites.
• Harness-specific design:
  • Additive A/B: the mechanism appends one text part, so the pre-fix artifact is exactly the same artifact with that part emptied; both variants share identical compaction runs.
  • Fact-disjoint filler sweep (--head-tokens): scales context size without disturbing planted facts.
  • Probe-model separation (--probe-model): a weak summarizer is measured by a strong judge.