Skip to content

feat: keep output that is usually read whole in view - #12

Merged
hardy30894 merged 3 commits into
mainfrom
feat-reopen-gate
Oct 23, 2026
Merged

hardy30894 merged 3 commits into
mainfrom
feat-reopen-gate

Conversation

@hardy30894

Copy link
Copy Markdown
Member

Stacked on #11 (the eval harness); review that first.

Why

On 1,491 recorded Claude Code sessions, how often the model needs an offloaded block back depends mostly on what produced it and how big it is:

class under 500 tokens 500 to 2k 2k to 8k
Bash:grep | 15% 38% 66%
Bash:sed 15% 48% 78%
Bash:cat 26% 52% 87%
Read 65% 79%

Every reopen costs an extra model turn, so offloading a large cat buys a turn and saves little.

What

  • A reopen policy (crates/optimize/src/policy.rs): Laplace-smoothed rates per command class and size band, falling back to the class, then to a prior. A block may be offloaded only when its predicted rate is under the threshold, 0.4. Otherwise the optimizer runs with offloading off, so only inline codecs, which never need a reopen, apply.
  • MCP optimize takes optional tool and command; exec uses its own command. Both classify with the same function eval uses.
  • secondwind eval --learn fits the rates on the even-numbered sessions, replays the odd ones with and without the gate, prints both, and writes ~/.secondwind/policy.json. eval --gate replays with the active policy. The built-in table holds the classes with at least 300 offloads in the corpus above.
  • The Claude Code mod passes the command with each block and compresses from 512 bytes instead of 6 KB.
  • The query description no longer claims it helps text output.

Results on held-out sessions (learned on the other half)

context saved reopen rate reopens per session unfaithful
no gate 24.8% 42.8% 23.6 0
learned gate 10.5% 22.1% 4.8 0

For the mod, Bash only: the old 6 KB rule saved 6.6% of tool output with 5.9 reopens per session; the gate at 512 bytes saves 9.7% with 4.8, and cuts billed cost by about 21% versus 5%.

Tests

Policy learning, backoff, round trip and refusal of a bad file; the gate in evaluate; end to end through the binary: a large cat stays in view over MCP and exec while the same block unnamed offloads, a learned policy replaces the built-in one, and eval --learn writes a policy and reports held-out sessions. Mod tests: the command reaches the binary, output under 512 bytes never does. cargo test --workspace: 363 passed. Clippy clean.

@hardy30894
hardy30894 added this pull request to stack #15 October 23, 2026 02:36
Base automatically changed from feat-eval-harness to main October 23, 2026 02:36
On 1,491 recorded sessions a block is reopened 15% of the time under 500 tokens and up to 87% at a few thousand, and every reopen costs an extra model turn. A reopen policy, per command class and size, now decides whether a block may be offloaded; when it may not, only inline codecs apply. MCP optimize takes optional tool and command arguments and exec uses its own command. The built-in rates come from that corpus; eval --learn fits rates on half the user's sessions, replays the other half with the gate, and writes ~/.secondwind/policy.json. On held-out sessions the learned gate cut reopens per session from 23.6 to 4.8 with no unfaithful block. The optimize description no longer claims query helps text.
…12 bytes

With the binary's reopen gate deciding what stays in view, the 6 KB floor only skipped savings: on held-out sessions it kept 13% of what the gate saves, and 512 bytes keeps 99.9%. A call to the binary takes about 30 ms.
Each state is drawn on the terminal and desktop surfaces: off offers to turn on, an empty session says where output will land, a net gain shows in the accent color and a net loss in red.
@hardy30894
hardy30894 merged commit 0c1a05d into main Oct 23, 2026
3 checks passed
@hardy30894
hardy30894 deleted the feat-reopen-gate branch October 23, 2026 02:39
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant