Repository navigation
Epic: UniiChat repo vs OptChat — compaction & caching review #202
Description
Victor Taelin open-sourced UniiChat and asked (on X) that ports check their compaction and caching against the real code. This is that review: UniiChat 0500bf6 vs pi-optchat aae1caf (v0.8.6), code reading only, three independent passes (view decay / Memory.ts, prompt caching, prompts + tools + subagents + import). Earlier gist-based deviations are in #101 (comment 6061608959).
Headline: the core matches. Merge order, 128→64 KB chat view, 32→16 KB compactions' view, saved batch state, 4-line blocks with the last whole one marked and the tail unmarked, single flight on a cold prefix, and nearly every prompt rule. Every deviation we had already measured (#93, #108, #112, #129, #146, #153, #179, #190) holds up. What's below is new.
Sub-issues
Adopt (no measurement needed):
- Give up on a node the compactor refuses instead of retrying it forever; cap leaf input #193 Give up on a refused/empty/over-length node instead of retrying every 10 s forever; cap leaf input. Biggest one.
- Persist the compactions' view in view.json #195 Persist the compactions' view in view.json (one cold prefix per
/reloadtoday). - zoom: return the built half when the other isn't summarized yet; list roots on a bad id #200 zoom: show the built half when the other isn't summarized yet; list roots on a bad id.
- Add a fake-Anthropic cache replay test asserting hit rate and ≤4 marks #198 Fake-Anthropic cache replay test (≥95% read, ≤4 marks), including turns that add >25 blocks.
Adopt, verify offline or with a cheap live check:
- Change the view only in whole batches; decouple the compactions' view; order builds by position #194 Change the view only in whole batches; decouple the compactions' view from the chat's; order builds by message position.
- Single flight for sibling subagent spawns, so N children write the view once #196 Single flight for sibling subagent spawns (N children currently write the view N times).
- Leave the compaction task unmarked (A/B) #197 Leave the compaction task unmarked (+12–19% compaction input today, ~2–5% of its cost; ~$1 A/B).
Needs an A/B before adopting:
- A/B Unii's "small, mechanical job, be fast" clause in the compactor prompt #199 "Small, mechanical job, be fast, spend few tokens" clause in
COMPACT. - Trim the [Historical …] header on imported messages (measure, then A/B) #201 Trim the
[Historical …]header on imported messages (count bytes first).
Housekeeping:
- Fix four stale rows in the Epic: adopt Victor's revised OptChat gist (cache fix, compactor, prompts) #101 comment (done, comment edited): the merge
<chat>rule (Unii's code now does the same, Memory.ts:187-188); "nozoom("Name")" (we havezoom("<run id>")); "images = text placeholder" (zoom returns images); and the "97% vs 58–76%" credit for the three view marks (that figure came from batch merging in Merge the view in batches, as Victor's revised recipe does, so turns read it from the cache #96; the marks' own evidence is the tool-turn run, 0% vs 85.7%).
Full deviation table
Unii refs are /tmp/UniiChat 0500bf6; ours src/ at aae1caf. "Deliberate" means tied to an earlier PR/A-B.
View decay (Memory.ts)
| Area | Unii | OptChat | Deliberate? | Rec. |
|---|---|---|---|---|
| Merge order | (t+1)/2^l−i (Memory.ts:917), oldest on ties |
age-from-last /2^l, oldest on ties (memory.ts:45-46). Same order |
match | keep |
| Trigger/target | 128→64 KB chat, 32→16 KB seen (:147-148, :845, :940) |
same (memory.ts:137-138, :74-75) | match | keep |
| Budget counts | text bytes only; real view overshoots 2–6% | labels, newline, frame count (memory.ts:57-58) | #179 | keep |
| When a batch applies | at message boundaries, only whole batches (:847, :438) | on every append and every node built (:168, :301, :72-81) | no | #194 |
| Compactions' view trigger | own 32 KB sawtooth (:844-852) | forced to 16 KB whenever chat merges (:238) | no | #194 |
| Compactions' view end | cut at first unbuilt (:735, :842) | placeholders for unbuilt lines (:316-319), primeFirst handles prefix |
#108 | keep |
| Compactions' view persisted | yes (:241, :214) | rebuilt on load (:155-156, :211) | no | #195 |
| Turn view | view to from, no mid-turn refresh (:460-471) |
snapshot + previous exchange + 30% refresh (index.ts:447-464) | #101, #146 | keep |
| JOBS | 64 per side (:143) | jobs 8, setting 1–64 |
#190 | keep |
| Build order | by position, merges first on ties, ≤64th unbuilt line (:632-657) | leaves first, merges FIFO ungated (:266-276) | no | #194 |
| Too-long retry | resend view+task+one note (:739-746) | appends each reply + tooLong (compactor.ts:95-97) |
no | low-priority A/B, no issue |
| Refusal / no line | first 512 B of source, done (:747-754) | throws, retried every 10 s forever (:256-261) | no | #193 |
| Shortest try kept | yes (:754) | yes + 640 B tolerance (compactor.ts:100) | #101 | keep |
| Long messages | paged on write at 30K (Clip.ts:13) | paged on read at 25K; leaf input uncapped (:285-289) | #93 (measured vs ~8 KB split, not 30K) | keep paging; cap in #193 |
| Imports | same memory, own JOBS (:597-612) | separate job (#133) | #133 | keep |
Prompt caching
| Area | Unii | OptChat | Deliberate? | Rec. |
|---|---|---|---|---|
| View marks | 1 view mark + system + msg end + top-level (Memory.ts:1005-1017; Anthropic.ts:72-80) | 0/20/40 blocks back + end; system/tools unmarked (cache.ts:22,45-66) | #96 (0% vs 85.7% after 50 tool calls) | keep; tell Taelin (his cache.test caps turns at 6 tool steps, :232, so it never sees this) |
| System mark | marked (Anthropic.ts:77) | none | mark budget | keep (<1%) |
| Compaction task | unmarked, no top-level (Memory.ts:734; Anthropic.ts:72) | end mark on task (compactor.ts:57,68) | no, side effect of #187 | #197 |
| Single-flight scope | every call, per-block keys (AskAI.ts:218-279) | compactor on Anthropic only (compactor.ts:59-62) | listed as follow-up | #196 |
| Warm TTL | refreshed on read, LRU (AskAI.ts:240-242) | 4 min from primer, not refreshed (compactor.ts:16,25,39) | no; latency only | optional 1-line |
share |
compactions use chat system (Server.ts:338-340) | own COMPACT (#112) |
#112 | keep: 0 bytes shared with turns on either side (tools + model differ) |
| OpenAI | explicit prompt_cache_breakpoint, explicit mode for compactions, 30-min single flight (OpenAI.ts:46-64) |
auto prefix + prompt_cache_key (compactor.ts:66), no gate |
#96 saw 400 on breakpoints | needs live re-check; tracked in #79 |
| Retries | 8× backoff + 60 s forever (Agent.ts:231-271) | 2× + 10 s memory retry (#174) | #174 | keep; both re-read the same prefix |
| Hit-rate test | fake-Anthropic replay ≥95% (cache.test.ts) | shape + reach tests only | no | #198 |
Prompts, tools, subagents, import
| Area | Unii | OptChat | Deliberate? | Rec. |
|---|---|---|---|---|
| Density | "use the space up to the limit" (prompt.txt:75) | "ceiling, not a target" | #129 | keep |
| Quoting the user | near-verbatim however short (:77-78) | reasons/corrections/questions verbatim; bare "sure" logged as the decision | #129 | keep |
| Tool/echo leaves | no <call> block |
raw <call> ≤4K; "say what the output shows" |
#153, #130 | keep |
| Speed clause | "small, mechanical job … be fast" (:92-96) | none | no | #199 |
Merge uses <chat> |
same two sentences (Memory.ts:187-188) | same | #101 row stale | fix #101 |
| Compactor system | shared prompt + user rules, Haiku default | COMPACT alone, Sonnet |
#112, Haiku study | keep |
| zoom unbuilt half | returns built half + placeholder (Memory.ts:1039) | throws (memory.ts:372) | no | #200 |
| zoom bad id | lists roots (:1031-1033) | "No line id+n." | no | #200 |
| zoom head | id+0| |
id+1| |
#147 | keep |
| Paging | 30K pages (Clip.ts:13) | 25K offset/limit (#93, #117) |
yes | keep |
| search | none | optional, off by default | #101 | keep |
| Images | 1568 px JPEG copy (Memory.ts:160) | ≤2048 px / 2 MB (images.ts:7) | images PR; #101 row stale | keep; 1568 optional |
| Subagents | flat, main only, one effort per batch (Tools.ts:200,336-341) | levels, per-task model/thinking, models(), tell_parent, windows |
#101, #185 | keep |
| Subagent context | whole log as one text block, never compacted (Tools.ts:580-590) | Pi session with compaction | by design | keep |
| Reports | [Name] … Full chat: zoom("Name") |
[id] … Full chat: zoom("<id>") |
#101 row stale | fix #101 |
| Import sources | Claude Code, Codex, claude.ai + ChatGPT web zips | Claude Code (+memories), Codex, Pi/OMP, ChatGPT | — | claude.ai web source only if asked |
| Import content | all assistant text; tools opt-in | user messages + final answers | import v1 | keep |
| Import framing | one note per session; ai: kind |
[Historical …] on every message (sources.ts:119) |
no | #201 |
| Import dedup/order/subagent skip | same | same | yes | keep |
Full reports: /tmp/unii-compare-view.md, /tmp/unii-compare-cache.md, /tmp/unii-compare-rest.md (local).
Activity
[-]Epic: UniiChat 0500bf6 vs OptChat — compaction & caching review[/-][+]Epic: UniiChat repo vs OptChat — compaction & caching review[/+]