Rewrite the compaction summary as a handoff with fold rules - #68
Conversation
Codex Review SummaryThis comment shows the latest Codex review activity on this pull request.
ℹ️ About Codex in GitHubYour team has set up Codex to review pull requests in this repo. Reviews are triggered when you
Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings. |
Codecov Report✅ All modified and coverable lines are covered by tests. Additional details and impacted files@@ Coverage Diff @@
## main #68 +/- ##
=======================================
Coverage 94.37% 94.37%
=======================================
Files 93 93
Lines 7697 7701 +4
Branches 1604 1604
=======================================
+ Hits 7264 7268 +4
Misses 101 101
Partials 332 332 ☔ View full report in Codecov by Harness. 🚀 New features to boost your workflow:
|
![chatgpt-codex-connector[bot]](https://avatars.githubusercontent.com/in/1144995?s=60&v=4){kind=small}
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: d762783305
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
| - Start from the previous summary and keep its lines verbatim, including every step | ||
| and its figures, except where the new messages change or finish something. Do not | ||
| rewrite, merge, or compress what you carry forward. |
There was a problem hiding this comment.
When a previous summary is already near 1,000 words and a later fold adds a request or completed step without superseding anything, these rules require every existing line to remain verbatim while LENGTH_RULE still requires the result to stay under 1,000 words and forbids cutting requests, constraints, or exact values. No output can satisfy both requirements, so the summarizer must either discard required state or let the checkpoint grow across folds until it stops shrinking context or hits the provider output limit and is rejected. Permit older completed progress and narrative to be consolidated when needed to preserve the fixed bound.
Useful? React with 👍 / 👎.
| SUMMARY_PREFACE = "The earlier part of this conversation was compacted. Another model " \ | ||
| "wrote the handoff summary below from the full transcript; the " \ | ||
| "messages after it are the most recent turns, kept intact. Build on the " \ | ||
| "work already done, do not repeat it, and treat the user's most recent " \ |
There was a problem hiding this comment.
For every compaction, Compactor.call sends the summarizer only head, the messages before the retained cut; on subsequent compactions it sends the prior lossy summary plus part of the replay rather than the durable original transcript. The synthetic preface therefore gives the resumed model a false completeness guarantee by saying the handoff was written from the full transcript, even though unsummarized recent turns merely follow it and previously omitted details cannot be recovered. Describe the summary as covering the earlier compacted portion instead.
Useful? React with 👍 / 👎.
| Write the handoff summary for another model that will resume this work with only your | ||
| summary and the last few turns. It must be able to continue without asking the user to | ||
| repeat anything. Write these sections in this order, with these exact headings: | ||
|
|
||
| #{FORMAT} | ||
| #{SECTIONS} |
There was a problem hiding this comment.
This substantially expands every compaction request with the seven-section specification and fold rules, increasing provider input processing and potentially summary-generation latency, but neither the CHANGELOG nor the compaction documentation states that latency impact. Add the required latency assessment so hosts can evaluate the cost of this hot-path request change.
AGENTS.md reference: AGENTS.md:L18-L19
Useful? React with 👍 / 👎.
| digests.call(candidate).filter_map do |workload, digest| | ||
| workload if before.key?(workload) && before[workload] != digest | ||
| end |
There was a problem hiding this comment.
When a digest differs, this now returns a two-element [scenario, size] array, but coverage still interpolates the collection with changed.join(", "); Ruby recursively joins the nested arrays, so one changed workload renders as account_research, M and several render as an ambiguous flat sequence such as account_research, M, ledger_export, S. The comparison correctly excludes the cells, but its diagnostic no longer identifies which scenario/size pairs need a new baseline; format each pair explicitly (for example, scenario/size).
Useful? React with 👍 / 👎.
| your summary and the last few turns, using the same seven sections and headings as the | ||
| previous summary: | ||
|
|
||
| #{HEADINGS} |
There was a problem hiding this comment.
On a second or later checkpoint, the summarizer receives only the seven heading names here, not the definitions in SECTIONS. The fold rules say how to retain or replace existing lines but never tell it to add newly introduced constraints, decisions, unresolved failures, or exact identifiers from the new transcript, nor do they preserve the requirements for attribution and exact error text. Consequently, a user constraint or tool-produced ID first appearing after the initial checkpoint can be omitted even when the model follows the update instructions; include the full section definitions, or equivalent addition rules, in UPDATE_PROMPT.
Useful? React with 👍 / 👎.
# Conflicts: # CHANGELOG.md
Compaction eval comparisonbaseline 6ed9d01 (prompt 558eb0bcb2d4) vs candidate 55eddb0 (prompt 80fd19384428) Baseline cells the candidate did not run (not compared): claude-sonnet-5/account_research/S/full, claude-sonnet-5/account_research/S/tail, claude-sonnet-5/incident_debugging/S/full, claude-sonnet-5/incident_debugging/S/tail, claude-sonnet-5/ledger_export/S/full, claude-sonnet-5/ledger_export/S/tail, claude-sonnet-5/account_research/M/compacted, claude-sonnet-5/incident_debugging/M/compacted, claude-sonnet-5/ledger_export/M/compacted, gpt-5.6-sol/account_research/S/full, gpt-5.6-sol/account_research/S/tail, gpt-5.6-sol/incident_debugging/S/full, gpt-5.6-sol/incident_debugging/S/tail, gpt-5.6-sol/ledger_export/S/full, gpt-5.6-sol/ledger_export/S/tail, gpt-5.6-sol/account_research/M/compacted, gpt-5.6-sol/incident_debugging/M/compacted, gpt-5.6-sol/ledger_export/M/compacted, gemini-2.5-flash/account_research/S/full, gemini-2.5-flash/account_research/S/tail, gemini-2.5-flash/incident_debugging/S/full, gemini-2.5-flash/incident_debugging/S/tail, gemini-2.5-flash/ledger_export/S/full, gemini-2.5-flash/ledger_export/S/tail, gemini-2.5-flash/account_research/M/compacted, gemini-2.5-flash/incident_debugging/M/compacted, gemini-2.5-flash/ledger_export/M/compacted. Per model, matched compacted cells
Per cell
Passed: no model loses more than 3 points of aggregate probe accuracy or rejects more compactions, over matched cells. |
This branch was successfully deployed
1 participant
The compaction summary was a checklist template: goal, constraints, done and in-progress boxes, decisions, next steps, and a "critical context" bucket. It kept facts by narrating every step, which made summaries long on routine work and thin on the things a resuming model actually needs, and it had no rule for what happens when a later checkpoint folds new messages into an earlier summary.
The summary is now a handoff in seven fixed sections: the goal and the requests that shaped the work, quoted in the user's words and marked done or open; constraints; exact facts and references, current values only; decisions with reasons; progress as one line per step with its result and figures; open failures; and the current work with the single next step. The summarizer is told it is the record, not a reviewer: it reports what the user, the assistant, and the tools said as theirs and does not re-check, discount, or qualify it. Later checkpoints start from the previous summary and keep its lines verbatim, changing only what the new messages change or finish, so a fold cannot quietly rewrite earlier work. The replay preface presents the result as a handoff written by another model from the full transcript. The reply is the summary; nothing is extracted from it.
Measured with the compaction eval against the committed baseline, matched compacted cells at size S, the CI tier, two repetitions per cell:
The eval comparison also had a bug: scenario digests are per size, but the comparison keyed them by scenario name, so a baseline holding S and M rows read a candidate that had only finished S as three changed scenarios and found nothing comparable. It now tracks digests per scenario and size.
Verified with the full suite on Ruby 3.2 and 3.3, RuboCop, and the eval runs above.