Repository navigation
SWF(bug): restore compressed group replies in history - #1841
Conversation
Read unambiguous matches from the migrated group ledger instead of showing a torn save-token preview as a Bot reply.
|
I'll fix CI failures and address comments from users with write access that start with 'Devin'.
|
Codex Review Summary: ✅ CompletedThis comment shows the latest Codex review activity on this pull request.
ℹ️ About Codex in GitHubYour team has set up Codex to review pull requests in this repo. Reviews are triggered when you
Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings. |
Anchor the missing span to intact group ledger text on both sides of the cut, as in the reported preview.
Fei2411
left a comment
There was a problem hiding this comment.
Review of 158176ba (comments only; lulu makes the merge call).
Verdict: one fix needed before merge.
Needs fixing: truncated migrated replies are restored as complete (confirms the Codex P2)
The group- ledger importer (ds-bot/src/host/groups.js:422) drops length from a shortened room.log line before it tries group-lines/<room>/<seq>.txt. If that file is gone, the record keeps only the first 2,000 chars and has no length, so isKeptReply (:637) treats it as complete. In a window that spans several replies, that record can then fill the middle of the recovered span, and the client drops the "result was shortened" state.
Reproduced locally on this head: a 3-reply window (b, a, b), middle a reply kept short in room.log with its group-lines file removed. groupRelayReplies returned all three, with the middle one at 40 of 350 chars and no sign it was cut. Expected: null (fall back to today's shortened preview), or that one reply marked shortened.
Fix suggestion: keep length on the imported record when the file read fails (e.g. only drop it when text came from the file), so the existing record.text.length === record.length check rejects it; add that case to the new test. Note this changes what the ledger stores for those records, so check other readers of length (the group timeline already handles shortened lines from the live path).
Checked and fine
- Right replies / order / no duplicates: recovered replies match the ledger in ledger order for the PR's tests and my probes (4-reply window with a
workingrow, broken-head invalid-JSON window). A second round that repeats the same reply text does not get picked up twice or out of order (returnsnullrather than guessing). - No other group's data: lookup is per room via
roomOf(sessionId)/pastOwners; another room's session returnsnull(covered by the PR test). - Newer turns / no ledger: recovery only runs when a save-token notice is present and the preview has an elision marker, and only on records before the first ledger
anchor; groups with nogroup-book returnnulland render exactly as before. - Save-token format: notice and
...[+N chars]...regexes matchdsh-plugin-save-token@2.4.1(bothcompressedandlosslessly re-encoded, line and string elision). - Local checks on this head:
npm test,node scripts/build-client.mjs --check(rebuiltclient.jsis in the diff),npm run lintall pass. - Host smoke test: the PR leaves it unchecked although
src/host/changed. I ran it on this head (isolated.dsh-test-smoke, DSH 0.2.2-rc.1): all 52 steps passed. Please tick it in the description. - CI: 8 passed, 0 failed; Windows Node 24 test job, desktop packages and iOS simulator build still pending at review time. Devin Review: no issues found.
- PR rules: branch into
dev, up to date withdev,Fixes #852, changelog field filled.
Overlaps with open PRs
- #1055 (Agent presets): the only PR this one adds a new conflict to: one line, the
READSset inhost/channel.js('group-relay-replies'vs'presets'); keep both. Land #1841 first; #1055 resolves it when it merges dev. - #1074 (files under a member's group message) touches the same
shared/group-replies.js,client/cells.js,host/parts.js,host/groups.js; it already conflicts withdevin those files, and #1841 adds no new conflict. Whoever updates #1074 after this lands should keepparseRepliesdropping save-token-elided objects. - #888, #1381, #1201, #524, #736 touch
host/groups.js/parts.js/channel.jsbut already conflict withdev; #1841 adds nothing on top. #733, #1279 merge cleanly.
Keep the original length during import and treat old untagged rows at the former clipping limits as uncertain.
Keep original lengths for complete imports too, so a full reply at an old clipping limit remains recoverable.
Use the omitted length for a single trimmed reply without taking a suffix as proof of the source round.
Handle raw line splices before considering an exact single-reply match; use structural encoding only for its documented fragment shape.
Permit matching endpoints on one complete ledger row only when the elided JSON string length matches exactly.
Use raw fragments for the omitted-length calculation so re-encoding a lone surrogate does not inflate the visible length.
Compare the saved JSON suffix directly so a cut inside a backslash escape cannot point to another reply.
An apparent unique match is not enough when the first part of the log or an intervening sequence is missing.
|
Follow-up to Code Reaper’s truncated-reply finding: c071aaa4 preserves the original length and leaves the shortened preview in place when full text is missing. The regression covers both fresh imports and previously migrated rows; both preserved save-token fixtures also passed the Chrome reload retest. |
Fei2411
left a comment
There was a problem hiding this comment.
Re-review at 792cfa82: ready for lulu. My earlier finding is fixed, and I found nothing new.
The earlier finding is fixed. The importer now keeps length on every imported line, so a cut line whose group-lines file is missing fails hasCompleteText. Old migrated ledgers that already lost length are caught by the 2,000 / 20,000 cap check. Those caps have not changed since cc4fdb9, so the check is sound. The new tests cover both the room-log case and the already-migrated case.
The later commits look correct:
- JSON-codec cuts must now be exactly 220 / 80 with a matching total length. Line-codec cuts must be 260 / 100.
- A single-reply cut must match its encoded length exactly.
- The tail is matched on the encoded text, so a cut inside an escape still works.
- Recovery returns
nullunless the pre-anchor ledger holds every seq from 1 with no gap. Unknown or ambiguous cases fall back to the saved preview.
Order, no duplicates and no data from other groups still hold. Newer ledger-backed turns and rooms with no ledger behave as before.
One scope note for lulu (not a blocker). ROOM_LOG_MAX was 60 and group-logs only started on 2026-11-08. A group with more than 60 lines before then has a gap in its ledger, so its old compressed turns keep the old preview. That is the safe choice, and the PR description already says this history cannot be fully recovered.
Checks:
npm test,build-client.mjs --checkandnpm run lintpass locally.- The host smoke test passes all 52 steps (isolated
.dsh-test-smoke, pinned DSH 0.2.2-rc.1). - CI: 16/16 checks pass. Devin Review passes.
- The PR rules are followed, including the rebuilt
client.js.
Overlap: none new. 1841 merges cleanly into dev. It should still land before #1055: both add an entry to the READS set in ds-bot/src/host/channel.js, a one-line conflict for #1055 to resolve.
There was a problem hiding this comment.
The migration retains the original reply lengths and restores from the ledger only when the matching reply sequence is unique and complete. The targeted regressions passed, including clipped and ambiguous matches. #888 changes pagination rather than legacy reply recovery.
# Conflicts: # ds-bot/src/host/channel.js
Restore legacy group replies from the migrated ledger when save-token has cut their saved tool result.
Issue
Refs #852
What changed / 改了什么
Current group turns already store a short ledger reference. Older turns stored replies inline: save-token can splice one Bot’s reply to another’s, and the old parser either accepted that as one reply or could not parse the shortened JSON. Distinguish save-token 2.4.1’s whole-line splice from per-reply JSON trimming, then use a unique match only when the pre-ledger log is intact. If earlier sequences are missing or the surviving text cannot distinguish the original, keep the saved preview instead. Recovery keeps the shortened fallback if the ledger only retained a clipped copy. This does not rewrite session history or open spill-file paths.
How I checked it / 怎么验证的
Chrome and DSH Web, real save-token 2.4.1: reloaded the same v0.2.6 round after upgrading to current
devand then to this branch. The stored 1,031-byte result was unchanged. Ondev, one mangled Chief row appeared; on the branch, all four 350-character replies matched the ledger exactly. The structural JSON fixture also restored four exact 1,000-character replies without changing its saved result. An existing ledger-backed turn reloaded correctly.Chrome reload recording
Try it:
cd ds-bot && node --import ./test/harness-resolve.mjs --test --test-name-pattern='compressed pre-ledger group replies' test/work-dir.test.js(failed on thedevbase with one spliced reply; passed on this branch with all three fixture replies). Regression tests also cover invalid JSON, split escapes and emoji, clipped replies, and a later lookalike after the original aged out of the log.npm test(inds-bot/)node scripts/build-client.mjs --checknpm run lintThe Windows UI and the invalid-JSON, split-escape, and expired-log cases were not tested in Chrome; those cases have unit coverage. Legacy history with missing ledger rows remains unrecoverable.
Changelog
Group history restores legacy replies from the ledger when the saved result was shortened and the original messages can be matched.
Checklist
client.js.Implemented in session Restore compressed long turns in group history
Review with Devin · Open in Devin Desktop