Skip to content

SWF(bug): restore compressed group replies in history - #1841

Merged
Fei2411 merged 15 commits into
devfrom
fix/852-group-history
Nov 20, 2026
Merged

Fei2411 merged 15 commits into
devfrom
fix/852-group-history

Conversation

@Fei2411

@Fei2411 Fei2411 commented Nov 19, 2026 •

Copy link
Copy Markdown
Owner

Restore legacy group replies from the migrated ledger when save-token has cut their saved tool result.

Issue

Refs #852

What changed / 改了什么

Current group turns already store a short ledger reference. Older turns stored replies inline: save-token can splice one Bot’s reply to another’s, and the old parser either accepted that as one reply or could not parse the shortened JSON. Distinguish save-token 2.4.1’s whole-line splice from per-reply JSON trimming, then use a unique match only when the pre-ledger log is intact. If earlier sequences are missing or the surviving text cannot distinguish the original, keep the saved preview instead. Recovery keeps the shortened fallback if the ledger only retained a clipped copy. This does not rewrite session history or open spill-file paths.

How I checked it / 怎么验证的

Chrome and DSH Web, real save-token 2.4.1: reloaded the same v0.2.6 round after upgrading to current dev and then to this branch. The stored 1,031-byte result was unchanged. On dev, one mangled Chief row appeared; on the branch, all four 350-character replies matched the ledger exactly. The structural JSON fixture also restored four exact 1,000-character replies without changing its saved result. An existing ledger-backed turn reloaded correctly.

Same saved history: dev on the left; fix on the right

Chrome reload recording

Try it: cd ds-bot && node --import ./test/harness-resolve.mjs --test --test-name-pattern='compressed pre-ledger group replies' test/work-dir.test.js (failed on the dev base with one spliced reply; passed on this branch with all three fixture replies). Regression tests also cover invalid JSON, split escapes and emoji, clipped replies, and a later lookalike after the original aged out of the log.

  • npm test (in ds-bot/)
  • node scripts/build-client.mjs --check
  • npm run lint
  • Host smoke test: all 52 steps passed on pinned DSH.
  • Screenshot attached

The Windows UI and the invalid-JSON, split-escape, and expired-log cases were not tested in Chrome; those cases have unit coverage. Legacy history with missing ledger rows remains unrecoverable.

Changelog

Group history restores legacy replies from the ledger when the saved result was shortened and the original messages can be matched.

Checklist

  • One issue, one subject.
  • Rebuilt and committed client.js.
  • No keys, tokens, or personal data, including screenshots.
  • No AI author or co-author.

Software Factory opened this PR for Lulu
Implemented in session Restore compressed long turns in group history
Review with Devin · Open in Devin Desktop


Read unambiguous matches from the migrated group ledger instead of showing a torn save-token preview as a Bot reply.
@devin-ai-integration

Copy link
Copy Markdown
Contributor

I'll fix CI failures and address comments from users with write access that start with 'Devin'.

  • Disable automatic comment, CI, and merge conflict monitoring

@chatgpt-codex-connector

chatgpt-codex-connector Bot commented Nov 19, 2026 •

Copy link
Copy Markdown
Codex Review Summary: ✅ Completed

This comment shows the latest Codex review activity on this pull request.

Review Status Commit Review trigger
📝 Code Review ✅ Completed 2026-11-19T09:00:07.681663Z debdf9c PR opened
ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review" or "@codex security review".

Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings.

Anchor the missing span to intact group ledger text on both sides of the cut, as in the reported preview.
chatgpt-codex-connector[bot]

This comment was marked as resolved.

@devin-ai-integration devin-ai-integration Bot left a comment •

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

✅ Devin Review: No Issues Found

Devin Review analyzed this PR and found no bugs or issues to report.

Devin Review

@Fei2411 Fei2411 left a comment

Copy link
Copy Markdown
Owner Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Review of 158176ba (comments only; lulu makes the merge call).

Verdict: one fix needed before merge.

Needs fixing: truncated migrated replies are restored as complete (confirms the Codex P2)

The group- ledger importer (ds-bot/src/host/groups.js:422) drops length from a shortened room.log line before it tries group-lines/<room>/<seq>.txt. If that file is gone, the record keeps only the first 2,000 chars and has no length, so isKeptReply (:637) treats it as complete. In a window that spans several replies, that record can then fill the middle of the recovered span, and the client drops the "result was shortened" state.

Reproduced locally on this head: a 3-reply window (b, a, b), middle a reply kept short in room.log with its group-lines file removed. groupRelayReplies returned all three, with the middle one at 40 of 350 chars and no sign it was cut. Expected: null (fall back to today's shortened preview), or that one reply marked shortened.

Fix suggestion: keep length on the imported record when the file read fails (e.g. only drop it when text came from the file), so the existing record.text.length === record.length check rejects it; add that case to the new test. Note this changes what the ledger stores for those records, so check other readers of length (the group timeline already handles shortened lines from the live path).

Checked and fine

  • Right replies / order / no duplicates: recovered replies match the ledger in ledger order for the PR's tests and my probes (4-reply window with a working row, broken-head invalid-JSON window). A second round that repeats the same reply text does not get picked up twice or out of order (returns null rather than guessing).
  • No other group's data: lookup is per room via roomOf(sessionId) / pastOwners; another room's session returns null (covered by the PR test).
  • Newer turns / no ledger: recovery only runs when a save-token notice is present and the preview has an elision marker, and only on records before the first ledger anchor; groups with no group- book return null and render exactly as before.
  • Save-token format: notice and ...[+N chars]... regexes match dsh-plugin-save-token@2.4.1 (both compressed and losslessly re-encoded, line and string elision).
  • Local checks on this head: npm test, node scripts/build-client.mjs --check (rebuilt client.js is in the diff), npm run lint all pass.
  • Host smoke test: the PR leaves it unchecked although src/host/ changed. I ran it on this head (isolated .dsh-test-smoke, DSH 0.2.2-rc.1): all 52 steps passed. Please tick it in the description.
  • CI: 8 passed, 0 failed; Windows Node 24 test job, desktop packages and iOS simulator build still pending at review time. Devin Review: no issues found.
  • PR rules: branch into dev, up to date with dev, Fixes #852, changelog field filled.

Overlaps with open PRs

  • #1055 (Agent presets): the only PR this one adds a new conflict to: one line, the READS set in host/channel.js ('group-relay-replies' vs 'presets'); keep both. Land #1841 first; #1055 resolves it when it merges dev.
  • #1074 (files under a member's group message) touches the same shared/group-replies.js, client/cells.js, host/parts.js, host/groups.js; it already conflicts with dev in those files, and #1841 adds no new conflict. Whoever updates #1074 after this lands should keep parseReplies dropping save-token-elided objects.
  • #888, #1381, #1201, #524, #736 touch host/groups.js/parts.js/channel.js but already conflict with dev; #1841 adds nothing on top. #733, #1279 merge cleanly.

Keep the original length during import and treat old untagged rows at the former clipping limits as uncertain.
Keep original lengths for complete imports too, so a full reply at an old clipping limit remains recoverable.
Use the omitted length for a single trimmed reply without taking a suffix as proof of the source round.
Handle raw line splices before considering an exact single-reply match; use structural encoding only for its documented fragment shape.
Permit matching endpoints on one complete ledger row only when the elided JSON string length matches exactly.
Use raw fragments for the omitted-length calculation so re-encoding a lone surrogate does not inflate the visible length.
Compare the saved JSON suffix directly so a cut inside a backslash escape cannot point to another reply.
An apparent unique match is not enough when the first part of the log or an intervening sequence is missing.
@devin-ai-integration

Copy link
Copy Markdown
Contributor

Follow-up to Code Reaper’s truncated-reply finding: c071aaa4 preserves the original length and leaves the shortened preview in place when full text is missing. The regression covers both fresh imports and previously migrated rows; both preserved save-token fixtures also passed the Chrome reload retest.

@Fei2411 Fei2411 left a comment

Copy link
Copy Markdown
Owner Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Re-review at 792cfa82: ready for lulu. My earlier finding is fixed, and I found nothing new.

The earlier finding is fixed. The importer now keeps length on every imported line, so a cut line whose group-lines file is missing fails hasCompleteText. Old migrated ledgers that already lost length are caught by the 2,000 / 20,000 cap check. Those caps have not changed since cc4fdb9, so the check is sound. The new tests cover both the room-log case and the already-migrated case.

The later commits look correct:

  • JSON-codec cuts must now be exactly 220 / 80 with a matching total length. Line-codec cuts must be 260 / 100.
  • A single-reply cut must match its encoded length exactly.
  • The tail is matched on the encoded text, so a cut inside an escape still works.
  • Recovery returns null unless the pre-anchor ledger holds every seq from 1 with no gap. Unknown or ambiguous cases fall back to the saved preview.

Order, no duplicates and no data from other groups still hold. Newer ledger-backed turns and rooms with no ledger behave as before.

One scope note for lulu (not a blocker). ROOM_LOG_MAX was 60 and group-logs only started on 2026-11-08. A group with more than 60 lines before then has a gap in its ledger, so its old compressed turns keep the old preview. That is the safe choice, and the PR description already says this history cannot be fully recovered.

Checks:

  • npm test, build-client.mjs --check and npm run lint pass locally.
  • The host smoke test passes all 52 steps (isolated .dsh-test-smoke, pinned DSH 0.2.2-rc.1).
  • CI: 16/16 checks pass. Devin Review passes.
  • The PR rules are followed, including the rebuilt client.js.

Overlap: none new. 1841 merges cleanly into dev. It should still land before #1055: both add an entry to the READS set in ds-bot/src/host/channel.js, a one-line conflict for #1055 to resolve.

@devin-ai-integration devin-ai-integration Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The migration retains the original reply lengths and restores from the ledger only when the matching reply sequence is unique and complete. The targeted regressions passed, including clipped and ambiguous matches. #888 changes pagination rather than legacy reply recovery.

@Fei2411
Fei2411 merged commit e7d6b0b into dev Nov 20, 2026
15 checks passed
@Fei2411
Fei2411 deleted the fix/852-group-history branch November 20, 2026 13:47
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant