Skip to content

fix: bound LCM storage growth - #62

Merged
davidrobertson merged 2 commits into
mainfrom
codex/segmented-lossless-storage
Aug 8, 2026
Merged

fix: bound LCM storage growth#62
davidrobertson merged 2 commits into
mainfrom
codex/segmented-lossless-storage

Conversation

@davidrobertson

Copy link
Copy Markdown
Contributor

Summary

Codex LCM's raw log and SQLite index were storing the same event payloads. On a real 199,000-event store, that pushed the active index past 2 GB while the raw JSONL was about 955 MB.

This change:

  • rotates the authoritative raw log into 64 MiB segments and compresses closed segments with gzip level 1
  • stores byte locators in SQLite, then clears verified duplicate JSON payloads
  • migrates existing stores automatically with resumable, crash-safe state and malformed-record quarantine
  • keeps history forever by default, with optional CODEX_LCM_RETENTION_DAYS configuration in ~/.codex-lcm/.env
  • reports the storage layout and migration state through health and stats

Safety

The raw event log remains authoritative. SQLite is still rebuildable. Migration verifies segment hashes and event counts before removing the legacy source. If verification fails, the legacy file stays in place.

Retention invalidates the index checkpoint before deleting derived rows, so an interrupted cleanup rebuilds from the remaining raw source.

Validation

  • full Node test suite
  • TypeScript 7 typecheck
  • package smoke test
  • npm pack --dry-run
  • git diff --check
  • live-shaped rehearsal with 199,623 events and 1,593 sessions
  • SQLite PRAGMA integrity_check: ok
  • rebuilt archived and active event retrieval
  • rehearsal store reduced to 462 MB, including a 207 MB SQLite index and 267 MB compressed archive

The PR is intentionally draft while the branch build is tested as an installed local plugin.

@davidrobertson
davidrobertson marked this pull request as ready for review August 8, 2026 02:31
@davidrobertson
davidrobertson merged commit f7f4547 into main Aug 8, 2026
3 checks passed
@davidrobertson
davidrobertson deleted the codex/segmented-lossless-storage branch August 8, 2026 02:33
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant