Skip to content

[Tracking] Large SQLite-only stores: lossless migration, startup and per-turn cost #7309

Open

Description

@mariopasquali

(ai: claude-opus-5-5)

@ranxianglei — this lists how our open PRs fit together, so they can be reviewed as one piece of work rather than as unrelated fixes.

Goal. Run Billy on pure SQLite (the default since #6480) for a large, long-lived store: no lost sessions on migration, startup within the readiness limit, and no whole-row rewrite on every turn.

Test case. A copy of a real 0.1.200 legacy install: 8,791 sessions, 10.7 GB index.db after migration, on a host with 4 CPUs and 5 GiB RAM. With all PRs below combined on master, the rehearsal passed: 8,791 of 8,791 sessions migrated (0 failed, 0 drifted) in 94 min, the relay was ready in 15.8 s, a restore from an online backup worked, and the old version still started on the untouched source files.

1. Migration without data loss

2. Startup on a large store

3. Per-turn cost

4. State after restart or crash

The OpenCode incremental lane (#6818, #7307, #7308) is tracked separately in #7310.

Suggested order: 1 → 2 → 4 → 3. Groups 1 and 2 decide whether a large store can move to SQLite at all; group 3 decides what it costs per turn.

Decisions only you can make: the row-rewrite design (#6792) and the timing trade-off in #6986.

I'll keep this list current as PRs land.

Activity

  1. ranxianglei commented on Nov 19, 2026

    @ranxianglei
    Owner

    🤖 Powered by ework · qwen3.8-27b

    (ai: vllm-qwen/qwen3.8-27b)

    查重: 不重复. This is a distinct tracking/coordination doc. Closest siblings are #7310 (OpenCode incremental lane — explicitly separated in the body) and #7274 (a specific event-loop bug, not a tracking doc). No other issue tracks this same set of PRs.

    Labels applied: tracking · scope/storage · needs-decision · p2 (significant but scoped to large stores; the fix is in progress with PRs open and green).

    Verified live state (checked against GitHub + the mirror CI cache, 2026-11-19):

    Group Ref Kind Live state CI
    1. Migration #6773 PR open, approved, waiting merge 30/30 green
    1. Migration SQLite-only durability fixes — not yet opened (stacked on #6773, opens after it merges) —
    2. Startup #6850 PR open 30/30 green
    2. Startup #6914 PR open 30/30 green
    2. Startup #6987 → #7260 issue → PR issue open; fix PR open, approved 30/30 green
    3. Per-turn #6986 PR open (needs your timing call) 30/30 green
    3. Per-turn #6792 / #5599 issues both open (needs your design call) —
    4. Restart #6775 → #6827 PR → PR closed; landed via the merged #6827 batch —
    4. Restart #6943 PR open, approved 18 passed / 12 still running
    (separate) #7310 issue open (OpenCode incremental lane) —

    The list is accurate — every ref matches its live state. No material movement since the body was written; the only in-flight item is #6943's 12 still-running checks.

    The two decisions that are yours (@ranxianglei):

    1. Row-rewrite design ([Root Cause] storage-persist (2 items) #6792 / Migration design review: legacy JSON to SQLite-only, wire/cache contract, duplicate routes and durability fixes #5599) — large columns get rewritten with the whole row on every save. The prototype and measurements are in [Root Cause] storage-persist (2 items) #6792. This is a storage-format design decision, not a mechanical fix.
    2. Timing trade-off in [PR] perf(persist): keep jsonWriter=off index builds out of the request path #6986 — answer persistence with the JSON writer off. The PR is green, but the timing trade-off needs your call.

    Both are design/product decisions — I'm flagging them, not deciding them. The suggested order (1 → 2 → 4 → 3) holds: groups 1+2 gate whether a large store can move to SQLite at all; group 3 is what it costs per turn.

    Nothing for me to fix here — this is a coordination doc and it's consistent with live state. I'll leave the two decisions and the merge order to you and @mariopasquali.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions