perf(graph): stage source graph concurrently with embedding

[jmagar]

Problem

Large repository ingestion built the graph after vector publication, leaving graph work on the ingestion completion path. The last measured Labby run spent 108.817 seconds in graphing and took 404.986 seconds overall.

Resolution

Build a private generation graph alongside embedding and activate graph changes in the same SQLite transaction as ledger publication. Keep uncommitted graph data invisible and dispose private stage files through the existing durable cleanup worker.

Approach

• Graph domain owns private SQLite stages, bounded sanitized candidate journals, immutable sealing, merge-preserving activation/replay, durable indexed receipts, and owner-safe disposal.
• Ledger exposes existing publication logic to a caller-owned transaction without changing its generation fences or cleanup-debt semantics.
• Services coordinate provider reservations, concurrent embedding/staging, publication lease checks, and cancellation settlement. Transports remain unchanged.
• A shared generation budget accounts for graph journals and other deferred side effects. Staged candidates are released from prepared batches.
• Activation logs distinguish bulk activation from revision-triggered replay and report candidate counts, journal bytes, and elapsed time.

Reviewer considerations

The graph-wide revision deliberately selects conservative replay after any concurrent graph mutation. This preserves concurrent shared claims but may reduce speedup when unrelated cleanup occurs. Publication ownership survives caller cancellation and scheduler renewal abandonment; cleanup waits for settlement and preserves unknown or committed vector generations. Journals receive canonical redaction before disk persistence.

Verification

• Lavra security, Rust, simplicity, architecture, and performance reviews completed; all findings addressed.
• Graph: 116 passed, 1 ignored. Forced idle-plus-held SQLite connections reproduced premature disposal in both drop and explicit disposal; both paths now retain ownership until connection settlement.
• Ledger: 96 passed.
• Services: 1,214 passed, including discovery-only mapping without embedding or document claims.
• CLI mapping: 17 passed.
• Isolated upgrade of a consistent 1.5 GB production SQLite backup applied graph migration 6 and preserved all source, node, edge, and evidence counts.
• Jobs: 272 passed.
• Database schema tests: 6 passed.
• Actual publisher caller-drop and finalizer settlement tests: 2 passed.
• Affected crate Clippy with warnings denied, formatting, generated contracts/docs, and changed-file monolith checks passed.
• Deployment and new live ingestion benchmarks are pending CI and merge. No speedup is claimed for this branch yet.

Risk

Adds graph migration 6 and private staging storage beside the unified database. Physical disposal is asynchronous and retains durable retry records. The coarse revision fallback can replay a complete candidate journal under the publication writer; live measurements will distinguish that path.

[chatgpt-codex-connector]

You have reached your Codex usage limits for code reviews. You can see your limits in the Codex usage dashboard.
To continue using code reviews, add credits to your account and enable them for code reviews in your settings.