rfc: make schema apply, branch create and delete, and cleanup safe across writer processes #1018

Activity

1. github-actions

   added

   needs-triageAwaiting maintainer triageAwaiting maintainer triage

   on Oct 16, 2026
2. ragnorc commented on Oct 20, 2026

   

   ragnorc

   on Oct 20, 2026 – with Cursor

   ContributorAuthor

   More actions

   Summary

   Schema apply, branch create, branch delete, and cleanup are documented as safe from one writer process. The engine does not enforce that boundary, and it does not warn. Two processes can both return success and leave a graph that a later read cannot use: two live copies of one branch name, a commit that no live branch contains, or a new branch that still has the old schema.

   Row writes are different. Across processes, one write wins and the other returns ReadSetChanged. The loser's staged table versions stay on disk and stay unreachable.

   The correct design is option (a): each of these operations decides on the __manifest version compare-and-swap, then checks that decision again under publication. That is the design already written in draft pull request #1017. It is not implemented. Option (b), a cooperative lock on one machine, does not make the operations safe.

   Checked on main b8e82abfbaa02a0e14b65e3bb034d1a81f281864 (feat(branch): start a branch at a graph commit, 2026-10-20). The issue was still open, with no comments, at the time of this note. No merged pull request implements option (a). Open pull request #1017 specifies it and changes no engine code.

   Environment and method

   • Linux, rustc 1.97.1, Lance 11.0.0 as pinned in Cargo.toml.
   • Local temporary directory. No S3 run and no Azure run.
   • Proof was a temporary omnigraph-engine test built with --features failpoints. It is not in the tree.
   • Each race used two OS processes. The child re-executed the test binary. A failpoint seam held each side at a named point until both sides were inside the window, then both continued. One trial per race. The outcome is the seam order, not a timing sample.

   Command for the suite:

   cargo test -p omnigraph-engine --features failpoints --test issue_1018_repro --locked -- --nocapture
   

   The schema case below is a second run of schema_apply_during_branch_create after the schema text was corrected. The first schema attempt used an invalid schema string, returned a parse error, and never reached the seam. It is not evidence about the engine.

   Verdict per claim

   (1) Reproduced. docs/dev/invariants.md lines 186–190 say schema apply keeps its main-only, single-live-branch restriction and its process-local gate, and that branch creation, branch deletion, and cleanup keep the single-writer-process boundary. The same boundary is in docs/user/branching/index.md lines 98–100 and docs/user/operations/maintenance.md lines 258–264. Maintenance also says ordinary writes and merges may overlap cleanup.

   (2) Reproduced, with one correction. WriteQueueManager::for_root (crates/omnigraph/src/db/write_queue.rs lines 407–418) is a process-wide registry. Two handles in one process share it. Two OS processes do not. The comment at lines 367–375 is that same-process rule: a second handle in the same process must serialize a schema apply or a ref deletion with a live writer. Lines 14–15 already say the rest: cross-process writers have only the one-winner compare-and-swap at publish. The issue quotes line 371 as the cross-process gap. The line is the in-process requirement. The gap is that nothing else exists once the process boundary is crossed.

   Same-process evidence, one trial: two handles in one process created the branch race. One returned success. The other returned branch 'race' already exists. One native ref file remained, and branch_list returned main and race.

   (3) Partially reproduced. Nothing stopped the pairs below, and nothing warned at the call that succeeded. The damage shows up on a later read.

   • Two processes created race. Both returned success. The root then held race.01M5BPR55ERWY77FHD6H9XB5W5.json and race.01M5BPR55EVTGRQ51EY0BNFWVY.json. branch_list then failed: branch 'race' has more than one live native incarnation; run cleanup before using it. The single-process run above prevents this.
   • Branch delete beside a publish on feat. The writer was held at publish.pre_merge, which is after the publisher has already checked its precondition (crates/omnigraph-catalog/src/publisher.rs line 1205, then merge_rows at line 1069). The other process deleted feat. The writer returned success for commit hb1.01M5BPN9XBPRT514AXG5X8WEX1.3.01M5BPNAAZ3F4YSTRRMGXSX5SD. Delete returned success. branch_list returned only main. A later lookup of that commit returned NotFound with publication_in_doubt: false. A read of feat returned branch 'feat' not found.
   • Schema apply beside branch create. Apply was held at schema_apply.post_lock_pre_effect, after the only-main check and before table effects. The other process created side. Apply returned ok applied=true version=3. Create returned success. main has the new nickname column. side does not (No field named nickname). branch_merge of side into main returned ReadSetChanged on schema_ir_hash.
   • Cleanup beside a live insert did not show this class of damage. Cleanup was held at cleanup.table_gc with a keep-one policy. The other process inserted a row and published. Cleanup returned success for 4 tables and no per-table error. The new row was still present. That overlap matches the maintenance doc. It is not the unsafe cleanup window. The unsafe window is Lance's two-phase create_branch, and this run did not reach it. See limitations.

   Schema apply against a row write was not run as its own race. A row write moves main's manifest version, and schema apply already publishes with ExactGraphVersion (crates/omnigraph/src/db/omnigraph/schema_apply.rs lines 1031–1038). The unsafe schema case is a create, because a create does not move that version. That case is the one observed above.

   (4) Reproduced. Two processes updated Alice's age, both held after detached table commits and before publish (mutation.post_finalize_pre_publisher). One returned success and published age 11 on main at graph version 2. The other returned ReadSetChanged for graph_head:main (expected version 1, current version 2). The Person table pin was version 3, and 4 detached manifests were left under that dataset path. The extra detached manifest is the loser's effect. It is not named by the published pin.

   Root cause

   The gates are local to the process. The decisions that change schema, branch names, or deleted storage are not all on the manifest compare-and-swap.

   • Branch create checks the logical name by listing live refs (crates/omnigraph-core/src/branch_control.rs lines 147–189), then creates a native ref named {logical}.{ULID}. The seam comment at lines 791–796 says no compare-and-swap covers that window, and that the exclusive schema permit is what keeps a second create out. That permit exists only inside one process. Lance Branches::create is put-if-absent on the native path, so two different ULIDs do not conflict.
   • Branch delete checks the branch identifier, then calls replace_metadata (branch_control.rs lines 534–536 and 611). Lance 11 Branches::replace_metadata (lance-11.0.0 src/dataset/refs.rs lines 528–549) checks that the ref exists and then does an ordinary put. A row write's precondition does not include "this branch is still the live ref", and the overwrite at publish.pre_merge can still succeed after the ref is gone.
   • Schema apply rejects a graph that has a branch other than main while it plans (schema_apply.rs lines 153–163). It does not check that list again at publication. Publication requires the captured manifest version. A create does not change that version, so the apply still commits, and the new branch keeps the contract it cloned.
   • A row write's publication requires the captured graph head and the captured table versions. The loser reloads, sees the new head, and returns ReadSetChanged (crates/omnigraph/src/exec/mutation.rs lines 1324–1348). Detached effects from before that publish stay unreachable. Insert-only mutations may retry a pre-effect mismatch. Update and delete return ReadSetChanged to the caller (lines 1200–1207). The trial above was an update, and the ReadSetChanged came after the detached commits.

   Invariants at stake: one publication door (2), one coherent view that is checked again before effects (3), one publication per mutation (4), pre-publication effects stay unreachable (5), and a loud typed outcome instead of a success that a later read cannot find (8 and 11). The deny-list at docs/dev/invariants.md line 155 forbids presenting a process-local lock as distributed writer fencing.

   The full class, not only the four names in the issue:

   • Already decided by the manifest compare-and-swap. The in-process gates only avoid wasted staging. This includes row mutate and load, merge, schema apply against any writer that moves main's version, system-column upgrade against that same writer, ensure_indices, and optimize. Merged pull request fix(engine): optimize survives a cross-process write race on the same table #297 is this class: optimize retries when its manifest equality check loses to a concurrent table write.
   • Not decided by that compare-and-swap. Branch create and branch create-from. Branch delete. Cleanup against a tree that Lance has written and not yet named with a ref. Proposal drop against publish was not run here. Draft rfc: graph access and cross-process writes #1017 describes that fifth race. Contract install, discard, and reload take the exclusive schema gate. That gate is process-local. Those three were not raced here.
   • System-column upgrade has the same create window as schema apply. It publishes with ExactGraphHead (system_column_upgrade.rs line 424). A create does not move that head. This was not raced separately.

   Cluster admission is a separate rule. docs/user/deployment.md lines 288–298 say one mutation-capable writer per cluster, and that the lock cannot fence older binaries, raw storage tools, or embedded writers. The runs above used Omnigraph::open on a standalone directory. No cluster lock was involved.

   Recommendation

   Use option (a). Do not use option (b) as the correctness mechanism.

   Option (a) is already specified in draft pull request #1017, section "Engine: control operations on the version CAS" (head e15d2f5b on ragnorc/omnigraph, checked 2026-10-20). Retirement becomes a head-preserving publication on the branch chain, so a concurrent publisher loses that version or wins it and the delete retries. The logical branch name is registered on main after the native ref exists, so two creates race on main's version and the loser removes its own unregistered tree. Because that registration moves main's version, schema apply and system-column upgrade lose to a create that happened after their only-main check. Cleanup deletes an unregistered tree only after the existing seven-day orphan age, and it never calls Lance's forced branch delete. Drop becomes a publication before any tag release. The in-process gates stay as a way to avoid wasted work.

   The issue text says #1017 Appendix A1 contains both options. That is no longer true. In the current draft, Appendix A1 is WWW-Authenticate on 401. The lock option is in the alternatives table, and the table rejects it. The draft also says the one-writer boundary stays in force until the engine phase ships.

   Option (b) protects only processes of one account on one machine that choose to take the lock. A second machine, a second account, an embedded writer, or any process that does not take the lock is outside it. Calling that lock writer fencing would match the deny-list. A same-machine warning can still be useful as a hint. It must fail open, and it must not be what the write path trusts. #1017 rejects the lock even as a write-path stopgap.

   What each option costs:

   • The compare-and-swap that works on local disk, S3, and Azure is Lance's put-if-absent of the next numbered __manifest version, with zero retries (lance-io-11.0.0 src/object_store.rs lines 890–930). Local disk stages the object and uses rename_if_not_exists. S3 and Azure send PutMode::Create, which is If-None-Match: * (object_store-0.13.2 aws/client.rs line 796 and azure/client.rs line 559). Local PutMode::Update is NotImplemented (local.rs line 336). There is no conditional delete.
   • Option (a), per the draft, adds one such publication on main for create, one on the branch plus one on main for delete, and one on the target branch for drop. Schema apply, system-column upgrade, row writes, merges, optimize, and index publication already pay their one publication. They gain no new put. Cleanup gains no put. Its cost is a stricter choice of what it may delete.
   • Option (b) costs a local lock and no object-store request. It does not cover a shared object-store root.

   What option (a) needs from Lance: no new primitive. It needs the numbered manifest create-if-absent that Lance 11 already provides. It must not depend on a conditional rewrite of a branch ref, because replace_metadata has none, and it must not depend on a conditional delete, because the object stores have none. Old-version retention stays the collector's current rule: delete a version only when no retained manifest still names it.

   Liability: option (a) removes silent lost commits and two live copies of one name. It adds __manifest rows under the next storage stamp. Older binaries refuse that stamp. The way back is export and load. That stamp is irreversible, and #963 also wants a stamp, so the two need an explicit order. Option (b) adds a lock operators can mistake for safety, and it leaves the silent outcomes in place for every writer that does not hold it.

   Lance

   Lance does not fix this. The pin is 11.0.0. Open pull request #1072 upgrades the pin to 13.0.0 and does not add a ref compare-and-swap. Its branch_control.rs change is a test double for paginated list. At tag v13.0.0, Branches::replace_metadata is still an exists-check and an unconditional put (rust/lance/src/dataset/refs.rs lines 531–556), and Dataset::create_branch still writes the tree and then the ref (lines 609–645). A search of open Lance pull requests and issues for replace_metadata and branch-ref compare-and-swap returned no matches on 2026-10-20. https://github.com/lance-format/lance/releases/tag/v13.0.0

   Related work

   • rfc: graph access and cross-process writes #1017, draft RFC. Specifies option (a). Implements nothing. Says the current boundary stays until the engine phase. rfc: graph access and cross-process writes #1017
   • rfc: storage outcomes per operation and restartable ownership #1093, draft RFC for storage outcomes and restartable ownership. Related recovery vocabulary. Not this compare-and-swap. rfc: storage outcomes per operation and restartable ownership #1093
   • feat(cli): serve a target for you on loopback #1083, feat(server): serve a cluster to the same account on loopback #1077, feat(server): serve one standalone graph to the same account on loopback #1075. Loopback serve. These concentrate cooperating clients in one process. They do not make two direct writers safe. feat(cli): serve a target for you on loopback #1083 feat(server): serve a cluster to the same account on loopback #1077 feat(server): serve one standalone graph to the same account on loopback #1075
   • CLI: omnigraph studio launcher and graph owner discovery #980. Studio launcher. The client that rfc: graph access and cross-process writes #1017 sends through that one local server. CLI: omnigraph studio launcher and graph owner discovery #980
   • rfc: schema apply with live branches #963. Schema apply with live branches. A different restriction, the main-only rule. A cross-process create is one way a second branch appears. Both want a storage stamp. rfc: schema apply with live branches #963
   • perf(writes): hold the version a write committed in the read-handle cache #818 and docs(rfc): group commit, with the Lance composition guard #785. Hold a committed version, and group commit. Performance on the publication path. Not the control-operation gap. perf(writes): hold the version a write committed in the read-handle cache #818 docs(rfc): group commit, with the Lance composition guard #785
   • fix(engine): optimize survives a cross-process write race on the same table #297, merged. Optimize survives a cross-process table write by losing the manifest compare-and-swap and retrying. That is the mechanism option (a) extends. It does not cover branch create, branch delete, or cleanup. fix(engine): optimize survives a cross-process write race on the same table #297
   • bug: cleanup reclaims table versions still held by active reads #958, storage: safely reclaim obsolete physical __manifest versions and files #644, branches: safe deletion of unmerged work and a supported recovery window #647, server: coordinated optimize and cleanup operations with resumable status #646. Cleanup of versions held by readers, reclaim of old manifest files, safe deletion and recovery, server-coordinated optimize and cleanup. Adjacent, still open, not this race. bug: cleanup reclaims table versions still held by active reads #958 storage: safely reclaim obsolete physical __manifest versions and files #644 branches: safe deletion of unmerged work and a supported recovery window #647 server: coordinated optimize and cleanup operations with resumable status #646
   • bug: one storage write with an unknown outcome stops the server until a manual unlock #1067. One unknown storage outcome stops the server. rfc: graph access and cross-process writes #1017 points its control operations at that outcome vocabulary. bug: one storage write with an unknown outcome stops the server until a manual unlock #1067
   • performance: capture a write from the process's own last publication instead of re-reading __manifest #837 and rfc: delegated, branch-scoped credentials for agents #978. Read the last publication from the same process, and delegated branch credentials. Not this gap. performance: capture a write from the process's own last publication instead of re-reading __manifest #837 rfc: delegated, branch-scoped credentials for agents #978

   Supersession

   Do not close this issue. #1017 is the draft design for the same need, and it is wider, but it is not merged and it implements nothing. It says the one-writer boundary remains until its engine phase. This issue stays the open record of that unimplemented boundary.

   Limitations

   • One deterministic trial per race, on a local disk. S3 and Azure were not run. The conditional-create behavior above is from the object_store 0.13.2 sources this build uses.
   • Cleanup against Lance's two-phase create was not run. No OmniGraph seam sits between the tree write and the ref write (lance-11.0.0 src/dataset.rs lines 590–603).
   • Proposal drop against publish was not run.
   • System-column upgrade against create, and contract install, discard, and reload, were not run.
   • The temporary test was removed after the runs and was not committed.