Skip to content

rfc: relocated graph roots - #1234

Open
ragnorc wants to merge 3 commits into
ModernRelay:mainfrom
ragnorc:rfc/1233-relocated-root-branch-bases
Open

ragnorc wants to merge 3 commits into
ModernRelay:mainfrom
ragnorc:rfc/1233-relocated-root-branch-bases

Conversation

@ragnorc

@ragnorc ragnorc commented Nov 3, 2026 •

Copy link
Copy Markdown
Contributor

Why

Issue #1233 is labeled needs-rfc. The fix changes when Omnigraph trusts a stored Lance field (BasePath.path) for one class of bases. It also changes what RFC 0029 and RFC 0064 say about copying a graph root to a new path. GOVERNANCE.md makes that size L: the decision comes before the code.

What changed

There is one new draft RFC, docs/rfcs/2026-11-03-relocated-graph-roots.md, and its registry row. Nothing outside docs/rfcs/ changes.

Why a new RFC, not amendments to RFC 0029 and RFC 0064. The rule is a compatibility decision that neither RFC owns. Amending both would state it twice. docs/rfcs/README.md prescribes a disposition note when an accepted RFC replaces part of an earlier one. The draft does not edit RFC 0029 or RFC 0064: a draft cannot change an accepted contract. The exact disposition notes are in the RFC's Rollout section, and they land with acceptance. That matches how PR #872 added disposition notes after acceptance.

Decisions the RFC takes:

  • Contract. A whole-root copy on the same kind of backend (local, S3 with another prefix or bucket, Azure with another prefix or container) opens at its new location and serves every branch, tag, and retained snapshot. It reads only objects under the opened root. A missing copied object fails with NotFound and names the opened root. A base that is not graph-internal refuses with a Permanent failure before anything is read through it.
  • Outside the contract. Moves between backend kinds, cluster roots at a new location (that is adoption), external Blob URIs, copies taken while a writer runs, and older binaries.
  • Base provenance. Every graph-dataset base comes from Lance's shallow clone of a branch of the same dataset, through the single lance_clone::create_branch call that forbidden_apis.rs already pins. The RFC explains why this is consistent with invariant 8 and with lance.md's "names never establish ownership".
  • A mismatched-name base fails closed. Lance writes the name and the path from one ref_name. It has no branch rename, and Omnigraph never calls add_bases. So a mismatch comes only from a foreign writer or corruption. Keeping the recorded path would be the cross-tree read the contract removes.
  • Pre-v11 table forks. The rule extends to tables and the collector, rather than refusing relocation while forks exist. Detection costs the same either way. Forks already open through the one opener. Without the extension, the collector's borrowed_origins would protect the original location's files on a relocated root, and could delete files the copy's retained forks still borrow.
  • Where the rule lives, and what persists. It lives in one door (the dataset opener plus one checkout function), guarded against bare checkouts and against Lance base-adding APIs. Nothing new is persisted, and there is no format stamp.
  • Lance fence row. The row text and its drop condition are given in full.
  • Base-store cache (revision 2). Lance keeps a base's store in a pub(crate) cache that clones share and that a swap of the public manifest Arc does not clear. The door therefore installs the rebound map on a fresh clone with an empty cache (Dataset::with_object_store) before any base resolves.
  • One owner per path rule (revision 2). A table pin's manifest_path is rebuilt by full_manifest_object_store_path. A Lance BasePath is rebound by branch_base, which is also the only classifier: the refusal, the collector's borrowed origins, and the relocation bit.

Evidence

The prototype on ragnorc/omnigraph:proto/1233-branch-base-follows-root passes the issue's acceptance list on local files. It also shows a silent cross-tree read on main: a copy missing one inherited object returned 4 rows read from the original path. The RFC lists the evidence the implementation must add. The draft code PR is #1235. It implements these decisions and waits for this RFC to be accepted.

Checks

  • python3 scripts/check-docs.py: OK (450 Markdown files).

Not covered

The status stays draft. blocked_on names two evidence gates: the RustFS and Azurite relocation owners, and the genuine predecessor journeys passing with the fail-closed rule. The upstream Lance issue is filed as lance#10661.

Draft RFC for issue ModernRelay#1233. A whole-root copy opened at a new location on
the same backend kind serves every branch from its own objects; a base
outside the root or a missing copied object refuses. Settles the
base-provenance rule, fail-closed handling of mismatched names, the
extension to pre-v11 table forks and the collector, object-store scope,
and the Lance fence row.

ragnorc commented Nov 3, 2026

Copy link
Copy Markdown
Contributor Author

Recommendation: revise.

This RFC is the right decision for issue #1233, and it is not ready to accept. A whole-root copy must read only the root it was opened at. Today a retained branch keeps the absolute path from create_branch. A copy then reads the original tree when that path still exists, and it fails when the path is gone. The repair belongs in the opener, in memory, for the one base shape Omnigraph writes. It must fail closed for every other base. It must not write a new manifest, a stamp, or a second location.

Five sentences do not match Lance 13.0.0 or the code at main 36d3ea08. Accepting them would freeze a wrong substrate model. Draft #1235 (25f7eedb) follows those sentences: rebind_internal_bases replaces dataset.manifest and leaves the base-store cache in place.

Blocking

1. The base-store cache outlives the public manifest Arc

Lance 13.0.0 Dataset::object_store(Some(id)) builds the store on first use and keeps it in base_object_stores (lance dataset.rs). The field is pub(crate). The struct comment says those stores are normally shared across clones. set_manifest is private. It is the only clear, and it clears only when base_paths change. Assigning the public manifest Arc does not clear the cache.

checkout_version calls checkout_by_ref. When already_checked_out is true (same branch, version, naming scheme, and e-tag), that function returns self.clone() and shares the cache. It does not always build a new handle.

The door must rewrite the map before the first base resolution. The handle must have an empty cache, and it must not share that cache with a handle that still has the recorded paths. A later swap of the public Arc cannot repair a filled cache. The first resolver wins. One handle can then compute a path from the new map and read it through a store opened at the old URI.

State this limit in "Inherited Lance behavior". Make the door obey it. The fence row that says every base_id file resolves through Dataset::manifest.base_paths needs the same sentence. Directory helpers (data_file_dir_for_base, dataset_dir_for_deletion, indice_files_dir) read the map on each call. The store does not.

2. The storage upgrade does not convert branches through lance_clone::create_branch

publish_conversion in crates/omnigraph/src/db/upgrade.rs checks out the existing branch dataset and overwrites it. Manifest::new_from_previous copies the predecessor base_paths. The create_branch_recoverably call in crates/omnigraph-catalog/src/legacy.rs is the test fixture fork. An upgraded graph has the old binary's clone bytes, copied forward by that overwrite. The fail-closed proof is the predecessor journeys in blocked_on. Replace "The storage upgrade converts branches through the same call."

3. A publish and a new branch do not persist ancestor bases the same way

"What new manifests persist" is right. A publish from a rebound handle copies the in-memory map, so the new version carries the opened URI for every base, ancestors included. commit_transaction (lance io/commit.rs) uses that map only when dataset.manifest.version == read_version. Otherwise Lance checks out read_version from disk, and the recorded bases return. create_branch always reloads the source manifest. The new branch records the opened URI for its direct source and keeps ancestor URIs from disk.

The compatibility sentence "It still fails on inherited ancestor bases, as it does today" matches a new branch and every version that was not rewritten. It does not match a same-version publish of a rebound handle. An older binary can read that new version, ancestors included. Split the two cases.

4. The portability quote stops before the shallow-clone paragraph

The layout page says a dataset whose files are relative to the root functions at the new location with no manifest update. The next paragraph says a shallow clone keeps the original base locations until base_paths is updated. A branch is that shallow clone. This rule is that port step, done in memory. Say that. A branch under tree/ does not break a promise the same page already qualifies. The invariants section already names the port step. The motivation should match it.

5. The relocation bit and the pin-path rebuild are a second statement of this fact

manifest_has_external_base_paths (crates/omnigraph-catalog/src/lib.rs, and Omnigraph::manifest_has_external_base_paths) is true when base_paths is non-empty. relocation_self_contained in crates/omnigraph-bench/src/real_graph.rs is the negation, for the manifest and for each user table (SnapshotDataset::has_external_base_paths). The comments say a directory copy is self-contained only when those paths are empty. After this contract a branched graph is relocatable and those entries remain, so the bit stays false. The evidence list does not name this surface. Draft #1235 does not either.

full_manifest_object_store_path in crates/omnigraph-core/src/metadata.rs already rebuilds an absolute path from the opened root for table pins. Name it. Give both facts one owner, or write the split in the RFC: an Omnigraph pin path is rebuilt at read, and a Lance BasePath is rebound at the open door. Five more path rewrites should not grow beside these two.

Optional

  • The alternatives row says "The upstream issue is filed as well." The PR body says the issue is drafted and not filed. Rollout step 3 is future tense. A search of lance-format/lance returned no issue for this defect. Say "drafted, not filed."
  • DatasetBuilder::with_serialized_manifest does not drop ManifestLocation for an attached version. load_by_uri calls resolve_version_location. An attached version HEADs the manifest and fills the e-tag and the size (default_resolve_version in lance-table io/commit.rs). A detached version returns e_tag: None and size: None with no HEAD. Reject the builder because it opens the dataset again and does I/O. The in-memory door is what can keep the DST cost at zero, and only when it runs before any base read.
  • The reproduced NotFound is code: internal and action: refresh (lance-io local.rs:273). A missing copied object does not heal on refresh. The RFC keeps that class. Say so. After the rebind, the path in the message must be under the opened root. Today it names the create-time path.
  • The suffix text uses / only. Lance accepts \ in get_root_path. Draft fix(core): graph-internal branch bases follow the opened root #1235 normalized() trims a trailing slash and, on Windows, maps \ to /. Put that rule in this RFC.
  • A base that Omnigraph did not write, whose path still ends with the suffix and whose name is main, is rebound. That is the residual of this discriminator. The inventory says it does not occur. One sentence in the design is enough.
  • blocked_on is the right gate for the unknown predecessor shape. Those journeys are the upgrade proof. The RFC already says they do not block acceptance. That matches the RFC guide when the gates are explicit.
  • The registry row is last, draft, not-started. That matches "in progress only after acceptance." Point at draft fix(core): graph-internal branch bases follow the opened root #1235 in the rollout so a reader can find the code. Open RFC PRs rfc: the GQ 3 grammar, one break and the slots it freezes #1202, rfc: the GQ 3 programme, sequence, slot owners and the open RFCs #1213, and rfc: remote optimize through the serving owner #1214 also append registry rows. That is a mechanical conflict, not a design conflict. fix(cluster): never take a live server's S3 lock #1217 does not touch this contract. Copies taken while a writer runs stay outside it, which matches rfc: remote optimize through the serving owner #1214.

Tradeoff and liability

The change removes a second frozen location. The opened root is the authority (invariants 3 and 12). A same-kind backup can stand alone. There is no new persisted state, no stamp, and no Lance patch. The reader rule stays for old bytes. Drop it for new bytes only when a clone records a location relative to the dataset.

The change adds a permanent in-memory reading of one persisted shape, a fail-closed refusal that can stop a graph whose bases are outside the inventory, and a door on every open and checkout. The private base-store cache is how the same cause returns if the door is late. The relocation bit is how a second definition returns if the evidence list ignores it.

Correctness of the read contract comes before a format change. Lance specifies an absolute BasePath.path, and uri_to_url resolves a relative path against the working directory. An on-disk rewrite is set aside on purpose: historical versions would still need the reader rule, and e-tags and branch identity would change. Keeping the recorded path for an unexplained base would leave the cross-tree read.

The concept to own is "the absolute location of a graph dataset follows the opened root." One matcher should be the only reader of base_paths for this decision, including the collector. branch_base should own the cache rule as well, so a caller cannot swap the Arc after a base store exists.

These claims in the RFC match the code at main and Lance 13.0.0: create_branch records the absolute source URI; shallow_clone copies bases and sets is_dataset_root; later versions copy the map; find_main and find_branch do no I/O; there is no branch rename; production has one Lance create_branch, in lance_clone.rs; catalog publishes pass initial_bases: None; external Blob URIs stay outside bases because table writes set allow_external_blob_outside_bases; local reads open the absolute path; collect_native_forks follows refs; borrowed_origins reads base paths; unattached tag names are ULIDs; GQT cannot copy a root; cluster adoption stays unsupported; a move between backend kinds stays on export and import. The registry row, the front matter, and the decision to leave RFC 0029 and RFC 0064 untouched until acceptance are correct.

What I checked

Head 6cdebae5fd821f1902cc8372a45d097ad51dee15. Merge base 36d3ea08c5289b40653a26752cd0e6c8ef85e6af. No code ships, so there is no behavior diff to run on this head. GQT cannot copy a root. The owner named in the RFC, integration/branching.rs, is the right place for the regression when the code lands.

Reproduced on the merge-base workspace with target/debug/omnigraph (version 0.14.1, commit null; the binary is newer than lance_clone.rs). One local graph, one Item, branch retained at that commit, then a whole-root copy with the original path moved away:

  • Snapshot query on the copy: exit 0, one row.
  • Branch query on the copy: exit 1. storage: Not found: tmp/og-repro-1233/demo/__manifest/data/11011011100101100000110125108f4e0e9a1d3ae329e4841c.lance, lance-io-13.0.0 local.rs:273:40, code: internal, action: refresh. The same file exists under the copy.
  • Original path restored: the copy's branch query returns the row. That is the silent cross-tree read.
  • Original data files hidden: the copy's branch query fails on the original path again.
  • After a write on retained and a child branch, with the original path moved away: the written retained head on the copy returned both rows. The child failed NotFound on .../demo/__manifest/tree/retained.<id>/data/<file>.lance.

The branch manifest at the merge base contains the absolute URI /tmp/og-repro-1233/demo/__manifest.

Docs, on a detached worktree of 6cdebae5: PYTHONPATH=/tmp/og-docs-py python3 scripts/check-docs.py with markdown-it-py==4.0.0 and mdurl==0.1.2. Result: Documentation OK (450 Markdown files checked). python3-venv is not installed in this environment, so this was not the documented venv. The same pins were installed. CI job Check AGENTS.md Links on this head runs scripts/check-docs.py and passed. GQT jobs passed by skipping (No GQT input changed). The heavy test jobs skipped. That is docs-only CI, not a corpus run.

Limitations

I did not run S3, Azure, or the predecessor journeys. I did not re-run the prototype's GQT, DST, or Clippy claims. I did not re-check Lance 14 or 15 for branch materialize. The CLI commit field is null. I did not push, and I did not change this pull request's branch.

Inline comments

docs/rfcs/2026-11-03-relocated-graph-roots.md:152

This pair of sentences does not match Lance 13.0.0.

object_store(Some(id)) builds the store on first use and keeps it in base_object_stores (pub(crate)). The struct comment says those stores are normally shared across clones. set_manifest is private and is the only clear, and it clears only when base_paths change. Assigning the public manifest Arc does not clear the cache.

checkout_version calls checkout_by_ref. When already_checked_out is true, that returns self.clone() and shares the cache. It does not always build a new handle.

The door must rewrite the map before the first base resolution, on a handle with an empty cache that is not shared with a handle that still has the recorded paths. Draft #1235 (25f7eedb) rebind_internal_bases only replaces the Arc. The fence row needs the same limit: directory helpers re-read the map, and the store does not.

docs/rfcs/2026-11-03-relocated-graph-roots.md:168

publish_conversion in crates/omnigraph/src/db/upgrade.rs checks out the existing branch dataset and overwrites it. Manifest::new_from_previous copies the predecessor base_paths. It does not call lance_clone::create_branch. The create_branch_recoverably in crates/omnigraph-catalog/src/legacy.rs is the test fixture fork.

An upgraded graph has the old binary's clone bytes, copied forward by that overwrite. The fail-closed proof is the predecessor journeys in blocked_on. Replace "the same call."

docs/rfcs/2026-11-03-relocated-graph-roots.md:341

Split a publish from create_branch.

A same-version publish of a rebound handle copies the whole in-memory map (new_from_previous). An older binary can read that new version, ancestors included. commit_transaction uses the handle map only when dataset.manifest.version == read_version. Otherwise Lance reloads read_version from disk.

create_branch always reloads the source manifest. The new branch records the opened URI for its direct source and keeps ancestor URIs from disk. "It still fails on inherited ancestor bases, as it does today" matches that case and every version that was not rewritten. It does not match the publish.

docs/rfcs/2026-11-03-relocated-graph-roots.md:69

The layout page's next paragraph qualifies this sentence. A dataset whose files are relative to the root functions at the new location with no manifest update. A shallow clone, which a branch is, keeps the original base locations until base_paths is updated. This rule is that port step, done in memory. The motivation should say that. A branch under tree/ does not break a promise the same page already qualifies.

docs/rfcs/2026-11-03-relocated-graph-roots.md:376

Add the existing relocation bit to this list. manifest_has_external_base_paths is true when base_paths is non-empty (omnigraph-catalog and Omnigraph::manifest_has_external_base_paths). relocation_self_contained in crates/omnigraph-bench/src/real_graph.rs is the negation, including each user table. After this contract a branched graph is relocatable and those entries remain, so the bit stays false. The comments still say a copy is self-contained only when base_paths is empty.

Also name full_manifest_object_store_path in crates/omnigraph-core/src/metadata.rs. It already rebuilds an absolute pin path from the opened root. One owner, or an explicit split between a pin path and a Lance BasePath, so the two cannot drift.

docs/rfcs/2026-11-03-relocated-graph-roots.md:359

Optional. The rejection is right. The reason is not. load_by_uri still calls resolve_version_location when a serialized manifest is supplied. An attached version HEADs the object and fills the e-tag and the size. A detached version returns e_tag: None and size: None with no HEAD. Reject the builder because it opens the dataset again and does I/O. The in-memory door is what can keep the DST cost at zero, and only when it runs before any base read.

docs/rfcs/2026-11-03-relocated-graph-roots.md:362

Optional. This row says the upstream issue is filed. The PR body says it is drafted and not filed, and rollout step 3 is future tense. A search of lance-format/lance returned no issue for this defect. Say "drafted, not filed."

State the base-store cache limit and that the door installs the rebound
map on a fresh, unshared cache before any base resolves; the storage
upgrade overwrites rather than clones; what a same-version publish, a
stale-handle publish and a new branch persist; the layout page's
shallow-clone port step; one owner each for table pin paths and Lance
bases, the relocation bit included. Also the builder's I/O, the NotFound
class and message path, separator normalization, the main-name residual,
and links to lance#10661 and draft ModernRelay#1235.
@ragnorc

ragnorc commented Nov 3, 2026

Copy link
Copy Markdown
Contributor Author

Thanks. I checked every point against Lance 13.0.0 and main 36d3ea08. All of them hold, so I changed the text rather than disputing any. RFC commits: 4da4c206, 92eb847f. Code commits on #1235: 6bba8f48, ade1fe4c, 04605ccc.

Blocking

  1. Base-store cache. Confirmed: base_object_stores is pub(crate) and shared by clones, set_manifest is the only clear, and already_checked_out returns self.clone().
    • RFC: "Inherited Lance behavior" and the fence row now state the limit. The design says the door installs the rebound map on a fresh clone with an empty cache (Dataset::with_object_store) before any base resolves.
    • Code (6bba8f48): follow_opened_root now does exactly that, and it is async.
    • Test (ade1fe4c): branch_base::tests::a_base_store_resolved_before_the_rebind_is_not_read_through uses two isolated shared-memory:// buckets. The RustFS and Azurite journeys add the same step on the never-written child branch. With the old behavior (an Arc swap on the incoming handle), both fail Not found reading the emptied old bucket.
    • A forbidden_apis check now refuses any production manifest = or base_paths read outside branch_base.rs.
  2. Storage upgrade. Confirmed: publish_conversion checks out and overwrites, and the legacy.rs call is the test-only LegacyHistory::fork. The sentence is replaced. The predecessor journeys stay the proof in blocked_on; they passed on fix(core): graph-internal branch bases follow the opened root #1235.
  3. What persists. Confirmed in commit_transaction: the in-memory map is used only when dataset.manifest.version == read_version. The RFC now splits a same-version publish, a stale-handle publish, and a new branch, both in "What new manifests persist" and in compatibility.
  4. Layout page. The motivation now quotes the shallow-clone paragraph and says this rule is that port step, done in memory.
  5. Relocation bit and pin path. The RFC gains "Two path rules, one owner each": the pin path stays with full_manifest_object_store_path, and Lance bases with branch_base.
    • Code: manifest_has_external_base_paths and SnapshotDataset::has_external_base_paths (and so relocation_self_contained) now ask branch_base::has_external_bases, so a branched graph is self-contained.
    • The collector, the full-text certificate, the range memo, the merge key and the instrumentation checks read bases only through branch_base accessors.
    • The branching test asserts the bit is false for every branch of a relocated root.

Optional

One addition, from fixing #1235. At first the accessor that resolves a base path also re-classified bases, and that broke fts_compat_tests::certificate_file_is_bounded_and_follows_shallow_clone_ownership. That test feeds Lance's non-root base shape on purpose. Now only the door and the relocation bit classify, and the accessors read the map the door rebound.

Still open. The CI wiring for the other-bucket and Azurite halves (a second bucket and container, and the required cells) needs a token with workflow scope, which my push token lacks. Those halves pass locally against the CI-pinned RustFS and Azurite.

ragnorc commented Nov 3, 2026

Copy link
Copy Markdown
Contributor Author

Recommendation: ready.

This RFC is the right decision for issue #1233. A whole-root copy must read only the root it was opened at. The text at 92eb847f matches Lance 13.0.0 and main 36d3ea08. PR #1235 at 04605ccc follows that text. The delta has no blocking defect.

Prior findings

All 5 blocking findings are resolved. All 6 optional findings are resolved.

  1. Base-store cache. Resolved. "Inherited Lance behavior" and the fence row state the limit. The door installs the rebound map on a fresh handle (Dataset::with_object_store) before any base resolves. That call sets base_object_stores to empty (lance dataset.rs). follow_opened_root at 04605ccc does this. It does not write the map on the handle that came in.
  2. Storage upgrade. Resolved. The text says publish_conversion checks out each branch and overwrites it. Manifest::new_from_previous copies base_paths. The predecessor journeys stay the proof in blocked_on.
  3. What a new version persists. Resolved. The text splits a same-version publish, a stale-handle publish, and a new branch. That split matches commit_transaction and the clone path in do_commit_new_dataset.
  4. Layout page. Resolved. The motivation quotes both portability paragraphs. This rule is that port step, done in memory.
  5. Relocation bit and pin path. Resolved. "Two path rules, one owner each" names full_manifest_object_store_path and branch_base. The bit asks branch_base::has_external_bases. An openable graph is self-contained. fix(core): graph-internal branch bases follow the opened root #1235 does this.
  6. Upstream issue. Resolved. lance#10661 is open. It states this defect.
  7. with_serialized_manifest. Resolved. The rejection is the extra open. resolve_version_location HEADs an attached manifest.
  8. NotFound. Resolved. The class stays a NotFound storage failure. The server reports every storage failure as internal, and that code asks for refresh. The message names the opened root.
  9. Separators. Resolved. The text drops the query and a trailing /. On Windows it reads \\ as /, which is what get_root_path accepts.
  10. Main-name residual. Resolved. One sentence states it.
  11. Rollout. Resolved. The rollout points at fix(core): graph-internal branch bases follow the opened root #1235. The registry row is still last, draft, not-started. Main is still 36d3ea08.

Optional

One sentence overstates a clone. A same-version publish does record the opened URI for every base, ancestors included, because new_from_previous copies the in-memory map. The next sentence says that is what Lance writes for any clone at that location. A clone does not rewrite ancestors. do_commit_new_dataset reloads the source manifest from ref_path, and shallow_clone copies those disk bases. Only the new base uses ref_path. The next two bullets already say this. Narrow that one sentence. The three cases stay the contract.

Tradeoff and liability

The change removes a second frozen location. The opened root is the authority (invariants 3 and 12). A same-kind backup can stand alone. There is no new persisted state, no stamp, and no Lance patch.

The change adds a permanent in-memory reading of one persisted shape, a fail-closed refusal, and a door on every open and checkout. The door owns the cache rule. A late swap of the public manifest cannot restore the cross-tree read. The pin path and the Lance base each have one owner. The text forbids a third path rule. A new location fact names one of those two owners.

The read contract comes first. An on-disk rewrite stays rejected. Old versions would still need the reader rule, and e-tags and branch identity would change.

What I checked

Head 92eb847f55e6a881c9aaaeeece65d37ec92ff863. Merge base 36d3ea08c5289b40653a26752cd0e6c8ef85e6af. Delta 6cdebae5..92eb847f is one file, +152/−43.

Lance 13.0.0:

  • object_store(Some(id)) fills base_object_stores on first use. The field comment says clones share it. set_manifest clears it only when base_paths change.
  • checkout_by_ref returns self.clone() when already_checked_out is true. checkout_manifest and Dataset::with_object_store start an empty cache.
  • commit_transaction uses the in-memory manifest when version == read_version. It also uses that manifest when read_version is 0 and the commit is not a strict overwrite. Omnigraph manifest publishes use with_max_retries(0), so that arm does not apply to them.
  • The clone path reloads the source manifest from disk.

Main 36d3ea08: publish_conversion overwrites each branch. manifest_has_external_base_paths is true when base_paths is non-empty. full_manifest_object_store_path rebuilds a pin path from the opened root. The server maps every OmniError::Storage to internal. ErrorCode::Internal asks for refresh.

#1235 at 04605ccc: follow_opened_root calls with_object_store on the primary store, then assigns the new map on that fresh handle. has_external_bases is the relocation bit.

Open RFCs #1213, #1202, #1214, #1224, and #996 do not mention this contract. Their diffs have no base_path text and no relocation text. A registry-row overlap with #1202, #1213, and #1214 is only a merge conflict.

The stated tests can fail before the fix and pass after it. On main, a branched graph has a non-empty base_paths, so a self-contained assertion fails. On main, a missing inherited file is reported at the create-time path. The cache test on #1235 empties the source bucket after the store is resolved. An Arc swap then reads that empty bucket and fails. A fresh cache reads the copy and passes. I did not execute those tests. This pull request ships no code. GQT cannot copy a root.

Docs, on a detached worktree of 92eb847f: python3 scripts/check-docs.py with markdown-it-py==4.0.0 and mdurl==0.1.2. Result: Documentation OK (450 Markdown files checked).

Limitations

I did not run the implementation tests, S3, Azure, or the predecessor journeys. I did not re-check Lance 14 or 15. I did not push, and I did not change this pull request.

Inline comment on docs/rfcs/2026-11-03-relocated-graph-roots.md line 341 (RIGHT):

Optional. This sentence now sits on the claim that a same-version publish records the opened URI for every base, ancestors included.

A clone does not do that. do_commit_new_dataset (lance io/commit.rs) reloads the source manifest from ref_path. shallow_clone copies those disk bases and adds one base whose path is ref_path. Only that new base is the opened URI. Ancestors stay as they were on disk.

The next two bullets already split a stale publish from a new branch. Narrow this sentence so it does not say a clone rewrites ancestors. The three cases stay the contract.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant