Repository navigation
Conversation
* rfc: remote optimize through the serving owner A bounded, manually requested optimize of a served graph's main branch, executed by the process that serves and publishes it: today's gate shape and one publication, a gate-hold bound with an exact outcome per table, a durable operation record under a caller-chosen id before any effect, lookup by that id after a lost response, and restart resolution from the graph's history. Replaces the closed served-maintenance proposal in the form its review asked for. * rfc: link the remote optimize RFC to its discussion * rfc: revise remote optimize after review Record the intent nonce and resolve by commit_id_answers; keep an in-doubt publication non-terminal; release the writer gates before the artifact scan; add a resume position, resource ceilings and per-table failure isolation; require maintain on every route; depend on ModernRelay#1175 and ModernRelay#1176. * rfc: settle remote optimize resume, ceilings and resolution after second review * rfc: settle remote optimize resolution reads and the artifact bound after third review * rfc: resolve through the buffered commit lookup and count both offset sides after fourth review * rfc: state the history cost of a released commit and reuse the collector's path resolution after fifth review
Adapt rather than take ModernRelay#1186's ceilings and list every difference; share the record's bounds with the deployment receipts; cap the resolution walk per attempt; name the exit from a full unresolved list; justify the callback over a two-phase engine API; narrow the containment claim; skip the artifact write under a pending shutdown and size its limit against the grace at qualification.
…path The HTTP request carries tables and max_gate_hold only; resource ceilings and the run budget are serve settings. The record is keyed by kind and holds no event history, progress, pause or resume. The out-of-process worker is deferred rather than rejected: Lance's compaction task API already separates planning, execute and the commit, and a later slice moves execute alone.
A §3.6 table names each hazard, the mechanism that owns it and where it is decided: the cluster lock, the S3 lease, the manifest version CAS, the record's conditional puts, detached staging, pinned reads, the in-process gates, standalone roots, raw writers, the other maintenance operations and a later worker. No mechanism is added.
|
Recommendation: revise. WhyA served graph cannot run WhatThis pull request is a design document. It adds the RFC and three pending amendments. It changes no executable code. The current head is HowAcceptance is a conditional put of the operation record. The run holds today's writer gates, works one table at a time, and publishes the finished tables once. The record stores the intent nonce before any manifest request. A ImpactThe serving process is the right owner. The findings on #1190, through the ready review of Two sentences still disagree with the catalog on this merge base. An implementation that follows them builds a second parent-chain walk, and it can leave a full unresolved list stuck after a write on the wrong graph. Fix those sentences before an implementation follows them. Keep #1186 closed. Two accepted optimize RFCs would leave two contracts. What the earlier discussion still holdsChecked again on merge base
Blocking1. Resolution needs the catalog's parent-chain page§2.4 walks main by calling
The resume sentence has a second defect. "Resumes the walk below that commit" drops the stored commit. That is sound only when the stored commit was already tested and was not the input head's child. A skipped child makes the walk pass the input head. §2.4 then stays in Call 2. One write settles the records of that graph onlyThe unresolved list is cluster-wide, with a cap of four. §2.4 already says that only a commit on this graph's main settles a record. A commit on another graph does not. The exit paragraph then says that any mutation, load, or merge that publishes on main settles every unresolved record at once. That is true for the records of the graph that moved. It is false for a record whose graph did not move. One cluster can hold doubts for more than one graph. A write on the wrong graph leaves the list full, and admission stays at 409. Say the graph in the operator instruction. A move of graph G's main head settles the unresolved records whose input head is on G. The same sentence should cover the earlier line that another maintenance publication "on main" settles the doubt. The active slot is cluster-wide. Settlement is per graph. Optional1. A late manifest put can still create the absent version§3.6 says a late manifest put meets a held version and fails as a conflict, so it never overwrites the successor's commit. The put-if-absent in 2.
|
|
Design spike for this RFC: correct with listed amendments. Branch: https://github.com/ragnorc/omnigraph/tree/spike/remote-optimize-1214. The report is VerdictCorrect with listed amendments. The overall design holds under every
Amendments
Per-claim resultsTest names are in
Baseline runs on this branch: engine Not verified
What was builtThe spike starts from the three stacked implementation drafts: #1204
Stubs and limits.
Running the spikecargo test -p omnigraph-server --test spike_remote_optimize -- --test-threads=1 --nocapture
cargo test -p omnigraph-engine --features failpoints --test integration spike_remote_optimize:: -- --nocapture
cargo test -p omnigraph-cluster --test spike_record_bounds -- --nocapture |
Walk main with the catalog's first-parent page and store its unvisited next as the cursor; settle unresolved records per graph; let a graph's deletion settle them as graph_deleted; write the acceptance from the registered operation; make the record bounds compose; capture authority again under the held gates; refuse an id at or before the eviction mark; state when a late manifest put lands; answer an over-ceiling max_gate_hold with 400.
Name the control object operations.json and the id operation_id; make lookup and cancel generic routes under /operations/ while each kind keeps its start route; add §2.6 stating what the record owns and what a kind supplies, with optimize as the first kind and an asynchronous merge as the named next one; record why a job registry with a JSON spec, events and pause stays with the control plane.
|
Recommendation: revise. WhyA served graph cannot run WhatThis pull request is a design document. The reviewed range adds no executable code. The current head is Prior findings
Blocking1. Graph deletion can restore the full-list wedge, and it can close a doubt while history can still answer§3.4 settles each unresolved record of the deleted graph as The deletion protocol already has one resume owner. The outstanding deployment stores the root and the schema contract. It writes A crash after the root is gone and before the operation-record put, or a resume that completes The set of records is also short. From step 3 of §2.3 until step 4, a
Name the lifetime Required rule:
The testing row that deletes the graph should cover the resume after 2. The generic record does not fit the second kind used to justify it§2.6 says §2.1 to §2.5 are generic, a kind cannot change the walk, and a later kind adds a value and a row, not a file. The merge column is there so the split can be checked. The check fails.
Pick one shape before implementation freezes the file. Either the record in this RFC is the maintenance record: one active slot, settlement on main, and §2.6's merge column is a later change to the header and to §2.4. Or the header stores active operations by admission scope, and §2.4 takes the kind's settlement branch. A format written the first way and described the second way is the liability five later kinds will pay. 3. Two submissions can both pass before either acceptance put§3.3 registers the owned operation first and makes the acceptance write that operation's first step. That closes the spike's hole, where a disconnect between the put and registration left an accepted id that never ran. The same paragraph says the admission slot is the record's active field, and that the record and the registry agree. Those two slots do not agree until the put lands. Deployment does not use the ledger field for this exclusion. Say the same thing here. The object the next admission reads has to include the registered operation, and the check and the registration have to hold one gate across the acceptance put. The record field becomes the slot after that put. Until then, the in-process owner is the slot. 4. The eviction mark does not stop a retry of the evicted id§2.5 stores the server acceptance time of the most recently evicted record. It compares that time with the ULID time inside the caller's A client clock ahead of the server mints an id whose ULID time is later than the acceptance time the server stores. After that record is evicted, a retry of the same id has a ULID time after the mark. The id is no longer in the file, so admission treats it as new and runs the work again. That is the replay this mark exists to stop. A client clock behind the server mints a fresh id whose ULID time is at or before the mark. That fresh id is refused. The "never a fresh one" sentence does not hold. Store the ULID time of the evicted Optional1. A missing link is not a short page
2. The pending server-runtime amendment still names the old objectThis pull request's amendment on 3. The new authority test drops the served limit§1.2 recaptures authority under the held gates for the served run. The direct path keeps today's capture, and a mismatch there still refuses. The new testing row says a write that lands between the request and the gates does not refuse the run, with owner
Tradeoff and liabilityThe serving process remains the right owner. One publication remains the right door. Keeping today's gates, with a finite hold, remains the right first cut. This slice gives up low write latency during a run. A new gate shape would be a second design. The resolution walk now has one owner, The new generic layer adds liability the optimize-only record did not have. The durable header, the walk, the cap, and the grant are each specified twice, and the two specifications disagree. Five later kinds that follow the header will not fit the §2.6 rows, and five kinds that follow the rows will each revise the file. The callback on Graph deletion adds a second writer of a terminal fact beside history, outside the deletion state machine. A terminal The eviction mark adds a clock comparison that does not own the anti-replay rule. The rule is "this id, once evicted, does not run again". The id's own ULID time is the value that states that. The server acceptance time states a different fact. No GQT case fits this RFC. The contract is the control object, the HTTP admission, and completion evidence. A ValidationReviewed head
Code and Lance reads on
Open work read as context, not as shipped behavior: #1175 and #1176 are still open and are still the dependencies this RFC names. #1184 is still open. #1204 is the open engine draft. #1205 and #1206 are not open. #1217 and #1221 change lease fencing and are not a requirement on this text. LimitationsEngine tests were not run. They do not exercise this document. The design spike's file:// runs were not re-run here. S3 and Azure were not executed. I did not re-fetch the Lance transaction page; the pinned crate source is the check for the late-put sentence. I read the deletion and admission paths this delta now depends on. I did not re-review #1175 or #1176 line by line. Recommendation: revise. |
The review of the generic record found the header optimize-shaped while the prose called it generic. The header now keys active operations by admission scope and each kind's state by graph and kind, a record stores its settlement branch, and the resolution walk, the exit from a full unresolved list and the grant on lookup and cancel follow the record. Graph deletion settles records as a step of complete_graph_deletion on both passes, before Deleted, covering an active publishing record, walking while the root is readable and writing graph_deleted only after verified absence; the lifetime is schema_identity_domain. Each scope has an in-process gate held across the acceptance put, so two submissions cannot both pass the active-slot check. The eviction mark is the evicted id's own ULID time. Ok(None) and Err from first_parent_page both keep a record publishing. The pending server-runtime amendment names operations.json, and the authority testing row is the served path.
The record, file, routes and id are operations. Maintenance names one admission scope, the kinds the maintain grant covers; seven lines still read as if it named the record.
|
Recommendation: revise. WhyA served graph cannot run WhatThis pull request is a design document. The reviewed range adds no executable code. The current head is Prior findings
Blocking1. Deletion can close a doubt the history can still answer§3.4 walks while the root is readable, then writes
Two endings leave the record
The deletion row should add two cases. The child sits deeper than one page, and the walk reaches it before the root is removed. A failed history read with the root still present leaves 2. The eviction mark can move backward, and it can refuse a record the file still holds§2.5 stores the ULID time of the most recently evicted One sequence:
The mark is the greatest ULID time of any id this file has evicted. A put that evicts ids sets the mark to the greater of the stored mark and those times. The mark never decreases. An absent mark still refuses nothing. Admission matches an id the file still holds before it reads the mark. That POST returns the stored operation. The mark applies to an id the file does not hold. The same id is then at or before the mark and is refused. A fresh id at or before the mark is refused too, and that caller mints an id with a later ULID time. The testing row should add both cases. Thirty-two later ids from one clock leave the mark equal to the last evicted id, so that row stays green on this text. Optional1. Branch deletion is named and has no owner§2.4 says deletion of a non-main settlement branch settles that record. §2.6 names the reason The later kind adds the step there, in the shape of §3.4. Walk while the branch is readable. Write The merge row also says the kind bounds the number of scopes, and it gives no number. The size proof in §2.2 is one scope plus four unresolved records. The later RFC names the number. Until then the proof covers the maintenance scope only. Tradeoff and liabilityThe serving process remains the right owner. One publication remains the right door. Today's gates, with a finite hold, remain the right first cut. This slice gives up low write latency during a run. A new gate shape would be a second design. The header change removes the split between generic prose and an optimize-shaped file. Cleanup, repair, and full-text rebuild add a kind and a start route inside the maintenance scope. They pass the same callback. They call The deletion step belongs in The eviction mark should stay one field. A list of evicted ids would be a second retention store. The field is a high-water mark, because the terminal list and the ULID time are different orders. An id the file still holds is answered by the file. The mark is only the rule for an id the file has dropped. The scope gate copies the deployment gate. It adds no lock type. No ValidationReviewed head
This RFC changes no behavior a
Code reads on
Main since the merge base renames Open work read as context: #1175 and #1176 are still the dependencies this RFC names. #1184 is still open and still does not touch this walk. #1204 is the open engine draft. #1205 and #1206 are not open. #1217 and #1221 change lease fencing and are not a requirement on this text. LimitationsEngine tests were not run. They do not exercise this document. The design spike's file:// runs were not re-run here. S3 and Azure were not executed. I read the deletion, admission, and commit-graph paths this delta depends on. I did not re-review #1175 or #1176 line by line. Inline comments
Blocking. The walk has to finish, or fail closed, before the root is removed.
A page that stores Repeat pages until the record is
Blocking. The mark has to be the greatest evicted ULID time, and a held id has to match first. The terminal list evicts the oldest entry. ULID time follows the caller clock. Replacing the mark with the latest eviction lets the mark move backward. A retry of an earlier id is then newer than the mark and runs again. The same replacement refuses a POST of an id the file still holds, because §3.2 step 2 runs before the same-id match. Set the mark to the greater of its current value and the ULID times evicted in that put. Match an id the file holds before reading the mark. Apply the mark only to an id the file does not hold. Recommendation: revise. |
Reopens the remote-optimize RFC for review. It merged early as #1190 (still
status: draft) and #1203 reverted it; this is the same text plus the fixes its post-merge review asked for, recorded in the decision log under 2026-10-29 "Reopened":resolution_walk_limit), resumed below the stop commit and reported by lookup;before_publishcallback was preferred over a two-phase engine API;RecoveryRequiredonly;Trimmed after comparing with LanceDB's served maintenance (its embedded
optimize()is a client-side call, its remote tables refuse it, and its managed service runs compaction off the request path as registry jobs with durable ids, cancel and reattach):tablesandmax_gate_holdonly; resource ceilings and the run budget are serve settings alone, as docs(rfc): optimize graphs through the running server #1186 had it;kindand holds no event history, progress, pause or resume; a job registry is the control plane's;plan_compaction, serializableCompactionTask,execute, then the commit) already separates the stepsTableStoreuses, so a later slice movesexecutealone while the owner keeps the plan and the one publication.A new §3.6 states every cross-process hazard once, in one table: the cluster lock, the S3 lease, the manifest version CAS, the record's conditional puts, detached staging, pinned reads, the in-process gates, standalone roots, raw writers, the other maintenance operations and a later worker. No mechanism is added.
A bounded, manually requested
optimizeof a served graph's main branch, executed by the process that serves and publishes it: today's gate shape and one publication, a gate-hold bound with an exact outcome per table, a durable operation record under a caller-chosen id before any effect, lookup by that id after a lost response, and restart resolution from the graph's history.Competing draft: #1186 (a synchronous call with no record). Only one of the two should be accepted; the comparison is in this RFC's Relationship to open work and Alternatives, and on #1186.
Blocked on #1175, #1176 and an in-region measurement, as the front matter says. Implementation drafts: #1204 (engine); the server and CLI slices reopen after acceptance.
Docs-only;
python3 scripts/check-docs.pypasses.2026-11-01. Revised after the reopened review and the design spike (both comments on this PR): the resolution walk is
CommitGraph::first_parent_pagewith the page's unvisitednextas the cursor; settlement is per graph; a graph's deletion settles its unresolved records asgraph_deleted(chosen over refusing the deployment); the registered operation writes the acceptance; the record bounds compose (terminal-only total, 512-bytedetail, outcomes summarized past 1,024 tables); authority is captured again under the held gates; an id at or before the eviction mark is refusedmaintenance_id_expired; §3.6 says when a late manifest put lands; an over-ceilingmax_gate_holdis 400. Amendment 5 of the spike (artifact write checked at issue) was already in §1.8 and §3.4 and is an implementation note for #1204.2026-11-01, second revision. The record is now generic:
__cluster/operations.json,operation_id, generic lookup and cancel routes under/graphs/{graph_id}/operations/, and a per-kind start route. New §2.6 states what the record owns (id, acceptance before effect, states, cancel, lookup, retention, eviction mark, nonce and the history walk) and what a kind supplies (start route and typed request, grant, admission scope, settlement branch, typed outcomes, kind-local state, settlement when what it names is gone), withoptimizeas the first kind and an asynchronous merge as the named next one, not in this RFC. §Alternatives records why a job registry with a JSON spec, events, pause and resume (LanceDB Distributed's/v1/jobs) stays with the control plane.2026-11-01, third revision. After the review of head 2413b5a, all seven findings applied. The header now matches §2.6: active operations keyed by admission scope, kind state keyed by graph and kind, the settlement branch stored on the record and walked by §2.4, lookup and cancel authorized by the record's kind. Graph deletion settles every non-terminal record as a step of
complete_graph_deletionon both passes beforeDeleted, walking while the root is readable and writinggraph_deletedonly after verified absence; the lifetime isschema_identity_domain. Each scope has an in-process gate held across the acceptance put. The eviction mark is the evicted id's own ULID time. Bothfirst_parent_pagefailures keep a recordpublishing. The server-runtime pending amendment namesoperations.json, and the authority testing row is the served path.