[ragnorc]

Summary

Rollout step 1 (Engine) of remote optimize through the serving owner. It is stacked on #1175 and includes its commits; review the six commits after it, one per slice. The server record and routes (step 2, #1205, closed until acceptance) and the CLI served path (step 3, #1206, closed until acceptance) stack on it.

The RFC is back under review as #1214 after #1203 reverted its early merge. This PR stays a draft until the RFC is accepted, and follows any change the review makes.

optimize_as(actor, OptimizeRequest) returns an OptimizeReport. OptimizeRequest::default() is today's full run.

1. Gate-hold bound and per-table outcomes (feat(engine)!).
   • Order. A full-catalog run visits tables in type-key order from start_at and wraps. A tables request works in its own order and neither reads nor reports a resume position. Bad requests are refused before any effect.
   • Pipeline. Plans run ahead, and rewrites start in visit order within the planning and rewrite concurrency.
   • Gate hold. Once the gate-hold deadline passes, every table whose turn has not come is deferred(gate_hold_limit). The finished tables publish once, and resume_at names the first deferred table.
   • Tail. The configuration clear and the adjacency artifact report their own outcomes and never fail a published run. Every writer gate is released before the artifact scan.
   • CLI. --max-gate-hold and --tables are added. JSON reports the outcomes and drops the __manifest row and the always-null version fields (breaking note).
2. Before-publish hook. The shared publication helper calls an optional hook once: after the updates are prepared, and before the coordinator lock and the publisher's first attempt. The hook receives the intent nonce and the captured head. Its failure publishes nothing and is Unpublished. Other callers pass no hook.
3. Per-table failure isolation. A table's own planning failure (BeforeEffect) or staging failure (Unpublished) reports it failed with its error, and the other tables still publish. An uncertain failure still fails the run. The DST harnesses treat a failed table as the op's failure, so their obligations keep their meaning.
4. Cancellation.
   • Checks. OptimizeRequest::with_cancellation is checked before every plan, before every rewrite, and before the publication. It is also awaited in between, so a table in progress is dropped where it stands.
   • A cancelled run defers every table as cancelled and publishes nothing. It skips the configuration clear and the artifact, and leaves the resume position where it was.
   • A late cancellation arrives after the last check. The run finishes and reports arrived_late.
5. Resource ceilings and the run byte budget.
   • Ceilings. TableCeilings covers source fragments, rows and bytes, managed Blob bytes, and index rows. Each is measured from the table's compaction plan before its first write, fragments first, and only when set.
   • Sizes. Bytes are recorded data-file sizes, or a HEAD request through the collector's base rule, now shared as data_file_location and deletion_file_location. Lance 13 records no deletion-file size, so deletion files always take a HEAD.
   • Blob bytes come from the descriptor scan that compaction sizing already runs, refactored into one visitor. External references count nothing.
   • Over a ceiling. The table is deferred(resource_limit) with nothing written for it.
   • Budget. run_source_bytes is also the default table byte ceiling, and a larger explicit ceiling is refused. The first table that would pass the budget, and every later one, is deferred(byte_budget), and resume_at names it.
6. Adjacency artifact bound.
   • Pre-check. artifact_bytes is checked first against a lower bound: the fixed bytes at the shortest header, plus 8 bytes per live edge row.
   • Counting build. The graph-index build now streams one batch at a time and counts each batch's growth from the widest header. It stops before keeping another batch or allocating the adjacency arrays.
   • Outcome. Either stop is skipped(size_limit) and writes nothing.
   • Write. The artifact write runs in a spawned task the run awaits, so a dropped run cannot drop an issued write.

Tests

• maintenance.rs tests for:
  • the gate hold and resume position;
  • table lists and their refusals, including a table ceiling above the budget;
  • the before-publish hook: called once, its nonce answered by the commit, and a failing hook publishing nothing;
  • cancellation before the run and late cancellation;
  • each table ceiling, with a deletion file in the plan;
  • the byte budget, measured through a refusing hook, and its resume;
  • Blob bytes ignoring external references;
  • the artifact limit, both before the scan and during the counting build, writing nothing, then rewriting at its exact size.
• failpoints.rs tests for:
  • a failed configuration clear after publication;
  • one table failing while the other publishes;
  • cancellation while a table is parked mid-staging.
• Matrix: the optimize post-detached cell now asserts the run succeeds.
• Unit: a persist.rs test brackets the serialized size between the two bounds.
• CLI: the renderer test covers every outcome, including a failed table.
• Ran locally:
  • engine maintenance::, traversal_adaptive::, traversal_indexed:: and merge_cleanup::;
  • the optimize, cleanup and collector failpoints;
  • detached_commit_matrix, policy_engine_chassis, forbidden_apis and the seam guard;
  • the CLI renderer and optimize integration tests;
  • the DST cost golden (unchanged) and the maintenance obligation tests;
  • the GQT traversal cases;
  • Clippy in both feature graphs, cargo fmt --check, check-docs.py and typos.

[ragnorc]

Why, what, how, impact

Why. Remote optimize (RFC #1214) needs optimize to be bounded, cancellable and to report what it did per table.
What. optimize_as(actor, OptimizeRequest) returns an OptimizeReport. It adds a gate-hold bound, a before-publish hook, per-table failure isolation, cancellation, per-table ceilings plus a run byte budget, and an adjacency-artifact size bound.
How. One pipeline (run_table_pipeline) plans ahead, starts rewrites in visit order, and publishes once. Every stop becomes a typed outcome (Deferred(..), Skipped(..), Failed).
Impact. Direct optimize keeps its behavior under OptimizeRequest::default(). The CLI JSON output changes (breaking note present).

Reviewed head: 6db66e1e (merge base with main 14815c6e; main is now dd7a9f75). I read the six slice commits after #1175 in optimize.rs, table_ops.rs, graph_index/{mod,persist}.rs, collector.rs, table_store.rs and the CLI change.

Blocking

1. The branch conflicts with main. A merge-tree against dd7a9f75 conflicts in db/mod.rs, db/omnigraph/optimize.rs, db/omnigraph/repair.rs, tests/forbidden_apis.rs and tests/integration/maintenance.rs. #1270 changed the write path. Rebase, then re-run the optimize failpoint and DST maintenance tests, because the conflicts sit in the pipeline file.
2. Order. This PR contains #1175's commits and waits on RFC #1214. It cannot merge before either. This is a process block, not a code defect.

I found no correctness defect in the code at this head.

What I checked in the code

• Gate-hold stop: Bound(turns, ..) names the first unvisited table. turns counts tables popped in visit order, and ready is ordered, so resume_at is correct. The first table always starts.
• Byte budget: when a budget is set, every table is measured (table_source_bytes() falls back to the budget). A table ceiling above the budget is refused. The first table with work therefore always fits.
• Cancellation: a cancelled run publishes nothing and skips the configuration clear. The staged tables are deferred(cancelled). Dropped staging futures leave only unreachable detached commits. A late cancel is ArrivedLate.
• Failure isolation: only BeforeEffect and Unpublished evidence stays with the table. Any other evidence fails the run.
• Artifact bound: ArtifactCount is an upper bound on the written bytes (my reading, and the PR's unit test shows it within the header digits). The pre-check is a true lower bound. The spawned write task keeps an issued write from being dropped.
• Blob refactor: visit_row_blob_bytes keeps the old early stop. A table with no Blob column now returns early and gives Some(0), the same answer as before.

Optional

1. The before-publish hook runs under all writer gates and has no bound. max_gate_hold does not cover it. The doc says the writer wait is at most the hold, plus the last table, the publication and the configuration clear. A slow hook (a server record write) breaks that sentence. Either bound the hook with a timeout that fails as Unpublished, or add the hook to the documented wait.
2. index_rows may not see a rebuild that compaction causes. measure_plan gets index_work from the plan before any write. apply_optimize_table_effects calls stage_index_fold after the rewrite, on the new tip. If a rewrite alone makes an index lag, that rebuild can run over more rows than the ceiling allows. I did not test this; it is a question about the contract. A test with a vector or scalar index and a compaction-only table would settle it.
3. Liability of five parallel ceilings. Each ceiling touches an enum variant, as_str, a builder, a measure_plan arm, and the CLI and JSON output. A sixth measure repeats all of that. A small table of (measure, limit, measure_fn) would give one owner. Keep the typed output tokens.
4. Repeat cost of the artifact skip. When the artifact is over its limit and none exists, every later run repeats the count_rows pre-check and the counting scan, up to the limit. The cost is bounded by the limit. Say so in the maintenance doc, or cache the verdict by edge-table stamps.

Validation

Local, at the exact head 6db66e1e (shared target dir, 2 cores):

• cargo test -p omnigraph-engine --locked --test integration maintenance::: 61 passed, 0 failed.
• cargo test -p omnigraph-engine --locked --features failpoints --test failpoints optimize: 11 passed, 0 failed (includes cancel mid-run, per-table failure, failed configuration clear).

CI on this head: every check in statusCheckRollup is SUCCESS or SKIPPED, including CI Gate, Test Workspace, DST pinned suite and the GQT jobs.

Not run by me: clippy, DST cost golden, CLI tests, the persist.rs unit test. I rely on CI for those.

Limitation on the regression proof. This is a new API, not a bug fix. The base has no OptimizeRequest, so the new tests cannot compile there, and a compile failure is not a before/after proof. GQT cannot express optimize outcomes, gate holds or cancellation, so the Rust integration and failpoint owners are the right place. I did not run the performance and storage-cost claims; the PR says the DST cost golden is unchanged, and CI is green on it.