performance: a client cannot bound how long one change-feed page takes

Summary

A client cannot bound how long one change-feed page takes. GET /graphs/{id}/changes keeps adding commits to a page until it reaches the branch head, 128 commits, or the change and byte budgets. The page takes as long as all of its commits together, and on a hosted graph one commit can take 14–26 s. limit counts changes, so a small limit does not help when the first commit is the expensive one. The engine already has a commit ceiling, but no HTTP or CLI caller can set it. A client with a request timeout therefore fails on some pages, and a retry does the same work again.

Evidence

Measured on 2026-10-25 against a hosted server running 0.14.1-edge+7711e4a on S3-compatible object storage. The graph has 656 tables. Artifact, which several of these commits write, has 100,058 rows. Times are client wall time (curl time_total). Sync writers were committing on main during the runs.

One page of GET /changes?branch=main&limit=8192&start=after:<commit N back>:

Commits in the page	Changes	Time
1	349	25.8 s
4	706	51.7 s
16	722	53.6 s
32	744	49.9 s

The four newest commits read alone with GET /commits/{id}/changes, newest first:

Commit	Changes	Time
Merge	349	14.3 s, 18.4 s
Linear	1	1.2 s
Merge	350	5.9 s, 6.3 s
Merge	6, 3 of them on `Artifact`	17.8 s

The 1-commit page holds the newest merge. Through the feed it took 25.8 s; read alone it took 14.3 s and 18.4 s. I did not look into the difference.

Commit count does not predict cost. The 16- and 32-commit pages took no longer than the 4-commit page, because the older commits were cheap. What a page costs depends on which commits it holds. The expensive ones here are merges that touch Artifact, which take the full-table path of #969.

limit does not bound the time. The same 4-commit range with limit=50 took 21.1 s and returned 50 changes. It ended inside the second commit of the range, after the 6-change merge that costs about 18 s on its own. The rest of that commit then resumes mid-block, and each later page plans the commit again (#970).

Earlier the same day, on a branch with 9 commits since its fork, one page from the merge base took 56.4 s for 366 changes.

Effect on a client

OmniGraph Studio reads a branch review through this feed with a 60 s request timeout. Whether a review opens depends on which commits land in its first page. The server sets no deadline of its own, so the client's timeout is the only limit.

Code

At 7711e4a3:

The HTTP handler passes max_commits: None (handlers.rs#L4862), and ChangeFeedQuery has no commit parameter (lib.rs#L1214). The CLI's embedded path passes None too (client.rs#L1038).
None becomes CHANGE_FEED_DEFAULT_COMMITS_PER_POLL, 128 (feed.rs#L233, model.rs#L36).
The poll stops before a commit only on that ceiling or on an exhausted change or byte budget (feed.rs#L313). Nothing measures the time a page has used.

Directions

Expose the commit ceiling. Add a commit parameter (1 to CHANGE_FEED_MAX_COMMITS_PER_POLL, 512) to GET /changes, the MCP changes_poll tool and the CLI, and pass it through as max_commits. A client that needs short requests can ask for one commit per page. A page that stops on the ceiling ends at a block boundary, so its continuation is a cursor and pays no mid-commit restart. Page contents stay deterministic. It cannot shorten a single commit that takes 26 s.
A time budget per page. Stop adding commits once a page has used a set amount of time, at a block boundary, and always after at least one block. The continuation is the same block-boundary cursor. Clients would no longer need to guess a commit count. The cost is that where a page ends depends on timing, so two runs can split the same range differently. If page boundaries must be reproducible (DST replay, for example), the budget has to read the simulated clock, or option 1 is the safer one.

Neither option reduces what a commit costs. #969 and #970 do that. These options only let a client keep each request under its timeout while those costs remain.

performance: graph diff and branch review read both full tables unless the endpoints are one transaction apart #969: the full-table comparison that makes one commit cost seconds.
performance: every change page restarts its comparison (re-plan, re-resolve endpoints, re-sort keys) #970: every page restarts its comparison, so splitting a heavy commit with a small limit costs more in total.
performance: a commit lookup that misses the current branch checks out every live branch's __manifest #972: commit-id lookups that scan every live branch. They cost seconds on the diff route; I did not check whether a feed poll pays them.
changes: wait for a branch to move instead of polling #1056: wait for a branch to move instead of polling.

Limitations

One client and a live graph with writers committing during the runs. Each row ran once or twice, and I did not control cache state. The 9-commit branch was deleted later the same day, so that run cannot be repeated. The main series can be repeated, but main keeps moving, so "N commits back" names different commits on another day.

Summary

A client cannot bound the time of one GET /graphs/{id}/changes page. The engine already stops a poll on a commit ceiling. The HTTP handler, the CLI and MCP never set that ceiling. limit counts changes. On a full-table commit it does not shorten the page. On a touched-fragment commit a small limit can make the same commit slower, because the scan batch is limit + 1.

This is a partial reproduction. The code claims match current main. The local page cost matches the sum of its commits, and limit does not cut the full-table commit. The hosted 14–26 s S3 times were not repeated.

Reproduction

Verdict: partially reproduced.

Latest main at comment time is acb1f8c3f5d6034577a55146194881128300b932 (fix(server): remove crypto-bigint 0.5 for RUSTSEC-2026-0387, #1151). The measurements ran on d5bf9fe4e31adb4ea15624b05eb42f5a50dbe66e, which was main when the run started. git diff d5bf9fe4..acb1f8c3 is empty for the change-feed, handler, API-type, CLI and MCP files cited below. Those files are also unchanged since the reported server commit 7711e4a3.

Environment: Linux, Rust 1.97.1, local temporary directory, debug (unoptimized) cargo test. Not the hosted 0.14.1-edge+7711e4a server, not S3, not 656 tables, not 100,058 rows, and no concurrent writers. The CLI check used target/debug/omnigraph, which prints omnigraph 0.14.1.

Code claims (confirmed)

server_changes_feed passes max_commits: None and max_bytes: None (handlers.rs).
ChangeFeedQuery has limit and no commit parameter (lib.rs). CHANGE_FEED_PARAMS is the same list. An unknown query name, including max_commits, is rejected with 400 (handlers.rs).
The embedded CLI passes max_commits: None (client.rs). changes poll has --limit and no commit flag (args.rs). MCP changes_poll has the same gap: its schema has limit, and the tool calls server_changes_feed (catalog.rs, mcp.rs).
None becomes CHANGE_FEED_DEFAULT_COMMITS_PER_POLL (128). The hard ceiling is 512 (model.rs, feed.rs).
Before it opens the next commit, the poll stops only when the commit index hits that ceiling or the change or byte budget is already spent (feed.rs). feed.rs has no clock. The server timeout covers the request body before admission, not this handler.
The engine comment states the gap directly: the byte and commit ceilings exist for SDK callers and tests, and the transports do not expose them (model.rs).

CLI, on this checkout:

$ target/debug/omnigraph changes poll --max-commits 1 --start now
error: unexpected argument '--max-commits' found

--help lists --limit and does not list a commit ceiling.

Local measurements

Temporary test, not committed: cargo test -p omnigraph-engine --test repro_issue_1157 --locked -- --nocapture --test-threads=1 on d5bf9fe4. The test calls poll_change_feed the way the handler does (max_commits: None, max_bytes: None). It fails because that call returns every remaining commit.

Graph: 40,000 Person rows in five loads of 8,000 (one load cannot exceed 8,192 keyed rows). Branch feature appends two new people in two commits. main appends one Note, so the histories diverge. The merge commit's first parent is that note. Three more one-row Note commits follow the merge. The merge Person registration moves from published version 6 (40,000 rows) to published version 8 (40,002 rows).

commit changes of the merge commit, second samples in one process:

`limit`	Time	Changes	Stop
1	258 ms	1	truncated
2	254 ms	2	complete
16	254 ms	2	complete
256	254 ms	2	complete
8192	257 ms	2	complete

A one-row Note commit at limit=8192 takes 3.9 ms. The merge has two changes and takes about 65 times longer. limit does not change that time. The full ordered merge opens both sides with the default scan target, not the page limit (candidate_scan.rs). The key sort still covers the table. That is the #969 shape: the branch tip is more than one transaction from main's pin.

One feed poll after the merge's first parent:

Call	Time	Blocks	Changes	Stop
`limit=8192`, `max_commits=None` (the handler)	266 ms	4	5	head
`limit=1`, `max_commits=None`	258 ms	1	1	mid-block
`limit=8192`, `max_commits=Some(1)`	260 ms	1	2	block boundary

The handler-shaped page time (266 ms) matches the merge (257 ms) plus three one-row commits (about 4 ms each). The page cost is the sum of its commits. max_commits=1 keeps one commit and stops on a block-boundary cursor. The handler cannot ask for that. limit=1 also returns one change, in the same ~260 ms, and stops mid-block. A retry of that token plans the commit again (#970).

The failing assertion on d5bf9fe4:

assertion `left == right` failed: client cannot bound the page to one commit:
handler-shaped poll returned 4 blocks in 266.02486ms
(merge alone 256.579713ms, cheap 3.910158ms)
  left: 4
 right: 1

A second graph, same 40,000 rows, used one feature commit that updated p000000 and inserted z-new. The merge Person registration moved only from published version 6 to 7. That commit follows the touched-fragment batch, which is limit + 1 (row_compare.rs):

`limit`	Time	Changes
1	2.33 s	1, truncated
2	1.66 s	2, complete
16	324 ms	2, complete
256	47 ms	2, complete
8192	24 ms	2, complete

A smaller limit made this commit slower. It did not bound the time.

What differed from the report

The hosted series (1 commit 25.8 s, 4 commits 51.7 s, limit=50 still 21.1 s, one merge 14–26 s on S3) was not repeated. Local debug on 40,000 rows puts the full-table merge at about 0.26 s. The shape matches: commit count does not set the cost, the expensive commit dominates the page, and limit does not remove it. Studio's 60 s timeout was not measured here. No live HTTP request was sent. The handler source is the HTTP proof. #972's branch-list cost was not measured on this graph.

Root cause

One poll walks commits in order. It pays each commit before it looks at the next one. The stop before the next commit is the commit ceiling or an exhausted change or byte budget. The transports leave the ceiling at the default 128 and leave the byte budget at the default 4 MiB. limit is only the change budget. The clock is not a budget.

A full-table commit pays the ordered key walk up front, so a small limit still waits for that walk and then stops mid-block. A touched-fragment commit sizes its scan from limit, so a small limit can increase the number of batches. Neither behavior gives the client a time bound.

The engine path is already correct for a caller who can set max_commits. change_feed_commits_examined_ceiling_bounds_sparse_polls and change_feed_small_ceiling_poll_is_bounded_across_backlog_depths lock that. The defect is the boundary: HTTP, CLI and MCP do not pass the value.

Recommended fix

Expose the existing commit ceiling. Add max_commits (1 to 512, default 128) to GET /changes, changes poll, and MCP changes_poll, and pass it through as ChangeFeedRequest.max_commits. A page that stops on the ceiling already ends on a block-boundary cursor. The next poll does not open that commit again, so it does not pay a mid-commit restart. The split is deterministic: the same range, the same ceiling, and the same cursor. DST can replay it. A client that must stay under a timeout asks for one commit per request.

This does not shorten one commit. The full-table merge above costs ~260 ms at every limit, including limit=1. #969 is the change that removes that cost (prove a chain of row-set-preserving transactions, or skip rows the key scan shows unchanged). #970 is the change that makes a mid-block token cheap. Until those land, max_commits=1 is how a client keeps each request to one commit and still resumes at a boundary.

A server time budget can sit in the same stop, before the next commit opens, after at least one block. It must return the same block-boundary cursor. It must not stop mid-commit on the clock. A mid-commit token makes the retry re-plan the commit (#970), so a timeout retry does more work than the first request. One commit that itself exceeds the deadline still runs until its change or byte budget. Only #969 shortens that commit. A clock split is not deterministic. Tests that replay page boundaries should use max_commits. A server deadline is a backstop for a caller that leaves the ceiling at 128, so one request cannot add up many expensive commits. It must return a cursor. A 408 with no cursor repeats the same work.

Do not use limit as a time control. Do not abort the handler with no cursor. Do not queue the page. The feed is one read of one captured cut. A queue would be a second authority for work the manifest already defines.

What this choice gives up: it does not make one commit cheaper. That is #969. It also does not use the clock as the primary contract. A clock split is harder to test and harder to explain, and two runs can divide one range in two ways. The commit ceiling costs more round trips when a client sets it to 1. Those round trips are the bound. Each one stays inside the client timeout and resumes at a boundary. That is the right trade for a client that fails at 60 s. Clarity of the cursor and a hard request bound come first. The speed of one merge diff stays with #969 and #970.

Invariant 11 requires a bound on resource use. The change budget, the byte budget and the hidden 128-commit ceiling bound how much one poll examines. The caller cannot choose the commit bound, and nothing bounds time. A client timeout fails the request and returns no cursor. The failure is loud, and the client still cannot make progress on that page without a larger timeout. Exposing the ceiling uses the mechanism that is already tested. It adds no storage format and no second record of the feed. A time budget that returns a boundary cursor has the same property. A time budget that returns a mid-block token increases the work of a retry. That is the wrong failure.

Lance

Pinned Lance is unmodified 13.0.0 (v13.0.0, 2026-10-07). Unreleased v14.0.0-rc.2 and v15.0.0-beta.1 (both 2026-10-22) do not add this page bound. The bound is OmniGraph's. Scan work in the v14 notes can change the cost of one Lance scan, for example lance#9750 (late materialization under a top-k), lance#9409 (CDF fragment prune by row version) and lance#10042 (one filter plan per scan). The full-table change path does not pass a row limit into the ordered scan, so a top-k improvement does not apply until OmniGraph asks for one. None of these replace #969 or the missing parameter. Open Lance pull requests reviewed for this (cleanup retention, scan coalescing, full-text) do not implement the contract.

Related work

performance: graph diff and branch review read both full tables unless the endpoints are one transaction apart #969: full-table comparison when the two pins are not one transaction apart. This reproduction used that path. It does not let the client stop before the next commit.
performance: every change page restarts its comparison (re-plan, re-resolve endpoints, re-sort keys) #970: every continuation re-plans. A limit stop inside the expensive commit pays that cost again.
performance: a commit lookup that misses the current branch checks out every live branch's __manifest #972: a commit lookup that misses the branch buffer checks every live branch. Not measured on this local graph. The feed poll here addressed commits by the captured cut.
changes: wait for a branch to move instead of polling #1056: wait for the branch to move. That does not bound one page.
Open pull requests: feat(mcp): expose every non-streaming graph operation as a tool #1139 adds MCP tools. Its changes_poll schema still has limit and no commit ceiling. rfc: request-aware plan cost #996 prices planner requests. It does not bound this feed. No open pull request implements this issue. fix(server): remove crypto-bigint 0.5 for RUSTSEC-2026-0387 #1151, which moved main during this run, changes a server dependency only.

No other open issue covers this defect with a wider scope. #969 and #970 explain the cost of one comparison. They do not supersede this issue. This issue stays open.

Limitations

One local debug process, 40,000 rows, two node types, a temporary directory. Each timed call was repeated in-process, so the figures above are warm. The first full-table read in a fresh process was in the same band (about 0.25 s), not the hosted 14–26 s. The two-commit append merge is one way to reach the full-table path. It is not a copy of the hosted merge history. The candidate-path table is a separate graph. The failing test was removed and was not pushed.

Navigation Menu

Repository navigation

performance: a client cannot bound how long one change-feed page takes #1157

Summary

Evidence

Effect on a client

Code

Directions

Related

Limitations

Summary

Reproduction

Code claims (confirmed)

Local measurements

What differed from the report

Root cause

Recommended fix

Lance

Related work

Limitations

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

performance: a client cannot bound how long one change-feed page takes #1157

Description

Summary

Evidence

Effect on a client

Code

Directions

Related

Limitations

Activity

ragnorc commented on Oct 25, 2026

Summary

Reproduction

Code claims (confirmed)

Local measurements

What differed from the report

Root cause

Recommended fix

Lance

Related work

Limitations

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions