From f58b7b77f7a7dd975dbda280bfc121ef078a7205 Mon Sep 17 00:00:00 2001 From: Ragnor Comerford Date: Mon, 12 Oct 2026 12:16:09 -0700 Subject: [PATCH 01/11] rfc: add graph diff, branch review and merge dry run to the change feed contract Amend RFC 0030 with a public graph diff over two read targets, a branch review form against the merge base, `branch merge-base`, and a read-only `branch merge --dry-run`. Keep `commit changes` as the cause-carrying unit, retire the table-shaped `ChangeSet`, and reshape C6 around the per-commit schema content storage format 14 retains. Adds D0 evidence gates, D1-D3 phases and a decision log recording how each premise was checked. --- docs/rfcs/0030-cdc-time-travel.md | 371 +++++++++++++++++++++++++----- 1 file changed, 311 insertions(+), 60 deletions(-) diff --git a/docs/rfcs/0030-cdc-time-travel.md b/docs/rfcs/0030-cdc-time-travel.md index 115121721..4abbeb858 100644 --- a/docs/rfcs/0030-cdc-time-travel.md +++ b/docs/rfcs/0030-cdc-time-travel.md @@ -7,7 +7,7 @@ implementation: partial authors: - OmniGraph maintainers created: 2026-08-05 -updated: 2026-10-04 +updated: 2026-10-12 discussion: null supersedes: [] superseded_by: [] @@ -28,7 +28,8 @@ blocked_on: [] > authority and fallback, cursors, and retention gaps. C0–C3 are implemented. The §4.4 ordering, client-pagination, and -continuation-size gates are closed as recorded in §14; C4+ remain design-stage. +continuation-size gates are closed as recorded in §14. The graph diff phases +D1–D3 (§8) and C4–C6 remain design-stage. **Depends on:** RFC 0013 Phase 7 graph lineage, RFC 0022 snapshot capture and publication, RFC 0023 exact-`id` table fencing, and RFC 0028 immutable table @@ -50,7 +51,9 @@ exact logical before/after values. Public results do not expose tables, datasets, fragments, physical versions, row addresses, or storage ordering keys. -Two public surfaces share one private adjacent-commit enumerator: +Two shipped public surfaces share one private adjacent-commit enumerator, and a +third, design-stage surface shares its type-lifetime pairing and image +comparison: 1. A **finite commit entity diff** compares one exact graph commit with its first parent. It is an inspection result. A `next_page_token` may continue a @@ -59,6 +62,9 @@ Two public surfaces share one private adjacent-commit enumerator: first-parent order. Its cursor means the caller has consumed every complete commit through one position; it advances only after the final page of that commit. +3. A **graph diff** (§8, design-stage) compares any two graph read targets, or + a branch with its merge base, and returns the net entity changes between + them in the same entity shape, without a cause. A page token and feed cursor are deliberately separate. The token means "continue this response." The cursor means "resume the durable feed after this @@ -91,10 +97,11 @@ carry their endpoints as graph references. Entity-data diff and graph-schema diff are separate contracts. A schema change is not translated into synthetic row inserts, updates, or deletes. Historical schema selection used to decode a retained row is private correctness -machinery, not a user-facing schema-history feature. Because current lineage -does not retain accepted SchemaIR for every historical commit, the initial -entity surfaces fail with a typed error at an unproven schema boundary. A -separate schema contract is deferred to §10. +machinery, not a user-facing schema-history feature. Every commit written at +storage format 14 or later names the schema content it was written under, so +the authority historical decoding needs now exists (§2, §10.1). The shipped +entity surfaces still fail with a typed error at a schema boundary; crossing +one is phase C6. ## 1. What Lance gives us—and what it does not @@ -159,9 +166,18 @@ The required durable authority already exists: not a second registry or a table identifier. - Every current v6 graph table is created with stable row IDs and exact non-null logical `id` as its unenforced primary key. -- `diff_between` and `diff_commits` already know how to compare two graph - snapshots, including table creation, removal, rename, and cross-lineage ID - comparison. +- `diff_between` and `diff_commits` compare two graph snapshots, including + table creation, removal, rename, and cross-lineage ID comparison. They are + embedded Rust APIs only: no CLI, HTTP or MCP surface calls them, and they + return the table-shaped `ChangeSet` that §8 retires. +- Every graph commit record written at storage format 14 or later carries the + accepted schema identity it was written under and names its schema content + (`schema_content_hash`), an immutable `__history/schemas/` object holding + the contract head, the `.pg` source and the IR. A graph upgraded from an + older stamp shares one contract across its pre-upgrade commits + ([schema contract in the manifest](2026-09-30-schema-contract-in-manifest.md)). +- `CommitGraph::merge_base` computes the merge base branch merge classifies + against; no public surface reports it. Two version names require precise scope: @@ -620,34 +636,156 @@ cold history cost and state explicitly that writer-clock timestamps are not a linearizable real-time oracle. It may not introduce a binary search without a proven monotonic index. -## 8. Relationship to existing diff - -`diff_between` remains the direct net-current comparison API while the feed is -built. It does not receive optional cause fields: a range collapsed across -multiple commits has no single honest actor or commit. - -Its current table-shaped `ChangeSet` / `EntityChange` fields are transitional. -The canonical public entity shape in §3 does not carry `table_key`, table or -incarnation IDs, manifest/table versions, or physical ordering fields. C2 -introduces the finite adjacent-commit entity diff with that graph shape rather -than expanding the table-shaped result into HTTP/OpenAPI/SDK contracts. - -C0 may still use immutable table identity to locate data internally. Public -ordering and output are by graph type identity and logical entity ID. - -Once the graph feed is proven, its exact adjacent enumerator supplies most of -the same machinery. That does **not** make arbitrary-range net diff a free -operation algebra. Update-then-revert must disappear; delete-then-reinsert may -be update relative to the range baseline; table lifetimes can change; and -intermediate history may be reclaimed while both endpoint snapshots remain -readable. - -Any future feed-backed net diff must therefore reduce against baseline -membership/images and final membership/images, not merely fold operation -labels. The direct endpoint snapshot algorithm remains a valid and often -lower-cost reconciliation path. Both paths share graph type-lifetime interval -construction, canonical image comparison, and a conformance suite; they do not -grow separate definitions of insert, update, or delete. +## 8. Graph diff and the existing net diff + +### 8.1 What the shipped surfaces leave open + +The commit diff and the feed answer what one commit changed and what a branch's +first-parent history changed, commit by commit. Two questions have no public +answer: + +- **Reviewing a branch.** Branches exist so that a change can be inspected + before it merges, but no surface reports what a branch changed since it left + its target. A caller can find the fork commit by comparing two `commit list` + outputs and poll the branch's feed after it. That result is per commit, not + net, and once the target has been merged into the branch, the merge commit's + block carries the target's changes. The merge base that branch merge + computes is not exposed. +- **Comparing two points.** "What changed on main this week" or "how does this + snapshot differ from that one" needs a net comparison of two endpoints. + `diff_between` and `diff_commits` compute one, but only for embedded Rust + callers and in the table-shaped `ChangeSet`. Their same-lineage path + classifies every row whose `_row_last_updated_at_version` falls in the + interval as an update without comparing images, so a rewrite with equal + values is reported as an update. That path runs only on linear table + intervals, which a graph holds only in history written before detached-only + tables. + +`branch merge --as-commit` followed by `commit changes ` reports the +exact changes a merge would publish, because commit resolution resolves a +proposal before the first-parent diff. It is a write, though: it needs merge +rights on the target and records a durable proposal. It is also short-lived by +design: once any commit moves the target's head, the proposal expires, and +reads of it, `commit changes` included, refuse until it is dropped. + +### 8.2 Contract + +A graph diff compares two **read targets**. A read target is a branch name, +meaning its head at capture, or a graph commit id: a published commit, an +unpublished proposal, or an unattached snapshot. Both endpoints are resolved to +exact graph commits once, at the first request. The response names them and +continuation reuses them, so a branch that moves between pages does not change +the result. A string that names both a live branch and a resolvable commit is +refused with a typed ambiguity error, never resolved by precedence. Most commit +ids have the form `hb1...`, but the genesis commit and +commits from before history blocks have bare ULID ids, and a bare ULID is a +legal branch name. + +There are two forms: + +- **Two-point:** `from` and `to` are read targets, and the diff is the net + transition from `from`'s state to `to`'s state. +- **Branch:** a source branch and a target branch, `main` by default. `from` is + their merge base, computed by the rule branch merge uses, and `to` is the + source head; both are captured from one coherent view. The result is what the + source changed since it left the target, and changes the target made after + the base do not appear. A source with no commits past the base yields an + empty result. + +Semantics are §4's definition applied to the two endpoints: absent or present +decides insert and delete, a typed image comparison decides update, and equal +images are no change. Intermediate history is never folded. An update that was +later reverted produces nothing, and a delete followed by a reinsert is an +update when the images differ and nothing when they are equal. Type lifetimes +pair by stable identity (§4.1), and the same schema gate applies: a paired +lifetime whose user schema differs, or a non-empty lifetime present on one side +only, refuses with `schema_boundary` until C6 defines how to cross it. + +Changes use the §3.1 entity shape and §4.4 order, with no cause: a range +collapsed across commits has no single honest actor or commit. The response +carries the resolved `from` and `to` commit ids and, for the branch form, +`base`. A page token of a new `diff-page` kind binds the graph identity, both +resolved endpoints, the filter digest and the position; it is never accepted +as a feed cursor. Filters are the §3.3 kind, type and operation filters. An +unreadable endpoint maps to the §6 typed outcomes. + +Authorization is `read` on the branch each endpoint resolves to. A commit +endpoint authorizes against its authored branch, or against the recorded +target of a proposal or snapshot, and a forbidden endpoint is reported as not +found, as the commit diff does. + +Cost: a type lifetime whose `to` pin is one proving transaction past its `from` +pin may use the §14 candidate pruning. Every other changed lifetime takes the +exact ordered comparison, whose cost grows with the size of that type's table +rather than with the number of changes. Lifetimes whose pins are equal at both +endpoints are skipped, so a branch diff compares only the types the branch +wrote. The user documentation states this cost before the surface ships. + +### 8.3 Relationship to the commit diff and the feed + +The commit diff stays. It is the cause-carrying block for one commit and the +unit the feed delivers. A graph diff from a commit's first parent to that +commit returns the same changes without the cause; that equality is a +conformance test, not a second definition. + +The graph diff does not derive a range by folding feed blocks. It compares +endpoint membership and images directly, which stays correct when intermediate +history has been reclaimed while both endpoints remain readable, and it is +often the cheaper reconciliation path. The commit diff, the feed and the graph +diff share type-lifetime interval construction, canonical image comparison and +one conformance suite; they do not grow separate definitions of insert, update +or delete. + +### 8.4 Merge base and merge preview + +`branch merge-base` reports the commit a merge of the source into the target +would classify against, by the rule the +[merge guide](../user/branching/merge.md) documents. It reads lineage only. It +also lets a caller replay a branch's own commits through the feed with +`--start after:`. + +`branch merge --dry-run` runs the merge's base resolution, three-way +classification and constraint validation, reports the outcome +(`already_up_to_date`, `fast_forward` or `merged`), the structured conflict +list and change counts per type and operation, and stops. It is a read: + +- it requires `read` on both branches, not merge rights; +- it captures its inputs as an ordinary read snapshot, without merge-input + tags, so a cleanup that reclaims an input mid-run yields the typed gap; +- it stages only to process-local scratch and commits and publishes nothing. + +Its outcome and conflict list must equal a real merge's on the same captured +inputs. A real merge already validates before its first detached commit, so +the dry run is that prefix without the input tags. The exact result a merge +would publish remains `--as-commit` followed by `commit changes`; the dry run +answers whether the merge would succeed, without a write. + +### 8.5 Surfaces + +| CLI | HTTP | Answers | +|---|---|---| +| `diff --from --to ` | `GET /graphs/{id}/diff?from=&to=` | Net changes between two read targets | +| `branch diff [--into ]` | `GET /graphs/{id}/branches/diff?source=&target=` | What the source changed since its merge base with the target | +| `branch merge-base [--into ]` | `GET /graphs/{id}/branches/merge-base?source=&target=` | The merge base commit | +| `branch merge --dry-run` | `POST /graphs/{id}/branches/merge` with `dry_run: true` | Merge outcome, conflicts and counts, without a write | +| `commit changes `, unchanged | `GET /graphs/{id}/commits/{commit_id}/changes`, unchanged | One commit's changes with its cause | + +The diff commands take the existing `--kind`, `--type`, `--op`, `--limit`, +`--page-token` and `--json` options and paginate automatically, as +`commit changes` does. `--into`, `source` and `target` follow `branch merge`. +These commands are not GQ statements, because their results are paged change +records rather than query rows. + +### 8.6 Retiring the table-shaped diff + +`diff_between` and `diff_commits` become one engine call that returns pages of +the §3.1 entity shape, which the CLI and server use. The table-shaped +`ChangeSet`, `EntityChange` and `ChangeStats`, and with them the stamp path's +update rule, are removed once their callers, the DST harness and engine tests, +move to it. The planner's `SnapshotDiff` operation already models this pair; +its registry routes no diff until an integration lands with differential +evidence, so routing D1 through it is an implementation choice that needs that +evidence. ## 9. Evidence gates @@ -730,6 +868,37 @@ Extend existing owners before adding a new test silo: `changes.rs`, - Direct snapshot access maps reclaimed participant versions to `HistoricalDataReclaimed` rather than leaking a raw Lance error. +### D0 — graph diff + +Owners: engine `integration/changes.rs` and `integration/merge_truth_table.rs`; +server `integration/data_routes.rs` and `integration/openapi.rs`; CLI +`integration/parity_matrix.rs`. + +- Conformance: for every commit in the G0 fixtures, the graph diff from its + first parent equals `commit changes` for it, without the cause. +- Net semantics: an update later reverted yields nothing; a delete and + reinsert yields one update when the images differ and nothing when they are + equal; a range whose intermediate history was reclaimed succeeds while both + endpoints are readable. +- A same-value rewrite across a linear interval of pre-detached history + reports no change, proving the stamp path's update rule is gone. +- Branch form: the result excludes target commits after the base; the + reported base equals branch merge's base on the merge truth-table fixtures, + ties included; after the target was merged into the source, only the + source's own net changes since the newest base appear. +- Endpoints: a branch, a published commit, a proposal and an unattached + snapshot all resolve; a string naming both a branch and a commit is refused; + pages after a concurrent commit on either branch still answer for the pinned + endpoints. +- `commit changes` on a proposal returns the changes its merge would publish, + and refuses once the target's head has moved. Both were checked by hand on + 2026-10-12 (§15); no test covers them yet. +- Dry run: outcome and conflict list equal the real merge's on the merge + truth-table rows; no `__manifest` version, merge-input tag or table version + is written; an actor with `read` alone can run it and still cannot merge. +- Authorization and not-found parity with the commit diff; the vocabulary + gates cover the new DTOs; the parity matrix covers local and remote runs. + ### Cost evidence C0 pins adjacent first-parent classification as a pure comparison against the @@ -772,23 +941,32 @@ physical schema rather than today's catalog. That is internal correctness machinery: the result contains the graph entity image, not a schema object, schema version, table field ID, or user-selectable decoder mode. -Today's lineage does not retain accepted SchemaIR for every commit. Therefore -the entity diff/feed phases cross only snapshot pairs whose compatible logical -schema can be proven; an unprovable schema boundary returns a typed refusal. It -is not guessed from table layout and does not silently fall back to the current -schema. - -A schema-feed extension must decide: - -- historical SchemaIR identity and schema-change event encoding; -- bootstrap semantics for a consumer starting from an empty store; -- whether a full schema snapshot rides every change or only schema commits; -- retention and rebuild behavior for schema history; -- whether the required authority earns an internal-format strand. +Since storage format 14, lineage retains the schema each commit was written +under (§2). The commit diff, the feed and the graph diff nevertheless cross only +snapshot pairs whose paired lifetimes share one user schema; a schema boundary +returns a typed refusal. It is not guessed from table layout and does not +silently fall back to the current schema. The cost is real: adding one nullable +property to a populated type refuses that commit's diff and stops every feed +at it, so each such schema apply forces every consumer to take a new baseline. + +C6 crosses a boundary by decoding each endpoint under its own archived schema. +Its first step covers additive changes. An added nullable property compares as +null on the side that lacks it, so the apply itself emits no entity changes; an +added type is an empty lifetime, which already passes. The block then carries +one schema record naming the commit's schema identity and content hash. C6 must +still decide: + +- the schema record's encoding, and how a consumer opts in through its cursor + and page-token scope, so that an older consumer never skips one silently; +- image rules for property and type renames and drops and for enum or + nullability changes, each of which keeps refusing until it has one; +- bootstrap for a consumer starting from an empty store, including whether a + baseline carries the schema content; +- retention of archived schema content relative to `cleanup`. It must not infer graph identity from Lance field IDs or physical column names. -Until that extension lands, the entity feed is replayable only within a proven -compatible graph schema established out of band. +Until C6 lands, the entity feed is replayable only within one graph schema +established out of band. ### 10.2 Other exclusions @@ -806,7 +984,7 @@ compatible graph schema established out of band. ## 11. Format and compatibility audit -The C0–C4 read surfaces below persist nothing and therefore require no +The C0–C4, C6 and D1–D3 read surfaces persist nothing and therefore require no internal-schema or recovery-schema bump: - lineage and table pins already exist; @@ -833,9 +1011,13 @@ tombstone. Opaque page tokens and cursors have separate wire versions and decoders. An unsupported version or cross-use is a typed error, not best-effort decoding. -Any implementation that proposes a stored watermark, feed offset, a NON-advisory -operation summary, delete tombstone, or historical SchemaIR changes this -conclusion and must return to this RFC's format audit before landing. +C6 reads the per-commit schema content that the +[schema contract in the manifest](2026-09-30-schema-contract-in-manifest.md) +already persists; the graph diff's page token is a caller-owned wire value, and +the merge dry run writes nothing. Any implementation that proposes a stored +watermark, feed offset, a NON-advisory operation summary, delete tombstone, or +schema-history authority beyond that archived content changes this conclusion +and must return to this RFC's format audit before landing. ## 12. Phasing @@ -847,7 +1029,10 @@ conclusion and must return to this RFC's format audit before landing. | C3 — public entity feed | First-parent feed cursor plus exact snapshot/cursor baseline and at-least-once consumer contract | Useful caller-owned feed within a proven compatible schema | | C4 — entity history | Newest-first history derived from the same per-commit enumerator, with separate backward continuation domains | Investigation surface; no new storage authority | | C5 — publication time, optional | Derived `published_at` and possibly a measured as-of-time selector | Lands only after its semantics and cold-history cost pass §7 | -| C6 — schema replay, separate decision | Historical SchemaIR authority and schema-change events | Requires its own format conclusion before implementation | +| C6 — schema boundary crossing | Decode each endpoint under its archived schema; cross additive changes with one schema record per block and no synthetic entity changes; other schema changes keep refusing until §10.1's decisions are made | No new persisted authority; consumers opt in through cursor and page-token scope | +| D1 — two-point graph diff | `diff --from --to` and `GET /diff` over one engine call in the §3.1 shape; ambiguity refusal; `diff-page` token; retirement of the table-shaped `ChangeSet` | Net comparison of any two read targets | +| D2 — branch review | `branch merge-base` and `branch diff`, with base and source head captured together | Branches reviewable without a write | +| D3 — merge dry run | Read-only base resolution, classification and validation reporting outcome, conflicts and counts | Merge feasibility without merge rights | C0 deliberately does **not** add a second coordinator or an O(history log history) ancestry index. `CommitGraph` already holds the warm lineage @@ -866,8 +1051,8 @@ whose cost is justified by C1. 6. Delete authority: exact begin/end logical-ID comparison; transaction history may only prove that comparison unnecessary. 7. Retention: validate concrete participant versions; no scalar watermark. -8. Existing `diff`: distinct net-current API with shared primitives, no - optional multi-commit attribution. +8. Net diff: a public graph diff over two read targets in the §3.1 shape, + without a cause; the table-shaped `ChangeSet` retires (§8). 9. Row images: insert after, update before and after, delete before; edge endpoints follow each image. 10. Entity-data and graph-schema diffs are separate. Historical schema decoding @@ -876,6 +1061,14 @@ whose cost is justified by C1. captured together; a bare head ID is not a safe bootstrap. 12. Format: no bump for the entity feed; revisit before persisting any new history authority. +13. Branch review compares the merge base with the source head; changes the + target made after the base are excluded. +14. A read target string that names both a branch and a commit is refused, + never resolved by precedence. +15. The commit diff remains the cause-carrying unit; the graph diff never + carries a cause. +16. Merge preview without a write is a dry run that persists nothing; the + exact publishable result remains a `--as-commit` proposal. --- @@ -1109,3 +1302,61 @@ implementation, recorded here so later phases inherit them: `_rowid` and `_rowaddr` enter the sort, and complete rows hydrate in bounded, compacted chunks. The `O(N log N)` sort and full-table key read on the exact fallback remain. + +## 15. Decision log + +Amendments before 2026-10-12 are recorded in §14 and in the dispositions above. + +- **2026-10-12: graph diff.** Adds the graph diff (§8), its evidence gates + (§9 D0) and phases D1–D3 (§12), and reshapes C6 around the per-commit schema + content that storage format 14 retains (§10.1). This entry supersedes these + sentences, which the body no longer contains: + - status line: "C4+ remain design-stage." + - §0: "Two public surfaces share one private adjacent-commit enumerator:" + - §0: "Because current lineage does not retain accepted SchemaIR for every + historical commit, the initial entity surfaces fail with a typed error at + an unproven schema boundary. A separate schema contract is deferred to + §10." + - §2: "`diff_between` and `diff_commits` already know how to compare two + graph snapshots, including table creation, removal, rename, and + cross-lineage ID comparison." + - §8, all of "Relationship to existing diff", including "`diff_between` + remains the direct net-current comparison API while the feed is built." + and "Its current table-shaped `ChangeSet` / `EntityChange` fields are + transitional." Its rules on cause, endpoint comparison and shared + definitions carry into §8.2 and §8.3. + - §10.1: "Today's lineage does not retain accepted SchemaIR for every + commit." and the list of decisions a schema-feed extension must make. + - §11: "The C0–C4 read surfaces below persist nothing" and "…delete + tombstone, or historical SchemaIR changes this conclusion…" + - §12: the row "C6 — schema replay, separate decision". + - §13 item 8: "Existing `diff`: distinct net-current API with shared + primitives, no optional multi-commit attribution." + + The facts the amendment rests on were checked against `main` at `47dfe7b4`. + - By reading the code: no CLI, HTTP, MCP or GQ surface calls `diff_between` + or `diff_commits` or reports a merge base; each commit record carries + `schema_content_hash`; commit resolution resolves proposals and unattached + snapshots before the first-parent diff; branch merge validates before its + first detached commit but pins its inputs with merge-input tags first; the + planner registry routes no diff; `export` and `snapshot` accept only a + branch; commit ids are `hb1...` or bare ULIDs; a merge + `as commit` is the `branch_merge` action. + - With a CLI built from `47dfe7b4` on a scratch graph, repeated with the + 0.13.0 release for every step that release supports, with the same + results: + - the feed of a branch after its fork commit returns only the branch's + commits; + - a commit off the branch's first-parent chain is refused as a start; + - merging the target into the branch puts the target's changes in that + merge commit's block; + - a conflicting merge returns its conflict list, publishes nothing and + leaves no merge-input tag; + - adding a nullable property to a populated type refuses that commit's + diff and stops the feed at it, while the feed after the apply works; + - `commit changes` on a `branch merge --as-commit` proposal returns exactly + the changes the merge would publish and none the target made; + - once a commit moves the target's head, `commit changes` and + `query --snapshot` on the proposal refuse with "expired and awaits + cleanup", `commit publish` fails its head precondition, and + `commit drop` succeeds. From 10db8e9367fe05dc1d8f567fc57420c046120a0a Mon Sep 17 00:00:00 2001 From: Ragnor Comerford Date: Mon, 12 Oct 2026 12:17:37 -0700 Subject: [PATCH 02/11] rfc: retitle RFC 0030 to graph diffs, change feeds and retained history --- docs/rfcs/0030-cdc-time-travel.md | 11 +++++++---- docs/rfcs/README.md | 2 +- 2 files changed, 8 insertions(+), 5 deletions(-) diff --git a/docs/rfcs/0030-cdc-time-travel.md b/docs/rfcs/0030-cdc-time-travel.md index 4abbeb858..b645a2a8d 100644 --- a/docs/rfcs/0030-cdc-time-travel.md +++ b/docs/rfcs/0030-cdc-time-travel.md @@ -1,6 +1,6 @@ --- rfc: "0030" -title: "Graph change feed and retained-history contract" +title: "Graph diffs, change feeds and retained history" track: maintainer status: accepted implementation: partial @@ -14,7 +14,7 @@ superseded_by: [] blocked_on: [] --- -# RFC 0030: Graph change feed and retained-history contract +# RFC 0030: Graph diffs, change feeds and retained history > **Detached-only tables disposition:** > [RFC: Detached-only tables](2026-09-21-detached-only-tables.md) changed the @@ -1309,8 +1309,11 @@ Amendments before 2026-10-12 are recorded in §14 and in the dispositions above. - **2026-10-12: graph diff.** Adds the graph diff (§8), its evidence gates (§9 D0) and phases D1–D3 (§12), and reshapes C6 around the per-commit schema - content that storage format 14 retains (§10.1). This entry supersedes these - sentences, which the body no longer contains: + content that storage format 14 retains (§10.1). The RFC now covers more than + the feed, so it is retitled from "Graph change feed and retained-history + contract" to "Graph diffs, change feeds and retained history"; the filename + is unchanged. This entry supersedes these sentences, which the body no + longer contains: - status line: "C4+ remain design-stage." - §0: "Two public surfaces share one private adjacent-commit enumerator:" - §0: "Because current lineage does not retain accepted SchemaIR for every diff --git a/docs/rfcs/README.md b/docs/rfcs/README.md index 402e51a25..f2d6e04c8 100644 --- a/docs/rfcs/README.md +++ b/docs/rfcs/README.md @@ -194,7 +194,7 @@ then dated RFCs by date. | [0027](0027-lineage-merge-deltas.md) | Lineage-based merge deltas | maintainer | draft | not-started | | [0028](0028-stable-schema-identity.md) | Stable schema identity and table incarnation | maintainer | accepted | complete | | [0029](0029-azure-blob-storage.md) | Native Azure Blob storage | public | accepted | in-progress | -| [0030](0030-cdc-time-travel.md) | Graph change feed and retained-history contract | maintainer | accepted | partial | +| [0030](0030-cdc-time-travel.md) | Graph diffs, change feeds and retained history | maintainer | accepted | partial | | [0031](0031-comparative-cost-harness.md) | Comparative cost harness | maintainer | draft | not-started | | [0032](0032-adversarial-correctness-harness.md) | Adversarial correctness harness | maintainer | draft | not-started | | [0033](0033-blob-management.md) | Blob management | maintainer | accepted | partial | From ae01158c556ebe1dfc0ca746f9dc30e0b20a4719 Mon Sep 17 00:00:00 2001 From: Ragnor Comerford Date: Mon, 12 Oct 2026 16:01:47 -0700 Subject: [PATCH 03/11] feat(changes): graph diff, branch review and merge dry run Implement RFC 0030 D1-D3 on the engine, HTTP and the CLI. - Omnigraph::diff_page reports the net entity changes between two branches or commits in the commit-changes shape, with a diff-page token that keeps the first page's resolved endpoints. A side naming both a branch and a commit is AmbiguousReadTarget; one naming neither is ReadTargetNotFound. - Omnigraph::merge_base returns the commit a merge would classify against, by the merge's own rule. - Session::branch_merge_dry_run runs a merge's capture, classification and validation and stops before any effect. - GET /diff, GET /branches/diff, GET /branches/merge-base and dry_run on POST /branches/merge; CLI diff, branch diff, branch merge-base and branch merge --dry-run. A /diff side the actor cannot read answers the same 404 as an unknown side. - The table-shaped diff_between/diff_commits and ChangeSet are removed; the DST oracle and tests move to diff_page. - GQT gains --- diff, --- merge-base and --- merge-dry-run steps (RFC 0045) with four cases on the engine, DST and served targets. --- .../graph-diff-and-branch-review.added.md | 14 + .../table-shaped-diff-removed.removed.md | 11 + crates/omnigraph-api-types/src/lib.rs | 126 ++ crates/omnigraph-bench/src/gqt_runner.rs | 6 + crates/omnigraph-cli/src/cli.rs | 79 ++ crates/omnigraph-cli/src/client.rs | 172 ++- crates/omnigraph-cli/src/main.rs | 153 ++- crates/omnigraph-cli/src/output.rs | 134 ++ crates/omnigraph-cli/src/planes.rs | 10 +- .../tests/integration/parity_matrix.rs | 116 ++ crates/omnigraph-cli/tests/support/mod.rs | 13 +- crates/omnigraph-core/src/error.rs | 11 + crates/omnigraph-dst/src/harness.rs | 60 +- crates/omnigraph-dst/tests/scenarios.rs | 2 +- crates/omnigraph-gqt-core/src/lib.rs | 650 +++++++++- crates/omnigraph-gqt-core/src/served.rs | 243 +++- crates/omnigraph-gqt-core/src/tests.rs | 125 ++ crates/omnigraph-gqt/README.md | 16 + .../cases/graph_diff_branch_review.gqt | 100 ++ .../cases/graph_diff_merge_proposal.gqt | 83 ++ .../cases/graph_diff_net_range.gqt | 128 ++ .../cases/graph_diff_refusals.gqt | 72 ++ crates/omnigraph-planner/src/operation.rs | 3 +- crates/omnigraph-server/src/handlers.rs | 271 ++++ crates/omnigraph-server/src/http_contract.rs | 5 +- crates/omnigraph-server/src/lib.rs | 8 + crates/omnigraph-server/src/redacted_cause.rs | 2 + .../tests/integration/auth_policy.rs | 114 ++ .../tests/integration/data_routes.rs | 229 ++++ .../tests/integration/openapi.rs | 29 +- crates/omnigraph-server/tests/support/mod.rs | 2 + crates/omnigraph/src/changes/mod.rs | 765 +---------- crates/omnigraph/src/changes/model.rs | 49 + crates/omnigraph/src/changes/token.rs | 146 ++- crates/omnigraph/src/db/graph_coordinator.rs | 99 +- crates/omnigraph/src/db/omnigraph.rs | 405 +++++- crates/omnigraph/src/exec/merge.rs | 95 +- crates/omnigraph/tests/failpoints.rs | 50 +- crates/omnigraph/tests/forbidden_apis.rs | 16 +- crates/omnigraph/tests/helpers/cost.rs | 4 + crates/omnigraph/tests/helpers/mod.rs | 116 +- crates/omnigraph/tests/integration/changes.rs | 877 ++++++++++--- .../tests/integration/legacy_columns.rs | 24 +- .../tests/integration/merge_truth_table.rs | 68 +- .../tests/integration/warm_read_cost.rs | 18 +- docs/dev/merge.md | 9 + docs/rfcs/0030-cdc-time-travel.md | 90 +- docs/rfcs/0045-gq-logic-tests.md | 22 +- docs/user/branching/changes.md | 69 +- docs/user/branching/index.md | 9 +- docs/user/branching/merge.md | 18 + docs/user/cli/reference.md | 6 +- docs/user/index.md | 2 +- docs/user/operations/server.md | 14 +- openapi.json | 1138 +++++++++++++++-- 55 files changed, 5701 insertions(+), 1395 deletions(-) create mode 100644 changelog.d/graph-diff-and-branch-review.added.md create mode 100644 changelog.d/table-shaped-diff-removed.removed.md create mode 100644 crates/omnigraph-gqt/cases/graph_diff_branch_review.gqt create mode 100644 crates/omnigraph-gqt/cases/graph_diff_merge_proposal.gqt create mode 100644 crates/omnigraph-gqt/cases/graph_diff_net_range.gqt create mode 100644 crates/omnigraph-gqt/cases/graph_diff_refusals.gqt diff --git a/changelog.d/graph-diff-and-branch-review.added.md b/changelog.d/graph-diff-and-branch-review.added.md new file mode 100644 index 000000000..4191fb34a --- /dev/null +++ b/changelog.d/graph-diff-and-branch-review.added.md @@ -0,0 +1,14 @@ +- **Compare two points, review a branch, and preview a merge.** + `omnigraph diff --from --to ` (`GET /graphs/{id}/diff`) reports + the net entity changes between two branches or commits, in the + `commit changes` change shape and without a cause. `branch diff ` + (`GET /branches/diff`) reports what a branch changed since its merge base + with a target branch, and `branch merge-base` (`GET /branches/merge-base`) + prints that base. `branch merge --dry-run` (`dry_run: true` on + `POST /branches/merge`) reports the outcome or the conflicts a merge would + have and writes nothing; it needs `read` on both branches, not + `branch_merge`. See [changes, diffs and change feeds][graph-diff-changes] + and [previewing a merge][graph-diff-merge-preview]. + +[graph-diff-changes]: ../docs/user/branching/changes.md +[graph-diff-merge-preview]: ../docs/user/branching/merge.md#preview-a-merge diff --git a/changelog.d/table-shaped-diff-removed.removed.md b/changelog.d/table-shaped-diff-removed.removed.md new file mode 100644 index 000000000..06ef3545b --- /dev/null +++ b/changelog.d/table-shaped-diff-removed.removed.md @@ -0,0 +1,11 @@ +- The embedded Rust `Omnigraph::diff_between` and `Omnigraph::diff_commits` + and their table-shaped `ChangeSet`, `EntityChange`, `ChangeStats`, + `ChangeFilter` and `ChangeOp` types are removed. Use `Omnigraph::diff_page`, + which compares any two branches or commits page by page in the + `commit changes` change shape with a `ChangeFeedScope` filter. Its updates + compare row images, so a row rewritten with equal values is no longer + reported, and a pair across a dropped and re-added type refuses with + `ChangeSchemaBoundary` instead of reporting deletes and inserts. See + [changes, diffs and change feeds][table-shaped-diff-removed-guide]. + +[table-shaped-diff-removed-guide]: ../docs/user/branching/changes.md#compare-two-points diff --git a/crates/omnigraph-api-types/src/lib.rs b/crates/omnigraph-api-types/src/lib.rs index 239a724a1..c15837a9d 100644 --- a/crates/omnigraph-api-types/src/lib.rs +++ b/crates/omnigraph-api-types/src/lib.rs @@ -336,6 +336,13 @@ pub struct BranchMergeRequest { /// with `delete_branch`, whose source the proposal still holds. #[serde(default)] pub as_commit: bool, + /// Run the merge through classification and validation and stop: the + /// response reports the outcome the merge would have now, a conflict + /// fails with the same `409` a merge returns, and nothing is written. + /// A read of both branches; refused together with `as_commit` and + /// `delete_branch`. + #[serde(default, skip_serializing_if = "std::ops::Not::not")] + pub dry_run: bool, /// Session settings for this request (the Session settings RFC); see [`SettingsRequest`]. #[serde(default, skip_serializing_if = "Option::is_none")] pub settings: Option, @@ -390,6 +397,10 @@ pub struct BranchMergeOutput { /// `branch_deleted` is `false`. #[serde(default, skip_serializing_if = "Option::is_none")] pub branch_delete_error_details: Option, + /// `true` for a dry run: `outcome` is what the merge would do now, and + /// `commit` is `null` because nothing was written. + #[serde(default, skip_serializing_if = "std::ops::Not::not")] + pub dry_run: bool, } #[derive(Debug, Clone, Copy, PartialEq, Eq, Serialize, Deserialize, ToSchema)] @@ -880,6 +891,97 @@ pub struct CommitChangesQuery { pub set: Vec, } +/// One bounded page of a graph diff: the net entity changes between two +/// resolved commits, in the commit diff's entity shape and order. There is no +/// cause: a range has no single author. +#[derive(Debug, Clone, Serialize, Deserialize, ToSchema)] +pub struct DiffOutput { + /// The commit the `from` side resolved to; for a branch diff, the merge + /// base. + pub from_commit_id: String, + /// The commit the `to` side resolved to; for a branch diff, the source + /// head. + pub to_commit_id: String, + /// The merge base, present only for a branch diff. + #[serde(default, skip_serializing_if = "Option::is_none")] + pub base_commit_id: Option, + pub changes: Vec, + /// Continue THIS bounded response; later pages answer for the same two + /// commits. Absent on the final page. + #[serde(default, skip_serializing_if = "Option::is_none")] + pub next_page_token: Option, +} + +/// Query parameters for a two-point graph diff. +#[derive(Debug, Clone, Default, Deserialize, IntoParams)] +#[into_params(parameter_in = Query)] +pub struct DiffQuery { + /// Read target the diff starts from: a branch name (its head now) or a + /// graph commit id. A string naming both is refused. + pub from: String, + /// Read target the diff ends at, spelled as `from`. + pub to: String, + /// Opaque continuation from the preceding page of this response. + pub page_token: Option, + /// Maximum changes per page. Server default applies when absent; above + /// the public ceiling the request fails with 413. + pub limit: Option, + /// Repeatable filter: node | edge. + #[serde(default)] + pub kind: Vec, + /// Repeatable filter: accepted-schema type name. + #[serde(default)] + pub r#type: Vec, + /// Repeatable filter: insert | update | delete. + #[serde(default)] + pub op: Vec, +} + +/// Query parameters for a branch diff: what `source` changed since its merge +/// base with `target`. +#[derive(Debug, Clone, Default, Deserialize, IntoParams)] +#[into_params(parameter_in = Query)] +pub struct BranchDiffQuery { + /// Branch whose changes are reported. + pub source: String, + /// Branch the source is compared against. Defaults to `main`. + pub target: Option, + /// Opaque continuation from the preceding page of this response. + pub page_token: Option, + /// Maximum changes per page, as for `GET /diff`. + pub limit: Option, + /// Repeatable filter: node | edge. + #[serde(default)] + pub kind: Vec, + /// Repeatable filter: accepted-schema type name. + #[serde(default)] + pub r#type: Vec, + /// Repeatable filter: insert | update | delete. + #[serde(default)] + pub op: Vec, +} + +/// Query parameters for the merge base of two branches. +#[derive(Debug, Clone, Default, Deserialize, IntoParams)] +#[into_params(parameter_in = Query)] +pub struct MergeBaseQuery { + /// Branch that would be merged. + pub source: String, + /// Branch that would receive the merge. Defaults to `main`. + pub target: Option, +} + +/// The commit a merge of `source` into `target` would classify against, and +/// the two heads it was computed from. +#[derive(Debug, Clone, Serialize, Deserialize, ToSchema)] +pub struct MergeBaseOutput { + pub source: String, + pub target: String, + pub source_commit_id: String, + pub target_commit_id: String, + pub merge_base: CommitOutput, +} + /// Query parameters for the change feed poll. #[derive(Debug, Clone, Default, Deserialize, IntoParams)] #[into_params(parameter_in = Query)] @@ -2169,6 +2271,30 @@ pub fn commit_changes_output(page: &omnigraph::changes::CommitChangesPage) -> Co } } +pub fn diff_output(page: &omnigraph::changes::GraphDiffPage) -> DiffOutput { + DiffOutput { + from_commit_id: page.from_commit_id.clone(), + to_commit_id: page.to_commit_id.clone(), + base_commit_id: page.base_commit_id.clone(), + changes: page.changes.iter().map(entity_change_output).collect(), + next_page_token: page.next_page_token.clone(), + } +} + +pub fn merge_base_output( + source: &str, + target: &str, + base: &omnigraph::changes::MergeBase, +) -> MergeBaseOutput { + MergeBaseOutput { + source: source.to_string(), + target: target.to_string(), + source_commit_id: base.source_commit_id.clone(), + target_commit_id: base.target_commit_id.clone(), + merge_base: commit_output(&base.merge_base), + } +} + pub fn change_feed_output(page: &omnigraph::changes::ChangeFeedPage) -> ChangeFeedOutput { let (next_page_token, cursor, caught_up) = match &page.continuation { omnigraph::changes::ChangeFeedContinuation::MidBlock { page_token } => { diff --git a/crates/omnigraph-bench/src/gqt_runner.rs b/crates/omnigraph-bench/src/gqt_runner.rs index cf730732e..16f8c404d 100644 --- a/crates/omnigraph-bench/src/gqt_runner.rs +++ b/crates/omnigraph-bench/src/gqt_runner.rs @@ -48,6 +48,9 @@ pub enum GqtOperationKind { Show, Restart, Concurrent, + Diff, + MergeBase, + MergeDryRun, } impl From for GqtOperationKind { fn from(kind: StepKind) -> Self { @@ -65,6 +68,9 @@ impl From for GqtOperationKind { StepKind::Show => Self::Show, StepKind::Restart => Self::Restart, StepKind::Concurrent => Self::Concurrent, + StepKind::Diff => Self::Diff, + StepKind::MergeBase => Self::MergeBase, + StepKind::MergeDryRun => Self::MergeDryRun, } } } diff --git a/crates/omnigraph-cli/src/cli.rs b/crates/omnigraph-cli/src/cli.rs index 3ab98d2a8..36a27f81c 100644 --- a/crates/omnigraph-cli/src/cli.rs +++ b/crates/omnigraph-cli/src/cli.rs @@ -252,6 +252,38 @@ pub(crate) enum Command { #[command(subcommand)] command: BlobCommand, }, + /// Net entity changes between two read targets: a branch (its head now) + /// or a graph commit id. Auto-paginates; `branch diff` reviews a branch + /// against its merge base, `commit changes` reports one commit with its + /// cause. + Diff { + /// Graph URI + uri: Option, + /// Read target the diff starts from + #[arg(long)] + from: String, + /// Read target the diff ends at + #[arg(long)] + to: String, + /// Changes per page (the server clamps at its public ceiling) + #[arg(long)] + limit: Option, + /// Fetch exactly one page starting at this opaque token + /// (auto-paginates when omitted) + #[arg(long)] + page_token: Option, + /// Filter by entity kind (repeatable): node | edge + #[arg(long = "kind", value_enum)] + kinds: Vec, + /// Filter by type name (repeatable) + #[arg(long = "type")] + types: Vec, + /// Filter by operation (repeatable): insert | update | delete + #[arg(long = "op", value_enum)] + ops: Vec, + #[arg(long)] + json: bool, + }, /// Commit history operations Commit { #[command(subcommand)] @@ -872,6 +904,11 @@ pub(crate) enum BranchCommand { /// Exclusive with --delete-branch, whose source the commit still holds. #[arg(long, conflicts_with = "delete_branch")] as_commit: bool, + /// Run the merge through classification and validation and stop: + /// report the outcome, or fail with the conflicts, and write nothing. + /// A read of both branches. + #[arg(long, conflicts_with_all = ["delete_branch", "as_commit"])] + dry_run: bool, /// Session setting for this invocation (repeatable): `name=value` in /// GQ spelling, e.g. `--set merge_lineage=off` — the request's /// `settings` field (the Session settings RFC). @@ -880,6 +917,48 @@ pub(crate) enum BranchCommand { #[arg(long)] json: bool, }, + /// What a branch changed since its merge base with a target branch: the + /// net entity changes from the base to the branch head. Auto-paginates. + Diff { + /// Graph URI + #[arg(long)] + uri: Option, + source: String, + /// Branch the source is compared against (default main) + #[arg(long)] + into: Option, + /// Changes per page (the server clamps at its public ceiling) + #[arg(long)] + limit: Option, + /// Fetch exactly one page starting at this opaque token + /// (auto-paginates when omitted) + #[arg(long)] + page_token: Option, + /// Filter by entity kind (repeatable): node | edge + #[arg(long = "kind", value_enum)] + kinds: Vec, + /// Filter by type name (repeatable) + #[arg(long = "type")] + types: Vec, + /// Filter by operation (repeatable): insert | update | delete + #[arg(long = "op", value_enum)] + ops: Vec, + #[arg(long)] + json: bool, + }, + /// The commit a merge of a source branch into a target branch would + /// classify against + MergeBase { + /// Graph URI + #[arg(long)] + uri: Option, + source: String, + /// Branch the source would merge into (default main) + #[arg(long)] + into: Option, + #[arg(long)] + json: bool, + }, } #[derive(Debug, Subcommand)] diff --git a/crates/omnigraph-cli/src/client.rs b/crates/omnigraph-cli/src/client.rs index 22d2de3dc..c02846dda 100644 --- a/crates/omnigraph-cli/src/client.rs +++ b/crates/omnigraph-cli/src/client.rs @@ -34,12 +34,13 @@ use omnigraph_api_types::{ BranchCreateRequest, BranchDeleteOutput, BranchListOutput, BranchMergeOutcome, BranchMergeOutput, BranchMergeRequest, BranchOutcomeOutput, ChangeBaselineOutput, ChangeBaselineRecord, ChangeBaselineRequest, ChangeFeedOutput, ChangeOpOutput, ChangeOutput, - ChangeRequest, CommitChangesOutput, CommitListOutput, CommitOutput, EntityKindOutput, - ExportRequest, GraphBatchLoadOutput, GraphDiscoveryResponse, GraphListResponse, - InvokeStoredQueryRequest, QueryRequest, ReadOutput, SchemaApplyOutput, SchemaOutput, - SettingsRequest, SnapshotOutput, branch_list_read_output, change_baseline_output, - change_feed_output, change_scope, commit_changes_output, commit_output, read_output, - schema_apply_output, show_read_output, snapshot_payload, + ChangeRequest, CommitChangesOutput, CommitListOutput, CommitOutput, DiffOutput, + EntityKindOutput, ExportRequest, GraphBatchLoadOutput, GraphDiscoveryResponse, + GraphListResponse, InvokeStoredQueryRequest, MergeBaseOutput, QueryRequest, ReadOutput, + SchemaApplyOutput, SchemaOutput, SettingsRequest, SnapshotOutput, branch_list_read_output, + change_baseline_output, change_feed_output, change_scope, commit_changes_output, commit_output, + diff_output, merge_base_output, read_output, schema_apply_output, show_read_output, + snapshot_payload, }; use omnigraph_compiler::catalog::Catalog; use omnigraph_compiler::query::ast::{BranchWrite, CommitWrite}; @@ -107,12 +108,30 @@ fn validate_merge_output( target: &str, delete_branch: bool, as_commit: bool, + dry_run: bool, ) -> Result<()> { if output.source != source || output.target != target { bail!( "invalid merge response: source or target differs from the request; effects are unknown; reconcile before retrying" ); } + if output.dry_run != dry_run { + bail!( + "invalid merge response: the dry-run flag contradicts the request; effects are unknown; reconcile before retrying" + ); + } + if dry_run { + // A dry run writes nothing: no commit, no deletion. + if output.commit.is_some() + || output.branch_deleted.is_some() + || output.branch_delete_error_details.is_some() + { + bail!( + "invalid merge response: a dry run reported a write; effects are unknown; reconcile before retrying" + ); + } + return Ok(()); + } validate_merge_receipt( output.outcome, output.commit.as_ref(), @@ -1724,12 +1743,14 @@ impl GraphClient { } } + #[allow(clippy::too_many_arguments)] pub(crate) async fn branch_merge( &self, source: &str, into: &str, delete_branch: bool, as_commit: bool, + dry_run: bool, settings: &[(SettingId, SettingValue)], ) -> Result { // Use the engine's canonical spelling for dispatch, receipt validation, @@ -1755,12 +1776,28 @@ impl GraphClient { target: Some(into.to_string()), delete_branch, as_commit, + dry_run, settings: Self::remote_settings(settings)?, })?), token.as_deref(), ) .await? } + GraphClient::Embedded { uri, actor } if dry_run => { + // A read of both branches, composed as the server's dry run. + let session = Self::open_read_session(uri, settings).await?; + let result = session.branch_merge_dry_run(source, into).await?; + BranchMergeOutput { + source: source.to_string(), + target: into.to_string(), + outcome: result.outcome.into(), + commit: None, + actor_id: actor.clone(), + branch_deleted: None, + branch_delete_error_details: None, + dry_run: true, + } + } GraphClient::Embedded { uri, actor } => { let session = Self::open_write_session(uri, settings).await?; let actor = actor.as_deref(); @@ -1790,13 +1827,121 @@ impl GraphClient { actor_id: actor.map(String::from), branch_deleted, branch_delete_error_details, + dry_run: false, } } }; - validate_merge_output(&output, source, into, delete_branch, as_commit)?; + validate_merge_output(&output, source, into, delete_branch, as_commit, dry_run)?; Ok(output) } + /// Fetch one bounded page of a graph diff. Auto-pagination is owned by the + /// command output loop, as for `commit_changes_page`. + pub(crate) async fn diff_page( + &self, + sides: DiffSidesArg<'_>, + page_token: Option<&str>, + limit: Option, + filter: &ChangeFilterArgs<'_>, + ) -> Result { + match self { + GraphClient::Remote { + http, + base_url, + token, + .. + } => { + let limit_value = limit.map(|limit| limit.to_string()); + let (segments, sides): (&[&str], [(&str, &str); 2]) = match sides { + DiffSidesArg::Between { from, to } => (&["diff"], [("from", from), ("to", to)]), + DiffSidesArg::Branch { source, target } => ( + &["branches", "diff"], + [("source", source), ("target", target)], + ), + }; + let mut query: Vec<(&str, &str)> = sides.to_vec(); + if let Some(page_token) = page_token { + query.push(("page_token", page_token)); + } + if let Some(limit) = limit_value.as_deref() { + query.push(("limit", limit)); + } + let filter_pairs = filter.query_pairs(); + query.extend( + filter_pairs + .iter() + .map(|(name, value)| (*name, value.as_str())), + ); + remote_json( + http, + Method::GET, + remote_url(base_url, segments, &query)?, + None, + token.as_deref(), + ) + .await + } + GraphClient::Embedded { uri, .. } => { + let session = Self::open_read_session(uri, &[]).await?; + let sides = match sides { + DiffSidesArg::Between { from, to } => { + omnigraph::changes::GraphDiffSides::Between { + from: from.to_string(), + to: to.to_string(), + } + } + DiffSidesArg::Branch { source, target } => { + omnigraph::changes::GraphDiffSides::Branch { + source: source.to_string(), + target: target.to_string(), + } + } + }; + let page = session + .diff_page(&omnigraph::changes::GraphDiffRequest { + sides, + scope: change_scope(filter.kinds, filter.types, filter.ops), + page_token: page_token.map(str::to_string), + max_changes: limit, + max_bytes: None, + }) + .await?; + Ok(diff_output(&page)) + } + } + } + + /// The merge base of `source` and `target`, with the heads it was + /// computed from. + pub(crate) async fn merge_base(&self, source: &str, target: &str) -> Result { + match self { + GraphClient::Remote { + http, + base_url, + token, + .. + } => { + remote_json( + http, + Method::GET, + remote_url( + base_url, + &["branches", "merge-base"], + &[("source", source), ("target", target)], + )?, + None, + token.as_deref(), + ) + .await + } + GraphClient::Embedded { uri, .. } => { + let session = Self::open_read_session(uri, &[]).await?; + let base = session.merge_base(source, target).await?; + Ok(merge_base_output(source, target, &base)) + } + } + } + /// Apply a standalone graph's schema with the caller's catalog validator. /// Served schemas are deployed through the cluster configuration. pub(crate) async fn apply_schema( @@ -2236,6 +2381,15 @@ fn verified_blob_precondition(error: color_eyre::eyre::Report) -> color_eyre::ey } } +/// The two sides of a graph diff, as the command spelled them. +#[derive(Debug, Clone, Copy)] +pub(crate) enum DiffSidesArg<'a> { + /// Two read targets: branches or commit ids. + Between { from: &'a str, to: &'a str }, + /// A source branch against its merge base with a target branch. + Branch { source: &'a str, target: &'a str }, +} + /// Shared spelling of the change-surface filters across the three verbs; one /// translation (`change_scope`) is used by the embedded arm and the server. pub(crate) struct ChangeFilterArgs<'a> { @@ -2996,7 +3150,7 @@ mod tests { ( "branch merge", client - .branch_merge("review", "main", false, false, &settings) + .branch_merge("review", "main", false, false, false, &settings) .await .map(|_| ()), ), @@ -3103,7 +3257,7 @@ mod tests { .await .unwrap(); client - .branch_merge("review", "main", false, false, &settings) + .branch_merge("review", "main", false, false, false, &settings) .await .unwrap(); diff --git a/crates/omnigraph-cli/src/main.rs b/crates/omnigraph-cli/src/main.rs index d3af3b26b..bfecc8e28 100644 --- a/crates/omnigraph-cli/src/main.rs +++ b/crates/omnigraph-cli/src/main.rs @@ -771,6 +771,7 @@ async fn run(cli: Cli) -> Result<()> { into, delete_branch, as_commit, + dry_run, settings, json, } => { @@ -786,9 +787,11 @@ async fn run(cli: Cli) -> Result<()> { ) .await?; let into = resolve_branch(into, None, "main"); - echo_write_target(cli.quiet, "branch merge", client.uri(), client.is_remote()); + if !dry_run { + echo_write_target(cli.quiet, "branch merge", client.uri(), client.is_remote()); + } let payload = client - .branch_merge(&source, &into, delete_branch, as_commit, &settings) + .branch_merge(&source, &into, delete_branch, as_commit, dry_run, &settings) .await?; // Keep the successful merge receipt on stdout; optional // deletion has its own structured result and human warning. @@ -800,6 +803,13 @@ async fn run(cli: Cli) -> Result<()> { } if json { print_json(&payload)?; + } else if payload.dry_run { + println!( + "dry run: merging {} into {} would be: {} (nothing was written)", + payload.source, + payload.target, + payload.outcome.as_str() + ); } else { match payload.commit.as_ref().filter(|commit| !commit.published) { Some(commit) => println!( @@ -821,7 +831,83 @@ async fn run(cli: Cli) -> Result<()> { } } } + BranchCommand::Diff { + uri, + source, + into, + limit, + page_token, + kinds, + types, + ops, + json, + } => { + let client = client::GraphClient::resolve( + capability, + cli.server.as_deref(), + cli.graph.as_deref(), + uri, + cli.profile.as_deref(), + cli.store.as_deref(), + ) + .await?; + let into = resolve_branch(into, None, "main"); + let sides = client::DiffSidesArg::Branch { + source: &source, + target: &into, + }; + run_diff(&client, sides, page_token, limit, kinds, types, ops, json).await?; + } + BranchCommand::MergeBase { + uri, + source, + into, + json, + } => { + let client = client::GraphClient::resolve( + capability, + cli.server.as_deref(), + cli.graph.as_deref(), + uri, + cli.profile.as_deref(), + cli.store.as_deref(), + ) + .await?; + let into = resolve_branch(into, None, "main"); + let output = client.merge_base(&source, &into).await?; + if json { + print_json(&output)?; + } else { + print_merge_base_human(&output); + } + } }, + Command::Diff { + uri, + from, + to, + limit, + page_token, + kinds, + types, + ops, + json, + } => { + let client = client::GraphClient::resolve( + capability, + cli.server.as_deref(), + cli.graph.as_deref(), + uri, + cli.profile.as_deref(), + cli.store.as_deref(), + ) + .await?; + let sides = client::DiffSidesArg::Between { + from: &from, + to: &to, + }; + run_diff(&client, sides, page_token, limit, kinds, types, ops, json).await?; + } Command::Commit { command } => match command { CommitCommand::List { uri, @@ -2451,3 +2537,66 @@ fn report_deployment_lock(status: &omnigraph_cluster::DeploymentStatus) { ); } } + +/// Print a graph diff: one page when `page_token` is given (the raw escape +/// hatch), otherwise every page in turn, each emitted before the next is +/// fetched. +#[allow(clippy::too_many_arguments)] +async fn run_diff( + client: &client::GraphClient, + sides: client::DiffSidesArg<'_>, + page_token: Option, + limit: Option, + kinds: Vec, + types: Vec, + ops: Vec, + json: bool, +) -> Result<()> { + let kinds: Vec = + kinds.into_iter().map(Into::into).collect(); + let ops: Vec = ops.into_iter().map(Into::into).collect(); + let filter = client::ChangeFilterArgs { + kinds: &kinds, + types: &types, + ops: &ops, + }; + if let Some(page_token) = page_token.as_deref() { + let page = client + .diff_page(sides, Some(page_token), limit, &filter) + .await?; + if json { + print_json(&page)?; + } else { + print_diff_human(&page); + } + return Ok(()); + } + let mut next_page_token: Option = None; + if json { + let mut output = DiffJsonStream::new(io::BufWriter::new(io::stdout())); + loop { + let page = client + .diff_page(sides, next_page_token.as_deref(), limit, &filter) + .await?; + next_page_token = page.next_page_token.clone(); + output.write_page(&page)?; + if next_page_token.is_none() { + break; + } + } + let _ = output.finish()?; + } else { + let mut output = DiffHumanStream::new(); + loop { + let page = client + .diff_page(sides, next_page_token.as_deref(), limit, &filter) + .await?; + next_page_token = page.next_page_token.clone(); + output.write_page(&page)?; + if next_page_token.is_none() { + break; + } + } + } + Ok(()) +} diff --git a/crates/omnigraph-cli/src/output.rs b/crates/omnigraph-cli/src/output.rs index 36af7d0d4..a861531fc 100644 --- a/crates/omnigraph-cli/src/output.rs +++ b/crates/omnigraph-cli/src/output.rs @@ -1357,6 +1357,140 @@ impl CommitChangesHumanStream { } } +/// The endpoints a graph diff page names; every page of one auto-paginated +/// diff must name the same ones. +#[derive(Debug, Clone, PartialEq, Eq, Serialize)] +struct DiffEndpointsHeader { + from_commit_id: String, + to_commit_id: String, + #[serde(skip_serializing_if = "Option::is_none")] + base_commit_id: Option, +} + +impl DiffEndpointsHeader { + fn of(page: &omnigraph_api_types::DiffOutput) -> Self { + Self { + from_commit_id: page.from_commit_id.clone(), + to_commit_id: page.to_commit_id.clone(), + base_commit_id: page.base_commit_id.clone(), + } + } +} + +fn print_diff_endpoints_human(header: &DiffEndpointsHeader) { + if let Some(base_commit_id) = &header.base_commit_id { + println!("merge_base: {base_commit_id}"); + } + println!("from: {}", header.from_commit_id); + println!("to: {}", header.to_commit_id); +} + +pub(crate) fn print_diff_human(page: &omnigraph_api_types::DiffOutput) { + print_diff_endpoints_human(&DiffEndpointsHeader::of(page)); + for change in &page.changes { + print_entity_change_row(change); + } + if let Some(next_page_token) = &page.next_page_token { + println!("next_page_token: {}", next_page_token); + } +} + +/// Incremental JSON rendering for an auto-paginated graph diff: the endpoint +/// fields once, then one open `changes` array, each change serialized as its +/// page arrives. +pub(crate) struct DiffJsonStream { + writer: W, + header: Option, + first_change: bool, +} + +impl DiffJsonStream { + pub(crate) fn new(writer: W) -> Self { + Self { + writer, + header: None, + first_change: true, + } + } + + pub(crate) fn write_page(&mut self, page: &omnigraph_api_types::DiffOutput) -> Result<()> { + let header = DiffEndpointsHeader::of(page); + match &self.header { + None => { + self.writer.write_all(b"{\n \"from_commit_id\": ")?; + serde_json::to_writer(&mut self.writer, &header.from_commit_id)?; + self.writer.write_all(b",\n \"to_commit_id\": ")?; + serde_json::to_writer(&mut self.writer, &header.to_commit_id)?; + if let Some(base_commit_id) = &header.base_commit_id { + self.writer.write_all(b",\n \"base_commit_id\": ")?; + serde_json::to_writer(&mut self.writer, base_commit_id)?; + } + self.writer.write_all(b",\n \"changes\": [")?; + self.header = Some(header); + } + Some(seen) if seen == &header => {} + Some(_) => bail!("diff continuation changed its endpoints while auto-paginating"), + } + for change in &page.changes { + if !self.first_change { + self.writer.write_all(b",")?; + } + self.writer.write_all(b"\n ")?; + write_pretty_inline(&mut self.writer, change, 4)?; + self.first_change = false; + } + self.writer.flush()?; + Ok(()) + } + + pub(crate) fn finish(mut self) -> Result { + if self.header.is_none() { + bail!("diff auto-pagination completed without a page"); + } + if self.first_change { + self.writer.write_all(b"]\n}\n")?; + } else { + self.writer.write_all(b"\n ]\n}\n")?; + } + self.writer.flush()?; + Ok(self.writer) + } +} + +/// Incremental human rendering for an auto-paginated graph diff. +pub(crate) struct DiffHumanStream { + header: Option, +} + +impl DiffHumanStream { + pub(crate) fn new() -> Self { + Self { header: None } + } + + pub(crate) fn write_page(&mut self, page: &omnigraph_api_types::DiffOutput) -> Result<()> { + let header = DiffEndpointsHeader::of(page); + match &self.header { + None => { + print_diff_endpoints_human(&header); + self.header = Some(header); + } + Some(seen) if seen == &header => {} + Some(_) => bail!("diff continuation changed its endpoints while auto-paginating"), + } + for change in &page.changes { + print_entity_change_row(change); + } + io::stdout().flush()?; + Ok(()) + } +} + +pub(crate) fn print_merge_base_human(output: &omnigraph_api_types::MergeBaseOutput) { + println!("merge_base: {}", output.merge_base.graph_commit_id); + println!("source: {} at {}", output.source, output.source_commit_id); + println!("target: {} at {}", output.target, output.target_commit_id); +} + /// Incremental JSON rendering for one auto-paginated feed poll. /// /// A block may straddle pages. Keeping its JSON object open lets the renderer diff --git a/crates/omnigraph-cli/src/planes.rs b/crates/omnigraph-cli/src/planes.rs index 4fc1c6fc4..87fd7df1a 100644 --- a/crates/omnigraph-cli/src/planes.rs +++ b/crates/omnigraph-cli/src/planes.rs @@ -13,8 +13,8 @@ use color_eyre::Result; use color_eyre::eyre::bail; use crate::cli::{ - BlobCommand, ChangesCommand, Cli, ClusterCommand, Command, GraphsCommand, QueriesCommand, - SchemaCommand, + BlobCommand, BranchCommand, ChangesCommand, Cli, ClusterCommand, Command, GraphsCommand, + QueriesCommand, SchemaCommand, }; #[derive(Debug, Clone, Copy, PartialEq, Eq)] @@ -201,6 +201,10 @@ fn flag_applies(flag: ScopeFlag, capability: Capability, cmd: &Command) -> bool command: BlobCommand::Get { .. } | BlobCommand::Stat { .. } } | Command::Changes { .. } | Command::Commit { .. } + | Command::Diff { .. } + | Command::Branch { + command: BranchCommand::Diff { .. } | BranchCommand::MergeBase { .. } + } | Command::Query { .. } ), Control => matches!( @@ -271,6 +275,7 @@ pub(crate) fn command_plane(cmd: &Command) -> Plane { | Command::Export { .. } | Command::Blob { .. } | Command::Commit { .. } + | Command::Diff { .. } | Command::Changes { .. } | Command::Graphs { .. } => Plane::Data, Command::Schema { @@ -338,6 +343,7 @@ pub(crate) fn command_label(cmd: &Command) -> &'static str { BlobCommand::Clear { .. } => "blob clear", }, Command::Commit { .. } => "commit", + Command::Diff { .. } => "diff", Command::Changes { command } => match command { ChangesCommand::Poll { .. } => "changes poll", ChangesCommand::Baseline { .. } => "changes baseline", diff --git a/crates/omnigraph-cli/tests/integration/parity_matrix.rs b/crates/omnigraph-cli/tests/integration/parity_matrix.rs index ed0f87516..06fcd8ca3 100644 --- a/crates/omnigraph-cli/tests/integration/parity_matrix.rs +++ b/crates/omnigraph-cli/tests/integration/parity_matrix.rs @@ -1406,6 +1406,122 @@ fn parity_commit_changes_filtered() { assert_parity("commit changes filtered", &l, &r); } +#[test] +fn parity_diff_over_shared_history() { + let p = parity(); + let commit_id = fixture_load_commit(&p); + let (l, _r) = p.run(&["commit", "show", &commit_id, "--json"]); + let parent = parse_stdout_json(&l)["parent_commit_id"] + .as_str() + .unwrap() + .to_string(); + // --limit 1 walks page tokens on both arms; the shared history makes the + // endpoints, type ids and rows byte-identical. + let (l, r) = p.run(&[ + "diff", "--from", &parent, "--to", "main", "--limit", "1", "--json", + ]); + assert_parity("diff", &l, &r); + let (local, remote) = (parse_stdout_json(&l), parse_stdout_json(&r)); + assert_eq!(local, remote, "diff JSON diverges unscrubbed"); + assert_eq!(local["from_commit_id"], parent.as_str()); + assert!(!local["changes"].as_array().unwrap().is_empty()); + // The commit form and the two-point form from its parent agree. + let (commit, _) = p.run(&["commit", "changes", &commit_id, "--json"]); + let (between, _) = p.run(&["diff", "--from", &parent, "--to", &commit_id, "--json"]); + assert_eq!( + parse_stdout_json(&commit)["changes"], + parse_stdout_json(&between)["changes"] + ); + // A side naming nothing is the same refusal on both arms. + let (l, r) = p.run(&["diff", "--from", "main", "--to", "nowhere"]); + assert_eq!(l.status.code(), r.status.code()); + for output in [&l, &r] { + assert!(!output.status.success(), "{output:?}"); + assert!( + String::from_utf8_lossy(&output.stderr) + .contains("'nowhere' names no branch and no commit"), + "{output:?}" + ); + } +} + +/// Each arm writes its own commits, so commit ids differ between the arms; +/// only those are normalized before the outputs are compared. +fn without_commit_ids(output: &std::process::Output) -> serde_json::Value { + fn normalize(value: &mut serde_json::Value) { + match value { + serde_json::Value::Object(map) => { + for (key, value) in map.iter_mut() { + if key.ends_with("commit_id") && value.is_string() { + *value = serde_json::Value::String(format!("<{key}>")); + } else { + normalize(value); + } + } + } + serde_json::Value::Array(items) => items.iter_mut().for_each(normalize), + _ => {} + } + } + let mut payload = parse_stdout_json(output); + normalize(&mut payload); + if let Some(base) = payload.get_mut("merge_base") { + for key in ["graph_manifest_version", "created_at"] { + base[key] = serde_json::Value::Null; + } + } + payload +} + +#[test] +fn parity_branch_review() { + let p = parity(); + let (l, r) = p.run(&["branch", "create", "--from", "main", "feature", "--json"]); + assert_parity("branch create (review setup)", &l, &r); + let (l, r) = p.run(&[ + "mutate", + "--branch", + "feature", + "-e", + "query add() { insert Person { name: \"Reviewed\", age: 31 } }", + "--json", + ]); + assert_write_parity("review source write", &l, &r); + for args in [ + &["branch", "merge-base", "feature", "--json"][..], + &[ + "branch", "diff", "feature", "--into", "main", "--limit", "1", "--json", + ][..], + &["diff", "--from", "main", "--to", "feature", "--json"][..], + &["branch", "merge", "feature", "--dry-run", "--json"][..], + ] { + let (l, r) = p.run(args); + assert!( + l.status.success() && r.status.success(), + "{args:?}: {l:?} {r:?}" + ); + assert_eq!( + without_commit_ids(&l), + without_commit_ids(&r), + "{args:?}: JSON diverges beyond commit ids" + ); + } + let (l, _r) = p.run(&["branch", "diff", "feature", "--json"]); + let review = parse_stdout_json(&l); + assert_eq!(review["changes"][0]["id"], "Reviewed"); + assert_eq!(review["changes"][0]["op"], "insert"); + assert_eq!(review["base_commit_id"], review["from_commit_id"]); + let (l, _r) = p.run(&["branch", "merge", "feature", "--dry-run", "--json"]); + let dry = parse_stdout_json(&l); + assert_eq!(dry["outcome"], "fast_forward"); + assert_eq!(dry["dry_run"], true); + assert!(dry["commit"].is_null()); + // The dry run wrote nothing: the real merge still has work to do. + let (l, r) = p.run(&["branch", "merge", "feature", "--json"]); + assert_write_parity("merge after the dry run", &l, &r); + assert_eq!(parse_stdout_json(&l)["outcome"], "fast_forward"); +} + #[test] fn parity_changes_poll_beginning() { let p = parity(); diff --git a/crates/omnigraph-cli/tests/support/mod.rs b/crates/omnigraph-cli/tests/support/mod.rs index 1a1751506..971c225b9 100644 --- a/crates/omnigraph-cli/tests/support/mod.rs +++ b/crates/omnigraph-cli/tests/support/mod.rs @@ -1420,14 +1420,15 @@ pub fn run_both( if args.first() == Some(&"blob") { local.env("NO_COLOR", "1"); } - // The read commands (blob get/stat, query, changes, commit) reject `--as`: - // a served read resolves the actor from its bearer token and a direct read - // attributes none, so passing `--as` there is an error. Only the write - // verbs, `blob put` and `blob clear` among them, carry the parity actor on - // the direct arm. + // The read commands (blob get/stat, branch diff/merge-base, query, + // changes, commit, diff) reject `--as`: a served read resolves the actor + // from its bearer token and a direct read attributes none, so passing + // `--as` there is an error. Only the write verbs, `blob put` and + // `blob clear` among them, carry the parity actor on the direct arm. let read = match args { ["blob", verb, ..] => matches!(*verb, "get" | "stat"), - [verb, ..] => matches!(*verb, "query" | "changes" | "commit"), + ["branch", verb, ..] => matches!(*verb, "diff" | "merge-base"), + [verb, ..] => matches!(*verb, "query" | "changes" | "commit" | "diff"), [] => false, }; if !read { diff --git a/crates/omnigraph-core/src/error.rs b/crates/omnigraph-core/src/error.rs index f8f409357..957ba4a6a 100644 --- a/crates/omnigraph-core/src/error.rs +++ b/crates/omnigraph-core/src/error.rs @@ -231,6 +231,17 @@ pub enum OmniError { /// an arbitrary storage `NotFound` as branch absence. #[error("branch '{branch}' not found")] BranchNotFound { branch: String }, + /// A graph diff endpoint names neither a live branch nor a resolvable + /// graph commit. Carries only the caller's own spelling. + #[error("'{target}' names no branch and no commit")] + ReadTargetNotFound { target: String }, + /// A graph diff endpoint names both a live branch and a resolvable graph + /// commit. Refused rather than resolved by precedence, so the same string + /// never silently means a different snapshot. + #[error( + "'{target}' names both a branch and a commit; rename the branch, or name its head by commit id" + )] + AmbiguousReadTarget { target: String }, /// A change page or feed can no longer be reconstructed contiguously: /// cleanup reclaimed a published dataset version one of its commits pins. Recovery is /// the exact baseline handshake, never a retried continuation. diff --git a/crates/omnigraph-dst/src/harness.rs b/crates/omnigraph-dst/src/harness.rs index 5d004c777..010f3a705 100644 --- a/crates/omnigraph-dst/src/harness.rs +++ b/crates/omnigraph-dst/src/harness.rs @@ -12,7 +12,10 @@ use std::io::Write as _; use std::sync::{Arc, Mutex}; use omnigraph::Session; -use omnigraph::changes::{ChangeFilter, ChangeOp, EntityKind}; +use omnigraph::changes::{ + ChangeEntityKind, ChangeFeedScope, ChangeOpKind, GraphDiffRequest, GraphDiffSides, + GraphEntityChange, +}; use omnigraph::db::{InitOptions, Omnigraph, ReadTarget, SnapshotId}; use omnigraph::error::{OmniError, Result as OmniResult}; use omnigraph::loader::LoadMode; @@ -3467,9 +3470,9 @@ async fn capture_history(db: &Omnigraph, main: &Model, history: &mut Vec<(String /// must equal the recorded model state — for EVERY commit in the /// universe's history. A recovery or maintenance op that corrupts the /// PAST while preserving the head passed every oracle before this one. -/// 2. **Conservative Person diff** over adjacent commits via -/// `diff_commits`: every model-changed name must appear in the engine's -/// ChangeSet, every engine-reported Delete must be a model delete, and +/// 2. **Conservative Person diff** over adjacent commits via the graph diff +/// (`diff_page`, every page): every model-changed name must appear in the +/// engine's changes, every engine-reported Delete must be a model delete, and /// engine deletes may not exceed model deletes. Deliberately one-sided /// where semantics are ambiguous (a value-equal upsert may legitimately /// surface as an engine Update the model-diff can't see) — no guessing, @@ -3501,12 +3504,12 @@ async fn assert_history_matches(db: &Session, history: &[(String, Model)], where for pair in history.windows(2) { let (a_id, a_m) = &pair[0]; let (b_id, b_m) = &pair[1]; - let filter = ChangeFilter { - kinds: Some(vec![EntityKind::Node]), + let scope = ChangeFeedScope { + kinds: Some(vec![ChangeEntityKind::Node]), type_names: Some(vec!["Person".to_string()]), ops: None, }; - let cs = match Box::pin(db.diff_commits(a_id, b_id, &filter)).await { + let changes = match Box::pin(graph_diff_changes(db, a_id, b_id, &scope)).await { Ok(changes) => changes, Err(OmniError::ChangeSchemaBoundary { graph_commit_id, @@ -3518,7 +3521,7 @@ async fn assert_history_matches(db: &Session, history: &[(String, Model)], where continue; } Err(error) => panic!( - "{where_}: diff_commits over recorded history failed without a matching \ + "{where_}: the graph diff over recorded history failed without a matching \ accepted schema transition: {error:?}" ), }; @@ -3540,34 +3543,61 @@ async fn assert_history_matches(db: &Session, history: &[(String, Model)], where model_changed.insert(name.clone()); } } - let engine_ids: BTreeSet<&str> = cs.changes.iter().map(|c| c.id.as_str()).collect(); - let engine_deletes: BTreeSet<&str> = cs - .changes + let engine_ids: BTreeSet<&str> = changes.iter().map(|c| c.id.as_str()).collect(); + let engine_deletes: BTreeSet<&str> = changes .iter() - .filter(|c| matches!(c.op, ChangeOp::Delete)) + .filter(|c| matches!(c.op, ChangeOpKind::Delete)) .map(|c| c.id.as_str()) .collect(); for name in model_changed.iter().chain(model_deleted.iter()) { assert!( engine_ids.contains(name.as_str()), - "{where_}: diff_commits {a_id}→{b_id} is MISSING model-changed Person '{name}'" + "{where_}: the graph diff {a_id}→{b_id} is MISSING model-changed Person '{name}'" ); } for name in &model_deleted { assert!( engine_deletes.contains(name.as_str()), - "{where_}: diff_commits {a_id}→{b_id} did not report model-deleted Person '{name}' as Delete" + "{where_}: the graph diff {a_id}→{b_id} did not report model-deleted Person '{name}' as Delete" ); } for id in &engine_deletes { assert!( model_deleted.contains(*id), - "{where_}: diff_commits {a_id}→{b_id} reports a Delete of '{id}' the model never deleted" + "{where_}: the graph diff {a_id}→{b_id} reports a Delete of '{id}' the model never deleted" ); } } } +/// Every change of the graph diff `from` → `to` within `scope`, page by page. +async fn graph_diff_changes( + db: &Session, + from: &str, + to: &str, + scope: &ChangeFeedScope, +) -> omnigraph::error::Result> { + let mut request = GraphDiffRequest { + sides: GraphDiffSides::Between { + from: from.to_string(), + to: to.to_string(), + }, + scope: scope.clone(), + page_token: None, + max_changes: None, + max_bytes: None, + }; + let mut changes = Vec::new(); + loop { + let page = db.diff_page(&request).await?; + changes.extend(page.changes); + match page.next_page_token { + Some(token) => request.page_token = Some(token), + None => return Ok(changes), + } + } +} + /// PHYSICAL-CHANNEL ORACLE (third audit channel, the one that counts rows): per /// branch, `export_jsonl` (no query machinery) must equal the model's physical /// expectation — persons exactly, Knows = every row ∪ ghosts, duplicates kept. diff --git a/crates/omnigraph-dst/tests/scenarios.rs b/crates/omnigraph-dst/tests/scenarios.rs index 825fdd169..4121dd9a0 100644 --- a/crates/omnigraph-dst/tests/scenarios.rs +++ b/crates/omnigraph-dst/tests/scenarios.rs @@ -413,7 +413,7 @@ fn dst_open_crash_is_effect_free() { /// model's state at that moment), re-read EVERY recorded commit through /// `ReadTarget::Snapshot` at final audit (snapshot equality: persons via raw /// scan + edges via a real traversal), pass the conservative Person -/// `diff_commits` check over every adjacent pair — and replay identically, +/// graph diff check over every adjacent pair — and replay identically, /// history commit ids included. #[test] #[serial] diff --git a/crates/omnigraph-gqt-core/src/lib.rs b/crates/omnigraph-gqt-core/src/lib.rs index c3d0018d0..f02090be4 100644 --- a/crates/omnigraph-gqt-core/src/lib.rs +++ b/crates/omnigraph-gqt-core/src/lib.rs @@ -123,6 +123,9 @@ pub fn step_kind(step: &Step) -> &'static str { Step::Show(_) => "show", Step::Restart { .. } => "restart", Step::Concurrent(_) => "concurrent", + Step::Diff(_) => "diff", + Step::MergeBase(_) => "merge-base", + Step::MergeDryRun(_) => "merge-dry-run", } } @@ -147,6 +150,9 @@ pub enum Step { Show(ShowStep), Restart { ordinal: usize }, Concurrent(ConcurrentStep), + Diff(DiffStep), + MergeBase(MergeBaseStep), + MergeDryRun(MergeDryRunStep), } impl Step { @@ -161,6 +167,9 @@ impl Step { Self::Show(s) => s.ordinal, Self::Restart { ordinal } => *ordinal, Self::Concurrent(s) => s.ordinal, + Self::Diff(s) => s.ordinal, + Self::MergeBase(s) => s.ordinal, + Self::MergeDryRun(s) => s.ordinal, } } @@ -176,7 +185,10 @@ impl Step { | Step::Control(_) | Step::Settings(_) | Step::Restart { .. } - | Step::Concurrent(_) => None, + | Step::Concurrent(_) + | Step::Diff(_) + | Step::MergeBase(_) + | Step::MergeDryRun(_) => None, } } @@ -190,7 +202,10 @@ impl Step { | Step::Control(_) | Step::Settings(_) | Step::Restart { .. } - | Step::Concurrent(_) => None, + | Step::Concurrent(_) + | Step::Diff(_) + | Step::MergeBase(_) + | Step::MergeDryRun(_) => None, } } } @@ -321,6 +336,99 @@ pub enum MergeExpect { Outcome(MergeOutcome), } +/// A `--- diff` step (RFC 0030 §8): one graph diff or one commit's own diff, +/// run to its last page, its changes compared as JSON rows. +#[derive(Debug)] +pub struct DiffStep { + ordinal: usize, + pub sides: DiffSides, + expect: DiffExpect, +} + +/// What a `--- diff` step compares. A side is a branch name, or `$name` for +/// the commit a `--- bind commit $name` captured. +#[derive(Debug, Clone, PartialEq, Eq)] +pub enum DiffSides { + /// `from: to: `: the net changes between two read targets. + Between { from: String, to: String }, + /// `source: ` and `target: ` (`main` when unspelled): what + /// the source changed since its merge base with the target. + Branch { source: String, target: String }, + /// `commit: `: one commit's own changes (`commit changes`). + Commit { commit: String }, +} + +/// A diff step's expect: the change rows, or an error. +#[derive(Debug)] +pub enum DiffExpect { + /// One JSON object per change: `kind`, `type` (the type's name), `id`, + /// `op`, and `before` / `after` images as the wire spells them. + Rows { + ordered: bool, + rows: Vec, + }, + Error { + needle: String, + }, +} + +/// `--- merge-base source: [target: ]`: the commit a merge +/// of the source into the target would classify against. +#[derive(Debug)] +pub struct MergeBaseStep { + ordinal: usize, + pub source: String, + pub target: String, + expect: MergeBaseExpect, +} + +/// A merge-base step's expect: `commit $name` or an error. +#[derive(Debug)] +pub enum MergeBaseExpect { + Commit { name: String }, + Error { needle: String }, +} + +/// `--- merge-dry-run source: [target: ]`: the merge run +/// through validation without a write, judged with a merge's own expects. +#[derive(Debug)] +pub struct MergeDryRunStep { + ordinal: usize, + pub source: String, + pub target: String, + expect: MergeExpect, +} + +impl DiffStep { + pub fn ordinal(&self) -> usize { + self.ordinal + } + + pub fn expect(&self) -> &DiffExpect { + &self.expect + } +} + +impl MergeBaseStep { + pub fn ordinal(&self) -> usize { + self.ordinal + } + + pub fn expect(&self) -> &MergeBaseExpect { + &self.expect + } +} + +impl MergeDryRunStep { + pub fn ordinal(&self) -> usize { + self.ordinal + } + + pub fn expect(&self) -> &MergeExpect { + &self.expect + } +} + /// A `--- query` step holding `branch list`: one `name` column, rows in /// byte order, so its expect is a read expect like a declaration's. #[derive(Debug)] @@ -601,8 +709,13 @@ enum ExpectHeader { Ordered, Ok, Error(String), - Affected { nodes: usize, edges: usize }, + Affected { + nodes: usize, + edges: usize, + }, Outcome(MergeOutcome), + /// `commit $name`: the commit a `--- bind commit $name` captured. + Commit(String), } /// The one argument a `--- query` or `--- mutate` header takes, @@ -810,6 +923,12 @@ fn parse_expect_header(rest: &str) -> Result { }; return Ok(ExpectHeader::Outcome(outcome)); } + if let Some(var) = rest.strip_prefix("commit ") { + let var = var.trim(); + let name = parse_loop_var(var) + .map_err(|_| format!("`--- expect commit {var}`: a name is $[a-z][a-z0-9_]*"))?; + return Ok(ExpectHeader::Commit(name)); + } if rest == "unordered" { return Ok(ExpectHeader::Unordered); } @@ -1240,6 +1359,20 @@ enum Pending { /// A `--- concurrent` block awaiting the bare `--- expect` that names /// each session's outcome. Concurrent(ConcurrentStep), + Diff { + ordinal: usize, + sides: DiffSides, + }, + MergeBase { + ordinal: usize, + source: String, + target: String, + }, + MergeDryRun { + ordinal: usize, + source: String, + target: String, + }, } /// Whether `text` holds the statement ` ` at any position: @@ -1339,7 +1472,10 @@ fn complete_list_step( needle: needle.clone(), } } - ExpectHeader::Ok | ExpectHeader::Affected { .. } | ExpectHeader::Outcome(_) => { + ExpectHeader::Ok + | ExpectHeader::Affected { .. } + | ExpectHeader::Outcome(_) + | ExpectHeader::Commit(_) => { return Err("`branch list` takes `unordered`, `ordered`, or `error:`".into()); } }; @@ -1400,7 +1536,8 @@ fn complete_show_step( ExpectHeader::Error(_) | ExpectHeader::Ok | ExpectHeader::Affected { .. } - | ExpectHeader::Outcome(_) => { + | ExpectHeader::Outcome(_) + | ExpectHeader::Commit(_) => { return Err(format!( "`{}` takes `unordered` or `ordered`; it returns the settings rows and does not fail", show_statement_name(id) @@ -1438,12 +1575,115 @@ fn complete_settings_step( | ExpectHeader::Ordered | ExpectHeader::Error(_) | ExpectHeader::Affected { .. } - | ExpectHeader::Outcome(_) => { + | ExpectHeader::Outcome(_) + | ExpectHeader::Commit(_) => { Err("a settings step takes `ok`; `set` and `reset` return no rows and no counts".into()) } } } +fn commit_expect_refusal() -> String { + "`expect commit $name` is accepted on a `--- merge-base` step only".into() +} + +/// The `key: value` arguments of a step header, each key once and from +/// `allowed`. +fn parse_step_arguments( + kind: &str, + rest: &str, + allowed: &[&str], +) -> Result, String> { + let mut args = HashMap::new(); + let mut tokens = rest.split_whitespace(); + while let Some(token) = tokens.next() { + let Some(key) = token.strip_suffix(':').filter(|key| allowed.contains(key)) else { + let keys: Vec = allowed.iter().map(|key| format!("`{key}:`")).collect(); + return Err(format!( + "`--- {kind}` takes {} as `key: value`, got `{token}`", + keys.join(", ") + )); + }; + let Some(value) = tokens.next() else { + return Err(format!("`--- {kind}` argument `{key}:` needs a value")); + }; + if args.insert(key.to_string(), value.to_string()).is_some() { + return Err(format!("`--- {kind}` names `{key}:` twice")); + } + } + Ok(args) +} + +/// A diff step's expect: change rows, one JSON object per line, or an error. +fn complete_diff_expect(mode: &ExpectHeader, section: &Section<'_>) -> Result { + match mode { + ExpectHeader::Ordered | ExpectHeader::Unordered => { + let mut rows = Vec::new(); + for (index, line) in §ion.body { + let line = line.trim(); + if line.is_empty() { + continue; + } + let row: Value = serde_json::from_str(line).map_err(|e| { + format!("line {}: a diff row is one JSON object: {e}", index + 1) + })?; + if !row.is_object() { + return Err(format!("line {}: a diff row is one JSON object", index + 1)); + } + rows.push(row); + } + Ok(DiffExpect::Rows { + ordered: matches!(mode, ExpectHeader::Ordered), + rows, + }) + } + ExpectHeader::Error(needle) => { + refuse_nonempty_body(§ion.body, "an `expect error:` section")?; + Ok(DiffExpect::Error { + needle: needle.clone(), + }) + } + ExpectHeader::Ok + | ExpectHeader::Affected { .. } + | ExpectHeader::Outcome(_) + | ExpectHeader::Commit(_) => { + Err("a `--- diff` step takes `ordered` or `unordered` rows, or `error:`".into()) + } + } +} + +/// A merge-base step's expect: `commit $name` naming a bound commit, or an +/// error. +fn complete_merge_base_expect( + mode: &ExpectHeader, + section: &Section<'_>, + bound_names: &HashSet, +) -> Result { + match mode { + ExpectHeader::Commit(name) => { + refuse_nonempty_body(§ion.body, "an `expect commit` section")?; + if !bound_names.contains(name) { + return Err(format!( + "`expect commit ${name}`: `${name}` is not bound; `--- bind commit ${name}` must precede this step" + )); + } + Ok(MergeBaseExpect::Commit { name: name.clone() }) + } + ExpectHeader::Error(needle) => { + refuse_nonempty_body(§ion.body, "an `expect error:` section")?; + Ok(MergeBaseExpect::Error { + needle: needle.clone(), + }) + } + ExpectHeader::Ok + | ExpectHeader::Ordered + | ExpectHeader::Unordered + | ExpectHeader::Affected { .. } + | ExpectHeader::Outcome(_) => { + Err("a `--- merge-base` step takes `commit $name` or `error:`".into()) + } + } +} + fn affected_refusal(name: &str) -> String { format!("`expect affected:` is refused on a control write; `{name}` carries no counts") } @@ -1473,6 +1713,7 @@ fn complete_write_expect( ExpectHeader::Outcome(_) => { Err("`expect outcome:` is accepted on a `branch merge` step only".into()) } + ExpectHeader::Commit(_) => Err(commit_expect_refusal()), ExpectHeader::Affected { .. } => Err(affected_refusal(name)), ExpectHeader::Unordered | ExpectHeader::Ordered => Err(rows_refusal(name)), } @@ -1588,6 +1829,7 @@ fn complete_decl_step( (ExpectHeader::Outcome(_), _) => { Err("`expect outcome:` is accepted on a `branch merge` step only".into()) } + (ExpectHeader::Commit(_), _) => Err(commit_expect_refusal()), (ExpectHeader::Unordered | ExpectHeader::Ordered, true) => Err( "a mutate step takes `ok`, `affected:`, or `error:`; mutation results carry no rows" .into(), @@ -1807,7 +2049,17 @@ pub fn parse_case(stem: &str, text: &str) -> Result { "invalid_case: a seam must directly precede its mutate step (a GQ mutation or a branch statement) or load step; no seam is crossed by a query step yet".into(), ); } - if matches!(kind, "query" | "mutate" | "load" | "restart" | "concurrent") { + if matches!( + kind, + "query" + | "mutate" + | "load" + | "restart" + | "concurrent" + | "diff" + | "merge-base" + | "merge-dry-run" + ) { source_lines.insert(ordinal + 1, section.header_line + 1); awaiting_seam_step = false; } @@ -2165,6 +2417,15 @@ pub fn parse_case(stem: &str, text: &str) -> Result { Some(Pending::Load { .. }) => { return Err("a generated load takes no params".into()); } + Some( + Pending::Diff { .. } + | Pending::MergeBase { .. } + | Pending::MergeDryRun { .. }, + ) => { + return Err( + "a diff, merge-base or merge-dry-run step takes no params".into() + ); + } None => { return Err(format!( "line {}: `--- params` must directly follow a query or mutate section", @@ -2342,6 +2603,31 @@ pub fn parse_case(stem: &str, text: &str) -> Result { Some(Pending::Concurrent(_)) => { unreachable!("a pending concurrent block is completed above") } + Some(Pending::Diff { ordinal, sides }) => Step::Diff(DiffStep { + ordinal, + sides, + expect: complete_diff_expect(&mode, section)?, + }), + Some(Pending::MergeBase { + ordinal, + source, + target, + }) => Step::MergeBase(MergeBaseStep { + ordinal, + source, + target, + expect: complete_merge_base_expect(&mode, section, &bound_names)?, + }), + Some(Pending::MergeDryRun { + ordinal, + source, + target, + }) => Step::MergeDryRun(MergeDryRunStep { + ordinal, + source, + target, + expect: complete_merge_expect(&mode, section)?, + }), None => { return Err(format!( "line {}: `--- expect` has no query or mutate step to bind to", @@ -2356,6 +2642,83 @@ pub fn parse_case(stem: &str, text: &str) -> Result { push_step(&mut items, &mut open_loop, completed); } } + "diff" | "merge-base" | "merge-dry-run" => { + if pending.is_some() { + return Err(format!( + "line {}: the previous step is missing its `--- expect`", + section.header_line + 1 + )); + } + refuse_nonempty_body(§ion.body, &format!("`--- {kind}`"))?; + // A `$name` side names a bound commit; it must be bound before. + let side = |value: &String| -> Result { + if let Some(name) = value.strip_prefix('$') + && !bound_names.contains(name) + { + return Err(format!( + "`--- {kind}` side `{value}` is not bound; `--- bind commit {value}` must precede this step" + )); + } + Ok(value.clone()) + }; + ordinal += 1; + pending = Some(if kind == "diff" { + let args = parse_step_arguments( + kind, + rest, + &["from", "to", "source", "target", "commit"], + )?; + let sides = match ( + args.get("from"), + args.get("to"), + args.get("source"), + args.get("target"), + args.get("commit"), + ) { + (Some(from), Some(to), None, None, None) => DiffSides::Between { + from: side(from)?, + to: side(to)?, + }, + (None, None, Some(source), target, None) => DiffSides::Branch { + source: source.clone(), + target: target.cloned().unwrap_or_else(|| MAIN_BRANCH.to_string()), + }, + (None, None, None, None, Some(commit)) => DiffSides::Commit { + commit: side(commit)?, + }, + _ => { + return Err( + "`--- diff` takes `from:` and `to:`, `source:` with an optional `target:`, or `commit:`" + .into(), + ); + } + }; + Pending::Diff { ordinal, sides } + } else { + let args = parse_step_arguments(kind, rest, &["source", "target"])?; + let source = args + .get("source") + .cloned() + .ok_or_else(|| format!("`--- {kind}` needs `source:`"))?; + let target = args + .get("target") + .cloned() + .unwrap_or_else(|| MAIN_BRANCH.to_string()); + if kind == "merge-base" { + Pending::MergeBase { + ordinal, + source, + target, + } + } else { + Pending::MergeDryRun { + ordinal, + source, + target, + } + } + }); + } "bind" => { if pending.is_some() { return Err(format!( @@ -3523,6 +3886,249 @@ async fn run_control_step( } } +/// A diff side as the engine and server take it: `$name` replaced by the +/// commit it names, anything else as spelled. +pub fn resolve_side(side: &str, bindings: &Bindings) -> Result { + match side.strip_prefix('$') { + Some(name) => bindings + .get(name) + .cloned() + .ok_or_else(|| format!("`{side}` is not bound at this step")), + None => Ok(side.to_string()), + } +} + +/// One change as a diff step compares it: its wire spelling with the type +/// named rather than identified, since the opaque type id differs between +/// graphs. +pub fn diff_row(change: &omnigraph_api_types::EntityChangeOutput) -> Value { + let mut row = serde_json::to_value(change).expect("change outputs serialize"); + row["type"] = Value::String(change.r#type.name.clone()); + row +} + +/// Judge a diff step's rows (in order or as a multiset) or its error. +pub fn check_diff_expect( + host: &impl ExecutionHost, + expect: &DiffExpect, + outcome: Result, String>, +) -> Result<(), String> { + match expect { + DiffExpect::Rows { ordered, rows } => { + let got = outcome.map_err(|e| format!("`diff` failed: {e}"))?; + let render = |rows: &[Value]| -> Vec { + let mut lines: Vec = rows.iter().map(canonical_json).collect(); + if !ordered { + lines.sort(); + } + lines + }; + let (want, got) = (render(rows), render(&got)); + if want == got { + Ok(()) + } else { + Err(format!( + "diff rows mismatch\n expected:\n {}\n actual:\n {}", + want.join("\n "), + got.join("\n ") + )) + } + } + DiffExpect::Error { needle } => { + check_error_expect(host, needle, outcome, "the diff succeeded") + } + } +} + +/// Every change a diff step names, page by page through the engine. +async fn engine_diff_rows( + session: &Session, + sides: &DiffSides, + bindings: &Bindings, +) -> Result, String> { + let scope = omnigraph::changes::ChangeFeedScope::default(); + let mut rows = Vec::new(); + let mut token: Option = None; + if let DiffSides::Commit { commit } = sides { + let id = resolve_side(commit, bindings)?; + loop { + let page = session + .commit_changes_page(&id, &scope, token.as_deref(), None, None) + .await + .map_err(|e| e.to_string())?; + rows.extend( + page.block + .changes + .iter() + .map(|change| diff_row(&omnigraph_api_types::entity_change_output(change))), + ); + match page.next_page_token { + Some(next) => token = Some(next), + None => return Ok(rows), + } + } + } + let sides = match sides { + DiffSides::Between { from, to } => omnigraph::changes::GraphDiffSides::Between { + from: resolve_side(from, bindings)?, + to: resolve_side(to, bindings)?, + }, + DiffSides::Branch { source, target } => omnigraph::changes::GraphDiffSides::Branch { + source: source.clone(), + target: target.clone(), + }, + DiffSides::Commit { .. } => unreachable!("the commit form returned above"), + }; + loop { + let page = session + .diff_page(&omnigraph::changes::GraphDiffRequest { + sides: sides.clone(), + scope: scope.clone(), + page_token: token.clone(), + max_changes: None, + max_bytes: None, + }) + .await + .map_err(|e| e.to_string())?; + rows.extend( + page.changes + .iter() + .map(|change| diff_row(&omnigraph_api_types::entity_change_output(change))), + ); + match page.next_page_token { + Some(next) => token = Some(next), + None => return Ok(rows), + } + } +} + +async fn run_diff_step( + host: &impl ExecutionHost, + session: &Session, + step: &DiffStep, + binding: Option<(&str, &str)>, + bindings: &Bindings, +) -> Result<(), StepFail> { + let label = step_label(step.ordinal, "diff", binding); + let fail = |message: String| StepFail { + label: label.clone(), + message, + bless_lines: None, + }; + let outcome = operation( + host, + step.ordinal, + engine_diff_rows(session, &step.sides, bindings), + ) + .await + .map_err(&fail)?; + host.observe(|| format!("actual diff: {outcome:?}")); + check_diff_expect(host, &step.expect, outcome).map_err(fail) +} + +/// Judge a merge-base step's commit against the bound one, or its error. +pub fn check_merge_base_expect( + host: &impl ExecutionHost, + expect: &MergeBaseExpect, + outcome: Result, + bindings: &Bindings, +) -> Result<(), String> { + match expect { + MergeBaseExpect::Commit { name } => { + let want = bindings + .get(name) + .ok_or_else(|| format!("`${name}` is not bound at this step"))?; + let got = outcome.map_err(|e| format!("`merge-base` failed: {e}"))?; + if &got == want { + Ok(()) + } else { + Err(format!( + "merge base mismatch: expected `${name}` ({want}), got {got}" + )) + } + } + MergeBaseExpect::Error { needle } => { + check_error_expect(host, needle, outcome, "`merge-base` succeeded") + } + } +} + +async fn run_merge_base_step( + host: &impl ExecutionHost, + session: &Session, + step: &MergeBaseStep, + binding: Option<(&str, &str)>, + bindings: &Bindings, +) -> Result<(), StepFail> { + let label = step_label(step.ordinal, "merge-base", binding); + let fail = |message: String| StepFail { + label: label.clone(), + message, + bless_lines: None, + }; + let outcome = operation( + host, + step.ordinal, + session.merge_base(&step.source, &step.target), + ) + .await + .map_err(&fail)? + .map(|base| base.merge_base.graph_commit_id) + .map_err(|e| e.to_string()); + host.observe(|| format!("actual merge base: {outcome:?}")); + check_merge_base_expect(host, &step.expect, outcome, bindings).map_err(fail) +} + +/// Judge a dry run as a merge step is judged; a dry run that reports a +/// commit fails whatever was expected. +pub fn check_dry_run_expect( + host: &impl ExecutionHost, + expect: &MergeExpect, + outcome: Result<(MergeOutcome, bool), String>, +) -> Result<(), String> { + if let Ok((_, true)) = &outcome { + return Err("a dry run reported a commit".into()); + } + let outcome = outcome.map(|(outcome, _)| outcome); + match expect { + MergeExpect::Write(expect) => check_write_expect(host, "merge-dry-run", expect, outcome), + MergeExpect::Outcome(want) => match outcome { + Ok(got) if got == *want => Ok(()), + Ok(got) => Err(format!( + "merge outcome mismatch: expected `{}`, got `{}`", + merge_outcome_word(*want), + merge_outcome_word(got) + )), + Err(e) => Err(format!("`merge-dry-run` failed: {e}")), + }, + } +} + +async fn run_merge_dry_run_step( + host: &impl ExecutionHost, + session: &Session, + step: &MergeDryRunStep, + binding: Option<(&str, &str)>, +) -> Result<(), StepFail> { + let label = step_label(step.ordinal, "merge-dry-run", binding); + let fail = |message: String| StepFail { + label: label.clone(), + message, + bless_lines: None, + }; + let outcome = operation( + host, + step.ordinal, + session.branch_merge_dry_run(&step.source, &step.target), + ) + .await + .map_err(&fail)? + .map(|result| (result.outcome, result.commit.is_some())) + .map_err(|e| e.to_string()); + host.observe(|| format!("actual dry run: {outcome:?}")); + check_dry_run_expect(host, &step.expect, outcome).map_err(fail) +} + async fn run_mutate_step( host: &impl ExecutionHost, session: &Session, @@ -3820,6 +4426,13 @@ async fn execute_steps_inner( Step::Settings(s) => run_settings_step(host, &mut session, s, binding), Step::Show(s) => run_show_step(host, &session, s, binding), Step::Concurrent(c) => host.concurrent_step(&session, case, c).await, + Step::Diff(d) => run_diff_step(host, &session, d, binding, &bindings).await, + Step::MergeBase(m) => { + run_merge_base_step(host, &session, m, binding, &bindings).await + } + Step::MergeDryRun(m) => { + run_merge_dry_run_step(host, &session, m, binding).await + } Step::Restart { ordinal } => { generation += 1; host.observe(|| format!("lifetime: reopen generation {generation}")); @@ -3968,6 +4581,23 @@ fn expectation_evidence(step: &Step) -> Value { } Step::Show(s) => read_expect_evidence(&s.expect), Step::Restart { .. } => serde_json::json!({"kind": "restart", "storage": "preserved"}), + Step::Diff(d) => serde_json::json!({ + "kind": "diff", + "sides": format!("{:?}", d.sides), + "expect": format!("{:?}", d.expect), + }), + Step::MergeBase(m) => serde_json::json!({ + "kind": "merge-base", + "source": m.source, + "target": m.target, + "expect": format!("{:?}", m.expect), + }), + Step::MergeDryRun(m) => serde_json::json!({ + "kind": "merge-dry-run", + "source": m.source, + "target": m.target, + "expect": format!("{:?}", m.expect), + }), Step::Concurrent(c) => serde_json::json!({ "kind": "concurrent", "sessions": c.sessions.iter().map(|s| serde_json::json!({"label": s.label, "branch": s.branch, "kind": s.kind.name(), "expect": format!("{:?}", s.expect)})).collect::>(), @@ -4075,6 +4705,9 @@ pub enum StepKind { Show, Restart, Concurrent, + Diff, + MergeBase, + MergeDryRun, } #[derive(Clone, Debug)] @@ -4145,6 +4778,9 @@ impl Step { Self::Show(_) => StepKind::Show, Self::Restart { .. } => StepKind::Restart, Self::Concurrent(_) => StepKind::Concurrent, + Self::Diff(_) => StepKind::Diff, + Self::MergeBase(_) => StepKind::MergeBase, + Self::MergeDryRun(_) => StepKind::MergeDryRun, } } } diff --git a/crates/omnigraph-gqt-core/src/served.rs b/crates/omnigraph-gqt-core/src/served.rs index a8df0ac21..1fdb7350c 100644 --- a/crates/omnigraph-gqt-core/src/served.rs +++ b/crates/omnigraph-gqt-core/src/served.rs @@ -8,8 +8,9 @@ use futures::FutureExt as _; use omnigraph::db::MergeOutcome; use omnigraph::loader::LoadMode; use omnigraph_api_types::{ - BranchMergeOutcome, BranchOutcomeOutput, ChangeOutput, ChangeRequest, HTTP_API_CONTRACT, - HTTP_API_CONTRACT_HEADER, IngestRequest, QueryRequest, ReadOutput, SchemaOutput, + BranchMergeOutcome, BranchMergeOutput, BranchMergeRequest, BranchOutcomeOutput, ChangeOutput, + ChangeRequest, CommitChangesOutput, DiffOutput, HTTP_API_CONTRACT, HTTP_API_CONTRACT_HEADER, + IngestRequest, MergeBaseOutput, QueryRequest, ReadOutput, SchemaOutput, }; use omnigraph_compiler::query::ast::show_statement_name; use serde::de::DeserializeOwned; @@ -18,11 +19,13 @@ use serde_json::Value; use std::time::Duration; use crate::{ - Bindings, Case, ControlStep, ControlWrite, ExecutionHost, Fixture, Item, ListStep, LoadStep, - MAIN_BRANCH, MergeExpect, MutateExpect, MutateStep, ParamsInput, QueryExpect, QueryStep, Seed, - ShowStep, Step, StepFail, WriteExpect, bound_snapshot, bound_source, check_error_expect, - check_rows_json, expectation_evidence, merge_outcome_word, operation, record_binding, - resolve_operand, step_kind, step_label, substitute_all, + Bindings, Case, ControlStep, ControlWrite, DiffSides, DiffStep, ExecutionHost, Fixture, Item, + ListStep, LoadStep, MAIN_BRANCH, MergeBaseStep, MergeDryRunStep, MergeExpect, MutateExpect, + MutateStep, ParamsInput, QueryExpect, QueryStep, Seed, ShowStep, Step, StepFail, WriteExpect, + bound_snapshot, bound_source, check_diff_expect, check_dry_run_expect, check_error_expect, + check_merge_base_expect, check_rows_json, diff_row, expectation_evidence, merge_outcome_word, + operation, record_binding, resolve_operand, resolve_side, step_kind, step_label, + substitute_all, }; /// The server a `--server` run addresses: its base URL, the graph id under @@ -92,7 +95,14 @@ pub fn admit_served(case: &Case) -> Result<(), String> { Step::Show(show) if !show.prefix.is_empty() => { refused.push(format!("step {ordinal} (show with a settings prefix)")); } - Step::Mutate(_) | Step::Load(_) | Step::Control(_) | Step::List(_) | Step::Show(_) => {} + Step::Mutate(_) + | Step::Load(_) + | Step::Control(_) + | Step::List(_) + | Step::Show(_) + | Step::Diff(_) + | Step::MergeBase(_) + | Step::MergeDryRun(_) => {} } } if refused.is_empty() { @@ -157,10 +167,10 @@ async fn execute_served_inner( case.source_lines.get(&ordinal).map(|line| *line as u64), step_kind(step), ); - let outcome = - match step { - Step::Query(q) => query(host, &client, q, binding, &bindings).await, - Step::Mutate(m) => mutate(host, &client, m, binding, &bindings) + let outcome = match step { + Step::Query(q) => query(host, &client, q, binding, &bindings).await, + Step::Mutate(m) => { + mutate(host, &client, m, binding, &bindings) .await .and_then(|commit| { record_binding( @@ -169,27 +179,31 @@ async fn execute_served_inner( commit, &step_label(ordinal, "mutate", binding), ) - }), - Step::Load(l) => load(host, &client, l, binding).await, - Step::Control(c) => control(host, &client, c, binding, &bindings) - .await - .and_then(|commit| { - record_binding( - &mut bindings, - c.bind.as_deref(), - commit, - &step_label(ordinal, c.name, binding), - ) - }), - Step::List(l) => list(host, &client, l, binding).await, - Step::Show(s) => show(host, &client, s, binding).await, - Step::Settings(_) | Step::Restart { .. } | Step::Concurrent(_) => { - Err(StepFail::new( - step_label(ordinal, step_kind(step), binding), - "unsupported_environment: refused by served admission".into(), - )) - } - }; + }) + } + Step::Load(l) => load(host, &client, l, binding).await, + Step::Control(c) => control(host, &client, c, binding, &bindings) + .await + .and_then(|commit| { + record_binding( + &mut bindings, + c.bind.as_deref(), + commit, + &step_label(ordinal, c.name, binding), + ) + }), + Step::List(l) => list(host, &client, l, binding).await, + Step::Show(s) => show(host, &client, s, binding).await, + Step::Diff(d) => diff(host, &client, d, binding, &bindings).await, + Step::MergeBase(m) => merge_base(host, &client, m, binding, &bindings).await, + Step::MergeDryRun(m) => merge_dry_run(host, &client, m, binding).await, + Step::Settings(_) | Step::Restart { .. } | Step::Concurrent(_) => { + Err(StepFail::new( + step_label(ordinal, step_kind(step), binding), + "unsupported_environment: refused by served admission".into(), + )) + } + }; host.measure_step_end(ordinal as u64); host.record("assertion", || match &outcome { Ok(()) => serde_json::json!({"status": "passed"}), @@ -355,6 +369,20 @@ impl ServerClient { .await } + /// A GET whose query parameters the client encodes. + async fn get_query( + &self, + route: &str, + query: &[(&str, &str)], + ) -> Result, String> { + self.send( + self.http + .get(format!("{}{route}", self.graph_base)) + .query(query), + ) + .await + } + async fn load(&self, branch: &str, mode: LoadMode, data: String) -> Result<(), String> { let request = IngestRequest { branch: Some(branch.to_string()), @@ -887,3 +915,152 @@ async fn statement( }; client.read(&request).await } + +/// Every change a diff step names, page by page through its route: +/// `GET /diff`, `GET /branches/diff` or `GET /commits/{id}/changes`. +async fn served_diff_rows( + client: &ServerClient, + sides: &DiffSides, + bindings: &Bindings, +) -> Result, String>, String> { + let mut rows = Vec::new(); + let mut token: Option = None; + loop { + let from: String; + let to: String; + let mut query: Vec<(&str, &str)> = Vec::new(); + let page_changes = match sides { + DiffSides::Commit { commit } => { + let id = resolve_side(commit, bindings)?; + if let Some(token) = token.as_deref() { + query.push(("page_token", token)); + } + let route = format!("/commits/{id}/changes"); + match client + .get_query::(&route, &query) + .await? + { + Ok(page) => Ok((page.changes, page.next_page_token)), + Err(error) => Err(error), + } + } + DiffSides::Between { from: f, to: t } => { + from = resolve_side(f, bindings)?; + to = resolve_side(t, bindings)?; + query.extend([("from", from.as_str()), ("to", to.as_str())]); + if let Some(token) = token.as_deref() { + query.push(("page_token", token)); + } + match client.get_query::("/diff", &query).await? { + Ok(page) => Ok((page.changes, page.next_page_token)), + Err(error) => Err(error), + } + } + DiffSides::Branch { source, target } => { + query.extend([("source", source.as_str()), ("target", target.as_str())]); + if let Some(token) = token.as_deref() { + query.push(("page_token", token)); + } + match client + .get_query::("/branches/diff", &query) + .await? + { + Ok(page) => Ok((page.changes, page.next_page_token)), + Err(error) => Err(error), + } + } + }; + let (changes, next) = match page_changes { + Ok(page) => page, + Err(error) => return Ok(Err(error)), + }; + rows.extend(changes.iter().map(diff_row)); + match next { + Some(next) => token = Some(next), + None => return Ok(Ok(rows)), + } + } +} + +async fn diff( + host: &impl ExecutionHost, + client: &ServerClient, + step: &DiffStep, + binding: Option<(&str, &str)>, + bindings: &Bindings, +) -> Result<(), StepFail> { + let label = step_label(step.ordinal(), "diff", binding); + let fail = |message: String| StepFail::new(label.clone(), message); + let outcome = operation( + host, + step.ordinal(), + served_diff_rows(client, &step.sides, bindings), + ) + .await + .map_err(&fail)? + .map_err(&fail)?; + host.observe(|| format!("actual diff: {outcome:?}")); + check_diff_expect(host, step.expect(), outcome).map_err(fail) +} + +async fn merge_base( + host: &impl ExecutionHost, + client: &ServerClient, + step: &MergeBaseStep, + binding: Option<(&str, &str)>, + bindings: &Bindings, +) -> Result<(), StepFail> { + let label = step_label(step.ordinal(), "merge-base", binding); + let fail = |message: String| StepFail::new(label.clone(), message); + let query = [ + ("source", step.source.as_str()), + ("target", step.target.as_str()), + ]; + let outcome = operation( + host, + step.ordinal(), + client.get_query::("/branches/merge-base", &query), + ) + .await + .map_err(&fail)? + .map_err(&fail)? + .map(|output| output.merge_base.graph_commit_id); + host.observe(|| format!("actual merge base: {outcome:?}")); + check_merge_base_expect(host, step.expect(), outcome, bindings).map_err(fail) +} + +async fn merge_dry_run( + host: &impl ExecutionHost, + client: &ServerClient, + step: &MergeDryRunStep, + binding: Option<(&str, &str)>, +) -> Result<(), StepFail> { + let label = step_label(step.ordinal(), "merge-dry-run", binding); + let fail = |message: String| StepFail::new(label.clone(), message); + let request = BranchMergeRequest { + source: step.source.clone(), + target: Some(step.target.clone()), + delete_branch: false, + as_commit: false, + dry_run: true, + settings: None, + }; + let outcome = operation( + host, + step.ordinal(), + client.post::<_, BranchMergeOutput>("/branches/merge", &request), + ) + .await + .map_err(&fail)? + .map_err(&fail)? + .map(|output| { + let outcome = match output.outcome { + BranchMergeOutcome::AlreadyUpToDate => MergeOutcome::AlreadyUpToDate, + BranchMergeOutcome::FastForward => MergeOutcome::FastForward, + BranchMergeOutcome::Merged => MergeOutcome::Merged, + }; + (outcome, output.commit.is_some() || !output.dry_run) + }); + host.observe(|| format!("actual dry run: {outcome:?}")); + check_dry_run_expect(host, step.expect(), outcome).map_err(fail) +} diff --git a/crates/omnigraph-gqt-core/src/tests.rs b/crates/omnigraph-gqt-core/src/tests.rs index 03f7aa021..91556befe 100644 --- a/crates/omnigraph-gqt-core/src/tests.rs +++ b/crates/omnigraph-gqt-core/src/tests.rs @@ -666,6 +666,131 @@ fn refuses_outcome_on_any_step_but_a_merge() { } } +#[test] +fn diff_steps_parse_their_sides_and_expects() { + let bind = "--- mutate\nquery add() {\n insert Person { name: \"bob\" }\n}\n--- expect ok\n--- bind commit $c\n"; + let text = format!( + "{HDR}{SCHEMA}{SEED}{bind}\ + --- diff from: $c to: main\n--- expect ordered\n{{\"id\": \"bob\"}}\n\ + --- diff source: feature\n--- expect unordered\n\ + --- diff commit: $c\n--- expect error: nope\n\ + --- merge-base source: feature target: main\n--- expect commit $c\n\ + --- merge-dry-run source: feature\n--- expect outcome: merged\n" + ); + let case = parse_case("x", &text).unwrap(); + let steps: Vec<&Step> = case + .items + .iter() + .map(|item| match item { + Item::Step(step) => step, + Item::Loop { .. } => panic!("no loops here"), + }) + .collect(); + let [ + _, + Step::Diff(between), + Step::Diff(branch), + Step::Diff(commit), + Step::MergeBase(base), + Step::MergeDryRun(dry), + ] = steps.as_slice() + else { + panic!("unexpected steps: {steps:?}"); + }; + assert_eq!( + between.sides, + DiffSides::Between { + from: "$c".into(), + to: "main".into() + } + ); + assert!(matches!( + between.expect(), + DiffExpect::Rows { ordered: true, rows } if rows.len() == 1 + )); + assert_eq!( + branch.sides, + DiffSides::Branch { + source: "feature".into(), + target: "main".into() + } + ); + assert!(matches!( + branch.expect(), + DiffExpect::Rows { ordered: false, rows } if rows.is_empty() + )); + assert_eq!( + commit.sides, + DiffSides::Commit { + commit: "$c".into() + } + ); + assert!(matches!(commit.expect(), DiffExpect::Error { needle } if needle == "nope")); + assert_eq!( + (base.source.as_str(), base.target.as_str()), + ("feature", "main") + ); + assert!(matches!(base.expect(), MergeBaseExpect::Commit { name } if name == "c")); + assert_eq!( + (dry.source.as_str(), dry.target.as_str()), + ("feature", "main") + ); + assert!(matches!( + dry.expect(), + MergeExpect::Outcome(MergeOutcome::Merged) + )); +} + +#[test] +fn diff_steps_refuse_bad_arguments_and_expects() { + for (steps, want) in [ + ( + "--- diff from: $c to: main\n--- expect ordered\n", + "`--- diff` side `$c` is not bound", + ), + ( + "--- diff from: main source: feature\n--- expect ordered\n", + "`--- diff` takes `from:` and `to:`, `source:` with an optional `target:`, or `commit:`", + ), + ( + "--- diff branch: main\n--- expect ordered\n", + "`--- diff` takes `from:`, `to:`, `source:`, `target:`, `commit:` as `key: value`, got `branch:`", + ), + ( + "--- diff from: main to: main\n--- expect ok\n", + "a `--- diff` step takes `ordered` or `unordered` rows, or `error:`", + ), + ( + "--- diff from: main to: main\n--- expect ordered\n[1]\n", + "a diff row is one JSON object", + ), + ( + "--- merge-base target: main\n--- expect error: x\n", + "`--- merge-base` needs `source:`", + ), + ( + "--- merge-base source: feature\n--- expect commit $c\n", + "`expect commit $c`: `$c` is not bound", + ), + ( + "--- merge-dry-run source: feature\n--- expect ordered\n", + "rows", + ), + ( + "--- mutate\nbranch create b0\n--- expect commit $c\n", + "`expect commit $name` is accepted on a `--- merge-base` step only", + ), + ( + "--- diff from: main to: main\nbody\n--- expect ordered\n", + "`--- diff`", + ), + ] { + let text = format!("{HDR}{SCHEMA}{SEED}{steps}"); + let error = refusal("x", &text); + assert!(error.contains(want), "{steps}: {error}"); + } +} + #[test] fn outcome_takes_the_three_merge_words() { for (word, want) in [ diff --git a/crates/omnigraph-gqt/README.md b/crates/omnigraph-gqt/README.md index c9f2e8d44..9d6641342 100644 --- a/crates/omnigraph-gqt/README.md +++ b/crates/omnigraph-gqt/README.md @@ -130,6 +130,22 @@ inherits its base's recorded target. A name is bound once, before every use, outside a loop, and survives `--- restart`; the served path binds from the `ChangeOutput.commit` the route answered and sends `snapshot` on the read. +Three steps cover the graph diff surfaces (RFC 0030 §8). A `--- diff` step +names its sides as `from: to: ` (a two-point diff), +`source: ` with an optional `target: ` (a branch diff against +the merge base; `main` when unspelled), or `commit: ` (one commit's own +changes, `commit changes`). A side is a branch name or a bound `$name`. Its +expect is `ordered` or `unordered` rows, one JSON object per change in the +wire shape with `type` spelled as the type's name, or `error:`. Rows compare +as JSON whatever their key order; the step follows every page. +`--- merge-base source: [target: ]` expects +`--- expect commit $name`, the bound commit the merge base must be, or `error:`. +`--- merge-dry-run source: [target: ]` takes a merge's own +`outcome:` and `error:` expects and fails if the dry run reports a commit. The +engine and DST paths call the engine; the served path sends `GET /diff`, +`GET /branches/diff`, `GET /commits/{id}/changes`, `GET /branches/merge-base` +and `POST /branches/merge` with `dry_run: true`. + The configuration accepts 1–16 distinct environment parameter sets, 1–600000 milliseconds, and 1–64 distinct unsigned 64-bit seeds per DST environment. The required corpus refuses budgets above 10000 milliseconds; diff --git a/crates/omnigraph-gqt/cases/graph_diff_branch_review.gqt b/crates/omnigraph-gqt/cases/graph_diff_branch_review.gqt new file mode 100644 index 000000000..3181c5e7a --- /dev/null +++ b/crates/omnigraph-gqt/cases/graph_diff_branch_review.gqt @@ -0,0 +1,100 @@ +# issue: none +# notes: Reviewing a branch (RFC 0030 §8): the merge base, the branch diff from it to the source head, +# notes: the two-point diff of the heads, and a dry run that writes nothing. + +--- runner +timeout_ms: 10000 +environments: + - target: omnigraph-engine + storage: local-filesystem + - target: omnigraph-engine-dst + storage: in-memory-object-store + seeds: [0, 42] + - target: omnigraph-server + storage: local-filesystem + +--- schema +node Person { + name: String @key + age: I32 +} + +--- seed +{"type":"Person","data":{"name":"alice","age":30}} +{"type":"Person","data":{"name":"bob","age":40}} + +--- mutate +query insert_fork() { + insert Person { name: "fork", age: 1 } +} + +--- expect affected: nodes=1 edges=0 + +--- bind commit $fork + +--- mutate +branch create feature + +--- expect ok + +--- mutate branch: feature +query age_alice() { + update Person set { age: 31 } where name = "alice" +} + +--- expect affected: nodes=1 edges=0 + +--- mutate branch: feature +query insert_carol() { + insert Person { name: "carol", age: 22 } +} + +--- expect affected: nodes=1 edges=0 + +--- mutate +query insert_dave() { + insert Person { name: "dave", age: 50 } +} + +--- expect affected: nodes=1 edges=0 + +--- merge-base source: feature + +--- expect commit $fork + +--- diff source: feature target: main + +--- expect ordered +{"kind":"node","type":"Person","id":"alice","op":"update","before":{"properties":{"name":"alice","age":30}},"after":{"properties":{"name":"alice","age":31}}} +{"kind":"node","type":"Person","id":"carol","op":"insert","after":{"properties":{"name":"carol","age":22}}} + +--- diff from: main to: feature + +--- expect ordered +{"kind":"node","type":"Person","id":"alice","op":"update","before":{"properties":{"name":"alice","age":30}},"after":{"properties":{"name":"alice","age":31}}} +{"kind":"node","type":"Person","id":"carol","op":"insert","after":{"properties":{"name":"carol","age":22}}} +{"kind":"node","type":"Person","id":"dave","op":"delete","before":{"properties":{"name":"dave","age":50}}} + +--- diff from: $fork to: main + +--- expect ordered +{"kind":"node","type":"Person","id":"dave","op":"insert","after":{"properties":{"name":"dave","age":50}}} + +--- merge-dry-run source: feature + +--- expect outcome: merged + +--- query +query names_on_main() { + match { $p: Person } + return { $p.name } +} + +--- expect unordered +{"p.name": "alice"} +{"p.name": "bob"} +{"p.name": "dave"} +{"p.name": "fork"} + +--- expect shape +p.name: String diff --git a/crates/omnigraph-gqt/cases/graph_diff_merge_proposal.gqt b/crates/omnigraph-gqt/cases/graph_diff_merge_proposal.gqt new file mode 100644 index 000000000..e11164e77 --- /dev/null +++ b/crates/omnigraph-gqt/cases/graph_diff_merge_proposal.gqt @@ -0,0 +1,83 @@ +# issue: none +# notes: A merge proposal under review (RFC 0030 §8): its own diff is the branch's changes, a diff from +# notes: main to the proposal names it by commit id, and once published the branch has nothing left to +# notes: merge and nothing left to review. + +--- runner +timeout_ms: 10000 +environments: + - target: omnigraph-engine + storage: local-filesystem + - target: omnigraph-engine-dst + storage: in-memory-object-store + seeds: [0, 42] + - target: omnigraph-server + storage: local-filesystem + +--- schema +node Person { + name: String @key + age: I32 +} + +--- seed +{"type":"Person","data":{"name":"alice","age":30}} +{"type":"Person","data":{"name":"bob","age":40}} + +--- mutate +branch create feature + +--- expect ok + +--- mutate branch: feature +query age_alice() { + update Person set { age: 31 } where name = "alice" +} + +--- expect affected: nodes=1 edges=0 + +--- mutate branch: feature +query insert_carol() { + insert Person { name: "carol", age: 22 } +} + +--- expect affected: nodes=1 edges=0 + +--- mutate +query insert_dave() { + insert Person { name: "dave", age: 50 } +} + +--- expect affected: nodes=1 edges=0 + +--- mutate +branch merge feature into main as commit + +--- expect outcome: merged + +--- bind commit $proposal + +--- diff commit: $proposal + +--- expect ordered +{"kind":"node","type":"Person","id":"alice","op":"update","before":{"properties":{"name":"alice","age":30}},"after":{"properties":{"name":"alice","age":31}}} +{"kind":"node","type":"Person","id":"carol","op":"insert","after":{"properties":{"name":"carol","age":22}}} + +--- diff from: main to: $proposal + +--- expect ordered +{"kind":"node","type":"Person","id":"alice","op":"update","before":{"properties":{"name":"alice","age":30}},"after":{"properties":{"name":"alice","age":31}}} +{"kind":"node","type":"Person","id":"carol","op":"insert","after":{"properties":{"name":"carol","age":22}}} + +--- mutate +commit publish "${proposal}" + +--- expect ok + +--- merge-dry-run source: feature + +--- expect outcome: already_up_to_date + +--- diff source: feature + +--- expect ordered diff --git a/crates/omnigraph-gqt/cases/graph_diff_net_range.gqt b/crates/omnigraph-gqt/cases/graph_diff_net_range.gqt new file mode 100644 index 000000000..2c6673849 --- /dev/null +++ b/crates/omnigraph-gqt/cases/graph_diff_net_range.gqt @@ -0,0 +1,128 @@ +# issue: none +# notes: A graph diff compares its endpoints and never folds history (RFC 0030 §8.2): a reverted update, +# notes: an insert later deleted and an equal reinsert produce nothing; a changed reinsert is one update; +# notes: an insert later updated is one insert with the final image. A commit's own diff is its change. + +--- runner +timeout_ms: 10000 +environments: + - target: omnigraph-engine + storage: local-filesystem + - target: omnigraph-engine-dst + storage: in-memory-object-store + seeds: [0, 42] + - target: omnigraph-server + storage: local-filesystem + +--- schema +node Person { + name: String @key + age: I32 +} + +--- seed +{"type":"Person","data":{"name":"alice","age":30}} +{"type":"Person","data":{"name":"hank","age":40}} +{"type":"Person","data":{"name":"ivan","age":20}} + +--- mutate +query mark() { + insert Person { name: "start", age: 0 } +} + +--- expect affected: nodes=1 edges=0 + +--- bind commit $start + +--- mutate +query alice_ages() { + update Person set { age: 31 } where name = "alice" +} + +--- expect affected: nodes=1 edges=0 + +--- mutate +query alice_reverts() { + update Person set { age: 30 } where name = "alice" +} + +--- expect affected: nodes=1 edges=0 + +--- mutate +query gina_arrives() { + insert Person { name: "gina", age: 50 } +} + +--- expect affected: nodes=1 edges=0 + +--- mutate +query gina_leaves() { + delete Person where name = "gina" +} + +--- expect affected: nodes=1 edges=0 + +--- mutate +query hank_leaves() { + delete Person where name = "hank" +} + +--- expect affected: nodes=1 edges=0 + +--- mutate +query hank_returns() { + insert Person { name: "hank", age: 40 } +} + +--- expect affected: nodes=1 edges=0 + +--- mutate +query ivan_leaves() { + delete Person where name = "ivan" +} + +--- expect affected: nodes=1 edges=0 + +--- mutate +query ivan_returns_older() { + insert Person { name: "ivan", age: 21 } +} + +--- expect affected: nodes=1 edges=0 + +--- mutate +query jill_arrives() { + insert Person { name: "jill", age: 18 } +} + +--- expect affected: nodes=1 edges=0 + +--- mutate +query jill_ages() { + update Person set { age: 19 } where name = "jill" +} + +--- expect affected: nodes=1 edges=0 + +--- bind commit $last + +--- diff from: $start to: main + +--- expect ordered +{"kind":"node","type":"Person","id":"ivan","op":"update","before":{"properties":{"name":"ivan","age":20}},"after":{"properties":{"name":"ivan","age":21}}} +{"kind":"node","type":"Person","id":"jill","op":"insert","after":{"properties":{"name":"jill","age":19}}} + +--- diff commit: $last + +--- expect ordered +{"kind":"node","type":"Person","id":"jill","op":"update","before":{"properties":{"name":"jill","age":18}},"after":{"properties":{"name":"jill","age":19}}} + +--- diff from: main to: $start + +--- expect unordered +{"kind":"node","type":"Person","id":"jill","op":"delete","before":{"properties":{"name":"jill","age":19}}} +{"kind":"node","type":"Person","id":"ivan","op":"update","before":{"properties":{"name":"ivan","age":21}},"after":{"properties":{"name":"ivan","age":20}}} + +--- diff from: main to: main + +--- expect ordered diff --git a/crates/omnigraph-gqt/cases/graph_diff_refusals.gqt b/crates/omnigraph-gqt/cases/graph_diff_refusals.gqt new file mode 100644 index 000000000..1ef1411d6 --- /dev/null +++ b/crates/omnigraph-gqt/cases/graph_diff_refusals.gqt @@ -0,0 +1,72 @@ +# issue: none +# notes: Graph diff refusals (RFC 0030 §8): a side naming nothing is a typed not-found, a merge dry run +# notes: answers a conflicting merge's own conflict list and writes nothing, and a missing branch has no +# notes: merge base. + +--- runner +timeout_ms: 10000 +environments: + - target: omnigraph-engine + storage: local-filesystem + - target: omnigraph-engine-dst + storage: in-memory-object-store + seeds: [0, 42] + - target: omnigraph-server + storage: local-filesystem + +--- schema +node Person { + name: String @key + age: I32 +} + +--- seed +{"type":"Person","data":{"name":"bob","age":40}} + +--- diff from: main to: nowhere + +--- expect error: 'nowhere' names no branch and no commit + +--- merge-base source: nowhere + +--- expect error: not found + +--- mutate +branch create feature + +--- expect ok + +--- mutate branch: feature +query feature_bob() { + update Person set { age: 41 } where name = "bob" +} + +--- expect affected: nodes=1 edges=0 + +--- mutate +query main_bob() { + update Person set { age: 42 } where name = "bob" +} + +--- expect affected: nodes=1 edges=0 + +--- merge-dry-run source: feature + +--- expect error: merge conflicts + +--- query +query bob_on_main() { + match { $p: Person { name: "bob" } } + return { $p.age } +} + +--- expect unordered +{"p.age": 42} + +--- expect shape +p.age: I32 + +--- mutate +branch merge feature into main + +--- expect error: merge conflicts diff --git a/crates/omnigraph-planner/src/operation.rs b/crates/omnigraph-planner/src/operation.rs index db99318b9..1ff256329 100644 --- a/crates/omnigraph-planner/src/operation.rs +++ b/crates/omnigraph-planner/src/operation.rs @@ -59,7 +59,8 @@ pub enum Operation { resume: Option, budget: PageBudgetSpec, }, - /// The unpaged `diff_commits` pair, arbitrary and cross-branch. + /// The graph diff's two endpoints (`Omnigraph::diff_page`), arbitrary and + /// cross-branch. SnapshotDiff { from: Side, to: Side }, /// The three merge cursors (milestone 2). MergeClassify { diff --git a/crates/omnigraph-server/src/handlers.rs b/crates/omnigraph-server/src/handlers.rs index ae6d72c54..cae570c5c 100644 --- a/crates/omnigraph-server/src/handlers.rs +++ b/crates/omnigraph-server/src/handlers.rs @@ -2884,6 +2884,11 @@ async fn branch_delete_body( /// carries this merge's own target publication, including for a fast-forward; /// an already-up-to-date merge returns `commit: null`. /// +/// With `dry_run: true` the merge stops after classification and validation: +/// the response reports the outcome it would have now (`commit: null`, +/// `dry_run: true`), a conflict answers the same 409, and nothing is written. +/// A dry run is a read of both branches and needs no `branch_merge` grant. +/// /// With `delete_branch: true` the source branch is deleted after a successful /// merge, under its own `branch_delete` policy check. The merge is durable by /// then, so a deletion refusal or failure never fails the request; it is @@ -2905,9 +2910,37 @@ pub(crate) async fn server_branch_merge( "a merge `as_commit` keeps its source until the commit is published or dropped; drop `delete_branch`", )); } + if request.dry_run && (request.as_commit || request.delete_branch) { + return Err(ApiError::bad_request( + "a dry run writes nothing; drop `as_commit` and `delete_branch`", + )); + } let target = request.target.unwrap_or_else(|| "main".to_string()); let actor_ref = actor.as_ref().map(|Extension(actor)| actor); let session = state.session(&handle, request.settings.as_ref())?; + if request.dry_run { + // A read of both branches: no merge admission, no write gate, no + // `branch_merge` grant. Conflicts answer the same 409 a merge does. + authorize_branch_reads( + actor_ref, + handle.policy.as_deref(), + [&request.source, &target], + )?; + let result = session + .branch_merge_dry_run(&request.source, &target) + .await + .map_err(ApiError::from_omni)?; + return Ok(Json(BranchMergeOutput { + source: request.source, + target, + outcome: result.outcome.into(), + commit: None, + actor_id: actor_ref.map(|actor| actor.actor_id.as_ref().to_string()), + branch_deleted: None, + branch_delete_error_details: None, + dry_run: true, + })); + } let admission = admit_branch_merge(&state, &handle, actor_ref, &request.source, &target)?; let actor = actor.map(|Extension(actor)| actor); state @@ -2950,6 +2983,7 @@ pub(crate) async fn server_branch_merge( .map(|actor| actor.actor_id.as_ref().to_string()), branch_deleted, branch_delete_error_details, + dry_run: false, })), } })? @@ -3747,6 +3781,12 @@ mod blob_error_tests { #[derive(Default)] pub(crate) struct ParsedChangeParams { pub branch: Option, + /// The graph diff's read targets (`GET /diff`). + pub from: Option, + pub to: Option, + /// The branch diff's and merge base's branches. + pub source: Option, + pub target: Option, pub cursor: Option, pub start: Option, pub page_token: Option, @@ -3778,6 +3818,18 @@ fn parse_set_parameter(raw: &str) -> std::result::Result<(SettingId, SettingValu pub(crate) const COMMIT_CHANGES_PARAMS: &[&str] = &["page_token", "limit", "kind", "type", "op", "set"]; +pub(crate) const DIFF_PARAMS: &[&str] = + &["from", "to", "page_token", "limit", "kind", "type", "op"]; +pub(crate) const BRANCH_DIFF_PARAMS: &[&str] = &[ + "source", + "target", + "page_token", + "limit", + "kind", + "type", + "op", +]; +pub(crate) const MERGE_BASE_PARAMS: &[&str] = &["source", "target"]; pub(crate) fn parse_change_query( raw: Option<&str>, allowed: &[&str], @@ -3809,6 +3861,10 @@ pub(crate) fn parse_change_query( let value = value.into_owned(); match name { "branch" => set_single(&mut params.branch, name, value)?, + "from" => set_single(&mut params.from, name, value)?, + "to" => set_single(&mut params.to, name, value)?, + "source" => set_single(&mut params.source, name, value)?, + "target" => set_single(&mut params.target, name, value)?, "cursor" => set_single(&mut params.cursor, name, value)?, "start" => set_single(&mut params.start, name, value)?, "page_token" => set_single(&mut params.page_token, name, value)?, @@ -3939,6 +3995,219 @@ fn validate_change_http_limit(limit: Option) -> std::result::Result<(), A Ok(()) } +/// A side of a graph diff the caller must spell. +fn required_diff_param(value: Option, name: &str) -> std::result::Result { + match value { + Some(value) if !value.trim().is_empty() => Ok(value.trim().to_string()), + _ => Err(ApiError::bad_request(format!( + "query parameter '{name}' is required" + ))), + } +} + +/// Authorize `read` on the branch one graph diff side resolves to: a branch +/// name, or the authored branch (a proposal's or snapshot's recorded target) +/// of a commit. The side is resolved before the policy check, so a denial is +/// reported exactly as a side that names nothing: the route is not an +/// existence oracle for branches or commits the actor cannot read. +async fn authorize_diff_side( + db: &Omnigraph, + actor: Option<&AuthenticatedActor>, + policy: Option<&PolicyEngine>, + side: &str, +) -> std::result::Result<(), ApiError> { + let not_found = || ApiError::not_found(format!("'{side}' names no branch and no commit")); + let branch = match db.diff_read_target_branch(side).await { + Ok(branch) => branch, + // A branch deleted between resolution steps is the same fixed 404. + Err(OmniError::BranchNotFound { .. }) => return Err(not_found()), + Err(error) => return Err(change_route_error(error)), + }; + match authorize( + actor, + policy, + PolicyRequest { + action: PolicyAction::Read, + branch: Some(branch), + target_branch: None, + }, + )? { + Authz::Allowed => Ok(()), + Authz::Denied(_) => Err(not_found()), + } +} + +#[utoipa::path( + get, + path = "/diff", + tag = "changes", + operation_id = "getDiff", + params(api::DiffQuery), + responses( + (status = 200, description = "Net entity changes between the two resolved commits, in frozen (kind, type, id, op) order, without a cause", body = api::DiffOutput), + (status = 400, description = "A missing side, an invalid filter or limit, a side naming both a branch and a commit, or a rejected page token", body = api::ChangeErrorOutput), + (status = 401, description = "Unauthorized", body = api::ChangeErrorOutput), + // No 403: a side the actor cannot read is indistinguishable from a + // side that names nothing (404). + (status = 404, description = "A side names no branch and no commit, or the actor cannot read the branch it resolves to", body = api::ChangeErrorOutput), + (status = 409, description = "The two sides cross a schema boundary; see change_diff_refusal", body = api::ChangeErrorOutput), + (status = 410, description = "Retained history for an endpoint is no longer readable; see change_feed_gap", body = api::ChangeErrorOutput), + (status = 413, description = "Requested limit exceeds the public change ceiling", body = api::ChangeErrorOutput), + (status = 500, description = "Internal failure while reading changes", body = api::ChangeErrorOutput), + (status = 503, description = "Recovery required before changes can be read", body = api::ChangeErrorOutput), + ), + security(("bearer_token" = [])), +)] +/// Net entity changes between two read targets. +/// +/// `from` and `to` each name a branch (its head now) or a graph commit +/// (published, an unpublished proposal, or an unattached snapshot); a string +/// naming both is refused. The first page resolves both sides to exact +/// commits and names them; a large result continues via the opaque +/// `page_token`, and every page answers for those same commits. +pub(crate) async fn server_diff( + Extension(handle): Extension, + actor: Option>, + axum::extract::RawQuery(raw): axum::extract::RawQuery, +) -> std::result::Result, ApiError> { + let params = parse_change_query(raw.as_deref(), DIFF_PARAMS)?; + validate_change_http_limit(params.limit)?; + let from = required_diff_param(params.from, "from")?; + let to = required_diff_param(params.to, "to")?; + let db = &handle.engine; + let actor = actor.as_ref().map(|Extension(actor)| actor); + for side in [&from, &to] { + authorize_diff_side(db, actor, handle.policy.as_deref(), side).await?; + } + let scope = api::change_scope(¶ms.kinds, ¶ms.types, ¶ms.ops); + let page = db + .diff_page(&omnigraph::changes::GraphDiffRequest { + sides: omnigraph::changes::GraphDiffSides::Between { from, to }, + scope, + page_token: params.page_token, + max_changes: params.limit, + max_bytes: None, + }) + .await + .map_err(change_route_error)?; + Ok(Json(api::diff_output(&page))) +} + +/// Authorize `read` on each named branch of a branch diff or merge base. +fn authorize_branch_reads( + actor: Option<&AuthenticatedActor>, + policy: Option<&PolicyEngine>, + branches: [&str; 2], +) -> std::result::Result<(), ApiError> { + for branch in branches { + authorize_request( + actor, + policy, + PolicyRequest { + action: PolicyAction::Read, + branch: Some(branch.to_string()), + target_branch: None, + }, + )?; + } + Ok(()) +} + +#[utoipa::path( + get, + path = "/branches/diff", + tag = "changes", + operation_id = "getBranchDiff", + params(api::BranchDiffQuery), + responses( + (status = 200, description = "What `source` changed since its merge base with `target`: the net entity changes from the base to the source head", body = api::DiffOutput), + (status = 400, description = "A missing source, an invalid filter or limit, or a rejected page token", body = api::ChangeErrorOutput), + (status = 401, description = "Unauthorized", body = api::ChangeErrorOutput), + (status = 403, description = "The actor cannot read one of the branches", body = api::ChangeErrorOutput), + (status = 404, description = "A branch is not found", body = api::ChangeErrorOutput), + (status = 409, description = "The base and the source head cross a schema boundary; see change_diff_refusal", body = api::ChangeErrorOutput), + (status = 410, description = "Retained history for the base is no longer readable; see change_feed_gap", body = api::ChangeErrorOutput), + (status = 413, description = "Requested limit exceeds the public change ceiling", body = api::ChangeErrorOutput), + (status = 500, description = "Internal failure while reading changes", body = api::ChangeErrorOutput), + (status = 503, description = "Recovery required before changes can be read", body = api::ChangeErrorOutput), + ), + security(("bearer_token" = [])), +)] +/// What a branch changed since its merge base with a target. +/// +/// The merge base is the commit `POST /branches/merge` would classify +/// against; it and the source head are captured together. Changes the target +/// made after the base do not appear. Paged like `GET /diff`. +pub(crate) async fn server_branch_diff( + Extension(handle): Extension, + actor: Option>, + axum::extract::RawQuery(raw): axum::extract::RawQuery, +) -> std::result::Result, ApiError> { + let params = parse_change_query(raw.as_deref(), BRANCH_DIFF_PARAMS)?; + validate_change_http_limit(params.limit)?; + let source = normalize_change_branch(Some(&required_diff_param(params.source, "source")?))?; + let target = normalize_change_branch(params.target.as_deref())?; + authorize_branch_reads( + actor.as_ref().map(|Extension(actor)| actor), + handle.policy.as_deref(), + [&source, &target], + )?; + let scope = api::change_scope(¶ms.kinds, ¶ms.types, ¶ms.ops); + let page = handle + .engine + .diff_page(&omnigraph::changes::GraphDiffRequest { + sides: omnigraph::changes::GraphDiffSides::Branch { source, target }, + scope, + page_token: params.page_token, + max_changes: params.limit, + max_bytes: None, + }) + .await + .map_err(change_route_error)?; + Ok(Json(api::diff_output(&page))) +} + +#[utoipa::path( + get, + path = "/branches/merge-base", + tag = "changes", + operation_id = "getMergeBase", + params(api::MergeBaseQuery), + responses( + (status = 200, description = "The commit a merge of `source` into `target` would classify against, with the two heads", body = api::MergeBaseOutput), + (status = 400, description = "A missing or unknown parameter", body = api::ChangeErrorOutput), + (status = 401, description = "Unauthorized", body = api::ChangeErrorOutput), + (status = 403, description = "The actor cannot read one of the branches", body = api::ChangeErrorOutput), + (status = 404, description = "A branch is not found", body = api::ChangeErrorOutput), + (status = 500, description = "Internal failure while reading lineage", body = api::ChangeErrorOutput), + (status = 503, description = "Recovery required before lineage can be read", body = api::ChangeErrorOutput), + ), + security(("bearer_token" = [])), +)] +/// The merge base of two branches. +/// +/// Reads lineage only, by the rule `POST /branches/merge` uses. +pub(crate) async fn server_merge_base( + Extension(handle): Extension, + actor: Option>, + axum::extract::RawQuery(raw): axum::extract::RawQuery, +) -> std::result::Result, ApiError> { + let params = parse_change_query(raw.as_deref(), MERGE_BASE_PARAMS)?; + let source = normalize_change_branch(Some(&required_diff_param(params.source, "source")?))?; + let target = normalize_change_branch(params.target.as_deref())?; + authorize_branch_reads( + actor.as_ref().map(|Extension(actor)| actor), + handle.policy.as_deref(), + [&source, &target], + )?; + let base = handle + .engine + .merge_base(&source, &target) + .await + .map_err(change_route_error)?; + Ok(Json(api::merge_base_output(&source, &target, &base))) +} + /// Contextual 404 projection. `Manifest::NotFound` is a broad internal /// category, so its original text is never reused; the handler supplies the /// exact public resource spelling it attempted to resolve. @@ -3998,6 +4267,8 @@ fn change_route_error(error: OmniError) -> ApiError { } safe @ (OmniError::ChangeCursorRejected { .. } | OmniError::BranchNotFound { .. } + | OmniError::ReadTargetNotFound { .. } + | OmniError::AmbiguousReadTarget { .. } | OmniError::ChangeFeedGap { .. } | OmniError::CommitHasNoParent { .. } | OmniError::ChangeSchemaBoundary { .. }) => ApiError::from_omni(safe), diff --git a/crates/omnigraph-server/src/http_contract.rs b/crates/omnigraph-server/src/http_contract.rs index 2edcfda2d..d36019563 100644 --- a/crates/omnigraph-server/src/http_contract.rs +++ b/crates/omnigraph-server/src/http_contract.rs @@ -125,7 +125,10 @@ pub(crate) fn describe_contract(doc: &mut utoipa::openapi::OpenApi) { response.content.insert( "application/json".into(), Content::new(Some(Ref::from_schema_name( - if path.contains("/changes") { + if path.contains("/changes") + || path.ends_with("/diff") + || path.ends_with("/merge-base") + { "ChangeErrorOutput" } else { "ErrorOutput" diff --git a/crates/omnigraph-server/src/lib.rs b/crates/omnigraph-server/src/lib.rs index fabdcf375..406b22cc6 100644 --- a/crates/omnigraph-server/src/lib.rs +++ b/crates/omnigraph-server/src/lib.rs @@ -144,6 +144,9 @@ fn hash_bearer_token(token: &str) -> BearerTokenHash { handlers::server_commit_publish, handlers::server_commit_drop, handlers::server_commit_changes, + handlers::server_diff, + handlers::server_branch_diff, + handlers::server_merge_base, handlers::server_changes_feed, handlers::server_changes_baseline, ), @@ -1474,6 +1477,8 @@ impl ApiError { OmniError::BranchNotFound { branch } => { Self::not_found(format!("branch '{branch}' not found")) } + error @ OmniError::ReadTargetNotFound { .. } => Self::not_found(error.to_string()), + error @ OmniError::AmbiguousReadTarget { .. } => Self::bad_request(error.to_string()), // Retention loss under a change continuation: 410 with the // structured resume hint. Recovery is the baseline handshake. OmniError::ChangeFeedGap { @@ -2652,6 +2657,8 @@ pub fn build_app(state: AppState) -> Router { ) .route("/branches/{branch}", delete(server_branch_delete)) .route("/branches/merge", post(server_branch_merge)) + .route("/branches/diff", get(server_branch_diff)) + .route("/branches/merge-base", get(server_merge_base)) .route("/commits", get(server_commit_list)) .route( "/commits/{commit_id}", @@ -2659,6 +2666,7 @@ pub fn build_app(state: AppState) -> Router { ) .route("/commits/{commit_id}/publish", post(server_commit_publish)) .route("/commits/{commit_id}/changes", get(server_commit_changes)) + .route("/diff", get(server_diff)) .route("/changes", get(server_changes_feed)) .route("/changes/baseline", post(server_changes_baseline)) .route_layer(middleware::from_fn_with_state( diff --git a/crates/omnigraph-server/src/redacted_cause.rs b/crates/omnigraph-server/src/redacted_cause.rs index fc6233bb7..a125aac9b 100644 --- a/crates/omnigraph-server/src/redacted_cause.rs +++ b/crates/omnigraph-server/src/redacted_cause.rs @@ -38,6 +38,8 @@ impl RedactedCause { OmniError::ResourceLimitExceeded { .. } => "ResourceLimitExceeded", OmniError::ChangeCursorRejected { .. } => "ChangeCursorRejected", OmniError::BranchNotFound { .. } => "BranchNotFound", + OmniError::ReadTargetNotFound { .. } => "ReadTargetNotFound", + OmniError::AmbiguousReadTarget { .. } => "AmbiguousReadTarget", OmniError::ChangeFeedGap { .. } => "ChangeFeedGap", OmniError::CommitHasNoParent { .. } => "CommitHasNoParent", OmniError::ChangeSchemaBoundary { .. } => "ChangeSchemaBoundary", diff --git a/crates/omnigraph-server/tests/integration/auth_policy.rs b/crates/omnigraph-server/tests/integration/auth_policy.rs index bc57f83a3..f0a92a0e0 100644 --- a/crates/omnigraph-server/tests/integration/auth_policy.rs +++ b/crates/omnigraph-server/tests/integration/auth_policy.rs @@ -443,6 +443,9 @@ async fn signed_data_requires_cedar_and_rejects_forgery_on_every_protected_route (Method::GET, "/commits"), (Method::GET, "/commits/unknown"), (Method::GET, "/commits/unknown/changes"), + (Method::GET, "/diff"), + (Method::GET, "/branches/diff"), + (Method::GET, "/branches/merge-base"), (Method::GET, "/changes"), (Method::POST, "/changes/baseline"), ] { @@ -1683,6 +1686,114 @@ async fn policy_commit_diff_forbidden_is_indistinguishable_from_unknown() { forbidden.code, unknown.code, "forbidden and unknown must share the same error code" ); + + // A graph diff resolves each side before it authorizes the branch the side + // resolves to, so a forbidden side, commit or branch, answers exactly as a + // side that names nothing. + let get = |uri: String| { + Request::builder() + .header(HTTP_API_CONTRACT_HEADER, HTTP_API_CONTRACT) + .uri(g(&uri)) + .method(Method::GET) + .header("authorization", "Bearer team-token") + .body(Body::empty()) + .unwrap() + }; + let (status, readable) = json_response(&app, get("/diff?from=main&to=main".into())).await; + assert_eq!(status, StatusCode::OK, "{readable}"); + for (forbidden_side, unknown_side) in [ + (feature_commit.as_str(), unknown_id), + ("feature", "no-such-branch"), + ] { + let (forbidden_status, forbidden) = + json_response(&app, get(format!("/diff?from=main&to={forbidden_side}"))).await; + let (unknown_status, unknown) = + json_response(&app, get(format!("/diff?from=main&to={unknown_side}"))).await; + assert_eq!(forbidden_status, StatusCode::NOT_FOUND, "{forbidden}"); + assert_eq!(unknown_status, StatusCode::NOT_FOUND, "{unknown}"); + assert_eq!( + forbidden["error"], + format!("'{forbidden_side}' names no branch and no commit") + ); + assert_eq!(forbidden["code"], unknown["code"]); + } + // The branch routes name their branches outright: a read denial is a 403, + // and a dry run is a read of both branches. + for uri in [ + "/branches/diff?source=feature", + "/branches/merge-base?source=feature", + ] { + let (status, body) = json_response(&app, get(uri.into())).await; + assert_eq!(status, StatusCode::FORBIDDEN, "{uri}: {body}"); + } + let (status, body) = json_response( + &app, + Request::builder() + .header(HTTP_API_CONTRACT_HEADER, HTTP_API_CONTRACT) + .uri(g("/branches/merge")) + .method(Method::POST) + .header("authorization", "Bearer team-token") + .header("content-type", "application/json") + .body(Body::from( + json!({"source": "feature", "dry_run": true}).to_string(), + )) + .unwrap(), + ) + .await; + assert_eq!(status, StatusCode::FORBIDDEN, "{body}"); +} + +/// A merge dry run is a read of both branches: an actor with `read` and no +/// `branch_merge` grant may run it, and still may not merge. +#[tokio::test(flavor = "multi_thread")] +async fn policy_merge_dry_run_needs_read_and_not_branch_merge() { + let temp = init_loaded_graph().await; + let graph = graph_path(temp.path()); + let db = session(Omnigraph::open(graph.to_str().unwrap()).await.unwrap()); + db.branch_create_from(ReadTarget::branch("main"), "feature") + .await + .unwrap(); + db.load_with_receipt( + "feature", + r#"{"type":"Person","data":{"name":"feature-only","age":40}}"#, + omnigraph::loader::LoadMode::Merge, + ) + .await + .unwrap(); + drop(db); + let policy_path = temp.path().join("policy.yaml"); + fs::write( + &policy_path, + "version: 1\ngroups:\n team: [act-bruno]\nrules:\n - id: read\n allow: {actors: {group: team}, actions: [read], branch_scope: any}\n", + ) + .unwrap(); + let app = build_app( + AppState::open_with_bearer_tokens_and_policy( + graph.to_string_lossy().to_string(), + vec![("act-bruno".to_string(), "team-token".to_string())], + Some(&policy_path), + ) + .await + .unwrap(), + ); + let merge = |dry_run: bool| { + Request::builder() + .header(HTTP_API_CONTRACT_HEADER, HTTP_API_CONTRACT) + .uri(g("/branches/merge")) + .method(Method::POST) + .header("authorization", "Bearer team-token") + .header("content-type", "application/json") + .body(Body::from( + json!({"source": "feature", "dry_run": dry_run}).to_string(), + )) + .unwrap() + }; + let (status, body) = json_response(&app, merge(true)).await; + assert_eq!(status, StatusCode::OK, "{body}"); + assert_eq!(body["outcome"], "fast_forward"); + assert_eq!(body["dry_run"], true); + let (status, body) = json_response(&app, merge(false)).await; + assert_eq!(status, StatusCode::FORBIDDEN, "{body}"); } #[tokio::test(flavor = "multi_thread")] @@ -1841,6 +1952,7 @@ async fn policy_blocks_non_admin_merge_to_main_and_allows_admin() { target: Some("main".to_string()), delete_branch: false, as_commit: false, + dry_run: false, settings: None, }; let (deny_status, deny_body) = json_response( @@ -2219,6 +2331,7 @@ async fn authenticated_branch_merge_stamps_merge_actor_on_head_commit() { target: Some("main".to_string()), delete_branch: false, as_commit: false, + dry_run: false, settings: None, }; let (merge_status, merge_body) = json_response( @@ -2292,6 +2405,7 @@ async fn branch_merge_delete_branch_policy_denial_is_non_fatal() { target: Some("main".to_string()), delete_branch: true, as_commit: false, + dry_run: false, settings: None, }; let (merge_status, merge_body) = json_response( diff --git a/crates/omnigraph-server/tests/integration/data_routes.rs b/crates/omnigraph-server/tests/integration/data_routes.rs index 71570129f..7cdabda4b 100644 --- a/crates/omnigraph-server/tests/integration/data_routes.rs +++ b/crates/omnigraph-server/tests/integration/data_routes.rs @@ -1989,6 +1989,7 @@ async fn branch_merge_conflict_response_includes_structured_conflicts() { target: Some("main".to_string()), delete_branch: false, as_commit: false, + dry_run: false, settings: None, }; let (status, body) = json_response( @@ -3485,6 +3486,7 @@ async fn branch_merge_statement_conflict_matches_the_route_409() { target: Some("main".to_string()), delete_branch: false, as_commit: false, + dry_run: false, settings: None, }; let (route_status, route_body) = json_response( @@ -5225,6 +5227,7 @@ async fn remote_branch_list_create_merge_flow_works() { target: Some("main".to_string()), delete_branch: false, as_commit: false, + dry_run: false, settings: None, }; let (merge_status, merge_body) = json_response( @@ -5378,6 +5381,7 @@ async fn branch_merge_delete_branch_retires_parent_with_live_child() { target: Some("main".to_string()), delete_branch: true, as_commit: false, + dry_run: false, settings: None, }; let (merge_status, merge_body) = json_response( @@ -5438,6 +5442,7 @@ async fn branch_merge_delete_branch_refusal_is_non_fatal() { target: Some("feature".to_string()), delete_branch: true, as_commit: false, + dry_run: false, settings: None, }; let (merge_status, merge_body) = json_response( @@ -6765,6 +6770,230 @@ async fn mutate_graph_commit_precondition_issue_365() { ); } +// ─── Graph diff routes ────────────────────────────────────────────────────── + +/// One `insert_person` through `POST /mutate` on `branch`. +async fn insert_person_on(app: &axum::Router, branch: &str, name: &str) { + let (status, body) = json_response( + app, + json_post( + "/mutate", + &json!({ + "query": MUTATION_QUERIES, + "name": "insert_person", + "params": {"name": name, "age": 30}, + "branch": branch + }), + ), + ) + .await; + assert_eq!(status, StatusCode::OK, "{body}"); +} + +fn change_ids(page: &Value) -> Vec<(String, String)> { + page["changes"] + .as_array() + .expect("a diff page lists its changes") + .iter() + .map(|change| { + ( + change["id"].as_str().unwrap().to_string(), + change["op"].as_str().unwrap().to_string(), + ) + }) + .collect() +} + +#[tokio::test(flavor = "multi_thread")] +async fn diff_routes_review_a_branch_against_its_merge_base() { + let (_temp, app) = app_for_loaded_graph().await; + let fork = main_head_commit_id(&app).await; + branch_one_commit_ahead(&app, "feature", "Featured").await; + insert_person_on(&app, "feature", "Second").await; + insert_person_on(&app, "main", "MainOnly").await; + + let (status, base) = get_json(&app, g("/branches/merge-base?source=feature")).await; + assert_eq!(status, StatusCode::OK, "{base}"); + assert_eq!(base["source"], "feature"); + assert_eq!(base["target"], "main"); + assert_eq!(base["merge_base"]["graph_commit_id"], fork.as_str()); + + // The branch form pages like any change surface; every page names the + // base and the source head it resolved first. + let (status, first) = get_json(&app, g("/branches/diff?source=feature&limit=1")).await; + assert_eq!(status, StatusCode::OK, "{first}"); + assert_eq!(first["base_commit_id"], fork.as_str()); + assert_eq!(first["from_commit_id"], fork.as_str()); + assert_eq!(first["to_commit_id"], base["source_commit_id"]); + assert!(first.get("cause").is_none(), "a diff carries no cause"); + let token = first["next_page_token"] + .as_str() + .expect("two changes, pages of one"); + insert_person_on(&app, "feature", "Late").await; + let (status, second) = get_json( + &app, + g(&format!( + "/branches/diff?source=feature&target=main&limit=1&page_token={token}" + )), + ) + .await; + assert_eq!(status, StatusCode::OK, "{second}"); + assert_eq!(second["to_commit_id"], first["to_commit_id"]); + let mut reviewed = change_ids(&first); + reviewed.extend(change_ids(&second)); + assert!(second["next_page_token"].is_null()); + assert_eq!( + reviewed, + vec![ + ("Featured".to_string(), "insert".to_string()), + ("Second".to_string(), "insert".to_string()), + ] + ); + + // The two-point form compares the heads, so main's own insert reads as a + // delete; a commit id names a side as well as a branch does. + let (status, heads) = get_json(&app, g("/diff?from=main&to=feature")).await; + assert_eq!(status, StatusCode::OK, "{heads}"); + assert!( + change_ids(&heads).contains(&("MainOnly".to_string(), "delete".to_string())), + "{heads}" + ); + let (status, since_fork) = get_json(&app, g(&format!("/diff?from={fork}&to=main"))).await; + assert_eq!(status, StatusCode::OK, "{since_fork}"); + assert_eq!( + change_ids(&since_fork), + vec![("MainOnly".to_string(), "insert".to_string())] + ); +} + +#[tokio::test(flavor = "multi_thread")] +async fn diff_routes_refuse_malformed_and_unknown_sides() { + let (_temp, app) = app_for_loaded_graph().await; + for (uri, status, needle) in [ + ( + "/diff?from=main", + StatusCode::BAD_REQUEST, + "query parameter 'to' is required", + ), + ( + "/diff?from=main&to=main&branch=main", + StatusCode::BAD_REQUEST, + "unknown query parameter 'branch'", + ), + ( + "/diff?from=main&to=main&limit=0", + StatusCode::BAD_REQUEST, + "greater than zero", + ), + ( + "/diff?from=main&to=nowhere", + StatusCode::NOT_FOUND, + "'nowhere' names no branch and no commit", + ), + ( + "/branches/diff?target=main", + StatusCode::BAD_REQUEST, + "query parameter 'source' is required", + ), + ( + "/branches/diff?source=nowhere", + StatusCode::NOT_FOUND, + "branch 'nowhere' not found", + ), + ( + "/branches/merge-base?source=feature&limit=1", + StatusCode::BAD_REQUEST, + "unknown query parameter 'limit'", + ), + ] { + let (got, body) = get_json(&app, g(uri)).await; + assert_eq!(got, status, "{uri}: {body}"); + assert!( + body["error"].as_str().unwrap_or_default().contains(needle), + "{uri}: {body}" + ); + } +} + +#[tokio::test(flavor = "multi_thread")] +async fn merge_dry_run_answers_the_merge_without_writing() { + let (_temp, app) = app_for_loaded_graph().await; + branch_one_commit_ahead(&app, "feature", "Featured").await; + let head = main_head_commit_id(&app).await; + let (status, body) = json_response( + &app, + json_post( + "/branches/merge", + &json!({"source": "feature", "target": "main", "dry_run": true}), + ), + ) + .await; + assert_eq!(status, StatusCode::OK, "{body}"); + assert_eq!(body["outcome"], "fast_forward"); + assert!(body["commit"].is_null(), "{body}"); + assert_eq!(body["dry_run"], true); + assert_eq!( + main_head_commit_id(&app).await, + head, + "a dry run moves nothing" + ); + + let (status, body) = json_response( + &app, + json_post( + "/branches/merge", + &json!({"source": "feature", "dry_run": true, "as_commit": true}), + ), + ) + .await; + assert_eq!(status, StatusCode::BAD_REQUEST, "{body}"); + assert!( + body["error"] + .as_str() + .unwrap() + .contains("a dry run writes nothing") + ); + + // The real merge after the dry run answers the outcome the dry run did. + let (status, body) = json_response( + &app, + json_post("/branches/merge", &json!({"source": "feature"})), + ) + .await; + assert_eq!(status, StatusCode::OK, "{body}"); + assert_eq!(body["outcome"], "fast_forward"); + assert!( + body.get("dry_run").is_none(), + "a merge omits the dry-run flag" + ); +} + +#[tokio::test(flavor = "multi_thread")] +async fn merge_dry_run_conflict_matches_the_merge_409() { + let temp = divergent_alice_graph().await; + let graph = graph_path(temp.path()); + let app = build_app( + AppState::open(graph.to_string_lossy().to_string()) + .await + .unwrap(), + ); + let mut bodies = Vec::new(); + for dry_run in [true, false] { + let (status, body) = json_response( + &app, + json_post( + "/branches/merge", + &json!({"source": "feature", "target": "main", "dry_run": dry_run}), + ), + ) + .await; + assert_eq!(status, StatusCode::CONFLICT, "{body}"); + bodies.push(body); + } + assert_eq!(bodies[0]["merge_conflicts"], bodies[1]["merge_conflicts"]); + assert_eq!(bodies[0]["error"], bodies[1]["error"]); +} + // ─── Commit entity changes route ──────────────────────────────────────────── async fn load_commit(app: &axum::Router, ndjson: &str) -> String { diff --git a/crates/omnigraph-server/tests/integration/openapi.rs b/crates/omnigraph-server/tests/integration/openapi.rs index ed492dba0..d8cbcbd7f 100644 --- a/crates/omnigraph-server/tests/integration/openapi.rs +++ b/crates/omnigraph-server/tests/integration/openapi.rs @@ -214,6 +214,9 @@ const EXPECTED_PATHS: &[&str] = &[ "/graphs/{graph_id}/branches", "/graphs/{graph_id}/branches/{branch}", "/graphs/{graph_id}/branches/merge", + "/graphs/{graph_id}/branches/diff", + "/graphs/{graph_id}/branches/merge-base", + "/graphs/{graph_id}/diff", "/graphs/{graph_id}/commits", "/graphs/{graph_id}/commits/{commit_id}", "/graphs/{graph_id}/commits/{commit_id}/publish", @@ -222,6 +225,12 @@ const EXPECTED_PATHS: &[&str] = &[ "/graphs/{graph_id}/changes/baseline", ]; +/// The routes that answer in the change vocabulary: the change feed, a +/// commit's changes, and the graph diff, branch diff and merge base. +fn is_change_route(path: &str) -> bool { + path.contains("/changes") || path.ends_with("/diff") || path.ends_with("/merge-base") +} + #[test] fn openapi_contains_all_expected_paths() { let doc = openapi_json(); @@ -481,12 +490,12 @@ fn openapi_change_operations_reach_only_graph_vocabulary_schemas() { .as_object() .unwrap() .iter() - .filter(|(path, _)| path.contains("/changes")) + .filter(|(path, _)| is_change_route(path)) .collect(); assert_eq!( change_paths.len(), - 3, - "the three change routes are registered: {change_paths:?}" + 6, + "the six change routes are registered: {change_paths:?}" ); let mut frontier = std::collections::BTreeSet::new(); @@ -1787,6 +1796,9 @@ fn protected_endpoints_reference_bearer_token_security() { ("/graphs/{graph_id}/branches", "post"), ("/graphs/{graph_id}/branches/{branch}", "delete"), ("/graphs/{graph_id}/branches/merge", "post"), + ("/graphs/{graph_id}/branches/diff", "get"), + ("/graphs/{graph_id}/branches/merge-base", "get"), + ("/graphs/{graph_id}/diff", "get"), ("/graphs/{graph_id}/commits", "get"), ("/graphs/{graph_id}/commits/{commit_id}", "get"), ("/graphs/{graph_id}/commits/{commit_id}", "delete"), @@ -2240,6 +2252,9 @@ const EXPECTED_CLUSTER_PATHS: &[&str] = &[ "/graphs/{graph_id}/branches", "/graphs/{graph_id}/branches/{branch}", "/graphs/{graph_id}/branches/merge", + "/graphs/{graph_id}/branches/diff", + "/graphs/{graph_id}/branches/merge-base", + "/graphs/{graph_id}/diff", "/graphs/{graph_id}/commits", "/graphs/{graph_id}/commits/{commit_id}", "/graphs/{graph_id}/commits/{commit_id}/publish", @@ -2319,6 +2334,9 @@ async fn multi_mode_openapi_drops_flat_protected_paths() { "/branches", "/branches/{branch}", "/branches/merge", + "/branches/diff", + "/branches/merge-base", + "/diff", "/commits", "/commits/{commit_id}", "/commits/{commit_id}/publish", @@ -2551,6 +2569,9 @@ async fn served_spec_always_nests_under_cluster_prefix() { "/branches", "/branches/{branch}", "/branches/merge", + "/branches/diff", + "/branches/merge-base", + "/diff", "/commits", "/commits/{commit_id}", "/commits/{commit_id}/publish", @@ -2614,7 +2635,7 @@ fn openapi_describes_api_contract_admission_and_response_identity() { } else { // The admission envelope is {error, code}, also valid under // the change feed's narrower graph-vocabulary projection. - let error_schema = if path.contains("/changes") { + let error_schema = if is_change_route(path) { "#/components/schemas/ChangeErrorOutput" } else { "#/components/schemas/ErrorOutput" diff --git a/crates/omnigraph-server/tests/support/mod.rs b/crates/omnigraph-server/tests/support/mod.rs index a2450d3a5..11db1aca3 100644 --- a/crates/omnigraph-server/tests/support/mod.rs +++ b/crates/omnigraph-server/tests/support/mod.rs @@ -877,6 +877,7 @@ pub mod matrix { target: Some(target), delete_branch: false, as_commit: false, + dry_run: false, settings: None, }) .unwrap(); @@ -1159,6 +1160,7 @@ pub async fn http_merge_decision( target: Some("main".to_string()), delete_branch: false, as_commit: false, + dry_run: false, settings: None, }; let (status, _body) = json_response( diff --git a/crates/omnigraph/src/changes/mod.rs b/crates/omnigraph/src/changes/mod.rs index 1cd336c15..fc4668730 100644 --- a/crates/omnigraph/src/changes/mod.rs +++ b/crates/omnigraph/src/changes/mod.rs @@ -11,95 +11,26 @@ pub use model::{ COMMIT_CHANGES_MAX_CHANGES, ChangeBaseline, ChangeCause, ChangeEntityKind, ChangeFeedContinuation, ChangeFeedPage, ChangeFeedPosition, ChangeFeedRequest, ChangeFeedScope, ChangeFeedStart, ChangeOpKind, CommitChangesPage, EntityEndpoints, EntityImage, - GraphChangeBlock, GraphEntityChange, GraphTypeRef, + GraphChangeBlock, GraphDiffPage, GraphDiffRequest, GraphDiffSides, GraphEntityChange, + GraphTypeRef, MergeBase, }; -use std::collections::{BTreeMap, HashSet}; +use std::collections::BTreeMap; -use arrow_array::{Array, RecordBatch, StringArray, UInt64Array}; -use omnigraph_compiler::SystemColumns; - -use self::candidate_scan::{EmitSource, interval_candidate_plan}; -use self::enumerate::Emit; -use self::row_compare::{ - OrderedRows, RawRow, ScanTargets, rows_equal_across_vintages, user_schema_fingerprint, -}; use crate::db::DatasetEntry; use crate::db::Snapshot; -use crate::db::manifest::{TableIdentity, system_columns_at_image}; -use crate::error::{OmniError, Result}; -use crate::staging::is_detached_version; -use crate::storage_layer::{SnapshotHandle, TableStorage}; -use crate::table_store::TableStore; +use crate::db::manifest::TableIdentity; // ─── Types ────────────────────────────────────────────────────────────────── +/// The namespace of a graph entity: node or edge. #[derive(Debug, Clone, Copy, PartialEq, Eq)] pub enum EntityKind { Node, Edge, } -#[derive(Debug, Clone, Copy, PartialEq, Eq)] -pub enum ChangeOp { - Insert, - Update, - Delete, -} - -#[derive(Debug, Clone, PartialEq, Eq)] -pub struct Endpoints { - pub src: String, - pub dst: String, -} - -#[derive(Debug, Clone)] -pub struct EntityChange { - pub kind: EntityKind, - pub type_name: String, - pub id: String, - pub op: ChangeOp, - pub published_dataset_version: u64, - pub endpoints: Option, -} - -impl EntityChange { - /// Canonical qualified graph type key derived from the authoritative kind - /// and type name (for example, `node:Person`). - pub fn type_key(&self) -> String { - let prefix = match self.kind { - EntityKind::Node => "node", - EntityKind::Edge => "edge", - }; - format!("{prefix}:{}", self.type_name) - } -} - -#[derive(Debug, Clone, Default)] -pub struct ChangeFilter { - pub kinds: Option>, - pub type_names: Option>, - pub ops: Option>, -} - -#[derive(Debug, Clone, Default)] -pub struct ChangeStats { - pub inserts: usize, - pub updates: usize, - pub deletes: usize, - pub types_affected: Vec, -} - -#[derive(Debug, Clone)] -pub struct ChangeSet { - pub from_graph_manifest_version: u64, - pub to_graph_manifest_version: u64, - pub graph_branch: Option, - pub changes: Vec, - pub stats: ChangeStats, -} - -// ─── Filter helpers ───────────────────────────────────────────────────────── +// ─── Table lifetimes ──────────────────────────────────────────────────────── fn parse_table_key(table_key: &str) -> (EntityKind, &str) { if let Some(name) = table_key.strip_prefix("node:") { @@ -111,32 +42,6 @@ fn parse_table_key(table_key: &str) -> (EntityKind, &str) { } } -impl ChangeFilter { - fn matches_table(&self, table_key: &str) -> bool { - let (kind, type_name) = parse_table_key(table_key); - if let Some(ref kinds) = self.kinds { - if !kinds.contains(&kind) { - return false; - } - } - if let Some(ref names) = self.type_names { - if !names.iter().any(|n| n == type_name) { - return false; - } - } - true - } - - fn wants_op(&self, op: ChangeOp) -> bool { - match &self.ops { - Some(ops) => ops.contains(&op), - None => true, - } - } -} - -// ─── Core diff ────────────────────────────────────────────────────────────── - /// One immutable table lifetime whose physical state differs between two /// graph snapshots. /// @@ -193,115 +98,6 @@ pub(crate) fn changed_table_intervals<'a>( intervals } -/// Net-current diff between two snapshots. -/// -/// Uses a three-level algorithm: -/// 1. Manifest diff — skip unchanged sub-tables -/// 2. Lineage check — same branch → version-column diff; different → ID-based diff -/// 3. Row-level diff -pub(crate) async fn diff_snapshots( - table_store: &TableStore, - from: &Snapshot, - to: &Snapshot, - filter: &ChangeFilter, - branch: Option, - // Graph-vocabulary identity of the `to` side, used only to name a - // cross-branch schema boundary. `None`/empty when the target is a raw - // snapshot with no resolved commit. - to_commit_id: Option, -) -> Result { - let mut changes = Vec::new(); - - for interval in changed_table_intervals(from, to) { - let from_entry = interval.from; - let to_entry = interval.to; - // Prefer the destination alias for a rename; a removed table has only - // its source alias. Logical pairing never depends on either name. - let table_key = &to_entry - .or(from_entry) - .expect("identity came from one snapshot") - .type_key; - debug_assert!( - from_entry - .into_iter() - .chain(to_entry) - .all(|entry| entry.identity == interval.identity), - "table interval endpoints must retain their immutable identity" - ); - if !filter.matches_table(table_key) { - continue; - } - - let (kind, type_name) = parse_table_key(table_key); - let is_edge = kind == EntityKind::Edge; - - let table_changes = match (from_entry, to_entry) { - // Table added — all rows are inserts - (None, Some(to)) => diff_table_added(table_store, to, is_edge, filter).await?, - // Table removed — all rows are deletes - (Some(from), None) => diff_table_removed(table_store, from, is_edge, filter).await?, - // Fast path: version-column diff - (Some(from), Some(to)) if same_lineage(from_entry, to_entry) => { - match diff_table_same_lineage(table_store, from, to, is_edge, filter).await? { - Some(changes) => changes, - // RFC 0067: the `to` endpoint is a pending pin's detached - // version. Rows staged on a detached predecessor carry - // stamps outside the linear interval, so compare exactly. - None => { - diff_table_cross_branch( - table_store, - from, - to, - is_edge, - filter, - type_name, - to_commit_id.as_deref().unwrap_or_default(), - ) - .await? - } - } - } - // Cross-branch path: streaming ID-based diff - (Some(from), Some(to)) => { - diff_table_cross_branch( - table_store, - from, - to, - is_edge, - filter, - type_name, - to_commit_id.as_deref().unwrap_or_default(), - ) - .await? - } - // Unreachable: `same_state` above already skipped absent-on-both-sides tables. - (None, None) => continue, - }; - - let fallback_published_dataset_version = to_entry - .or(from_entry) - .expect("identity came from one snapshot") - .published_dataset_version; - for mut c in table_changes { - c.kind = kind; - c.type_name = type_name.to_string(); - if c.published_dataset_version == 0 { - c.published_dataset_version = fallback_published_dataset_version; - } - changes.push(c); - } - } - - let stats = compute_stats(&changes); - Ok(ChangeSet { - from_graph_manifest_version: from.graph_manifest_version(), - to_graph_manifest_version: to.graph_manifest_version(), - graph_branch: branch, - changes, - stats, - }) -} - /// Whether two registrations name one table state, the pin included: two /// branches staging detached on one dataset share `published_dataset_version` /// while their pins differ (RFC "Detached-only tables", Branches). @@ -317,552 +113,3 @@ fn same_state(a: Option<&DatasetEntry>, b: Option<&DatasetEntry>) -> bool { _ => false, } } - -fn same_lineage(from: Option<&DatasetEntry>, to: Option<&DatasetEntry>) -> bool { - match (from, to) { - (Some(f), Some(t)) => f.native_dataset_branch == t.native_dataset_branch, - _ => false, - } -} - -fn compute_stats(changes: &[EntityChange]) -> ChangeStats { - let mut stats = ChangeStats::default(); - let mut types = HashSet::new(); - for c in changes { - match c.op { - ChangeOp::Insert => stats.inserts += 1, - ChangeOp::Update => stats.updates += 1, - ChangeOp::Delete => stats.deletes += 1, - } - types.insert(c.type_name.clone()); - } - stats.types_affected = types.into_iter().collect(); - stats.types_affected.sort(); - stats -} - -// ─── Fast path: version-column diff ───────────────────────────────────────── - -/// The stamp path over a forward interval of one chain, or `None` for the -/// exact compare: a detached `to` its transaction cannot prove, or a `from` -/// at or past a linear `to` (another branch's chain of the same dataset). -async fn diff_table_same_lineage( - table_store: &TableStore, - from_entry: &DatasetEntry, - to_entry: &DatasetEntry, - is_edge: bool, - filter: &ChangeFilter, -) -> Result>> { - let vf = from_entry.published_dataset_version; - let vt = to_entry.published_dataset_version; - let storage: &dyn TableStorage = table_store; - let to_ds = storage.open_snapshot_at_entry(to_entry).await?; - if is_detached_version(to_ds.dataset().version().version) { - return diff_table_detached_commit(storage, from_entry, to_entry, &to_ds, is_edge, filter) - .await; - } - if vf >= vt { - return Ok(None); - } - let to_columns = system_columns_at_image(to_ds.dataset().schema(), &to_entry.type_key)?; - - let cols: Vec<&str> = if is_edge { - vec![ - to_columns.id, - to_columns.src, - to_columns.dst, - "_row_last_updated_at_version", - ] - } else { - vec![to_columns.id, "_row_last_updated_at_version"] - }; - - let wants_inserts = filter.wants_op(ChangeOp::Insert); - let wants_updates = filter.wants_op(ChangeOp::Update); - let wants_deletes = filter.wants_op(ChangeOp::Delete); - - let mut changes = Vec::new(); - - // Inserts + Updates: use _row_last_updated_at_version to find all rows - // touched since Vf, then classify by checking whether the ID existed at Vf. - // - // We key on _row_last_updated_at_version because one scan over it catches - // every row touched in the window — inserts and updates alike — regardless - // of write mode, and ID-set membership at Vf then distinguishes inserts from - // updates. (lance#6774 made merge_insert stamp new rows' _row_created_at_version - // with the commit version, so created_at became reliable too; last_updated - // stays the right key since it also covers updates.) - let changed_rows = if wants_inserts || wants_updates { - let filter_sql = format!( - "_row_last_updated_at_version > {} AND _row_last_updated_at_version <= {}", - vf, vt - ); - scan_with_filter(storage, &to_ds, &cols, &filter_sql, to_columns).await? - } else { - Vec::new() - }; - if changed_rows.is_empty() && !wants_deletes { - return Ok(Some(changes)); - } - let from_ds = storage.open_snapshot_at_entry(from_entry).await?; - let from_columns = system_columns_at_image(from_ds.dataset().schema(), &from_entry.type_key)?; - if !changed_rows.is_empty() { - let from_ids: HashSet = - scan_id_set(storage, &from_ds, &[from_columns.id], from_columns) - .await? - .into_iter() - .map(|r| r.id) - .collect(); - for row in changed_rows { - if from_ids.contains(&row.id) { - if wants_updates { - changes.push(entity_change_from_row(&row, ChangeOp::Update, is_edge)); - } - } else if wants_inserts { - changes.push(entity_change_from_row(&row, ChangeOp::Insert, is_edge)); - } - } - } - if wants_deletes { - changes.extend( - deleted_ids_by_set_diff(storage, &from_ds, &to_ds, is_edge, from_columns, to_columns) - .await?, - ); - } - - Ok(Some(changes)) -} - -/// The commit path of a same-lineage interval whose `to` side is a detached -/// pin (RFC "Detached-only tables"): its own transaction proves a one-commit -/// interval, else `None` for the exact compare; changes carry the pin's target. -async fn diff_table_detached_commit( - storage: &dyn TableStorage, - from_entry: &DatasetEntry, - to_entry: &DatasetEntry, - to_ds: &SnapshotHandle, - is_edge: bool, - filter: &ChangeFilter, -) -> Result>> { - let from_ds = storage.open_snapshot_at_entry(from_entry).await?; - let Some(plan) = - interval_candidate_plan(from_entry, to_entry, from_ds.dataset(), to_ds.dataset()).await? - else { - return Ok(None); - }; - let from_columns = system_columns_at_image(from_ds.dataset().schema(), &from_entry.type_key)?; - let to_columns = system_columns_at_image(to_ds.dataset().schema(), &to_entry.type_key)?; - let mut source = EmitSource::plan( - from_entry, - from_ds.dataset().clone(), - to_ds.dataset().clone(), - Some(plan), - None, - &ChangeFeedScope::default(), - ScanTargets::default(), - from_columns, - to_columns, - ) - .await?; - let mut changes = Vec::new(); - while let Some(emit) = source.next().await? { - let (raw, op, columns) = match &emit { - Emit::Insert(raw) => (raw, ChangeOp::Insert, to_columns), - Emit::Update { after, .. } => (after, ChangeOp::Update, to_columns), - Emit::Delete(raw) => (raw, ChangeOp::Delete, from_columns), - }; - if !filter.wants_op(op) { - continue; - } - let mut change = entity_change_from_raw(raw, op, is_edge, columns); - change.published_dataset_version = to_entry.published_dataset_version; - changes.push(change); - } - Ok(Some(changes)) -} - -// ─── Cross-branch path: streaming ID-based diff ──────────────────────────── - -async fn diff_table_cross_branch( - table_store: &TableStore, - from_entry: &DatasetEntry, - to_entry: &DatasetEntry, - is_edge: bool, - filter: &ChangeFilter, - type_name: &str, - to_commit_id: &str, -) -> Result> { - // Stream both snapshots id-ordered and merge them, using the SAME typed - // row-equality that the per-commit enumerator uses (`row_compare`). The - // former display-string signature rendered null and `""` identically and - // skipped every `_row_`-prefixed column, so a `""`↔null flip or a legal - // `_row_notes` change was silently dropped; typed Arrow comparison keeps - // them distinct and only skips the five reserved virtual columns. - let from_dataset = table_store.open_at_entry(from_entry).await?; - let to_dataset = table_store.open_at_entry(to_entry).await?; - let from_columns = system_columns_at_image(from_dataset.schema(), &from_entry.type_key)?; - let to_columns = system_columns_at_image(to_dataset.schema(), &to_entry.type_key)?; - - // Schema-boundary gate, symmetric with the per-commit enumerator. The typed - // row equality below walks the left row's fields, so it is only sound when - // both sides share one user schema. Today no two branch lifetimes of one - // table can diverge in user schema (schema apply requires a graph with only - // main, so a branch cannot be evolved separately), which is why this gate - // does not fire on any supported operation. It is load-bearing for future - // branch-scoped schema evolution: it turns a divergent-schema pair into a - // loud typed refusal instead of a silently dropped update (extra column on - // the right) or a `manifest_internal` error (extra column on the left). - if user_schema_fingerprint(&from_dataset, from_columns, is_edge) - != user_schema_fingerprint(&to_dataset, to_columns, is_edge) - { - return Err(OmniError::ChangeSchemaBoundary { - graph_commit_id: to_commit_id.to_string(), - type_name: type_name.to_string(), - }); - } - - let mut from = OrderedRows::open(from_dataset, None, from_columns.id).await?; - let mut to = OrderedRows::open(to_dataset, None, to_columns.id).await?; - - let mut changes = Vec::new(); - loop { - let from_id = from.peek().await?.map(|row| row.id.clone()); - let to_id = to.peek().await?.map(|row| row.id.clone()); - match (from_id, to_id) { - (None, None) => break, - // ID only in from → Delete - (Some(_), None) => { - let row = from.pop().await?.expect("peeked row present"); - if filter.wants_op(ChangeOp::Delete) { - changes.push(entity_change_from_raw( - &row, - ChangeOp::Delete, - is_edge, - from_columns, - )); - } - } - // ID only in to → Insert - (None, Some(_)) => { - let row = to.pop().await?.expect("peeked row present"); - if filter.wants_op(ChangeOp::Insert) { - changes.push(entity_change_from_raw( - &row, - ChangeOp::Insert, - is_edge, - to_columns, - )); - } - } - (Some(fid), Some(tid)) if fid < tid => { - let row = from.pop().await?.expect("peeked row present"); - if filter.wants_op(ChangeOp::Delete) { - changes.push(entity_change_from_raw( - &row, - ChangeOp::Delete, - is_edge, - from_columns, - )); - } - } - (Some(fid), Some(tid)) if fid > tid => { - let row = to.pop().await?.expect("peeked row present"); - if filter.wants_op(ChangeOp::Insert) { - changes.push(entity_change_from_raw( - &row, - ChangeOp::Insert, - is_edge, - to_columns, - )); - } - } - // Same ID — typed structural comparison, Blob-descriptor aware. The - // managed-Blob identity is qualified by the immutable data-file UUID, - // which is globally unique, so cross-branch equality is sound without - // a scope hint. - (Some(_), Some(_)) => { - let left = from.pop().await?.expect("peeked row present"); - let right = to.pop().await?.expect("peeked row present"); - if filter.wants_op(ChangeOp::Update) - && !rows_equal_across_vintages( - from.dataset(), - &left, - to.dataset(), - &right, - from_columns, - to_columns, - is_edge, - ) - .await? - { - changes.push(entity_change_from_raw( - &right, - ChangeOp::Update, - is_edge, - to_columns, - )); - } - } - } - } - - Ok(changes) -} - -// ─── Table added/removed ──────────────────────────────────────────────────── - -async fn diff_table_added( - table_store: &TableStore, - to_entry: &DatasetEntry, - is_edge: bool, - filter: &ChangeFilter, -) -> Result> { - if !filter.wants_op(ChangeOp::Insert) { - return Ok(Vec::new()); - } - drain_all_rows(table_store, to_entry, ChangeOp::Insert, is_edge).await -} - -async fn diff_table_removed( - table_store: &TableStore, - from_entry: &DatasetEntry, - is_edge: bool, - filter: &ChangeFilter, -) -> Result> { - if !filter.wants_op(ChangeOp::Delete) { - return Ok(Vec::new()); - } - drain_all_rows(table_store, from_entry, ChangeOp::Delete, is_edge).await -} - -/// Enumerate every row of one table snapshot as the given op. Used for an -/// added table (all inserts) or a removed one (all deletes); the streamed -/// `OrderedRows` shares the enumerator's scan shape so there is one row reader. -async fn drain_all_rows( - table_store: &TableStore, - entry: &DatasetEntry, - op: ChangeOp, - is_edge: bool, -) -> Result> { - let dataset = table_store.open_at_entry(entry).await?; - let system_columns = system_columns_at_image(dataset.schema(), &entry.type_key)?; - let mut rows = OrderedRows::open(dataset, None, system_columns.id).await?; - let mut changes = Vec::new(); - while let Some(row) = rows.pop().await? { - changes.push(entity_change_from_raw(&row, op, is_edge, system_columns)); - } - Ok(changes) -} - -// ─── Helpers ──────────────────────────────────────────────────────────────── - -/// Scan with a SQL filter, projecting specific columns. -async fn scan_with_filter( - storage: &dyn TableStorage, - ds: &SnapshotHandle, - cols: &[&str], - filter_sql: &str, - system_columns: SystemColumns, -) -> Result> { - let batches = storage.scan(ds, Some(cols), Some(filter_sql), None).await?; - extract_rows(&batches, system_columns) -} - -/// Compute deleted IDs: scan id at from and to, set-difference. -async fn deleted_ids_by_set_diff( - storage: &dyn TableStorage, - from_ds: &SnapshotHandle, - to_ds: &SnapshotHandle, - is_edge: bool, - from_columns: SystemColumns, - to_columns: SystemColumns, -) -> Result> { - let cols: Vec<&str> = if is_edge { - vec![from_columns.id, from_columns.src, from_columns.dst] - } else { - vec![from_columns.id] - }; - - let from_rows = scan_id_set(storage, from_ds, &cols, from_columns).await?; - let to_ids: HashSet = scan_id_set(storage, to_ds, &[to_columns.id], to_columns) - .await? - .into_iter() - .map(|r| r.id) - .collect(); - - Ok(from_rows - .into_iter() - .filter(|r| !to_ids.contains(&r.id)) - .map(|r| entity_change_from_row(&r, ChangeOp::Delete, is_edge)) - .collect()) -} - -async fn scan_id_set( - storage: &dyn TableStorage, - ds: &SnapshotHandle, - cols: &[&str], - system_columns: SystemColumns, -) -> Result> { - let batches = storage.scan(ds, Some(cols), None, None).await?; - extract_rows(&batches, system_columns) -} - -// ─── Row extraction ───────────────────────────────────────────────────────── - -#[derive(Debug, Clone)] -struct ScannedRow { - id: String, - src: Option, - dst: Option, - change_version: Option, -} - -fn extract_rows(batches: &[RecordBatch], system_columns: SystemColumns) -> Result> { - let mut rows = Vec::new(); - for batch in batches { - let ids = batch - .column_by_name(system_columns.id) - .and_then(|c| c.as_any().downcast_ref::()) - .filter(|ids| ids.null_count() == 0) - .ok_or_else(|| { - OmniError::manifest_internal(format!( - "change scan requires non-null Utf8 identity column '{}'", - system_columns.id - )) - })?; - let srcs = batch - .column_by_name(system_columns.src) - .and_then(|c| c.as_any().downcast_ref::()); - let dsts = batch - .column_by_name(system_columns.dst) - .and_then(|c| c.as_any().downcast_ref::()); - for i in 0..ids.len() { - rows.push(ScannedRow { - id: ids.value(i).to_string(), - src: srcs.map(|a| a.value(i).to_string()), - dst: dsts.map(|a| a.value(i).to_string()), - change_version: batch - .column_by_name("_row_last_updated_at_version") - .and_then(|c| c.as_any().downcast_ref::()) - .map(|versions| versions.value(i)), - }); - } - } - Ok(rows) -} - -/// Build a change from a scanned key row (same-lineage inserts/updates and the -/// deleted-set path, which classify by id membership and never compare images). -fn entity_change_from_row(row: &ScannedRow, op: ChangeOp, is_edge: bool) -> EntityChange { - EntityChange { - kind: if is_edge { - EntityKind::Edge - } else { - EntityKind::Node - }, - type_name: String::new(), - id: row.id.clone(), - op, - published_dataset_version: row.change_version.unwrap_or(0), - endpoints: if is_edge { - Some(Endpoints { - src: row.src.clone().unwrap_or_default(), - dst: row.dst.clone().unwrap_or_default(), - }) - } else { - None - }, - } -} - -/// Build a change using the system spellings of the image that supplied `raw`. -/// A missing change version leaves 0 for `diff_snapshots` to fill in. -fn entity_change_from_raw( - raw: &RawRow, - op: ChangeOp, - is_edge: bool, - system_columns: SystemColumns, -) -> EntityChange { - let string_col = |name: &str| -> Option { - raw.slice - .column_by_name(name) - .and_then(|c| c.as_any().downcast_ref::()) - .filter(|array| array.is_valid(0)) - .map(|array| array.value(0).to_string()) - }; - let change_version = raw - .slice - .column_by_name("_row_last_updated_at_version") - .and_then(|c| c.as_any().downcast_ref::()) - .filter(|array| array.is_valid(0)) - .map(|array| array.value(0)); - EntityChange { - kind: if is_edge { - EntityKind::Edge - } else { - EntityKind::Node - }, - type_name: String::new(), - id: raw.id.clone(), - op, - published_dataset_version: change_version.unwrap_or(0), - endpoints: if is_edge { - Some(Endpoints { - src: string_col(system_columns.src).unwrap_or_default(), - dst: string_col(system_columns.dst).unwrap_or_default(), - }) - } else { - None - }, - } -} - -#[cfg(test)] -mod tests { - use super::*; - use arrow_schema::{DataType, Field, Schema}; - use omnigraph_compiler::SYSTEM_COLUMNS_V3; - use std::sync::Arc; - - #[test] - fn change_scan_refuses_missing_or_invalid_identity_columns() { - for (name, column) in [ - ( - "id", - Arc::new(StringArray::from(vec![Some("a")])) as Arc, - ), - ( - "__id", - Arc::new(UInt64Array::from(vec![1])) as Arc, - ), - ( - "__id", - Arc::new(StringArray::from(vec![None::<&str>])) as Arc, - ), - ] { - let schema = Arc::new(Schema::new(vec![Field::new( - name, - column.data_type().clone(), - true, - )])); - let batch = RecordBatch::try_new(schema, vec![column]).unwrap(); - let error = extract_rows(&[batch], SYSTEM_COLUMNS_V3).unwrap_err(); - assert!( - error - .to_string() - .contains("requires non-null Utf8 identity column '__id'"), - "{error}" - ); - } - let empty = RecordBatch::new_empty(Arc::new(Schema::new(vec![Field::new( - "__id", - DataType::Utf8, - false, - )]))); - assert!( - extract_rows(&[empty], SYSTEM_COLUMNS_V3) - .unwrap() - .is_empty() - ); - } -} diff --git a/crates/omnigraph/src/changes/model.rs b/crates/omnigraph/src/changes/model.rs index 03e3ed2c4..38b16ce3a 100644 --- a/crates/omnigraph/src/changes/model.rs +++ b/crates/omnigraph/src/changes/model.rs @@ -194,6 +194,55 @@ pub struct CommitChangesPage { pub next_page_token: Option, } +/// The two sides of a graph diff, as the caller names them. A side is a read +/// target: a live branch name, meaning its head when the diff is captured, or +/// a graph commit id (published, an unpublished proposal, or an unattached +/// snapshot). +#[derive(Debug, Clone, PartialEq, Eq)] +pub enum GraphDiffSides { + /// The net transition from `from`'s state to `to`'s state. + Between { from: String, to: String }, + /// What `source` changed since its merge base with `target`: the merge + /// base and the source head, captured together. + Branch { source: String, target: String }, +} + +/// One graph diff request. `None` limits take the server-owned defaults; a +/// page token continues the response whose sides it was minted for and keeps +/// the endpoints that response resolved. +#[derive(Debug, Clone)] +pub struct GraphDiffRequest { + pub sides: GraphDiffSides, + pub scope: ChangeFeedScope, + pub page_token: Option, + pub max_changes: Option, + pub max_bytes: Option, +} + +/// One bounded page of a graph diff: net entity changes between two resolved +/// graph commits, without a cause (a range has no single author). The +/// endpoints are the exact commits the first page resolved; every +/// continuation answers for them, whatever the named branches did since. +#[derive(Debug, Clone)] +pub struct GraphDiffPage { + pub from_commit_id: String, + pub to_commit_id: String, + /// The merge base, present for [`GraphDiffSides::Branch`]; equal to + /// `from_commit_id` there. + pub base_commit_id: Option, + pub changes: Vec, + pub next_page_token: Option, +} + +/// The merge base of a source and a target branch, with the heads it was +/// computed from. +#[derive(Debug, Clone)] +pub struct MergeBase { + pub source_commit_id: String, + pub target_commit_id: String, + pub merge_base: GraphCommit, +} + /// Graph-vocabulary feed filter. Empty dimensions match everything; the /// canonical digest of this scope binds page tokens and feed cursors. #[derive(Debug, Clone, Default, PartialEq, Eq)] diff --git a/crates/omnigraph/src/changes/token.rs b/crates/omnigraph/src/changes/token.rs index b8759d694..3e8d4320a 100644 --- a/crates/omnigraph/src/changes/token.rs +++ b/crates/omnigraph/src/changes/token.rs @@ -1,10 +1,12 @@ //! Opaque continuation tokens and feed cursors. //! -//! Three payload kinds share one envelope: `base64url_no_pad(json ‖ sha256)`. -//! A commit page token continues one bounded commit diff; a feed cursor is the -//! caller-owned durable position after a complete commit; a feed page token -//! continues one interrupted poll at its captured cut. Kind tags make every -//! cross-use a typed rejection instead of a silently misread position. +//! Four payload kinds share one envelope: `base64url_no_pad(json ‖ sha256)`. +//! A commit page token continues one bounded commit diff; a diff page token +//! continues one bounded graph diff between the two commits its first page +//! resolved; a feed cursor is the caller-owned durable position after a +//! complete commit; a feed page token continues one interrupted poll at its +//! captured cut. Kind tags make every cross-use a typed rejection instead of a +//! silently misread position. //! //! Payload contents are private transport state. They never expose table //! aliases, dataset paths, Lance versions, or row addresses; the numeric table @@ -35,6 +37,7 @@ pub(crate) const CONTINUATION_EXACT_ID_MAX_BYTES: usize = 256; const CONTINUATION_ID_PREFIX_MAX_BYTES: usize = 64; pub(crate) const KIND_COMMIT_PAGE: &str = "commit-page"; +pub(crate) const KIND_DIFF_PAGE: &str = "diff-page"; pub(crate) const KIND_FEED_CURSOR: &str = "feed-cursor"; pub(crate) const KIND_FEED_PAGE: &str = "feed-page"; @@ -199,6 +202,30 @@ pub(crate) struct CommitPageTokenV1 { pub change_index: usize, } +/// Continues one bounded graph diff. Binds the request's sides (as a digest, +/// so a branch name never rides in the token), the two commits the first page +/// resolved them to, and the canonical filter scope. Continuations answer for +/// those commits whatever the named branches did since. +#[derive(Debug, Clone, PartialEq, Eq, Serialize, Deserialize)] +pub(crate) struct DiffPageTokenV1 { + pub version: u8, + pub kind: String, + /// Hashed graph identity (schema identity domain), never the raw domain. + pub graph_identity: String, + /// Digest of the request's sides: their form and the names it spelled. + pub sides: String, + pub from_commit_id: String, + pub to_commit_id: String, + /// The merge base, present for the branch form; equal to `from_commit_id`. + pub base_commit_id: Option, + pub filter_digest: String, + /// Opaque graph type identity of the last emitted change. + pub type_id: String, + pub position: IdPositionV1, + pub operation_rank: u8, + pub change_index: usize, +} + /// Durable caller-owned feed position: everything needed to prove the resumed /// chain is the same graph, history incarnation, branch incarnation, and /// filter contract, plus the last completed commit. @@ -251,6 +278,7 @@ pub(crate) struct FeedPageTokenV1 { fn kind_label(kind: &str) -> &'static str { match kind { KIND_COMMIT_PAGE => "commit changes page token", + KIND_DIFF_PAGE => "diff page token", KIND_FEED_CURSOR => "change feed cursor", KIND_FEED_PAGE => "change feed page token", _ => "unrecognized continuation", @@ -277,6 +305,27 @@ pub(crate) fn opaque_type_id(schema_identity_domain: &str, identity: TableIdenti base64::engine::general_purpose::URL_SAFE_NO_PAD.encode(hasher.finalize()) } +/// Digest of a graph diff request's sides. The form is part of the digest, so +/// a two-point request and a branch request over the same names never share a +/// continuation. +pub(crate) fn sides_digest(sides: &super::model::GraphDiffSides) -> String { + let canonical = match sides { + super::model::GraphDiffSides::Between { from, to } => serde_json::json!({ + "v": 1, + "form": "between", + "from": from, + "to": to, + }), + super::model::GraphDiffSides::Branch { source, target } => serde_json::json!({ + "v": 1, + "form": "branch", + "source": source, + "target": target, + }), + }; + hashed_identity(&canonical.to_string()) +} + /// Canonical digest of one feed scope. Sorted and deduplicated per dimension, /// with an explicit all-variants selection collapsing to the same digest as an /// absent filter, so equivalent scopes always bind the same continuation. @@ -403,6 +452,19 @@ pub(crate) fn decode_commit_page_token(token: &str) -> Result Ok(decoded) } +pub(crate) fn decode_diff_page_token(token: &str) -> Result { + let decoded: DiffPageTokenV1 = decode_token(token, KIND_DIFF_PAGE)?; + if decoded.type_id.is_empty() + || decoded.operation_rank > ChangeOpKind::Delete.rank() + || decoded.from_commit_id.is_empty() + || decoded.to_commit_id.is_empty() + { + return Err(cursor_rejected("invalid diff page token identity")); + } + decoded.position.validate("diff page token")?; + Ok(decoded) +} + pub(crate) fn decode_feed_cursor(token: &str) -> Result { let decoded: FeedCursorV1 = decode_token(token, KIND_FEED_CURSOR)?; if decoded.purpose != FEED_PURPOSE { @@ -494,12 +556,42 @@ mod tests { } } + fn diff_page_token() -> DiffPageTokenV1 { + DiffPageTokenV1 { + version: TOKEN_VERSION, + kind: KIND_DIFF_PAGE.to_string(), + graph_identity: hashed_identity("graph"), + sides: sides_digest(&crate::changes::model::GraphDiffSides::Branch { + source: "feature".to_string(), + target: "main".to_string(), + }), + from_commit_id: "base".to_string(), + to_commit_id: "head".to_string(), + base_commit_id: Some("base".to_string()), + filter_digest: filter_digest(&ChangeFeedScope::default()), + type_id: opaque_type_id( + "graph", + TableIdentity { + stable_table_id: 3, + table_incarnation_id: 7, + }, + ), + position: IdPositionV1::for_id("alice"), + operation_rank: 0, + change_index: 2, + } + } + #[test] fn tokens_round_trip() { let token = commit_page_token(); let decoded = decode_commit_page_token(&encode_token(&token).unwrap()).unwrap(); assert_eq!(decoded, token); + let diff = diff_page_token(); + let decoded = decode_diff_page_token(&encode_token(&diff).unwrap()).unwrap(); + assert_eq!(decoded, diff); + let cursor = feed_cursor(); let decoded = decode_feed_cursor(&encode_token(&cursor).unwrap()).unwrap(); assert_eq!(decoded, cursor); @@ -677,6 +769,50 @@ mod tests { let err = decode_feed_page_token(&page).unwrap_err(); assert!(matches!(err, OmniError::ChangeCursorRejected { .. })); + + // A commit page token and a diff page token never stand in for each + // other, even though both continue a bounded entity-change response. + let err = decode_diff_page_token(&page).unwrap_err(); + let OmniError::ChangeCursorRejected { reason } = err else { + panic!("expected rejection"); + }; + assert!( + reason.contains("commit changes page token") && reason.contains("diff page token"), + "{reason}" + ); + let diff = encode_token(&diff_page_token()).unwrap(); + let err = decode_commit_page_token(&diff).unwrap_err(); + assert!(matches!(err, OmniError::ChangeCursorRejected { .. })); + assert!(matches!( + decode_feed_cursor(&diff).unwrap_err(), + OmniError::ChangeCursorRejected { .. } + )); + } + + #[test] + fn sides_digest_binds_the_form_and_the_spelled_names() { + use crate::changes::model::GraphDiffSides; + let between = sides_digest(&GraphDiffSides::Between { + from: "main".to_string(), + to: "feature".to_string(), + }); + let branch = sides_digest(&GraphDiffSides::Branch { + source: "main".to_string(), + target: "feature".to_string(), + }); + let swapped = sides_digest(&GraphDiffSides::Between { + from: "feature".to_string(), + to: "main".to_string(), + }); + assert_ne!(between, branch); + assert_ne!(between, swapped); + assert_eq!( + between, + sides_digest(&GraphDiffSides::Between { + from: "main".to_string(), + to: "feature".to_string(), + }) + ); } #[test] diff --git a/crates/omnigraph/src/db/graph_coordinator.rs b/crates/omnigraph/src/db/graph_coordinator.rs index a360737f6..17a7cbbe9 100644 --- a/crates/omnigraph/src/db/graph_coordinator.rs +++ b/crates/omnigraph/src/db/graph_coordinator.rs @@ -9,8 +9,7 @@ use crate::error::{OmniError, Result}; use crate::storage::{StorageAdapter, normalize_root_uri}; use super::commit_graph::{ - CommitGraph, FirstParentEdge, GraphCommit, HistoryCache, Lineage, - graph_commit_from_manifest_row, + CommitGraph, GraphCommit, HistoryCache, Lineage, graph_commit_from_manifest_row, }; use super::manifest::{ BranchRecords, CapturedManifestProbe, CommitBuffer, DatasetUpdate, ExpectedTableVersions, @@ -95,25 +94,6 @@ pub struct ResolvedTarget { pub snapshot: Snapshot, } -/// Internal lineage classification for an existing two-commit diff request. -/// Arbitrary ranges retain net-current semantics; direct adjacency is derived -/// only from the child's persisted first-parent pointer. -pub(crate) enum ResolvedCommitRange { - FirstParent(FirstParentEdge), - Arbitrary { from: GraphCommit, to: GraphCommit }, -} - -fn classify_commit_range(from: GraphCommit, to: GraphCommit) -> ResolvedCommitRange { - if to.parent_commit_id.as_deref() == Some(from.graph_commit_id.as_str()) { - ResolvedCommitRange::FirstParent(FirstParentEdge { - parent: from, - child: to, - }) - } else { - ResolvedCommitRange::Arbitrary { from, to } - } -} - #[derive(Debug, Clone)] pub(crate) struct PublishedSnapshot { pub graph_manifest_version: u64, @@ -715,22 +695,6 @@ impl GraphCoordinator { } } - /// Resolve both endpoints and classify direct first-parent adjacency from - /// the child's persisted parent pointer. - /// - /// This is deliberately O(1) after the two commits are resolved: it adds - /// no ancestry index or history walk. Arbitrary ranges retain the existing - /// net-current diff semantics. - pub(crate) async fn resolve_commit_range( - &self, - from_id: &SnapshotId, - to_id: &SnapshotId, - ) -> Result { - let from = self.resolve_commit(from_id).await?; - let to = self.resolve_commit(to_id).await?; - Ok(classify_commit_range(from, to)) - } - pub(crate) async fn head_commit_id(&self) -> Result> { Ok(Some(SnapshotId::new( self.manifest.head().graph_commit_id.clone(), @@ -1300,67 +1264,6 @@ mod tests { assert_eq!(&found.graph_commit_id, released); } - fn commit( - id: &str, - parent_commit_id: Option<&str>, - merged_parent_commit_id: Option<&str>, - ) -> GraphCommit { - GraphCommit { - graph_commit_id: id.to_string(), - graph_branch: None, - graph_manifest_version: 1, - generation: 0, - parent_commit_id: parent_commit_id.map(str::to_string), - merged_parent_commit_id: merged_parent_commit_id.map(str::to_string), - actor_id: None, - created_at: 0, - published: true, - kind: crate::db::GraphCommitKind::Published, - root_commit_id: None, - } - } - - #[test] - fn commit_range_classification_uses_only_the_child_first_parent_pointer() { - let root = commit("root", None, None); - let child = commit("child", Some("root"), None); - match classify_commit_range(root.clone(), child.clone()) { - ResolvedCommitRange::FirstParent(edge) => { - assert_eq!(edge.parent.graph_commit_id, "root"); - assert_eq!(edge.child.graph_commit_id, "child"); - } - ResolvedCommitRange::Arbitrary { .. } => { - panic!("a direct child must classify as a first-parent edge") - } - } - assert!(matches!( - classify_commit_range(child.clone(), root.clone()), - ResolvedCommitRange::Arbitrary { .. } - )); - assert!(matches!( - classify_commit_range(root, commit("grandchild", Some("child"), None)), - ResolvedCommitRange::Arbitrary { .. } - )); - - let left = commit("left", Some("root"), None); - let right = commit("right", Some("root"), None); - let merge = commit("merge", Some("left"), Some("right")); - match classify_commit_range(left, merge.clone()) { - ResolvedCommitRange::FirstParent(edge) => { - assert_eq!(edge.parent.graph_commit_id, "left"); - assert_eq!(edge.child.graph_commit_id, "merge"); - assert_eq!(edge.child.merged_parent_commit_id.as_deref(), Some("right")); - } - ResolvedCommitRange::Arbitrary { .. } => { - panic!("a merge must be adjacent only to its persisted first parent") - } - } - assert!(matches!( - classify_commit_range(right, merge), - ResolvedCommitRange::Arbitrary { .. } - )); - } - #[tokio::test] async fn prepared_cold_open_refuses_a_legacy_stamp_published_after_admission() { #[cfg(feature = "failpoints")] diff --git a/crates/omnigraph/src/db/omnigraph.rs b/crates/omnigraph/src/db/omnigraph.rs index 07a148ac9..e048bd30e 100644 --- a/crates/omnigraph/src/db/omnigraph.rs +++ b/crates/omnigraph/src/db/omnigraph.rs @@ -18,9 +18,9 @@ use omnigraph_compiler::{ }; use crate::db::graph_coordinator::{ - CapturedLineage, GraphCoordinator, PublishedSnapshot, ResolvedCommitRange, commit_not_found, + CapturedLineage, GraphCoordinator, PublishedSnapshot, commit_not_found, }; -use crate::error::{OmniError, Result, dataset_subject}; +use crate::error::{ManifestErrorKind, OmniError, Result, dataset_subject}; use crate::runtime_cache::RuntimeCache; use crate::seams::{decide_seam, fail}; use crate::storage::{ @@ -2459,70 +2459,6 @@ impl Omnigraph { // ─── Change detection ──────────────────────────────────────────────── - pub async fn diff_between( - &self, - from: impl Into, - to: impl Into, - filter: &crate::changes::ChangeFilter, - ) -> Result { - let from_resolved = self.resolved_target(from).await?; - let to_resolved = self.resolved_target(to).await?; - crate::changes::diff_snapshots( - &self.table_store, - &from_resolved.snapshot, - &to_resolved.snapshot, - filter, - to_resolved.branch.clone().or(from_resolved.branch.clone()), - to_resolved.graph_commit_id.clone(), - ) - .await - } - - /// Diff two graph commits. Resolves each commit to - /// `(graph_branch, graph_manifest_version)` - /// and creates branch-aware snapshots. Supports cross-branch comparison. - pub async fn diff_commits( - &self, - from_commit_id: &str, - to_commit_id: &str, - filter: &crate::changes::ChangeFilter, - ) -> Result { - let coord = self.coordinator.read().await; - let range = coord - .resolve_commit_range( - &SnapshotId::new(from_commit_id), - &SnapshotId::new(to_commit_id), - ) - .await?; - // Classify direct adjacency from the child's persisted first-parent - // pointer without changing this API's net-current result shape. The - // future feed can reuse that relationship without an ancestry index. - let (from_commit, to_commit) = match range { - ResolvedCommitRange::FirstParent(edge) => (edge.parent, edge.child), - ResolvedCommitRange::Arbitrary { from, to } => (from, to), - }; - let from_snap = coord - .resolve_target(&ReadTarget::Snapshot(SnapshotId::new( - from_commit.graph_commit_id.clone(), - ))) - .await?; - let to_snap = coord - .resolve_target(&ReadTarget::Snapshot(SnapshotId::new( - to_commit.graph_commit_id.clone(), - ))) - .await?; - drop(coord); - crate::changes::diff_snapshots( - &self.table_store, - &from_snap.snapshot, - &to_snap.snapshot, - filter, - to_snap.branch.clone().or(from_snap.branch.clone()), - to_snap.graph_commit_id.clone(), - ) - .await - } - /// One bounded, deterministic page of the exact entity changes a graph /// commit made relative to its first parent, in graph vocabulary with /// exact before/after images. The parentless genesis commit has no diff; @@ -2740,6 +2676,327 @@ impl Omnigraph { .await } + /// One bounded page of a graph diff (RFC 0030 §8): the net entity changes + /// between two read targets, or between a source branch's merge base with + /// a target and the source head, in the commit diff's entity shape and + /// order, without a cause. The first page resolves both endpoints to + /// exact graph commits; its page token keeps them, so later pages answer + /// for the same commits whatever the named branches did since. + pub async fn diff_page( + &self, + request: &crate::changes::GraphDiffRequest, + ) -> Result { + use crate::changes::token; + + let max_changes = request + .max_changes + .unwrap_or(crate::changes::COMMIT_CHANGES_DEFAULT_CHANGES); + let max_bytes = request + .max_bytes + .unwrap_or(crate::changes::COMMIT_CHANGES_DEFAULT_BYTES); + crate::changes::enumerate::validate_change_page_limits(max_changes, max_bytes)?; + + let graph_identity = self.schema_view.load().schema_identity_domain.clone(); + let filter_digest = token::filter_digest(&request.scope); + let sides_digest = token::sides_digest(&request.sides); + let (endpoints, resume) = match request.page_token.as_deref() { + None => (self.resolve_diff_endpoints(&request.sides).await?, None), + Some(encoded) => { + let decoded = token::decode_diff_page_token(encoded)?; + if decoded.graph_identity != token::hashed_identity(&graph_identity) + || decoded.sides != sides_digest + { + return Err(token::cursor_rejected( + "diff page token does not match this graph and these sides", + )); + } + if decoded.filter_digest != filter_digest { + return Err(token::cursor_rejected( + "diff page token was minted for a different filter scope", + )); + } + let endpoints = self + .reopen_diff_endpoints( + &decoded.from_commit_id, + &decoded.to_commit_id, + decoded.base_commit_id.clone(), + ) + .await?; + let resume = crate::changes::enumerate::ContinuationKey { + type_id: decoded.type_id, + position: decoded.position, + operation_rank: decoded.operation_rank, + change_index: decoded.change_index, + }; + (endpoints, Some(resume)) + } + }; + + // Reclaimed history under either endpoint is a typed gap naming the + // diff's `to` commit; a diff has no feed cursor to resume. + let to_commit_id = endpoints.to_commit_id.clone(); + let map_gap = |error| match error { + OmniError::HistoricalVersionReclaimed { .. } => OmniError::ChangeFeedGap { + cursor: None, + first_unreadable_commit_id: to_commit_id.clone(), + }, + error => error, + }; + let mut budget = crate::changes::enumerate::PageBudget::new(max_changes, max_bytes); + let mut changes = Vec::new(); + let outcome = crate::changes::enumerate::enumerate_commit_changes( + &self.table_store, + &endpoints.from_snapshot, + &endpoints.to_snapshot, + &graph_identity, + &endpoints.to_commit_id, + &request.scope, + resume.as_ref(), + &mut budget, + &mut changes, + ) + .await + .map_err(map_gap)?; + + let next_page_token = match outcome { + crate::changes::enumerate::CommitEnumeration::Complete => None, + crate::changes::enumerate::CommitEnumeration::Truncated(key) => { + Some(token::encode_token(&token::DiffPageTokenV1 { + version: token::TOKEN_VERSION, + kind: token::KIND_DIFF_PAGE.to_string(), + graph_identity: token::hashed_identity(&graph_identity), + sides: sides_digest, + from_commit_id: endpoints.from_commit_id.clone(), + to_commit_id: endpoints.to_commit_id.clone(), + base_commit_id: endpoints.base_commit_id.clone(), + filter_digest, + type_id: key.type_id, + position: key.position, + operation_rank: key.operation_rank, + change_index: key.change_index, + })?) + } + crate::changes::enumerate::CommitEnumeration::Exhausted { required_bytes } => { + return Err(OmniError::resource_limit( + "commit_changes_page_bytes", + max_bytes, + required_bytes, + )); + } + }; + + Ok(crate::changes::GraphDiffPage { + from_commit_id: endpoints.from_commit_id, + to_commit_id: endpoints.to_commit_id, + base_commit_id: endpoints.base_commit_id, + changes, + next_page_token, + }) + } + + /// The merge base of `source` and `target` (RFC 0030 §8.4): the commit a + /// merge of `source` into `target` would classify against, by the rule + /// branch merge uses, with the two heads it was computed from. Reads + /// lineage only. + pub async fn merge_base( + &self, + source: &str, + target: &str, + ) -> Result { + let (source_head, target_head, merge_base, _, _) = + self.capture_merge_base(source, target).await?; + Ok(crate::changes::MergeBase { + source_commit_id: source_head, + target_commit_id: target_head, + merge_base, + }) + } + + /// Capture both branches from one view each and resolve their merge base, + /// returning the heads, the base commit and the source head's snapshot. + async fn capture_merge_base( + &self, + source: &str, + target: &str, + ) -> Result<( + String, + String, + crate::db::commit_graph::GraphCommit, + Snapshot, + Snapshot, + )> { + let source_branch = normalize_branch_name(source)?; + let target_branch = normalize_branch_name(target)?; + let (source_txn, target_txn, source_commits, target_commits) = self + .open_merge_write_txns(source_branch.as_deref(), target_branch.as_deref()) + .await?; + let source_head = source_txn + .effective_graph_head + .clone() + .ok_or_else(|| OmniError::manifest("source branch has no head commit".to_string()))?; + let target_head = target_txn + .effective_graph_head + .clone() + .ok_or_else(|| OmniError::manifest("target branch has no head commit".to_string()))?; + let merge_base = Self::resolve_merge_base( + &source_commits.graph, + &target_commits.graph, + &source_head, + &target_head, + ) + .await?; + Ok(( + source_head, + target_head, + merge_base, + source_txn.base, + target_txn.base, + )) + } + + /// Resolve a graph diff's sides to exact commits and their snapshots. + async fn resolve_diff_endpoints( + &self, + sides: &crate::changes::GraphDiffSides, + ) -> Result { + match sides { + crate::changes::GraphDiffSides::Between { from, to } => { + let (from_commit_id, from_snapshot) = self.resolve_diff_read_target(from).await?; + let (to_commit_id, to_snapshot) = self.resolve_diff_read_target(to).await?; + Ok(DiffEndpoints { + from_commit_id, + to_commit_id, + base_commit_id: None, + from_snapshot, + to_snapshot, + }) + } + crate::changes::GraphDiffSides::Branch { source, target } => { + let (source_head, target_head, merge_base, source_snapshot, target_snapshot) = + self.capture_merge_base(source, target).await?; + let base_commit_id = merge_base.graph_commit_id.clone(); + // The base is usually an ancestor neither head is at; when it + // is one of the captured heads, its captured snapshot is the + // base snapshot. + let base_snapshot = if base_commit_id == source_head { + source_snapshot.clone() + } else if base_commit_id == target_head { + target_snapshot + } else { + self.commit_snapshot(&base_commit_id).await? + }; + Ok(DiffEndpoints { + from_commit_id: base_commit_id.clone(), + to_commit_id: source_head, + base_commit_id: Some(base_commit_id), + from_snapshot: base_snapshot, + to_snapshot: source_snapshot, + }) + } + } + } + + /// Reopen the exact commits a diff page token names. + async fn reopen_diff_endpoints( + &self, + from_commit_id: &str, + to_commit_id: &str, + base_commit_id: Option, + ) -> Result { + Ok(DiffEndpoints { + from_commit_id: from_commit_id.to_string(), + to_commit_id: to_commit_id.to_string(), + base_commit_id, + from_snapshot: self.commit_snapshot(from_commit_id).await?, + to_snapshot: self.commit_snapshot(to_commit_id).await?, + }) + } + + /// The graph as of one commit, published or not. + async fn commit_snapshot(&self, commit_id: &str) -> Result { + Ok(self + .coordinator + .read() + .await + .resolve_target(&ReadTarget::Snapshot(SnapshotId::new(commit_id))) + .await? + .snapshot) + } + + /// The branch a transport authorizes `read` on for one graph diff side: + /// the branch itself, or the authored branch of the commit the side names + /// (an unpublished proposal's or unattached snapshot's recorded target). + /// Refuses exactly as the diff would: ambiguity and absence are typed. + pub async fn diff_read_target_branch(&self, target: &str) -> Result { + Ok(match self.classify_diff_read_target(target).await? { + DiffReadTarget::Branch(resolved) => { + resolved.branch.unwrap_or_else(|| "main".to_string()) + } + DiffReadTarget::Commit(commit) => { + commit.graph_branch.unwrap_or_else(|| "main".to_string()) + } + }) + } + + /// Resolve one diff endpoint to its commit and the graph as of it. + async fn resolve_diff_read_target(&self, target: &str) -> Result<(String, Snapshot)> { + match self.classify_diff_read_target(target).await? { + DiffReadTarget::Branch(resolved) => { + let commit_id = resolved.graph_commit_id.ok_or_else(|| { + OmniError::manifest(format!("branch '{}' has no head commit", target.trim())) + })?; + Ok((commit_id, resolved.snapshot)) + } + DiffReadTarget::Commit(commit) => { + let snapshot = self.commit_snapshot(&commit.graph_commit_id).await?; + Ok((commit.graph_commit_id, snapshot)) + } + } + } + + /// Classify one diff endpoint: a live branch (its head now) or a graph + /// commit. A string naming both is refused, never resolved by precedence; + /// one naming neither is a typed not-found carrying only the caller's + /// spelling. + async fn classify_diff_read_target(&self, target: &str) -> Result { + let target = target.trim(); + if target.is_empty() { + return Err(OmniError::manifest("a diff endpoint cannot be empty")); + } + // A string the branch-name rules refuse cannot name a branch; that is + // not an error here, since it may still name a commit. + let as_branch = match normalize_branch_name(target) { + Err(_) => None, + Ok(_) => match self.resolved_target(ReadTarget::branch(target)).await { + Ok(resolved) => Some(resolved), + Err(OmniError::BranchNotFound { .. }) => None, + Err(error) => return Err(error), + }, + }; + let as_commit = match self + .coordinator + .read() + .await + .resolve_commit(&SnapshotId::new(target)) + .await + { + Ok(commit) => Some(commit), + Err(OmniError::Manifest(error)) if error.kind == ManifestErrorKind::NotFound => None, + Err(OmniError::BranchNotFound { .. }) => None, + Err(error) => return Err(error), + }; + match (as_branch, as_commit) { + (Some(_), Some(_)) => Err(OmniError::AmbiguousReadTarget { + target: target.to_string(), + }), + (Some(resolved), None) => Ok(DiffReadTarget::Branch(Box::new(resolved))), + (None, Some(commit)) => Ok(DiffReadTarget::Commit(commit)), + (None, None) => Err(OmniError::ReadTargetNotFound { + target: target.to_string(), + }), + } + } + pub async fn entity_at_target( &self, target: impl Into, @@ -4167,6 +4424,22 @@ pub(crate) fn schema_for_table_key(catalog: &Catalog, table_key: &str) -> Result ))) } +/// What one graph diff endpoint names. +enum DiffReadTarget { + Branch(Box), + Commit(GraphCommit), +} + +/// A graph diff's two endpoints, resolved to exact commits and their +/// snapshots. +struct DiffEndpoints { + from_commit_id: String, + to_commit_id: String, + base_commit_id: Option, + from_snapshot: Snapshot, + to_snapshot: Snapshot, +} + #[cfg(test)] mod tests { use arrow_array::{Int32Array, new_null_array}; diff --git a/crates/omnigraph/src/exec/merge.rs b/crates/omnigraph/src/exec/merge.rs index 004145af4..5ab5fffa7 100644 --- a/crates/omnigraph/src/exec/merge.rs +++ b/crates/omnigraph/src/exec/merge.rs @@ -4977,6 +4977,11 @@ impl Session { // Keep the planning/publication future out of the public API's callers; // deeply composed loads and merges otherwise retain large debug // construction frames throughout execution. Poll it in the same task. + let mode = if as_commit { + MergeMode::Propose + } else { + MergeMode::Publish + }; let result = Box::pin(self.branch_merge_impl( source, target, @@ -4984,7 +4989,7 @@ impl Session { self.settings().merge_lineage(), HistoryReleaseBytes(self.settings().history_release_bytes()), WriteBudget::from_settings(self.settings()), - as_commit, + mode, )) .await; if let Err(error) = fail(&BRANCH_MERGE_PRE_RETURN) { @@ -5000,12 +5005,43 @@ impl Session { } result } + + /// Run `branch merge` through base resolution, three-way classification + /// and constraint validation, then stop (RFC 0030 §8.4): the outcome, or + /// the conflict list, that merging `source` into `target` would produce + /// against the branches as captured now. It is a read: it takes no branch + /// gate and no merge-input tag, stages only to process-local scratch, and + /// commits and publishes nothing, so `commit` is always `None`. Transports + /// authorize it as a read of both branches. + pub async fn branch_merge_dry_run(&self, source: &str, target: &str) -> Result { + Box::pin(self.branch_merge_impl( + source, + target, + None, + self.settings().merge_lineage(), + HistoryReleaseBytes(self.settings().history_release_bytes()), + WriteBudget::from_settings(self.settings()), + MergeMode::DryRun, + )) + .await + } +} + +/// What a branch merge does with its validated result. +#[derive(Debug, Clone, Copy, PartialEq, Eq)] +enum MergeMode { + /// Publish onto the target: the ordinary merge. + Publish, + /// Record an unpublished commit (`branch merge … as commit`). + Propose, + /// Stop after validation and write nothing. + DryRun, } impl Omnigraph { /// Prove a base from captured records, then read only addressed history /// blocks needed by the exact ancestry walk. - async fn resolve_merge_base( + pub(crate) async fn resolve_merge_base( source: &crate::db::commit_graph::CommitGraph, target: &crate::db::commit_graph::CommitGraph, source_commit_id: &str, @@ -5033,8 +5069,10 @@ impl Omnigraph { lineage: MergeLineage, history_release_bytes: HistoryReleaseBytes, write_budget: WriteBudget, - as_commit: bool, + mode: MergeMode, ) -> Result { + let as_commit = mode == MergeMode::Propose; + let dry_run = mode == MergeMode::DryRun; let outer_prepare_timing = crate::instrumentation::start_merge_timing( crate::instrumentation::MergeTimingPhase::OuterPrepare, ); @@ -5046,7 +5084,13 @@ impl Omnigraph { )); } - let _schema_permit = self.write_queue().acquire_schema_shared().await; + // A dry run is a read: it takes neither the schema permit nor the + // branch gates, so it never waits behind (or blocks) a writer. + let _schema_permit = if dry_run { + None + } else { + Some(self.write_queue().acquire_schema_shared().await) + }; // The target gate is held through publication: it keeps a target // delete/recreate from reusing the name underneath the plan (ABA) and // serializes merges into one target. The source gate is held only @@ -5056,17 +5100,19 @@ impl Omnigraph { // Both are taken in the sorted order `acquire_branches` uses, so the // pair cannot deadlock with another multi-branch acquirer. let queue = self.write_queue(); - let (source_gate, _target_gate) = if source_branch <= target_branch { + let (source_gate, _target_gate) = if dry_run { + (None, None) + } else if source_branch <= target_branch { let source_gate = queue.acquire_branch(source_branch.as_deref()).await; ( - source_gate, - queue.acquire_branch(target_branch.as_deref()).await, + Some(source_gate), + Some(queue.acquire_branch(target_branch.as_deref()).await), ) } else { let target_gate = queue.acquire_branch(target_branch.as_deref()).await; ( - queue.acquire_branch(source_branch.as_deref()).await, - target_gate, + Some(queue.acquire_branch(source_branch.as_deref()).await), + Some(target_gate), ) }; // Capture each branch as one coherent RFC-022 authority token plus @@ -5119,14 +5165,21 @@ impl Omnigraph { &witness, ); let preparation = async { - input_guard.pin(source_txn.manifest_probe.dataset()).await?; - input_guard.pin(target_txn.manifest_probe.dataset()).await?; + // A dry run holds its inputs only as this read's snapshots: a + // cleanup that reclaims one mid-run surfaces as a typed gap, + // never as a tag the run left behind. + if !dry_run { + input_guard.pin(source_txn.manifest_probe.dataset()).await?; + input_guard.pin(target_txn.manifest_probe.dataset()).await?; + } let base_snapshot = if head_is_base { target_txn.base.clone() } else { let base = ManifestCoordinator::pinned_graph_commit(self.uri(), &base_commit).await?; - input_guard.pin(&base.dataset).await?; + if !dry_run { + input_guard.pin(&base.dataset).await?; + } Snapshot::wrap(base.snapshot) }; if !source_txn.manifest_probe.is_current().await? @@ -5194,10 +5247,12 @@ impl Omnigraph { history_release_bytes, write_budget, publication, + dry_run, )) .await; let proposal_holds_inputs = as_commit && merge_result.is_ok(); - if !proposal_holds_inputs + if !dry_run + && !proposal_holds_inputs && !merge_result .as_ref() .is_err_and(|error| error.is_manifest_publish_in_doubt()) @@ -5244,6 +5299,7 @@ impl Omnigraph { history_release_bytes: HistoryReleaseBytes, write_budget: WriteBudget, publication: crate::db::omnigraph::Publication, + dry_run: bool, ) -> Result { let source_snapshot = &source_txn.base; let target_snapshot = &target_txn.base; @@ -5572,6 +5628,19 @@ impl Omnigraph { validation_timing.finish(); } fail(&BRANCH_MERGE_POST_CANDIDATE_VALIDATION)?; + if dry_run { + // Everything a merge decides is decided: the classification found + // no conflict and the merged candidates validate. Stop before the + // first effect. + return Ok(MergeResult { + outcome: if is_fast_forward { + MergeOutcome::FastForward + } else { + MergeOutcome::Merged + }, + commit: None, + }); + } // Every `RewriteMerged` / `AdoptWithDelta` / `AdoptPureInserts` logical // data step commits one detached link of its table's chain (RFC 0067). diff --git a/crates/omnigraph/tests/failpoints.rs b/crates/omnigraph/tests/failpoints.rs index f997ac450..a5fe45157 100644 --- a/crates/omnigraph/tests/failpoints.rs +++ b/crates/omnigraph/tests/failpoints.rs @@ -8377,14 +8377,14 @@ async fn rfc_0067_detached_pin_diff_keeps_the_acknowledged_insert() { .contains(&"detached-diff".to_string()) ); let after = helpers::snapshot_id(&db, "main").await.unwrap(); - let changes = db - .diff_between( - ReadTarget::Snapshot(before), - ReadTarget::Snapshot(after), - &omnigraph::changes::ChangeFilter::default(), - ) - .await - .unwrap(); + let changes = helpers::graph_diff( + &db, + before.as_str(), + after.as_str(), + &omnigraph::changes::ChangeFeedScope::default(), + ) + .await + .unwrap(); assert_eq!( changes.stats.inserts, 1, "acknowledged insert absent from public diff: {changes:?}" @@ -8415,14 +8415,14 @@ async fn rfc_0067_foreign_linear_commit_diff_keeps_the_acknowledged_insert() { collect_column_strings(&read_table(&db, "node:Person").await, "name") .contains(&"blocked-diff-second".to_string()) ); - let changes = db - .diff_between( - ReadTarget::Snapshot(before), - ReadTarget::Snapshot(after), - &omnigraph::changes::ChangeFilter::default(), - ) - .await - .unwrap(); + let changes = helpers::graph_diff( + &db, + before.as_str(), + after.as_str(), + &omnigraph::changes::ChangeFeedScope::default(), + ) + .await + .unwrap(); assert_eq!( changes.stats.inserts, 1, "acknowledged chain insert absent from public diff: {changes:?}" @@ -9260,16 +9260,16 @@ async fn rfc_0067_merge_diff_keeps_every_chunk() { } let after = helpers::snapshot_id(&db, "main").await.unwrap(); let after_rows = count_rows(&db, "node:Person").await; - let changes = db - .diff_between( - ReadTarget::Snapshot(before), - ReadTarget::Snapshot(after), - &omnigraph::changes::ChangeFilter::default(), - ) - .await - .unwrap(); + let changes = helpers::graph_diff( + &db, + before.as_str(), + after.as_str(), + &omnigraph::changes::ChangeFeedScope::default(), + ) + .await + .unwrap(); assert_eq!( - changes.stats.inserts as usize, + changes.stats.inserts, after_rows - before_rows, "diff omitted acknowledged merge chunks: {} inserts for {} added rows", changes.stats.inserts, diff --git a/crates/omnigraph/tests/forbidden_apis.rs b/crates/omnigraph/tests/forbidden_apis.rs index 1e42e9157..870a8a724 100644 --- a/crates/omnigraph/tests/forbidden_apis.rs +++ b/crates/omnigraph/tests/forbidden_apis.rs @@ -310,8 +310,10 @@ const READ_ONLY_SURFACES: &[(&str, &str)] = &[ ("db/omnigraph.rs", "resolved_branch_of"), ("db/omnigraph.rs", "sync_branch"), ("db/omnigraph.rs", "resolve_snapshot"), - ("db/omnigraph.rs", "diff_between"), - ("db/omnigraph.rs", "diff_commits"), + ("db/omnigraph.rs", "diff_page"), + ("db/omnigraph.rs", "merge_base"), + ("db/omnigraph.rs", "diff_read_target_branch"), + ("exec/merge.rs", "branch_merge_dry_run"), ("db/omnigraph.rs", "commit_changes_page"), ("db/omnigraph.rs", "poll_change_feed"), ("db/omnigraph.rs", "capture_change_baseline"), @@ -459,11 +461,6 @@ const LOW_LEVEL_READ_ONLY_SURFACES: &[(&str, &str, &str)] = &[ "GraphCoordinator", "resolve_commit", ), - ( - "db/graph_coordinator.rs", - "GraphCoordinator", - "resolve_commit_range", - ), ( "db/graph_coordinator.rs", "GraphCoordinator", @@ -1100,11 +1097,6 @@ durable_calls! { // candidate scan + parent before-image probe and the full-merge fallback. // Read-only — it stages and publishes nothing. ("changes/candidate_scan.rs", ".dataset()", 5, WriteProtocol::ReadOnlyAccess), - // Net-diff cross-branch path: the same typed row comparison over two - // pinned snapshot handles, plus the same-lineage path reading each - // side's schema to resolve its system column spellings (RFC 0040 - // Historical reads). Read-only — the diff stages and publishes nothing. - ("changes/mod.rs", ".dataset()", 11, WriteProtocol::ReadOnlyAccess), ("db/omnigraph/collector.rs", ".dataset()", 17, WriteProtocol::ReadOnlyAccess), ("db/omnigraph/collector.rs", ".delete(", 1, WriteProtocol::Composed("detached-only sweep through the table's own Lance store: freed files, then the published manifests no retained `__manifest` version pins (links before tips) and the dead stagings; an interrupted pass is re-swept by the next")), ("db/omnigraph/schema_apply.rs", ".dataset()", 1, SCHEMA_V9), diff --git a/crates/omnigraph/tests/helpers/cost.rs b/crates/omnigraph/tests/helpers/cost.rs index abb90c08c..d95629206 100644 --- a/crates/omnigraph/tests/helpers/cost.rs +++ b/crates/omnigraph/tests/helpers/cost.rs @@ -244,6 +244,9 @@ pub struct IoCounts { /// `__manifest` registry scans (publish state; includes the graph-lineage rows /// folded into `__manifest` by RFC-013 Phase 7). pub manifest_reads: u64, + /// `__manifest` object-store writes: published versions, transaction files + /// and tags (merge-input holds and proposals among them). + pub manifest_writes: u64, /// Bytes returned by those real object-store reads. This distinguishes a /// physical-address take from scanning a compacted history-sized fragment /// even when both use a similar number of requests. @@ -639,6 +642,7 @@ impl OpProbes { data_opener_reads: t.opener_reads, data_scan_reads: t.scan_reads, manifest_reads: manifest.read_iops, + manifest_writes: manifest.write_iops, manifest_read_bytes: manifest.read_bytes, version_probes: self.probe_count.load(Ordering::Relaxed), data_open_count: self.data_open_count.load(Ordering::Relaxed), diff --git a/crates/omnigraph/tests/helpers/mod.rs b/crates/omnigraph/tests/helpers/mod.rs index b919fbdce..7c70fc129 100644 --- a/crates/omnigraph/tests/helpers/mod.rs +++ b/crates/omnigraph/tests/helpers/mod.rs @@ -15,7 +15,7 @@ use arrow_array::{Array, RecordBatch, StringArray}; use futures::TryStreamExt; pub use omnigraph::Session; -use omnigraph::changes::{ChangeFilter, ChangeSet}; +use omnigraph::changes::{ChangeEntityKind, ChangeFeedScope, ChangeOpKind}; use omnigraph::db::{Omnigraph, ReadTarget, Snapshot, SnapshotId}; use omnigraph::error::Result; use omnigraph::loader::LoadMode; @@ -568,18 +568,116 @@ pub async fn snapshot_id(db: &Omnigraph, branch: &str) -> Result { db.resolve_snapshot(branch).await } +/// One change of a graph diff flattened for assertions: the entity's kind, +/// type, id and operation, and an edge's endpoints from its newest image. +#[derive(Debug, Clone, PartialEq, Eq)] +pub struct NetChange { + pub kind: ChangeEntityKind, + pub type_name: String, + pub id: String, + pub op: ChangeOpKind, + pub endpoints: Option, +} + +#[derive(Debug, Clone, PartialEq, Eq)] +pub struct NetEndpoints { + pub src: String, + pub dst: String, +} + +impl NetChange { + /// `node:` or `edge:`. + pub fn type_key(&self) -> String { + let prefix = match self.kind { + ChangeEntityKind::Node => "node", + ChangeEntityKind::Edge => "edge", + }; + format!("{prefix}:{}", self.type_name) + } +} + +#[derive(Debug, Clone, Copy, Default, PartialEq, Eq)] +pub struct NetStats { + pub inserts: usize, + pub updates: usize, + pub deletes: usize, +} + +/// A whole graph diff: its resolved endpoints and every change of every page. +#[derive(Debug, Clone)] +pub struct NetDiff { + pub from_commit_id: String, + pub to_commit_id: String, + pub changes: Vec, + pub stats: NetStats, +} + +/// The graph diff `from` → `to` (branch names or commit ids) within `scope`, +/// every page. +pub async fn graph_diff( + db: &Omnigraph, + from: &str, + to: &str, + scope: &ChangeFeedScope, +) -> Result { + let mut request = omnigraph::changes::GraphDiffRequest { + sides: omnigraph::changes::GraphDiffSides::Between { + from: from.to_string(), + to: to.to_string(), + }, + scope: scope.clone(), + page_token: None, + max_changes: None, + max_bytes: None, + }; + let mut diff = NetDiff { + from_commit_id: String::new(), + to_commit_id: String::new(), + changes: Vec::new(), + stats: NetStats::default(), + }; + loop { + let page = db.diff_page(&request).await?; + diff.from_commit_id = page.from_commit_id; + diff.to_commit_id = page.to_commit_id; + for change in page.changes { + match change.op { + ChangeOpKind::Insert => diff.stats.inserts += 1, + ChangeOpKind::Update => diff.stats.updates += 1, + ChangeOpKind::Delete => diff.stats.deletes += 1, + } + let endpoints = change + .after + .as_ref() + .or(change.before.as_ref()) + .and_then(|image| image.endpoints.as_ref()) + .map(|endpoints| NetEndpoints { + src: endpoints.from.clone(), + dst: endpoints.to.clone(), + }); + diff.changes.push(NetChange { + kind: change.kind, + type_name: change.entity_type.name, + id: change.id, + op: change.op, + endpoints, + }); + } + match page.next_page_token { + Some(token) => request.page_token = Some(token), + None => return Ok(diff), + } + } +} + +/// The graph diff from `from_snapshot` to the head of `branch`. pub async fn diff_since_branch( db: &Omnigraph, branch: &str, from_snapshot: SnapshotId, - filter: &ChangeFilter, -) -> Result { - db.diff_between( - ReadTarget::Snapshot(from_snapshot), - ReadTarget::branch(branch), - filter, - ) - .await + scope: &ChangeFeedScope, +) -> Result { + graph_diff(db, from_snapshot.as_str(), branch, scope).await } /// Advance a Lance dataset HEAD directly from tests without going through diff --git a/crates/omnigraph/tests/integration/changes.rs b/crates/omnigraph/tests/integration/changes.rs index df340932e..90645cec6 100644 --- a/crates/omnigraph/tests/integration/changes.rs +++ b/crates/omnigraph/tests/integration/changes.rs @@ -4,7 +4,7 @@ use crate::helpers; -use omnigraph::changes::{ChangeFilter, ChangeOp, EntityKind}; +use omnigraph::changes::{ChangeEntityKind, ChangeFeedScope, ChangeOpKind}; use omnigraph::db::commit_graph::CommitGraph; use omnigraph::db::{MergeOutcome, Omnigraph, ReadTarget}; use omnigraph::instrumentation::{MergeWriteProbes, with_merge_write_probes}; @@ -42,7 +42,7 @@ async fn head_commit_id(uri: &str, branch: Option<&str>) -> String { commit_graph.head_commit_id().await.unwrap().unwrap() } -fn change_tuples(change_set: &omnigraph::changes::ChangeSet) -> Vec<(String, String, ChangeOp)> { +fn change_tuples(change_set: &NetDiff) -> Vec<(String, String, ChangeOpKind)> { let mut tuples: Vec<_> = change_set .changes .iter() @@ -51,14 +51,14 @@ fn change_tuples(change_set: &omnigraph::changes::ChangeSet) -> Vec<(String, Str tuples.sort_by(|a, b| { a.0.cmp(&b.0).then_with(|| a.1.cmp(&b.1)).then_with(|| { let a_op = match a.2 { - ChangeOp::Insert => 0, - ChangeOp::Update => 1, - ChangeOp::Delete => 2, + ChangeOpKind::Insert => 0, + ChangeOpKind::Update => 1, + ChangeOpKind::Delete => 2, }; let b_op = match b.2 { - ChangeOp::Insert => 0, - ChangeOp::Update => 1, - ChangeOp::Delete => 2, + ChangeOpKind::Insert => 0, + ChangeOpKind::Update => 1, + ChangeOpKind::Delete => 2, }; a_op.cmp(&b_op) }) @@ -89,21 +89,18 @@ async fn write_receipts_identify_exact_commits_and_mutation_noops() { assert_eq!(stored.graph_commit_id, commit.graph_commit_id); assert_eq!(stored.graph_manifest_version, commit.graph_manifest_version); - let changes = db - .diff_commits( - commit.parent_commit_id.as_deref().unwrap(), - &commit.graph_commit_id, - &ChangeFilter::default(), - ) - .await - .unwrap(); - assert_eq!( - changes.to_graph_manifest_version, - commit.graph_manifest_version - ); + let changes = graph_diff( + &db, + commit.parent_commit_id.as_deref().unwrap(), + &commit.graph_commit_id, + &ChangeFeedScope::default(), + ) + .await + .unwrap(); + assert_eq!(changes.to_commit_id, commit.graph_commit_id); assert_eq!( change_tuples(&changes), - vec![("node:Person".into(), "Eve".into(), ChangeOp::Insert)] + vec![("node:Person".into(), "Eve".into(), ChangeOpKind::Insert)] ); let head_before_no_op = snapshot_id(&db, "main").await.unwrap(); @@ -172,12 +169,7 @@ async fn diff_empty_when_nothing_changed() { let dir = tempfile::tempdir().unwrap(); let db = init_and_load(&dir).await; let v = snapshot_id(&db, "main").await.unwrap(); - let cs = db - .diff_between( - ReadTarget::Snapshot(v.clone()), - ReadTarget::Snapshot(v), - &ChangeFilter::default(), - ) + let cs = graph_diff(&db, v.as_str(), v.as_str(), &ChangeFeedScope::default()) .await .unwrap(); assert!(cs.changes.is_empty()); @@ -187,7 +179,7 @@ async fn diff_empty_when_nothing_changed() { } #[tokio::test] -async fn diff_pairs_type_renames_by_identity_and_separates_reincarnations() { +async fn diff_pairs_type_renames_by_identity_and_refuses_reincarnations() { let dir = tempfile::tempdir().unwrap(); let uri = dir.path().to_str().unwrap(); let db = helpers::session( @@ -220,14 +212,14 @@ node Anchor { name: String @key } .unwrap(); let after_rename = snapshot_id(&db, "main").await.unwrap(); - let rename_diff = db - .diff_between( - ReadTarget::Snapshot(before_rename), - ReadTarget::Snapshot(after_rename.clone()), - &ChangeFilter::default(), - ) - .await - .unwrap(); + let rename_diff = graph_diff( + &db, + before_rename.as_str(), + after_rename.as_str(), + &ChangeFeedScope::default(), + ) + .await + .unwrap(); assert!( rename_diff.changes.is_empty(), "a pure alias change on one table identity must not become delete+insert: {:?}", @@ -255,33 +247,18 @@ node Anchor { name: String @key } .await .unwrap(); - let reincarnation_diff = diff_since_branch(&db, "main", after_rename, &ChangeFilter::default()) + // Drop/re-add is schema evolution, not entity data (RFC 0030 §4.1): the + // old lifetime leaves with Alice still in it, so the pair is a schema + // boundary, refused rather than reported as a delete and an insert. + match diff_since_branch(&db, "main", after_rename, &ChangeFeedScope::default()) .await - .unwrap(); - let reincarnation_changes = reincarnation_diff - .changes - .iter() - .map(|change| (change.type_key().clone(), change.id.clone(), change.op)) - .collect::>(); - assert_eq!( - reincarnation_changes, - vec![ - ( - "node:Human".to_string(), - "Alice".to_string(), - ChangeOp::Delete, - ), - ( - "node:Human".to_string(), - "Bob".to_string(), - ChangeOp::Insert, - ), - ], - "drop/re-add under one alias must visit the old identity before the newly allocated identity" - ); - assert_eq!(reincarnation_diff.stats.deletes, 1); - assert_eq!(reincarnation_diff.stats.inserts, 1); - assert_eq!(reincarnation_diff.stats.updates, 0); + .unwrap_err() + { + omnigraph::error::OmniError::ChangeSchemaBoundary { type_name, .. } => { + assert_eq!(type_name, "Human") + } + other => panic!("a drop/re-add pair is a schema boundary, got: {other:?}"), + } } #[tokio::test] @@ -299,13 +276,13 @@ async fn diff_detects_node_insert() { .await .unwrap(); - let cs = diff_since_branch(&db, "main", v_before, &ChangeFilter::default()) + let cs = diff_since_branch(&db, "main", v_before, &ChangeFeedScope::default()) .await .unwrap(); let inserts: Vec<_> = cs .changes .iter() - .filter(|c| c.op == ChangeOp::Insert && c.type_key() == "node:Person") + .filter(|c| c.op == ChangeOpKind::Insert && c.type_key() == "node:Person") .collect(); assert!( !inserts.is_empty(), @@ -320,7 +297,7 @@ async fn diff_detects_node_insert() { "Insert should contain Eve. Got: {:?}", inserts.iter().map(|c| &c.id).collect::>() ); - assert_eq!(inserts[0].kind, EntityKind::Node); + assert_eq!(inserts[0].kind, ChangeEntityKind::Node); assert_eq!(inserts[0].endpoints, None); } @@ -339,13 +316,13 @@ async fn diff_detects_node_update() { .await .unwrap(); - let cs = diff_since_branch(&db, "main", v_before, &ChangeFilter::default()) + let cs = diff_since_branch(&db, "main", v_before, &ChangeFeedScope::default()) .await .unwrap(); let updates: Vec<_> = cs .changes .iter() - .filter(|c| c.op == ChangeOp::Update && c.type_key() == "node:Person") + .filter(|c| c.op == ChangeOpKind::Update && c.type_key() == "node:Person") .collect(); assert!( !updates.is_empty(), @@ -372,24 +349,40 @@ async fn diff_detects_node_delete_with_cascade() { .await .unwrap(); - let cs = diff_since_branch(&db, "main", v_before, &ChangeFilter::default()) + let cs = diff_since_branch(&db, "main", v_before, &ChangeFeedScope::default()) .await .unwrap(); + // Multi-type changes follow the frozen block order (RFC 0030 §4.4): nodes + // before edges, and each type's changes contiguous, in an order its + // opaque identity fixes rather than its name. let table_keys = cs .changes .iter() .map(|change| change.type_key()) .collect::>(); assert!( - table_keys.windows(2).all(|pair| pair[0] <= pair[1]), - "multi-table changes must follow graph-visible table-key order: {table_keys:?}" + cs.changes + .windows(2) + .all(|pair| !(pair[0].kind == ChangeEntityKind::Edge + && pair[1].kind == ChangeEntityKind::Node)), + "nodes precede edges: {table_keys:?}" ); + let mut seen_types = Vec::new(); + for key in &table_keys { + if seen_types.last() != Some(key) { + assert!( + !seen_types.contains(key), + "one type's changes are contiguous: {table_keys:?}" + ); + seen_types.push(key.clone()); + } + } // Should have node:Person delete let person_deletes: Vec<_> = cs .changes .iter() - .filter(|c| c.op == ChangeOp::Delete && c.type_key() == "node:Person") + .filter(|c| c.op == ChangeOpKind::Delete && c.type_key() == "node:Person") .collect(); assert!( !person_deletes.is_empty(), @@ -404,7 +397,7 @@ async fn diff_detects_node_delete_with_cascade() { let edge_deletes: Vec<_> = cs .changes .iter() - .filter(|c| c.op == ChangeOp::Delete && c.type_key() == "edge:Knows") + .filter(|c| c.op == ChangeOpKind::Delete && c.type_key() == "edge:Knows") .collect(); assert!( !edge_deletes.is_empty(), @@ -439,14 +432,14 @@ async fn diff_detects_edge_insert_with_endpoints() { .await .unwrap(); - let cs = diff_since_branch(&db, "main", v_before, &ChangeFilter::default()) + let cs = diff_since_branch(&db, "main", v_before, &ChangeFeedScope::default()) .await .unwrap(); let edge_inserts: Vec<_> = cs .changes .iter() - .filter(|c| c.op == ChangeOp::Insert && c.type_key() == "edge:Knows") + .filter(|c| c.op == ChangeOpKind::Insert && c.type_key() == "edge:Knows") .collect(); assert!( !edge_inserts.is_empty(), @@ -458,7 +451,7 @@ async fn diff_detects_edge_insert_with_endpoints() { ); let e = &edge_inserts[0]; - assert_eq!(e.kind, EntityKind::Edge); + assert_eq!(e.kind, ChangeEntityKind::Edge); let ep = e .endpoints .as_ref() @@ -486,7 +479,7 @@ async fn filter_by_type_name_skips_non_matching() { .unwrap(); // Filter to Company only — should not see Person changes - let filter = ChangeFilter { + let filter = ChangeFeedScope { type_names: Some(vec!["Company".to_string()]), ..Default::default() }; @@ -529,8 +522,8 @@ async fn filter_by_op_skips_unwanted_operations() { .unwrap(); // Filter to Insert only - let filter = ChangeFilter { - ops: Some(vec![ChangeOp::Insert]), + let filter = ChangeFeedScope { + ops: Some(vec![ChangeOpKind::Insert]), ..Default::default() }; let cs = diff_since_branch(&db, "main", v_before, &filter) @@ -541,7 +534,7 @@ async fn filter_by_op_skips_unwanted_operations() { for c in &cs.changes { assert_eq!( c.op, - ChangeOp::Insert, + ChangeOpKind::Insert, "Filter for Insert-only should not include {:?} for {} ({})", c.op, c.type_key(), @@ -588,7 +581,7 @@ async fn diff_after_merge_reports_actual_changes() { assert_eq!(outcome.outcome, MergeOutcome::Merged); // Diff from pre-branch to post-merge on main - let cs = diff_since_branch(&main, "main", v_before_branch, &ChangeFilter::default()) + let cs = diff_since_branch(&main, "main", v_before_branch, &ChangeFeedScope::default()) .await .unwrap(); @@ -604,11 +597,11 @@ async fn diff_after_merge_reports_actual_changes() { let person_inserts: Vec<_> = person_changes .iter() - .filter(|c| c.op == ChangeOp::Insert) + .filter(|c| c.op == ChangeOpKind::Insert) .collect(); let person_updates: Vec<_> = person_changes .iter() - .filter(|c| c.op == ChangeOp::Update) + .filter(|c| c.op == ChangeOpKind::Update) .collect(); // There should be exactly 1 insert (Eve) not all persons @@ -628,7 +621,7 @@ async fn diff_after_merge_reports_actual_changes() { } #[tokio::test] -async fn diff_commits_resolves_feature_commit_from_main_handle() { +async fn graph_diff_resolves_feature_commit_from_main_handle() { let dir = tempfile::tempdir().unwrap(); let uri = dir.path().to_str().unwrap(); let main = init_and_load(&dir).await; @@ -662,14 +655,18 @@ async fn diff_commits_resolves_feature_commit_from_main_handle() { .unwrap() .graph_commit_id; - let cs = main - .diff_commits(&main_head, &feature_head, &ChangeFilter::default()) - .await - .unwrap(); + let cs = graph_diff( + &main, + &main_head, + &feature_head, + &ChangeFeedScope::default(), + ) + .await + .unwrap(); assert!( cs.changes .iter() - .any(|change| change.op == ChangeOp::Insert && change.id == "Eve"), + .any(|change| change.op == ChangeOpKind::Insert && change.id == "Eve"), "expected feature-only insert to be diffable from a main handle" ); } @@ -709,24 +706,23 @@ async fn cross_branch_diff_honors_insert_only_filter() { .unwrap() .graph_commit_id; - let filter = ChangeFilter { - ops: Some(vec![ChangeOp::Insert]), + let filter = ChangeFeedScope { + ops: Some(vec![ChangeOpKind::Insert]), ..Default::default() }; - let cs = main - .diff_commits(&main_head, &feature_head, &filter) + let cs = graph_diff(&main, &main_head, &feature_head, &filter) .await .unwrap(); assert!(!cs.changes.is_empty()); assert!( cs.changes .iter() - .all(|change| change.op == ChangeOp::Insert) + .all(|change| change.op == ChangeOpKind::Insert) ); } #[tokio::test] -async fn diff_commits_resolves_commits_across_branches_from_any_handle() { +async fn graph_diff_resolves_commits_across_branches_from_any_handle() { let dir = tempfile::tempdir().unwrap(); let uri = dir.path().to_str().unwrap(); let main = init_and_load(&dir).await; @@ -745,18 +741,28 @@ async fn diff_commits_resolves_commits_across_branches_from_any_handle() { .unwrap(); let feature_commit = head_commit_id(uri, Some("feature")).await; - let from_main = main - .diff_commits(&base_commit, &feature_commit, &ChangeFilter::default()) - .await - .unwrap(); - let from_feature = feature - .diff_commits(&base_commit, &feature_commit, &ChangeFilter::default()) - .await - .unwrap(); + let from_main = graph_diff( + &main, + &base_commit, + &feature_commit, + &ChangeFeedScope::default(), + ) + .await + .unwrap(); + let from_feature = graph_diff( + &feature, + &base_commit, + &feature_commit, + &ChangeFeedScope::default(), + ) + .await + .unwrap(); assert_eq!(change_tuples(&from_main), change_tuples(&from_feature)); assert!(from_main.changes.iter().any(|change| { - change.type_key() == "node:Person" && change.id == "Eve" && change.op == ChangeOp::Insert + change.type_key() == "node:Person" + && change.id == "Eve" + && change.op == ChangeOpKind::Insert })); } @@ -788,8 +794,8 @@ async fn cross_lineage_diff_honors_delete_only_filter() { .await .unwrap(); - let filter = ChangeFilter { - ops: Some(vec![ChangeOp::Delete]), + let filter = ChangeFeedScope { + ops: Some(vec![ChangeOpKind::Delete]), ..Default::default() }; let change_set = diff_since_branch(&feature, "feature", before, &filter) @@ -804,7 +810,7 @@ async fn cross_lineage_diff_honors_delete_only_filter() { change_set .changes .iter() - .all(|change| change.op == ChangeOp::Delete) + .all(|change| change.op == ChangeOpKind::Delete) ); } @@ -827,16 +833,18 @@ async fn same_branch_diff_across_first_lazy_fork_detects_update() { .await .unwrap(); - let change_set = diff_since_branch(&feature, "feature", before, &ChangeFilter::default()) + let change_set = diff_since_branch(&feature, "feature", before, &ChangeFeedScope::default()) .await .unwrap(); assert!(change_set.changes.iter().any(|change| { - change.type_key() == "node:Person" && change.id == "Bob" && change.op == ChangeOp::Update + change.type_key() == "node:Person" + && change.id == "Bob" + && change.op == ChangeOpKind::Update })); } #[tokio::test] -async fn diff_commits_cross_branch_reports_property_only_updates() { +async fn graph_diff_cross_branch_reports_property_only_updates() { let dir = tempfile::tempdir().unwrap(); let uri = dir.path().to_str().unwrap(); let main = init_and_load(&dir).await; @@ -855,21 +863,29 @@ async fn diff_commits_cross_branch_reports_property_only_updates() { .unwrap(); let feature_commit = head_commit_id(uri, Some("feature")).await; - let change_set = main - .diff_commits(&base_commit, &feature_commit, &ChangeFilter::default()) - .await - .unwrap(); + let change_set = graph_diff( + &main, + &base_commit, + &feature_commit, + &ChangeFeedScope::default(), + ) + .await + .unwrap(); assert!(change_set.changes.iter().any(|change| { - change.type_key() == "node:Person" && change.id == "Bob" && change.op == ChangeOp::Update + change.type_key() == "node:Person" + && change.id == "Bob" + && change.op == ChangeOpKind::Update })); assert!(!change_set.changes.iter().any(|change| { - change.type_key() == "node:Person" && change.id == "Bob" && change.op == ChangeOp::Insert + change.type_key() == "node:Person" + && change.id == "Bob" + && change.op == ChangeOpKind::Insert })); } #[tokio::test] -async fn diff_commits_ignores_row_version_only_differences() { +async fn graph_diff_ignores_row_version_only_differences() { let dir = tempfile::tempdir().unwrap(); let uri = dir.path().to_str().unwrap(); let main = init_and_load(&dir).await; @@ -898,10 +914,14 @@ async fn diff_commits_ignores_row_version_only_differences() { .unwrap(); let main_commit = head_commit_id(uri, None).await; - let change_set = main - .diff_commits(&main_commit, &feature_commit, &ChangeFilter::default()) - .await - .unwrap(); + let change_set = graph_diff( + &main, + &main_commit, + &feature_commit, + &ChangeFeedScope::default(), + ) + .await + .unwrap(); assert!( change_set.changes.is_empty(), @@ -949,14 +969,18 @@ query set_body($slug: String, $body: String) { .unwrap(); let feature_commit = head_commit_id(uri, Some("feature")).await; - let change_set = db - .diff_commits(&main_commit, &feature_commit, &ChangeFilter::default()) - .await - .unwrap(); + let change_set = graph_diff( + &db, + &main_commit, + &feature_commit, + &ChangeFeedScope::default(), + ) + .await + .unwrap(); assert!( change_set.changes.iter().any(|change| { - change.type_key() == "node:Doc" && change.id == "x" && change.op == ChangeOp::Update + change.type_key() == "node:Doc" && change.id == "x" && change.op == ChangeOpKind::Update }), "null → empty-string must surface as an update, not be conflated: {:?}", change_set.changes @@ -1000,14 +1024,18 @@ query set_notes($slug: String, $notes: String) { .unwrap(); let feature_commit = head_commit_id(uri, Some("feature")).await; - let change_set = db - .diff_commits(&main_commit, &feature_commit, &ChangeFilter::default()) - .await - .unwrap(); + let change_set = graph_diff( + &db, + &main_commit, + &feature_commit, + &ChangeFeedScope::default(), + ) + .await + .unwrap(); assert!( change_set.changes.iter().any(|change| { - change.type_key() == "node:Doc" && change.id == "x" && change.op == ChangeOp::Update + change.type_key() == "node:Doc" && change.id == "x" && change.op == ChangeOpKind::Update }), "a change to a legal _row_-prefixed property must be detected: {:?}", change_set.changes @@ -1345,7 +1373,7 @@ async fn change_feed_describes_ranged_external_blob_and_advances_past_it() { /// Acceptance #7: the cross-branch net diff shares the same comparator, so a /// same-length Blob-only update on a forked branch must surface through -/// `diff_commits` too. +/// the graph diff too. #[tokio::test] async fn cross_branch_diff_detects_same_length_blob_only_update() { let dir = tempfile::tempdir().unwrap(); @@ -1372,15 +1400,19 @@ async fn cross_branch_diff_detects_same_length_blob_only_update() { .unwrap(); let feature_commit = head_commit_id(uri, Some("feature")).await; - let change_set = db - .diff_commits(&main_commit, &feature_commit, &ChangeFilter::default()) - .await - .unwrap(); + let change_set = graph_diff( + &db, + &main_commit, + &feature_commit, + &ChangeFeedScope::default(), + ) + .await + .unwrap(); assert!( change_set.changes.iter().any(|change| { change.type_key() == "node:Document" && change.id == "doc" - && change.op == ChangeOp::Update + && change.op == ChangeOpKind::Update }), "cross-branch diff dropped the Blob-only update: {:?}", change_set.changes @@ -1434,15 +1466,14 @@ async fn cross_branch_diff_detects_same_length_blob_update_between_sibling_branc .unwrap(); let b_commit = head_commit_id(uri, Some("branch-b")).await; - let change_set = db - .diff_commits(&a_commit, &b_commit, &ChangeFilter::default()) + let change_set = graph_diff(&db, &a_commit, &b_commit, &ChangeFeedScope::default()) .await .unwrap(); assert!( change_set.changes.iter().any(|change| { change.type_key() == "node:Document" && change.id == "doc" - && change.op == ChangeOp::Update + && change.op == ChangeOpKind::Update }), "cross-branch diff dropped the sibling-branch Blob update (branch-local \ fragment-id collision): {:?}", @@ -2503,6 +2534,31 @@ node Ghost { OmniError::ChangeSchemaBoundary { type_name, .. } => assert_eq!(type_name, "Person"), other => panic!("expected a typed schema boundary, got: {other:?}"), } + // A graph diff whose endpoints straddle the apply refuses the same way: + // the gate compares the two endpoints, whatever lies between them. + let add_parent = db + .get_commit(&add_commit) + .await + .unwrap() + .parent_commit_id + .unwrap(); + match db + .diff_page(&omnigraph::changes::GraphDiffRequest { + sides: omnigraph::changes::GraphDiffSides::Between { + from: add_parent, + to: "main".to_string(), + }, + scope: scope.clone(), + page_token: None, + max_changes: None, + max_bytes: None, + }) + .await + .unwrap_err() + { + OmniError::ChangeSchemaBoundary { type_name, .. } => assert_eq!(type_name, "Person"), + other => panic!("expected the graph diff's typed schema boundary, got: {other:?}"), + } // The add kept every data file: the fragment holding Alice and Carol // physically lacks `note` and `attachment`, and Lance reads both as null @@ -2607,7 +2663,7 @@ query set_attachment($name: String, $attachment: Blob) { ), ] ); - let since_add = diff_since_branch(&db, "main", after_add, &ChangeFilter::default()) + let since_add = diff_since_branch(&db, "main", after_add, &ChangeFeedScope::default()) .await .unwrap(); assert_eq!( @@ -2616,17 +2672,17 @@ query set_attachment($name: String, $attachment: Blob) { ( "node:Person".to_string(), "Alice".to_string(), - ChangeOp::Update + ChangeOpKind::Update ), ( "node:Person".to_string(), "Bob".to_string(), - ChangeOp::Insert + ChangeOpKind::Insert ), ( "node:Person".to_string(), "Carol".to_string(), - ChangeOp::Update + ChangeOpKind::Update ), ] ); @@ -3829,7 +3885,7 @@ struct StateAtCommit { struct MainHistory { commits: Vec, states: Vec, - diff: Vec<(String, String, ChangeOp)>, + diff: Vec<(String, String, ChangeOpKind)>, feed: Vec, } @@ -3872,14 +3928,14 @@ async fn main_history(db: &Session) -> MainHistory { }); } let (newest, oldest) = (commits.first().unwrap(), commits.last().unwrap()); - let diff = db - .diff_commits( - &oldest.graph_commit_id, - &newest.graph_commit_id, - &ChangeFilter::default(), - ) - .await - .unwrap(); + let diff = graph_diff( + db, + &oldest.graph_commit_id, + &newest.graph_commit_id, + &ChangeFeedScope::default(), + ) + .await + .unwrap(); let feed = db .poll_change_feed(feed_request( None, @@ -3945,26 +4001,26 @@ async fn history_reads_are_the_same_after_a_merged_branch_is_deleted_and_recreat assert_eq!(feature_commits[0].graph_commit_id, feature_head.as_str()); let feature_head_commit = db.get_commit(feature_head.as_str()).await.unwrap(); assert_eq!(feature_head_commit, feature_commits[0]); - let feature_diff = db - .diff_commits( - fork_point.as_str(), - feature_head.as_str(), - &ChangeFilter::default(), - ) - .await - .unwrap(); + let feature_diff = graph_diff( + &db, + fork_point.as_str(), + feature_head.as_str(), + &ChangeFeedScope::default(), + ) + .await + .unwrap(); assert_eq!( change_tuples(&feature_diff), vec![ ( "node:Person".to_string(), "FeatureHead".to_string(), - ChangeOp::Insert + ChangeOpKind::Insert ), ( "node:Person".to_string(), "OnFeature".to_string(), - ChangeOp::Insert + ChangeOpKind::Insert ), ] ); @@ -4041,3 +4097,500 @@ async fn history_reads_are_the_same_after_a_merged_branch_is_deleted_and_recreat ); } } + +// ─── Graph diff (RFC 0030 §8) ────────────────────────────────────────────── + +fn between(from: &str, to: &str) -> omnigraph::changes::GraphDiffSides { + omnigraph::changes::GraphDiffSides::Between { + from: from.to_string(), + to: to.to_string(), + } +} + +fn diff_request( + sides: omnigraph::changes::GraphDiffSides, + page_size: Option, +) -> omnigraph::changes::GraphDiffRequest { + omnigraph::changes::GraphDiffRequest { + sides, + scope: omnigraph::changes::ChangeFeedScope::default(), + page_token: None, + max_changes: page_size, + max_bytes: None, + } +} + +/// Every page of one graph diff, `page_size` changes at a time. Every page +/// must name the endpoints the first one resolved. +async fn graph_diff_all( + db: &Omnigraph, + sides: omnigraph::changes::GraphDiffSides, + page_size: Option, +) -> omnigraph::error::Result { + let mut request = diff_request(sides, page_size); + let mut all: Option = None; + loop { + let page = db.diff_page(&request).await?; + let next = page.next_page_token.clone(); + match all.as_mut() { + None => all = Some(page), + Some(all) => { + assert_eq!( + (&all.from_commit_id, &all.to_commit_id, &all.base_commit_id), + ( + &page.from_commit_id, + &page.to_commit_id, + &page.base_commit_id + ), + "every page answers for the first page's endpoints" + ); + all.changes.extend(page.changes); + } + } + match next { + Some(token) => request.page_token = Some(token), + None => break, + } + } + let mut all = all.expect("a diff has at least one page"); + all.next_page_token = None; + Ok(all) +} + +/// Every change of one commit's own diff, `page_size` at a time. +async fn commit_changes_all( + db: &Omnigraph, + commit_id: &str, + page_size: Option, +) -> Vec { + let scope = omnigraph::changes::ChangeFeedScope::default(); + let mut changes = Vec::new(); + let mut token: Option = None; + loop { + let page = db + .commit_changes_page(commit_id, &scope, token.as_deref(), page_size, None) + .await + .unwrap(); + changes.extend(page.block.changes); + match page.next_page_token { + Some(next) => token = Some(next), + None => return changes, + } + } +} + +fn diff_ops( + changes: &[omnigraph::changes::GraphEntityChange], +) -> Vec<(String, String, omnigraph::changes::ChangeOpKind)> { + changes + .iter() + .map(|change| { + ( + change.entity_type.name.clone(), + change.id.clone(), + change.op, + ) + }) + .collect() +} + +fn person(name: &str, age: i64) -> omnigraph_compiler::ir::ParamMap { + mixed_params(&[("$name", name)], &[("$age", age)]) +} + +async fn write_on( + db: &Session, + branch: &str, + query: &str, + params: &omnigraph_compiler::ir::ParamMap, +) -> omnigraph::db::commit_graph::GraphCommit { + db.mutate_with_receipt(branch, MUTATION_QUERIES, query, params) + .await + .unwrap() + .commit + .expect("an effectful write publishes one commit") +} + +/// The graph diff and the commit diff share one definition of a change: from +/// a commit's first parent to the commit, the graph diff is exactly that +/// commit's changes without the cause, for node and edge inserts, updates and +/// deletes and for a merge commit. +#[tokio::test] +async fn graph_diff_from_a_commit_parent_is_that_commits_changes() { + use omnigraph::changes::ChangeOpKind::*; + + let dir = tempfile::tempdir().unwrap(); + let db = init_and_load(&dir).await; + db.branch_create("feature").await.unwrap(); + write_on(&db, "feature", "insert_person", &person("Frank", 60)).await; + let mut commits = vec![ + write_on(&db, "main", "insert_person", &person("Eve", 22)).await, + write_on(&db, "main", "set_age", &person("Alice", 31)).await, + write_on( + &db, + "main", + "add_friend", + ¶ms(&[("$from", "Eve"), ("$to", "Alice")]), + ) + .await, + write_on( + &db, + "main", + "remove_friendship", + ¶ms(&[("$from", "Bob")]), + ) + .await, + ]; + let merge = db.branch_merge("feature", "main").await.unwrap(); + assert_eq!(merge.outcome, MergeOutcome::Merged); + commits.push(merge.commit.expect("a merge publishes one commit")); + + let mut seen_ops = Vec::new(); + for commit in &commits { + let parent = commit.parent_commit_id.clone().unwrap(); + let diff = graph_diff_all(&db, between(&parent, &commit.graph_commit_id), Some(1)) + .await + .unwrap(); + assert_eq!(diff.from_commit_id, parent); + assert_eq!(diff.to_commit_id, commit.graph_commit_id); + assert_eq!(diff.base_commit_id, None); + let own = commit_changes_all(&db, &commit.graph_commit_id, Some(1)).await; + assert_eq!(diff.changes, own, "commit {}", commit.graph_commit_id); + seen_ops.extend(diff_ops(&diff.changes).into_iter().map(|(_, _, op)| op)); + } + for op in [Insert, Update, Delete] { + assert!(seen_ops.contains(&op), "the fixture exercises {op:?}"); + } +} + +/// A graph diff compares its endpoints and never folds history: a reverted +/// update, an insert later deleted and a delete followed by an equal reinsert +/// produce nothing, a reinsert with another image is one update, and an +/// insert later updated is one insert with the final image. +#[tokio::test] +async fn graph_diff_reports_the_net_change_over_its_range() { + use omnigraph::changes::ChangeOpKind::*; + + let dir = tempfile::tempdir().unwrap(); + let db = init_and_load(&dir).await; + write_on(&db, "main", "insert_person", &person("Hank", 40)).await; + write_on(&db, "main", "insert_person", &person("Ivan", 20)).await; + let start = snapshot_id(&db, "main").await.unwrap().as_str().to_string(); + + write_on(&db, "main", "set_age", &person("Alice", 31)).await; + write_on(&db, "main", "set_age", &person("Alice", 30)).await; + write_on(&db, "main", "insert_person", &person("Gina", 50)).await; + write_on(&db, "main", "remove_person", ¶ms(&[("$name", "Gina")])).await; + write_on(&db, "main", "remove_person", ¶ms(&[("$name", "Hank")])).await; + write_on(&db, "main", "insert_person", &person("Hank", 40)).await; + write_on(&db, "main", "remove_person", ¶ms(&[("$name", "Ivan")])).await; + write_on(&db, "main", "insert_person", &person("Ivan", 21)).await; + write_on(&db, "main", "insert_person", &person("Jill", 18)).await; + write_on(&db, "main", "set_age", &person("Jill", 19)).await; + + let diff = graph_diff_all(&db, between(&start, "main"), None) + .await + .unwrap(); + assert_eq!( + diff_ops(&diff.changes), + vec![ + ("Person".to_string(), "Ivan".to_string(), Update), + ("Person".to_string(), "Jill".to_string(), Insert), + ] + ); + let ivan = &diff.changes[0]; + assert_eq!(ivan.before.as_ref().unwrap().properties["age"], 20); + assert_eq!(ivan.after.as_ref().unwrap().properties["age"], 21); + let jill = &diff.changes[1]; + assert!(jill.before.is_none()); + assert_eq!(jill.after.as_ref().unwrap().properties["age"], 19); +} + +/// The branch form reports what the source changed since its merge base with +/// the target, the base `branch_merge` classifies against: commits the target +/// made after the base never appear, and once the target is merged into the +/// source, only the source's own net changes since that newer base remain. +#[tokio::test] +async fn branch_diff_reports_what_the_source_changed_since_its_merge_base() { + use omnigraph::changes::ChangeOpKind::*; + use omnigraph::changes::GraphDiffSides; + + let dir = tempfile::tempdir().unwrap(); + let db = init_and_load(&dir).await; + let fork = write_on(&db, "main", "insert_person", &person("Fork", 1)).await; + db.branch_create("feature").await.unwrap(); + write_on(&db, "feature", "set_age", &person("Alice", 31)).await; + let feature_head = write_on(&db, "feature", "insert_person", &person("Eve", 22)).await; + let main_head = write_on(&db, "main", "insert_person", &person("Zed", 70)).await; + + let base = db.merge_base("feature", "main").await.unwrap(); + assert_eq!(base.merge_base.graph_commit_id, fork.graph_commit_id); + assert_eq!(base.source_commit_id, feature_head.graph_commit_id); + assert_eq!(base.target_commit_id, main_head.graph_commit_id); + + let branch = || GraphDiffSides::Branch { + source: "feature".to_string(), + target: "main".to_string(), + }; + let review = graph_diff_all(&db, branch(), Some(1)).await.unwrap(); + assert_eq!( + review.base_commit_id.as_deref(), + Some(fork.graph_commit_id.as_str()) + ); + assert_eq!(review.from_commit_id, fork.graph_commit_id); + assert_eq!(review.to_commit_id, feature_head.graph_commit_id); + let own_changes = vec![ + ("Person".to_string(), "Alice".to_string(), Update), + ("Person".to_string(), "Eve".to_string(), Insert), + ]; + assert_eq!(diff_ops(&review.changes), own_changes); + + // The two-point diff of the heads carries the target's later change + // reversed, which is why review has its own form. + let heads = graph_diff_all(&db, between("main", "feature"), None) + .await + .unwrap(); + assert!( + diff_ops(&heads.changes).contains(&("Person".to_string(), "Zed".to_string(), Delete)), + "{:?}", + diff_ops(&heads.changes) + ); + + // Merge the target into the source: the base advances to the target head. + assert_eq!( + db.branch_merge("main", "feature").await.unwrap().outcome, + MergeOutcome::Merged + ); + let base = db.merge_base("feature", "main").await.unwrap(); + assert_eq!(base.merge_base.graph_commit_id, main_head.graph_commit_id); + let review = graph_diff_all(&db, branch(), None).await.unwrap(); + assert_eq!( + review.base_commit_id.as_deref(), + Some(main_head.graph_commit_id.as_str()) + ); + assert_eq!(diff_ops(&review.changes), own_changes); +} + +/// Every read target kind resolves: a branch, a published commit, an +/// unpublished merge proposal and an unattached snapshot. A proposal's own +/// commit diff is the change its merge would publish, and once the target +/// head moves the proposal expires: its commit diff and graph diff refuse. +#[tokio::test] +async fn graph_diff_resolves_every_read_target_kind() { + use omnigraph::changes::ChangeOpKind::*; + use omnigraph::error::OmniError; + + let dir = tempfile::tempdir().unwrap(); + let db = init_and_load(&dir).await; + let main_head = snapshot_id(&db, "main").await.unwrap().as_str().to_string(); + db.branch_create("feature").await.unwrap(); + let eve = write_on(&db, "feature", "insert_person", &person("Eve", 22)).await; + let eve_insert = vec![("Person".to_string(), "Eve".to_string(), Insert)]; + + let by_commit = graph_diff_all(&db, between(&main_head, &eve.graph_commit_id), None) + .await + .unwrap(); + assert_eq!(diff_ops(&by_commit.changes), eve_insert); + let by_branch = graph_diff_all(&db, between("main", "feature"), None) + .await + .unwrap(); + assert_eq!(by_branch.changes, by_commit.changes); + + let proposal = db + .branch_merge_as_with_options("feature", "main", None, true) + .await + .unwrap() + .commit + .expect("a merge as commit records its proposal"); + assert!(!proposal.published); + let proposed = commit_changes_all(&db, &proposal.graph_commit_id, None).await; + assert_eq!(diff_ops(&proposed), eve_insert); + let to_proposal = graph_diff_all(&db, between("main", &proposal.graph_commit_id), None) + .await + .unwrap(); + assert_eq!(to_proposal.changes, proposed); + + let snapshot = db + .mutate_with_receipt( + "main", + r#" +query insert_root($name: String, $age: I32) as unattached branch { + insert Person { name: $name, age: $age } +} +"#, + "insert_root", + &person("Una", 9), + ) + .await + .unwrap() + .commit + .expect("an unattached write reports its snapshot"); + assert!(!snapshot.published); + let to_snapshot = graph_diff_all(&db, between("main", &snapshot.graph_commit_id), None) + .await + .unwrap(); + assert_eq!( + diff_ops(&to_snapshot.changes), + vec![("Person".to_string(), "Una".to_string(), Insert)] + ); + + // The proposal expires when the target head moves. + write_on(&db, "main", "insert_person", &person("Moe", 5)).await; + let scope = omnigraph::changes::ChangeFeedScope::default(); + let error = db + .commit_changes_page(&proposal.graph_commit_id, &scope, None, None, None) + .await + .unwrap_err(); + assert!(error.to_string().contains("expired"), "{error}"); + match graph_diff_all(&db, between("main", &proposal.graph_commit_id), None) + .await + .unwrap_err() + { + OmniError::ReadTargetNotFound { target } => { + assert_eq!(target, proposal.graph_commit_id) + } + other => panic!("expected ReadTargetNotFound, got {other:?}"), + } +} + +/// A side that names both a live branch and a commit is refused rather than +/// resolved by precedence, and a side that names neither is a typed +/// not-found. The genesis commit id is a bare ULID, which is a legal branch +/// name. +#[tokio::test] +async fn graph_diff_refuses_an_ambiguous_or_unknown_side() { + use omnigraph::error::OmniError; + + let dir = tempfile::tempdir().unwrap(); + let db = init_and_load(&dir).await; + let genesis = db + .list_commits(Some("main")) + .await + .unwrap() + .into_iter() + .find(|commit| commit.parent_commit_id.is_none()) + .unwrap() + .graph_commit_id; + // Before the branch exists, the id names only the commit. + graph_diff_all(&db, between(&genesis, "main"), None) + .await + .unwrap(); + + db.branch_create(&genesis).await.unwrap(); + match graph_diff_all(&db, between(&genesis, "main"), None) + .await + .unwrap_err() + { + OmniError::AmbiguousReadTarget { target } => assert_eq!(target, genesis), + other => panic!("expected AmbiguousReadTarget, got {other:?}"), + } + assert!(matches!( + db.diff_read_target_branch(&genesis).await.unwrap_err(), + OmniError::AmbiguousReadTarget { .. } + )); + + match graph_diff_all(&db, between("main", "no-such-side"), None) + .await + .unwrap_err() + { + OmniError::ReadTargetNotFound { target } => assert_eq!(target, "no-such-side"), + other => panic!("expected ReadTargetNotFound, got {other:?}"), + } +} + +/// Continuations answer for the commits the first page resolved, even after +/// the named branch moves, and a token works only for the sides and filter it +/// was minted for and never as another continuation kind. +#[tokio::test] +async fn graph_diff_pages_answer_for_the_first_pages_endpoints() { + use omnigraph::changes::{ChangeFeedScope, ChangeOpKind}; + use omnigraph::error::OmniError; + + let dir = tempfile::tempdir().unwrap(); + let db = init_and_load(&dir).await; + let start = snapshot_id(&db, "main").await.unwrap().as_str().to_string(); + for i in 0..5 { + write_on(&db, "main", "insert_person", &person(&format!("P{i}"), i)).await; + } + let mut request = diff_request(between(&start, "main"), Some(2)); + let first = db.diff_page(&request).await.unwrap(); + assert_eq!(first.changes.len(), 2); + let token = first + .next_page_token + .clone() + .expect("five changes, pages of two"); + + write_on(&db, "main", "insert_person", &person("Late", 9)).await; + let mut ids: Vec = first.changes.iter().map(|c| c.id.clone()).collect(); + request.page_token = Some(token.clone()); + loop { + let page = db.diff_page(&request).await.unwrap(); + assert_eq!(page.to_commit_id, first.to_commit_id); + ids.extend(page.changes.iter().map(|c| c.id.clone())); + match page.next_page_token { + Some(next) => request.page_token = Some(next), + None => break, + } + } + assert_eq!(ids, ["P0", "P1", "P2", "P3", "P4"]); + + let mut other_sides = diff_request(between(&start, "feature"), Some(2)); + other_sides.page_token = Some(token.clone()); + assert!(matches!( + db.diff_page(&other_sides).await.unwrap_err(), + OmniError::ChangeCursorRejected { .. } + )); + let mut other_scope = diff_request(between(&start, "main"), Some(2)); + other_scope.page_token = Some(token.clone()); + other_scope.scope = ChangeFeedScope { + ops: Some(vec![ChangeOpKind::Delete]), + ..ChangeFeedScope::default() + }; + assert!(matches!( + db.diff_page(&other_scope).await.unwrap_err(), + OmniError::ChangeCursorRejected { .. } + )); + + // A commit changes page token is not a diff page token, nor the reverse. + let pair = db + .mutate_with_receipt( + "main", + MUTATION_QUERIES, + "insert_person_and_friend", + &mixed_params(&[("$name", "Pat"), ("$friend", "Alice")], &[("$age", 3)]), + ) + .await + .unwrap() + .commit + .unwrap(); + let commit_token = db + .commit_changes_page( + &pair.graph_commit_id, + &ChangeFeedScope::default(), + None, + Some(1), + None, + ) + .await + .unwrap() + .next_page_token + .expect("two changes, pages of one"); + let mut cross = diff_request(between(&start, "main"), Some(2)); + cross.page_token = Some(commit_token); + assert!(matches!( + db.diff_page(&cross).await.unwrap_err(), + OmniError::ChangeCursorRejected { .. } + )); + assert!(matches!( + db.commit_changes_page( + &pair.graph_commit_id, + &ChangeFeedScope::default(), + Some(&token), + Some(1), + None, + ) + .await + .unwrap_err(), + OmniError::ChangeCursorRejected { .. } + )); +} diff --git a/crates/omnigraph/tests/integration/legacy_columns.rs b/crates/omnigraph/tests/integration/legacy_columns.rs index 2051830fa..e8d4c1a32 100644 --- a/crates/omnigraph/tests/integration/legacy_columns.rs +++ b/crates/omnigraph/tests/integration/legacy_columns.rs @@ -4,7 +4,7 @@ use crate::helpers; use arrow_array::{StringArray, StructArray}; -use omnigraph::changes::{ChangeFeedScope, ChangeFilter, ChangeOp, ChangeOpKind}; +use omnigraph::changes::{ChangeFeedScope, ChangeOpKind}; use omnigraph::db::{Omnigraph, ReadTarget}; use omnigraph::loader::LoadMode; @@ -403,31 +403,31 @@ async fn legacy_vintage_graph_mutates_diffs_evolves_and_reopens() { ); let changed_commit = snapshot_id(&db, "main").await.unwrap(); - let diff = db - .diff_commits( - loaded_commit.as_str(), - changed_commit.as_str(), - &ChangeFilter::default(), - ) - .await - .unwrap(); + let diff = helpers::graph_diff( + &db, + loaded_commit.as_str(), + changed_commit.as_str(), + &ChangeFeedScope::default(), + ) + .await + .unwrap(); assert_eq!(diff.changes.len(), 3); assert!( diff.changes .iter() - .any(|change| change.id == "Alice" && change.op == ChangeOp::Update) + .any(|change| change.id == "Alice" && change.op == ChangeOpKind::Update) ); assert!( diff.changes .iter() - .any(|change| change.id == "Bob" && change.op == ChangeOp::Delete) + .any(|change| change.id == "Bob" && change.op == ChangeOpKind::Delete) ); let deleted_edge = diff .changes .iter() .find(|change| change.id == "works-bob") .unwrap(); - assert_eq!(deleted_edge.op, ChangeOp::Delete); + assert_eq!(deleted_edge.op, ChangeOpKind::Delete); let endpoints = deleted_edge.endpoints.as_ref().unwrap(); assert_eq!(endpoints.src, "Bob"); assert_eq!(endpoints.dst, "company-1"); diff --git a/crates/omnigraph/tests/integration/merge_truth_table.rs b/crates/omnigraph/tests/integration/merge_truth_table.rs index 295cb1da4..1d2a96bad 100644 --- a/crates/omnigraph/tests/integration/merge_truth_table.rs +++ b/crates/omnigraph/tests/integration/merge_truth_table.rs @@ -27,6 +27,11 @@ //! [`OmniError::MergeConflicts`] error), and //! * the affected graph state on `main` after a successful merge. //! +//! Before the merge, every cell runs `branch_merge_dry_run` on the same +//! inputs (RFC 0030 §8.4): it must report the merge's own outcome or conflict +//! list while writing nothing to the object store and leaving `main` where it +//! was. +//! //! `branch_merge(source, target)` is directional: target acts as the merge //! base anchor while source's diff is replayed. The matrix already //! enumerates both `(L, R)` and `(R, L)` as independent cells (e.g. @@ -862,19 +867,41 @@ async fn run_direction( apply(&db, "feature", left_op).await; apply(&db, "main", right_op).await; - let merge_result = db.branch_merge("feature", "main").await; - let outcome = match merge_result.map(|result| result.outcome) { - Ok(MergeOutcome::AlreadyUpToDate) => ActualOutcome::AlreadyUpToDate, - Ok(MergeOutcome::FastForward) => ActualOutcome::FastForward, - Ok(MergeOutcome::Merged) => ActualOutcome::Merged, - Err(OmniError::MergeConflicts(conflicts)) => ActualOutcome::Conflicts( - conflicts - .into_iter() - .map(|c| (c.type_key, c.kind, c.entity_id)) - .collect(), - ), - Err(other) => panic!("[{label}] unexpected merge error: {other:?}"), - }; + let main_before = helpers::snapshot_id(&db, "main").await.unwrap(); + let (dry_run, dry_io) = + helpers::cost::measure(db.branch_merge_dry_run("feature", "main")).await; + if let Ok(result) = &dry_run { + assert!( + result.commit.is_none(), + "[{label}] a dry run returned a commit" + ); + } + let dry_outcome = actual_outcome(label, dry_run.map(|result| result.outcome)); + assert_eq!( + (dry_io.manifest_writes, dry_io.data_writes), + (0, 0), + "[{label}] a dry run wrote to the object store: {dry_io:?}" + ); + assert_eq!( + helpers::snapshot_id(&db, "main").await.unwrap(), + main_before, + "[{label}] a dry run moved main" + ); + + let (merge_result, merge_io) = helpers::cost::measure(db.branch_merge("feature", "main")).await; + let outcome = actual_outcome(label, merge_result.map(|result| result.outcome)); + assert!( + dry_outcome == outcome, + "[{label}] the dry run answered {dry_outcome:?}, the merge {outcome:?}" + ); + if matches!(outcome, ActualOutcome::FastForward | ActualOutcome::Merged) { + // The counter the dry run is held to zero on sees a real merge's + // publication, so a zero above is evidence and not a dead probe. + assert!( + merge_io.manifest_writes > 0, + "[{label}] the merge published without a counted __manifest write: {merge_io:?}" + ); + } if let Expected::Merged(assert) | Expected::FastForward(assert) = expected && matches!( @@ -902,6 +929,21 @@ async fn run_direction( } } +fn actual_outcome(label: &str, result: Result) -> ActualOutcome { + match result { + Ok(MergeOutcome::AlreadyUpToDate) => ActualOutcome::AlreadyUpToDate, + Ok(MergeOutcome::FastForward) => ActualOutcome::FastForward, + Ok(MergeOutcome::Merged) => ActualOutcome::Merged, + Err(OmniError::MergeConflicts(conflicts)) => ActualOutcome::Conflicts( + conflicts + .into_iter() + .map(|c| (c.type_key, c.kind, c.entity_id)) + .collect(), + ), + Err(other) => panic!("[{label}] unexpected merge error: {other:?}"), + } +} + /// Expected state on `main` after `action` alone runs on the base fixture. /// /// Used by the conflict-cell invariant in [`run_direction`] to assert diff --git a/crates/omnigraph/tests/integration/warm_read_cost.rs b/crates/omnigraph/tests/integration/warm_read_cost.rs index 25c9b73d2..2a68d8764 100644 --- a/crates/omnigraph/tests/integration/warm_read_cost.rs +++ b/crates/omnigraph/tests/integration/warm_read_cost.rs @@ -2023,7 +2023,7 @@ async fn people_as_of(db: &Session, commit: &omnigraph::db::GraphCommit) -> usiz #[tokio::test] async fn by_id_reads_of_held_commits_issue_no_request_to_history() { use omnigraph::changes::{ - ChangeFeedPosition, ChangeFeedRequest, ChangeFeedScope, ChangeFeedStart, ChangeFilter, + ChangeFeedPosition, ChangeFeedRequest, ChangeFeedScope, ChangeFeedStart, }; let dir = tempfile::tempdir().unwrap(); @@ -2072,14 +2072,14 @@ async fn by_id_reads_of_held_commits_issue_no_request_to_history() { "state of the commit {age} publishes before the head" ); } - reader - .diff_commits( - &main[2].graph_commit_id, - &main[1].graph_commit_id, - &ChangeFilter::default(), - ) - .await - .unwrap(); + helpers::graph_diff( + &reader, + &main[2].graph_commit_id, + &main[1].graph_commit_id, + &ChangeFeedScope::default(), + ) + .await + .unwrap(); assert_eq!( history.requests(), (0, 0), diff --git a/docs/dev/merge.md b/docs/dev/merge.md index 16a7eb296..2c9c41d86 100644 --- a/docs/dev/merge.md +++ b/docs/dev/merge.md @@ -59,6 +59,15 @@ An optimization miss is not a merge failure. Missing transaction history, unknown certificate fields, or an unfamiliar Lance shape falls back to the general route. +A dry run (`Session::branch_merge_dry_run`, RFC 0030 §8.4) takes the same +path through base resolution, classification and constraint validation, then +returns before the queue guards and the first detached commit. It is a read: +it takes neither the schema permit nor the branch gates, pins no merge-input +tag (a reclaimed input surfaces as a typed gap), and stages only to the +process-local scratch directory. The merge truth table runs it before every +cell's merge and holds it to the merge's outcome and to zero object-store +writes. + On the adopt route, every target, main included, selects the source's exact table ref, version and pin: the target's registration takes the source's `staged_version` in the same dataset, with no fenced insert, keyed update or diff --git a/docs/rfcs/0030-cdc-time-travel.md b/docs/rfcs/0030-cdc-time-travel.md index b645a2a8d..5b1e3c561 100644 --- a/docs/rfcs/0030-cdc-time-travel.md +++ b/docs/rfcs/0030-cdc-time-travel.md @@ -27,9 +27,9 @@ blocked_on: [] > the feed contract, which is entity changes, the exact ordered comparison as > authority and fallback, cursors, and retention gaps. -C0–C3 are implemented. The §4.4 ordering, client-pagination, and -continuation-size gates are closed as recorded in §14. The graph diff phases -D1–D3 (§8) and C4–C6 remain design-stage. +C0–C3 and the graph diff phases D1–D3 (§8) are implemented. The §4.4 +ordering, client-pagination, and continuation-size gates are closed as +recorded in §14. C4–C6 remain design-stage. **Depends on:** RFC 0013 Phase 7 graph lineage, RFC 0022 snapshot capture and publication, RFC 0023 exact-`id` table fencing, and RFC 0028 immutable table @@ -51,9 +51,8 @@ exact logical before/after values. Public results do not expose tables, datasets, fragments, physical versions, row addresses, or storage ordering keys. -Two shipped public surfaces share one private adjacent-commit enumerator, and a -third, design-stage surface shares its type-lifetime pairing and image -comparison: +Three public surfaces share one private enumerator of the entity changes +between two pinned graph snapshots: 1. A **finite commit entity diff** compares one exact graph commit with its first parent. It is an inspection result. A `next_page_token` may continue a @@ -62,7 +61,7 @@ comparison: first-parent order. Its cursor means the caller has consumed every complete commit through one position; it advances only after the final page of that commit. -3. A **graph diff** (§8, design-stage) compares any two graph read targets, or +3. A **graph diff** (§8) compares any two graph read targets, or a branch with its merge base, and returns the net entity changes between them in the same entity shape, without a cause. @@ -166,10 +165,10 @@ The required durable authority already exists: not a second registry or a table identifier. - Every current v6 graph table is created with stable row IDs and exact non-null logical `id` as its unenforced primary key. -- `diff_between` and `diff_commits` compare two graph snapshots, including - table creation, removal, rename, and cross-lineage ID comparison. They are - embedded Rust APIs only: no CLI, HTTP or MCP surface calls them, and they - return the table-shaped `ChangeSet` that §8 retires. +- `Omnigraph::diff_page` compares any two graph snapshots, including renamed + and cross-lineage type lifetimes, through the commit diff's enumerator + (§8). It replaced the embedded-only, table-shaped `diff_between` and + `diff_commits`. - Every graph commit record written at storage format 14 or later carries the accepted schema identity it was written under and names its schema content (`schema_content_hash`), an immutable `__history/schemas/` object holding @@ -177,7 +176,7 @@ The required durable authority already exists: older stamp shares one contract across its pre-upgrade commits ([schema contract in the manifest](2026-09-30-schema-contract-in-manifest.md)). - `CommitGraph::merge_base` computes the merge base branch merge classifies - against; no public surface reports it. + against; `branch merge-base` reports it (§8.4). Two version names require precise scope: @@ -638,11 +637,11 @@ proven monotonic index. ## 8. Graph diff and the existing net diff -### 8.1 What the shipped surfaces leave open +### 8.1 What the commit diff and the feed left open The commit diff and the feed answer what one commit changed and what a branch's -first-parent history changed, commit by commit. Two questions have no public -answer: +first-parent history changed, commit by commit. Before D1–D3, two questions had +no public answer: - **Reviewing a branch.** Branches exist so that a change can be inspected before it merges, but no surface reports what a branch changed since it left @@ -704,15 +703,18 @@ only, refuses with `schema_boundary` until C6 defines how to cross it. Changes use the §3.1 entity shape and §4.4 order, with no cause: a range collapsed across commits has no single honest actor or commit. The response carries the resolved `from` and `to` commit ids and, for the branch form, -`base`. A page token of a new `diff-page` kind binds the graph identity, both -resolved endpoints, the filter digest and the position; it is never accepted -as a feed cursor. Filters are the §3.3 kind, type and operation filters. An +`base`. A page token of a new `diff-page` kind binds the graph identity, a +digest of the request's sides (their form and spelled names, never a raw +branch name), both resolved endpoints, the filter digest and the position; it +is never accepted as a feed cursor or a commit page token. Filters are the §3.3 kind, type and operation filters. An unreadable endpoint maps to the §6 typed outcomes. Authorization is `read` on the branch each endpoint resolves to. A commit endpoint authorizes against its authored branch, or against the recorded -target of a proposal or snapshot, and a forbidden endpoint is reported as not -found, as the commit diff does. +target of a proposal or snapshot. The two-point route resolves a side before +it authorizes it, so a forbidden side is reported exactly as a side that names +nothing (`404`), as the commit diff does. The branch form and the merge base +name their branches outright and answer a read denial with `403`. Cost: a type lifetime whose `to` pin is one proving transaction past its `from` pin may use the §14 candidate pruning. Every other changed lifetime takes the @@ -745,9 +747,11 @@ also lets a caller replay a branch's own commits through the feed with `--start after:`. `branch merge --dry-run` runs the merge's base resolution, three-way -classification and constraint validation, reports the outcome -(`already_up_to_date`, `fast_forward` or `merged`), the structured conflict -list and change counts per type and operation, and stops. It is a read: +classification and constraint validation and stops. It reports the outcome +(`already_up_to_date`, `fast_forward` or `merged`), or fails with the merge's +own structured conflict list. It reports no change counts: a merge that adopts +a table's pointer computes no row delta for it, so counts would need a second +diff, and `branch diff` already answers what changes. It is a read: - it requires `read` on both branches, not merge rights; - it captures its inputs as an ordinary read snapshot, without merge-input @@ -767,7 +771,7 @@ answers whether the merge would succeed, without a write. | `diff --from --to ` | `GET /graphs/{id}/diff?from=&to=` | Net changes between two read targets | | `branch diff [--into ]` | `GET /graphs/{id}/branches/diff?source=&target=` | What the source changed since its merge base with the target | | `branch merge-base [--into ]` | `GET /graphs/{id}/branches/merge-base?source=&target=` | The merge base commit | -| `branch merge --dry-run` | `POST /graphs/{id}/branches/merge` with `dry_run: true` | Merge outcome, conflicts and counts, without a write | +| `branch merge --dry-run` | `POST /graphs/{id}/branches/merge` with `dry_run: true` | Merge outcome or conflicts, without a write | | `commit changes `, unchanged | `GET /graphs/{id}/commits/{commit_id}/changes`, unchanged | One commit's changes with its cause | The diff commands take the existing `--kind`, `--type`, `--op`, `--limit`, @@ -778,11 +782,12 @@ records rather than query rows. ### 8.6 Retiring the table-shaped diff -`diff_between` and `diff_commits` become one engine call that returns pages of -the §3.1 entity shape, which the CLI and server use. The table-shaped -`ChangeSet`, `EntityChange` and `ChangeStats`, and with them the stamp path's -update rule, are removed once their callers, the DST harness and engine tests, -move to it. The planner's `SnapshotDiff` operation already models this pair; +`diff_between` and `diff_commits` became one engine call, +`Omnigraph::diff_page`, which returns pages of the §3.1 entity shape and which +the CLI and server use. The table-shaped `ChangeSet`, `EntityChange`, +`ChangeStats`, `ChangeFilter` and `ChangeOp`, the stamp path with its update +rule, and the adjacency classifier only `diff_commits` read are removed; the +DST harness and the engine tests moved to the graph diff. The planner's `SnapshotDiff` operation already models this pair; its registry routes no diff until an integration lands with differential evidence, so routing D1 through it is an implementation choice that needs that evidence. @@ -883,9 +888,9 @@ server `integration/data_routes.rs` and `integration/openapi.rs`; CLI - A same-value rewrite across a linear interval of pre-detached history reports no change, proving the stamp path's update rule is gone. - Branch form: the result excludes target commits after the base; the - reported base equals branch merge's base on the merge truth-table fixtures, - ties included; after the target was merged into the source, only the - source's own net changes since the newest base appear. + reported base is the one branch merge classifies against, because one + function computes both; after the target was merged into the source, only + the source's own net changes since the newest base appear. - Endpoints: a branch, a published commit, a proposal and an unattached snapshot all resolve; a string naming both a branch and a commit is refused; pages after a concurrent commit on either branch still answer for the pinned @@ -1032,7 +1037,7 @@ and must return to this RFC's format audit before landing. | C6 — schema boundary crossing | Decode each endpoint under its archived schema; cross additive changes with one schema record per block and no synthetic entity changes; other schema changes keep refusing until §10.1's decisions are made | No new persisted authority; consumers opt in through cursor and page-token scope | | D1 — two-point graph diff | `diff --from --to` and `GET /diff` over one engine call in the §3.1 shape; ambiguity refusal; `diff-page` token; retirement of the table-shaped `ChangeSet` | Net comparison of any two read targets | | D2 — branch review | `branch merge-base` and `branch diff`, with base and source head captured together | Branches reviewable without a write | -| D3 — merge dry run | Read-only base resolution, classification and validation reporting outcome, conflicts and counts | Merge feasibility without merge rights | +| D3 — merge dry run | Read-only base resolution, classification and validation reporting the outcome or the conflicts | Merge feasibility without merge rights | C0 deliberately does **not** add a second coordinator or an O(history log history) ancestry index. `CommitGraph` already holds the warm lineage @@ -1363,3 +1368,22 @@ Amendments before 2026-10-12 are recorded in §14 and in the dispositions above. `query --snapshot` on the proposal refuse with "expired and awaits cleanup", `commit publish` fails its head precondition, and `commit drop` succeeds. +- **2026-10-12: D1–D3 implemented.** `Omnigraph::diff_page` (two-point and + branch forms, the `diff-page` token, typed `AmbiguousReadTarget` and + `ReadTargetNotFound`), `Omnigraph::merge_base`, + `Session::branch_merge_dry_run`, `GET /diff`, `GET /branches/diff`, + `GET /branches/merge-base`, `dry_run` on `POST /branches/merge`, and the CLI + `diff`, `branch diff`, `branch merge-base` and `branch merge --dry-run`. + The table-shaped diff is retired (§8.6). Two decisions changed in + implementation: the dry run reports no change counts (§8.4, whose sentence + "reports the outcome …, the structured conflict list and change counts per + type and operation, and stops" it supersedes), and the `diff-page` token also + binds a digest of the request's sides. Evidence: engine `changes.rs` + graph-diff tests and the dry run inside every `merge_truth_table` cell (zero + object-store writes, the merge's own outcome); server `data_routes.rs` and + `auth_policy.rs` route and policy tests; CLI `parity_matrix.rs` + `parity_diff_over_shared_history` and `parity_branch_review`; GQT cases + `graph_diff_branch_review.gqt`, `graph_diff_merge_proposal.gqt`, + `graph_diff_net_range.gqt` and `graph_diff_refusals.gqt` on the engine, DST + and served targets (RFC 0045's new `--- diff`, `--- merge-base` and + `--- merge-dry-run` steps). diff --git a/docs/rfcs/0045-gq-logic-tests.md b/docs/rfcs/0045-gq-logic-tests.md index a8fa29043..9bce21f92 100644 --- a/docs/rfcs/0045-gq-logic-tests.md +++ b/docs/rfcs/0045-gq-logic-tests.md @@ -7,7 +7,7 @@ implementation: partial authors: - azimafroozeh created: 2026-08-29 -updated: 2026-09-28 +updated: 2026-10-12 discussion: https://github.com/ModernRelay/omnigraph/pull/584 supersedes: [] superseded_by: [] @@ -600,6 +600,18 @@ in Seams at an explicit step. A step is one of: there is no row expect on a mutate step. - `--- restart`, body empty: drop the store handle and reopen it from the same URI before the next step. +- `--- diff`, `--- merge-base` and `--- merge-dry-run`, body empty: the graph + diff surfaces of RFC 0030 §8, which are not GQ statements. `--- diff` + names its sides as `from: to: `, as `source: ` with an + optional `target: ` (`main` when absent), or as `commit: `, + where a side is a branch name or a bound `$name`; it follows every page and + takes `ordered` or `unordered` rows, one JSON object per change in the wire + shape with `type` spelled as the type's name and compared whatever its key + order, or `error: `. `--- merge-base source: + [target: ]` takes `commit $name`, the bound commit the base must be, + or `error: `. `--- merge-dry-run source: + [target: ]` takes the `branch merge` expects (`ok`, `outcome:`, + `error:`) and fails if the dry run reports a commit. - A settings step (a `--- mutate` whose body is a `set ...;` statement) and a show step (a `--- query` whose body is `show ...;`) take no `via`: their headers stay bare, a `via` on them is refused with the grammar, they @@ -2662,3 +2674,11 @@ historical command and configuration descriptions are not migration aliases. `order_key_aggregate_or_node_binds_to_return_item.gqt` (§Evidence and tests) and `engine_v1_refuses_compound_predicates.gqt` to `compound_and_bare_boolean_filters.gqt`. +- 2026-10-12, graph diff steps: `--- diff`, `--- merge-base` and + `--- merge-dry-run` join the step kinds (§File format) for the surfaces of + RFC 0030 §8, with the `expect commit $name` mode for a merge base. A + `--- query` step cannot hold them: its rows answer a GQ statement under a + typed result schema, and a diff is paged change records that RFC 0030 keeps + out of GQ. The engine and DST targets call the engine, and the served target + sends the HTTP routes, so one case pins all three. Additive: no earlier + sentence is superseded. diff --git a/docs/user/branching/changes.md b/docs/user/branching/changes.md index 6a9035c7c..b7aa4316f 100644 --- a/docs/user/branching/changes.md +++ b/docs/user/branching/changes.md @@ -1,8 +1,9 @@ -# Changes and Change Feeds +# Changes, Diffs and Change Feeds -OmniGraph can describe what one commit changed or deliver an ordered feed of -changes on a branch. Both surfaces report logical nodes and edges, not storage -details. +OmniGraph can describe what one commit changed, compare any two points in +history, review what a branch changed, or deliver an ordered feed of changes +on a branch. Every surface reports logical nodes and edges, not storage +details, in one change shape. HTTP calls follow the [HTTP contract](../operations/server.md#http-contract), including the required request header and response check before consuming a @@ -41,6 +42,58 @@ that crosses a schema boundary returns `409` with reason `schema_boundary`; take a new baseline instead of treating either response as an empty change. +## Compare two points + +```bash +omnigraph diff --from --to main --store graph.omni --json +``` + +Each side is a branch name, meaning its head when the diff starts, or a commit +id: a published commit, an unpublished proposal, or an unattached snapshot. A +string that names both a branch and a commit is refused; name the branch's +head by its commit id instead. The equivalent HTTP route is +`GET /graphs/{graph_id}/diff?from=&to=`. + +The result is the net change from the `from` state to the `to` state, in the +`commit changes` change shape and order, with `from_commit_id` and +`to_commit_id` naming the exact commits the sides resolved to. There is no +`cause`: a range of commits has no single author. Intermediate history is not +replayed, so an update that was later reverted produces nothing, and a delete +followed by an equal reinsert produces nothing. A diff from a commit's first +parent to the commit equals that commit's `commit changes`, without the cause. + +Comparing two branch heads also shows what the target changed since the +branches diverged, reversed. To review a branch, use the branch form below. + +The kind, type and operation filters, `--limit` and `--page-token` work as for +`commit changes`. A page token keeps the commits the first page resolved, so +later pages answer for them even after the branches move. A diff across a +schema change returns `409` with reason `schema_boundary`, and a side whose +history was reclaimed returns `410` with `change_feed_gap`. + +Cost grows with the size of each changed type, not with the number of +changes: apart from a single commit, every changed type is compared in full. +Types unchanged between the two points are skipped. + +## Review a branch + +```bash +omnigraph branch diff feature --into main --store graph.omni --json +omnigraph branch merge-base feature --into main --store graph.omni --json +``` + +`branch diff` reports what `feature` changed since its merge base with `main` +(`main` when `--into` is omitted): the net change from the merge base to the +head of `feature`. Changes `main` made after the merge base do not appear. The +response also names the base as `base_commit_id`. `branch merge-base` prints +that commit alone, the one a merge of `feature` into `main` would classify +against. It is useful for replaying the branch's own commits with +`changes poll --branch feature --start after:`. + +The HTTP routes are `GET /graphs/{graph_id}/branches/diff?source=&target=` and +`GET /graphs/{graph_id}/branches/merge-base?source=&target=`. To see whether +the merge would succeed, run it as a [dry run](merge.md#preview-a-merge). + ## Follow a branch ```bash @@ -83,8 +136,8 @@ and `page_token` parameters are mutually exclusive. ## Recover from a retention gap -A cursor does not prevent `cleanup` from reclaiming old history. When a feed -or commit diff can no longer be read, the server returns `410` with +A cursor does not prevent `cleanup` from reclaiming old history. When a feed, +commit diff or diff can no longer be read, the server returns `410` with `change_feed_gap`, including `first_unreadable_commit_id` and, when available, the rejected `cursor`. Retrying the same cursor cannot close the gap; install a new baseline: @@ -118,7 +171,9 @@ resumed feed begins with commits after the captured snapshot. Baseline kind and type filters select the snapshot contents. An operation filter applies only to the resumed feed. Baselines require the `export` -policy action; commit changes and feed polling require `read`. +policy action; commit changes, diffs and feed polling require `read`. A diff +needs `read` on the branch each side resolves to; a side the actor cannot read +answers `404` exactly as a side that names nothing. The CLI's durable `--out` installation is available on POSIX platforms. The file contains only snapshot entity records, not the terminal handshake. The CLI diff --git a/docs/user/branching/index.md b/docs/user/branching/index.md index 4b9bb8cda..bc4213a1d 100644 --- a/docs/user/branching/index.md +++ b/docs/user/branching/index.md @@ -216,10 +216,13 @@ needs. Branch deletion can likewise end access to branch-only history. ## Changes and feeds -Inspect the logical changes made by one commit: +Inspect the logical changes made by one commit, compare two points, or review +what a branch changed since it left `main`: ```bash omnigraph commit changes --store graph.omni --json +omnigraph diff --from --to main --store graph.omni --json +omnigraph branch diff feature --store graph.omni --json ``` For continuous consumption, `omnigraph changes poll` follows complete commits @@ -227,7 +230,7 @@ on one branch and returns an opaque resume cursor. If cleanup has reclaimed required history, create a coherent snapshot and new cursor with `omnigraph changes baseline`. -See [Changes and Change Feeds](changes.md) for filters, pagination, cursor -checkpointing, and `410` retention-gap recovery. +See [Changes, Diffs and Change Feeds](changes.md) for filters, pagination, +cursor checkpointing, and `410` retention-gap recovery. See [Merging Branches](merge.md) for merge outcomes and conflict handling. diff --git a/docs/user/branching/merge.md b/docs/user/branching/merge.md index 74c58227a..9f3b8be00 100644 --- a/docs/user/branching/merge.md +++ b/docs/user/branching/merge.md @@ -91,6 +91,24 @@ string. Inspect and retry deletion separately; do not replay a successful merge. Deleting a branch is irreversible and may make branch-only commits unavailable. It also releases old history for a later cleanup. +## Preview a merge + +```bash +omnigraph branch merge feature --into main --dry-run --store graph.omni --json +``` + +A dry run runs the merge through classification and constraint validation, +then stops. It reports the outcome the merge would have now, or fails with the +same structured conflict list, and writes nothing: `commit` is `null` and +`dry_run` is `true`. It is a read of both branches, so it needs `read` on each +and no `branch_merge` grant. It refuses `--as-commit` and `--delete-branch`. +Over HTTP, send `dry_run: true` to `POST /graphs/{graph_id}/branches/merge`. + +A dry run answers whether the merge would succeed. To see what it would change, +use [`branch diff`](changes.md#review-a-branch). To review the exact result +before it lands, merge `--as-commit` and inspect the proposal with +`commit changes`; the proposal expires once the target branch moves. + ## Conflicts When both sides changed the same logical data incompatibly, the merge returns a diff --git a/docs/user/cli/reference.md b/docs/user/cli/reference.md index 7d094cf19..ab4e144db 100644 --- a/docs/user/cli/reference.md +++ b/docs/user/cli/reference.md @@ -44,9 +44,10 @@ server resolves the actor from the bearer token. Drop it, or use `--store ` | `mutate` | Run an insert/update/delete query (`as commit` leaves it unpublished; `as unattached branch [on ""]` records a snapshot that never publishes), or a `branch create`, `branch delete`, `branch merge`, `commit publish`, or `commit drop` statement | direct or served | | `load` | Load graph JSONL in `overwrite`, `append`, or `merge` mode | direct or served | | `blob get/stat/put/clear` | Read, inspect, replace or clear one Blob cell | direct or served | -| `branch create/list/delete/merge` | Manage graph branches; `merge --as-commit` leaves the merge unpublished | direct or served | +| `branch create/list/delete/merge/diff/merge-base` | Manage graph branches; `merge --as-commit` leaves the merge unpublished, `merge --dry-run` reports the outcome or conflicts and writes nothing, `diff` reviews what a branch changed since its merge base, `merge-base` prints that base | direct or served | | `snapshot` | Show a branch snapshot: `internal_schema_version`, `graph_manifest_version`, and the datasets of one captured graph version | direct or served | | `commit list/show/changes/publish/drop` | Inspect history (`list --unpublished` for a branch's unpublished commits, `list --unattached` for its live unattached snapshots; the two flags exclude each other) or one commit's entity changes; publish an unpublished commit onto its recorded target branch (an unattached snapshot refuses with `unattached_cannot_publish`), or drop it (an unattached snapshot takes every descendant with it) | direct or served | +| `diff` | Net entity changes between two points: branches or commit ids (`--from`, `--to`) | direct or served | | `changes poll/baseline` | Consume a branch change feed or establish a new baseline | direct or served | | `export` | Stream a branch as JSONL | direct or served | | `schema show` | Read the accepted schema | direct or served | @@ -169,8 +170,7 @@ and one request to 300 seconds; uncertain writes are never automatically retried. See [managed bulk loading](managed-data.md#bulk-loading) for limits, permissions, ordinary addressing, and reconciliation. -Change-feed commands, cursor checkpointing, and baseline recovery are described -in [Changes and Change Feeds](../branching/changes.md). +Diffs, branch review and change feeds: [Changes, Diffs and Change Feeds](../branching/changes.md). ## Blob commands diff --git a/docs/user/index.md b/docs/user/index.md index 901c1f7eb..9277096c8 100644 --- a/docs/user/index.md +++ b/docs/user/index.md @@ -24,7 +24,7 @@ deployment. ## History and collaboration - [Branches, commits, and snapshots](branching/index.md) -- [Commit changes, change feeds, and baselines](branching/changes.md) +- [Commit changes, diffs, branch review, change feeds, and baselines](branching/changes.md) - [Merging and conflicts](branching/merge.md) ## CLI and HTTP diff --git a/docs/user/operations/server.md b/docs/user/operations/server.md index 494870bd0..aeeea526b 100644 --- a/docs/user/operations/server.md +++ b/docs/user/operations/server.md @@ -229,9 +229,9 @@ keep their existing routes and do not expose MCP. | `/graphs/{id}/queries` | List and invoke stored queries, including conditional mutations | | `/graphs/{id}/load`, `/load/ndjson` | Bounded batch loading | | `/graphs/{id}/blob` | GET/HEAD one Blob cell; PUT/DELETE to replace or clear it | -| `/graphs/{id}/branches` | Branch management and merge | +| `/graphs/{id}/branches` | Branch management and merge, a merge dry run (`dry_run: true`), a branch's diff against its merge base (`GET /branches/diff`) and that merge base (`GET /branches/merge-base`) | | `/graphs/{id}/snapshot`, `/commits` | Snapshot (`internal_schema_version`, `graph_manifest_version`, and the datasets of one captured graph version), history, per-commit changes, and the unpublished commits of a branch (`GET /commits?unpublished=true`, `POST /commits/{commit_id}/publish`, `DELETE /commits/{commit_id}`) | -| `/graphs/{id}/changes` | Poll a branch feed or establish a baseline | +| `/graphs/{id}/changes`, `/diff` | Poll a branch feed, establish a baseline, or diff two points (branches or commit ids) | | `/graphs/{id}/schema` | Show the accepted schema | | `/graphs/{id}/export` | Stream a branch snapshot as JSONL | @@ -332,11 +332,11 @@ them. `PUT` and `DELETE` replace or clear the cell; see [Blob values](../blobs.m ## Changes and baselines `GET /graphs/{id}/commits/{commit_id}/changes` compares a commit with its first -parent. `GET /graphs/{id}/changes` polls complete branch commits; the terminal -response supplies its durable cursor. Unreadable history returns `410 change_feed_gap`. -`POST /graphs/{id}/changes/baseline` streams entities, then a snapshot commit and -resume cursor. See [Changes and Change Feeds](../branching/changes.md) for -pagination, checkpointing and recovery. +parent, `GET /graphs/{id}/diff?from=&to=` any two points, and `/branches/diff` +a branch with its merge base. `GET /graphs/{id}/changes` polls branch commits +and returns a durable cursor; unreadable history returns `410 change_feed_gap`. +`POST /graphs/{id}/changes/baseline` streams entities, a snapshot commit and a +resume cursor ([Changes, Diffs and Change Feeds](../branching/changes.md)). ## Errors and retries diff --git a/openapi.json b/openapi.json index 8cd64cf10..b621e3a6d 100644 --- a/openapi.json +++ b/openapi.json @@ -3089,13 +3089,363 @@ ] } }, + "/graphs/{graph_id}/branches/diff": { + "get": { + "tags": [ + "changes" + ], + "summary": "What a branch changed since its merge base with a target.", + "description": "The merge base is the commit `POST /branches/merge` would classify\nagainst; it and the source head are captured together. Changes the target\nmade after the base do not appear. Paged like `GET /diff`.", + "operationId": "cluster_getBranchDiff", + "parameters": [ + { + "name": "graph_id", + "in": "path", + "description": "Graph id to route the request to.", + "required": true, + "schema": { + "type": "string" + } + }, + { + "name": "source", + "in": "query", + "description": "Branch whose changes are reported.", + "required": true, + "schema": { + "type": "string" + } + }, + { + "name": "target", + "in": "query", + "description": "Branch the source is compared against. Defaults to `main`.", + "required": false, + "schema": { + "type": "string" + } + }, + { + "name": "page_token", + "in": "query", + "description": "Opaque continuation from the preceding page of this response.", + "required": false, + "schema": { + "type": "string" + } + }, + { + "name": "limit", + "in": "query", + "description": "Maximum changes per page, as for `GET /diff`.", + "required": false, + "schema": { + "type": "integer", + "minimum": 0 + } + }, + { + "name": "kind", + "in": "query", + "description": "Repeatable filter: node | edge.", + "required": false, + "schema": { + "type": "array", + "items": { + "$ref": "#/components/schemas/EntityKindOutput" + } + } + }, + { + "name": "type", + "in": "query", + "description": "Repeatable filter: accepted-schema type name.", + "required": false, + "schema": { + "type": "array", + "items": { + "type": "string" + } + } + }, + { + "name": "op", + "in": "query", + "description": "Repeatable filter: insert | update | delete.", + "required": false, + "schema": { + "type": "array", + "items": { + "$ref": "#/components/schemas/ChangeOpOutput" + } + } + }, + { + "name": "omnigraph-http-api", + "in": "header", + "description": "exactly one omnigraph-http-api header with value 0.13 is required", + "required": true, + "schema": { + "type": "string", + "enum": [ + "0.13" + ] + } + } + ], + "responses": { + "200": { + "description": "What `source` changed since its merge base with `target`: the net entity changes from the base to the source head", + "headers": { + "omnigraph-http-api": { + "schema": { + "type": "string", + "enum": [ + "0.13" + ] + }, + "description": "Supported OmniGraph HTTP API contract." + } + }, + "content": { + "application/json": { + "schema": { + "$ref": "#/components/schemas/DiffOutput" + } + } + } + }, + "400": { + "description": "A missing source, an invalid filter or limit, or a rejected page token; api_contract_mismatch when the required API contract header is missing, duplicated or unsupported", + "headers": { + "omnigraph-http-api": { + "schema": { + "type": "string", + "enum": [ + "0.13" + ] + }, + "description": "Supported OmniGraph HTTP API contract." + } + }, + "content": { + "application/json": { + "schema": { + "$ref": "#/components/schemas/ChangeErrorOutput" + } + } + } + }, + "401": { + "description": "Unauthorized", + "headers": { + "omnigraph-http-api": { + "schema": { + "type": "string", + "enum": [ + "0.13" + ] + }, + "description": "Supported OmniGraph HTTP API contract." + } + }, + "content": { + "application/json": { + "schema": { + "$ref": "#/components/schemas/ChangeErrorOutput" + } + } + } + }, + "403": { + "description": "The actor cannot read one of the branches", + "headers": { + "omnigraph-http-api": { + "schema": { + "type": "string", + "enum": [ + "0.13" + ] + }, + "description": "Supported OmniGraph HTTP API contract." + } + }, + "content": { + "application/json": { + "schema": { + "$ref": "#/components/schemas/ChangeErrorOutput" + } + } + } + }, + "404": { + "description": "A branch is not found", + "headers": { + "omnigraph-http-api": { + "schema": { + "type": "string", + "enum": [ + "0.13" + ] + }, + "description": "Supported OmniGraph HTTP API contract." + } + }, + "content": { + "application/json": { + "schema": { + "$ref": "#/components/schemas/ChangeErrorOutput" + } + } + } + }, + "409": { + "description": "The base and the source head cross a schema boundary; see change_diff_refusal", + "headers": { + "omnigraph-http-api": { + "schema": { + "type": "string", + "enum": [ + "0.13" + ] + }, + "description": "Supported OmniGraph HTTP API contract." + } + }, + "content": { + "application/json": { + "schema": { + "$ref": "#/components/schemas/ChangeErrorOutput" + } + } + } + }, + "410": { + "description": "Retained history for the base is no longer readable; see change_feed_gap", + "headers": { + "omnigraph-http-api": { + "schema": { + "type": "string", + "enum": [ + "0.13" + ] + }, + "description": "Supported OmniGraph HTTP API contract." + } + }, + "content": { + "application/json": { + "schema": { + "$ref": "#/components/schemas/ChangeErrorOutput" + } + } + } + }, + "413": { + "description": "Requested limit exceeds the public change ceiling", + "headers": { + "omnigraph-http-api": { + "schema": { + "type": "string", + "enum": [ + "0.13" + ] + }, + "description": "Supported OmniGraph HTTP API contract." + } + }, + "content": { + "application/json": { + "schema": { + "$ref": "#/components/schemas/ChangeErrorOutput" + } + } + } + }, + "429": { + "description": "Server observer, ingress or operation admission capacity exhausted; honor Retry-After", + "headers": { + "Retry-After": { + "schema": { + "type": "string" + }, + "description": "Minimum delay before a caller-bounded retry of this refused request; not a scheduled server retry." + }, + "omnigraph-http-api": { + "schema": { + "type": "string", + "enum": [ + "0.13" + ] + }, + "description": "Supported OmniGraph HTTP API contract." + } + }, + "content": { + "application/json": { + "schema": { + "$ref": "#/components/schemas/ChangeErrorOutput" + } + } + } + }, + "500": { + "description": "Internal failure while reading changes", + "headers": { + "omnigraph-http-api": { + "schema": { + "type": "string", + "enum": [ + "0.13" + ] + }, + "description": "Supported OmniGraph HTTP API contract." + } + }, + "content": { + "application/json": { + "schema": { + "$ref": "#/components/schemas/ChangeErrorOutput" + } + } + } + }, + "503": { + "description": "Recovery required before changes can be read; Known graph unavailable (graph_unavailable) or server operation admission closed; reconcile any earlier write before retrying", + "headers": { + "omnigraph-http-api": { + "schema": { + "type": "string", + "enum": [ + "0.13" + ] + }, + "description": "Supported OmniGraph HTTP API contract." + } + }, + "content": { + "application/json": { + "schema": { + "$ref": "#/components/schemas/ChangeErrorOutput" + } + } + } + } + }, + "security": [ + { + "bearer_token": [] + } + ] + } + }, "/graphs/{graph_id}/branches/merge": { "post": { "tags": [ "branches" ], "summary": "Merge one branch into another.", - "description": "Merges `source` into `target` (defaults to `main`). Outcome is one of\n`already_up_to_date`, `fast_forward`, or `merged`. Returns 409 with the\nlist of conflicts if the merge cannot be completed; the target is left\nunchanged in that case. **Destructive** to `target` on success. `commit`\ncarries this merge's own target publication, including for a fast-forward;\nan already-up-to-date merge returns `commit: null`.\n\nWith `delete_branch: true` the source branch is deleted after a successful\nmerge, under its own `branch_delete` policy check. The merge is durable by\nthen, so a deletion refusal or failure never fails the request; it is\nreported via `branch_deleted: false` + `branch_delete_error_details`.\n\nThe GQ statement `branch merge` on `POST /mutate` runs the same body\n(without the deletion composition) and answers the same 409 on conflict.", + "description": "Merges `source` into `target` (defaults to `main`). Outcome is one of\n`already_up_to_date`, `fast_forward`, or `merged`. Returns 409 with the\nlist of conflicts if the merge cannot be completed; the target is left\nunchanged in that case. **Destructive** to `target` on success. `commit`\ncarries this merge's own target publication, including for a fast-forward;\nan already-up-to-date merge returns `commit: null`.\n\nWith `dry_run: true` the merge stops after classification and validation:\nthe response reports the outcome it would have now (`commit: null`,\n`dry_run: true`), a conflict answers the same 409, and nothing is written.\nA dry run is a read of both branches and needs no `branch_merge` grant.\n\nWith `delete_branch: true` the source branch is deleted after a successful\nmerge, under its own `branch_delete` policy check. The merge is durable by\nthen, so a deletion refusal or failure never fails the request; it is\nreported via `branch_deleted: false` + `branch_delete_error_details`.\n\nThe GQ statement `branch merge` on `POST /mutate` runs the same body\n(without the deletion composition) and answers the same 409 on conflict.", "operationId": "cluster_mergeBranches", "parameters": [ { @@ -3302,12 +3652,244 @@ "429": { "description": "Per-actor admission cap exceeded; honor `Retry-After` header; Server observer, ingress or operation admission capacity exhausted; honor Retry-After", "headers": { - "Retry-After": { - "schema": { - "type": "string" - }, - "description": "Minimum delay before a caller-bounded retry of this refused request; not a scheduled server retry." - }, + "Retry-After": { + "schema": { + "type": "string" + }, + "description": "Minimum delay before a caller-bounded retry of this refused request; not a scheduled server retry." + }, + "omnigraph-http-api": { + "schema": { + "type": "string", + "enum": [ + "0.13" + ] + }, + "description": "Supported OmniGraph HTTP API contract." + } + }, + "content": { + "application/json": { + "schema": { + "$ref": "#/components/schemas/ErrorOutput" + } + } + } + }, + "503": { + "description": "An overlapping durable recovery intent must be resolved before retry; Known graph unavailable (graph_unavailable) or server operation admission closed; reconcile any earlier write before retrying", + "headers": { + "omnigraph-http-api": { + "schema": { + "type": "string", + "enum": [ + "0.13" + ] + }, + "description": "Supported OmniGraph HTTP API contract." + } + }, + "content": { + "application/json": { + "schema": { + "$ref": "#/components/schemas/ErrorOutput" + } + } + } + } + }, + "security": [ + { + "bearer_token": [] + } + ] + } + }, + "/graphs/{graph_id}/branches/merge-base": { + "get": { + "tags": [ + "changes" + ], + "summary": "The merge base of two branches.", + "description": "Reads lineage only, by the rule `POST /branches/merge` uses.", + "operationId": "cluster_getMergeBase", + "parameters": [ + { + "name": "graph_id", + "in": "path", + "description": "Graph id to route the request to.", + "required": true, + "schema": { + "type": "string" + } + }, + { + "name": "source", + "in": "query", + "description": "Branch that would be merged.", + "required": true, + "schema": { + "type": "string" + } + }, + { + "name": "target", + "in": "query", + "description": "Branch that would receive the merge. Defaults to `main`.", + "required": false, + "schema": { + "type": "string" + } + }, + { + "name": "omnigraph-http-api", + "in": "header", + "description": "exactly one omnigraph-http-api header with value 0.13 is required", + "required": true, + "schema": { + "type": "string", + "enum": [ + "0.13" + ] + } + } + ], + "responses": { + "200": { + "description": "The commit a merge of `source` into `target` would classify against, with the two heads", + "headers": { + "omnigraph-http-api": { + "schema": { + "type": "string", + "enum": [ + "0.13" + ] + }, + "description": "Supported OmniGraph HTTP API contract." + } + }, + "content": { + "application/json": { + "schema": { + "$ref": "#/components/schemas/MergeBaseOutput" + } + } + } + }, + "400": { + "description": "A missing or unknown parameter; api_contract_mismatch when the required API contract header is missing, duplicated or unsupported", + "headers": { + "omnigraph-http-api": { + "schema": { + "type": "string", + "enum": [ + "0.13" + ] + }, + "description": "Supported OmniGraph HTTP API contract." + } + }, + "content": { + "application/json": { + "schema": { + "$ref": "#/components/schemas/ChangeErrorOutput" + } + } + } + }, + "401": { + "description": "Unauthorized", + "headers": { + "omnigraph-http-api": { + "schema": { + "type": "string", + "enum": [ + "0.13" + ] + }, + "description": "Supported OmniGraph HTTP API contract." + } + }, + "content": { + "application/json": { + "schema": { + "$ref": "#/components/schemas/ChangeErrorOutput" + } + } + } + }, + "403": { + "description": "The actor cannot read one of the branches", + "headers": { + "omnigraph-http-api": { + "schema": { + "type": "string", + "enum": [ + "0.13" + ] + }, + "description": "Supported OmniGraph HTTP API contract." + } + }, + "content": { + "application/json": { + "schema": { + "$ref": "#/components/schemas/ChangeErrorOutput" + } + } + } + }, + "404": { + "description": "A branch is not found", + "headers": { + "omnigraph-http-api": { + "schema": { + "type": "string", + "enum": [ + "0.13" + ] + }, + "description": "Supported OmniGraph HTTP API contract." + } + }, + "content": { + "application/json": { + "schema": { + "$ref": "#/components/schemas/ChangeErrorOutput" + } + } + } + }, + "429": { + "description": "Server observer, ingress or operation admission capacity exhausted; honor Retry-After", + "headers": { + "Retry-After": { + "schema": { + "type": "string" + }, + "description": "Minimum delay before a caller-bounded retry of this refused request; not a scheduled server retry." + }, + "omnigraph-http-api": { + "schema": { + "type": "string", + "enum": [ + "0.13" + ] + }, + "description": "Supported OmniGraph HTTP API contract." + } + }, + "content": { + "application/json": { + "schema": { + "$ref": "#/components/schemas/ChangeErrorOutput" + } + } + } + }, + "500": { + "description": "Internal failure while reading lineage", + "headers": { "omnigraph-http-api": { "schema": { "type": "string", @@ -3321,13 +3903,13 @@ "content": { "application/json": { "schema": { - "$ref": "#/components/schemas/ErrorOutput" + "$ref": "#/components/schemas/ChangeErrorOutput" } } } }, "503": { - "description": "An overlapping durable recovery intent must be resolved before retry; Known graph unavailable (graph_unavailable) or server operation admission closed; reconcile any earlier write before retrying", + "description": "Recovery required before lineage can be read; Known graph unavailable (graph_unavailable) or server operation admission closed; reconcile any earlier write before retrying", "headers": { "omnigraph-http-api": { "schema": { @@ -3342,7 +3924,7 @@ "content": { "application/json": { "schema": { - "$ref": "#/components/schemas/ErrorOutput" + "$ref": "#/components/schemas/ChangeErrorOutput" } } } @@ -4931,13 +5513,257 @@ "content": { "application/json": { "schema": { - "$ref": "#/components/schemas/ChangeErrorOutput" + "$ref": "#/components/schemas/ChangeErrorOutput" + } + } + } + }, + "401": { + "description": "Unauthorized", + "headers": { + "omnigraph-http-api": { + "schema": { + "type": "string", + "enum": [ + "0.13" + ] + }, + "description": "Supported OmniGraph HTTP API contract." + } + }, + "content": { + "application/json": { + "schema": { + "$ref": "#/components/schemas/ChangeErrorOutput" + } + } + } + }, + "404": { + "description": "Commit not found, or the actor cannot read the commit's branch", + "headers": { + "omnigraph-http-api": { + "schema": { + "type": "string", + "enum": [ + "0.13" + ] + }, + "description": "Supported OmniGraph HTTP API contract." + } + }, + "content": { + "application/json": { + "schema": { + "$ref": "#/components/schemas/ChangeErrorOutput" + } + } + } + }, + "409": { + "description": "Commit cannot be entity-diffed (parentless commit or schema boundary); see change_diff_refusal", + "headers": { + "omnigraph-http-api": { + "schema": { + "type": "string", + "enum": [ + "0.13" + ] + }, + "description": "Supported OmniGraph HTTP API contract." + } + }, + "content": { + "application/json": { + "schema": { + "$ref": "#/components/schemas/ChangeErrorOutput" + } + } + } + }, + "410": { + "description": "Required retained history is no longer readable; see change_feed_gap and capture a new baseline", + "headers": { + "omnigraph-http-api": { + "schema": { + "type": "string", + "enum": [ + "0.13" + ] + }, + "description": "Supported OmniGraph HTTP API contract." + } + }, + "content": { + "application/json": { + "schema": { + "$ref": "#/components/schemas/ChangeErrorOutput" + } + } + } + }, + "413": { + "description": "Requested limit exceeds the public change ceiling", + "headers": { + "omnigraph-http-api": { + "schema": { + "type": "string", + "enum": [ + "0.13" + ] + }, + "description": "Supported OmniGraph HTTP API contract." + } + }, + "content": { + "application/json": { + "schema": { + "$ref": "#/components/schemas/ChangeErrorOutput" + } + } + } + }, + "429": { + "description": "Server observer, ingress or operation admission capacity exhausted; honor Retry-After", + "headers": { + "Retry-After": { + "schema": { + "type": "string" + }, + "description": "Minimum delay before a caller-bounded retry of this refused request; not a scheduled server retry." + }, + "omnigraph-http-api": { + "schema": { + "type": "string", + "enum": [ + "0.13" + ] + }, + "description": "Supported OmniGraph HTTP API contract." + } + }, + "content": { + "application/json": { + "schema": { + "$ref": "#/components/schemas/ChangeErrorOutput" + } + } + } + }, + "500": { + "description": "Internal failure while reading changes", + "headers": { + "omnigraph-http-api": { + "schema": { + "type": "string", + "enum": [ + "0.13" + ] + }, + "description": "Supported OmniGraph HTTP API contract." + } + }, + "content": { + "application/json": { + "schema": { + "$ref": "#/components/schemas/ChangeErrorOutput" + } + } + } + }, + "503": { + "description": "Recovery required before changes can be read; Known graph unavailable (graph_unavailable) or server operation admission closed; reconcile any earlier write before retrying", + "headers": { + "omnigraph-http-api": { + "schema": { + "type": "string", + "enum": [ + "0.13" + ] + }, + "description": "Supported OmniGraph HTTP API contract." + } + }, + "content": { + "application/json": { + "schema": { + "$ref": "#/components/schemas/ChangeErrorOutput" + } + } + } + } + }, + "security": [ + { + "bearer_token": [] + } + ] + } + }, + "/graphs/{graph_id}/commits/{commit_id}/publish": { + "post": { + "tags": [ + "commits" + ], + "summary": "Publish an unpublished commit onto its recorded target branch.", + "description": "The target is the branch the commit recorded when it was written `as\ncommit`, so the request carries no body. Runs under `commit_publish` on\nthat branch; a denial answers the 404 an unknown id answers. An\nunattached snapshot (`as unattached branch`) never publishes: 409 with\ncode `unattached_cannot_publish`, after the same authorization. The GQ\nstatement `commit publish \"\"` on `POST /mutate` runs the same body.", + "operationId": "cluster_publishCommit", + "parameters": [ + { + "name": "graph_id", + "in": "path", + "description": "Graph id to route the request to.", + "required": true, + "schema": { + "type": "string" + } + }, + { + "name": "commit_id", + "in": "path", + "description": "Unpublished commit identifier", + "required": true, + "schema": { + "type": "string" + } + }, + { + "name": "omnigraph-http-api", + "in": "header", + "description": "exactly one omnigraph-http-api header with value 0.13 is required", + "required": true, + "schema": { + "type": "string", + "enum": [ + "0.13" + ] + } + } + ], + "responses": { + "200": { + "description": "The commit as published on its recorded target branch; a published id is answered as it stands", + "headers": { + "omnigraph-http-api": { + "schema": { + "type": "string", + "enum": [ + "0.13" + ] + }, + "description": "Supported OmniGraph HTTP API contract." + } + }, + "content": { + "application/json": { + "schema": { + "$ref": "#/components/schemas/CommitOutput" } } } }, - "401": { - "description": "Unauthorized", + "400": { + "description": "Invalid request; api_contract_mismatch when the required API contract header is missing, duplicated or unsupported", "headers": { "omnigraph-http-api": { "schema": { @@ -4952,13 +5778,13 @@ "content": { "application/json": { "schema": { - "$ref": "#/components/schemas/ChangeErrorOutput" + "$ref": "#/components/schemas/ErrorOutput" } } } }, - "404": { - "description": "Commit not found, or the actor cannot read the commit's branch", + "401": { + "description": "Unauthorized", "headers": { "omnigraph-http-api": { "schema": { @@ -4973,13 +5799,13 @@ "content": { "application/json": { "schema": { - "$ref": "#/components/schemas/ChangeErrorOutput" + "$ref": "#/components/schemas/ErrorOutput" } } } }, - "409": { - "description": "Commit cannot be entity-diffed (parentless commit or schema boundary); see change_diff_refusal", + "404": { + "description": "Commit not found, or its publication is not permitted", "headers": { "omnigraph-http-api": { "schema": { @@ -4994,13 +5820,13 @@ "content": { "application/json": { "schema": { - "$ref": "#/components/schemas/ChangeErrorOutput" + "$ref": "#/components/schemas/ErrorOutput" } } } }, - "410": { - "description": "Required retained history is no longer readable; see change_feed_gap and capture a new baseline", + "409": { + "description": "The target's state moved under the proposal's authority (code `conflict`), or the commit is unattached and cannot publish (code `unattached_cannot_publish`, no durable effect)", "headers": { "omnigraph-http-api": { "schema": { @@ -5015,13 +5841,13 @@ "content": { "application/json": { "schema": { - "$ref": "#/components/schemas/ChangeErrorOutput" + "$ref": "#/components/schemas/ErrorOutput" } } } }, - "413": { - "description": "Requested limit exceeds the public change ceiling", + "412": { + "description": "The target head moved past the commit's captured parent; the commit can only be dropped", "headers": { "omnigraph-http-api": { "schema": { @@ -5036,13 +5862,13 @@ "content": { "application/json": { "schema": { - "$ref": "#/components/schemas/ChangeErrorOutput" + "$ref": "#/components/schemas/ErrorOutput" } } } }, "429": { - "description": "Server observer, ingress or operation admission capacity exhausted; honor Retry-After", + "description": "Per-actor admission cap exceeded; honor `Retry-After` header; Server observer, ingress or operation admission capacity exhausted; honor Retry-After", "headers": { "Retry-After": { "schema": { @@ -5063,34 +5889,13 @@ "content": { "application/json": { "schema": { - "$ref": "#/components/schemas/ChangeErrorOutput" - } - } - } - }, - "500": { - "description": "Internal failure while reading changes", - "headers": { - "omnigraph-http-api": { - "schema": { - "type": "string", - "enum": [ - "0.13" - ] - }, - "description": "Supported OmniGraph HTTP API contract." - } - }, - "content": { - "application/json": { - "schema": { - "$ref": "#/components/schemas/ChangeErrorOutput" + "$ref": "#/components/schemas/ErrorOutput" } } } }, "503": { - "description": "Recovery required before changes can be read; Known graph unavailable (graph_unavailable) or server operation admission closed; reconcile any earlier write before retrying", + "description": "An overlapping durable recovery intent must be resolved before retry; Known graph unavailable (graph_unavailable) or server operation admission closed; reconcile any earlier write before retrying", "headers": { "omnigraph-http-api": { "schema": { @@ -5105,7 +5910,7 @@ "content": { "application/json": { "schema": { - "$ref": "#/components/schemas/ChangeErrorOutput" + "$ref": "#/components/schemas/ErrorOutput" } } } @@ -5118,14 +5923,14 @@ ] } }, - "/graphs/{graph_id}/commits/{commit_id}/publish": { - "post": { + "/graphs/{graph_id}/diff": { + "get": { "tags": [ - "commits" + "changes" ], - "summary": "Publish an unpublished commit onto its recorded target branch.", - "description": "The target is the branch the commit recorded when it was written `as\ncommit`, so the request carries no body. Runs under `commit_publish` on\nthat branch; a denial answers the 404 an unknown id answers. An\nunattached snapshot (`as unattached branch`) never publishes: 409 with\ncode `unattached_cannot_publish`, after the same authorization. The GQ\nstatement `commit publish \"\"` on `POST /mutate` runs the same body.", - "operationId": "cluster_publishCommit", + "summary": "Net entity changes between two read targets.", + "description": "`from` and `to` each name a branch (its head now) or a graph commit\n(published, an unpublished proposal, or an unattached snapshot); a string\nnaming both is refused. The first page resolves both sides to exact\ncommits and names them; a large result continues via the opaque\n`page_token`, and every page answers for those same commits.", + "operationId": "cluster_getDiff", "parameters": [ { "name": "graph_id", @@ -5137,14 +5942,78 @@ } }, { - "name": "commit_id", - "in": "path", - "description": "Unpublished commit identifier", + "name": "from", + "in": "query", + "description": "Read target the diff starts from: a branch name (its head now) or a\ngraph commit id. A string naming both is refused.", + "required": true, + "schema": { + "type": "string" + } + }, + { + "name": "to", + "in": "query", + "description": "Read target the diff ends at, spelled as `from`.", "required": true, "schema": { "type": "string" } }, + { + "name": "page_token", + "in": "query", + "description": "Opaque continuation from the preceding page of this response.", + "required": false, + "schema": { + "type": "string" + } + }, + { + "name": "limit", + "in": "query", + "description": "Maximum changes per page. Server default applies when absent; above\nthe public ceiling the request fails with 413.", + "required": false, + "schema": { + "type": "integer", + "minimum": 0 + } + }, + { + "name": "kind", + "in": "query", + "description": "Repeatable filter: node | edge.", + "required": false, + "schema": { + "type": "array", + "items": { + "$ref": "#/components/schemas/EntityKindOutput" + } + } + }, + { + "name": "type", + "in": "query", + "description": "Repeatable filter: accepted-schema type name.", + "required": false, + "schema": { + "type": "array", + "items": { + "type": "string" + } + } + }, + { + "name": "op", + "in": "query", + "description": "Repeatable filter: insert | update | delete.", + "required": false, + "schema": { + "type": "array", + "items": { + "$ref": "#/components/schemas/ChangeOpOutput" + } + } + }, { "name": "omnigraph-http-api", "in": "header", @@ -5160,7 +6029,7 @@ ], "responses": { "200": { - "description": "The commit as published on its recorded target branch; a published id is answered as it stands", + "description": "Net entity changes between the two resolved commits, in frozen (kind, type, id, op) order, without a cause", "headers": { "omnigraph-http-api": { "schema": { @@ -5175,13 +6044,13 @@ "content": { "application/json": { "schema": { - "$ref": "#/components/schemas/CommitOutput" + "$ref": "#/components/schemas/DiffOutput" } } } }, "400": { - "description": "Invalid request; api_contract_mismatch when the required API contract header is missing, duplicated or unsupported", + "description": "A missing side, an invalid filter or limit, a side naming both a branch and a commit, or a rejected page token; api_contract_mismatch when the required API contract header is missing, duplicated or unsupported", "headers": { "omnigraph-http-api": { "schema": { @@ -5196,7 +6065,7 @@ "content": { "application/json": { "schema": { - "$ref": "#/components/schemas/ErrorOutput" + "$ref": "#/components/schemas/ChangeErrorOutput" } } } @@ -5217,13 +6086,13 @@ "content": { "application/json": { "schema": { - "$ref": "#/components/schemas/ErrorOutput" + "$ref": "#/components/schemas/ChangeErrorOutput" } } } }, "404": { - "description": "Commit not found, or its publication is not permitted", + "description": "A side names no branch and no commit, or the actor cannot read the branch it resolves to", "headers": { "omnigraph-http-api": { "schema": { @@ -5238,13 +6107,13 @@ "content": { "application/json": { "schema": { - "$ref": "#/components/schemas/ErrorOutput" + "$ref": "#/components/schemas/ChangeErrorOutput" } } } }, "409": { - "description": "The target's state moved under the proposal's authority (code `conflict`), or the commit is unattached and cannot publish (code `unattached_cannot_publish`, no durable effect)", + "description": "The two sides cross a schema boundary; see change_diff_refusal", "headers": { "omnigraph-http-api": { "schema": { @@ -5259,13 +6128,13 @@ "content": { "application/json": { "schema": { - "$ref": "#/components/schemas/ErrorOutput" + "$ref": "#/components/schemas/ChangeErrorOutput" } } } }, - "412": { - "description": "The target head moved past the commit's captured parent; the commit can only be dropped", + "410": { + "description": "Retained history for an endpoint is no longer readable; see change_feed_gap", "headers": { "omnigraph-http-api": { "schema": { @@ -5280,13 +6149,34 @@ "content": { "application/json": { "schema": { - "$ref": "#/components/schemas/ErrorOutput" + "$ref": "#/components/schemas/ChangeErrorOutput" + } + } + } + }, + "413": { + "description": "Requested limit exceeds the public change ceiling", + "headers": { + "omnigraph-http-api": { + "schema": { + "type": "string", + "enum": [ + "0.13" + ] + }, + "description": "Supported OmniGraph HTTP API contract." + } + }, + "content": { + "application/json": { + "schema": { + "$ref": "#/components/schemas/ChangeErrorOutput" } } } }, "429": { - "description": "Per-actor admission cap exceeded; honor `Retry-After` header; Server observer, ingress or operation admission capacity exhausted; honor Retry-After", + "description": "Server observer, ingress or operation admission capacity exhausted; honor Retry-After", "headers": { "Retry-After": { "schema": { @@ -5307,13 +6197,34 @@ "content": { "application/json": { "schema": { - "$ref": "#/components/schemas/ErrorOutput" + "$ref": "#/components/schemas/ChangeErrorOutput" + } + } + } + }, + "500": { + "description": "Internal failure while reading changes", + "headers": { + "omnigraph-http-api": { + "schema": { + "type": "string", + "enum": [ + "0.13" + ] + }, + "description": "Supported OmniGraph HTTP API contract." + } + }, + "content": { + "application/json": { + "schema": { + "$ref": "#/components/schemas/ChangeErrorOutput" } } } }, "503": { - "description": "An overlapping durable recovery intent must be resolved before retry; Known graph unavailable (graph_unavailable) or server operation admission closed; reconcile any earlier write before retrying", + "description": "Recovery required before changes can be read; Known graph unavailable (graph_unavailable) or server operation admission closed; reconcile any earlier write before retrying", "headers": { "omnigraph-http-api": { "schema": { @@ -5328,7 +6239,7 @@ "content": { "application/json": { "schema": { - "$ref": "#/components/schemas/ErrorOutput" + "$ref": "#/components/schemas/ChangeErrorOutput" } } } @@ -8670,6 +9581,10 @@ } ] }, + "dry_run": { + "type": "boolean", + "description": "`true` for a dry run: `outcome` is what the merge would do now, and\n`commit` is `null` because nothing was written." + }, "outcome": { "$ref": "#/components/schemas/BranchMergeOutcome" }, @@ -8695,6 +9610,10 @@ "type": "boolean", "description": "Delete the source branch after a successful merge. The deletion runs\nunder its own `branch_delete` policy check; a refusal or failure is\nreported via `branch_deleted` / `branch_delete_error_details` on the response\nand never fails the already-landed merge." }, + "dry_run": { + "type": "boolean", + "description": "Run the merge through classification and validation and stop: the\nresponse reports the outcome the merge would have now, a conflict\nfails with the same `409` a merge returns, and nothing is written.\nA read of both branches; refused together with `as_commit` and\n`delete_branch`." + }, "settings": { "oneOf": [ { @@ -9485,6 +10404,45 @@ } } }, + "DiffOutput": { + "type": "object", + "description": "One bounded page of a graph diff: the net entity changes between two\nresolved commits, in the commit diff's entity shape and order. There is no\ncause: a range has no single author.", + "required": [ + "from_commit_id", + "to_commit_id", + "changes" + ], + "properties": { + "base_commit_id": { + "type": [ + "string", + "null" + ], + "description": "The merge base, present only for a branch diff." + }, + "changes": { + "type": "array", + "items": { + "$ref": "#/components/schemas/EntityChangeOutput" + } + }, + "from_commit_id": { + "type": "string", + "description": "The commit the `from` side resolved to; for a branch diff, the merge\nbase." + }, + "next_page_token": { + "type": [ + "string", + "null" + ], + "description": "Continue THIS bounded response; later pages answer for the same two\ncommits. Absent on the final page." + }, + "to_commit_id": { + "type": "string", + "description": "The commit the `to` side resolved to; for a branch diff, the source\nhead." + } + } + }, "EntityChangeOutput": { "type": "object", "description": "One entity change. Cause is stated once on the enclosing block, never here.\nAn insert carries only `after`, an update exact `before` and `after`, a\ndelete only `before`.", @@ -10219,6 +11177,34 @@ "merge" ] }, + "MergeBaseOutput": { + "type": "object", + "description": "The commit a merge of `source` into `target` would classify against, and\nthe two heads it was computed from.", + "required": [ + "source", + "target", + "source_commit_id", + "target_commit_id", + "merge_base" + ], + "properties": { + "merge_base": { + "$ref": "#/components/schemas/CommitOutput" + }, + "source": { + "type": "string" + }, + "source_commit_id": { + "type": "string" + }, + "target": { + "type": "string" + }, + "target_commit_id": { + "type": "string" + } + } + }, "MergeConflictKindOutput": { "type": "string", "enum": [ From 5169758a661cf21b090085e00afe0e4ab590ae66 Mon Sep 17 00:00:00 2001 From: Ragnor Comerford Date: Mon, 12 Oct 2026 19:19:16 -0700 Subject: [PATCH 04/11] rfc: limit the merge-base feed recipe and authorize both branches of a branch diff Address the review of the graph diff amendment. - A merge base is a valid feed start only on the source's first-parent chain. After the target was merged into the source, the base is the imported target commit and the feed refuses it; point at branch diff for the net change and add the shape to the D0 gates. - The branch diff and merge base require read on both named branches, continuations included: their endpoints, the base and the source head, can both be readable while the target is not. Add the D0 gate. - Defer GQ exposure of the diff surfaces to RFC 0056 rather than reject it. --- docs/rfcs/0030-cdc-time-travel.md | 68 +++++++++++++++++++++++++++---- 1 file changed, 59 insertions(+), 9 deletions(-) diff --git a/docs/rfcs/0030-cdc-time-travel.md b/docs/rfcs/0030-cdc-time-travel.md index b645a2a8d..267cccddf 100644 --- a/docs/rfcs/0030-cdc-time-travel.md +++ b/docs/rfcs/0030-cdc-time-travel.md @@ -709,10 +709,16 @@ resolved endpoints, the filter digest and the position; it is never accepted as a feed cursor. Filters are the §3.3 kind, type and operation filters. An unreadable endpoint maps to the §6 typed outcomes. -Authorization is `read` on the branch each endpoint resolves to. A commit -endpoint authorizes against its authored branch, or against the recorded -target of a proposal or snapshot, and a forbidden endpoint is reported as not -found, as the commit diff does. +Authorization of the two-point form is `read` on the branch each endpoint +resolves to. A commit endpoint authorizes against its authored branch, or +against the recorded target of a proposal or snapshot, and a forbidden +endpoint is reported as not found, as the commit diff does. The branch form +requires `read` on both named branches, source and target, on every request, +a continuation included, before it reads lineage. Its endpoints are the base +and the source head, which can both be readable while the target is not, and +the base and the result would then reveal whether the target has incorporated +the source. A named branch the actor cannot read is refused as any branch +route refuses it. Cost: a type lifetime whose `to` pin is one proving transaction past its `from` pin may use the §14 candidate pruning. Every other changed lifetime takes the @@ -740,9 +746,19 @@ or delete. `branch merge-base` reports the commit a merge of the source into the target would classify against, by the rule the -[merge guide](../user/branching/merge.md) documents. It reads lineage only. It -also lets a caller replay a branch's own commits through the feed with -`--start after:`. +[merge guide](../user/branching/merge.md) documents. It reads lineage only, and +like the branch diff it requires `read` on both branches (§8.2). + +When the base lies on the source's first-parent chain, as it does for a source +that has never merged the target in, a caller can replay the source's commits +since the base through the feed with `--start after:`. A merge commit in +that range still carries the changes it imported in its block (§3.2), so the +replay is the source's history, not only its own edits. Once the target has +been merged into the source, the base is the target commit that merge imported, +which the source's first-parent chain does not hold, and the feed refuses it as +a start (§5.3). The feed does not walk second parents, because that would +change its replay contract; `branch diff` reports the net change from any +base. `branch merge --dry-run` runs the merge's base resolution, three-way classification and constraint validation, reports the outcome @@ -773,8 +789,11 @@ answers whether the merge would succeed, without a write. The diff commands take the existing `--kind`, `--type`, `--op`, `--limit`, `--page-token` and `--json` options and paginate automatically, as `commit changes` does. `--into`, `source` and `target` follow `branch merge`. -These commands are not GQ statements, because their results are paged change -records rather than query rows. +Exposing them as GQ statements is deferred, not rejected. +[RFC 0056](https://github.com/ModernRelay/omnigraph/pull/670) proposes carrying +`commit changes` and `changes poll` as entity-change rows with an optional page +object, and the graph diff should take whatever envelope that RFC settles +rather than decide a second one here. D1–D3 ship through the CLI and HTTP. ### 8.6 Retiring the table-shaped diff @@ -886,6 +905,11 @@ server `integration/data_routes.rs` and `integration/openapi.rs`; CLI reported base equals branch merge's base on the merge truth-table fixtures, ties included; after the target was merged into the source, only the source's own net changes since the newest base appear. +- Merge base and the feed: after the target was merged into the source, the + base is off the source's first-parent chain and the source's feed refuses + `after:` with `ChangeCursorRejected`, while the branch diff still + reports the source's net change. Owner: the existing merge-and-feed test in + `integration/changes.rs`. - Endpoints: a branch, a published commit, a proposal and an unattached snapshot all resolve; a string naming both a branch and a commit is refused; pages after a concurrent commit on either branch still answer for the pinned @@ -898,6 +922,9 @@ server `integration/data_routes.rs` and `integration/openapi.rs`; CLI is written; an actor with `read` alone can run it and still cannot merge. - Authorization and not-found parity with the commit diff; the vocabulary gates cover the new DTOs; the parity matrix covers local and remote runs. +- Branch-form authorization: an actor who can read the base and the source but + not the target is refused the branch diff, on its first page and on a + continuation, and the merge base. ### Cost evidence @@ -1363,3 +1390,26 @@ Amendments before 2026-10-12 are recorded in §14 and in the dispositions above. `query --snapshot` on the proposal refuse with "expired and awaits cleanup", `commit publish` fails its head precondition, and `commit drop` succeeds. +- **2026-10-12: review of the graph diff amendment.** Three corrections from + the review of ModernRelay/omnigraph#964: + - The merge base is not always a valid feed start. Once the target has been + merged into the source, the base is the target commit that merge + imported, off the source's first-parent chain, and the feed refuses it. + The review reproduced the refusal with `CommitGraph::merge_base` and the + feed in the existing merge-and-feed test. §8.4 now limits the recipe to a + base on that chain and points at `branch diff` for the net change, and D0 + gains the shape. + - The branch form's endpoints are the base and the source head, so + authorizing them alone lets an actor who cannot read the target learn + whether it has incorporated the source. §8.2 now requires `read` on both + named branches, continuations included, and D0 gains the gate. + - GQ exposure of the diff surfaces is deferred to RFC 0056 + (ModernRelay/omnigraph#670), not rejected (§8.5). + + This entry supersedes these sentences: + - §8.2: "Authorization is `read` on the branch each endpoint resolves to." + It now covers the two-point form only. + - §8.4: "It also lets a caller replay a branch's own commits through the + feed with `--start after:`." + - §8.5: "These commands are not GQ statements, because their results are + paged change records rather than query rows." From 3bb077542571eba1a038d7fe40cf389406f4b3b7 Mon Sep 17 00:00:00 2001 From: Ragnor Comerford Date: Mon, 12 Oct 2026 19:28:56 -0700 Subject: [PATCH 05/11] fix(changes): document the merge-base feed limit and test target-denied branch review Follow the RFC 0030 review corrections: - The user guide's merge-base recipe now holds only while the source has never merged the target in; after that the base is off the source's first-parent chain, the feed refuses it, and branch diff gives the net change. The existing merge-and-feed test asserts that shape. - The branch diff and merge base already authorize read on source and target. A new policy test pins the case where both diff endpoints are readable but the target is not, on the first page, a continuation and the merge base. --- .../tests/integration/auth_policy.rs | 91 +++++++++++++++++++ crates/omnigraph/tests/integration/changes.rs | 51 +++++++++++ docs/user/branching/changes.md | 13 ++- 3 files changed, 152 insertions(+), 3 deletions(-) diff --git a/crates/omnigraph-server/tests/integration/auth_policy.rs b/crates/omnigraph-server/tests/integration/auth_policy.rs index f0a92a0e0..718810189 100644 --- a/crates/omnigraph-server/tests/integration/auth_policy.rs +++ b/crates/omnigraph-server/tests/integration/auth_policy.rs @@ -1796,6 +1796,97 @@ async fn policy_merge_dry_run_needs_read_and_not_branch_merge() { assert_eq!(status, StatusCode::FORBIDDEN, "{body}"); } +/// A branch review's endpoints are the merge base and the source head, which +/// can both be readable while the target is not; the base and the result +/// would then reveal whether the target has incorporated the source. So the +/// branch diff, on every page, and the merge base need `read` on the target +/// as well as on the source. +#[tokio::test(flavor = "multi_thread")] +async fn policy_branch_review_needs_read_on_the_target() { + let temp = init_loaded_graph().await; + let graph = graph_path(temp.path()); + let db = session(Omnigraph::open(graph.to_str().unwrap()).await.unwrap()); + for branch in ["feature", "private"] { + db.branch_create_from(ReadTarget::branch("main"), branch) + .await + .unwrap(); + } + db.load_with_receipt( + "feature", + concat!( + r#"{"type":"Person","data":{"name":"feature-one","age":40}}"#, + "\n", + r#"{"type":"Person","data":{"name":"feature-two","age":41}}"#, + ), + omnigraph::loader::LoadMode::Merge, + ) + .await + .unwrap(); + drop(db); + // act-bruno reads main and feature but not private; act-ada reads all. + let policy_path = temp.path().join("policy.yaml"); + fs::write( + &policy_path, + "version: 1\ngroups:\n team: [act-bruno]\n admins: [act-ada]\nprotected_branches: [main, feature]\nrules:\n - id: team-read\n allow: {actors: {group: team}, actions: [read], branch_scope: protected}\n - id: admin-read\n allow: {actors: {group: admins}, actions: [read], branch_scope: any}\n", + ) + .unwrap(); + let app = build_app( + AppState::open_with_bearer_tokens_and_policy( + graph.to_string_lossy().to_string(), + vec![ + ("act-bruno".to_string(), "team-token".to_string()), + ("act-ada".to_string(), "admin-token".to_string()), + ], + Some(&policy_path), + ) + .await + .unwrap(), + ); + let get = |uri: &str, token: &str| { + Request::builder() + .header(HTTP_API_CONTRACT_HEADER, HTTP_API_CONTRACT) + .uri(g(uri)) + .method(Method::GET) + .header("authorization", format!("Bearer {token}")) + .body(Body::empty()) + .unwrap() + }; + + // Against main the base and feature's head are readable, and so is the + // target. + let (status, body) = json_response( + &app, + get("/branches/diff?source=feature&target=main", "team-token"), + ) + .await; + assert_eq!(status, StatusCode::OK, "{body}"); + // Against private the endpoints are the same base on main and the same + // feature head; only the target is unreadable. + let (status, first) = json_response( + &app, + get( + "/branches/diff?source=feature&target=private&limit=1", + "admin-token", + ), + ) + .await; + assert_eq!(status, StatusCode::OK, "{first}"); + assert_eq!(first["from_commit_id"], body["from_commit_id"]); + assert_eq!(first["to_commit_id"], body["to_commit_id"]); + let token = first["next_page_token"] + .as_str() + .expect("two changes page at one") + .to_string(); + for uri in [ + "/branches/diff?source=feature&target=private".to_string(), + format!("/branches/diff?source=feature&target=private&limit=1&page_token={token}"), + "/branches/merge-base?source=feature&target=private".to_string(), + ] { + let (status, body) = json_response(&app, get(&uri, "team-token")).await; + assert_eq!(status, StatusCode::FORBIDDEN, "{uri}: {body}"); + } +} + #[tokio::test(flavor = "multi_thread")] async fn policy_blocks_change_on_protected_main_but_allows_unprotected_branch() { let temp = init_graph_with_schema_and_data( diff --git a/crates/omnigraph/tests/integration/changes.rs b/crates/omnigraph/tests/integration/changes.rs index 90645cec6..ad657870d 100644 --- a/crates/omnigraph/tests/integration/changes.rs +++ b/crates/omnigraph/tests/integration/changes.rs @@ -3248,6 +3248,57 @@ async fn change_feed_merge_commit_is_first_parent_relative_with_merged_parent_on merge_block.changes.iter().all(|change| change.id != "Bob"), "main's own divergence is not re-attributed to the merge commit" ); + + // Main merged side in, so the merge base of main into side is the side + // head that merge imported, which main's first-parent chain does not + // hold: main's feed refuses it as a start, and the branch diff still + // reports main's net change since it. + let base = main.merge_base("main", "side").await.unwrap(); + assert_eq!( + merge_block.cause.merged_parent_commit_id.as_deref(), + Some(base.merge_base.graph_commit_id.as_str()), + "the base is the merged parent" + ); + let refused = main + .poll_change_feed(feed_request( + None, + ChangeFeedPosition::Start(ChangeFeedStart::AfterCommit( + base.merge_base.graph_commit_id.clone(), + )), + )) + .await + .unwrap_err(); + assert!( + matches!( + &refused, + omnigraph::error::OmniError::ChangeCursorRejected { reason } + if reason.contains("first-parent chain") + ), + "{refused:?}" + ); + let review = graph_diff_all( + &main, + omnigraph::changes::GraphDiffSides::Branch { + source: "main".to_string(), + target: "side".to_string(), + }, + None, + ) + .await + .unwrap(); + assert_eq!( + review.base_commit_id.as_deref(), + Some(base.merge_base.graph_commit_id.as_str()) + ); + assert_eq!( + diff_ops(&review.changes), + vec![( + "Person".to_string(), + "Bob".to_string(), + omnigraph::changes::ChangeOpKind::Update + )], + "only main's own divergence is main's change since the base" + ); } #[tokio::test] diff --git a/docs/user/branching/changes.md b/docs/user/branching/changes.md index b7aa4316f..2a671d7ae 100644 --- a/docs/user/branching/changes.md +++ b/docs/user/branching/changes.md @@ -87,8 +87,13 @@ omnigraph branch merge-base feature --into main --store graph.omni --json head of `feature`. Changes `main` made after the merge base do not appear. The response also names the base as `base_commit_id`. `branch merge-base` prints that commit alone, the one a merge of `feature` into `main` would classify -against. It is useful for replaying the branch's own commits with -`changes poll --branch feature --start after:`. +against. While `feature` has never merged `main` in, the base is on its +first-parent history, and `changes poll --branch feature --start +after:` replays its commits since then; a merge commit in that +range still carries the changes it brought in. Once `main` has been merged into +`feature`, the base is the `main` commit that merge brought in, which +`feature`'s feed refuses as a start; `branch diff` gives the net change from +any base. The HTTP routes are `GET /graphs/{graph_id}/branches/diff?source=&target=` and `GET /graphs/{graph_id}/branches/merge-base?source=&target=`. To see whether @@ -173,7 +178,9 @@ Baseline kind and type filters select the snapshot contents. An operation filter applies only to the resumed feed. Baselines require the `export` policy action; commit changes, diffs and feed polling require `read`. A diff needs `read` on the branch each side resolves to; a side the actor cannot read -answers `404` exactly as a side that names nothing. +answers `404` exactly as a side that names nothing. `branch diff` and +`branch merge-base` need `read` on both branches, every page included, and +answer `403` otherwise. The CLI's durable `--out` installation is available on POSIX platforms. The file contains only snapshot entity records, not the terminal handshake. The CLI From 52a881c2793a98e8363d8b0572946adcea126ec7 Mon Sep 17 00:00:00 2001 From: Ragnor Comerford Date: Mon, 12 Oct 2026 20:18:43 -0700 Subject: [PATCH 06/11] perf(changes): resolve diff sides once, skip commit lookups for branch names, use full CLI pages - A diff side whose spelling is neither a ULID nor an hb1 address cannot be a commit id, so classification no longer looks it up as one. The lookup's miss path checked out every live branch's __manifest, so a diff between two branch names cost about 22 requests per live branch. A cost gate pins the reads flat in idle branches. - Omnigraph::prepare_diff_page resolves a request once and reports the branch each two-point side reads; the server authorizes those and runs the page with run_diff_page, so authorization and the read share one resolution. diff_read_target_branch is removed. - When the CLI follows every page of commit changes, changes poll, diff or branch diff, it asks for 8,192 changes per page unless --limit is set: every page restarts its comparison, so fewer pages cost less. --- ...change-pages-at-the-maximum.performance.md | 9 + crates/omnigraph-cli/src/cli.rs | 12 +- crates/omnigraph-cli/src/main.rs | 30 ++- crates/omnigraph-server/src/handlers.rs | 45 ++-- crates/omnigraph/src/db/mod.rs | 6 +- crates/omnigraph/src/db/omnigraph.rs | 219 +++++++++++++----- crates/omnigraph/tests/forbidden_apis.rs | 3 +- crates/omnigraph/tests/integration/changes.rs | 39 +++- .../tests/integration/changes_cost.rs | 67 ++++++ docs/user/branching/changes.md | 12 +- 10 files changed, 343 insertions(+), 99 deletions(-) create mode 100644 changelog.d/cli-change-pages-at-the-maximum.performance.md diff --git a/changelog.d/cli-change-pages-at-the-maximum.performance.md b/changelog.d/cli-change-pages-at-the-maximum.performance.md new file mode 100644 index 000000000..8be84600d --- /dev/null +++ b/changelog.d/cli-change-pages-at-the-maximum.performance.md @@ -0,0 +1,9 @@ +- **Fewer, larger change pages from the CLI.** When `commit changes` or + `changes poll` follows every page itself, it now asks for 8,192 changes + per page instead of the 1,000 the HTTP routes default to. Every page + resumes its comparison where the last one stopped, so a large result + costs fewer requests and less time; the output is unchanged. `--limit` + still sets the size, and a single `--page-token` fetch still defaults to + 1,000. See [changes and change feeds][cli-change-pages]. + +[cli-change-pages]: ../docs/user/branching/changes.md#inspect-one-commit diff --git a/crates/omnigraph-cli/src/cli.rs b/crates/omnigraph-cli/src/cli.rs index 36a27f81c..6202c9fa8 100644 --- a/crates/omnigraph-cli/src/cli.rs +++ b/crates/omnigraph-cli/src/cli.rs @@ -265,7 +265,8 @@ pub(crate) enum Command { /// Read target the diff ends at #[arg(long)] to: String, - /// Changes per page (the server clamps at its public ceiling) + /// Changes per page, at most 8192: 8192 when the CLI follows every + /// page, 1000 for one `--page-token` fetch #[arg(long)] limit: Option, /// Fetch exactly one page starting at this opaque token @@ -927,7 +928,8 @@ pub(crate) enum BranchCommand { /// Branch the source is compared against (default main) #[arg(long)] into: Option, - /// Changes per page (the server clamps at its public ceiling) + /// Changes per page, at most 8192: 8192 when the CLI follows every + /// page, 1000 for one `--page-token` fetch #[arg(long)] limit: Option, /// Fetch exactly one page starting at this opaque token @@ -1063,7 +1065,8 @@ pub(crate) enum CommitCommand { #[arg(long)] uri: Option, commit_id: String, - /// Changes per page (the server clamps at its public ceiling) + /// Changes per page, at most 8192: 8192 when the CLI follows every + /// page, 1000 for one `--page-token` fetch #[arg(long)] limit: Option, /// Fetch exactly one page starting at this opaque token @@ -1137,7 +1140,8 @@ pub(crate) enum ChangesCommand { /// Start mode: now | beginning | after: (default now) #[arg(long, conflicts_with = "cursor")] start: Option, - /// Changes per page (the server clamps at its public ceiling) + /// Changes per page, at most 8192 (default 8192; the CLI follows + /// every page) #[arg(long)] limit: Option, /// Filter by entity kind (repeatable): node | edge diff --git a/crates/omnigraph-cli/src/main.rs b/crates/omnigraph-cli/src/main.rs index bfecc8e28..58118491b 100644 --- a/crates/omnigraph-cli/src/main.rs +++ b/crates/omnigraph-cli/src/main.rs @@ -1082,7 +1082,7 @@ async fn run(cli: Cli) -> Result<()> { .commit_changes_page( &commit_id, next_page_token.as_deref(), - limit, + auto_page_limit(limit), &filter, &settings, ) @@ -1102,7 +1102,7 @@ async fn run(cli: Cli) -> Result<()> { .commit_changes_page( &commit_id, next_page_token.as_deref(), - limit, + auto_page_limit(limit), &filter, &settings, ) @@ -1158,7 +1158,7 @@ async fn run(cli: Cli) -> Result<()> { cursor.as_deref(), start.as_deref(), next_page_token.as_deref(), - limit, + auto_page_limit(limit), &filter, &settings, ) @@ -1180,7 +1180,7 @@ async fn run(cli: Cli) -> Result<()> { cursor.as_deref(), start.as_deref(), next_page_token.as_deref(), - limit, + auto_page_limit(limit), &filter, &settings, ) @@ -2538,6 +2538,14 @@ fn report_deployment_lock(status: &omnigraph_cluster::DeploymentStatus) { } } +/// The page size the CLI asks for when it follows every page itself: the +/// public maximum unless `--limit` sets one. Every page resolves its commits +/// again and restarts its ordered walk after the previous page's last change, +/// so the same changes cost less in fewer, larger pages. +fn auto_page_limit(limit: Option) -> Option { + Some(limit.unwrap_or(omnigraph::changes::COMMIT_CHANGES_MAX_CHANGES)) +} + /// Print a graph diff: one page when `page_token` is given (the raw escape /// hatch), otherwise every page in turn, each emitted before the next is /// fetched. @@ -2576,7 +2584,12 @@ async fn run_diff( let mut output = DiffJsonStream::new(io::BufWriter::new(io::stdout())); loop { let page = client - .diff_page(sides, next_page_token.as_deref(), limit, &filter) + .diff_page( + sides, + next_page_token.as_deref(), + auto_page_limit(limit), + &filter, + ) .await?; next_page_token = page.next_page_token.clone(); output.write_page(&page)?; @@ -2589,7 +2602,12 @@ async fn run_diff( let mut output = DiffHumanStream::new(); loop { let page = client - .diff_page(sides, next_page_token.as_deref(), limit, &filter) + .diff_page( + sides, + next_page_token.as_deref(), + auto_page_limit(limit), + &filter, + ) .await?; next_page_token = page.next_page_token.clone(); output.write_page(&page)?; diff --git a/crates/omnigraph-server/src/handlers.rs b/crates/omnigraph-server/src/handlers.rs index cae570c5c..c9d03d170 100644 --- a/crates/omnigraph-server/src/handlers.rs +++ b/crates/omnigraph-server/src/handlers.rs @@ -4010,30 +4010,25 @@ fn required_diff_param(value: Option, name: &str) -> std::result::Result /// of a commit. The side is resolved before the policy check, so a denial is /// reported exactly as a side that names nothing: the route is not an /// existence oracle for branches or commits the actor cannot read. -async fn authorize_diff_side( - db: &Omnigraph, +fn authorize_diff_side( actor: Option<&AuthenticatedActor>, policy: Option<&PolicyEngine>, side: &str, + branch: &str, ) -> std::result::Result<(), ApiError> { - let not_found = || ApiError::not_found(format!("'{side}' names no branch and no commit")); - let branch = match db.diff_read_target_branch(side).await { - Ok(branch) => branch, - // A branch deleted between resolution steps is the same fixed 404. - Err(OmniError::BranchNotFound { .. }) => return Err(not_found()), - Err(error) => return Err(change_route_error(error)), - }; match authorize( actor, policy, PolicyRequest { action: PolicyAction::Read, - branch: Some(branch), + branch: Some(branch.to_string()), target_branch: None, }, )? { Authz::Allowed => Ok(()), - Authz::Denied(_) => Err(not_found()), + Authz::Denied(_) => Err(ApiError::not_found(format!( + "'{side}' names no branch and no commit" + ))), } } @@ -4076,19 +4071,35 @@ pub(crate) async fn server_diff( let to = required_diff_param(params.to, "to")?; let db = &handle.engine; let actor = actor.as_ref().map(|Extension(actor)| actor); - for side in [&from, &to] { - authorize_diff_side(db, actor, handle.policy.as_deref(), side).await?; - } let scope = api::change_scope(¶ms.kinds, ¶ms.types, ¶ms.ops); - let page = db - .diff_page(&omnigraph::changes::GraphDiffRequest { - sides: omnigraph::changes::GraphDiffSides::Between { from, to }, + // One resolution serves authorization and the read: each side is + // authorized on the branch it resolved to, before any of its data is read. + let prepared = db + .prepare_diff_page(&omnigraph::changes::GraphDiffRequest { + sides: omnigraph::changes::GraphDiffSides::Between { + from: from.clone(), + to: to.clone(), + }, scope, page_token: params.page_token, max_changes: params.limit, max_bytes: None, }) .await + .map_err(|error| match error { + // A branch deleted between resolution steps is that side's 404. + OmniError::BranchNotFound { branch } => { + let side = if branch == from.trim() { &from } else { &to }; + ApiError::not_found(format!("'{side}' names no branch and no commit")) + } + error => change_route_error(error), + })?; + for (side, branch) in prepared.side_branches() { + authorize_diff_side(actor, handle.policy.as_deref(), side, branch)?; + } + let page = db + .run_diff_page(prepared) + .await .map_err(change_route_error)?; Ok(Json(api::diff_output(&page))) } diff --git a/crates/omnigraph/src/db/mod.rs b/crates/omnigraph/src/db/mod.rs index f4235c14a..16efe1bbe 100644 --- a/crates/omnigraph/src/db/mod.rs +++ b/crates/omnigraph/src/db/mod.rs @@ -19,9 +19,9 @@ pub use omnigraph::{ CleanupPolicyOptions, CollectorCost, CollectorPathSnapshot, CollectorReport, CollectorRowSummary, DatasetCleanupStats, DatasetOptimizeStats, DatasetRepairStats, EXPORT_CHUNK_MAX_BYTES, ExportCut, FullTextIndexRebuildResult, GraphCreateReconciliation, - InitOptions, MergeOutcome, MergeResult, Omnigraph, OpenMode, PendingIndex, PreparedGraphCreate, - PreparedSchemaApply, PreparedSchemaSettlement, RebuiltFullTextIndex, RepairAction, - RepairClassification, RepairOptions, RepairStats, RetainedManifestVersions, + InitOptions, MergeOutcome, MergeResult, Omnigraph, OpenMode, PendingIndex, PreparedDiffPage, + PreparedGraphCreate, PreparedSchemaApply, PreparedSchemaSettlement, RebuiltFullTextIndex, + RepairAction, RepairClassification, RepairOptions, RepairStats, RetainedManifestVersions, SYSTEM_COLUMNS_PREFLIGHT, SchemaApplyReconciliation, SchemaApplyResult, SchemaApplySettlement, SchemaContractDigest, SchemaNonPublicationProof, SkipReason, StagingVerdict, SystemColumnUpgradeFinding, SystemColumnUpgradeOptions, SystemColumnUpgradeOutcome, diff --git a/crates/omnigraph/src/db/omnigraph.rs b/crates/omnigraph/src/db/omnigraph.rs index 1b8655ba7..22ec54704 100644 --- a/crates/omnigraph/src/db/omnigraph.rs +++ b/crates/omnigraph/src/db/omnigraph.rs @@ -2702,6 +2702,29 @@ impl Omnigraph { &self, request: &crate::changes::GraphDiffRequest, ) -> Result { + let prepared = self.prepare_diff(request, false).await?; + self.run_diff_page(prepared).await + } + + /// Resolve a graph diff page request once, reading lineage only: its + /// endpoints, its continuation position, and the branch each two-point + /// side reads ([`PreparedDiffPage::side_branches`]). A transport + /// authorizes those branches and then runs the page with + /// [`Self::run_diff_page`], so authorization and execution see the one + /// resolution. A continuation resolves its endpoints from the page token + /// and classifies each spelled side again for authorization. + pub async fn prepare_diff_page( + &self, + request: &crate::changes::GraphDiffRequest, + ) -> Result { + self.prepare_diff(request, true).await + } + + async fn prepare_diff( + &self, + request: &crate::changes::GraphDiffRequest, + side_branches_on_continuation: bool, + ) -> Result { use crate::changes::token; let max_changes = request @@ -2715,8 +2738,12 @@ impl Omnigraph { let graph_identity = self.schema_view.load().schema_identity_domain.clone(); let filter_digest = token::filter_digest(&request.scope); let sides_digest = token::sides_digest(&request.sides); - let (endpoints, resume) = match request.page_token.as_deref() { - None => (self.resolve_diff_endpoints(&request.sides).await?, None), + let (endpoints, resume, side_branches) = match request.page_token.as_deref() { + None => { + let (endpoints, side_branches) = + self.resolve_diff_endpoints(&request.sides).await?; + (endpoints, None, side_branches) + } Some(encoded) => { let decoded = token::decode_diff_page_token(encoded)?; if decoded.graph_identity != token::hashed_identity(&graph_identity) @@ -2731,6 +2758,15 @@ impl Omnigraph { "diff page token was minted for a different filter scope", )); } + let mut side_branches = Vec::new(); + if side_branches_on_continuation + && let crate::changes::GraphDiffSides::Between { from, to } = &request.sides + { + for side in [from, to] { + let branch = self.classify_diff_read_target(side).await?.read_branch(); + side_branches.push((side.trim().to_string(), branch)); + } + } let endpoints = self .reopen_diff_endpoints( &decoded.from_commit_id, @@ -2744,10 +2780,40 @@ impl Omnigraph { operation_rank: decoded.operation_rank, change_index: decoded.change_index, }; - (endpoints, Some(resume)) + (endpoints, Some(resume), side_branches) } }; + Ok(PreparedDiffPage { + scope: request.scope.clone(), + max_changes, + max_bytes, + graph_identity, + filter_digest, + sides_digest, + endpoints, + resume, + side_branches, + }) + } + + /// Read one prepared graph diff page; see [`Self::prepare_diff_page`]. + pub async fn run_diff_page( + &self, + prepared: PreparedDiffPage, + ) -> Result { + use crate::changes::token; + let PreparedDiffPage { + scope, + max_changes, + max_bytes, + graph_identity, + filter_digest, + sides_digest, + endpoints, + resume, + side_branches: _, + } = prepared; // Reclaimed history under either endpoint is a typed gap naming the // diff's `to` commit; a diff has no feed cursor to resume. let to_commit_id = endpoints.to_commit_id.clone(); @@ -2766,7 +2832,7 @@ impl Omnigraph { &endpoints.to_snapshot, &graph_identity, &endpoints.to_commit_id, - &request.scope, + &scope, resume.as_ref(), &mut budget, &mut changes, @@ -2870,22 +2936,33 @@ impl Omnigraph { )) } - /// Resolve a graph diff's sides to exact commits and their snapshots. + /// Resolve a graph diff's sides to exact commits and their snapshots, + /// with the branch each two-point side reads (none for the branch form, + /// whose transport authorizes the two branches it names). async fn resolve_diff_endpoints( &self, sides: &crate::changes::GraphDiffSides, - ) -> Result { + ) -> Result<(DiffEndpoints, Vec<(String, String)>)> { match sides { crate::changes::GraphDiffSides::Between { from, to } => { - let (from_commit_id, from_snapshot) = self.resolve_diff_read_target(from).await?; - let (to_commit_id, to_snapshot) = self.resolve_diff_read_target(to).await?; - Ok(DiffEndpoints { - from_commit_id, - to_commit_id, - base_commit_id: None, - from_snapshot, - to_snapshot, - }) + let (from_commit_id, from_snapshot, from_branch) = + self.resolve_diff_read_target(from).await?; + let (to_commit_id, to_snapshot, to_branch) = + self.resolve_diff_read_target(to).await?; + let side_branches = vec![ + (from.trim().to_string(), from_branch), + (to.trim().to_string(), to_branch), + ]; + Ok(( + DiffEndpoints { + from_commit_id, + to_commit_id, + base_commit_id: None, + from_snapshot, + to_snapshot, + }, + side_branches, + )) } crate::changes::GraphDiffSides::Branch { source, target } => { let (source_head, target_head, merge_base, source_snapshot, target_snapshot) = @@ -2901,13 +2978,16 @@ impl Omnigraph { } else { self.commit_snapshot(&base_commit_id).await? }; - Ok(DiffEndpoints { - from_commit_id: base_commit_id.clone(), - to_commit_id: source_head, - base_commit_id: Some(base_commit_id), - from_snapshot: base_snapshot, - to_snapshot: source_snapshot, - }) + Ok(( + DiffEndpoints { + from_commit_id: base_commit_id.clone(), + to_commit_id: source_head, + base_commit_id: Some(base_commit_id), + from_snapshot: base_snapshot, + to_snapshot: source_snapshot, + }, + Vec::new(), + )) } } } @@ -2939,33 +3019,21 @@ impl Omnigraph { .snapshot) } - /// The branch a transport authorizes `read` on for one graph diff side: - /// the branch itself, or the authored branch of the commit the side names - /// (an unpublished proposal's or unattached snapshot's recorded target). - /// Refuses exactly as the diff would: ambiguity and absence are typed. - pub async fn diff_read_target_branch(&self, target: &str) -> Result { - Ok(match self.classify_diff_read_target(target).await? { - DiffReadTarget::Branch(resolved) => { - resolved.branch.unwrap_or_else(|| "main".to_string()) - } - DiffReadTarget::Commit(commit) => { - commit.graph_branch.unwrap_or_else(|| "main".to_string()) - } - }) - } - - /// Resolve one diff endpoint to its commit and the graph as of it. - async fn resolve_diff_read_target(&self, target: &str) -> Result<(String, Snapshot)> { - match self.classify_diff_read_target(target).await? { + /// Resolve one diff endpoint to its commit, the graph as of it, and the + /// branch it reads. + async fn resolve_diff_read_target(&self, target: &str) -> Result<(String, Snapshot, String)> { + let classified = self.classify_diff_read_target(target).await?; + let branch = classified.read_branch(); + match classified { DiffReadTarget::Branch(resolved) => { let commit_id = resolved.graph_commit_id.ok_or_else(|| { OmniError::manifest(format!("branch '{}' has no head commit", target.trim())) })?; - Ok((commit_id, resolved.snapshot)) + Ok((commit_id, resolved.snapshot, branch)) } DiffReadTarget::Commit(commit) => { let snapshot = self.commit_snapshot(&commit.graph_commit_id).await?; - Ok((commit.graph_commit_id, snapshot)) + Ok((commit.graph_commit_id, snapshot, branch)) } } } @@ -2989,17 +3057,27 @@ impl Omnigraph { Err(error) => return Err(error), }, }; - let as_commit = match self - .coordinator - .read() - .await - .resolve_commit(&SnapshotId::new(target)) - .await - { - Ok(commit) => Some(commit), - Err(OmniError::Manifest(error)) if error.kind == ManifestErrorKind::NotFound => None, - Err(OmniError::BranchNotFound { .. }) => None, - Err(error) => return Err(error), + // Every graph commit id is minted canonical, a ULID or an `hb1` + // address, so any other spelling names no commit and needs no lookup. + // The lookup is not free: on a miss it checks out every live branch's + // `__manifest`, which a branch name like `main` would otherwise pay. + let as_commit = if !omnigraph_core::graph_commit_id::is_valid_graph_commit_id(target) { + None + } else { + match self + .coordinator + .read() + .await + .resolve_commit(&SnapshotId::new(target)) + .await + { + Ok(commit) => Some(commit), + Err(OmniError::Manifest(error)) if error.kind == ManifestErrorKind::NotFound => { + None + } + Err(OmniError::BranchNotFound { .. }) => None, + Err(error) => return Err(error), + } }; match (as_branch, as_commit) { (Some(_), Some(_)) => Err(OmniError::AmbiguousReadTarget { @@ -4446,6 +4524,41 @@ enum DiffReadTarget { Commit(GraphCommit), } +impl DiffReadTarget { + /// The branch a transport authorizes `read` on: the branch itself, or the + /// authored branch of the commit (an unpublished proposal's or unattached + /// snapshot's recorded target). + fn read_branch(&self) -> String { + match self { + Self::Branch(resolved) => resolved.branch.clone(), + Self::Commit(commit) => commit.graph_branch.clone(), + } + .unwrap_or_else(|| "main".to_string()) + } +} + +/// A graph diff page request resolved once; see +/// [`Omnigraph::prepare_diff_page`]. +pub struct PreparedDiffPage { + scope: crate::changes::ChangeFeedScope, + max_changes: usize, + max_bytes: u64, + graph_identity: String, + filter_digest: String, + sides_digest: String, + endpoints: DiffEndpoints, + resume: Option, + side_branches: Vec<(String, String)>, +} + +impl PreparedDiffPage { + /// Each two-point side as the caller spelled it, with the branch it + /// reads; empty for the branch form. + pub fn side_branches(&self) -> &[(String, String)] { + &self.side_branches + } +} + /// A graph diff's two endpoints, resolved to exact commits and their /// snapshots. struct DiffEndpoints { diff --git a/crates/omnigraph/tests/forbidden_apis.rs b/crates/omnigraph/tests/forbidden_apis.rs index 870a8a724..48272dd35 100644 --- a/crates/omnigraph/tests/forbidden_apis.rs +++ b/crates/omnigraph/tests/forbidden_apis.rs @@ -312,7 +312,8 @@ const READ_ONLY_SURFACES: &[(&str, &str)] = &[ ("db/omnigraph.rs", "resolve_snapshot"), ("db/omnigraph.rs", "diff_page"), ("db/omnigraph.rs", "merge_base"), - ("db/omnigraph.rs", "diff_read_target_branch"), + ("db/omnigraph.rs", "prepare_diff_page"), + ("db/omnigraph.rs", "run_diff_page"), ("exec/merge.rs", "branch_merge_dry_run"), ("db/omnigraph.rs", "commit_changes_page"), ("db/omnigraph.rs", "poll_change_feed"), diff --git a/crates/omnigraph/tests/integration/changes.rs b/crates/omnigraph/tests/integration/changes.rs index ad657870d..096099444 100644 --- a/crates/omnigraph/tests/integration/changes.rs +++ b/crates/omnigraph/tests/integration/changes.rs @@ -4535,18 +4535,37 @@ async fn graph_diff_refuses_an_ambiguous_or_unknown_side() { OmniError::AmbiguousReadTarget { target } => assert_eq!(target, genesis), other => panic!("expected AmbiguousReadTarget, got {other:?}"), } - assert!(matches!( - db.diff_read_target_branch(&genesis).await.unwrap_err(), - OmniError::AmbiguousReadTarget { .. } - )); - match graph_diff_all(&db, between("main", "no-such-side"), None) - .await - .unwrap_err() - { - OmniError::ReadTargetNotFound { target } => assert_eq!(target, "no-such-side"), - other => panic!("expected ReadTargetNotFound, got {other:?}"), + // A spelling no commit id has skips the commit lookup: an unknown name, + // and a malformed `hb1` address, are the same typed not-found. + for side in ["no-such-side", "hb1.not.a.commit"] { + match graph_diff_all(&db, between("main", side), None) + .await + .unwrap_err() + { + OmniError::ReadTargetNotFound { target } => assert_eq!(target, side), + other => panic!("expected ReadTargetNotFound for {side}, got {other:?}"), + } } + + // One resolution names the branch each side reads: a branch itself, and + // a commit's authored branch. + db.branch_create("feature").await.unwrap(); + let on_feature = write_on(&db, "feature", "insert_person", &person("Fay", 30)).await; + let prepared = db + .prepare_diff_page(&diff_request( + between("main", &on_feature.graph_commit_id), + None, + )) + .await + .unwrap(); + assert_eq!( + prepared.side_branches(), + [ + ("main".to_string(), "main".to_string()), + (on_feature.graph_commit_id.clone(), "feature".to_string()), + ] + ); } /// Continuations answer for the commits the first page resolved, even after diff --git a/crates/omnigraph/tests/integration/changes_cost.rs b/crates/omnigraph/tests/integration/changes_cost.rs index 340d502c3..6f6a9dbe9 100644 --- a/crates/omnigraph/tests/integration/changes_cost.rs +++ b/crates/omnigraph/tests/integration/changes_cost.rs @@ -725,6 +725,73 @@ async fn change_feed_caught_up_poll_manifest_reads_are_flat_in_history() { .await; } +/// Resolving a graph diff's sides reads no `__manifest` per live branch. A +/// side spelled as a branch name cannot be a commit id, so it skips the +/// commit lookup, whose miss path checks out every live branch's +/// `__manifest`. Both sides here are branch names; the sweep adds idle +/// branches, which neither side reads. +#[tokio::test] +async fn graph_diff_side_resolution_is_flat_in_live_branches() { + use omnigraph::changes::{GraphDiffRequest, GraphDiffSides}; + + cost_harness(async { + let mut curve: Vec<(u64, IoCounts)> = Vec::new(); + for idle in [1u64, 9] { + let dir = tempfile::tempdir().unwrap(); + let db = Omnigraph::init( + dir.path().to_str().unwrap(), + "node Person {\n name: String @key\n age: I32?\n}\n", + ) + .await + .unwrap(); + let db = helpers::session(db); + db.load_with_receipt( + "main", + r#"{"type":"Person","data":{"name":"base","age":1}}"#, + LoadMode::Merge, + ) + .await + .unwrap(); + db.branch_create("feature").await.unwrap(); + db.load_with_receipt( + "feature", + r#"{"type":"Person","data":{"name":"feature","age":2}}"#, + LoadMode::Merge, + ) + .await + .unwrap(); + for i in 0..idle { + db.branch_create(&format!("idle{i}")).await.unwrap(); + } + + let (page, io) = measure(db.diff_page(&GraphDiffRequest { + sides: GraphDiffSides::Between { + from: "main".to_string(), + to: "feature".to_string(), + }, + scope: ChangeFeedScope::default(), + page_token: None, + max_changes: None, + max_bytes: None, + })) + .await; + assert_eq!(page.unwrap().changes.len(), 1); + eprintln!( + "DIFF idle_branches={idle}: manifest_reads={} data_open_count={}", + io.manifest_reads, io.data_open_count, + ); + curve.push((idle, io)); + } + assert_flat( + &curve, + |io| io.manifest_reads, + 1, + "a diff between two branch names must not read a __manifest per live branch", + ); + }) + .await; +} + /// A backlog walk reads the records of the commits it examines in one scan of /// `__history`, so its manifest reads are flat in the backlog, and pays at /// most two data opens per effectful commit, a term that grows with it. diff --git a/docs/user/branching/changes.md b/docs/user/branching/changes.md index 2a671d7ae..659d9b000 100644 --- a/docs/user/branching/changes.md +++ b/docs/user/branching/changes.md @@ -29,13 +29,15 @@ with its first parent. Each response includes: [Blobs](../blobs.md)). Filter with repeatable `--kind node|edge`, `--type `, and -`--op insert|update|delete` options. `--limit` defaults to 1,000 and may be -at most 8,192. Results are deterministic by entity kind, type identity, logical -id, and operation. +`--op insert|update|delete` options. A page holds at most 8,192 changes; the +HTTP routes default to 1,000. Results are deterministic by entity kind, type +identity, logical id, and operation. Large results are paginated. The CLI normally follows every -`next_page_token`; passing `--page-token` fetches exactly one page. A page -token continues that one commit result and is not a change-feed cursor. +`next_page_token` and asks for 8,192 changes per page, because every page +resumes its comparison where the last one stopped; passing `--page-token` +fetches exactly one page, of 1,000 changes unless `--limit` sets the size. A +page token continues that one commit result and is not a change-feed cursor. A parentless commit returns `409` with reason `parentless_commit`. A diff that crosses a schema boundary returns `409` with reason From 5d976ce9a369c2f7bedda312cdc03bd94c396887 Mon Sep 17 00:00:00 2001 From: Ragnor Comerford Date: Mon, 12 Oct 2026 21:15:29 -0700 Subject: [PATCH 07/11] test(dst): update the cost golden for the graph-diff audit oracle The DST history audit now checks each adjacent pair of recorded commits through diff_page instead of the retired diff_commits. diff_page resolves each side as a read target, which also checks the branch namespace so an ambiguous ULID is refused, and the audit pays 4 more gets in total (_audit l.get 724 -> 728). No engine operation's count changes. --- crates/omnigraph-dst/cost_table.txt | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/crates/omnigraph-dst/cost_table.txt b/crates/omnigraph-dst/cost_table.txt index 3520d83dd..dbe07bb49 100644 --- a/crates/omnigraph-dst/cost_table.txt +++ b/crates/omnigraph-dst/cost_table.txt @@ -35,7 +35,7 @@ _audit a.delete calls=1 _audit a.get calls=2 _audit a.list calls=2 _audit a.put_if_absent calls=1 -_audit l.get calls=724 +_audit l.get calls=728 _audit l.list calls=113 _close l.list calls=3 _history l.list calls=30 From 32c30788868dc3d67564f1c0e63c8f1b3cd2d173 Mon Sep 17 00:00:00 2001 From: Ragnor Comerford Date: Tue, 13 Oct 2026 09:45:02 -0700 Subject: [PATCH 08/11] rfc: state the off-chain merge base as a case, not a permanent rule A base off the source's first-parent chain is refused as a feed start. A merge of the target into a diverged source is one way to get there, and a later merge back can bring the base onto the chain again, so the RFC no longer says any past target-to-source merge prevents the recipe. --- docs/rfcs/0030-cdc-time-travel.md | 30 ++++++++++++++++-------------- 1 file changed, 16 insertions(+), 14 deletions(-) diff --git a/docs/rfcs/0030-cdc-time-travel.md b/docs/rfcs/0030-cdc-time-travel.md index 267cccddf..caf075c3b 100644 --- a/docs/rfcs/0030-cdc-time-travel.md +++ b/docs/rfcs/0030-cdc-time-travel.md @@ -753,12 +753,13 @@ When the base lies on the source's first-parent chain, as it does for a source that has never merged the target in, a caller can replay the source's commits since the base through the feed with `--start after:`. A merge commit in that range still carries the changes it imported in its block (§3.2), so the -replay is the source's history, not only its own edits. Once the target has -been merged into the source, the base is the target commit that merge imported, -which the source's first-parent chain does not hold, and the feed refuses it as -a start (§5.3). The feed does not walk second parents, because that would -change its replay contract; `branch diff` reports the net change from any -base. +replay is the source's history, not only its own edits. A base off that chain +is refused as a start (§5.3). For example, after the target is merged into a +source that had diverged from it, the base is the target commit that merge +imported, which the source's first-parent chain does not hold; a later merge +of the source into the target and back can bring the base onto the chain +again. The feed does not walk second parents, because that would change its +replay contract; `branch diff` reports the net change from any base. `branch merge --dry-run` runs the merge's base resolution, three-way classification and constraint validation, reports the outcome @@ -905,11 +906,11 @@ server `integration/data_routes.rs` and `integration/openapi.rs`; CLI reported base equals branch merge's base on the merge truth-table fixtures, ties included; after the target was merged into the source, only the source's own net changes since the newest base appear. -- Merge base and the feed: after the target was merged into the source, the - base is off the source's first-parent chain and the source's feed refuses - `after:` with `ChangeCursorRejected`, while the branch diff still - reports the source's net change. Owner: the existing merge-and-feed test in - `integration/changes.rs`. +- Merge base and the feed: when the target was merged into a diverged source, + the base is off the source's first-parent chain and the source's feed + refuses `after:` with `ChangeCursorRejected`, while the branch diff + still reports the source's net change. Owner: the existing merge-and-feed + test in `integration/changes.rs`. - Endpoints: a branch, a published commit, a proposal and an unattached snapshot all resolve; a string naming both a branch and a commit is refused; pages after a concurrent commit on either branch still answer for the pinned @@ -1392,9 +1393,10 @@ Amendments before 2026-10-12 are recorded in §14 and in the dispositions above. `commit drop` succeeds. - **2026-10-12: review of the graph diff amendment.** Three corrections from the review of ModernRelay/omnigraph#964: - - The merge base is not always a valid feed start. Once the target has been - merged into the source, the base is the target commit that merge - imported, off the source's first-parent chain, and the feed refuses it. + - The merge base is not always a valid feed start: a base off the source's + first-parent chain is refused. One way it gets there is a merge of the + target into a diverged source, which makes the base the target commit + that merge imported. The review reproduced the refusal with `CommitGraph::merge_base` and the feed in the existing merge-and-feed test. §8.4 now limits the recipe to a base on that chain and points at `branch diff` for the net change, and D0 From 46f9c96c072257ae2f4a76ba5f59e59d279653a3 Mon Sep 17 00:00:00 2001 From: Ragnor Comerford Date: Tue, 13 Oct 2026 09:45:29 -0700 Subject: [PATCH 09/11] docs(changes): state the off-chain merge base as a case in the branch review guide --- docs/user/branching/changes.md | 9 +++++---- 1 file changed, 5 insertions(+), 4 deletions(-) diff --git a/docs/user/branching/changes.md b/docs/user/branching/changes.md index 2a671d7ae..12734983e 100644 --- a/docs/user/branching/changes.md +++ b/docs/user/branching/changes.md @@ -90,10 +90,11 @@ that commit alone, the one a merge of `feature` into `main` would classify against. While `feature` has never merged `main` in, the base is on its first-parent history, and `changes poll --branch feature --start after:` replays its commits since then; a merge commit in that -range still carries the changes it brought in. Once `main` has been merged into -`feature`, the base is the `main` commit that merge brought in, which -`feature`'s feed refuses as a start; `branch diff` gives the net change from -any base. +range still carries the changes it brought in. A base off `feature`'s +first-parent history is refused as a start: for example, after `main` is +merged into a `feature` that had diverged from it, the base is the `main` +commit that merge brought in. `branch diff` gives the net change from any +base. The HTTP routes are `GET /graphs/{graph_id}/branches/diff?source=&target=` and `GET /graphs/{graph_id}/branches/merge-base?source=&target=`. To see whether From d3b05e305d0ec07f2836217744bff3493b0b3f6b Mon Sep 17 00:00:00 2001 From: Ragnor Comerford Date: Tue, 13 Oct 2026 11:07:19 -0700 Subject: [PATCH 10/11] feat(gq): make the graph diff, merge base and merge dry run GQ branch statements MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Review asked that the diff surfaces be GQ statements under branch_stmt, beside `branch merge ... as commit`, rather than GQT step kinds. - Grammar and AST: `branch diff [into ]`, `branch diff to ` with `after page ""` and `limit `, `branch merge-base [into ]`, and `branch merge [into ] dry run`, a BranchStmt::Read control read. A bare `dry run` is the clause, as `as commit` is. - Server: the statements enter POST /query and are refused at POST /mutate as reads. Each runs its route's body (shared helpers now behind GET /diff, /branches/diff, /branches/merge-base and the merge dry run), so authorization, the 403/404 decisions and the dry run's 409 match. - Engine and API types: Session::branch_read answers a statement for embedded callers; branch_read_output renders it. A branch diff answers one row per change, the type named by its name, with the compared commits and the continuation token in a new optional `page` object on ReadOutput, the shape RFC 0056 proposes. - CLI: `omnigraph query -e` runs the statements locally and remotely; the table, kv and jsonl renderings show the page state. - GQT: the three step kinds are withdrawn. Branch reads are `--- query` rows steps without a shape section, taking a bound `${name}` in a quoted side and in expected rows; the four graph_diff cases use them on the engine, DST and served targets. - RFC 0030 §8.5, RFC 0045, RFC 0055, user docs, the changelog note and openapi.json follow. --- .../graph-diff-and-branch-review.added.md | 31 +- crates/omnigraph-api-types/src/lib.rs | 146 ++- crates/omnigraph-bench/src/gqt_case.rs | 10 +- crates/omnigraph-bench/src/gqt_runner.rs | 8 +- crates/omnigraph-cli/src/cli.rs | 12 +- crates/omnigraph-cli/src/client.rs | 45 +- crates/omnigraph-cli/src/helpers.rs | 9 +- crates/omnigraph-cli/src/main.rs | 16 +- crates/omnigraph-cli/src/read_format.rs | 32 +- .../tests/integration/parity_matrix.rs | 33 + crates/omnigraph-compiler/src/query/ast.rs | 64 +- crates/omnigraph-compiler/src/query/parser.rs | 71 +- .../src/query/parser_tests.rs | 88 ++ .../omnigraph-compiler/src/query/query.pest | 28 +- crates/omnigraph-gqt-core/src/lib.rs | 901 +++++------------- crates/omnigraph-gqt-core/src/served.rs | 231 +---- crates/omnigraph-gqt-core/src/tests.rs | 174 +--- crates/omnigraph-gqt/README.md | 25 +- .../cases/graph_diff_branch_review.gqt | 29 +- .../cases/graph_diff_merge_proposal.gqt | 21 +- .../cases/graph_diff_net_range.gqt | 16 +- .../cases/graph_diff_refusals.gqt | 9 +- crates/omnigraph-server/src/handlers.rs | 149 ++- .../omnigraph-server/src/handlers/dispatch.rs | 80 +- .../tests/integration/auth_policy.rs | 48 + .../tests/integration/data_routes.rs | 106 +++ crates/omnigraph/src/changes/mod.rs | 2 +- crates/omnigraph/src/changes/model.rs | 17 + crates/omnigraph/src/session.rs | 54 +- crates/omnigraph/tests/forbidden_apis.rs | 1 + docs/rfcs/0030-cdc-time-travel.md | 72 +- docs/rfcs/0045-gq-logic-tests.md | 49 +- docs/rfcs/0055-gq-branch-statements.md | 8 +- docs/user/branching/changes.md | 15 +- docs/user/branching/index.md | 14 +- docs/user/branching/merge.md | 3 + docs/user/cli/reference.md | 2 +- docs/user/operations/server.md | 2 +- docs/user/queries/index.md | 6 +- openapi.json | 50 +- 40 files changed, 1522 insertions(+), 1155 deletions(-) diff --git a/changelog.d/graph-diff-and-branch-review.added.md b/changelog.d/graph-diff-and-branch-review.added.md index 4191fb34a..2f335005a 100644 --- a/changelog.d/graph-diff-and-branch-review.added.md +++ b/changelog.d/graph-diff-and-branch-review.added.md @@ -1,14 +1,23 @@ -- **Compare two points, review a branch, and preview a merge.** - `omnigraph diff --from --to ` (`GET /graphs/{id}/diff`) reports - the net entity changes between two branches or commits, in the - `commit changes` change shape and without a cause. `branch diff ` - (`GET /branches/diff`) reports what a branch changed since its merge base - with a target branch, and `branch merge-base` (`GET /branches/merge-base`) - prints that base. `branch merge --dry-run` (`dry_run: true` on - `POST /branches/merge`) reports the outcome or the conflicts a merge would - have and writes nothing; it needs `read` on both branches, not - `branch_merge`. See [changes, diffs and change feeds][graph-diff-changes] - and [previewing a merge][graph-diff-merge-preview]. +- **Compare two points, review a branch, and preview a merge.** Each is a GQ + branch statement at `POST /query` and `omnigraph query`, with a CLI command + and HTTP route that wrap the same engine call: + - `branch diff to ` (`omnigraph diff --from --to`, `GET /diff`) + reports the net entity changes between two branches or commits, in the + `commit changes` change shape and without a cause; + - `branch diff [into ]` (`omnigraph branch diff`, + `GET /branches/diff`) reports what a branch changed since its merge base; + - `branch merge-base [into ]` (`omnigraph branch + merge-base`, `GET /branches/merge-base`) names that base; + - `branch merge [into ] dry run` (`omnigraph branch merge + --dry-run`, `dry_run: true` on `POST /branches/merge`) reports the outcome + or the conflicts a merge would have and writes nothing; it needs `read` on + both branches, not `branch_merge`. + + A `branch diff` statement answers one row per change and carries the + compared commits and its continuation token in a new optional `page` object + on `ReadOutput`; `after page ""` continues it. See + [changes, diffs and change feeds][graph-diff-changes] and + [previewing a merge][graph-diff-merge-preview]. [graph-diff-changes]: ../docs/user/branching/changes.md [graph-diff-merge-preview]: ../docs/user/branching/merge.md#preview-a-merge diff --git a/crates/omnigraph-api-types/src/lib.rs b/crates/omnigraph-api-types/src/lib.rs index c15837a9d..6e5b190ba 100644 --- a/crates/omnigraph-api-types/src/lib.rs +++ b/crates/omnigraph-api-types/src/lib.rs @@ -487,6 +487,26 @@ pub struct ReadOutput { /// come from one pinned version, so no separate id fetch is needed. #[serde(skip_serializing_if = "Option::is_none")] pub graph_commit_id: Option, + /// Page state of a paged control read (`branch diff`); absent for every + /// other answer. + #[serde(default, skip_serializing_if = "Option::is_none")] + pub page: Option, +} + +/// The page state a `branch diff` answer carries: the exact commits it +/// compares, and the token `after page ""` continues it with, absent +/// on its last page. +#[derive(Debug, Clone, Default, PartialEq, Eq, Serialize, Deserialize, ToSchema)] +pub struct ReadPageOutput { + #[serde(default, skip_serializing_if = "Option::is_none")] + pub from_commit_id: Option, + #[serde(default, skip_serializing_if = "Option::is_none")] + pub to_commit_id: Option, + /// The merge base, for a branch's diff since it. + #[serde(default, skip_serializing_if = "Option::is_none")] + pub base_commit_id: Option, + #[serde(default, skip_serializing_if = "Option::is_none")] + pub next_page_token: Option, } /// The effect of a branch statement sent to `POST /mutate`, tagged by `kind`. @@ -1087,8 +1107,9 @@ pub struct QueryRequest { /// GQ read-query source. May declare one or more named queries; pick one /// with `name` when more than one is declared. Mutations /// (`insert`/`update`/`delete`) get 400 — use `POST /mutate` instead. May be the branch - /// statement `branch list`, sent with no `name`, `params`, `branch`, or - /// `snapshot`; or one `explain` statement (`explain query …`), which + /// statement `branch list` or a branch control read (`branch diff …`, + /// `branch merge-base …`, `branch merge … dry run`), sent with no `name`, + /// `params`, `branch`, or `snapshot`; or one `explain` statement (`explain query …`), which /// answers the v2 plan instead of /// running it, as one result per plan node (fields `tree`, `depth`, `node`, /// `detail`: the logical, physical and available DataFusion trees, then `plan` entries for @@ -2346,6 +2367,7 @@ pub fn read_output( columns, rows, graph_commit_id, + page: None, }) } @@ -2368,9 +2390,128 @@ pub fn branch_list_read_output(branches: &[String]) -> Result, + page: Option, +) -> Result { + Ok(ReadOutput { + query_name: query_name.to_string(), + target: ReadTargetOutput { + branch: None, + snapshot: None, + }, + row_count: rows.len(), + columns: columns.iter().map(|column| column.to_string()).collect(), + rows: serde_json::value::to_raw_value(&rows)?, + graph_commit_id: None, + page, }) } +/// The answer of any branch control read, rendered by its family. +pub fn branch_read_output( + answer: &omnigraph::changes::BranchReadAnswer, +) -> Result { + use omnigraph::changes::BranchReadAnswer; + match answer { + BranchReadAnswer::Diff(page) => branch_diff_read_output(page), + BranchReadAnswer::MergeBase { + source, + target, + base, + } => branch_merge_base_read_output(source, target, base), + BranchReadAnswer::MergeDryRun { + source, + target, + outcome, + } => branch_merge_dry_run_read_output(source, target, (*outcome).into()), + } +} + +/// The `branch diff` answer: one row per entity change in the diff's order, +/// columns `kind`, `type`, `id`, `op`, `before`, `after`, the HTTP change +/// shape with the type named by its name (`GET /diff` also carries its +/// per-graph opaque id); the compared commits and the continuation token +/// travel in `page`. +pub fn branch_diff_read_output( + page: &omnigraph::changes::GraphDiffPage, +) -> Result { + let rows = page + .changes + .iter() + .map(|change| { + let mut row = serde_json::to_value(entity_change_output(change))?; + row["type"] = serde_json::Value::String(change.entity_type.name.clone()); + Ok(row) + }) + .collect::, serde_json::Error>>()?; + branch_read_rows_output( + "branch diff", + &["kind", "type", "id", "op", "before", "after"], + rows, + Some(ReadPageOutput { + from_commit_id: Some(page.from_commit_id.clone()), + to_commit_id: Some(page.to_commit_id.clone()), + base_commit_id: page.base_commit_id.clone(), + next_page_token: page.next_page_token.clone(), + }), + ) +} + +/// The `branch merge-base` answer: one row naming the two branches, their +/// heads and the merge base commit. +pub fn branch_merge_base_read_output( + source: &str, + target: &str, + base: &omnigraph::changes::MergeBase, +) -> Result { + branch_read_rows_output( + "branch merge-base", + &[ + "source", + "target", + "source_commit_id", + "target_commit_id", + "merge_base_commit_id", + ], + vec![serde_json::json!({ + "source": source, + "target": target, + "source_commit_id": base.source_commit_id, + "target_commit_id": base.target_commit_id, + "merge_base_commit_id": base.merge_base.graph_commit_id, + })], + None, + ) +} + +/// The `branch merge … dry run` answer: one row naming the two branches and +/// the outcome the merge would have. +pub fn branch_merge_dry_run_read_output( + source: &str, + target: &str, + outcome: BranchMergeOutcome, +) -> Result { + branch_read_rows_output( + "branch merge dry run", + &["source", "target", "outcome"], + vec![serde_json::json!({ + "source": source, + "target": target, + "outcome": outcome, + })], + None, + ) +} + /// The `show` answer: one row per setting in the order given, the five /// string columns `name`, `value`, `default`, `source`, `scope`; no target /// and no graph commit (the statement reads the session, not the store). @@ -2406,6 +2547,7 @@ pub fn show_read_output(rows: &[SettingRow]) -> Result { + StepKind::Query | StepKind::BranchList | StepKind::BranchRead => { if reopened { return Err( "reads after preparation restart are not post-reopen treatment".into(), @@ -742,8 +742,12 @@ pub fn override_sources( } /// The only step kinds a served workload may contain anywhere in its program. -pub const SERVED_READ_KINDS: [StepKind; 3] = - [StepKind::Query, StepKind::BranchList, StepKind::Show]; +pub const SERVED_READ_KINDS: [StepKind; 4] = [ + StepKind::Query, + StepKind::BranchList, + StepKind::BranchRead, + StepKind::Show, +]; /// Engine preparation implied by the executed prefix reads, a restart, and the process lifecycle. pub(crate) fn engine_preparation( diff --git a/crates/omnigraph-bench/src/gqt_runner.rs b/crates/omnigraph-bench/src/gqt_runner.rs index 16f8c404d..079f0bfb0 100644 --- a/crates/omnigraph-bench/src/gqt_runner.rs +++ b/crates/omnigraph-bench/src/gqt_runner.rs @@ -46,11 +46,9 @@ pub enum GqtOperationKind { CommitDrop, Settings, Show, + BranchRead, Restart, Concurrent, - Diff, - MergeBase, - MergeDryRun, } impl From for GqtOperationKind { fn from(kind: StepKind) -> Self { @@ -66,11 +64,9 @@ impl From for GqtOperationKind { StepKind::CommitDrop => Self::CommitDrop, StepKind::Settings => Self::Settings, StepKind::Show => Self::Show, + StepKind::BranchRead => Self::BranchRead, StepKind::Restart => Self::Restart, StepKind::Concurrent => Self::Concurrent, - StepKind::Diff => Self::Diff, - StepKind::MergeBase => Self::MergeBase, - StepKind::MergeDryRun => Self::MergeDryRun, } } } diff --git a/crates/omnigraph-cli/src/cli.rs b/crates/omnigraph-cli/src/cli.rs index 36a27f81c..86a4939d1 100644 --- a/crates/omnigraph-cli/src/cli.rs +++ b/crates/omnigraph-cli/src/cli.rs @@ -101,7 +101,7 @@ pub(crate) struct Cli { #[derive(Debug, Subcommand)] pub(crate) enum Command { // ── Data plane ── run against a graph (embedded or via --server). - /// Execute a read query, `branch list`, or an `explain` statement against a branch or snapshot. + /// Execute a read query, a branch read (`branch list`, `branch diff`, `branch merge-base`, `branch merge … dry run`), or an `explain` statement against a branch or snapshot. /// /// Run a read query. Query { @@ -110,13 +110,15 @@ pub(crate) enum Command { /// `--query`/`-e`, selects which query in that ad-hoc source to run. name: Option, /// Ad-hoc query file (a `.gq` you're authoring / break-glass), one - /// `branch list` statement, or one `explain query …` statement, which - /// answers the plan the query would run under as results. + /// branch read statement (`branch list`, `branch diff`, + /// `branch merge-base`, `branch merge … dry run`), or one + /// `explain query …` statement, which answers the plan the query + /// would run under as results. #[arg(long, conflicts_with = "query_string")] query: Option, /// Inline ad-hoc GQ source — alternative to `--query `. May be - /// the `branch list` statement, which takes no name, params, --branch - /// or --snapshot, or an `explain query …` statement. + /// a branch read statement, which takes no name, params, --branch or + /// --snapshot, or an `explain query …` statement. #[arg( short = 'e', long = "query-string", diff --git a/crates/omnigraph-cli/src/client.rs b/crates/omnigraph-cli/src/client.rs index c02846dda..0ec753cab 100644 --- a/crates/omnigraph-cli/src/client.rs +++ b/crates/omnigraph-cli/src/client.rs @@ -38,12 +38,12 @@ use omnigraph_api_types::{ EntityKindOutput, ExportRequest, GraphBatchLoadOutput, GraphDiscoveryResponse, GraphListResponse, InvokeStoredQueryRequest, MergeBaseOutput, QueryRequest, ReadOutput, SchemaApplyOutput, SchemaOutput, SettingsRequest, SnapshotOutput, branch_list_read_output, - change_baseline_output, change_feed_output, change_scope, commit_changes_output, commit_output, - diff_output, merge_base_output, read_output, schema_apply_output, show_read_output, - snapshot_payload, + branch_read_output, change_baseline_output, change_feed_output, change_scope, + commit_changes_output, commit_output, diff_output, merge_base_output, read_output, + schema_apply_output, show_read_output, snapshot_payload, }; use omnigraph_compiler::catalog::Catalog; -use omnigraph_compiler::query::ast::{BranchWrite, CommitWrite}; +use omnigraph_compiler::query::ast::{BranchRead, BranchWrite, CommitWrite}; use omnigraph_compiler::query::parser::parse_query; use reqwest::header::{CONTENT_RANGE, RANGE}; use reqwest::{Method, StatusCode}; @@ -1526,6 +1526,43 @@ impl GraphClient { } } + /// A branch control read from `-e`/`--query` (`branch diff`, + /// `branch merge-base`, `branch merge … dry run`): `POST /query` with the + /// source and the `--set` values, or the embedded engine's answer in the + /// one `ReadOutput` shape the server renders. + pub(crate) async fn branch_read_statement( + &self, + query_source: &str, + read: BranchRead, + settings: &[(SettingId, SettingValue)], + ) -> Result { + match self { + GraphClient::Remote { + http, + base_url, + token, + response_limit, + } => { + let mut request = branch_statement_query_request(query_source); + request.settings = Self::remote_settings(settings)?; + remote_json_bounded( + http, + Method::POST, + remote_url(base_url, &["query"], &[])?, + Some(serde_json::to_value(request)?), + token.as_deref(), + None, + *response_limit, + ) + .await + } + GraphClient::Embedded { uri, .. } => { + let session = Self::open_read_session(uri, settings).await?; + Ok(branch_read_output(&session.branch_read(&read).await?)?) + } + } + } + /// The `show` statement from `-e`/`--query`: `POST /query` with the /// source and the `--set` values, or the embedded session's rows after /// the source's prefix, both as the one `ReadOutput` shape diff --git a/crates/omnigraph-cli/src/helpers.rs b/crates/omnigraph-cli/src/helpers.rs index 6cf90977c..ba80cb144 100644 --- a/crates/omnigraph-cli/src/helpers.rs +++ b/crates/omnigraph-cli/src/helpers.rs @@ -8,7 +8,7 @@ use omnigraph_api_types::{ ChangeRequest, QueryRequest, branch_statement_refusals, query_file_refusals, }; use omnigraph_compiler::query::ast::{ - BranchStmt, BranchWrite, CommitWrite, EXPLAIN_STATEMENT_NAME, QueryFile, show_statement_name, + BranchWrite, CommitWrite, EXPLAIN_STATEMENT_NAME, QueryFile, show_statement_name, }; use omnigraph_compiler::settings::SettingId; @@ -978,11 +978,12 @@ pub(crate) fn commit_write_at_read_door(write: &CommitWrite) -> String { ) } -/// The server's refusal for `branch list` sent through `mutate`. -pub(crate) fn read_at_write_door() -> String { +/// The server's refusal for a branch read (`branch list`, `branch diff`, …) +/// sent through `mutate`. +pub(crate) fn read_at_write_door(statement_name: &str) -> String { branch_statement_refusals::with_statement( branch_statement_refusals::READ_AT_WRITE_DOOR, - BranchStmt::List.statement_name(), + statement_name, ) } diff --git a/crates/omnigraph-cli/src/main.rs b/crates/omnigraph-cli/src/main.rs index bfecc8e28..7a464789a 100644 --- a/crates/omnigraph-cli/src/main.rs +++ b/crates/omnigraph-cli/src/main.rs @@ -1654,7 +1654,7 @@ async fn run(cli: Cli) -> Result<()> { .. }) => { refuse_process_prefix_when_remote(&client, &prefix)?; - run_branch_list_statement_cli( + run_branch_read_statement_cli( &client, &query_source, stmt, @@ -2412,7 +2412,7 @@ fn refuse_process_prefix_when_remote( /// The `query` door's branch-statement path: the door check, then the /// envelope refusals, then the round trip. The door rule itself is documented /// on `refuse_wrong_door` in `crates/omnigraph-server/src/handlers/dispatch.rs`. -async fn run_branch_list_statement_cli( +async fn run_branch_read_statement_cli( client: &client::GraphClient, query_source: &str, stmt: BranchStmt, @@ -2424,7 +2424,14 @@ async fn run_branch_list_statement_cli( bail!("{}", control_write_at_read_door(write)); } refuse_statement_envelope(has_target, has_name_or_params, false)?; - client.branch_list_statement(query_source, settings).await + match stmt { + BranchStmt::Read(read) => { + client + .branch_read_statement(query_source, read, settings) + .await + } + _ => client.branch_list_statement(query_source, settings).await, + } } /// The `query` door's `show` path: the envelope refusals `branch list` @@ -2457,8 +2464,9 @@ async fn run_branch_statement_cli( json: bool, settings: &[(SettingId, SettingValue)], ) -> Result { + let statement_name = stmt.statement_name(); let BranchStmt::Write(write) = stmt else { - bail!("{}", read_at_write_door()); + bail!("{}", read_at_write_door(statement_name)); }; refuse_statement_envelope(has_target, has_name_or_params, has_expected_head)?; if let BranchWrite::Delete { .. } = &write { diff --git a/crates/omnigraph-cli/src/read_format.rs b/crates/omnigraph-cli/src/read_format.rs index ae57fc765..d78e268e5 100644 --- a/crates/omnigraph-cli/src/read_format.rs +++ b/crates/omnigraph-cli/src/read_format.rs @@ -42,8 +42,8 @@ pub fn render_read( ReadOutputFormat::Json => Ok(serde_json::to_string_pretty(output)?), ReadOutputFormat::Jsonl => render_jsonl(output), ReadOutputFormat::Csv => render_csv(output), - ReadOutputFormat::Kv => render_kv(output), - ReadOutputFormat::Table => render_table(output, options), + ReadOutputFormat::Kv => Ok(render_kv(output)? + &page_footer(output)), + ReadOutputFormat::Table => Ok(render_table(output, options)? + &page_footer(output)), ReadOutputFormat::Arrow => color_eyre::eyre::bail!( "`--format arrow` writes an Arrow IPC byte stream to stdout and has no text rendering" ), @@ -57,6 +57,7 @@ fn render_jsonl(output: &ReadOutput) -> Result { "query_name": output.query_name, "target": output.target, "row_count": output.row_count, + "page": output.page, }))?); for row in rows(output)? { lines.push(serde_json::to_string(&row)?); @@ -200,12 +201,34 @@ fn header_line(output: &ReadOutput) -> String { .map(|branch| format!(" from branch {}", branch)) }) .unwrap_or_default(); + let compared = output + .page + .as_ref() + .and_then(|page| { + Some(( + page.from_commit_id.as_deref()?, + page.to_commit_id.as_deref()?, + )) + }) + .map(|(from, to)| format!(" comparing {from} to {to}")) + .unwrap_or_default(); format!( - "{} rows{} via {}", - output.row_count, from, output.query_name + "{} rows{}{} via {}", + output.row_count, from, compared, output.query_name ) } +/// The line a paged answer ends with when a further page exists: the clause +/// that continues it. +fn page_footer(output: &ReadOutput) -> String { + output + .page + .as_ref() + .and_then(|page| page.next_page_token.as_deref()) + .map(|token| format!("\nmore: continue with `after page \"{token}\"`")) + .unwrap_or_default() +} + fn rows(output: &ReadOutput) -> Result>> { let parsed: Value = serde_json::from_str(output.rows.get())?; Ok(parsed @@ -319,6 +342,7 @@ mod tests { ) .unwrap(), graph_commit_id: None, + page: None, } } diff --git a/crates/omnigraph-cli/tests/integration/parity_matrix.rs b/crates/omnigraph-cli/tests/integration/parity_matrix.rs index 06fcd8ca3..6adea2f65 100644 --- a/crates/omnigraph-cli/tests/integration/parity_matrix.rs +++ b/crates/omnigraph-cli/tests/integration/parity_matrix.rs @@ -1494,6 +1494,16 @@ fn parity_branch_review() { ][..], &["diff", "--from", "main", "--to", "feature", "--json"][..], &["branch", "merge", "feature", "--dry-run", "--json"][..], + // The same reads as GQ statements at the query door. + &["query", "-e", "branch merge-base feature", "--json"][..], + &[ + "query", + "-e", + "branch diff feature into main limit 1", + "--json", + ][..], + &["query", "-e", "branch diff main to feature", "--json"][..], + &["query", "-e", "branch merge feature dry run", "--json"][..], ] { let (l, r) = p.run(args); assert!( @@ -1516,6 +1526,29 @@ fn parity_branch_review() { assert_eq!(dry["outcome"], "fast_forward"); assert_eq!(dry["dry_run"], true); assert!(dry["commit"].is_null()); + let (l, _r) = p.run(&["query", "-e", "branch diff feature", "--json"]); + let rows = parse_stdout_json(&l); + assert_eq!(rows["query_name"], "branch diff"); + assert_eq!(rows["rows"][0]["id"], "Reviewed"); + assert_eq!(rows["rows"][0]["type"], "Person"); + assert_eq!(rows["page"]["base_commit_id"], review["base_commit_id"]); + let (l, r) = p.run(&["query", "-e", "branch merge feature dry run", "--json"]); + for output in [&l, &r] { + assert_eq!( + parse_stdout_json(output)["rows"][0]["outcome"], + "fast_forward" + ); + } + // A branch read sent through `mutate` is refused as a read on both arms. + let (l, r) = p.run(&["mutate", "-e", "branch diff feature"]); + for output in [&l, &r] { + assert!(!output.status.success(), "{output:?}"); + assert!( + String::from_utf8_lossy(&output.stderr) + .contains("statement 'branch diff' is a read; use POST /query"), + "{output:?}" + ); + } // The dry run wrote nothing: the real merge still has work to do. let (l, r) = p.run(&["branch", "merge", "feature", "--json"]); assert_write_parity("merge after the dry run", &l, &r); diff --git a/crates/omnigraph-compiler/src/query/ast.rs b/crates/omnigraph-compiler/src/query/ast.rs index ad5eb8a12..c9a322526 100644 --- a/crates/omnigraph-compiler/src/query/ast.rs +++ b/crates/omnigraph-compiler/src/query/ast.rs @@ -198,12 +198,67 @@ impl QueryFile { } } -/// A top-level branch statement: a control write, or `branch list`, the -/// one statement that changes no branch. +/// A top-level branch statement: a control write, `branch list`, or a +/// control read that compares or previews branches (RFC 0030 §8). Only a +/// control write changes a branch. #[derive(Debug, Clone, PartialEq, Eq)] pub enum BranchStmt { Write(BranchWrite), List, + Read(BranchRead), +} + +/// A branch control read: it changes no branch and publishes nothing. +#[derive(Debug, Clone, PartialEq, Eq)] +pub enum BranchRead { + /// `branch diff [into ]` or `branch diff to `, + /// one page of net entity changes. + Diff { + sides: BranchDiffSides, + page: DiffPageClause, + }, + /// `branch merge-base [into ]`. + MergeBase { + source: String, + into: Option, + }, + /// `branch merge [into ] dry run`: the merge decided, + /// nothing written. + MergeDryRun { + source: String, + into: Option, + }, +} + +/// The two sides of `branch diff`. `into` is `None` when unspelled. +#[derive(Debug, Clone, PartialEq, Eq)] +pub enum BranchDiffSides { + /// What `source` changed since its merge base with `into`. + Branch { + source: String, + into: Option, + }, + /// The net change from `from` to `to`, each a branch or a commit id. + Between { from: String, to: String }, +} + +/// The paging clauses of `branch diff`: `after page ""` continues a +/// previous answer, `limit ` bounds the page's changes. +#[derive(Debug, Clone, Default, PartialEq, Eq)] +pub struct DiffPageClause { + pub after_page: Option, + pub limit: Option, +} + +impl BranchRead { + /// The read's leading words. + pub fn statement_name(&self) -> &'static str { + match self { + BranchRead::Diff { .. } => "branch diff", + BranchRead::MergeBase { .. } => "branch merge-base", + BranchRead::MergeDryRun { .. } => "branch merge dry run", + } + } } /// A control write: `branch create [from ]`, @@ -238,16 +293,17 @@ impl BranchWrite { } impl BranchStmt { - /// `false` only for `branch list`. + /// `true` only for a control write. pub fn is_write(&self) -> bool { matches!(self, BranchStmt::Write(_)) } - /// The statement's two keywords, `branch create` through `branch list`. + /// The statement's leading words, `branch create` through `branch diff`. pub fn statement_name(&self) -> &'static str { match self { BranchStmt::Write(write) => write.statement_name(), BranchStmt::List => "branch list", + BranchStmt::Read(read) => read.statement_name(), } } diff --git a/crates/omnigraph-compiler/src/query/parser.rs b/crates/omnigraph-compiler/src/query/parser.rs index b2caccaf3..34d1ee73c 100644 --- a/crates/omnigraph-compiler/src/query/parser.rs +++ b/crates/omnigraph-compiler/src/query/parser.rs @@ -298,10 +298,15 @@ fn parse_branch_stmt( .find(|inner| inner.as_rule() != Rule::kw_branch) .expect("grammar: branch_stmt holds one form after kw_branch"); let rule = form.as_rule(); - let as_commit = form - .clone() - .into_inner() - .any(|inner| inner.as_rule() == Rule::as_commit_clause); + let has = |wanted: Rule| { + form.clone() + .into_inner() + .any(|inner| inner.as_rule() == wanted) + }; + let as_commit = has(Rule::as_commit_clause); + let dry_run = has(Rule::dry_run_clause); + let between = has(Rule::kw_to); + let page = parse_diff_page_clause(&form)?; let names = form .into_inner() .filter(|inner| inner.as_rule() == Rule::branch_name) @@ -315,18 +320,76 @@ fn parse_branch_stmt( (Rule::branch_delete, Some(name), None) => { Ok(BranchStmt::Write(BranchWrite::Delete { name })) } + (Rule::branch_merge, Some(source), into) if dry_run => { + Ok(BranchStmt::Read(BranchRead::MergeDryRun { source, into })) + } (Rule::branch_merge, Some(source), into) => Ok(BranchStmt::Write(BranchWrite::Merge { source, into, as_commit, })), (Rule::branch_list, None, None) => Ok(BranchStmt::List), + (Rule::branch_merge_base, Some(source), into) => { + Ok(BranchStmt::Read(BranchRead::MergeBase { source, into })) + } + (Rule::branch_diff, Some(from), Some(to)) if between => { + Ok(BranchStmt::Read(BranchRead::Diff { + sides: BranchDiffSides::Between { from, to }, + page, + })) + } + (Rule::branch_diff, Some(source), into) => Ok(BranchStmt::Read(BranchRead::Diff { + sides: BranchDiffSides::Branch { source, into }, + page, + })), (other, _, _) => { unreachable!("grammar: every branch form fixes its name arity, got {other:?}") } } } +/// The `after page ""` and `limit ` clauses of `branch diff`; both +/// absent on every other form. A limit is at least one (`Q004` otherwise). +fn parse_diff_page_clause( + form: &pest::iterators::Pair, +) -> std::result::Result { + let mut page = DiffPageClause::default(); + for clause in form.clone().into_inner() { + match clause.as_rule() { + Rule::diff_after_page => { + let literal = clause + .into_inner() + .find(|inner| inner.as_rule() == Rule::string_lit) + .expect("grammar: `after page` names one string literal"); + let token = decode_string_literal(literal.as_str()) + .map_err(|error| diagnostic_at(Q004, parse_error_message(error), &literal))?; + page.after_page = Some(token); + } + Rule::diff_limit => { + let integer = clause + .into_inner() + .find(|inner| inner.as_rule() == Rule::integer) + .expect("grammar: `limit` names one integer"); + let limit = integer + .as_str() + .parse::() + .ok() + .filter(|limit| *limit > 0) + .ok_or_else(|| { + diagnostic_at( + Q004, + format!("diff limit {} must be a positive integer", integer.as_str()), + &integer, + ) + })?; + page.limit = Some(limit); + } + _ => {} + } + } + Ok(page) +} + fn parse_commit_stmt( pair: pest::iterators::Pair, ) -> std::result::Result { diff --git a/crates/omnigraph-compiler/src/query/parser_tests.rs b/crates/omnigraph-compiler/src/query/parser_tests.rs index 227b68f52..c795f4ff0 100644 --- a/crates/omnigraph-compiler/src/query/parser_tests.rs +++ b/crates/omnigraph-compiler/src/query/parser_tests.rs @@ -1744,6 +1744,94 @@ fn branch_statements_parse_with_and_without_their_defaults() { ); } +#[test] +fn branch_control_reads_parse_as_reads() { + let read = |stmt: BranchStmt| match stmt { + BranchStmt::Read(read) => read, + other => panic!("expected a branch control read, got {other:?}"), + }; + let names = |source: &str, into: Option<&str>| (source.to_string(), into.map(str::to_string)); + + let diff = parse_branch("branch diff feature"); + assert!(!diff.is_write()); + assert_eq!(diff.statement_name(), "branch diff"); + assert_eq!( + read(diff), + BranchRead::Diff { + sides: BranchDiffSides::Branch { + source: "feature".to_string(), + into: None, + }, + page: DiffPageClause::default(), + } + ); + assert_eq!( + read(parse_branch( + r#"branch diff feature into main after page "tok" limit 50"# + )), + BranchRead::Diff { + sides: BranchDiffSides::Branch { + source: "feature".to_string(), + into: Some("main".to_string()), + }, + page: DiffPageClause { + after_page: Some("tok".to_string()), + limit: Some(50), + }, + } + ); + // A commit id is quoted: it is not an identifier. + assert_eq!( + read(parse_branch( + r#"branch diff "hb1.01ARZ3NDEKTSV4RRFFQ69G5FAV.0.01ARZ3NDEKTSV4RRFFQ69G5FAV" to main limit 1"# + )), + BranchRead::Diff { + sides: BranchDiffSides::Between { + from: "hb1.01ARZ3NDEKTSV4RRFFQ69G5FAV.0.01ARZ3NDEKTSV4RRFFQ69G5FAV".to_string(), + to: "main".to_string(), + }, + page: DiffPageClause { + after_page: None, + limit: Some(1), + }, + } + ); + + let base = parse_branch("branch merge-base feature into main"); + assert!(!base.is_write()); + assert_eq!(base.statement_name(), "branch merge-base"); + let (source, into) = names("feature", Some("main")); + assert_eq!(read(base), BranchRead::MergeBase { source, into }); + + let dry = parse_branch("branch merge feature dry run"); + assert!(!dry.is_write()); + assert_eq!(dry.statement_name(), "branch merge dry run"); + let (source, into) = names("feature", None); + assert_eq!(read(dry), BranchRead::MergeDryRun { source, into }); + let (source, into) = names("feature", Some("main")); + assert_eq!( + read(parse_branch("branch merge feature into main dry run")), + BranchRead::MergeDryRun { source, into } + ); + + // `dry run` is the clause, never a branch named `dry`; quoted, it is one. + assert_eq!(parse_branch(r#"branch merge "dry""#), merge("dry", None)); + assert!(parse_query("branch merge dry run").is_err()); + for input in [ + "branch merge feature dry run as commit", + "branch merge feature as commit dry run", + "branch merge-base feature dry run", + "branch diff feature to", + "branch diff feature into main to other", + "branch diff feature limit 0", + "branch diff feature limit 5 after page \"tok\"", + "branch diffx feature", + "branch merge-basefeature", + ] { + assert!(parse_query(input).is_err(), "{input}"); + } +} + #[test] fn branch_names_outside_the_identifier_alphabet_are_quoted() { assert_eq!( diff --git a/crates/omnigraph-compiler/src/query/query.pest b/crates/omnigraph-compiler/src/query/query.pest index 6c36eda91..b3eb0df0c 100644 --- a/crates/omnigraph-compiler/src/query/query.pest +++ b/crates/omnigraph-compiler/src/query/query.pest @@ -28,7 +28,7 @@ setting_value = { integer | ident | string_lit } // Statements: one per file, never beside a query declaration. Each keyword // is atomic and closed by a word boundary: an identifier character may not // follow it (`branch createb0` is a parse error), a quoted name may. -branch_stmt = { kw_branch ~ (branch_create | branch_delete | branch_merge | branch_list) } +branch_stmt = { kw_branch ~ (branch_create | branch_delete | branch_merge_base | branch_merge | branch_list | branch_diff) } // The two control writes on an unpublished commit, named by its full id. commit_stmt = { kw_commit ~ (commit_publish | commit_drop) } // `explain` wraps the one read declaration whose plan is answered as rows. @@ -38,11 +38,21 @@ explain_stmt = { kw_explain ~ query_decl } statement_trailer = { (setting_stmt | show_stmt | branch_stmt | commit_stmt | explain_stmt | query_decl)+ } branch_create = { kw_create ~ branch_name ~ (kw_from ~ branch_name)? } branch_delete = { kw_delete ~ branch_name } -branch_merge = { kw_merge ~ branch_name ~ (kw_into ~ branch_name)? ~ as_commit_clause? } +branch_merge = { kw_merge ~ branch_name ~ (kw_into ~ branch_name)? ~ (as_commit_clause | dry_run_clause)? } branch_list = { kw_list } -// A bare `as` followed by `commit` is the clause, never a branch named `as`; -// quote the name for that. -branch_name = { !(kw_as ~ kw_commit) ~ (ident | string_lit) } +// Control reads (RFC 0030 §8): the merge base of a merge, a branch's net +// change since that base (`into`), or the net change between two points, +// each a branch or a quoted commit id (`to`). `merge-base` is tried before +// `merge`, which would otherwise take `merge` and fail on `-base`. +branch_merge_base = { kw_merge_base ~ branch_name ~ (kw_into ~ branch_name)? } +branch_diff = { kw_diff ~ branch_name ~ ((kw_into | kw_to) ~ branch_name)? ~ diff_after_page? ~ diff_limit? } +diff_after_page = { kw_after ~ kw_page ~ string_lit } +diff_limit = { kw_limit ~ integer } +// `dry run`: the merge is decided and nothing is written. +dry_run_clause = { kw_dry ~ kw_run } +// A bare `as` followed by `commit`, or `dry` followed by `run`, is the +// clause, never a branch named `as` or `dry`; quote the name for that. +branch_name = { !(kw_as ~ kw_commit) ~ !(kw_dry ~ kw_run) ~ (ident | string_lit) } commit_publish = { kw_publish ~ string_lit } commit_drop = { kw_drop ~ string_lit } // `as commit`: the write's result is an unpublished commit. @@ -57,6 +67,14 @@ kw_explain = @{ "explain" ~ !(ASCII_ALPHANUMERIC | "_") } kw_create = @{ "create" ~ !(ASCII_ALPHANUMERIC | "_") } kw_delete = @{ "delete" ~ !(ASCII_ALPHANUMERIC | "_") } kw_merge = @{ "merge" ~ !(ASCII_ALPHANUMERIC | "_") } +kw_merge_base = @{ "merge-base" ~ !(ASCII_ALPHANUMERIC | "_") } +kw_diff = @{ "diff" ~ !(ASCII_ALPHANUMERIC | "_") } +kw_to = @{ "to" ~ !(ASCII_ALPHANUMERIC | "_") } +kw_after = @{ "after" ~ !(ASCII_ALPHANUMERIC | "_") } +kw_page = @{ "page" ~ !(ASCII_ALPHANUMERIC | "_") } +kw_limit = @{ "limit" ~ !(ASCII_ALPHANUMERIC | "_") } +kw_dry = @{ "dry" ~ !(ASCII_ALPHANUMERIC | "_") } +kw_run = @{ "run" ~ !(ASCII_ALPHANUMERIC | "_") } kw_list = @{ "list" ~ !(ASCII_ALPHANUMERIC | "_") } kw_from = @{ "from" ~ !(ASCII_ALPHANUMERIC | "_") } kw_into = @{ "into" ~ !(ASCII_ALPHANUMERIC | "_") } diff --git a/crates/omnigraph-gqt-core/src/lib.rs b/crates/omnigraph-gqt-core/src/lib.rs index 9059b831c..68c2fa7c3 100644 --- a/crates/omnigraph-gqt-core/src/lib.rs +++ b/crates/omnigraph-gqt-core/src/lib.rs @@ -16,8 +16,9 @@ use omnigraph::instrumentation::{QueryIoProbes, capture_query_io_probes, with_qu use omnigraph::loader::LoadMode; use omnigraph::{MutationReceipt, Session}; use omnigraph_compiler::query::ast::{ - BranchStmt, BranchWrite, Clause, CommitWrite, EmptyFile, Expr, FileBody, Literal, - MutationWriteMode, Param, QueryDecl, SettingStmt, UnattachedBase, show_statement_name, + BranchDiffSides, BranchRead, BranchStmt, BranchWrite, Clause, CommitWrite, EmptyFile, Expr, + FileBody, Literal, MutationWriteMode, Param, QueryDecl, SettingStmt, UnattachedBase, + show_statement_name, }; use omnigraph_compiler::query::parser::parse_query; use omnigraph_compiler::query::typecheck::{ @@ -121,11 +122,9 @@ pub fn step_kind(step: &Step) -> &'static str { Step::List(_) => "list", Step::Settings(_) => "settings", Step::Show(_) => "show", + Step::BranchRead(_) => "branch read", Step::Restart { .. } => "restart", Step::Concurrent(_) => "concurrent", - Step::Diff(_) => "diff", - Step::MergeBase(_) => "merge-base", - Step::MergeDryRun(_) => "merge-dry-run", } } @@ -148,11 +147,9 @@ pub enum Step { List(ListStep), Settings(SettingsStep), Show(ShowStep), + BranchRead(BranchReadStep), Restart { ordinal: usize }, Concurrent(ConcurrentStep), - Diff(DiffStep), - MergeBase(MergeBaseStep), - MergeDryRun(MergeDryRunStep), } impl Step { @@ -165,11 +162,9 @@ impl Step { Self::List(s) => s.ordinal, Self::Settings(s) => s.ordinal, Self::Show(s) => s.ordinal, + Self::BranchRead(s) => s.ordinal, Self::Restart { ordinal } => *ordinal, Self::Concurrent(s) => s.ordinal, - Self::Diff(s) => s.ordinal, - Self::MergeBase(s) => s.ordinal, - Self::MergeDryRun(s) => s.ordinal, } } @@ -180,15 +175,13 @@ impl Step { Step::Query(step) => Some(&step.expect), Step::List(step) => Some(&step.expect), Step::Show(step) => Some(&step.expect), + Step::BranchRead(step) => Some(&step.expect), Step::Mutate(_) | Step::Load(_) | Step::Control(_) | Step::Settings(_) | Step::Restart { .. } - | Step::Concurrent(_) - | Step::Diff(_) - | Step::MergeBase(_) - | Step::MergeDryRun(_) => None, + | Step::Concurrent(_) => None, } } @@ -197,15 +190,13 @@ impl Step { Step::Query(step) => Some(&mut step.expect), Step::List(step) => Some(&mut step.expect), Step::Show(step) => Some(&mut step.expect), + Step::BranchRead(step) => Some(&mut step.expect), Step::Mutate(_) | Step::Load(_) | Step::Control(_) | Step::Settings(_) | Step::Restart { .. } - | Step::Concurrent(_) - | Step::Diff(_) - | Step::MergeBase(_) - | Step::MergeDryRun(_) => None, + | Step::Concurrent(_) => None, } } } @@ -336,99 +327,6 @@ pub enum MergeExpect { Outcome(MergeOutcome), } -/// A `--- diff` step (RFC 0030 §8): one graph diff or one commit's own diff, -/// run to its last page, its changes compared as JSON rows. -#[derive(Debug)] -pub struct DiffStep { - ordinal: usize, - pub sides: DiffSides, - expect: DiffExpect, -} - -/// What a `--- diff` step compares. A side is a branch name, or `$name` for -/// the commit a `--- bind commit $name` captured. -#[derive(Debug, Clone, PartialEq, Eq)] -pub enum DiffSides { - /// `from: to: `: the net changes between two read targets. - Between { from: String, to: String }, - /// `source: ` and `target: ` (`main` when unspelled): what - /// the source changed since its merge base with the target. - Branch { source: String, target: String }, - /// `commit: `: one commit's own changes (`commit changes`). - Commit { commit: String }, -} - -/// A diff step's expect: the change rows, or an error. -#[derive(Debug)] -pub enum DiffExpect { - /// One JSON object per change: `kind`, `type` (the type's name), `id`, - /// `op`, and `before` / `after` images as the wire spells them. - Rows { - ordered: bool, - rows: Vec, - }, - Error { - needle: String, - }, -} - -/// `--- merge-base source: [target: ]`: the commit a merge -/// of the source into the target would classify against. -#[derive(Debug)] -pub struct MergeBaseStep { - ordinal: usize, - pub source: String, - pub target: String, - expect: MergeBaseExpect, -} - -/// A merge-base step's expect: `commit $name` or an error. -#[derive(Debug)] -pub enum MergeBaseExpect { - Commit { name: String }, - Error { needle: String }, -} - -/// `--- merge-dry-run source: [target: ]`: the merge run -/// through validation without a write, judged with a merge's own expects. -#[derive(Debug)] -pub struct MergeDryRunStep { - ordinal: usize, - pub source: String, - pub target: String, - expect: MergeExpect, -} - -impl DiffStep { - pub fn ordinal(&self) -> usize { - self.ordinal - } - - pub fn expect(&self) -> &DiffExpect { - &self.expect - } -} - -impl MergeBaseStep { - pub fn ordinal(&self) -> usize { - self.ordinal - } - - pub fn expect(&self) -> &MergeBaseExpect { - &self.expect - } -} - -impl MergeDryRunStep { - pub fn ordinal(&self) -> usize { - self.ordinal - } - - pub fn expect(&self) -> &MergeExpect { - &self.expect - } -} - /// A `--- query` step holding `branch list`: one `name` column, rows in /// byte order, so its expect is a read expect like a declaration's. #[derive(Debug)] @@ -456,6 +354,136 @@ pub struct ShowStep { expect: QueryExpect, } +/// A `--- query` step holding a branch control read (`branch diff`, +/// `branch merge-base`, `branch merge … dry run`, RFC 0030 §8). Its rows are +/// the statement's answer, compared as JSON; a side spelled `"${name}"`, and +/// a `"${name}"` in an expected row, stand for the commit `--- bind commit +/// $name` captured. Its shape is the statement's own, so the step carries no +/// shape section. +#[derive(Debug)] +pub struct BranchReadStep { + ordinal: usize, + read: BranchRead, + /// The statement text, for the served path, which sends it with its + /// bound names replaced. + source: String, + expect: QueryExpect, +} + +/// The operands of a branch control read that may name a bound commit: its +/// branch or commit sides. +fn branch_read_operands(read: &BranchRead) -> Vec<&str> { + match read { + BranchRead::Diff { + sides: BranchDiffSides::Branch { source, into }, + .. + } + | BranchRead::MergeBase { source, into } + | BranchRead::MergeDryRun { source, into } => std::iter::once(source.as_str()) + .chain(into.as_deref()) + .collect(), + BranchRead::Diff { + sides: BranchDiffSides::Between { from, to }, + .. + } => vec![from.as_str(), to.as_str()], + } +} + +/// `read` with every operand spelled `${name}` replaced by the bound id. +fn resolve_branch_read(read: &BranchRead, bindings: &Bindings) -> Result { + let resolve = |operand: &str| resolve_operand(operand, bindings); + let resolve_into = |into: &Option| into.as_deref().map(resolve).transpose(); + Ok(match read { + BranchRead::Diff { sides, page } => BranchRead::Diff { + sides: match sides { + BranchDiffSides::Branch { source, into } => BranchDiffSides::Branch { + source: resolve(source)?, + into: resolve_into(into)?, + }, + BranchDiffSides::Between { from, to } => BranchDiffSides::Between { + from: resolve(from)?, + to: resolve(to)?, + }, + }, + page: page.clone(), + }, + BranchRead::MergeBase { source, into } => BranchRead::MergeBase { + source: resolve(source)?, + into: resolve_into(into)?, + }, + BranchRead::MergeDryRun { source, into } => BranchRead::MergeDryRun { + source: resolve(source)?, + into: resolve_into(into)?, + }, + }) +} + +/// `text` with every bound commit name replaced by its id. +fn substitute_bound(text: &str, bindings: &Bindings) -> String { + bindings.iter().fold(text.to_string(), |text, (name, id)| { + text.replace(&format!("${{{name}}}"), id) + }) +} + +/// The step a branch control read becomes under `mode`: rows in the +/// statement's own order or as a multiset, or an error. A bound `${name}` +/// may appear in the expected rows. +fn complete_branch_read_step( + ordinal: usize, + read: BranchRead, + source: String, + mode: &ExpectHeader, + section: &Section<'_>, + loop_var: Option<&str>, + bound_names: &HashSet, +) -> Result { + let expect = match mode { + ExpectHeader::Unordered | ExpectHeader::Ordered => { + refuse_comment_lines(§ion.body, "expect")?; + let body: String = section + .body + .iter() + .map(|(_, l)| *l) + .collect::>() + .join("\n"); + validate_subst_tokens(&body, loop_var, bound_names)?; + QueryExpect::Rows { + ordered: matches!(mode, ExpectHeader::Ordered), + body_raw: body, + span: BodySpan { + start_line: section.header_line + 1, + len: section.body.len(), + }, + shape: ShapeExpect { + lines: Vec::new(), + span: BodySpan { + start_line: 0, + len: 0, + }, + }, + } + } + ExpectHeader::Error(needle) => { + refuse_nonempty_body(§ion.body, "an `expect error:` section")?; + QueryExpect::Error { + needle: needle.clone(), + } + } + ExpectHeader::Ok | ExpectHeader::Affected { .. } | ExpectHeader::Outcome(_) => { + return Err(format!( + "`{}` takes `unordered`, `ordered`, or `error:`", + read.statement_name() + )); + } + }; + Ok(BranchReadStep { + ordinal, + read, + source, + expect, + }) +} + /// `main`, the default wherever a branch is unspelled. const MAIN_BRANCH: &str = "main"; @@ -709,13 +737,8 @@ enum ExpectHeader { Ordered, Ok, Error(String), - Affected { - nodes: usize, - edges: usize, - }, + Affected { nodes: usize, edges: usize }, Outcome(MergeOutcome), - /// `commit $name`: the commit a `--- bind commit $name` captured. - Commit(String), } /// The one argument a `--- query` or `--- mutate` header takes, @@ -923,12 +946,6 @@ fn parse_expect_header(rest: &str) -> Result { }; return Ok(ExpectHeader::Outcome(outcome)); } - if let Some(var) = rest.strip_prefix("commit ") { - let var = var.trim(); - let name = parse_loop_var(var) - .map_err(|_| format!("`--- expect commit {var}`: a name is $[a-z][a-z0-9_]*"))?; - return Ok(ExpectHeader::Commit(name)); - } if rest == "unordered" { return Ok(ExpectHeader::Unordered); } @@ -1368,23 +1385,14 @@ enum Pending { id: Option, prefix: Vec, }, - /// A `--- concurrent` block awaiting the bare `--- expect` that names - /// each session's outcome. - Concurrent(ConcurrentStep), - Diff { - ordinal: usize, - sides: DiffSides, - }, - MergeBase { - ordinal: usize, - source: String, - target: String, - }, - MergeDryRun { + BranchRead { ordinal: usize, + read: BranchRead, source: String, - target: String, }, + /// A `--- concurrent` block awaiting the bare `--- expect` that names + /// each session's outcome. + Concurrent(ConcurrentStep), } /// Whether `text` holds the statement ` ` at any position: @@ -1484,10 +1492,7 @@ fn complete_list_step( needle: needle.clone(), } } - ExpectHeader::Ok - | ExpectHeader::Affected { .. } - | ExpectHeader::Outcome(_) - | ExpectHeader::Commit(_) => { + ExpectHeader::Ok | ExpectHeader::Affected { .. } | ExpectHeader::Outcome(_) => { return Err("`branch list` takes `unordered`, `ordered`, or `error:`".into()); } }; @@ -1548,8 +1553,7 @@ fn complete_show_step( ExpectHeader::Error(_) | ExpectHeader::Ok | ExpectHeader::Affected { .. } - | ExpectHeader::Outcome(_) - | ExpectHeader::Commit(_) => { + | ExpectHeader::Outcome(_) => { return Err(format!( "`{}` takes `unordered` or `ordered`; it returns the settings rows and does not fail", show_statement_name(id) @@ -1587,111 +1591,8 @@ fn complete_settings_step( | ExpectHeader::Ordered | ExpectHeader::Error(_) | ExpectHeader::Affected { .. } - | ExpectHeader::Outcome(_) - | ExpectHeader::Commit(_) => { - Err("a settings step takes `ok`; `set` and `reset` return no rows and no counts".into()) - } - } -} - -fn commit_expect_refusal() -> String { - "`expect commit $name` is accepted on a `--- merge-base` step only".into() -} - -/// The `key: value` arguments of a step header, each key once and from -/// `allowed`. -fn parse_step_arguments( - kind: &str, - rest: &str, - allowed: &[&str], -) -> Result, String> { - let mut args = HashMap::new(); - let mut tokens = rest.split_whitespace(); - while let Some(token) = tokens.next() { - let Some(key) = token.strip_suffix(':').filter(|key| allowed.contains(key)) else { - let keys: Vec = allowed.iter().map(|key| format!("`{key}:`")).collect(); - return Err(format!( - "`--- {kind}` takes {} as `key: value`, got `{token}`", - keys.join(", ") - )); - }; - let Some(value) = tokens.next() else { - return Err(format!("`--- {kind}` argument `{key}:` needs a value")); - }; - if args.insert(key.to_string(), value.to_string()).is_some() { - return Err(format!("`--- {kind}` names `{key}:` twice")); - } - } - Ok(args) -} - -/// A diff step's expect: change rows, one JSON object per line, or an error. -fn complete_diff_expect(mode: &ExpectHeader, section: &Section<'_>) -> Result { - match mode { - ExpectHeader::Ordered | ExpectHeader::Unordered => { - let mut rows = Vec::new(); - for (index, line) in §ion.body { - let line = line.trim(); - if line.is_empty() { - continue; - } - let row: Value = serde_json::from_str(line).map_err(|e| { - format!("line {}: a diff row is one JSON object: {e}", index + 1) - })?; - if !row.is_object() { - return Err(format!("line {}: a diff row is one JSON object", index + 1)); - } - rows.push(row); - } - Ok(DiffExpect::Rows { - ordered: matches!(mode, ExpectHeader::Ordered), - rows, - }) - } - ExpectHeader::Error(needle) => { - refuse_nonempty_body(§ion.body, "an `expect error:` section")?; - Ok(DiffExpect::Error { - needle: needle.clone(), - }) - } - ExpectHeader::Ok - | ExpectHeader::Affected { .. } - | ExpectHeader::Outcome(_) - | ExpectHeader::Commit(_) => { - Err("a `--- diff` step takes `ordered` or `unordered` rows, or `error:`".into()) - } - } -} - -/// A merge-base step's expect: `commit $name` naming a bound commit, or an -/// error. -fn complete_merge_base_expect( - mode: &ExpectHeader, - section: &Section<'_>, - bound_names: &HashSet, -) -> Result { - match mode { - ExpectHeader::Commit(name) => { - refuse_nonempty_body(§ion.body, "an `expect commit` section")?; - if !bound_names.contains(name) { - return Err(format!( - "`expect commit ${name}`: `${name}` is not bound; `--- bind commit ${name}` must precede this step" - )); - } - Ok(MergeBaseExpect::Commit { name: name.clone() }) - } - ExpectHeader::Error(needle) => { - refuse_nonempty_body(§ion.body, "an `expect error:` section")?; - Ok(MergeBaseExpect::Error { - needle: needle.clone(), - }) - } - ExpectHeader::Ok - | ExpectHeader::Ordered - | ExpectHeader::Unordered - | ExpectHeader::Affected { .. } | ExpectHeader::Outcome(_) => { - Err("a `--- merge-base` step takes `commit $name` or `error:`".into()) + Err("a settings step takes `ok`; `set` and `reset` return no rows and no counts".into()) } } } @@ -1725,7 +1626,6 @@ fn complete_write_expect( ExpectHeader::Outcome(_) => { Err("`expect outcome:` is accepted on a `branch merge` step only".into()) } - ExpectHeader::Commit(_) => Err(commit_expect_refusal()), ExpectHeader::Affected { .. } => Err(affected_refusal(name)), ExpectHeader::Unordered | ExpectHeader::Ordered => Err(rows_refusal(name)), } @@ -1841,7 +1741,6 @@ fn complete_decl_step( (ExpectHeader::Outcome(_), _) => { Err("`expect outcome:` is accepted on a `branch merge` step only".into()) } - (ExpectHeader::Commit(_), _) => Err(commit_expect_refusal()), (ExpectHeader::Unordered | ExpectHeader::Ordered, true) => Err( "a mutate step takes `ok`, `affected:`, or `error:`; mutation results carry no rows" .into(), @@ -2032,7 +1931,7 @@ pub fn parse_case(stem: &str, text: &str) -> Result { awaiting: Option, ) -> Result<(), String> { match awaiting { - Some(step) if matches!(step, Step::Show(_)) => { + Some(step) if matches!(step, Step::Show(_) | Step::BranchRead(_)) => { push_step(items, open_loop, step); Ok(()) } @@ -2061,17 +1960,7 @@ pub fn parse_case(stem: &str, text: &str) -> Result { "invalid_case: a seam must directly precede its mutate step (a GQ mutation or a branch statement) or load step; no seam is crossed by a query step yet".into(), ); } - if matches!( - kind, - "query" - | "mutate" - | "load" - | "restart" - | "concurrent" - | "diff" - | "merge-base" - | "merge-dry-run" - ) { + if matches!(kind, "query" | "mutate" | "load" | "restart" | "concurrent") { source_lines.insert(ordinal + 1, section.header_line + 1); awaiting_seam_step = false; } @@ -2217,10 +2106,10 @@ pub fn parse_case(stem: &str, text: &str) -> Result { ); } if kind == "mutate" && !stmt.is_write() { - return Err( - "`branch list` under `--- mutate` is refused; use `--- query`" - .into(), - ); + return Err(format!( + "`{}` under `--- mutate` is refused; use `--- query`", + stmt.statement_name() + )); } if branch.is_some() { return Err( @@ -2231,6 +2120,30 @@ pub fn parse_case(stem: &str, text: &str) -> Result { ordinal += 1; pending = Some(match stmt { BranchStmt::List => Pending::List { ordinal }, + BranchStmt::Read(read) => { + for operand in branch_read_operands(&read) { + if let Some(name) = bound_operand(operand) + && !bound_names.contains(name) + { + return Err(format!( + "`${{{name}}}` is not bound; `--- bind commit ${name}` must precede this step" + )); + } + } + let bound = branch_read_operands(&read) + .into_iter() + .filter(|operand| bound_operand(operand).is_some()) + .count(); + if bound > 0 && source.matches("${").count() == bound { + substitutable_lines + .extend(section.body.iter().map(|(i, _)| *i)); + } + Pending::BranchRead { + ordinal, + read, + source: source.clone(), + } + } BranchStmt::Write(write) => Pending::Control { ordinal, prefix: file.settings, @@ -2417,7 +2330,9 @@ pub fn parse_case(stem: &str, text: &str) -> Result { } let step = match pending.as_mut() { Some(Pending::Decl(step)) => step, - Some(Pending::List { .. } | Pending::Control { .. }) => { + Some( + Pending::List { .. } | Pending::Control { .. } | Pending::BranchRead { .. }, + ) => { return Err("a branch statement takes no params".into()); } Some(Pending::Settings { .. } | Pending::Show { .. }) => { @@ -2429,15 +2344,6 @@ pub fn parse_case(stem: &str, text: &str) -> Result { Some(Pending::Load { .. }) => { return Err("a generated load takes no params".into()); } - Some( - Pending::Diff { .. } - | Pending::MergeBase { .. } - | Pending::MergeDryRun { .. }, - ) => { - return Err( - "a diff, merge-base or merge-dry-run step takes no params".into() - ); - } None => { return Err(format!( "line {}: `--- params` must directly follow a query or mutate section", @@ -2514,6 +2420,13 @@ pub fn parse_case(stem: &str, text: &str) -> Result { push_step(&mut items, &mut open_loop, step); continue; } + if let Step::BranchRead(read) = &step { + return Err(format!( + "line {}: `{}` answers its own columns; drop the `--- expect shape` section", + section.header_line + 1, + read.read.statement_name() + )); + } let Some(QueryExpect::Rows { shape, .. }) = step.read_expect_mut() else { return Err(format!( "line {}: internal: the step awaiting a shape section carries no rows expect", @@ -2615,31 +2528,19 @@ pub fn parse_case(stem: &str, text: &str) -> Result { Some(Pending::Concurrent(_)) => { unreachable!("a pending concurrent block is completed above") } - Some(Pending::Diff { ordinal, sides }) => Step::Diff(DiffStep { - ordinal, - sides, - expect: complete_diff_expect(&mode, section)?, - }), - Some(Pending::MergeBase { + Some(Pending::BranchRead { ordinal, + read, source, - target, - }) => Step::MergeBase(MergeBaseStep { - ordinal, - source, - target, - expect: complete_merge_base_expect(&mode, section, &bound_names)?, - }), - Some(Pending::MergeDryRun { + }) => Step::BranchRead(complete_branch_read_step( ordinal, + read, source, - target, - }) => Step::MergeDryRun(MergeDryRunStep { - ordinal, - source, - target, - expect: complete_merge_expect(&mode, section)?, - }), + &mode, + section, + loop_var, + &bound_names, + )?), None => { return Err(format!( "line {}: `--- expect` has no query or mutate step to bind to", @@ -2654,83 +2555,6 @@ pub fn parse_case(stem: &str, text: &str) -> Result { push_step(&mut items, &mut open_loop, completed); } } - "diff" | "merge-base" | "merge-dry-run" => { - if pending.is_some() { - return Err(format!( - "line {}: the previous step is missing its `--- expect`", - section.header_line + 1 - )); - } - refuse_nonempty_body(§ion.body, &format!("`--- {kind}`"))?; - // A `$name` side names a bound commit; it must be bound before. - let side = |value: &String| -> Result { - if let Some(name) = value.strip_prefix('$') - && !bound_names.contains(name) - { - return Err(format!( - "`--- {kind}` side `{value}` is not bound; `--- bind commit {value}` must precede this step" - )); - } - Ok(value.clone()) - }; - ordinal += 1; - pending = Some(if kind == "diff" { - let args = parse_step_arguments( - kind, - rest, - &["from", "to", "source", "target", "commit"], - )?; - let sides = match ( - args.get("from"), - args.get("to"), - args.get("source"), - args.get("target"), - args.get("commit"), - ) { - (Some(from), Some(to), None, None, None) => DiffSides::Between { - from: side(from)?, - to: side(to)?, - }, - (None, None, Some(source), target, None) => DiffSides::Branch { - source: source.clone(), - target: target.cloned().unwrap_or_else(|| MAIN_BRANCH.to_string()), - }, - (None, None, None, None, Some(commit)) => DiffSides::Commit { - commit: side(commit)?, - }, - _ => { - return Err( - "`--- diff` takes `from:` and `to:`, `source:` with an optional `target:`, or `commit:`" - .into(), - ); - } - }; - Pending::Diff { ordinal, sides } - } else { - let args = parse_step_arguments(kind, rest, &["source", "target"])?; - let source = args - .get("source") - .cloned() - .ok_or_else(|| format!("`--- {kind}` needs `source:`"))?; - let target = args - .get("target") - .cloned() - .unwrap_or_else(|| MAIN_BRANCH.to_string()); - if kind == "merge-base" { - Pending::MergeBase { - ordinal, - source, - target, - } - } else { - Pending::MergeDryRun { - ordinal, - source, - target, - } - } - }); - } "bind" => { if pending.is_some() { return Err(format!( @@ -3644,6 +3468,56 @@ async fn run_list_step( ) } +/// Runs a branch control read through `Session::branch_read`, its bound +/// operands replaced, and holds its rows (rendered as the read door renders +/// them) or its error against the expect. +async fn run_branch_read_step( + host: &impl ExecutionHost, + session: &Session, + step: &BranchReadStep, + binding: Option<(&str, &str)>, + bindings: &Bindings, +) -> Result<(), StepFail> { + let statement = step.read.statement_name(); + let label = step_label(step.ordinal, statement, binding); + let fail = |message: String| StepFail { + label: label.clone(), + message, + bless_lines: None, + }; + let read = resolve_branch_read(&step.read, bindings).map_err(&fail)?; + let outcome = operation(host, step.ordinal, session.branch_read(&read)) + .await + .map_err(&fail)?; + match &step.expect { + QueryExpect::Rows { + ordered, + body_raw, + span, + .. + } => { + let answer = outcome.map_err(|e| fail(format!("`{statement}` failed: {e}")))?; + let output = omnigraph_api_types::branch_read_output(&answer) + .map_err(|e| fail(format!("`{statement}` rows failed to render: {e}")))?; + let rows: Vec = serde_json::from_str(output.rows.get()) + .map_err(|e| fail(format!("`{statement}` rows are not a JSON array: {e}")))?; + check_rows_json( + host, + &label, + &rows, + *ordered, + &substitute_bound(body_raw, bindings), + *span, + binding, + ) + } + QueryExpect::Error { needle } => { + check_error_expect(host, needle, outcome, &format!("`{statement}` succeeded")) + .map_err(fail) + } + } +} + /// Holds a read expect against a result the runner built itself (`branch /// list`, `show`): the result's own schema is the inferred one, so the /// shape check has no compiler to disagree with. @@ -3898,249 +3772,6 @@ async fn run_control_step( } } -/// A diff side as the engine and server take it: `$name` replaced by the -/// commit it names, anything else as spelled. -pub fn resolve_side(side: &str, bindings: &Bindings) -> Result { - match side.strip_prefix('$') { - Some(name) => bindings - .get(name) - .cloned() - .ok_or_else(|| format!("`{side}` is not bound at this step")), - None => Ok(side.to_string()), - } -} - -/// One change as a diff step compares it: its wire spelling with the type -/// named rather than identified, since the opaque type id differs between -/// graphs. -pub fn diff_row(change: &omnigraph_api_types::EntityChangeOutput) -> Value { - let mut row = serde_json::to_value(change).expect("change outputs serialize"); - row["type"] = Value::String(change.r#type.name.clone()); - row -} - -/// Judge a diff step's rows (in order or as a multiset) or its error. -pub fn check_diff_expect( - host: &impl ExecutionHost, - expect: &DiffExpect, - outcome: Result, String>, -) -> Result<(), String> { - match expect { - DiffExpect::Rows { ordered, rows } => { - let got = outcome.map_err(|e| format!("`diff` failed: {e}"))?; - let render = |rows: &[Value]| -> Vec { - let mut lines: Vec = rows.iter().map(canonical_json).collect(); - if !ordered { - lines.sort(); - } - lines - }; - let (want, got) = (render(rows), render(&got)); - if want == got { - Ok(()) - } else { - Err(format!( - "diff rows mismatch\n expected:\n {}\n actual:\n {}", - want.join("\n "), - got.join("\n ") - )) - } - } - DiffExpect::Error { needle } => { - check_error_expect(host, needle, outcome, "the diff succeeded") - } - } -} - -/// Every change a diff step names, page by page through the engine. -async fn engine_diff_rows( - session: &Session, - sides: &DiffSides, - bindings: &Bindings, -) -> Result, String> { - let scope = omnigraph::changes::ChangeFeedScope::default(); - let mut rows = Vec::new(); - let mut token: Option = None; - if let DiffSides::Commit { commit } = sides { - let id = resolve_side(commit, bindings)?; - loop { - let page = session - .commit_changes_page(&id, &scope, token.as_deref(), None, None) - .await - .map_err(|e| e.to_string())?; - rows.extend( - page.block - .changes - .iter() - .map(|change| diff_row(&omnigraph_api_types::entity_change_output(change))), - ); - match page.next_page_token { - Some(next) => token = Some(next), - None => return Ok(rows), - } - } - } - let sides = match sides { - DiffSides::Between { from, to } => omnigraph::changes::GraphDiffSides::Between { - from: resolve_side(from, bindings)?, - to: resolve_side(to, bindings)?, - }, - DiffSides::Branch { source, target } => omnigraph::changes::GraphDiffSides::Branch { - source: source.clone(), - target: target.clone(), - }, - DiffSides::Commit { .. } => unreachable!("the commit form returned above"), - }; - loop { - let page = session - .diff_page(&omnigraph::changes::GraphDiffRequest { - sides: sides.clone(), - scope: scope.clone(), - page_token: token.clone(), - max_changes: None, - max_bytes: None, - }) - .await - .map_err(|e| e.to_string())?; - rows.extend( - page.changes - .iter() - .map(|change| diff_row(&omnigraph_api_types::entity_change_output(change))), - ); - match page.next_page_token { - Some(next) => token = Some(next), - None => return Ok(rows), - } - } -} - -async fn run_diff_step( - host: &impl ExecutionHost, - session: &Session, - step: &DiffStep, - binding: Option<(&str, &str)>, - bindings: &Bindings, -) -> Result<(), StepFail> { - let label = step_label(step.ordinal, "diff", binding); - let fail = |message: String| StepFail { - label: label.clone(), - message, - bless_lines: None, - }; - let outcome = operation( - host, - step.ordinal, - engine_diff_rows(session, &step.sides, bindings), - ) - .await - .map_err(&fail)?; - host.observe(|| format!("actual diff: {outcome:?}")); - check_diff_expect(host, &step.expect, outcome).map_err(fail) -} - -/// Judge a merge-base step's commit against the bound one, or its error. -pub fn check_merge_base_expect( - host: &impl ExecutionHost, - expect: &MergeBaseExpect, - outcome: Result, - bindings: &Bindings, -) -> Result<(), String> { - match expect { - MergeBaseExpect::Commit { name } => { - let want = bindings - .get(name) - .ok_or_else(|| format!("`${name}` is not bound at this step"))?; - let got = outcome.map_err(|e| format!("`merge-base` failed: {e}"))?; - if &got == want { - Ok(()) - } else { - Err(format!( - "merge base mismatch: expected `${name}` ({want}), got {got}" - )) - } - } - MergeBaseExpect::Error { needle } => { - check_error_expect(host, needle, outcome, "`merge-base` succeeded") - } - } -} - -async fn run_merge_base_step( - host: &impl ExecutionHost, - session: &Session, - step: &MergeBaseStep, - binding: Option<(&str, &str)>, - bindings: &Bindings, -) -> Result<(), StepFail> { - let label = step_label(step.ordinal, "merge-base", binding); - let fail = |message: String| StepFail { - label: label.clone(), - message, - bless_lines: None, - }; - let outcome = operation( - host, - step.ordinal, - session.merge_base(&step.source, &step.target), - ) - .await - .map_err(&fail)? - .map(|base| base.merge_base.graph_commit_id) - .map_err(|e| e.to_string()); - host.observe(|| format!("actual merge base: {outcome:?}")); - check_merge_base_expect(host, &step.expect, outcome, bindings).map_err(fail) -} - -/// Judge a dry run as a merge step is judged; a dry run that reports a -/// commit fails whatever was expected. -pub fn check_dry_run_expect( - host: &impl ExecutionHost, - expect: &MergeExpect, - outcome: Result<(MergeOutcome, bool), String>, -) -> Result<(), String> { - if let Ok((_, true)) = &outcome { - return Err("a dry run reported a commit".into()); - } - let outcome = outcome.map(|(outcome, _)| outcome); - match expect { - MergeExpect::Write(expect) => check_write_expect(host, "merge-dry-run", expect, outcome), - MergeExpect::Outcome(want) => match outcome { - Ok(got) if got == *want => Ok(()), - Ok(got) => Err(format!( - "merge outcome mismatch: expected `{}`, got `{}`", - merge_outcome_word(*want), - merge_outcome_word(got) - )), - Err(e) => Err(format!("`merge-dry-run` failed: {e}")), - }, - } -} - -async fn run_merge_dry_run_step( - host: &impl ExecutionHost, - session: &Session, - step: &MergeDryRunStep, - binding: Option<(&str, &str)>, -) -> Result<(), StepFail> { - let label = step_label(step.ordinal, "merge-dry-run", binding); - let fail = |message: String| StepFail { - label: label.clone(), - message, - bless_lines: None, - }; - let outcome = operation( - host, - step.ordinal, - session.branch_merge_dry_run(&step.source, &step.target), - ) - .await - .map_err(&fail)? - .map(|result| (result.outcome, result.commit.is_some())) - .map_err(|e| e.to_string()); - host.observe(|| format!("actual dry run: {outcome:?}")); - check_dry_run_expect(host, &step.expect, outcome).map_err(fail) -} - async fn run_mutate_step( host: &impl ExecutionHost, session: &Session, @@ -4437,14 +4068,10 @@ async fn execute_steps_inner( Step::List(l) => run_list_step(host, &session, l, binding).await, Step::Settings(s) => run_settings_step(host, &mut session, s, binding), Step::Show(s) => run_show_step(host, &session, s, binding), - Step::Concurrent(c) => host.concurrent_step(&session, case, c).await, - Step::Diff(d) => run_diff_step(host, &session, d, binding, &bindings).await, - Step::MergeBase(m) => { - run_merge_base_step(host, &session, m, binding, &bindings).await - } - Step::MergeDryRun(m) => { - run_merge_dry_run_step(host, &session, m, binding).await + Step::BranchRead(r) => { + run_branch_read_step(host, &session, r, binding, &bindings).await } + Step::Concurrent(c) => host.concurrent_step(&session, case, c).await, Step::Restart { ordinal } => { generation += 1; host.observe(|| format!("lifetime: reopen generation {generation}")); @@ -4592,24 +4219,8 @@ fn expectation_evidence(step: &Step) -> Value { serde_json::json!({"kind": "settings", "statements": format!("{:?}", s.statements)}) } Step::Show(s) => read_expect_evidence(&s.expect), + Step::BranchRead(r) => read_expect_evidence(&r.expect), Step::Restart { .. } => serde_json::json!({"kind": "restart", "storage": "preserved"}), - Step::Diff(d) => serde_json::json!({ - "kind": "diff", - "sides": format!("{:?}", d.sides), - "expect": format!("{:?}", d.expect), - }), - Step::MergeBase(m) => serde_json::json!({ - "kind": "merge-base", - "source": m.source, - "target": m.target, - "expect": format!("{:?}", m.expect), - }), - Step::MergeDryRun(m) => serde_json::json!({ - "kind": "merge-dry-run", - "source": m.source, - "target": m.target, - "expect": format!("{:?}", m.expect), - }), Step::Concurrent(c) => serde_json::json!({ "kind": "concurrent", "sessions": c.sessions.iter().map(|s| serde_json::json!({"label": s.label, "branch": s.branch, "kind": s.kind.name(), "expect": format!("{:?}", s.expect)})).collect::>(), @@ -4715,11 +4326,9 @@ pub enum StepKind { CommitDrop, Settings, Show, + BranchRead, Restart, Concurrent, - Diff, - MergeBase, - MergeDryRun, } #[derive(Clone, Debug)] @@ -4788,11 +4397,9 @@ impl Step { Self::List(_) => StepKind::BranchList, Self::Settings(_) => StepKind::Settings, Self::Show(_) => StepKind::Show, + Self::BranchRead(_) => StepKind::BranchRead, Self::Restart { .. } => StepKind::Restart, Self::Concurrent(_) => StepKind::Concurrent, - Self::Diff(_) => StepKind::Diff, - Self::MergeBase(_) => StepKind::MergeBase, - Self::MergeDryRun(_) => StepKind::MergeDryRun, } } } diff --git a/crates/omnigraph-gqt-core/src/served.rs b/crates/omnigraph-gqt-core/src/served.rs index 1fdb7350c..514bddcf0 100644 --- a/crates/omnigraph-gqt-core/src/served.rs +++ b/crates/omnigraph-gqt-core/src/served.rs @@ -8,9 +8,8 @@ use futures::FutureExt as _; use omnigraph::db::MergeOutcome; use omnigraph::loader::LoadMode; use omnigraph_api_types::{ - BranchMergeOutcome, BranchMergeOutput, BranchMergeRequest, BranchOutcomeOutput, ChangeOutput, - ChangeRequest, CommitChangesOutput, DiffOutput, HTTP_API_CONTRACT, HTTP_API_CONTRACT_HEADER, - IngestRequest, MergeBaseOutput, QueryRequest, ReadOutput, SchemaOutput, + BranchMergeOutcome, BranchOutcomeOutput, ChangeOutput, ChangeRequest, HTTP_API_CONTRACT, + HTTP_API_CONTRACT_HEADER, IngestRequest, QueryRequest, ReadOutput, SchemaOutput, }; use omnigraph_compiler::query::ast::show_statement_name; use serde::de::DeserializeOwned; @@ -19,13 +18,12 @@ use serde_json::Value; use std::time::Duration; use crate::{ - Bindings, Case, ControlStep, ControlWrite, DiffSides, DiffStep, ExecutionHost, Fixture, Item, - ListStep, LoadStep, MAIN_BRANCH, MergeBaseStep, MergeDryRunStep, MergeExpect, MutateExpect, - MutateStep, ParamsInput, QueryExpect, QueryStep, Seed, ShowStep, Step, StepFail, WriteExpect, - bound_snapshot, bound_source, check_diff_expect, check_dry_run_expect, check_error_expect, - check_merge_base_expect, check_rows_json, diff_row, expectation_evidence, merge_outcome_word, - operation, record_binding, resolve_operand, resolve_side, step_kind, step_label, - substitute_all, + Bindings, BranchReadStep, Case, ControlStep, ControlWrite, ExecutionHost, Fixture, Item, + ListStep, LoadStep, MAIN_BRANCH, MergeExpect, MutateExpect, MutateStep, ParamsInput, + QueryExpect, QueryStep, Seed, ShowStep, Step, StepFail, WriteExpect, bound_snapshot, + bound_source, check_error_expect, check_rows_json, expectation_evidence, merge_outcome_word, + operation, record_binding, resolve_operand, step_kind, step_label, substitute_all, + substitute_bound, }; /// The server a `--server` run addresses: its base URL, the graph id under @@ -100,9 +98,7 @@ pub fn admit_served(case: &Case) -> Result<(), String> { | Step::Control(_) | Step::List(_) | Step::Show(_) - | Step::Diff(_) - | Step::MergeBase(_) - | Step::MergeDryRun(_) => {} + | Step::BranchRead(_) => {} } } if refused.is_empty() { @@ -194,9 +190,7 @@ async fn execute_served_inner( }), Step::List(l) => list(host, &client, l, binding).await, Step::Show(s) => show(host, &client, s, binding).await, - Step::Diff(d) => diff(host, &client, d, binding, &bindings).await, - Step::MergeBase(m) => merge_base(host, &client, m, binding, &bindings).await, - Step::MergeDryRun(m) => merge_dry_run(host, &client, m, binding).await, + Step::BranchRead(r) => branch_read(host, &client, r, binding, &bindings).await, Step::Settings(_) | Step::Restart { .. } | Step::Concurrent(_) => { Err(StepFail::new( step_label(ordinal, step_kind(step), binding), @@ -369,20 +363,6 @@ impl ServerClient { .await } - /// A GET whose query parameters the client encodes. - async fn get_query( - &self, - route: &str, - query: &[(&str, &str)], - ) -> Result, String> { - self.send( - self.http - .get(format!("{}{route}", self.graph_base)) - .query(query), - ) - .await - } - async fn load(&self, branch: &str, mode: LoadMode, data: String) -> Result<(), String> { let request = IngestRequest { branch: Some(branch.to_string()), @@ -899,6 +879,48 @@ async fn show( ) } +/// A branch control read at the `/query` door: the statement sent with its +/// bound names replaced, its rows held against the expect with the same +/// names replaced in the expected rows. +async fn branch_read( + host: &impl ExecutionHost, + client: &ServerClient, + step: &BranchReadStep, + binding: Option<(&str, &str)>, + bindings: &Bindings, +) -> Result<(), StepFail> { + let name = step.read.statement_name(); + let label = step_label(step.ordinal, name, binding); + let fail = |message: String| StepFail::new(label.clone(), message); + let text = substitute_bound(&step.source, bindings); + let outcome = operation(host, step.ordinal, statement(client, &text)) + .await + .map_err(&fail)? + .map_err(&fail)?; + match &step.expect { + QueryExpect::Rows { + ordered, + body_raw, + span, + .. + } => { + let actual = outcome.map_err(|e| fail(format!("`{name}` failed: {e}")))?; + check_rows_json( + host, + &label, + &actual, + *ordered, + &substitute_bound(body_raw, bindings), + *span, + binding, + ) + } + QueryExpect::Error { needle } => { + check_error_expect(host, needle, outcome, &format!("`{name}` succeeded")).map_err(fail) + } + } +} + /// A read statement (`branch list`, `show …`) at the `/query` door, sent /// bare: no name, params, branch or snapshot, as the route documents. async fn statement( @@ -915,152 +937,3 @@ async fn statement( }; client.read(&request).await } - -/// Every change a diff step names, page by page through its route: -/// `GET /diff`, `GET /branches/diff` or `GET /commits/{id}/changes`. -async fn served_diff_rows( - client: &ServerClient, - sides: &DiffSides, - bindings: &Bindings, -) -> Result, String>, String> { - let mut rows = Vec::new(); - let mut token: Option = None; - loop { - let from: String; - let to: String; - let mut query: Vec<(&str, &str)> = Vec::new(); - let page_changes = match sides { - DiffSides::Commit { commit } => { - let id = resolve_side(commit, bindings)?; - if let Some(token) = token.as_deref() { - query.push(("page_token", token)); - } - let route = format!("/commits/{id}/changes"); - match client - .get_query::(&route, &query) - .await? - { - Ok(page) => Ok((page.changes, page.next_page_token)), - Err(error) => Err(error), - } - } - DiffSides::Between { from: f, to: t } => { - from = resolve_side(f, bindings)?; - to = resolve_side(t, bindings)?; - query.extend([("from", from.as_str()), ("to", to.as_str())]); - if let Some(token) = token.as_deref() { - query.push(("page_token", token)); - } - match client.get_query::("/diff", &query).await? { - Ok(page) => Ok((page.changes, page.next_page_token)), - Err(error) => Err(error), - } - } - DiffSides::Branch { source, target } => { - query.extend([("source", source.as_str()), ("target", target.as_str())]); - if let Some(token) = token.as_deref() { - query.push(("page_token", token)); - } - match client - .get_query::("/branches/diff", &query) - .await? - { - Ok(page) => Ok((page.changes, page.next_page_token)), - Err(error) => Err(error), - } - } - }; - let (changes, next) = match page_changes { - Ok(page) => page, - Err(error) => return Ok(Err(error)), - }; - rows.extend(changes.iter().map(diff_row)); - match next { - Some(next) => token = Some(next), - None => return Ok(Ok(rows)), - } - } -} - -async fn diff( - host: &impl ExecutionHost, - client: &ServerClient, - step: &DiffStep, - binding: Option<(&str, &str)>, - bindings: &Bindings, -) -> Result<(), StepFail> { - let label = step_label(step.ordinal(), "diff", binding); - let fail = |message: String| StepFail::new(label.clone(), message); - let outcome = operation( - host, - step.ordinal(), - served_diff_rows(client, &step.sides, bindings), - ) - .await - .map_err(&fail)? - .map_err(&fail)?; - host.observe(|| format!("actual diff: {outcome:?}")); - check_diff_expect(host, step.expect(), outcome).map_err(fail) -} - -async fn merge_base( - host: &impl ExecutionHost, - client: &ServerClient, - step: &MergeBaseStep, - binding: Option<(&str, &str)>, - bindings: &Bindings, -) -> Result<(), StepFail> { - let label = step_label(step.ordinal(), "merge-base", binding); - let fail = |message: String| StepFail::new(label.clone(), message); - let query = [ - ("source", step.source.as_str()), - ("target", step.target.as_str()), - ]; - let outcome = operation( - host, - step.ordinal(), - client.get_query::("/branches/merge-base", &query), - ) - .await - .map_err(&fail)? - .map_err(&fail)? - .map(|output| output.merge_base.graph_commit_id); - host.observe(|| format!("actual merge base: {outcome:?}")); - check_merge_base_expect(host, step.expect(), outcome, bindings).map_err(fail) -} - -async fn merge_dry_run( - host: &impl ExecutionHost, - client: &ServerClient, - step: &MergeDryRunStep, - binding: Option<(&str, &str)>, -) -> Result<(), StepFail> { - let label = step_label(step.ordinal(), "merge-dry-run", binding); - let fail = |message: String| StepFail::new(label.clone(), message); - let request = BranchMergeRequest { - source: step.source.clone(), - target: Some(step.target.clone()), - delete_branch: false, - as_commit: false, - dry_run: true, - settings: None, - }; - let outcome = operation( - host, - step.ordinal(), - client.post::<_, BranchMergeOutput>("/branches/merge", &request), - ) - .await - .map_err(&fail)? - .map_err(&fail)? - .map(|output| { - let outcome = match output.outcome { - BranchMergeOutcome::AlreadyUpToDate => MergeOutcome::AlreadyUpToDate, - BranchMergeOutcome::FastForward => MergeOutcome::FastForward, - BranchMergeOutcome::Merged => MergeOutcome::Merged, - }; - (outcome, output.commit.is_some() || !output.dry_run) - }); - host.observe(|| format!("actual dry run: {outcome:?}")); - check_dry_run_expect(host, step.expect(), outcome).map_err(fail) -} diff --git a/crates/omnigraph-gqt-core/src/tests.rs b/crates/omnigraph-gqt-core/src/tests.rs index 1a47a3b91..842543d95 100644 --- a/crates/omnigraph-gqt-core/src/tests.rs +++ b/crates/omnigraph-gqt-core/src/tests.rs @@ -568,6 +568,55 @@ fn refuses_mutation_declaration_under_query() { assert!(refusal("x", &text).contains("use `--- mutate`")); } +/// A branch control read is a `--- query` rows step whose shape is its own: +/// it carries no shape section, takes `${name}` for a bound commit in its +/// sides and its expected rows, and is refused under `--- mutate`, with an +/// unbound name, or with a shape section. +#[test] +fn branch_read_steps_parse_and_refuse() { + const BIND: &str = "--- mutate\nquery ins() {\n insert Person { name: \"bob\" }\n}\n--- expect ok\n--- bind commit $c\n"; + let accepted = format!( + "{HDR}{SCHEMA}{SEED}{BIND}\ + --- query\nbranch diff \"${{c}}\" to main\n--- expect ordered\n\ + --- query\nbranch merge-base main\n--- expect ordered\n\ + {{\"source\": \"main\", \"target\": \"main\", \"source_commit_id\": \"${{c}}\", \"target_commit_id\": \"${{c}}\", \"merge_base_commit_id\": \"${{c}}\"}}\n\ + --- query\nbranch merge main dry run\n--- expect error: nothing\n" + ); + let case = parse_case("x", &accepted).unwrap(); + let reads = case + .items + .iter() + .filter(|item| matches!(item, Item::Step(Step::BranchRead(_)))) + .count(); + assert_eq!(reads, 3); + + for (step, want) in [ + ( + "--- mutate\nbranch diff feature\n--- expect ok\n".to_string(), + "`branch diff` under `--- mutate` is refused; use `--- query`", + ), + ( + "--- query\nbranch diff \"${x}\" to main\n--- expect ordered\n".to_string(), + "`${x}` is not bound; `--- bind commit $x` must precede this step", + ), + ( + "--- query\nbranch merge-base feature\n--- expect ok\n".to_string(), + "`branch merge-base` takes `unordered`, `ordered`, or `error:`", + ), + ] { + let text = format!("{HDR}{SCHEMA}{SEED}{step}"); + assert_eq!(refusal("x", &text), want, "{step}"); + } + let shaped = format!( + "{HDR}{SCHEMA}{SEED}--- query\nbranch merge feature dry run\n--- expect ordered\n--- expect shape\noutcome: String\n" + ); + assert!( + refusal("x", &shaped) + .contains("answers its own columns; drop the `--- expect shape` section"), + "{shaped}" + ); +} + const CREATE: &str = "--- mutate\nbranch create b0\n--- expect ok\n"; const LIST: &str = "--- query\nbranch list\n--- expect unordered\n{\"name\": \"b0\"}\n{\"name\": \"main\"}\n--- expect shape\nname: String\n"; @@ -666,131 +715,6 @@ fn refuses_outcome_on_any_step_but_a_merge() { } } -#[test] -fn diff_steps_parse_their_sides_and_expects() { - let bind = "--- mutate\nquery add() {\n insert Person { name: \"bob\" }\n}\n--- expect ok\n--- bind commit $c\n"; - let text = format!( - "{HDR}{SCHEMA}{SEED}{bind}\ - --- diff from: $c to: main\n--- expect ordered\n{{\"id\": \"bob\"}}\n\ - --- diff source: feature\n--- expect unordered\n\ - --- diff commit: $c\n--- expect error: nope\n\ - --- merge-base source: feature target: main\n--- expect commit $c\n\ - --- merge-dry-run source: feature\n--- expect outcome: merged\n" - ); - let case = parse_case("x", &text).unwrap(); - let steps: Vec<&Step> = case - .items - .iter() - .map(|item| match item { - Item::Step(step) => step, - Item::Loop { .. } => panic!("no loops here"), - }) - .collect(); - let [ - _, - Step::Diff(between), - Step::Diff(branch), - Step::Diff(commit), - Step::MergeBase(base), - Step::MergeDryRun(dry), - ] = steps.as_slice() - else { - panic!("unexpected steps: {steps:?}"); - }; - assert_eq!( - between.sides, - DiffSides::Between { - from: "$c".into(), - to: "main".into() - } - ); - assert!(matches!( - between.expect(), - DiffExpect::Rows { ordered: true, rows } if rows.len() == 1 - )); - assert_eq!( - branch.sides, - DiffSides::Branch { - source: "feature".into(), - target: "main".into() - } - ); - assert!(matches!( - branch.expect(), - DiffExpect::Rows { ordered: false, rows } if rows.is_empty() - )); - assert_eq!( - commit.sides, - DiffSides::Commit { - commit: "$c".into() - } - ); - assert!(matches!(commit.expect(), DiffExpect::Error { needle } if needle == "nope")); - assert_eq!( - (base.source.as_str(), base.target.as_str()), - ("feature", "main") - ); - assert!(matches!(base.expect(), MergeBaseExpect::Commit { name } if name == "c")); - assert_eq!( - (dry.source.as_str(), dry.target.as_str()), - ("feature", "main") - ); - assert!(matches!( - dry.expect(), - MergeExpect::Outcome(MergeOutcome::Merged) - )); -} - -#[test] -fn diff_steps_refuse_bad_arguments_and_expects() { - for (steps, want) in [ - ( - "--- diff from: $c to: main\n--- expect ordered\n", - "`--- diff` side `$c` is not bound", - ), - ( - "--- diff from: main source: feature\n--- expect ordered\n", - "`--- diff` takes `from:` and `to:`, `source:` with an optional `target:`, or `commit:`", - ), - ( - "--- diff branch: main\n--- expect ordered\n", - "`--- diff` takes `from:`, `to:`, `source:`, `target:`, `commit:` as `key: value`, got `branch:`", - ), - ( - "--- diff from: main to: main\n--- expect ok\n", - "a `--- diff` step takes `ordered` or `unordered` rows, or `error:`", - ), - ( - "--- diff from: main to: main\n--- expect ordered\n[1]\n", - "a diff row is one JSON object", - ), - ( - "--- merge-base target: main\n--- expect error: x\n", - "`--- merge-base` needs `source:`", - ), - ( - "--- merge-base source: feature\n--- expect commit $c\n", - "`expect commit $c`: `$c` is not bound", - ), - ( - "--- merge-dry-run source: feature\n--- expect ordered\n", - "rows", - ), - ( - "--- mutate\nbranch create b0\n--- expect commit $c\n", - "`expect commit $name` is accepted on a `--- merge-base` step only", - ), - ( - "--- diff from: main to: main\nbody\n--- expect ordered\n", - "`--- diff`", - ), - ] { - let text = format!("{HDR}{SCHEMA}{SEED}{steps}"); - let error = refusal("x", &text); - assert!(error.contains(want), "{steps}: {error}"); - } -} - #[test] fn outcome_takes_the_three_merge_words() { for (word, want) in [ diff --git a/crates/omnigraph-gqt/README.md b/crates/omnigraph-gqt/README.md index 9d6641342..2710b924c 100644 --- a/crates/omnigraph-gqt/README.md +++ b/crates/omnigraph-gqt/README.md @@ -130,21 +130,16 @@ inherits its base's recorded target. A name is bound once, before every use, outside a loop, and survives `--- restart`; the served path binds from the `ChangeOutput.commit` the route answered and sends `snapshot` on the read. -Three steps cover the graph diff surfaces (RFC 0030 §8). A `--- diff` step -names its sides as `from: to: ` (a two-point diff), -`source: ` with an optional `target: ` (a branch diff against -the merge base; `main` when unspelled), or `commit: ` (one commit's own -changes, `commit changes`). A side is a branch name or a bound `$name`. Its -expect is `ordered` or `unordered` rows, one JSON object per change in the -wire shape with `type` spelled as the type's name, or `error:`. Rows compare -as JSON whatever their key order; the step follows every page. -`--- merge-base source: [target: ]` expects -`--- expect commit $name`, the bound commit the merge base must be, or `error:`. -`--- merge-dry-run source: [target: ]` takes a merge's own -`outcome:` and `error:` expects and fails if the dry run reports a commit. The -engine and DST paths call the engine; the served path sends `GET /diff`, -`GET /branches/diff`, `GET /commits/{id}/changes`, `GET /branches/merge-base` -and `POST /branches/merge` with `dry_run: true`. +A `--- query` step may hold a branch control read (RFC 0030 §8.5): +`branch diff [into ]`, `branch diff to `, +`branch merge-base [into ]` or +`branch merge [into ] dry run`. It takes `unordered`, +`ordered` or `error:`; its rows are the statement's answer, compared as JSON +whatever their key order, and it carries no shape section, since its columns +are the statement's. A side spelled `"${name}"` runs with the commit +`--- bind commit $name` captured, and `${name}` in an expected row stands for +the same id. The engine and DST paths call `Session::branch_read`; the served +path sends the statement to `POST /query`. The configuration accepts 1–16 distinct environment parameter sets, 1–600000 milliseconds, and 1–64 distinct unsigned 64-bit seeds per DST diff --git a/crates/omnigraph-gqt/cases/graph_diff_branch_review.gqt b/crates/omnigraph-gqt/cases/graph_diff_branch_review.gqt index 3181c5e7a..d89557a7e 100644 --- a/crates/omnigraph-gqt/cases/graph_diff_branch_review.gqt +++ b/crates/omnigraph-gqt/cases/graph_diff_branch_review.gqt @@ -1,6 +1,6 @@ # issue: none -# notes: Reviewing a branch (RFC 0030 §8): the merge base, the branch diff from it to the source head, -# notes: the two-point diff of the heads, and a dry run that writes nothing. +# notes: Reviewing a branch with the branch control reads (RFC 0030 §8): the merge base, the branch diff +# notes: from it to the source head, the two-point diff of the heads, and a dry run that writes nothing. --- runner timeout_ms: 10000 @@ -51,6 +51,8 @@ query insert_carol() { --- expect affected: nodes=1 edges=0 +--- bind commit $carol + --- mutate query insert_dave() { insert Person { name: "dave", age: 50 } @@ -58,31 +60,40 @@ query insert_dave() { --- expect affected: nodes=1 edges=0 ---- merge-base source: feature +--- bind commit $dave ---- expect commit $fork +--- query +branch merge-base feature ---- diff source: feature target: main +--- expect ordered +{"source": "feature", "target": "main", "source_commit_id": "${carol}", "target_commit_id": "${dave}", "merge_base_commit_id": "${fork}"} + +--- query +branch diff feature into main --- expect ordered {"kind":"node","type":"Person","id":"alice","op":"update","before":{"properties":{"name":"alice","age":30}},"after":{"properties":{"name":"alice","age":31}}} {"kind":"node","type":"Person","id":"carol","op":"insert","after":{"properties":{"name":"carol","age":22}}} ---- diff from: main to: feature +--- query +branch diff main to feature --- expect ordered {"kind":"node","type":"Person","id":"alice","op":"update","before":{"properties":{"name":"alice","age":30}},"after":{"properties":{"name":"alice","age":31}}} {"kind":"node","type":"Person","id":"carol","op":"insert","after":{"properties":{"name":"carol","age":22}}} {"kind":"node","type":"Person","id":"dave","op":"delete","before":{"properties":{"name":"dave","age":50}}} ---- diff from: $fork to: main +--- query +branch diff "${fork}" to main --- expect ordered {"kind":"node","type":"Person","id":"dave","op":"insert","after":{"properties":{"name":"dave","age":50}}} ---- merge-dry-run source: feature +--- query +branch merge feature dry run ---- expect outcome: merged +--- expect ordered +{"source": "feature", "target": "main", "outcome": "merged"} --- query query names_on_main() { diff --git a/crates/omnigraph-gqt/cases/graph_diff_merge_proposal.gqt b/crates/omnigraph-gqt/cases/graph_diff_merge_proposal.gqt index e11164e77..013a6d746 100644 --- a/crates/omnigraph-gqt/cases/graph_diff_merge_proposal.gqt +++ b/crates/omnigraph-gqt/cases/graph_diff_merge_proposal.gqt @@ -1,6 +1,6 @@ # issue: none -# notes: A merge proposal under review (RFC 0030 §8): its own diff is the branch's changes, a diff from -# notes: main to the proposal names it by commit id, and once published the branch has nothing left to +# notes: A merge proposal under review (RFC 0030 §8): the diff from its parent is the branch's changes, a +# notes: diff from main to the proposal names it by commit id, and once published the branch has nothing left to # notes: merge and nothing left to review. --- runner @@ -50,6 +50,8 @@ query insert_dave() { --- expect affected: nodes=1 edges=0 +--- bind commit $dave + --- mutate branch merge feature into main as commit @@ -57,13 +59,15 @@ branch merge feature into main as commit --- bind commit $proposal ---- diff commit: $proposal +--- query +branch diff "${dave}" to "${proposal}" --- expect ordered {"kind":"node","type":"Person","id":"alice","op":"update","before":{"properties":{"name":"alice","age":30}},"after":{"properties":{"name":"alice","age":31}}} {"kind":"node","type":"Person","id":"carol","op":"insert","after":{"properties":{"name":"carol","age":22}}} ---- diff from: main to: $proposal +--- query +branch diff main to "${proposal}" --- expect ordered {"kind":"node","type":"Person","id":"alice","op":"update","before":{"properties":{"name":"alice","age":30}},"after":{"properties":{"name":"alice","age":31}}} @@ -74,10 +78,13 @@ commit publish "${proposal}" --- expect ok ---- merge-dry-run source: feature +--- query +branch merge feature dry run ---- expect outcome: already_up_to_date +--- expect ordered +{"source": "feature", "target": "main", "outcome": "already_up_to_date"} ---- diff source: feature +--- query +branch diff feature --- expect ordered diff --git a/crates/omnigraph-gqt/cases/graph_diff_net_range.gqt b/crates/omnigraph-gqt/cases/graph_diff_net_range.gqt index 2c6673849..a6dd9aa92 100644 --- a/crates/omnigraph-gqt/cases/graph_diff_net_range.gqt +++ b/crates/omnigraph-gqt/cases/graph_diff_net_range.gqt @@ -1,7 +1,7 @@ # issue: none # notes: A graph diff compares its endpoints and never folds history (RFC 0030 §8.2): a reverted update, # notes: an insert later deleted and an equal reinsert produce nothing; a changed reinsert is one update; -# notes: an insert later updated is one insert with the final image. A commit's own diff is its change. +# notes: an insert later updated is one insert with the final image. A diff from a commit's parent is its change. --- runner timeout_ms: 10000 @@ -97,6 +97,8 @@ query jill_arrives() { --- expect affected: nodes=1 edges=0 +--- bind commit $jill + --- mutate query jill_ages() { update Person set { age: 19 } where name = "jill" @@ -106,23 +108,27 @@ query jill_ages() { --- bind commit $last ---- diff from: $start to: main +--- query +branch diff "${start}" to main --- expect ordered {"kind":"node","type":"Person","id":"ivan","op":"update","before":{"properties":{"name":"ivan","age":20}},"after":{"properties":{"name":"ivan","age":21}}} {"kind":"node","type":"Person","id":"jill","op":"insert","after":{"properties":{"name":"jill","age":19}}} ---- diff commit: $last +--- query +branch diff "${jill}" to "${last}" --- expect ordered {"kind":"node","type":"Person","id":"jill","op":"update","before":{"properties":{"name":"jill","age":18}},"after":{"properties":{"name":"jill","age":19}}} ---- diff from: main to: $start +--- query +branch diff main to "${start}" --- expect unordered {"kind":"node","type":"Person","id":"jill","op":"delete","before":{"properties":{"name":"jill","age":19}}} {"kind":"node","type":"Person","id":"ivan","op":"update","before":{"properties":{"name":"ivan","age":21}},"after":{"properties":{"name":"ivan","age":20}}} ---- diff from: main to: main +--- query +branch diff main to main --- expect ordered diff --git a/crates/omnigraph-gqt/cases/graph_diff_refusals.gqt b/crates/omnigraph-gqt/cases/graph_diff_refusals.gqt index 1ef1411d6..507393a9d 100644 --- a/crates/omnigraph-gqt/cases/graph_diff_refusals.gqt +++ b/crates/omnigraph-gqt/cases/graph_diff_refusals.gqt @@ -23,11 +23,13 @@ node Person { --- seed {"type":"Person","data":{"name":"bob","age":40}} ---- diff from: main to: nowhere +--- query +branch diff main to nowhere --- expect error: 'nowhere' names no branch and no commit ---- merge-base source: nowhere +--- query +branch merge-base nowhere --- expect error: not found @@ -50,7 +52,8 @@ query main_bob() { --- expect affected: nodes=1 edges=0 ---- merge-dry-run source: feature +--- query +branch merge feature dry run --- expect error: merge conflicts diff --git a/crates/omnigraph-server/src/handlers.rs b/crates/omnigraph-server/src/handlers.rs index cae570c5c..d76be84f3 100644 --- a/crates/omnigraph-server/src/handlers.rs +++ b/crates/omnigraph-server/src/handlers.rs @@ -43,7 +43,7 @@ where mod dispatch; use dispatch::{ - CommitControlOutcome, Door, ReadDispatch, classify, commit_control_body, + CommitControlOutcome, Door, ReadDispatch, branch_read_body, classify, commit_control_body, commit_write_at_read_door, control_write_at_read_door, explain_at_write_door, read_at_write_door, refuse_empty_file, refuse_explain, refuse_process_settings, refuse_statement_envelope, refuse_wrong_door, run_branch_statement, run_commit_statement, @@ -791,7 +791,14 @@ pub(crate) async fn server_snapshot( /// /// The GQ statement `branch list` is also served here, with no `branch`, /// `snapshot`, `name`, or `params`: it answers one result per branch (field -/// `name`, byte order) under the same `read` check as `GET /branches`. +/// `name`, byte order) under the same `read` check as `GET /branches`. So are +/// the branch control reads, each under its route's checks and errors: +/// `branch diff [into ]` and `branch diff to ` +/// (one result per entity change, fields `kind`, `type`, `id`, `op`, +/// `before`, `after`; the compared commits and the continuation token in +/// `page`; `after page ""` and `limit ` continue and bound it), +/// `branch merge-base [into ]` and +/// `branch merge [into ] dry run` (one result each). /// `branch create`, `branch delete`, and `branch merge` are rejected with /// 400; send them to `POST /mutate`. /// @@ -1567,7 +1574,9 @@ pub(crate) async fn run_mutate( let session = session_with_prefix(&session, &file.settings)?; run_branch_statement(&state, &handle, &session, actor, ingress, write).await } - BranchStmt::List => Err(read_at_write_door()), + BranchStmt::List | BranchStmt::Read(_) => { + Err(read_at_write_door(stmt.statement_name())) + } }; } FileBody::Commit(write) => { @@ -1753,6 +1762,9 @@ pub(crate) async fn run_query( BranchStmt::List => Ok(ReadDispatch::BranchList( branch_list_body(&handle, actor).await?, )), + BranchStmt::Read(read) => Ok(ReadDispatch::BranchRead( + branch_read_body(&handle, &session, actor, read).await?, + )), BranchStmt::Write(write) => Err(control_write_at_read_door(&write)), }; } @@ -1873,7 +1885,7 @@ fn engine_error_with_cause(error: OmniError) -> (ApiError, Option request_body = ChangeRequest, responses( (status = 200, description = "Mutation results", body = ChangeOutput), - (status = 400, description = "Bad request - also returned when `branch list` arrives here instead of POST /query, when a request target accompanies a branch statement, when a name or parameters accompany a branch statement, and when a commit precondition accompanies a branch statement", body = ErrorOutput), + (status = 400, description = "Bad request - also returned when `branch list` or a branch control read (`branch diff`, `branch merge-base`, `branch merge … dry run`) arrives here instead of POST /query, when a request target accompanies a branch statement, when a name or parameters accompany a branch statement, and when a commit precondition accompanies a branch statement", body = ErrorOutput), (status = 401, description = "Unauthorized", body = ErrorOutput), (status = 403, description = "Forbidden", body = ErrorOutput), (status = 404, description = "`branch delete` of a branch that does not exist, as DELETE /branches/{branch} answers", body = ErrorOutput), @@ -1903,8 +1915,9 @@ fn engine_error_with_cause(error: OmniError) -> (ApiError, Option /// here, with no `branch`, `name`, or `params`: each runs the handler body /// of its `/branches` route (same Cedar action, admission check, and errors, /// including the 409 of a conflicting merge) and answers a `ChangeOutput` -/// whose `outcome` names the effect. `branch list` is rejected with 400; -/// send it to `POST /query`. +/// whose `outcome` names the effect. `branch list` and the branch control +/// reads (`branch diff`, `branch merge-base`, `branch merge … dry run`) are +/// rejected with 400; send them to `POST /query`. pub(crate) async fn server_mutate( State(state): State, Extension(handle): Extension, @@ -4074,27 +4087,49 @@ pub(crate) async fn server_diff( validate_change_http_limit(params.limit)?; let from = required_diff_param(params.from, "from")?; let to = required_diff_param(params.to, "to")?; + let page = two_point_diff_page( + &handle, + actor.as_ref().map(|Extension(actor)| actor), + from, + to, + api::change_scope(¶ms.kinds, ¶ms.types, ¶ms.ops), + params.page_token, + params.limit, + ) + .await?; + Ok(Json(api::diff_output(&page))) +} + +/// One page of the net change between two read targets, for `GET /diff` and +/// the `branch diff to ` statement: each side is authorized on the +/// branch it resolves to, and a side the actor cannot read answers the 404 a +/// side naming nothing does. +pub(crate) async fn two_point_diff_page( + handle: &GraphRequest, + actor: Option<&AuthenticatedActor>, + from: String, + to: String, + scope: omnigraph::changes::ChangeFeedScope, + page_token: Option, + max_changes: Option, +) -> std::result::Result { let db = &handle.engine; - let actor = actor.as_ref().map(|Extension(actor)| actor); for side in [&from, &to] { authorize_diff_side(db, actor, handle.policy.as_deref(), side).await?; } - let scope = api::change_scope(¶ms.kinds, ¶ms.types, ¶ms.ops); - let page = db - .diff_page(&omnigraph::changes::GraphDiffRequest { - sides: omnigraph::changes::GraphDiffSides::Between { from, to }, - scope, - page_token: params.page_token, - max_changes: params.limit, - max_bytes: None, - }) - .await - .map_err(change_route_error)?; - Ok(Json(api::diff_output(&page))) + db.diff_page(&omnigraph::changes::GraphDiffRequest { + sides: omnigraph::changes::GraphDiffSides::Between { from, to }, + scope, + page_token, + max_changes, + max_bytes: None, + }) + .await + .map_err(change_route_error) } /// Authorize `read` on each named branch of a branch diff or merge base. -fn authorize_branch_reads( +pub(crate) fn authorize_branch_reads( actor: Option<&AuthenticatedActor>, policy: Option<&PolicyEngine>, branches: [&str; 2], @@ -4145,26 +4180,46 @@ pub(crate) async fn server_branch_diff( ) -> std::result::Result, ApiError> { let params = parse_change_query(raw.as_deref(), BRANCH_DIFF_PARAMS)?; validate_change_http_limit(params.limit)?; - let source = normalize_change_branch(Some(&required_diff_param(params.source, "source")?))?; - let target = normalize_change_branch(params.target.as_deref())?; - authorize_branch_reads( + let source = required_diff_param(params.source, "source")?; + let page = branch_diff_page( + &handle, actor.as_ref().map(|Extension(actor)| actor), - handle.policy.as_deref(), - [&source, &target], - )?; - let scope = api::change_scope(¶ms.kinds, ¶ms.types, ¶ms.ops); - let page = handle + &source, + params.target.as_deref(), + api::change_scope(¶ms.kinds, ¶ms.types, ¶ms.ops), + params.page_token, + params.limit, + ) + .await?; + Ok(Json(api::diff_output(&page))) +} + +/// One page of what `source` changed since its merge base with `target` +/// (`main` when absent), for `GET /branches/diff` and the `branch diff` +/// statement: `read` on both named branches before any lineage is read. +pub(crate) async fn branch_diff_page( + handle: &GraphRequest, + actor: Option<&AuthenticatedActor>, + source: &str, + target: Option<&str>, + scope: omnigraph::changes::ChangeFeedScope, + page_token: Option, + max_changes: Option, +) -> std::result::Result { + let source = normalize_change_branch(Some(source))?; + let target = normalize_change_branch(target)?; + authorize_branch_reads(actor, handle.policy.as_deref(), [&source, &target])?; + handle .engine .diff_page(&omnigraph::changes::GraphDiffRequest { sides: omnigraph::changes::GraphDiffSides::Branch { source, target }, scope, - page_token: params.page_token, - max_changes: params.limit, + page_token, + max_changes, max_bytes: None, }) .await - .map_err(change_route_error)?; - Ok(Json(api::diff_output(&page))) + .map_err(change_route_error) } #[utoipa::path( @@ -4193,19 +4248,35 @@ pub(crate) async fn server_merge_base( axum::extract::RawQuery(raw): axum::extract::RawQuery, ) -> std::result::Result, ApiError> { let params = parse_change_query(raw.as_deref(), MERGE_BASE_PARAMS)?; - let source = normalize_change_branch(Some(&required_diff_param(params.source, "source")?))?; - let target = normalize_change_branch(params.target.as_deref())?; - authorize_branch_reads( + let source = required_diff_param(params.source, "source")?; + let (source, target, base) = merge_base_body( + &handle, actor.as_ref().map(|Extension(actor)| actor), - handle.policy.as_deref(), - [&source, &target], - )?; + &source, + params.target.as_deref(), + ) + .await?; + Ok(Json(api::merge_base_output(&source, &target, &base))) +} + +/// The merge base of `source` into `target` (`main` when absent), for +/// `GET /branches/merge-base` and the `branch merge-base` statement: `read` +/// on both named branches first. Answers the normalized names with the base. +pub(crate) async fn merge_base_body( + handle: &GraphRequest, + actor: Option<&AuthenticatedActor>, + source: &str, + target: Option<&str>, +) -> std::result::Result<(String, String, omnigraph::changes::MergeBase), ApiError> { + let source = normalize_change_branch(Some(source))?; + let target = normalize_change_branch(target)?; + authorize_branch_reads(actor, handle.policy.as_deref(), [&source, &target])?; let base = handle .engine .merge_base(&source, &target) .await .map_err(change_route_error)?; - Ok(Json(api::merge_base_output(&source, &target, &base))) + Ok((source, target, base)) } /// Contextual 404 projection. `Manifest::NotFound` is a broad internal diff --git a/crates/omnigraph-server/src/handlers/dispatch.rs b/crates/omnigraph-server/src/handlers/dispatch.rs index c423a53f3..879a347ce 100644 --- a/crates/omnigraph-server/src/handlers/dispatch.rs +++ b/crates/omnigraph-server/src/handlers/dispatch.rs @@ -6,13 +6,16 @@ use super::*; use crate::api::branch_statement_refusals as refusals; use omnigraph_compiler::query::ast::{ - BranchStmt, BranchWrite, CommitWrite, EXPLAIN_STATEMENT_NAME, SettingStmt, show_statement_name, + BranchDiffSides, BranchRead, BranchStmt, BranchWrite, CommitWrite, EXPLAIN_STATEMENT_NAME, + SettingStmt, show_statement_name, }; use omnigraph_compiler::settings::{SettingId, SettingRow}; /// The HTTP entry a GQ source arrived through. A statement is served only at -/// its own door: `Query` serves `branch list` and `explain`, `Mutate` serves -/// `branch create`, `branch delete`, and `branch merge`. +/// its own door: `Query` serves `branch list`, the branch control reads +/// (`branch diff`, `branch merge-base`, `branch merge … dry run`) and +/// `explain`; `Mutate` serves `branch create`, `branch delete`, and +/// `branch merge`. #[derive(Debug, Clone, Copy, PartialEq, Eq)] pub(crate) enum Door { Query, @@ -20,8 +23,9 @@ pub(crate) enum Door { } /// What [`run_query`] answered: a declared query's rows, the branch names -/// of a `branch list` statement sorted in byte order, or the setting rows of -/// a `show` statement in definition order. +/// of a `branch list` statement sorted in byte order, the answer of a branch +/// control read, or the setting rows of a `show` statement in definition +/// order. pub(crate) enum ReadDispatch { Rows { query_name: String, @@ -30,6 +34,7 @@ pub(crate) enum ReadDispatch { graph_commit_id: Option, }, BranchList(Vec), + BranchRead(ReadOutput), Show(Vec), } @@ -48,6 +53,7 @@ impl ReadDispatch { ReadDispatch::BranchList(branches) => { api::branch_list_read_output(&branches).map_err(render_error) } + ReadDispatch::BranchRead(output) => Ok(output), ReadDispatch::Show(rows) => api::show_read_output(&rows).map_err(render_error), } } @@ -114,10 +120,10 @@ pub(super) fn commit_write_at_read_door(write: &CommitWrite) -> ApiError { )) } -pub(super) fn read_at_write_door() -> ApiError { +pub(super) fn read_at_write_door(statement_name: &str) -> ApiError { ApiError::bad_request(refusals::with_statement( refusals::READ_AT_WRITE_DOOR, - BranchStmt::List.statement_name(), + statement_name, )) } @@ -136,8 +142,64 @@ pub(super) fn refuse_wrong_door( ) -> std::result::Result<(), ApiError> { match (door, stmt) { (Door::Query, BranchStmt::Write(write)) => Err(control_write_at_read_door(write)), - (Door::Mutate, BranchStmt::List) => Err(read_at_write_door()), - (Door::Query, BranchStmt::List) | (Door::Mutate, BranchStmt::Write(_)) => Ok(()), + (Door::Mutate, BranchStmt::List | BranchStmt::Read(_)) => { + Err(read_at_write_door(stmt.statement_name())) + } + (Door::Query, BranchStmt::List | BranchStmt::Read(_)) + | (Door::Mutate, BranchStmt::Write(_)) => Ok(()), + } +} + +/// Answer a branch control read with the body of the route it shares: the +/// same authorization, engine call and error mapping as `GET /diff`, +/// `GET /branches/diff`, `GET /branches/merge-base` and the dry run of +/// `POST /branches/merge`. +pub(super) async fn branch_read_body( + handle: &GraphRequest, + session: &Session, + actor: Option<&AuthenticatedActor>, + read: BranchRead, +) -> std::result::Result { + match read { + BranchRead::Diff { sides, page } => { + let scope = omnigraph::changes::ChangeFeedScope::default(); + let diff = match sides { + BranchDiffSides::Branch { source, into } => { + branch_diff_page( + handle, + actor, + &source, + into.as_deref(), + scope, + page.after_page, + page.limit, + ) + .await? + } + BranchDiffSides::Between { from, to } => { + two_point_diff_page(handle, actor, from, to, scope, page.after_page, page.limit) + .await? + } + }; + api::branch_diff_read_output(&diff).map_err(render_error) + } + BranchRead::MergeBase { source, into } => { + let (source, target, base) = + merge_base_body(handle, actor, &source, into.as_deref()).await?; + api::branch_merge_base_read_output(&source, &target, &base).map_err(render_error) + } + BranchRead::MergeDryRun { source, into } => { + let target = into.unwrap_or_else(|| "main".to_string()); + // A read of both branches: no merge admission, no write gate, no + // `branch_merge` grant. Conflicts answer the 409 a merge does. + authorize_branch_reads(actor, handle.policy.as_deref(), [&source, &target])?; + let result = session + .branch_merge_dry_run(&source, &target) + .await + .map_err(ApiError::from_omni)?; + api::branch_merge_dry_run_read_output(&source, &target, result.outcome.into()) + .map_err(render_error) + } } } diff --git a/crates/omnigraph-server/tests/integration/auth_policy.rs b/crates/omnigraph-server/tests/integration/auth_policy.rs index 718810189..24d957baa 100644 --- a/crates/omnigraph-server/tests/integration/auth_policy.rs +++ b/crates/omnigraph-server/tests/integration/auth_policy.rs @@ -1716,6 +1716,23 @@ async fn policy_commit_diff_forbidden_is_indistinguishable_from_unknown() { format!("'{forbidden_side}' names no branch and no commit") ); assert_eq!(forbidden["code"], unknown["code"]); + // The `branch diff to ` statement answers the same 404. + let statement = |side: &str| { + statement_request( + "/query", + "team-token", + &format!(r#"branch diff main to "{side}""#), + ) + }; + let (forbidden_status, forbidden) = json_response(&app, statement(forbidden_side)).await; + let (unknown_status, unknown) = json_response(&app, statement(unknown_side)).await; + assert_eq!(forbidden_status, StatusCode::NOT_FOUND, "{forbidden}"); + assert_eq!(unknown_status, StatusCode::NOT_FOUND, "{unknown}"); + assert_eq!( + forbidden["error"], + format!("'{forbidden_side}' names no branch and no commit") + ); + assert_eq!(forbidden["code"], unknown["code"]); } // The branch routes name their branches outright: a read denial is a 403, // and a dry run is a read of both branches. @@ -1794,6 +1811,20 @@ async fn policy_merge_dry_run_needs_read_and_not_branch_merge() { assert_eq!(body["dry_run"], true); let (status, body) = json_response(&app, merge(false)).await; assert_eq!(status, StatusCode::FORBIDDEN, "{body}"); + // The statement form is the same read. + let (status, body) = json_response( + &app, + statement_request("/query", "team-token", "branch merge feature dry run"), + ) + .await; + assert_eq!(status, StatusCode::OK, "{body}"); + assert_eq!(body["rows"][0]["outcome"], "fast_forward"); + let (status, body) = json_response( + &app, + statement_request("/mutate", "team-token", "branch merge feature"), + ) + .await; + assert_eq!(status, StatusCode::FORBIDDEN, "{body}"); } /// A branch review's endpoints are the merge base and the source head, which @@ -1885,6 +1916,23 @@ async fn policy_branch_review_needs_read_on_the_target() { let (status, body) = json_response(&app, get(&uri, "team-token")).await; assert_eq!(status, StatusCode::FORBIDDEN, "{uri}: {body}"); } + // The statement forms take the same decision. + let (status, body) = json_response( + &app, + statement_request("/query", "team-token", "branch diff feature into main"), + ) + .await; + assert_eq!(status, StatusCode::OK, "{body}"); + for source in [ + "branch diff feature into private".to_string(), + format!(r#"branch diff feature into private after page "{token}" limit 1"#), + "branch merge-base feature into private".to_string(), + "branch merge feature into private dry run".to_string(), + ] { + let (status, body) = + json_response(&app, statement_request("/query", "team-token", &source)).await; + assert_eq!(status, StatusCode::FORBIDDEN, "{source}: {body}"); + } } #[tokio::test(flavor = "multi_thread")] diff --git a/crates/omnigraph-server/tests/integration/data_routes.rs b/crates/omnigraph-server/tests/integration/data_routes.rs index 7cdabda4b..dfc0b2229 100644 --- a/crates/omnigraph-server/tests/integration/data_routes.rs +++ b/crates/omnigraph-server/tests/integration/data_routes.rs @@ -6864,6 +6864,112 @@ async fn diff_routes_review_a_branch_against_its_merge_base() { change_ids(&since_fork), vec![("MainOnly".to_string(), "insert".to_string())] ); + + // The branch control reads answer the same through `POST /query`, as + // rows; a diff's commits and its continuation token travel in `page`. + let (_, base) = get_json(&app, g("/branches/merge-base?source=feature")).await; + let (status, base_rows) = query_statement(&app, "branch merge-base feature").await; + assert_eq!(status, StatusCode::OK, "{base_rows}"); + assert_eq!(base_rows["query_name"], "branch merge-base"); + assert_eq!( + base_rows["rows"], + json!([{ + "source": "feature", + "target": "main", + "source_commit_id": base["source_commit_id"], + "target_commit_id": base["target_commit_id"], + "merge_base_commit_id": fork, + }]) + ); + let row_ids = |answer: &Value| { + answer["rows"] + .as_array() + .unwrap() + .iter() + .map(|row| { + ( + row["id"].as_str().unwrap().to_string(), + row["op"].as_str().unwrap().to_string(), + ) + }) + .collect::>() + }; + let (_, review) = get_json(&app, g("/branches/diff?source=feature")).await; + let (status, first_rows) = query_statement(&app, "branch diff feature limit 1").await; + assert_eq!(status, StatusCode::OK, "{first_rows}"); + assert_eq!( + first_rows["columns"], + json!(["kind", "type", "id", "op", "before", "after"]) + ); + assert_eq!(first_rows["page"]["base_commit_id"], fork.as_str()); + assert_eq!(first_rows["page"]["to_commit_id"], review["to_commit_id"]); + assert_eq!(first_rows["rows"][0]["type"], "Person"); + let mut statement_review = row_ids(&first_rows); + let mut token = first_rows["page"]["next_page_token"] + .as_str() + .expect("several changes, pages of one") + .to_string(); + loop { + let (status, page) = query_statement( + &app, + &format!(r#"branch diff feature into main after page "{token}" limit 1"#), + ) + .await; + assert_eq!(status, StatusCode::OK, "{page}"); + assert_eq!(page["page"]["to_commit_id"], review["to_commit_id"]); + statement_review.extend(row_ids(&page)); + match page["page"]["next_page_token"].as_str() { + Some(next) => token = next.to_string(), + None => break, + } + } + assert_eq!(statement_review, change_ids(&review)); + let (status, head_rows) = query_statement(&app, "branch diff main to feature").await; + assert_eq!(status, StatusCode::OK, "{head_rows}"); + assert_eq!(row_ids(&head_rows), change_ids(&heads)); + let (status, dry) = query_statement(&app, "branch merge feature dry run").await; + assert_eq!(status, StatusCode::OK, "{dry}"); + assert_eq!( + dry["rows"], + json!([{"source": "feature", "target": "main", "outcome": "merged"}]) + ); + assert_eq!(main_head_commit_id(&app).await, since_fork["to_commit_id"]); + + // A control read is refused at the write door, under its own name. + for statement in [ + "branch diff feature", + "branch merge-base feature", + "branch merge feature dry run", + ] { + let (status, body) = + json_response(&app, json_post("/mutate", &json!({"query": statement}))).await; + assert_eq!(status, StatusCode::BAD_REQUEST, "{statement}: {body}"); + let name = statement.split(" feature").next().unwrap(); + let name = if statement.ends_with("dry run") { + "branch merge dry run" + } else { + name + }; + assert!( + body["error"] + .as_str() + .unwrap() + .contains(&format!("'{name}' is a read")), + "{statement}: {body}" + ); + } +} + +/// One statement at the read door, sent bare. +async fn query_statement(app: &axum::Router, statement: &str) -> (StatusCode, Value) { + json_response( + app, + json_post( + "/query", + &json!({"query": statement, "name": null, "params": null, "branch": null, "snapshot": null}), + ), + ) + .await } #[tokio::test(flavor = "multi_thread")] diff --git a/crates/omnigraph/src/changes/mod.rs b/crates/omnigraph/src/changes/mod.rs index fc4668730..6c1c7e5d7 100644 --- a/crates/omnigraph/src/changes/mod.rs +++ b/crates/omnigraph/src/changes/mod.rs @@ -6,7 +6,7 @@ pub(crate) mod row_compare; pub(crate) mod token; pub use model::{ - CHANGE_FEED_DEFAULT_COMMITS_PER_POLL, CHANGE_FEED_MAX_COMMITS_PER_POLL, + BranchReadAnswer, CHANGE_FEED_DEFAULT_COMMITS_PER_POLL, CHANGE_FEED_MAX_COMMITS_PER_POLL, COMMIT_CHANGES_DEFAULT_BYTES, COMMIT_CHANGES_DEFAULT_CHANGES, COMMIT_CHANGES_MAX_BYTES, COMMIT_CHANGES_MAX_CHANGES, ChangeBaseline, ChangeCause, ChangeEntityKind, ChangeFeedContinuation, ChangeFeedPage, ChangeFeedPosition, ChangeFeedRequest, ChangeFeedScope, diff --git a/crates/omnigraph/src/changes/model.rs b/crates/omnigraph/src/changes/model.rs index 38b16ce3a..8f4f0080a 100644 --- a/crates/omnigraph/src/changes/model.rs +++ b/crates/omnigraph/src/changes/model.rs @@ -243,6 +243,23 @@ pub struct MergeBase { pub merge_base: GraphCommit, } +/// What a branch control read answers (RFC 0030 §8), before a transport +/// renders it: one page of a diff, a merge base, or a merge's outcome. +#[derive(Debug, Clone)] +pub enum BranchReadAnswer { + Diff(GraphDiffPage), + MergeBase { + source: String, + target: String, + base: MergeBase, + }, + MergeDryRun { + source: String, + target: String, + outcome: crate::db::MergeOutcome, + }, +} + /// Graph-vocabulary feed filter. Empty dimensions match everything; the /// canonical digest of this scope binds page tokens and feed cursors. #[derive(Debug, Clone, Default, PartialEq, Eq)] diff --git a/crates/omnigraph/src/session.rs b/crates/omnigraph/src/session.rs index cb1d08a01..717430355 100644 --- a/crates/omnigraph/src/session.rs +++ b/crates/omnigraph/src/session.rs @@ -7,7 +7,7 @@ use std::ops::Deref; use std::sync::Arc; use omnigraph_compiler::error::CompilerError; -use omnigraph_compiler::query::ast::SettingStmt; +use omnigraph_compiler::query::ast::{BranchDiffSides, BranchRead, SettingStmt}; use omnigraph_compiler::query::codes::Q003; use omnigraph_compiler::query::diagnostic::QueryDiagnostic; use omnigraph_compiler::query::parser::{has_settings_prefix, parse_query}; @@ -173,6 +173,58 @@ impl Session { .collect() } + /// Answer a branch control read (`branch diff`, `branch merge-base`, + /// `branch merge … dry run`) with the engine call it names, an unspelled + /// `into` meaning `main`. It reads lineage and tables and writes nothing; + /// a transport authorizes the branches it names before calling it. + pub async fn branch_read(&self, read: &BranchRead) -> Result { + use crate::changes::{BranchReadAnswer, ChangeFeedScope, GraphDiffRequest, GraphDiffSides}; + + let target = |into: &Option| into.clone().unwrap_or_else(|| "main".to_string()); + Ok(match read { + BranchRead::Diff { sides, page } => { + let sides = match sides { + BranchDiffSides::Branch { source, into } => GraphDiffSides::Branch { + source: source.clone(), + target: target(into), + }, + BranchDiffSides::Between { from, to } => GraphDiffSides::Between { + from: from.clone(), + to: to.clone(), + }, + }; + BranchReadAnswer::Diff( + self.diff_page(&GraphDiffRequest { + sides, + scope: ChangeFeedScope::default(), + page_token: page.after_page.clone(), + max_changes: page.limit, + max_bytes: None, + }) + .await?, + ) + } + BranchRead::MergeBase { source, into } => { + let target = target(into); + let base = self.merge_base(source, &target).await?; + BranchReadAnswer::MergeBase { + source: source.clone(), + target, + base, + } + } + BranchRead::MergeDryRun { source, into } => { + let target = target(into); + let result = self.branch_merge_dry_run(source, &target).await?; + BranchReadAnswer::MergeDryRun { + source: source.clone(), + target, + outcome: result.outcome, + } + } + }) + } + /// The settings one call runs under: the session's, with the settings /// prefix of `source` applied in file order on a copy. The session is /// unchanged. A source without a prefix (`has_settings_prefix`) is not diff --git a/crates/omnigraph/tests/forbidden_apis.rs b/crates/omnigraph/tests/forbidden_apis.rs index 870a8a724..34b0e36c3 100644 --- a/crates/omnigraph/tests/forbidden_apis.rs +++ b/crates/omnigraph/tests/forbidden_apis.rs @@ -314,6 +314,7 @@ const READ_ONLY_SURFACES: &[(&str, &str)] = &[ ("db/omnigraph.rs", "merge_base"), ("db/omnigraph.rs", "diff_read_target_branch"), ("exec/merge.rs", "branch_merge_dry_run"), + ("session.rs", "branch_read"), ("db/omnigraph.rs", "commit_changes_page"), ("db/omnigraph.rs", "poll_change_feed"), ("db/omnigraph.rs", "capture_change_baseline"), diff --git a/docs/rfcs/0030-cdc-time-travel.md b/docs/rfcs/0030-cdc-time-travel.md index 4d85bb9c8..443c62102 100644 --- a/docs/rfcs/0030-cdc-time-travel.md +++ b/docs/rfcs/0030-cdc-time-travel.md @@ -7,7 +7,7 @@ implementation: partial authors: - OmniGraph maintainers created: 2026-08-05 -updated: 2026-10-12 +updated: 2026-10-13 discussion: null supersedes: [] superseded_by: [] @@ -782,22 +782,41 @@ answers whether the merge would succeed, without a write. ### 8.5 Surfaces -| CLI | HTTP | Answers | -|---|---|---| -| `diff --from --to ` | `GET /graphs/{id}/diff?from=&to=` | Net changes between two read targets | -| `branch diff [--into ]` | `GET /graphs/{id}/branches/diff?source=&target=` | What the source changed since its merge base with the target | -| `branch merge-base [--into ]` | `GET /graphs/{id}/branches/merge-base?source=&target=` | The merge base commit | -| `branch merge --dry-run` | `POST /graphs/{id}/branches/merge` with `dry_run: true` | Merge outcome or conflicts, without a write | -| `commit changes `, unchanged | `GET /graphs/{id}/commits/{commit_id}/changes`, unchanged | One commit's changes with its cause | - -The diff commands take the existing `--kind`, `--type`, `--op`, `--limit`, -`--page-token` and `--json` options and paginate automatically, as -`commit changes` does. `--into`, `source` and `target` follow `branch merge`. -Exposing them as GQ statements is deferred, not rejected. -[RFC 0056](https://github.com/ModernRelay/omnigraph/pull/670) proposes carrying -`commit changes` and `changes poll` as entity-change rows with an optional page -object, and the graph diff should take whatever envelope that RFC settles -rather than decide a second one here. D1–D3 ship through the CLI and HTTP. +Each surface is a GQ branch statement, a control read in the sense of +[RFC 0055](0055-gq-branch-statements.md): it enters through `POST /query` and +`omnigraph query`, standing alone in its file, and `POST /mutate` refuses it +as a read. The CLI verbs and HTTP routes wrap the same engine call, under the +same authorization and errors. + +| GQ statement | CLI | HTTP | Answers | +|---|---|---|---| +| `branch diff to ` | `diff --from --to ` | `GET /graphs/{id}/diff?from=&to=` | Net changes between two read targets | +| `branch diff [into ]` | `branch diff [--into ]` | `GET /graphs/{id}/branches/diff?source=&target=` | What the source changed since its merge base with the target | +| `branch merge-base [into ]` | `branch merge-base [--into ]` | `GET /graphs/{id}/branches/merge-base?source=&target=` | The merge base commit | +| `branch merge [into ] dry run` | `branch merge --dry-run` | `POST /graphs/{id}/branches/merge` with `dry_run: true` | Merge outcome or conflicts, without a write | +| none | `commit changes `, unchanged | `GET /graphs/{id}/commits/{commit_id}/changes`, unchanged | One commit's changes with its cause | + +A side is a branch name or a quoted commit id. A `branch diff` statement takes +`after page ""` and `limit `, in that order. `dry run` is a clause +like `as commit`: a bare `dry run` after the names is the clause, so a branch +named `dry` is quoted. The CLI and HTTP diff surfaces also take the `--kind`, +`--type` and `--op` filters; the statements take none yet. + +A statement answers a `ReadOutput` with no target and no graph commit: + +- `branch diff`: one row per change in the §4.4 order, columns `kind`, `type`, + `id`, `op`, `before` and `after`, the HTTP change shape with the type named + by its name. The opaque type id, unique per graph, stays on the HTTP routes, + where it lets a consumer follow a type across a rename. The compared commits + (`from_commit_id`, `to_commit_id`, `base_commit_id` for the branch form) and + the continuation token (`next_page_token`) travel in an optional `page` + object on `ReadOutput`, the shape RFC 0056 + ([#670](https://github.com/ModernRelay/omnigraph/pull/670)) proposes for + paged change reads. +- `branch merge-base`: one row, `source`, `target`, `source_commit_id`, + `target_commit_id` and `merge_base_commit_id`. +- `branch merge … dry run`: one row, `source`, `target` and `outcome`; a + conflict is the 409 a merge answers. ### 8.6 Retiring the table-shaped diff @@ -1440,5 +1459,20 @@ Amendments before 2026-10-12 are recorded in §14 and in the dispositions above. `parity_diff_over_shared_history` and `parity_branch_review`; GQT cases `graph_diff_branch_review.gqt`, `graph_diff_merge_proposal.gqt`, `graph_diff_net_range.gqt` and `graph_diff_refusals.gqt` on the engine, DST - and served targets (RFC 0045's new `--- diff`, `--- merge-base` and - `--- merge-dry-run` steps). + and served targets (as GQT steps at first, then, per the 2026-10-13 entry, + as the branch statements under `--- query`). +- **2026-10-13: the graph diff surfaces are GQ statements.** Review of + ModernRelay/omnigraph#966 asked that the diff, merge base and dry run be GQ + statements beside `branch merge … as commit` rather than GQT step kinds. + §8.5 now defines `branch diff`, `branch merge-base` and + `branch merge … dry run` as RFC 0055 control reads at `POST /query`, with the + CLI verbs and HTTP routes as wrappers of the same engine call. The + `branch diff` answer carries its compared commits and continuation token in + an optional `page` object on `ReadOutput`, the shape RFC 0056 proposes, and + names its rows' type by name. GQT runs the statements as ordinary + `--- query` steps; the three step kinds RFC 0045 had gained are withdrawn. + This entry supersedes these sentences: + - §8.5: "Exposing them as GQ statements is deferred, not rejected." and the + rest of that paragraph through "D1–D3 ship through the CLI and HTTP." + - The 2026-10-12 review entry's "GQ exposure of the diff surfaces is + deferred to RFC 0056 (ModernRelay/omnigraph#670), not rejected (§8.5)." diff --git a/docs/rfcs/0045-gq-logic-tests.md b/docs/rfcs/0045-gq-logic-tests.md index 9bce21f92..597110bc6 100644 --- a/docs/rfcs/0045-gq-logic-tests.md +++ b/docs/rfcs/0045-gq-logic-tests.md @@ -7,7 +7,7 @@ implementation: partial authors: - azimafroozeh created: 2026-08-29 -updated: 2026-10-12 +updated: 2026-10-13 discussion: https://github.com/ModernRelay/omnigraph/pull/584 supersedes: [] superseded_by: [] @@ -600,18 +600,6 @@ in Seams at an explicit step. A step is one of: there is no row expect on a mutate step. - `--- restart`, body empty: drop the store handle and reopen it from the same URI before the next step. -- `--- diff`, `--- merge-base` and `--- merge-dry-run`, body empty: the graph - diff surfaces of RFC 0030 §8, which are not GQ statements. `--- diff` - names its sides as `from: to: `, as `source: ` with an - optional `target: ` (`main` when absent), or as `commit: `, - where a side is a branch name or a bound `$name`; it follows every page and - takes `ordered` or `unordered` rows, one JSON object per change in the wire - shape with `type` spelled as the type's name and compared whatever its key - order, or `error: `. `--- merge-base source: - [target: ]` takes `commit $name`, the bound commit the base must be, - or `error: `. `--- merge-dry-run source: - [target: ]` takes the `branch merge` expects (`ok`, `outcome:`, - `error:`) and fails if the dry run reports a commit. - A settings step (a `--- mutate` whose body is a `set ...;` statement) and a show step (a `--- query` whose body is `show ...;`) take no `via`: their headers stay bare, a `via` on them is refused with the grammar, they @@ -667,10 +655,21 @@ counts exist). `branch list` is a rows step: it takes `unordered`, `ordered`, or `error: ` over rows `{"name": "…"}`, its rows are sorted by `name` in byte order (a total order, so `ordered` is accepted and the `order`-clause refusal for declarations does not apply), and it -carries the mandatory shape section, one line, `name: String`. A statement -step has no params, so an iterating value never reaches its body; `${` in a -quoted statement name parses and is refused by the runner's `${` fence, and -`${` anywhere else in a statement body does not parse. +carries the mandatory shape section, one line, `name: String`. A +`--- query` step may also hold a ***branch control read*** of RFC 0030 §8.5 +(`branch diff`, `branch merge-base`, `branch merge … dry run`), refused under +`--- mutate` as `branch list` is. It is a rows step over the statement's own +answer: `unordered`, `ordered`, or `error: `, the rows compared +as JSON whatever their key order. Its columns are the statement's, so it +carries no shape section, and one is refused (`` `branch diff` answers its +own columns; drop the `--- expect shape` section ``). A side spelled as a +quoted bound name (`branch diff "${c}" to main`) runs with the commit +`--- bind commit $c` captured, and a bound `${c}` in an expected row stands +for the same id, as in a params body; an unbound name is refused. A +statement step has no params, so an iterating value never reaches its body; +`${` in a quoted statement name parses and is refused by the runner's `${` +fence unless it is a bound commit, and `${` anywhere else in a statement body +does not parse. The read/mutation classification is the compiler's own (`query_body = { read_query_body | mutation_body }` in `query.pest`); the @@ -2674,11 +2673,11 @@ historical command and configuration descriptions are not migration aliases. `order_key_aggregate_or_node_binds_to_return_item.gqt` (§Evidence and tests) and `engine_v1_refuses_compound_predicates.gqt` to `compound_and_bare_boolean_filters.gqt`. -- 2026-10-12, graph diff steps: `--- diff`, `--- merge-base` and - `--- merge-dry-run` join the step kinds (§File format) for the surfaces of - RFC 0030 §8, with the `expect commit $name` mode for a merge base. A - `--- query` step cannot hold them: its rows answer a GQ statement under a - typed result schema, and a diff is paged change records that RFC 0030 keeps - out of GQ. The engine and DST targets call the engine, and the served target - sends the HTTP routes, so one case pins all three. Additive: no earlier - sentence is superseded. +- 2026-10-13, branch control reads: a `--- query` step holds the branch + control reads of RFC 0030 §8.5 (`branch diff`, `branch merge-base`, + `branch merge … dry run`) as rows steps without a shape section, their + columns being the statement's own, with a bound commit name in a quoted + side and in the expected rows. This supersedes "a `--- query` step may hold + `branch list`" as the only branch statement a query step holds, and "`${` + in a quoted statement name parses and is refused by the runner's `${` + fence" for a bound commit name in a branch control read. diff --git a/docs/rfcs/0055-gq-branch-statements.md b/docs/rfcs/0055-gq-branch-statements.md index cba92c47c..15d9a03ab 100644 --- a/docs/rfcs/0055-gq-branch-statements.md +++ b/docs/rfcs/0055-gq-branch-statements.md @@ -7,7 +7,7 @@ implementation: in-progress authors: - azimafroozeh created: 2026-09-04 -updated: 2026-09-06 +updated: 2026-10-13 discussion: https://github.com/ModernRelay/omnigraph/pull/626 supersedes: [] superseded_by: [] @@ -1493,3 +1493,9 @@ None. expectation in item 3 of the first-cases entry above, "both are red and held out of the corpus until their fixes" in the 2026-09-06 entry, and the Motivation's listing of seed 221206 among the engine findings. +- 2026-10-13: [RFC 0030](0030-cdc-time-travel.md) §8.5 adds the branch + control reads `branch diff`, `branch merge-base` and + `branch merge … dry run`, which enter `POST /query` and `omnigraph query` + beside `branch list` and are refused at `POST /mutate` as reads. This + supersedes "`branch list` is the only branch statement `POST /query` + accepts" in User and operational behavior; the read door stays read-only. diff --git a/docs/user/branching/changes.md b/docs/user/branching/changes.md index 12734983e..f87eb9a70 100644 --- a/docs/user/branching/changes.md +++ b/docs/user/branching/changes.md @@ -52,7 +52,9 @@ Each side is a branch name, meaning its head when the diff starts, or a commit id: a published commit, an unpublished proposal, or an unattached snapshot. A string that names both a branch and a commit is refused; name the branch's head by its commit id instead. The equivalent HTTP route is -`GET /graphs/{graph_id}/diff?from=&to=`. +`GET /graphs/{graph_id}/diff?from=&to=`, and the GQ statement +`branch diff to ` answers the same comparison at `POST /query`, +a commit id quoted (`branch diff "" to main`). The result is the net change from the `from` state to the `to` state, in the `commit changes` change shape and order, with `from_commit_id` and @@ -97,7 +99,16 @@ commit that merge brought in. `branch diff` gives the net change from any base. The HTTP routes are `GET /graphs/{graph_id}/branches/diff?source=&target=` and -`GET /graphs/{graph_id}/branches/merge-base?source=&target=`. To see whether +`GET /graphs/{graph_id}/branches/merge-base?source=&target=`; the GQ statements +are `branch diff feature into main` and `branch merge-base feature into main`. + +A statement answers rows through `POST /query` or `omnigraph query -e`. A +`branch diff` row has `kind`, `type` (the type's name), `id`, `op`, `before` +and `after`; the compared commits and the token for the next page travel in +the answer's `page` object, and `after page ""` (then `limit `) +continues it. A `branch merge-base` row names `source`, `target`, both heads +and `merge_base_commit_id`. Statements take no kind, type or operation filter +yet; use the routes or CLI commands for those. To see whether the merge would succeed, run it as a [dry run](merge.md#preview-a-merge). ## Follow a branch diff --git a/docs/user/branching/index.md b/docs/user/branching/index.md index 65b2283a1..dad7cc783 100644 --- a/docs/user/branching/index.md +++ b/docs/user/branching/index.md @@ -33,14 +33,17 @@ a missing target branch by combining `--branch ` with `--from `. Each branch operation is also a GQ statement, so a client that already sends `.gq` source needs no second transport. The three control writes go to -`POST /mutate` and the listing to `POST /query`, each with no request target, -name, or parameters: +`POST /mutate`, and the listing and the branch reads to `POST /query`, each +with no request target, name, or parameters: ```text branch create "review/add-benchmark" from main branch merge "review/add-benchmark" into main branch delete "review/abandoned" branch list +branch diff "review/add-benchmark" into main +branch merge-base "review/add-benchmark" into main +branch merge "review/add-benchmark" into main dry run ``` `from` defaults to `main` and `into` defaults to `main`. A name outside the @@ -53,7 +56,12 @@ that publishes a commit reports it in `commit`. `branch list` answers one row per branch in column `name`, sorted by name. Each statement runs under the same policy check as its HTTP route (`POST /branches`, `DELETE /branches/{branch}`, `POST /branches/merge`, `GET /branches`), and `branch list` is a read, so it is -refused on `POST /mutate` as a control write is refused on `POST /query`. +refused on `POST /mutate` as a control write is refused on `POST /query`. The +branch reads `branch diff`, `branch merge-base` and `branch merge … dry run` +answer rows as `branch list` does and take their routes' checks; see +[reviewing a branch](changes.md#review-a-branch) and +[previewing a merge](merge.md#preview-a-merge). A bare `dry run` after the +names is the clause, so a branch named `dry` is quoted. Branches are cheap until written: unchanged data remains shared with the source. A branch remains after a normal merge, so prefer `--delete-branch` or delete it diff --git a/docs/user/branching/merge.md b/docs/user/branching/merge.md index 9f3b8be00..b3d0addf5 100644 --- a/docs/user/branching/merge.md +++ b/docs/user/branching/merge.md @@ -103,6 +103,9 @@ same structured conflict list, and writes nothing: `commit` is `null` and `dry_run` is `true`. It is a read of both branches, so it needs `read` on each and no `branch_merge` grant. It refuses `--as-commit` and `--delete-branch`. Over HTTP, send `dry_run: true` to `POST /graphs/{graph_id}/branches/merge`. +The GQ statement `branch merge feature into main dry run` is the same read at +`POST /query`: it answers one row with `source`, `target` and `outcome`, and a +conflict is the 409 a merge answers. A dry run answers whether the merge would succeed. To see what it would change, use [`branch diff`](changes.md#review-a-branch). To review the exact result diff --git a/docs/user/cli/reference.md b/docs/user/cli/reference.md index ab4e144db..1721d8345 100644 --- a/docs/user/cli/reference.md +++ b/docs/user/cli/reference.md @@ -40,7 +40,7 @@ server resolves the actor from the bearer token. Drop it, or use `--store ` | Command | Purpose | Scope | |---|---|---| | `init` | Create an empty graph from a `.pg` schema | direct | -| `query` | Run a read query, or the `branch list`, `show`, or `explain` statement | direct or served | +| `query` | Run a read query, or a branch read (`branch list`, `diff`, `merge-base`, `merge … dry run`), `show`, or `explain` statement | direct or served | | `mutate` | Run an insert/update/delete query (`as commit` leaves it unpublished; `as unattached branch [on ""]` records a snapshot that never publishes), or a `branch create`, `branch delete`, `branch merge`, `commit publish`, or `commit drop` statement | direct or served | | `load` | Load graph JSONL in `overwrite`, `append`, or `merge` mode | direct or served | | `blob get/stat/put/clear` | Read, inspect, replace or clear one Blob cell | direct or served | diff --git a/docs/user/operations/server.md b/docs/user/operations/server.md index aeeea526b..2ffed9fa7 100644 --- a/docs/user/operations/server.md +++ b/docs/user/operations/server.md @@ -235,7 +235,7 @@ keep their existing routes and do not expose MCP. | `/graphs/{id}/schema` | Show the accepted schema | | `/graphs/{id}/export` | Stream a branch snapshot as JSONL | -`/query` also serves `branch list`, `show`, and [explain](../queries/explain.md). +`/query` also serves `branch list`, the branch reads (`branch diff`, `branch merge-base`, `branch merge … dry run`), `show`, and [explain](../queries/explain.md). `/mutate` serves `branch create`, `branch delete`, `branch merge`, `commit publish`, and `commit drop`. See [Branching](../branching/index.md). Each of `/query`, `/mutate`, `/mutate/if-graph-commit` and `/branches/merge` diff --git a/docs/user/queries/index.md b/docs/user/queries/index.md index b3b353ac9..0ccb248d1 100644 --- a/docs/user/queries/index.md +++ b/docs/user/queries/index.md @@ -4,9 +4,9 @@ A `.gq` file contains named, typed queries. Read queries match graph patterns and return columns; mutation queries use the same declaration form and are covered in [Mutations](../mutations/index.md). A file may instead hold exactly one statement, never beside a query declaration: a branch statement (`branch -create`, `branch delete`, `branch merge`, or `branch list`; see -[Branches, Commits, and History](../branching/index.md)), a `show` statement -(see [session settings](#session-settings)), or an `explain` statement, which +create`, `delete`, `merge`, `list`, `diff`, or `merge-base`; see [Branches, +Commits, and History](../branching/index.md)), a `show` statement (see +[session settings](#session-settings)), or an `explain` statement, which answers the plan a read query would run under instead of its rows (see [Explain](explain.md)). Any of them may open with [session settings](#session-settings) lines. diff --git a/openapi.json b/openapi.json index b621e3a6d..27928ac54 100644 --- a/openapi.json +++ b/openapi.json @@ -7156,7 +7156,7 @@ "mutations" ], "summary": "Apply a GQ mutation to a branch (canonical mutation endpoint).", - "description": "Writes to the named `branch` (defaults to `main`). Mutations are atomic\nper call and produce a new commit. Returns counts of nodes and edges\naffected. **Destructive**: on success the branch is updated; rejected\nmutations may still acquire locks briefly. Returns 409 when the prepared\nwrite authority changes before effects.\n\nConditional callers use `POST /mutate/if-graph-commit`, which requires and\nvalidates `Omnigraph-If-Graph-Commit`. This endpoint rejects that header.\n\nPairs with `POST /query` (read-only).\n\nThe GQ statements `branch create`, `branch delete`, and `branch merge`\n(grammar: `BranchStmt` in `omnigraph-compiler`) are also served\nhere, with no `branch`, `name`, or `params`: each runs the handler body\nof its `/branches` route (same Cedar action, admission check, and errors,\nincluding the 409 of a conflicting merge) and answers a `ChangeOutput`\nwhose `outcome` names the effect. `branch list` is rejected with 400;\nsend it to `POST /query`.", + "description": "Writes to the named `branch` (defaults to `main`). Mutations are atomic\nper call and produce a new commit. Returns counts of nodes and edges\naffected. **Destructive**: on success the branch is updated; rejected\nmutations may still acquire locks briefly. Returns 409 when the prepared\nwrite authority changes before effects.\n\nConditional callers use `POST /mutate/if-graph-commit`, which requires and\nvalidates `Omnigraph-If-Graph-Commit`. This endpoint rejects that header.\n\nPairs with `POST /query` (read-only).\n\nThe GQ statements `branch create`, `branch delete`, and `branch merge`\n(grammar: `BranchStmt` in `omnigraph-compiler`) are also served\nhere, with no `branch`, `name`, or `params`: each runs the handler body\nof its `/branches` route (same Cedar action, admission check, and errors,\nincluding the 409 of a conflicting merge) and answers a `ChangeOutput`\nwhose `outcome` names the effect. `branch list` and the branch control\nreads (`branch diff`, `branch merge-base`, `branch merge … dry run`) are\nrejected with 400; send them to `POST /query`.", "operationId": "cluster_mutate", "parameters": [ { @@ -7214,7 +7214,7 @@ } }, "400": { - "description": "Bad request - also returned when `branch list` arrives here instead of POST /query, when a request target accompanies a branch statement, when a name or parameters accompany a branch statement, and when a commit precondition accompanies a branch statement; api_contract_mismatch when the required API contract header is missing, duplicated or unsupported", + "description": "Bad request - also returned when `branch list` or a branch control read (`branch diff`, `branch merge-base`, `branch merge … dry run`) arrives here instead of POST /query, when a request target accompanies a branch statement, when a name or parameters accompany a branch statement, and when a commit precondition accompanies a branch statement; api_contract_mismatch when the required API contract header is missing, duplicated or unsupported", "headers": { "omnigraph-http-api": { "schema": { @@ -8587,7 +8587,7 @@ "queries" ], "summary": "Execute an inline read query.", - "description": "Designed for ad-hoc exploration and AI-agent tool-use: short field\nnames (`query`, `name`) match the CLI `-e` flag and the GQ `query`\nkeyword. Mutations (`insert`/`update`/`delete`) are rejected with 400\n-- use `POST /mutate` for writes. Select a branch or snapshot and obtain\nthe pinned graph-commit token with the response. Cedar authorizes Read.\n\nThe GQ statement `branch list` is also served here, with no `branch`,\n`snapshot`, `name`, or `params`: it answers one result per branch (field\n`name`, byte order) under the same `read` check as `GET /branches`.\n`branch create`, `branch delete`, and `branch merge` are rejected with\n400; send them to `POST /mutate`.\n\nThe GQ statement `explain query …` is served here too: the query is not\nrun, and the result describes its v2 logical, physical and available\nDataFusion trees, one result per node (fields `tree`, `depth`, `node`, `detail`), followed by\n`plan` entries for the passes and the document's other fields, under the\nrequest's target and `params`.", + "description": "Designed for ad-hoc exploration and AI-agent tool-use: short field\nnames (`query`, `name`) match the CLI `-e` flag and the GQ `query`\nkeyword. Mutations (`insert`/`update`/`delete`) are rejected with 400\n-- use `POST /mutate` for writes. Select a branch or snapshot and obtain\nthe pinned graph-commit token with the response. Cedar authorizes Read.\n\nThe GQ statement `branch list` is also served here, with no `branch`,\n`snapshot`, `name`, or `params`: it answers one result per branch (field\n`name`, byte order) under the same `read` check as `GET /branches`. So are\nthe branch control reads, each under its route's checks and errors:\n`branch diff [into ]` and `branch diff to `\n(one result per entity change, fields `kind`, `type`, `id`, `op`,\n`before`, `after`; the compared commits and the continuation token in\n`page`; `after page \"\"` and `limit ` continue and bound it),\n`branch merge-base [into ]` and\n`branch merge [into ] dry run` (one result each).\n`branch create`, `branch delete`, and `branch merge` are rejected with\n400; send them to `POST /mutate`.\n\nThe GQ statement `explain query …` is served here too: the query is not\nrun, and the result describes its v2 logical, physical and available\nDataFusion trees, one result per node (fields `tree`, `depth`, `node`, `detail`), followed by\n`plan` entries for the passes and the document's other fields, under the\nrequest's target and `params`.", "operationId": "cluster_query", "parameters": [ { @@ -11470,7 +11470,7 @@ }, "query": { "type": "string", - "description": "GQ read-query source. May declare one or more named queries; pick one\nwith `name` when more than one is declared. Mutations\n(`insert`/`update`/`delete`) get 400 — use `POST /mutate` instead. May be the branch\nstatement `branch list`, sent with no `name`, `params`, `branch`, or\n`snapshot`; or one `explain` statement (`explain query …`), which\nanswers the v2 plan instead of\nrunning it, as one result per plan node (fields `tree`, `depth`, `node`,\n`detail`: the logical, physical and available DataFusion trees, then `plan` entries for\nthe passes and the document's other fields), under the same `params`,\n`branch`, or `snapshot` as the query itself.", + "description": "GQ read-query source. May declare one or more named queries; pick one\nwith `name` when more than one is declared. Mutations\n(`insert`/`update`/`delete`) get 400 — use `POST /mutate` instead. May be the branch\nstatement `branch list` or a branch control read (`branch diff …`,\n`branch merge-base …`, `branch merge … dry run`), sent with no `name`,\n`params`, `branch`, or `snapshot`; or one `explain` statement (`explain query …`), which\nanswers the v2 plan instead of\nrunning it, as one result per plan node (fields `tree`, `depth`, `node`,\n`detail`: the logical, physical and available DataFusion trees, then `plan` entries for\nthe passes and the document's other fields), under the same `params`,\n`branch`, or `snapshot` as the query itself.", "example": "query get_person($name: String) {\n match {\n $p: Person { name: $name }\n }\n return { $p.name, $p.age }\n}" }, "settings": { @@ -11516,6 +11516,17 @@ ], "description": "Effective graph head commit id of the exact snapshot this read was\nserved from. On a fresh named branch this is the inherited source head,\nso it is immediately usable as `Omnigraph-If-Graph-Commit` (CLI:\n`--if-commit`) for the branch's first conditional write. The id and rows\ncome from one pinned version, so no separate id fetch is needed." }, + "page": { + "oneOf": [ + { + "type": "null" + }, + { + "$ref": "#/components/schemas/ReadPageOutput", + "description": "Page state of a paged control read (`branch diff`); absent for every\nother answer." + } + ] + }, "query_name": { "type": "string" }, @@ -11529,6 +11540,37 @@ } } }, + "ReadPageOutput": { + "type": "object", + "description": "The page state a `branch diff` answer carries: the exact commits it\ncompares, and the token `after page \"\"` continues it with, absent\non its last page.", + "properties": { + "base_commit_id": { + "type": [ + "string", + "null" + ], + "description": "The merge base, for a branch's diff since it." + }, + "from_commit_id": { + "type": [ + "string", + "null" + ] + }, + "next_page_token": { + "type": [ + "string", + "null" + ] + }, + "to_commit_id": { + "type": [ + "string", + "null" + ] + } + } + }, "ReadSetConflictOutput": { "type": "object", "description": "Structured authority mismatch for a prepared write. Values are\nstrings because members include optional graph commit ids and future\nauthority tokens, not only numeric published dataset versions.", From 337b6ed0bd3da94594a24505c24c14eb290ecfe9 Mon Sep 17 00:00:00 2001 From: Ragnor Comerford Date: Wed, 14 Oct 2026 09:17:51 -0700 Subject: [PATCH 11/11] fix(changes): authorize each diff side before the next is resolved `/diff` resolved both sides before it authorized either, so with a forbidden `from` an unknown `to` answered first and the error named whichever side the policy hid. The engine now takes the caller's read check (`diff_page_readable`) and authorizes each two-point side as soon as it is classified, before the next side is looked at or a snapshot opens; a refused side is the not-found of a side that names nothing. A continuation authorizes its sides again before it reopens the token's commits. The two-phase `prepare_diff_page` / `run_diff_page` API is gone, so no transport has to get the order right. A diff endpoint commit whose writing branch is gone is the not-found of the spelling that named it, mapped where the engine knows that spelling, instead of the transport guessing a side from a branch name. One helper decides a side's spelling. The CLI page-size note and the changes guide say that every page restarts its comparison, and the note names `diff` and `branch diff`. --- ...change-pages-at-the-maximum.performance.md | 15 +- crates/omnigraph-server/src/handlers.rs | 96 ++++---- .../tests/integration/auth_policy.rs | 44 +++- crates/omnigraph/src/db/mod.rs | 6 +- crates/omnigraph/src/db/omnigraph.rs | 222 +++++++++++------- crates/omnigraph/tests/forbidden_apis.rs | 3 +- crates/omnigraph/tests/integration/changes.rs | 61 +++-- docs/rfcs/0030-cdc-time-travel.md | 6 +- docs/user/branching/changes.md | 14 +- 9 files changed, 284 insertions(+), 183 deletions(-) diff --git a/changelog.d/cli-change-pages-at-the-maximum.performance.md b/changelog.d/cli-change-pages-at-the-maximum.performance.md index 8be84600d..a7393befa 100644 --- a/changelog.d/cli-change-pages-at-the-maximum.performance.md +++ b/changelog.d/cli-change-pages-at-the-maximum.performance.md @@ -1,9 +1,10 @@ -- **Fewer, larger change pages from the CLI.** When `commit changes` or - `changes poll` follows every page itself, it now asks for 8,192 changes - per page instead of the 1,000 the HTTP routes default to. Every page - resumes its comparison where the last one stopped, so a large result - costs fewer requests and less time; the output is unchanged. `--limit` - still sets the size, and a single `--page-token` fetch still defaults to - 1,000. See [changes and change feeds][cli-change-pages]. +- **Fewer, larger change pages from the CLI.** When `commit changes`, + `changes poll`, `diff` or `branch diff` follows every page itself, it now + asks for 8,192 changes per page instead of the 1,000 the HTTP routes + default to. Every page restarts its comparison after the last change the + previous page returned, so fewer, larger pages cost fewer requests and + less time; the output is unchanged. `--limit` still sets the size, and a + single `--page-token` fetch still defaults to 1,000. See + [changes and change feeds][cli-change-pages]. [cli-change-pages]: ../docs/user/branching/changes.md#inspect-one-commit diff --git a/crates/omnigraph-server/src/handlers.rs b/crates/omnigraph-server/src/handlers.rs index 10eeab434..55655cb12 100644 --- a/crates/omnigraph-server/src/handlers.rs +++ b/crates/omnigraph-server/src/handlers.rs @@ -4018,33 +4018,6 @@ fn required_diff_param(value: Option, name: &str) -> std::result::Result } } -/// Authorize `read` on the branch one graph diff side resolves to: a branch -/// name, or the authored branch (a proposal's or snapshot's recorded target) -/// of a commit. The side is resolved before the policy check, so a denial is -/// reported exactly as a side that names nothing: the route is not an -/// existence oracle for branches or commits the actor cannot read. -fn authorize_diff_side( - actor: Option<&AuthenticatedActor>, - policy: Option<&PolicyEngine>, - side: &str, - branch: &str, -) -> std::result::Result<(), ApiError> { - match authorize( - actor, - policy, - PolicyRequest { - action: PolicyAction::Read, - branch: Some(branch.to_string()), - target_branch: None, - }, - )? { - Authz::Allowed => Ok(()), - Authz::Denied(_) => Err(ApiError::not_found(format!( - "'{side}' names no branch and no commit" - ))), - } -} - #[utoipa::path( get, path = "/diff", @@ -4096,9 +4069,13 @@ pub(crate) async fn server_diff( } /// One page of the net change between two read targets, for `GET /diff` and -/// the `branch diff to ` statement: each side is authorized on the -/// branch it resolves to, and a side the actor cannot read answers the 404 a -/// side naming nothing does. +/// the `branch diff to ` statement. The engine authorizes each side +/// on the branch it resolves to (a branch name, or the authored branch, a +/// proposal's or snapshot's recorded target, of a commit) as soon as it +/// classifies that side, before it looks at the next side or opens a +/// snapshot. A side the actor cannot read is the 404 of a side that names +/// nothing, whichever side it is: the route is not an existence oracle for +/// branches or commits the actor cannot read. pub(crate) async fn two_point_diff_page( handle: &GraphRequest, actor: Option<&AuthenticatedActor>, @@ -4108,33 +4085,42 @@ pub(crate) async fn two_point_diff_page( page_token: Option, max_changes: Option, ) -> std::result::Result { - let db = &handle.engine; - // One resolution serves authorization and the read: each side is - // authorized on the branch it resolved to, before any of its data is read. - let prepared = db - .prepare_diff_page(&omnigraph::changes::GraphDiffRequest { - sides: omnigraph::changes::GraphDiffSides::Between { - from: from.clone(), - to: to.clone(), - }, - scope, - page_token, - max_changes, - max_bytes: None, - }) - .await - .map_err(|error| match error { - // A branch deleted between resolution steps is that side's 404. - OmniError::BranchNotFound { branch } => { - let side = if branch == from.trim() { &from } else { &to }; - ApiError::not_found(format!("'{side}' names no branch and no commit")) + let policy = handle.policy.as_deref(); + // A policy failure (a missing bearer, an evaluation error) is not a + // denial: it stops the read, and its own error answers. + let mut failure = None; + let mut can_read = |branch: &str| { + let request = PolicyRequest { + action: PolicyAction::Read, + branch: Some(branch.to_string()), + target_branch: None, + }; + match authorize(actor, policy, request) { + Ok(Authz::Allowed) => true, + Ok(Authz::Denied(_)) => false, + Err(error) => { + failure.get_or_insert(error); + false } - error => change_route_error(error), - })?; - for (side, branch) in prepared.side_branches() { - authorize_diff_side(actor, handle.policy.as_deref(), side, branch)?; + } + }; + let page = handle + .engine + .diff_page_readable( + &omnigraph::changes::GraphDiffRequest { + sides: omnigraph::changes::GraphDiffSides::Between { from, to }, + scope, + page_token, + max_changes, + max_bytes: None, + }, + &mut can_read, + ) + .await; + if let Some(error) = failure { + return Err(error); } - db.run_diff_page(prepared).await.map_err(change_route_error) + page.map_err(change_route_error) } /// Authorize `read` on each named branch of a branch diff or merge base. diff --git a/crates/omnigraph-server/tests/integration/auth_policy.rs b/crates/omnigraph-server/tests/integration/auth_policy.rs index 24d957baa..fb0c8fec3 100644 --- a/crates/omnigraph-server/tests/integration/auth_policy.rs +++ b/crates/omnigraph-server/tests/integration/auth_policy.rs @@ -1633,6 +1633,21 @@ async fn policy_commit_diff_forbidden_is_indistinguishable_from_unknown() { .await .unwrap(); let feature_commit = receipt.commit.graph_commit_id.clone(); + // A commit whose branch is deleted afterwards. + db.branch_create_from(ReadTarget::branch("main"), "gone") + .await + .unwrap(); + let gone_commit = db + .load_with_receipt( + "gone", + r#"{"type":"Person","data":{"name":"gone-only","age":41}}"#, + omnigraph::loader::LoadMode::Merge, + ) + .await + .unwrap() + .commit + .graph_commit_id; + db.branch_delete("gone").await.unwrap(); drop(db); let policy_path = temp.path().join("policy.yaml"); @@ -1687,9 +1702,9 @@ async fn policy_commit_diff_forbidden_is_indistinguishable_from_unknown() { "forbidden and unknown must share the same error code" ); - // A graph diff resolves each side before it authorizes the branch the side - // resolves to, so a forbidden side, commit or branch, answers exactly as a - // side that names nothing. + // A graph diff classifies each side before it authorizes the branch the + // side resolves to, so a forbidden side, commit or branch, answers exactly + // as a side that names nothing. let get = |uri: String| { Request::builder() .header(HTTP_API_CONTRACT_HEADER, HTTP_API_CONTRACT) @@ -1734,6 +1749,29 @@ async fn policy_commit_diff_forbidden_is_indistinguishable_from_unknown() { ); assert_eq!(forbidden["code"], unknown["code"]); } + // The sides are authorized in order, each before the next is looked at: a + // forbidden `from` answers as an unknown `from` does whatever `to` names, + // so an unknown `to` cannot reveal that `from` exists. A commit whose + // branch was deleted is no longer readable and answers the same way. + let unknown_from_id = "01ARZ3NDEKTSV4RRFFQ69G5FAW"; + for (forbidden_from, unknown_from, to) in [ + ("feature", "no-such-branch-2", "no-such-branch"), + (feature_commit.as_str(), unknown_from_id, unknown_id), + (gone_commit.as_str(), unknown_from_id, "main"), + ] { + for from in [forbidden_from, unknown_from] { + let expected = format!("'{from}' names no branch and no commit"); + let (status, body) = + json_response(&app, get(format!("/diff?from={from}&to={to}"))).await; + assert_eq!(status, StatusCode::NOT_FOUND, "{body}"); + assert_eq!(body["error"], expected, "/diff from {from} to {to}"); + let statement = format!(r#"branch diff "{from}" to "{to}""#); + let (status, body) = + json_response(&app, statement_request("/query", "team-token", &statement)).await; + assert_eq!(status, StatusCode::NOT_FOUND, "{body}"); + assert_eq!(body["error"], expected, "{statement}"); + } + } // The branch routes name their branches outright: a read denial is a 403, // and a dry run is a read of both branches. for uri in [ diff --git a/crates/omnigraph/src/db/mod.rs b/crates/omnigraph/src/db/mod.rs index 16efe1bbe..4764178c9 100644 --- a/crates/omnigraph/src/db/mod.rs +++ b/crates/omnigraph/src/db/mod.rs @@ -19,9 +19,9 @@ pub use omnigraph::{ CleanupPolicyOptions, CollectorCost, CollectorPathSnapshot, CollectorReport, CollectorRowSummary, DatasetCleanupStats, DatasetOptimizeStats, DatasetRepairStats, EXPORT_CHUNK_MAX_BYTES, ExportCut, FullTextIndexRebuildResult, GraphCreateReconciliation, - InitOptions, MergeOutcome, MergeResult, Omnigraph, OpenMode, PendingIndex, PreparedDiffPage, - PreparedGraphCreate, PreparedSchemaApply, PreparedSchemaSettlement, RebuiltFullTextIndex, - RepairAction, RepairClassification, RepairOptions, RepairStats, RetainedManifestVersions, + InitOptions, MergeOutcome, MergeResult, Omnigraph, OpenMode, PendingIndex, PreparedGraphCreate, + PreparedSchemaApply, PreparedSchemaSettlement, ReadCheck, RebuiltFullTextIndex, RepairAction, + RepairClassification, RepairOptions, RepairStats, RetainedManifestVersions, SYSTEM_COLUMNS_PREFLIGHT, SchemaApplyReconciliation, SchemaApplyResult, SchemaApplySettlement, SchemaContractDigest, SchemaNonPublicationProof, SkipReason, StagingVerdict, SystemColumnUpgradeFinding, SystemColumnUpgradeOptions, SystemColumnUpgradeOutcome, diff --git a/crates/omnigraph/src/db/omnigraph.rs b/crates/omnigraph/src/db/omnigraph.rs index 22ec54704..894bc5dca 100644 --- a/crates/omnigraph/src/db/omnigraph.rs +++ b/crates/omnigraph/src/db/omnigraph.rs @@ -2702,28 +2702,36 @@ impl Omnigraph { &self, request: &crate::changes::GraphDiffRequest, ) -> Result { - let prepared = self.prepare_diff(request, false).await?; + let prepared = self.prepare_diff(request, None).await?; self.run_diff_page(prepared).await } - /// Resolve a graph diff page request once, reading lineage only: its - /// endpoints, its continuation position, and the branch each two-point - /// side reads ([`PreparedDiffPage::side_branches`]). A transport - /// authorizes those branches and then runs the page with - /// [`Self::run_diff_page`], so authorization and execution see the one - /// resolution. A continuation resolves its endpoints from the page token - /// and classifies each spelled side again for authorization. - pub async fn prepare_diff_page( + /// [`Self::diff_page`] for a caller that may read only some branches. + /// `can_read` answers for the branch each two-point side reads: a branch + /// itself, or a commit's authored branch (an unpublished proposal's or + /// unattached snapshot's recorded target). Each side is authorized as + /// soon as it is classified, before the next side is looked at and before + /// its snapshot opens, and a side `can_read` refuses is + /// [`OmniError::ReadTargetNotFound`] naming it, exactly as a side that + /// names nothing: the answer is not an existence oracle. A continuation + /// classifies and authorizes its sides again before it reopens the + /// token's commits. The branch form names its branches outright, so its + /// caller authorizes them before the call. + pub async fn diff_page_readable( &self, request: &crate::changes::GraphDiffRequest, - ) -> Result { - self.prepare_diff(request, true).await + can_read: &mut ReadCheck<'_>, + ) -> Result { + let prepared = self.prepare_diff(request, Some(can_read)).await?; + self.run_diff_page(prepared).await } + /// Resolve a graph diff page request, reading lineage only: its + /// endpoints and its continuation position. async fn prepare_diff( &self, request: &crate::changes::GraphDiffRequest, - side_branches_on_continuation: bool, + can_read: Option<&mut ReadCheck<'_>>, ) -> Result { use crate::changes::token; @@ -2738,12 +2746,12 @@ impl Omnigraph { let graph_identity = self.schema_view.load().schema_identity_domain.clone(); let filter_digest = token::filter_digest(&request.scope); let sides_digest = token::sides_digest(&request.sides); - let (endpoints, resume, side_branches) = match request.page_token.as_deref() { - None => { - let (endpoints, side_branches) = - self.resolve_diff_endpoints(&request.sides).await?; - (endpoints, None, side_branches) - } + let (endpoints, resume) = match request.page_token.as_deref() { + None => ( + self.resolve_diff_endpoints(&request.sides, can_read) + .await?, + None, + ), Some(encoded) => { let decoded = token::decode_diff_page_token(encoded)?; if decoded.graph_identity != token::hashed_identity(&graph_identity) @@ -2758,13 +2766,12 @@ impl Omnigraph { "diff page token was minted for a different filter scope", )); } - let mut side_branches = Vec::new(); - if side_branches_on_continuation + if let Some(can_read) = can_read && let crate::changes::GraphDiffSides::Between { from, to } = &request.sides { for side in [from, to] { - let branch = self.classify_diff_read_target(side).await?.read_branch(); - side_branches.push((side.trim().to_string(), branch)); + self.authorized_diff_side(side, Some(&mut *can_read)) + .await?; } } let endpoints = self @@ -2780,7 +2787,7 @@ impl Omnigraph { operation_rank: decoded.operation_rank, change_index: decoded.change_index, }; - (endpoints, Some(resume), side_branches) + (endpoints, Some(resume)) } }; Ok(PreparedDiffPage { @@ -2792,12 +2799,11 @@ impl Omnigraph { sides_digest, endpoints, resume, - side_branches, }) } - /// Read one prepared graph diff page; see [`Self::prepare_diff_page`]. - pub async fn run_diff_page( + /// Read one prepared graph diff page. + async fn run_diff_page( &self, prepared: PreparedDiffPage, ) -> Result { @@ -2812,7 +2818,6 @@ impl Omnigraph { sides_digest, endpoints, resume, - side_branches: _, } = prepared; // Reclaimed history under either endpoint is a typed gap naming the // diff's `to` commit; a diff has no feed cursor to resume. @@ -2936,33 +2941,29 @@ impl Omnigraph { )) } - /// Resolve a graph diff's sides to exact commits and their snapshots, - /// with the branch each two-point side reads (none for the branch form, - /// whose transport authorizes the two branches it names). + /// Resolve a graph diff's sides to exact commits and their snapshots. + /// The two-point form takes its sides in order, each authorized by + /// `can_read` before the next is looked at; the branch form's caller + /// authorizes the two branches it names. async fn resolve_diff_endpoints( &self, sides: &crate::changes::GraphDiffSides, - ) -> Result<(DiffEndpoints, Vec<(String, String)>)> { + mut can_read: Option<&mut ReadCheck<'_>>, + ) -> Result { match sides { crate::changes::GraphDiffSides::Between { from, to } => { - let (from_commit_id, from_snapshot, from_branch) = - self.resolve_diff_read_target(from).await?; - let (to_commit_id, to_snapshot, to_branch) = - self.resolve_diff_read_target(to).await?; - let side_branches = vec![ - (from.trim().to_string(), from_branch), - (to.trim().to_string(), to_branch), - ]; - Ok(( - DiffEndpoints { - from_commit_id, - to_commit_id, - base_commit_id: None, - from_snapshot, - to_snapshot, - }, - side_branches, - )) + let (from_commit_id, from_snapshot) = self + .resolve_diff_read_target(from, can_read.as_deref_mut()) + .await?; + let (to_commit_id, to_snapshot) = + self.resolve_diff_read_target(to, can_read).await?; + Ok(DiffEndpoints { + from_commit_id, + to_commit_id, + base_commit_id: None, + from_snapshot, + to_snapshot, + }) } crate::changes::GraphDiffSides::Branch { source, target } => { let (source_head, target_head, merge_base, source_snapshot, target_snapshot) = @@ -2976,18 +2977,16 @@ impl Omnigraph { } else if base_commit_id == target_head { target_snapshot } else { - self.commit_snapshot(&base_commit_id).await? + self.diff_endpoint_snapshot(&base_commit_id, &base_commit_id) + .await? }; - Ok(( - DiffEndpoints { - from_commit_id: base_commit_id.clone(), - to_commit_id: source_head, - base_commit_id: Some(base_commit_id), - from_snapshot: base_snapshot, - to_snapshot: source_snapshot, - }, - Vec::new(), - )) + Ok(DiffEndpoints { + from_commit_id: base_commit_id.clone(), + to_commit_id: source_head, + base_commit_id: Some(base_commit_id), + from_snapshot: base_snapshot, + to_snapshot: source_snapshot, + }) } } } @@ -3003,8 +3002,12 @@ impl Omnigraph { from_commit_id: from_commit_id.to_string(), to_commit_id: to_commit_id.to_string(), base_commit_id, - from_snapshot: self.commit_snapshot(from_commit_id).await?, - to_snapshot: self.commit_snapshot(to_commit_id).await?, + from_snapshot: self + .diff_endpoint_snapshot(from_commit_id, from_commit_id) + .await?, + to_snapshot: self + .diff_endpoint_snapshot(to_commit_id, to_commit_id) + .await?, }) } @@ -3019,34 +3022,76 @@ impl Omnigraph { .snapshot) } - /// Resolve one diff endpoint to its commit, the graph as of it, and the - /// branch it reads. - async fn resolve_diff_read_target(&self, target: &str) -> Result<(String, Snapshot, String)> { - let classified = self.classify_diff_read_target(target).await?; - let branch = classified.read_branch(); + /// The graph as of one diff endpoint commit. A commit whose writing + /// branch is gone, or that a raced branch deletion hides mid-lookup, is + /// no longer readable: it is the not-found of the spelling that named it, + /// never the branch its record names, which the caller may not be allowed + /// to learn. + async fn diff_endpoint_snapshot(&self, commit_id: &str, spelling: &str) -> Result { + self.commit_snapshot(commit_id) + .await + .map_err(|error| match error { + OmniError::BranchNotFound { .. } => OmniError::ReadTargetNotFound { + target: spelling.to_string(), + }, + error => error, + }) + } + + /// Resolve one authorized diff endpoint to its commit and the graph as + /// of it. + async fn resolve_diff_read_target( + &self, + side: &str, + can_read: Option<&mut ReadCheck<'_>>, + ) -> Result<(String, Snapshot)> { + let (side, classified) = self.authorized_diff_side(side, can_read).await?; match classified { DiffReadTarget::Branch(resolved) => { let commit_id = resolved.graph_commit_id.ok_or_else(|| { - OmniError::manifest(format!("branch '{}' has no head commit", target.trim())) + OmniError::manifest(format!("branch '{side}' has no head commit")) })?; - Ok((commit_id, resolved.snapshot, branch)) + Ok((commit_id, resolved.snapshot)) } DiffReadTarget::Commit(commit) => { - let snapshot = self.commit_snapshot(&commit.graph_commit_id).await?; - Ok((commit.graph_commit_id, snapshot, branch)) + let snapshot = self + .diff_endpoint_snapshot(&commit.graph_commit_id, side) + .await?; + Ok((commit.graph_commit_id, snapshot)) } } } - /// Classify one diff endpoint: a live branch (its head now) or a graph - /// commit. A string naming both is refused, never resolved by precedence; - /// one naming neither is a typed not-found carrying only the caller's - /// spelling. - async fn classify_diff_read_target(&self, target: &str) -> Result { - let target = target.trim(); - if target.is_empty() { + /// Classify one diff side as the caller spelled it, then, when the caller + /// may read only some branches, authorize the branch it reads before + /// anything else is resolved: a refused side is the not-found of a side + /// that names nothing. Answers the side's spelling with what it names. + async fn authorized_diff_side<'a>( + &self, + side: &'a str, + can_read: Option<&mut ReadCheck<'_>>, + ) -> Result<(&'a str, DiffReadTarget)> { + // The one place a side's spelling is decided. + let side = side.trim(); + if side.is_empty() { return Err(OmniError::manifest("a diff endpoint cannot be empty")); } + let classified = self.classify_diff_read_target(side).await?; + if let Some(can_read) = can_read + && !can_read(&classified.read_branch()) + { + return Err(OmniError::ReadTargetNotFound { + target: side.to_string(), + }); + } + Ok((side, classified)) + } + + /// Classify one spelled diff endpoint: a live branch (its head now) or a + /// graph commit. A string naming both is refused, never resolved by + /// precedence; one naming neither is a typed not-found carrying only the + /// caller's spelling. + async fn classify_diff_read_target(&self, target: &str) -> Result { // A string the branch-name rules refuse cannot name a branch; that is // not an error here, since it may still name a commit. let as_branch = match normalize_branch_name(target) { @@ -4518,6 +4563,10 @@ pub(crate) fn schema_for_table_key(catalog: &Catalog, table_key: &str) -> Result ))) } +/// A caller's answer to "may it read this branch?", asked once per two-point +/// graph diff side; see [`Omnigraph::diff_page_readable`]. +pub type ReadCheck<'a> = dyn FnMut(&str) -> bool + Send + 'a; + /// What one graph diff endpoint names. enum DiffReadTarget { Branch(Box), @@ -4525,7 +4574,7 @@ enum DiffReadTarget { } impl DiffReadTarget { - /// The branch a transport authorizes `read` on: the branch itself, or the + /// The branch a caller authorizes `read` on: the branch itself, or the /// authored branch of the commit (an unpublished proposal's or unattached /// snapshot's recorded target). fn read_branch(&self) -> String { @@ -4537,9 +4586,9 @@ impl DiffReadTarget { } } -/// A graph diff page request resolved once; see -/// [`Omnigraph::prepare_diff_page`]. -pub struct PreparedDiffPage { +/// A graph diff page request resolved to its endpoints and continuation +/// position. +struct PreparedDiffPage { scope: crate::changes::ChangeFeedScope, max_changes: usize, max_bytes: u64, @@ -4548,15 +4597,6 @@ pub struct PreparedDiffPage { sides_digest: String, endpoints: DiffEndpoints, resume: Option, - side_branches: Vec<(String, String)>, -} - -impl PreparedDiffPage { - /// Each two-point side as the caller spelled it, with the branch it - /// reads; empty for the branch form. - pub fn side_branches(&self) -> &[(String, String)] { - &self.side_branches - } } /// A graph diff's two endpoints, resolved to exact commits and their diff --git a/crates/omnigraph/tests/forbidden_apis.rs b/crates/omnigraph/tests/forbidden_apis.rs index 3e5e5b612..aab972e2d 100644 --- a/crates/omnigraph/tests/forbidden_apis.rs +++ b/crates/omnigraph/tests/forbidden_apis.rs @@ -312,8 +312,7 @@ const READ_ONLY_SURFACES: &[(&str, &str)] = &[ ("db/omnigraph.rs", "resolve_snapshot"), ("db/omnigraph.rs", "diff_page"), ("db/omnigraph.rs", "merge_base"), - ("db/omnigraph.rs", "prepare_diff_page"), - ("db/omnigraph.rs", "run_diff_page"), + ("db/omnigraph.rs", "diff_page_readable"), ("exec/merge.rs", "branch_merge_dry_run"), ("session.rs", "branch_read"), ("db/omnigraph.rs", "commit_changes_page"), diff --git a/crates/omnigraph/tests/integration/changes.rs b/crates/omnigraph/tests/integration/changes.rs index 096099444..957742d21 100644 --- a/crates/omnigraph/tests/integration/changes.rs +++ b/crates/omnigraph/tests/integration/changes.rs @@ -4548,24 +4548,55 @@ async fn graph_diff_refuses_an_ambiguous_or_unknown_side() { } } - // One resolution names the branch each side reads: a branch itself, and - // a commit's authored branch. + // A caller that may read only some branches is asked, side by side, for + // the branch each side reads (a branch itself, or a commit's authored + // branch) as soon as the side is classified. A refused side is the + // not-found of a side that names nothing, and the next side is never + // looked at, so an unknown `to` cannot reveal that `from` exists. db.branch_create("feature").await.unwrap(); let on_feature = write_on(&db, "feature", "insert_person", &person("Fay", 30)).await; - let prepared = db - .prepare_diff_page(&diff_request( - between("main", &on_feature.graph_commit_id), - None, - )) + for from in ["feature", on_feature.graph_commit_id.as_str()] { + let mut asked = Vec::new(); + let refused = db + .diff_page_readable( + &diff_request(between(from, "no-such-side"), None), + &mut |branch: &str| { + asked.push(branch.to_string()); + branch == "main" + }, + ) + .await + .unwrap_err(); + match refused { + OmniError::ReadTargetNotFound { target } => assert_eq!(target, from), + other => panic!("expected ReadTargetNotFound for {from}, got {other:?}"), + } + assert_eq!(asked, ["feature"], "a refused side ends the resolution"); + } + let mut asked = Vec::new(); + db.diff_page_readable( + &diff_request(between(" main ", &on_feature.graph_commit_id), None), + &mut |branch: &str| { + asked.push(branch.to_string()); + true + }, + ) + .await + .unwrap(); + assert_eq!(asked, ["main", "feature"]); + + // A commit whose branch was deleted is no longer readable: it is that + // side's not-found, never the branch its record names. + db.branch_create("gone").await.unwrap(); + let on_gone = write_on(&db, "gone", "insert_person", &person("Gil", 31)).await; + db.branch_delete("gone").await.unwrap(); + match graph_diff_all(&db, between(&on_gone.graph_commit_id, "main"), None) .await - .unwrap(); - assert_eq!( - prepared.side_branches(), - [ - ("main".to_string(), "main".to_string()), - (on_feature.graph_commit_id.clone(), "feature".to_string()), - ] - ); + .unwrap_err() + { + OmniError::ReadTargetNotFound { target } => assert_eq!(target, on_gone.graph_commit_id), + other => panic!("expected ReadTargetNotFound, got {other:?}"), + } } /// Continuations answer for the commits the first page resolved, even after diff --git a/docs/rfcs/0030-cdc-time-travel.md b/docs/rfcs/0030-cdc-time-travel.md index 443c62102..d88b152e3 100644 --- a/docs/rfcs/0030-cdc-time-travel.md +++ b/docs/rfcs/0030-cdc-time-travel.md @@ -712,8 +712,12 @@ unreadable endpoint maps to the §6 typed outcomes. Authorization of the two-point form is `read` on the branch each endpoint resolves to. A commit endpoint authorizes against its authored branch, or against the recorded target of a proposal or snapshot. The two-point route -resolves a side before it authorizes it, so a forbidden side is reported +classifies a side before it authorizes it, so a forbidden side is reported exactly as a side that names nothing (`404`), as the commit diff does. The +engine authorizes the sides in order, each as soon as it is classified and +before the next side is looked at or a snapshot opens, so what the other side +names cannot reveal whether a forbidden side exists. A commit whose writing +branch is gone is no longer readable and is reported the same way. The branch form and the merge base require `read` on both named branches, source and target, on every request, a continuation included, before they read lineage. Their endpoints are the base and the source head, which can both be diff --git a/docs/user/branching/changes.md b/docs/user/branching/changes.md index 1de57fe9f..e325d1b9d 100644 --- a/docs/user/branching/changes.md +++ b/docs/user/branching/changes.md @@ -35,9 +35,10 @@ identity, logical id, and operation. Large results are paginated. The CLI normally follows every `next_page_token` and asks for 8,192 changes per page, because every page -resumes its comparison where the last one stopped; passing `--page-token` -fetches exactly one page, of 1,000 changes unless `--limit` sets the size. A -page token continues that one commit result and is not a change-feed cursor. +restarts its comparison and fewer, larger pages cost less; passing +`--page-token` fetches exactly one page, of 1,000 changes unless `--limit` +sets the size. A page token continues that one commit result and is not a +change-feed cursor. A parentless commit returns `409` with reason `parentless_commit`. A diff that crosses a schema boundary returns `409` with reason @@ -192,9 +193,10 @@ Baseline kind and type filters select the snapshot contents. An operation filter applies only to the resumed feed. Baselines require the `export` policy action; commit changes, diffs and feed polling require `read`. A diff needs `read` on the branch each side resolves to; a side the actor cannot read -answers `404` exactly as a side that names nothing. `branch diff` and -`branch merge-base` need `read` on both branches, every page included, and -answer `403` otherwise. +answers `404` exactly as a side that names nothing, whatever the other side +names. A commit whose branch was deleted answers the same `404`. +`branch diff` and `branch merge-base` need `read` on both branches, every +page included, and answer `403` otherwise. The CLI's durable `--out` installation is available on POSIX platforms. The file contains only snapshot entity records, not the terminal handshake. The CLI