embeddings: served backfill and re-embedding jobs with progress and resume #656

Activity

1. ragnorc

   added

   needs-triageAwaiting maintainer triageAwaiting maintainer triage

   featureFeature proposalFeature proposal

   on Sep 5, 2026
2. azimafroozeh

   removed

   needs-triageAwaiting maintainer triageAwaiting maintainer triage

   on Oct 1, 2026
3. ragnorc commented on Oct 18, 2026

   

   ragnorc

   on Oct 18, 2026 – with Cursor

   ContributorAuthor

   More actions

   Summary

   A served graph has no operation that fills or refreshes stored embeddings. POST /graphs/{id}/load and POST /graphs/{id}/mutate commit the source text and leave a missing vector null. They return no job id, no progress, and no coverage counts.

   The offline command omnigraph embed reads a JSONL file and writes a JSONL file. It does not open a graph. A later load --mode merge of that file publishes whole entities. On a mixed set it fills a null vector, and it leaves a stale vector and a same-dimension hand vector unchanged. --reembed-all replaces every selected vector, including vectors that already match the source text. There is no preview, no row budget, no cancel flag, and no resume cursor. A source edit or a delete between export and load is overwritten by the file: the newer source text is replaced, and a deleted row comes back.

   Verdict

   Reproduced. The gap in the report is still the gap on current main.

   SHA: be031fea766fc8b280137f9ad9f34d4d2b3087bc (feat(query): the ranked binding roots its component (#1069), 2026-10-18). git fetch origin main returned that SHA. The checkout was that commit. The issue is still open. It has no comments and no linked fix pull request.

   Environment

   omnigraph version reports 0.14.0, storage format 16 (serves v14 to v16), and HTTP contract 0.13. The binaries were target/debug/omnigraph and target/debug/omnigraph-server from this checkout. No .rs file under crates/ is newer than the CLI binary. rustc is 1.97.1. Cargo.lock pins Lance 11.0.0.

   The provider was mock only. The process had OMNIGRAPH_EMBEDDINGS_MOCK=1, OMNIGRAPH_EMBED_PROVIDER=mock, and OMNIGRAPH_EMBED_MODEL=repro-mock. OPENROUTER_API_KEY, OPENAI_API_KEY, and GEMINI_API_KEY were unset. The served graph used providers.embedding.local.kind: mock with model repro-mock. No provider secret was set.

   Schema:

   node Doc {
       slug: String @key
       body: String
       note: String?
       embedding: Vector(4)? @embed("body", model="repro-mock")
   }
   

   The direct graph was a local init at /tmp/repro-656/graph.omni. The served graph was a local cluster at /tmp/repro-656/cluster, graph id docs, started with omnigraph-server --cluster file:///tmp/repro-656/cluster --bind 127.0.0.1:18081 --unauthenticated. HTTP calls sent omnigraph-http-api: 0.13. /readyz reported status: serving and ready_graph_count: 1.

   Five rows:

   slug	body	vector
   missing	missing body	absent
   current	current body	mock of current body
   stale	stale body after	mock of stale body before
   other	other model body	[1.0, 0.0, 0.0, 0.0]
   manual	manual body	mock of manual body

   note was keep on every row. The spec was {"dimension":4,"types":{"Doc":{"target":"embedding","fields":["body"]}}}.

   Served API and MCP

   GET /openapi.json on the running server returned OpenAPI 3.1.0, package version 0.14.0, and 33 paths. None of the paths contain embed, backfill, reembed, or job. The graph paths are snapshot, blob, branches, changes, commits, diff, export, load, load/ndjson, mutate, queries, query, and schema.

   These calls returned HTTP 404 with an empty body:

   • POST /graphs/docs/embeddings
   • POST /graphs/docs/embeddings/backfill
   • POST /graphs/docs/embed
   • POST /graphs/docs/jobs
   • GET /graphs/docs/embeddings

   omnigraph embed rejects graph addressing. Exit 1:

   `embed` is a local command; --server addresses a served graph and does not apply. It does not address a graph.
   

   The same refusal is returned for --graph and for --store.

   POST /graphs/docs/load with one Doc row, body served text, and no vector, returned HTTP 200. embedding_generation was "unsupported". total_entities was 1. The JSON has a commit and no job id. The following query returned row_count 1. The row had d.slug and d.body. It had no embedding key.

   POST /mcp and GET /mcp returned HTTP 404 on this server. Unauthenticated boot does not mount /mcp, because that route requires a credential source (lib.rs). The tool table that boot would mount has 16 tools: health, graphs, schema, queries, query, read, snapshot, branches_list, commits_list, commits_get, commits_changes, changes_poll, mutate, load, branches_create, branches_delete, branches_merge (mcp.rs). There is no embed or backfill tool. This run did not start an authenticated MCP session.

   #655 already shows the same load and mutate facts on this SHA: a nullable @embed vector stays null, a source-text update keeps the old vector, and nearest omits null vectors with no coverage count. This note does not repeat that matrix. It covers the operator job the report asks for.

   Offline export, embed, load

   omnigraph export --store /tmp/repro-656/graph.omni wrote 5 lines. The missing line had keys body, note, slug. It had no embedding key. The other four lines had a length-4 embedding.

   Default embed, which the JSON calls fill_missing:

   records 5, selected_records 5, embedded_records 1, cleaned_records 0, mode fill_missing, dimension 4, model repro-mock
   

   The one rewritten row was missing. Its new vector was the mock of missing body. current, manual, stale, and other kept the exported vectors. stale stayed the mock of stale body before (first component 0.362332). other stayed [1.0, 0.0, 0.0, 0.0]. The command did not compare either vector with the source text or with the model name on @embed.

   --reembed-all reported embedded_records 5 and mode reembed_all. Every row, including current, was replaced with a fresh mock of its exported body. The fresh mock of current body differs from the exported f32 by at most 2.5e-8. The mode still rewrites that row. There is no "unchanged, skip" result.

   --select Doc:slug=stale with no --reembed-all reported mode reembed_selected, selected_records 1, embedded_records 1. Only stale changed. missing stayed null. The selector is a client filter. It forces a refresh of the named rows. It does not detect a stale vector.

   A second fill_missing over the already filled file reported embedded_records 0. The command read all 5 records again. It did not resume from a cursor.

   These flags are rejected with exit 2 and unexpected argument: --preview, --dry-run, --resume, --checkpoint, --cancel, --max-rows, --budget, --missing-only, --status. The JSON summary is one object after the file is renamed into place. There is no progress line.

   With the mock variables unset and no API key, embed exited 1:

   OPENROUTER_API_KEY or OPENAI_API_KEY is required for the openai-compatible embedding provider
   

   fail-out.jsonl was absent. fail-out.jsonl.tmp existed and had length 0. The temp file is created before the client is built (embed.rs).

   A 3-line input whose second body is the number 7 failed at line 2:

   line 2: Doc.body is not a string; an `@embed` source property is a String
   

   bad-out.jsonl was absent. bad-out.jsonl.tmp held 1 line, the first record only. A later successful run of a different input renamed a new temp file over the output and removed the temp name. The failed input has no resume record. The next run starts at line 1.

   There is no stored failed-row state. A provider or input failure stops the file. The graph is untouched because this command never opens it.

   Model and source identity

   schema plan against a copy of the schema whose @embed model is other-model exited 0 with supported: false and one step:

   kind: unsupported_change
   entity: node:Doc.embedding
   reason: changing @embed source or model for 'Doc.embedding' is not supported in schema migration v1; rebuild the graph so stored vectors and their declared embedding space cannot diverge
   

   That plan writes nothing. It does not list rows. The stored hand vector on other is still invisible to it.

   A text nearest for stale body after, with the mock provider resolving to repro-mock, returned row_count 4. The JSON keys were query_name, target, row_count, columns, rows, and graph_commit_id.

   slug	distance
   current	0.8288197
   manual	0.9060212
   stale	3.1541338
   other	3.2033029

   missing was absent. The row whose body equals the query text is stale, at distance 3.15. other is ranked in the same list. The query checks the schema model against the provider model (search.rs). It does not read a model id from the stored vector. An explicit vector update also skips that check.

   This update publishes a vector that is not the embedding of the body, and it leaves the body in place:

   update Doc set { embedding: $v } where slug = $s
   

   $v was [0.25, 0.25, 0.25, 0.25] and $s was manual. affected_nodes was 1. The receipt has commit and no coverage fields. After the commit, manual.body was still manual body and manual.note was still keep. The embedding was [0.25, 0.25, 0.25, 0.25].

   Edit or delete between export and load

   After the export and the fill file were written, three commits landed on the graph:

   • missing.body became missing body edited. Its vector stayed null.
   • current.note became edited note.
   • other was deleted.

   The read then had row_count 4. manual.embedding was still the [0.25, …] update from the previous step.

   load --mode merge of the filled export published one commit (graph_manifest_version 6 to 7). total_entities was 5. embedding_generation was "unsupported". The read after that commit:

   slug	body	note	vector
   missing	missing body (the edit is gone)	keep	mock of missing body
   current	current body	keep (the edit is gone)	exported vector
   manual	manual body	keep	original mock (the [0.25, …] update is gone)
   stale	stale body after	keep	still the mock of stale body before
   other	other model body	keep	[1.0, 0.0, 0.0, 0.0] (the deleted row is back)

   The published vector on missing matches the reverted source text. The newer source text is not left in place under an old vector. The whole exported entity replaces the current entity.

   A vector-only file (slug plus embedding, no body) against this schema exited 1:

   {"error":"non-nullable property 'body' has null or invalid values","code":"bad_request"}

   The graph commit id did not change.

   On a second graph whose body and note are nullable, the same vector-only merge published. The row keep then had embedding [0.5, 0.5, 0.5, 0.5]. The JSON row had no body and no note. The source text was cleared.

   Difference from the report at 02d182b6

   The report's gap is unchanged. There is still no served backfill, no preview, no budget, no cancel, and no resume. Clients still own checkpoints, retries, and concurrent edits.

   Three offline details are newer than that tree, from #929 (d6604a19):

   • The file command embeds the source string as stored. It no longer embeds a type / field template.
   • The spec fields list must name exactly one property. A blank source drops any target value.
   • --select selects rows and forces a refresh of those rows (reembed_selected). It is not a missing-only filter.

   Load JSON now includes embedding_generation. On this run it was "unsupported" for every load that touched Doc. That field is a capability flag. It is not attempted, completed, skipped, or failed work. #655 records the same flag.

   schema plan now refuses an @embed model change and tells the operator to rebuild. The issue did not mention that refusal. The refusal does not re-embed rows, and it does not see a hand vector of the same dimension.

   Root cause

   @embed is catalog metadata: a source property name and an optional model string (EmbedSource). The stored value is a fixed-size list of floats. The row has no model id, no source hash, and no source revision.

   omnigraph embed is a local-plane file transform (planes.rs). The default mode embeds a row only when the target is absent, null, or an empty array (embedding_missing, used at lines 307-315). Any other array counts as present. --reembed-all, and any use of --select, embeds every selected row that has source text. The summary model is the provider model for the process (lines 350-357). The output row does not store it. The temp file is renamed to the output only after the last line (lines 337-339).

   load --mode merge builds a full-schema batch. A missing JSON property becomes null (loader/mod.rs). The keyed write uses WhenMatched::UpdateAll and WhenNotMatched::InsertAll (table_store.rs). A deleted id in the file is inserted again. A required null fails the load. A nullable null is published.

   The affected class is every historical embedding repair: CLI, HTTP load, MCP load, and a hand update of the vector column. The steady-state hole, a new write that never schedules a vector, is #655. The missing per-value model and source identity is #657.

   Recommended fix

   One reconciler should own both the historical job and the steady-state backlog. RFC 0015 already specifies that shape (ensure_embeddings, explicit trigger, no network call on the content write). Its registry status is draft and its implementation is not-started. Extend it before code, because the acceptance criteria need a fact the RFC's NULL-only pending predicate cannot see.

   The predicate, in order:

   1. Store, on the vector, the model id and a hash of the source text that produced it. This is the contract in #657 (open, accepted, needs-rfc). A same-dimension model change is then a row whose stored model differs from the @embed model. A stale row is a row whose stored hash differs from the current source text. A missing row is a null vector with source text present. Legacy rows with no provenance stay unknown until an explicit refresh writes the marker. Do not treat an unknown marker as repro-mock.
   2. missing-only selects null vector and present source. refresh selects model mismatch, hash mismatch, and unknown provenance. preview runs the same predicate and returns counts. It publishes nothing.
   3. The write is a partial-schema update of the vector and its marker, keyed by the entity id, with when_not_matched = DoNothing. KnownPresentUpdate already has that not-matched arm (table_store.rs). The matched update must also fail closed when the live source hash differs from the hash the provider call used. That row is skipped and counted. It is not written. A deleted id is not inserted.
   4. One batch is one manifest publication on the named branch. The batch size is the work budget. The response counts attempted, completed, skipped, and failed rows, plus the model and the predicate. After a lost response or a restart, the next call reads the same predicate. Rows whose marker matches the current source and model are complete. There is no side job row.
   5. The HTTP route and the CLI graph mode call that one engine entry. The server resolves the actor from the bearer token and the mutating entry applies policy (invariant 10). omnigraph embed on files stays a file tool. It must not grow a second writer.
   6. A provider error fails that batch with a typed error. The existing client retry bound stays the bound. The batch does not publish a short vector list. Memory is one batch of source texts and one batch of vectors.
   7. The vector index stays derived (invariant 7). The reconciler does not build it inline. optimize remains the explicit index pass. nearest already returns rows the index does not cover.

   NULL-on-source-change from RFC 0015 still belongs on the content write, after the reconciler can see a null. It makes a later source edit look like a missing vector. It does not label rows that are already stale, and it does not label a model change. Those rows need the marker in step 1.

   This keeps invariant 3 (the batch pins one snapshot, and a retry starts from the live predicate), invariant 4 (each batch publishes once), invariant 8 (a mismatch is a counted skip, and a model change stays a loud schema refusal until the marker exists), invariant 11 (the batch is the bound), and invariant 12 (the manifest and the row are the record). A job queue for work that the predicate can see is on the deny-list.

   Tradeoff. This design spends correctness first. A full refresh without the marker would rewrite every current row, which this run already shows (embedded_records 5, including current). The marker is a storage decision and needs the RFC #657 asks for. What this choice defers is a background loop and a faster batched provider call. RFC 0015 already defers the loop. #874 owns batching and OpenAI-compatible role hints for the file command. The reconciler should call that same batch method when it exists, so document hints are not a second client.

   Alternatives

   • Add --preview and a byte offset to the file command only. Scripts get a count and a cruder resume. Concurrent edits and deleted rows stay unsafe, because the writer is still a full-row merge. Same-dimension model changes stay invisible.
   • Rely on NULL-on-source-change alone. Future edits become missing rows. Existing stale rows and hand vectors stay as they are in this run.
   • Keep export, embed, and load, and document the clobber. The behavior above is then the contract. A deleted row comes back. A newer note is lost. That is a large liability for a repair tool.
   • Add a durable job table or a queue. The predicate is derivable once the marker exists. A second authority can disagree with the rows after a crash.

   Liability. The file command can stay. The new liability is the marker format and the batch publication: a crash leaves a prefix of batches published, which is safe only while each batch checks the live source hash. The liability removed is larger. Today a repair load reverts source text, restores deleted rows, and nearest ranks a hand vector as if it were the schema model. The error text in resolve_nearest_query_vec already says the stored vectors were embedded with the schema model. This run shows that sentence is not enforced per row.

   #952 (draft, conflicting with main) makes the schema the spec for the file command and compares the provider model with @embed before it reads input. It does not write the graph. Its description says a --graph flag now would collide with the reconciler RFC 0015 names. Land the reconciler's flag with that RFC, and keep #952 on the file command.

   Lance

   Pinned Lance is 11.0.0 (released 2026-08-30). Latest stable is v13.0.0 (2026-10-07). v12.0.0 is between them. v14 is beta only (v14.0.0-beta.19 on 2026-10-16).

   No Lance release adds an embedding provider, a model id, or a served job. The pages below were read as text from lance.org. They match the pinned Rust sources.

   Lance 11 already has the column update a reconciler needs:

   • merge_insert with a partial source keeps target columns that the source omits, on matched rows. Inserts of a partial source require those columns to be nullable, or when_not_matched = DoNothing (merge_insert.rs around the partial-schema upsert and the non-nullable insert check). WhenMatched::UpdateIf is in the same enum.
   • Fragment::update_columns patches named columns.
   • The data evolution guide documents add_columns for a new column, including a Python UDF that can compute an embedding column and a checkpoint file for that UDF. That path adds a column. This graph already has the vector column. The checkpoint file is not an OmniGraph commit.
   • The read and write guide documents merge_insert as a bulk update keyed by a column (when_matched_update_all).
   • Vector index training skips null vectors (test_create_index_nulls in the v11 IVF source). A null stays out of the index. Newly written vectors still need OmniGraph's explicit index pass for full index coverage. Search correctness for an uncovered tail is the existing invariant 7 contract, not a Lance fix for this issue.
   • Transactions are per dataset version (transaction format). OmniGraph graph tables publish detached commits and one __manifest pin. A Lance rebase of a linear overwrite would not become the graph commit.

   Later Lance work changes the safety of a partial-schema update. It does not close this issue.

   • lance#8351 (merged, in v12.0.0 and v13.0.0) filters stale vector-index rows before top-k. Compare status against those tags is behind.
   • lance#9523 (merged 2026-09-30, in v13.0.0, not in v11 or v12) builds the partial-schema fill as one projection.
   • lance#9742 and lance#9765 repair partial blob sources and join columns on horizontal updates. Both are in v14.0.0-beta.19. Neither is in the v13.0.0 tag (diverged against v13, behind against that beta).
   • lance#9950 and lance#10043 are merged after v14.0.0-beta.19 (ahead of that tag). They fix UpdateIf struct names on indexed joins, and they prune unread target columns.
   • Open lance#10077 and lance#8374 bound merge_insert memory. They are performance bounds. They are not a job runner.

   A column-only backfill written on Lance 11 should carry a lance_surface_guards case for partial-schema update of a fixed-size list, with DoNothing on an absent key. A later bump that includes #9742, #9765, #9950, and #10043 reduces the chance that the indexed __id path drops or nulls a column the batch did not send. The bump still leaves model identity, the predicate, and the served operation as OmniGraph work.

   Related work

   • #655 is the steady-state contract: a content write should expose pending coverage and must not publish a stale vector as ready. Its reproduction comment on this SHA is the load, mutate, and nearest evidence. This issue is the operator pass over rows that already exist. RFC 0015 is the shared design. Neither issue replaces the other.
   • #657 is the marker this issue's model rule needs. It is needs-rfc. A refresh mode that cannot read a marker can only rewrite every row.
   • #874 is role hints and batched omnigraph embed. It records that a failed file run loses its work and points at fill-missing as a resume idea. The temp-file result above is that hole. Batching does not add the served operation.
   • #952 is the draft file-command change described above.
   • #550 is index readiness. Embeddings and indexes share the words pending and stale. They are different derived state. A vector can be present and still unindexed.
   • #648 is graph-wide search. Its reproduction shows the same holes from the query side: nulls omitted with no count, and an explicit vector skipping the model check.

   Supersession: none. #655, #657, and RFC 0015 are adjacent and required. They do not provide the served historical operation, the preview, or the pinned write. This issue stays open.

   Limitations

   • The graphs are local files. This run did not use S3 or Azure.
   • The embedder is the mock. There was no HTTP 429, no provider timeout, and no real model. A mid-file failure was a non-string source, plus a missing API key. Both stop before the output rename.
   • The set has 5 rows and dimension 4. The distances and the 2.5e-8 export drift belong to this seed.
   • The server was --unauthenticated. /mcp was therefore absent. The tool list is from source.
   • schema plan for the model change was direct. It was not sent through the server.
   • I did not add a test to the tree, and I did not run the workspace suite. The proof is this CLI and HTTP run. The existing file tests embed_seed_fills_missing_and_preserves_existing_vectors_by_default and embed_select_reembeds_only_matching_rows already pin fill-missing and selector refresh. They do not cover export, load, a stale vector, or a served route.