embeddings: managed population with observable pending, ready and stale coverage

Problem

@embed describes the source relationship, but loading a row without its vector does not populate it automatically. On 0.10.0 a nullable annotated vector remained null; a required vector caused load rejection. The offline omnigraph embed transform is available, but ingestion through the served graph has no managed population/coverage lifecycle.

Applications need to distinguish a content write that has committed from derived embeddings that are still pending, stale or failed.

Proposed change

Add managed embedding reconciliation for committed source content with observable state and coverage. Provider calls need not block the content-write transaction. Define the contract for nullable versus required derived vectors, source changes, model changes, failure/retry and how queries behave while coverage is incomplete.

Bind generated vectors to the exact source revision and model contract. A delayed job must not overwrite a newer source or attach a stale vector to the current row. Define branch/merge behavior and apply derived writes through normal graph authority.

Acceptance criteria

Load/mutate of eligible content reliably schedules or exposes pending derivation; no silent assumption that @embed means the vector is ready.
Status reports pending/ready/stale/failed counts and actionable failures without exposing provider secrets.
Duplicate work and provider retries converge; concurrent content edits, deletion, branch changes and restart cannot publish stale vectors as ready.
Search communicates its coverage contract, including missing embeddings.

Current code: embedding contract, offline embedding command. Query-time text embedding already exists; this request is about stored vector population.

Summary

@embed names the source text and the model for a vector. A load or a mutation does not create that vector. A nullable vector stays null. A required vector rejects the write. When the source text changes, the old vector stays, and nothing marks it stale. nearest returns only rows that have a vector. The result does not say that other rows were omitted. There is no pending, ready, stale, or failed count for embedding coverage.

Eligibility

This issue is still open. It has no comments and no linked fix pull request. #550 and #648 mention it. Those issues cover index readiness and graph-wide search. They do not add this contract.

Verdict

Reproduced. The gap on 0.10.0 is still the gap on current main.

SHA: be031fea766fc8b280137f9ad9f34d4d2b3087bc (feat(query): the ranked binding roots its component (#1069), 2026-10-18). git fetch origin main returned that SHA. The checkout was that commit.

Environment

omnigraph version reports 0.14.0, storage format 16 (serves v14 to v16), and HTTP contract 0.13. The binaries were target/debug/omnigraph and target/debug/omnigraph-server from this checkout. No .rs file under crates/ is newer than the CLI binary. rustc is 1.97.1. Cargo.lock pins Lance 11.0.0.

The provider was mock only. The process had OMNIGRAPH_EMBEDDINGS_MOCK=1, OMNIGRAPH_EMBED_PROVIDER=mock, and OMNIGRAPH_EMBED_MODEL=repro-mock. OPENROUTER_API_KEY, OPENAI_API_KEY, and GEMINI_API_KEY were unset. The served graph used providers.embedding.local.kind: mock with the same model. No provider secret was set.

Schema:

node Doc {
    slug: String @key
    body: String
    embedding: Vector(4)? @embed("body", model="repro-mock") @index
}
node RequiredDoc {
    slug: String @key
    body: String
    embedding: Vector(4) @embed("body", model="repro-mock")
}

The direct graph was a local init at /tmp/repro-655/graph.omni. The served graph was a local cluster at /tmp/repro-655-served, graph id docs, started with omnigraph-server --cluster /tmp/repro-655-served --bind 127.0.0.1:18080 --unauthenticated. HTTP calls sent omnigraph-http-api: 0.13.

Nullable vector, vector absent

Command:

omnigraph load --data nullable.jsonl --mode merge --json /tmp/repro-655/graph.omni

The file had two Doc rows. omitted had body set to missing vector and no embedding. supplied had embedding: [1.0, 0.0, 0.0, 0.0].

The load published one commit. total_entities was 2. embedding_generation was "unsupported". A read of both rows returned row_count 2. The omitted object had no embedding key. The supplied vector was still [1.0, 0.0, 0.0, 0.0].

POST /graphs/docs/load returned HTTP 200 with the same embedding_generation value and the same two-row read.

On a second graph that held only the null row, the human load line was:

Loads do not generate embeddings. Supplied vectors are preserved; omitted vectors follow schema nullability. Supply vectors in the input or prepare them with omnigraph embed.

That line is a capability note for the load. It is not a count of pending rows. It is printed even when every vector in the batch was supplied, which is what LoadEmbeddingGeneration::Unsupported means.

Required vector, vector absent

The same CLI load of one RequiredDoc row with body and no embedding exited 1:

{"error":"non-nullable vector property 'embedding' has null values","code":"bad_request"}

The next successful commit names the nullable load as its parent. The rejected load did not publish.

POST /graphs/docs/load returned HTTP 400 with that same error text.

A GQ insert that sets slug and body and omits embedding gets past the type checker. Execution rejects it. CLI exit 1, and POST /graphs/docs/mutate returned HTTP 400:

{"error":"missing required property 'embedding'","code":"bad_request"}

The type-check text says the insert must give the property or the @embed source. Execution does not embed. The source is not a substitute for the vector.

Source text change

update Doc set { body: $b } where slug = $s

supplied.body became source changed after the vector. The stored embedding stayed [1.0, 0.0, 0.0, 0.0]. The same update on RequiredDoc kept [0.0, 1.0, 0.0, 0.0] after the body change. The mutation JSON has affected_nodes, affected_edges, and commit. It has no stale mark and no embedding count. The served POST /graphs/docs/mutate did the same thing.

Search

Two Doc rows were committed. This query asks for up to 10 hits:

order { nearest($d.embedding, $q) } limit 10

$q was [1.0, 0.0, 0.0, 0.0]. row_count was 1. The only row was supplied. omitted was absent. The JSON keys were query_name, target, row_count, columns, rows, and graph_commit_id. There was no warning and no coverage field.

A text nearest for missing vector used the mock provider. That string is the body of the null row. The only hit was still supplied. That row has a hand-written vector, and its body is now different text. The row whose text equals the query was absent. The served query returned the same single hit.

explain for the vector query reported scan_row_estimate(node:Doc) as 2. It did not report how many of those rows have a null vector.

omnigraph optimize --json on the mixed graph returned pending_indexes: [] for node:Doc and for node:RequiredDoc. nearest after that optimize still returned 1 of 2 rows.

A second graph had one nullable row and no vector. A normal read returned row_count 1 and no embedding key. nearest returned row_count 0. optimize --json returned:

"pending_indexes": [
  {"property": "embedding", "reason": "property has no non-null vectors to train on yet"}
]

That reason means the vector index cannot train. It is not an embedding-coverage count. It appears only when the column has no non-null vector. A mix of null and non-null vectors does not produce it.

Offline command

omnigraph embed --input raw.jsonl --output embedded.jsonl --spec spec.json --json

With the mock provider this wrote one vector into the output file (embedded_records 1, model repro-mock). The graph row omitted still had no vector. The command writes a file. It does not publish the graph.

Difference from the 0.10.0 report

The reported behavior is unchanged. A nullable annotated vector stays null. A required annotated vector rejects the load. The file command exists. A served load does not fill vectors.

Three details are newer than that report:

A load that touches an @embed node type sets embedding_generation to "unsupported", including when every vector was supplied. A load that touches no such type sets it to null. This flag is not pending, ready, stale, or failed coverage.
The type checker accepts an insert that omits a required vector when the @embed source is assigned. Execution then rejects the missing vector. The 0.10.0 report described load rejection. It did not describe this split.
optimize --json can mark one index pending when a vector column has no non-null value. That signal is absent when some rows already have vectors, and those other rows are still missing from nearest.

Root cause

@embed is catalog metadata. The load writer and the mutation writer do not call the embedding client.

LoadEmbeddingGeneration::for_load returns Unsupported when the loaded node type has any embed_sources entry. It does not read row values.
The loader stores null for a nullable vector and rejects null for a required vector (loader/mod.rs).
Insert execution writes null into a nullable column. For a required column it returns missing required property. It does not read embed_sources (exec/mutation.rs).
The type checker does read embed_sources. It treats the source as enough for a required vector (typecheck.rs).
An update that assigns body publishes the new text and keeps the previous vector. Nothing clears that vector, and nothing records the source revision it came from.
nearest uses Lance distances. Lance 11 skips a null distance in the flat scan (lance-11.0.0 src/dataset/mem_wal/memtable/scanner/exec/brute_force_vector.rs, the distances.is_null check). Its index test test_create_index_nulls expects the hit count to equal the non-null count (src/index/vector/ivf.rs). OmniGraph does not add a count of the skipped rows.
The maintenance pending signal is PendingIndex when a vector index has nothing to train on (table_ops.rs).
omnigraph embed reads and writes JSONL (embed.rs).

Invariant 7 says a missing index may change cost and must not change the logical answer. A vector cell is not an index. It is a stored value. @embed describes it, and no writer maintains it. Search then treats a missing value as "not a hit" and does not report that fact.

Recommended fix

Write the contract in an RFC before code. This issue is labeled needs-rfc. RFC 0015 is the draft for this work. The RFC index marks it draft and not-started. Extend that RFC. Do not start with a queue. Do not call the provider inside load or mutate.

The rules this gap breaks, and the class each rule covers:

Invariant 7. Derived physical state may lag. It must not change the logical answer in silence. The class is every derived value, not only indexes. A stored embedding can disagree with its source text, and search can omit null rows with no report.
Invariant 9. A required value fails in the open. The engine does not invent a placeholder. The class is every non-null property. A required @embed vector must not become a silent null.
Invariants 2 and 4. One graph publication is the door for a load, a mutation, a merge, and a maintenance batch. A derived vector becomes visible only through that door.
Invariant 11. A provider call has a bound, a visible result, and a bounded retry. A failed embed must not look like success. The error must not contain a secret.
The deny-list rejects a job queue for state that a scan of committed rows can rebuild. It also rejects a synchronous vector rebuild on the content write. The class is every reconciler of derived state.

Put this contract in the RFC:

A content publication does not call the provider.
For a nullable @embed vector, a row with source text and a null vector is pending. A change to that source text, in the same publication, clears the vector. Search then stops using the old vector.
A reconciler reads pending rows, calls the provider in batches, and publishes vectors through the normal graph write. The publication is valid only when the committed source text and the recorded model are still the inputs that produced the vector. A late job that lost the race writes nothing. A restart reads the same committed rows. It does not trust a process-local queue.
For a required @embed vector, keep today's rejection until the RFC picks one rule. Either the content write still fails when the vector is absent, or the row stays unpublished until one later publication contains the source and a matching vector. Do not store null in the non-null column. Do not call the provider inside the content transaction. The type checker and the executor must use that same rule. Today they do not.
Counts come from committed rows. Ready means the stored vector names the current source revision and the current model. Pending means null, with source text, and no failed attempt. Failed means a committed attempt error, with the secret removed. Do not keep a second searchable vector and call it stale. Under the clear-on-change rule, an old vector is not a hit. The rows that need a new vector are the pending count. Do not report an old vector as ready.
nearest keeps the Lance rule that a null vector is not a hit. The query result states that contract. Use the retrieval descriptor already sketched as embedding_coverage in RFC 0047 and RFC 0048. That field is not in the query JSON from this run. Do not add a second coverage object.
A merge publishes a vector only when that vector is bound to the merged source text. If the merged text is not that source, the result vector is null and pending. Do not choose a vector by time.

This choice delays nearest freshness after a content write. That delay is the right trade. A provider call on the write path makes every content commit depend on network time and on a third party. The write stays local, and a provider failure does not fail the content commit. The cost is that a new or changed row is not a nearest hit until the reconciler publishes. The count, and an explicit wait on the reconciler, make that delay visible. RFC 0015 already records this choice. The addition here is the conditional publication. A null check alone does not stop a late job from storing embed(old text) on a newer source.

Other shapes, and why not

Call the provider inside load and mutate. The content commit then waits on the provider, and a provider failure fails the content write. The deny-list rejects this shape.
Treat "vector is null" as the only pending test, and write the new vector by row id with no source check. A job can then store embed(old text) after the text has changed. That publishes a stale vector as ready.
Keep the old vector and add a stale mark. Search must apply that mark, or it returns the wrong vector. The clear-on-change rule removes that window. It does not need a second searchable value.
Build the served job runner in embeddings: served backfill and re-embedding jobs with progress and resume #656 first. A queue would be a second authority for state the committed rows already show. Build the reconciler here. Let embeddings: served backfill and re-embedding jobs with progress and resume #656 report progress for that reconciler.

Liability

This removes a silent wrong vector and a silent missing hit. It adds provider calls, retries, and a second publication. Those calls need a deadline, a retry bound, and errors with no secret. The write-back must match the current source and model. A merge must not attach one side's vector to the other side's text. Those are the new obligations. They are smaller than a queue, a background scheduler, or a provider call on every content write.

Lance

Pinned Lance is 11.0.0. Later stable releases are v12.0.0 (2026-09-17) and v13.0.0 (2026-10-07). The newest prerelease seen here is v14.0.0-beta.19 (2026-10-16). None of these releases add managed embedding population or a pending, ready, stale, or failed row report.

Lance 11 already has the physical tools a reconciler would call:

A null vector is legal. Flat search skips a null distance. Index training drops nulls. test_create_index_nulls expects no null vectors in the hits.
The data evolution guide shows a new embedding column left null, then filled by a caller function. The read and write guide shows update and merge_insert for matched rows. The transaction spec defines Update with REWRITE_ROWS and REWRITE_COLUMNS, and it defines fragment conflicts. OmniGraph does not move Lance linear HEAD for a graph table. The graph manifest publication stays the door. A Lance background job would not become that door.
The vector index spec describes IVF layout. It does not define a count of rows that have no vector. The vector search quick start also does not define that count.

Later Lance changes that touch this area, and do not close the issue:

lance#9134, in v13. A vector index may exist when the table is too small to train. It covers no rows. A query skips that empty index. A null vector still gives the trainer nothing. A future OmniGraph bump could change the "no non-null vectors" pending_indexes path. It would not fill embeddings.
lance#8351, in v12. Vector search drops stale index-segment rows before top-k. On Lance 11, OmniGraph rebuilds a lagging vector index as a whole and does not add a delta segment (docs/dev/lance.md, index segments). A reconciler that changes vectors still needs that rebuild. Otherwise an index can rank an old vector until optimize. This fix is about index entries, not about the value stored on the row.
lance#9209, in v13. A bad vector update literal is rejected instead of stored as null. That is useful for a later write-back. It does not create vectors.

A search of open Lance issues and pull requests for this coverage lifecycle returned no match.

Related work

RFC 0015. Draft and not started. Extend it. Its null-on-change rule is safe only together with a conditional publication.
#656. Served backfill, progress, and resume. It should use this reconciler. It does not replace the contract.
#657. Model and source identity, beyond dimension. The write-back here must record the model and the source revision. Equal dimension is not that proof.
#874. Role hints for an OpenAI-compatible provider, and batching for the file command. Separate from coverage. A batch embed method is still useful for the reconciler.
#952. Draft. omnigraph embed would read @embed from a schema file. It still writes JSONL, not the graph. It does not add coverage.
#550 and #486. Index readiness, and a vector index that reports full coverage with a bad partition layout. Different from row embedding coverage. pending_indexes is the index signal.
#648. Graph-wide discovery. It asks search to show missing embeddings. That count belongs on the retrieval descriptor.
RFC 0047 and draft PR #793 (RFC 0048). They sketch embedding_coverage on a retrieval descriptor. The query JSON from this run has no retrievals field.

Supersession

No open issue replaces this one. #656 is the job surface. #657 is the identity check the publication needs. #952 changes the file command only. This issue stays open.

Limitations

This run did not merge branches, delete a row during a delayed job, or restart a process during publication. The stale vector above is one committed update on main. The conditional write-back and the merge rule are the design that follows from that result. They were not executed.
This run did not execute loads_report_unsupported_embedding_generation_without_changing_vectors. That server test already encodes the HTTP load matrix, including an Arrow null count of 1 for this load shape. This run used a live local server instead. A JSON read omits a null cell, so a missing embedding key is the visible sign. The Lance files were not opened to count nulls.
The mock provider did not leave this machine. A real provider error string was not exercised.
explain was not run again after optimize. This note does not claim a vector-index fragment bitmap.

Navigation Menu

Repository navigation

embeddings: managed population with observable pending, ready and stale coverage #655

Problem

Proposed change

Acceptance criteria

Summary

Eligibility

Verdict

Environment

Nullable vector, vector absent

Required vector, vector absent

Source text change

Search

Offline command

Difference from the 0.10.0 report

Root cause

Recommended fix

Other shapes, and why not

Liability

Lance

Related work

Supersession

Limitations

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

embeddings: managed population with observable pending, ready and stale coverage #655

Description

Problem

Proposed change

Acceptance criteria

Activity

ragnorc commented on Oct 18, 2026

Summary

Eligibility

Verdict

Environment

Nullable vector, vector absent

Required vector, vector absent

Source text change

Search

Offline command

Difference from the 0.10.0 report

Root cause

Recommended fix

Other shapes, and why not

Liability

Lance

Related work

Supersession

Limitations

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions