Skip to content

feat(console): show rejected and unaccounted records on the run page - #632

Open
tnunamak wants to merge 3 commits into
mainfrom
feat/console-run-rejected-records
Open

tnunamak wants to merge 3 commits into
mainfrom
feat/console-run-rejected-records

Conversation

@tnunamak

@tnunamak tnunamak commented Nov 5, 2026 •

Copy link
Copy Markdown
Contributor

What

The run detail page now shows records a run could not store or account for:

  • the run's rejected-record count and the rejected-record receipts for the run (stream, reason code, first input index, payload size);
  • the unaccounted count per STREAM_EVIDENCE stream.

Why

A run can succeed while the store permanently rejected some records (per-record isolation keeps the cursor moving), or while a stream left keys unaccounted (spec behavior). Both are recorded server-side, but the run page rendered neither, so such a run looked clean.

This PR keeps both behaviors and makes them visible. It does not hold the cursor and adds no table.

How

Data already on main:

  • run.batch_ingested carries records_permanently_rejected, and the terminal event carries the run total.
  • run.stream_evidence_declared carries outcomes.unaccounted.
  • GET /_ref/connections/:id/record-rejections lists receipts, oldest first, with no run filter.

One timeline page holds only part of a long run, and the terminal event is emitted last. A count summed from one page can therefore miss the batch and terminal events, and a page without them is not evidence of zero. So the run total comes from the run status, the same window-independent source as known_gaps:

  • Server (one small addition): GET /_ref/runs/:runId passes the terminal event's records_permanently_rejected through, only when it is a non-negative integer. It is absent while the run is active and for runs that predate the field.
  • Console: summarizeRunRejections returns one of four states and never reports a zero it cannot prove. A terminal event of any kind, including run.abandoned, can supply the total.
    • final: the run status total, or the terminal event's total. A reported zero renders no section.
    • so_far: the run is still active and the whole timeline is loaded, so the batch sum is the total to now. Liveness is the page's own (run status, else the timeline envelope's terminal status); the summary keeps no list of terminal events for it. A finished run with no total, such as an abandoned run, is never so_far.
    • at_least: part of the run is not on this page; the batch sum is a lower bound.
    • unknown: nothing readable. The section says the count is not available here.
  • Unaccounted streams come from evidence events, which have no window-independent source. When the page is not the whole timeline, the section says streams on other pages are not listed. Several evidence events for one stream are merged.
  • Receipts: read when the count is positive or unknown (an unreadable count is when they are the only evidence), and the connection is known; a known zero reads none. The pager follows the cursor up to 10 pages. If the cap stops it with pages left, the section says the list is partial. If receipts cannot be read (fetch failure, or no connection id), it says the list is unavailable.
  • The earlier "The other records were stored." is removed: retryable and unaccounted records are not stored either.

Mixed versions: a new console with an older server has no status total, so it shows so_far, at_least or unknown from the events it has, never a silent zero. An older console with a newer server ignores the extra field.

Tests

  • run-rejections.test.ts (18): each state, a reported zero, the terminal total beating the batch sum, an abandoned run with and without a total, a finished run whose terminal event is not on the page (never so_far), events without the field, evidence merged by stream, receipts narrowed to the run with the cursor kept, and the page's receipt read through loadRunRejectionView (old-server status with a partial timeline reads the receipts; a known zero, or no connection, reads none; a failed read gives no list).
  • rejected-records-section.test.ts (11): count and receipts, unaccounted count, no "other records" claim, lower bound, in progress, unavailable count, an unknown count that still lists receipts, other-page streams not listed, receipts unavailable, receipts partial, nothing rendered for a known zero.
  • run-status-body-builders-pure.test.ts and ref-run-status-route.test.ts: the total passes through from the terminal event (also a reported zero), and is omitted when absent or invalid. The route test uses the file-backed database; PostgreSQL was not run, since the new field reads terminal.data, the object known_gaps already comes from.

The new console tests fail on main, where the modules do not exist.

cd apps/console
npm run test:files -- "src/app/(console)/lib/run-rejections.test.ts" "src/app/(console)/syncs/[[]runId]/rejected-records-section.test.ts"   # 29 pass, 0 fail
npx tsc --noEmit                                                                                                                              # no errors in the touched files
cd reference-implementation
node --import tsx --test test/run-status-body-builders-pure.test.ts test/ref-run-status-route.test.ts                                           # 22 pass, 0 fail
npx tsc --noEmit                                                                                                                                # exit 0

ultracite check reports no lint-rule diagnostics for the touched files. It reports formatting errors (it expects tab indentation, while existing files in these directories, including unmodified ones, use spaces) and unused-suppression warnings, all present on main.

Limits

  • Receipts carry stream, reason, index and size, not record keys. A receipt stands for a distinct rejected payload, so the receipt count can differ from the rejected-record count.
  • The receipt pager stops after 10 pages and says so.
  • Other counts on this page are also summed from the loaded timeline page and are not changed here: checkpoint staged/advanced/commit-failed, skipped streams, interactions, {events.length} events, the terminal known gaps, and the failure and stderr sections.

Assisted-by: AI

A run can finish while some records were permanently rejected by the
store, or while a STREAM_EVIDENCE stream left keys unaccounted. The
server already records both: per-batch and run-total
records_permanently_rejected, outcomes.unaccounted on
run.stream_evidence_declared, and per-connection record-rejection
receipts. The run page showed none of them, so a run could read as
clean while records were missing.

The page now has a section that lists the rejected-record receipts for
the run (stream, reason, first input index, size) and the unaccounted
count per stream. No server change: the receipts route and the events
exist on main, so a new console works with an older server (the receipt
read fails soft and the count from run events remains) and an older
console is unaffected.

Assisted-by: AI
Signed-off-by: Tim Nunamaker <tnunamak@gmail.com>
The run page summed rejected records from one timeline page, and rendered
nothing when the sum was 0. For a long run, or for events that predate the
field, that looked the same as "nothing was rejected".

- Run status now carries the terminal event's records_permanently_rejected
  (window-independent, like known_gaps). The console reads it first.
- The summary states a count only when it can prove it: final, so far, at
  least, or unavailable. An unreadable count renders as unavailable.
- Unaccounted streams are flagged as not listed when the page is not the
  whole timeline, and a stream that declares evidence twice is merged.
- The receipt list reports a cut-short read and an unreadable list.
- Drop the claim that the other records were stored: retryable and
  unaccounted records are not.

Assisted-by: AI
Signed-off-by: Tim Nunamaker <tnunamak@gmail.com>
…nknown counts

The rejected-record summary kept its own list of terminal events and
left out run.abandoned. An abandoned run with no total on its event, or
a finished run whose terminal event was not on the loaded page, could be
reported as "so far", which says the run is still going. The summary now
takes the page's authoritative liveness as an input, reports "so far"
only for an active run with every event loaded, and reads the total from
run.abandoned like the other terminal events.

The page read the run's receipts only when the count was above zero, and
an unknown count counted as zero. An unknown count is when the receipts
are the only readable evidence, so they are now read when the count is
unknown or positive and the connection is known. A known zero still
reads none. The data flow moves into loadRunRejectionView so a test can
drive it without rendering the page.

Assisted-by: AI
Signed-off-by: Tim Nunamaker <tnunamak@gmail.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant