Some pipelines already record useful per-file diagnostics without keeping the
native parser payload. Implement the direct-column result format so those
pipelines reach the same coverage and reporting model.
Acceptance criteria
required identity fields are validated at configuration load; an unknown stored
status stays unknown and is not returned; a negative or type-incompatible
numeric diagnostic counts as invalid; no executable mapping language is
introduced.
Detail
The direct-column format maps logical diagnostic fields onto stored columns:
status, text-character count, reported page count, pages represented, and
structural-element counts where present. Absent mappings produce unavailable
diagnostics, distinct from an observed zero.
The status vocabulary is fixed at success, failure, partial, and
unknown. A stored value outside it counts as unknown without being echoed,
because a status column holds customer strings and a diagnostic report is not a
place to surface them.
Numeric diagnostics are validated. A negative page count or a value of an
incompatible type counts as invalid and is reported as such rather than being
coerced into a plausible number.
Coverage, currency, sampling, and limitations behave exactly as they do for a
native format. The two input styles differ in where the evidence is read from,
not in what the report means.
Mappings remain column identifiers. Adding an expression language here would
reintroduce the execution surface that the aggregate-only path exists to close.
Constraints
Two constraints govern this work wherever it touches the engine, and an issue
that satisfies its own tests while breaking either is implemented wrong:
- Document content never leaves the warehouse. Every content computation
runs inside a warehouse expression. No document body, excerpt, heading,
extracted value, image, file name, path, signed URL, individual file
identifier, or raw provider error string may reach stdout, stderr, the cache,
or an audit record.
- Nothing in this backlog invokes document processing. Only engine-authored
diagnostic queries over materialized tables execute. New document-processing
charges require a provider-enforced hard monetary ceiling that Dex can verify,
and no estimate, page limit, delayed quota, or user confirmation substitutes
for one.
Dependencies
Depends on #451, #452, and #454.
Some pipelines already record useful per-file diagnostics without keeping the
native parser payload. Implement the direct-column result format so those
pipelines reach the same coverage and reporting model.
Acceptance criteria
required identity fields are validated at configuration load; an unknown stored
status stays unknown and is not returned; a negative or type-incompatible
numeric diagnostic counts as invalid; no executable mapping language is
introduced.
Detail
The direct-column format maps logical diagnostic fields onto stored columns:
status, text-character count, reported page count, pages represented, and
structural-element counts where present. Absent mappings produce unavailable
diagnostics, distinct from an observed zero.
The status vocabulary is fixed at
success,failure,partial, andunknown. A stored value outside it counts as unknown without being echoed,because a status column holds customer strings and a diagnostic report is not a
place to surface them.
Numeric diagnostics are validated. A negative page count or a value of an
incompatible type counts as invalid and is reported as such rather than being
coerced into a plausible number.
Coverage, currency, sampling, and limitations behave exactly as they do for a
native format. The two input styles differ in where the evidence is read from,
not in what the report means.
Mappings remain column identifiers. Adding an expression language here would
reintroduce the execution surface that the aggregate-only path exists to close.
Constraints
Two constraints govern this work wherever it touches the engine, and an issue
that satisfies its own tests while breaking either is implemented wrong:
runs inside a warehouse expression. No document body, excerpt, heading,
extracted value, image, file name, path, signed URL, individual file
identifier, or raw provider error string may reach stdout, stderr, the cache,
or an audit record.
diagnostic queries over materialized tables execute. New document-processing
charges require a provider-enforced hard monetary ceiling that Dex can verify,
and no estimate, page limit, delayed quota, or user confirmation substitutes
for one.
Dependencies
Depends on #451, #452, and #454.