Databricks users need file assessment without exporting content and without
launching fresh processing. Support warehouse-side volume metadata aggregation
where it can be verified, explicit manifest bindings where it cannot, and
materialized native or mapped results.
Acceptance criteria
direct volume inventory ships only if integration tests prove metadata-only
behavior, and otherwise reports unavailable; the manifest binding remains usable
either way; no image is persisted and no parsing is triggered; the shared tests
and the internal dogfood scenario pass.
Detail
Direct volume inventory ships only if integration tests establish that metadata
aggregation happens warehouse-side without downloading content and without
emitting per-file records to Dex. If that cannot be established, the capability
reports unavailable with the reason.
The explicit manifest binding is the supported alternative: a materialized file
manifest, bound like any other collection. There is no client-side listing
fallback, because a fallback that reads object storage from the Dex process is
the boundary this design exists to hold.
The native result adapter reads materialized ai_parse_document output and
declares its supported shape, degrading explicitly on an unknown one.
Reuse the shared source and result-format conformance contracts, and repeat the
dogfood scenario against a Databricks collection.
Constraints
Two constraints govern this work wherever it touches the engine, and an issue
that satisfies its own tests while breaking either is implemented wrong:
- Document content never leaves the warehouse. Every content computation
runs inside a warehouse expression. No document body, excerpt, heading,
extracted value, image, file name, path, signed URL, individual file
identifier, or raw provider error string may reach stdout, stderr, the cache,
or an audit record.
- Nothing in this backlog invokes document processing. Only engine-authored
diagnostic queries over materialized tables execute. New document-processing
charges require a provider-enforced hard monetary ceiling that Dex can verify,
and no estimate, page limit, delayed quota, or user confirmation substitutes
for one.
Dependencies
Depends on #457, #458, and the BigQuery dogfood findings from #459.
Databricks users need file assessment without exporting content and without
launching fresh processing. Support warehouse-side volume metadata aggregation
where it can be verified, explicit manifest bindings where it cannot, and
materialized native or mapped results.
Acceptance criteria
direct volume inventory ships only if integration tests prove metadata-only
behavior, and otherwise reports unavailable; the manifest binding remains usable
either way; no image is persisted and no parsing is triggered; the shared tests
and the internal dogfood scenario pass.
Detail
Direct volume inventory ships only if integration tests establish that metadata
aggregation happens warehouse-side without downloading content and without
emitting per-file records to Dex. If that cannot be established, the capability
reports unavailable with the reason.
The explicit manifest binding is the supported alternative: a materialized file
manifest, bound like any other collection. There is no client-side listing
fallback, because a fallback that reads object storage from the Dex process is
the boundary this design exists to hold.
The native result adapter reads materialized
ai_parse_documentoutput anddeclares its supported shape, degrading explicitly on an unknown one.
Reuse the shared source and result-format conformance contracts, and repeat the
dogfood scenario against a Databricks collection.
Constraints
Two constraints govern this work wherever it touches the engine, and an issue
that satisfies its own tests while breaking either is implemented wrong:
runs inside a warehouse expression. No document body, excerpt, heading,
extracted value, image, file name, path, signed URL, individual file
identifier, or raw provider error string may reach stdout, stderr, the cache,
or an audit record.
diagnostic queries over materialized tables execute. New document-processing
charges require a provider-enforced hard monetary ceiling that Dex can verify,
and no estimate, page limit, delayed quota, or user confirmation substitutes
for one.
Dependencies
Depends on #457, #458, and the BigQuery dogfood findings from #459.