Skip to content

Add Databricks volume or manifest collections and ai_parse_document result assessment #461

Description

@marcociav-exmergo

Databricks users need file assessment without exporting content and without
launching fresh processing. Support warehouse-side volume metadata aggregation
where it can be verified, explicit manifest bindings where it cannot, and
materialized native or mapped results.

Acceptance criteria

direct volume inventory ships only if integration tests prove metadata-only
behavior, and otherwise reports unavailable; the manifest binding remains usable
either way; no image is persisted and no parsing is triggered; the shared tests
and the internal dogfood scenario pass.

Detail

Direct volume inventory ships only if integration tests establish that metadata
aggregation happens warehouse-side without downloading content and without
emitting per-file records to Dex. If that cannot be established, the capability
reports unavailable with the reason.

The explicit manifest binding is the supported alternative: a materialized file
manifest, bound like any other collection. There is no client-side listing
fallback, because a fallback that reads object storage from the Dex process is
the boundary this design exists to hold.

The native result adapter reads materialized ai_parse_document output and
declares its supported shape, degrading explicitly on an unknown one.

Reuse the shared source and result-format conformance contracts, and repeat the
dogfood scenario against a Databricks collection.

Constraints

Two constraints govern this work wherever it touches the engine, and an issue
that satisfies its own tests while breaking either is implemented wrong:

  • Document content never leaves the warehouse. Every content computation
    runs inside a warehouse expression. No document body, excerpt, heading,
    extracted value, image, file name, path, signed URL, individual file
    identifier, or raw provider error string may reach stdout, stderr, the cache,
    or an audit record.
  • Nothing in this backlog invokes document processing. Only engine-authored
    diagnostic queries over materialized tables execute. New document-processing
    charges require a provider-enforced hard monetary ceiling that Dex can verify,
    and no estimate, page limit, delayed quota, or user confirmation substitutes
    for one.

Dependencies

Depends on #457, #458, and the BigQuery dogfood findings from #459.

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions