Skip to content

Track provider eligibility for native document processing #464

Description

@marcociav-exmergo

The original use case, inspecting documents that nothing has processed yet,
stays blocked because no researched provider path documents a hard monetary
ceiling that Dex can verify. Keep the dated research matrix and the eligibility
checklist, and reassess when a provider documents a qualifying control.

Acceptance criteria

a path is admitted only with evidence for service coverage, enforcement timing,
concurrency, in-flight work, and verifiable configuration; otherwise this stays
blocked and the native-processing capability stays unavailable with its reason.

Detail

Research checked on September 9, 2026, across three paths. BigQuery to Document
AI: Document AI charges are separate from query processing, and Google Cloud
spend caps currently list other eligible services, omit Document AI, and permit
billable overages plus completion of in-flight requests. Snowflake Cortex
document parsing: per-user AI quotas can block new requests, but evaluation
occurs within minutes of spending. Databricks document parsing: Databricks warns
that Unity Gateway budgets are not absolute caps on final billed amounts, and
the documented budget coverage does not establish equivalent protection for
document parsing.

These findings do not establish that no customer arrangement offers protection.
They establish that Dex cannot presently claim the guarantee for these
documented integration paths.

A future integration must establish all of the following before admission: the
control covers the exact service, operation, principal, and billing scope; its
enforcement prevents monetary overrun including concurrent and in-flight work;
Dex can verify the active control and its applicability; missing permissions or
unreadable control state cause refusal; document content stays in the approved
provider environment; and cancellation, retries, and billing settlement have
documented semantics.

Page selection exists in all three provider interfaces and may be useful later
for bounded inspection. It is not a monetary cap and does not satisfy the
checklist on its own.

No backend is eligible today. This issue adds no bypass, no inference budget
flag, and no configurable override.

Constraints

Two constraints govern this work wherever it touches the engine, and an issue
that satisfies its own tests while breaking either is implemented wrong:

  • Document content never leaves the warehouse. Every content computation
    runs inside a warehouse expression. No document body, excerpt, heading,
    extracted value, image, file name, path, signed URL, individual file
    identifier, or raw provider error string may reach stdout, stderr, the cache,
    or an audit record.
  • Nothing in this backlog invokes document processing. Only engine-authored
    diagnostic queries over materialized tables execute. New document-processing
    charges require a provider-enforced hard monetary ceiling that Dex can verify,
    and no estimate, page limit, delayed quota, or user confirmation substitutes
    for one.

Dependencies

Blocked on a provider capability change. Depends on nothing in this backlog.

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions