Skip to content

Add the file-exploration contract: optional interfaces, capability model, and aggregate result types - #465

Open
marcociav-exmergo wants to merge 4 commits into
mainfrom
issue-450
Open

Add the file-exploration contract: optional interfaces, capability model, and aggregate result types#465
marcociav-exmergo wants to merge 4 commits into
mainfrom
issue-450

Conversation

@marcociav-exmergo

Copy link
Copy Markdown
Member

Closes #450.

Summary

Dex is being extended to document collections: BigQuery object tables,
Snowflake directory tables, and Databricks volumes or manifests, together with
the processing results some earlier pipeline already materialized about them.
This PR adds the contract every later piece of that work is built on (#451
through #463). It adds a new exmergo_dex_core.files package with two modules:

  • contract.py holds:
    • the fixed vocabularies: document families, format buckets, processing
      statuses, diagnostics, limitation codes, and unavailability reasons
    • the optional connector protocols
    • the request types
    • the capability model
  • results.py holds the aggregate types that are the only thing a file
    operation will ever return.

It adds no command and changes no existing behavior. No connector implements a
file source yet, so every connector reports file exploration unavailable with a
named reason.

The issue's four acceptance criteria are each held by a test rather than by
convention:

  • Existing connectors stay compatible. They implement nothing, and the
    Adapter protocol gains no member.
  • Unavailable and observed zero are distinct in every aggregate field. No
    field admits None.
  • Native processing is reported unavailable with a specific reason.
  • The result types are structurally incapable of carrying document content.

Decisions worth a reviewer's attention

Beside the adapter, not inside it, and tiered. Adapter is
runtime-checkable, so a new member there would demote every host-supplied adapter
that has not grown it. The file protocols live in their own module and follow the
tier idiom of adapters/project.py: a capability is read off isinstance, never
declared by a flag. There are three protocols:

  • FileCollectionSource: metadata aggregation.
  • DiscoveringFileSource: collection discovery. It sits beside the first tier
    rather than inside it, because a source reading an explicitly bound manifest
    table can aggregate it without being able to list anything.
  • FileResultSource: result assessment.

file_capabilities(adapter) derives the report and never calls a member. The
test fakes raise if one is called.

Native processing has no field to set. NativeProcessing.available and
.reason are computed constants:

  • Construction and deserialization refuse an available key.
  • model_copy(update={"available": True}) still reports false.
  • A confirmation flag has nothing to reach.

Opening it would be a reviewed code change, never a configuration. The reason
code is no_verified_hard_spend_cap.

"Structurally incapable" is a test.
tests/files/test_file_results.py walks every field of every aggregate model and
fails on any leaf that is not one of:

  • a strict non-negative int
  • a fixed enum
  • a timezone-aware datetime
  • another aggregate
  • RelationName

A second test shows the walker catching uri: str, note: str | None, a loose
int, and a dict. Every model also forbids unknown keys, so a uri cannot ride
along on an otherwise valid aggregate. RelationName refuses anything a path or
URL needs (/, :, ?, =, &, %, whitespace, quotes). It cannot tell a bare
scan.pdf from a two-part relation, and the docstring says so: the real
guarantee is that no field is meant to hold a file name.

Absent and zero never share a spelling. Count is a strict non-negative int,
or Unavailable(reason). It serializes as 0 or as {"reason": "not_reported"},
never null. Rates are Ratio(numerator, denominator), with a derived
fraction. A stored fraction is accepted on read-back only when it follows from
the two counts, which keeps the JSON round-trip the cache will need (#458) without
opening a hole in extra="forbid".

The models refuse inconsistent reports:

  • Coverage must partition the sample into matched, missing, and ambiguous, so a
    file with no result cannot leave the denominator.
  • sampled must equal min(requested, eligible).
  • Currency, statuses, payload validity, and every content distribution are
    measured over the matched files, which excludes ambiguous duplicates.
  • Every profile must state three limitations: that document PII was not
    screened, that no processing was invoked, and that the sample is not
    representative.
  • There is no readiness score.

Distributions use fixed bins, not percentiles. SUM(CASE ...) is the same
SQL on every dialect and percentile functions are not. Each diagnostic's first
bin is [0, 1), which is exactly the observed zeros, so "zero pages" and "no
page count" stay apart here too.

Format buckets, and other versus unknown. The bucket is read off the
reported content type alone, never the bytes or the file name:

  • The essence is lower-cased with parameters stripped, so
    Application/PDF; version=1.7 is pdf.
  • IANA types map to pdf/jpeg/png/tiff.
  • A well-formed type outside those is other.
  • A missing or malformed type is unknown, and so is an explicit
    application/octet-stream.
  • No aliases like image/jpg: guessing at aliases is how two connectors come to
    disagree.

format_for_content_type is the reference semantics that each connector's
bucketing SQL will be tested against.

What stays open for the next issues. Three things here are deliberately
provisional:

Dogfood

The contract was checked against a live BigQuery object table indexing 21
synthetic, non-sensitive files. Two synthetic results tables sit beside it:

  • one shaped like materialized ML.PROCESS_DOCUMENT output
  • one with mapped diagnostic columns

Both carry planted gaps: missing, failed, partial, stale, duplicate, and malformed
results. No document processing was run.

  • The inventory model holds real values. The warehouse-side aggregates
    (21 files, 5,725,700 known bytes, 1 zero-byte file, and pdf 11 / jpeg 2 / png 3
    / tiff 2 / other 2 / unknown 1) build a valid CollectionInventory, which
    passes envelope.sanitize. Its payload contains no file path.
  • The Python reference buckets match the warehouse. They agreed bucket for
    bucket over all eight distinct stored content types. A content type stored as
    Application/PDF; version=1.7 bucketed as pdf, and a PNG uploaded under a
    .pdf name bucketed as png.
  • AwareDatetime refused naive timestamps. That is the intended refusal, and
    the BigQuery source will need to keep the client's timezone-aware values.
  • Live connections report the right limitation. file_capabilities on a live
    BigQuery adapter and on the DuckDB demo warehouse reports no_file_source for
    all three capabilities, with native processing false.

Files

  • packages/dex-core/src/exmergo_dex_core/files/__init__.py, contract.py,
    results.py: new.
  • packages/dex-core/tests/files/test_file_contract.py, test_file_results.py:
    new. The first covers vocabulary, requests, compatibility across all seven
    shipped connectors, tier derivation, and native processing. The second covers
    the structural walker, absent versus zero, the consistency validators, JSON
    round-trips, and the sanitizer.
  • packages/dex-core/tests/test_safety_spine.py: two Family 5 tests. The first
    checks that paths, signed URLs, extracted text, and provider errors are refused
    by the string type and as extra keys. The second checks that native processing
    is unavailable on every connector.
  • references/file-exploration.md: new. It is the implementer-facing contract
    with a plain status line, and is deliberately not linked from AGENTS.md or
    the README until a command exists.
  • CHANGELOG.md: an [Unreleased] / Added entry.

Deliberately not changed:

@marcociav-exmergo marcociav-exmergo added the downstream-visible A consumer who stores or compares this command's output would observe a difference label Sep 10, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

downstream-visible A consumer who stores or compares this command's output would observe a difference

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Add the optional file-exploration interfaces, capability model, and typed aggregate results

1 participant