Add the file-exploration contract: optional interfaces, capability model, and aggregate result types - #465
Open
marcociav-exmergo wants to merge 4 commits into
Open
Add the file-exploration contract: optional interfaces, capability model, and aggregate result types#465marcociav-exmergo wants to merge 4 commits into
marcociav-exmergo wants to merge 4 commits into
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Closes #450.
Summary
Dex is being extended to document collections: BigQuery object tables,
Snowflake directory tables, and Databricks volumes or manifests, together with
the processing results some earlier pipeline already materialized about them.
This PR adds the contract every later piece of that work is built on (#451
through #463). It adds a new
exmergo_dex_core.filespackage with two modules:contract.pyholds:statuses, diagnostics, limitation codes, and unavailability reasons
results.pyholds the aggregate types that are the only thing a fileoperation will ever return.
It adds no command and changes no existing behavior. No connector implements a
file source yet, so every connector reports file exploration unavailable with a
named reason.
The issue's four acceptance criteria are each held by a test rather than by
convention:
Adapterprotocol gains no member.field admits
None.Decisions worth a reviewer's attention
Beside the adapter, not inside it, and tiered.
Adapterisruntime-checkable, so a new member there would demote every host-supplied adapter
that has not grown it. The file protocols live in their own module and follow the
tier idiom of
adapters/project.py: a capability is read offisinstance, neverdeclared by a flag. There are three protocols:
FileCollectionSource: metadata aggregation.DiscoveringFileSource: collection discovery. It sits beside the first tierrather than inside it, because a source reading an explicitly bound manifest
table can aggregate it without being able to list anything.
FileResultSource: result assessment.file_capabilities(adapter)derives the report and never calls a member. Thetest fakes raise if one is called.
Native processing has no field to set.
NativeProcessing.availableand.reasonare computed constants:availablekey.model_copy(update={"available": True})still reports false.Opening it would be a reviewed code change, never a configuration. The reason
code is
no_verified_hard_spend_cap."Structurally incapable" is a test.
tests/files/test_file_results.pywalks every field of every aggregate model andfails on any leaf that is not one of:
RelationNameA second test shows the walker catching
uri: str,note: str | None, a looseint, and adict. Every model also forbids unknown keys, so auricannot ridealong on an otherwise valid aggregate.
RelationNamerefuses anything a path orURL needs (
/,:,?,=,&,%, whitespace, quotes). It cannot tell a barescan.pdffrom a two-part relation, and the docstring says so: the realguarantee is that no field is meant to hold a file name.
Absent and zero never share a spelling.
Countis a strict non-negative int,or
Unavailable(reason). It serializes as0or as{"reason": "not_reported"},never
null. Rates areRatio(numerator, denominator), with a derivedfraction. A stored fraction is accepted on read-back only when it follows fromthe two counts, which keeps the JSON round-trip the cache will need (#458) without
opening a hole in
extra="forbid".The models refuse inconsistent reports:
file with no result cannot leave the denominator.
sampledmust equalmin(requested, eligible).measured over the matched files, which excludes ambiguous duplicates.
screened, that no processing was invoked, and that the sample is not
representative.
Distributions use fixed bins, not percentiles.
SUM(CASE ...)is the sameSQL on every dialect and percentile functions are not. Each diagnostic's first
bin is
[0, 1), which is exactly the observed zeros, so "zero pages" and "nopage count" stay apart here too.
Format buckets, and
otherversusunknown. The bucket is read off thereported content type alone, never the bytes or the file name:
Application/PDF; version=1.7ispdf.pdf/jpeg/png/tiff.other.unknown, and so is an explicitapplication/octet-stream.image/jpg: guessing at aliases is how two connectors come todisagree.
format_for_content_typeis the reference semantics that each connector'sbucketing SQL will be tested against.
What stays open for the next issues. Three things here are deliberately
provisional:
ResultBindingis the resolved engine-side form. The authored config section,its identifier-only validation, and source-scope checks land in Add explicit collection and materialized-result bindings to project configuration #451.
source_sample_sql,run_file_aggregate,quote_identifier) are unexported and have no implementer, so Aggregate-only execution and sanitized provider errors for file diagnostics #452 and Sample source files in the warehouse and match them against materialized results #454can refine them.
_EXPORTS. Exporting commits the publicAPI, so exports land with the commands in Expose
explore files inventoryandexplore files profilethrough the CLI and the engine #457.Dogfood
The contract was checked against a live BigQuery object table indexing 21
synthetic, non-sensitive files. Two synthetic results tables sit beside it:
ML.PROCESS_DOCUMENToutputBoth carry planted gaps: missing, failed, partial, stale, duplicate, and malformed
results. No document processing was run.
(21 files, 5,725,700 known bytes, 1 zero-byte file, and pdf 11 / jpeg 2 / png 3
/ tiff 2 / other 2 / unknown 1) build a valid
CollectionInventory, whichpasses
envelope.sanitize. Its payload contains no file path.bucket over all eight distinct stored content types. A content type stored as
Application/PDF; version=1.7bucketed aspdf, and a PNG uploaded under a.pdfname bucketed aspng.AwareDatetimerefused naive timestamps. That is the intended refusal, andthe BigQuery source will need to keep the client's timezone-aware values.
file_capabilitieson a liveBigQuery adapter and on the DuckDB demo warehouse reports
no_file_sourceforall three capabilities, with native processing false.
Files
packages/dex-core/src/exmergo_dex_core/files/__init__.py,contract.py,results.py: new.packages/dex-core/tests/files/test_file_contract.py,test_file_results.py:new. The first covers vocabulary, requests, compatibility across all seven
shipped connectors, tier derivation, and native processing. The second covers
the structural walker, absent versus zero, the consistency validators, JSON
round-trips, and the sanitizer.
packages/dex-core/tests/test_safety_spine.py: two Family 5 tests. The firstchecks that paths, signed URLs, extracted text, and provider errors are refused
by the string type and as extra keys. The second checks that native processing
is unavailable on every connector.
references/file-exploration.md: new. It is the implementer-facing contractwith a plain status line, and is deliberately not linked from
AGENTS.mdorthe README until a command exists.
CHANGELOG.md: an[Unreleased]/Addedentry.Deliberately not changed:
Adapterprotocol and every adapterconnect testoutput (thefilesentry is Add explicit collection and materialized-result bindings to project configuration #451)DexConfig, the cache and storage,_EXPORTS,AGENTS.md, andREADME.md