Repository navigation
#278: the §5.1 dtype ruling, and the register from the assimilation pass - #279
Merged
Merged
Conversation
Governance truth pass to main — no package change, no new tag
Release 1.1.1 — two delivery-path fixes from views-crafdapi, and the governance pass behind them
Five entries, none of which duplicated an existing one — checked against all 22 open and 80 resolved concerns before writing: - C-103 (Tier 2) the observed-range clip depends on `datafactory_query`, which is declared nowhere and is not installed; its absence is swallowed by a bare `except Exception`, so a missing dependency and "the producer publishes no boundary" leave through the same branch and ship fabricated months either way. Carries the verification question it cannot answer from this seat. - C-104 (Tier 3) a stale virtualenv takes 25 tests red, 20 of them the only tests that import either manager. Measured: 433 passed / 25 failed against pipeline-core 2.3.0 + pyarrow 23.0.1 where the lock pins 3.0.1 + 16.1.0. - C-105 (Tier 3) a run uploads file-by-file with no rollback and no idempotency. Added to Cluster J. - C-106 (Tier 4) `wire/header.build_header` is reachable only from tests; so is `gaul_schema.colrow`. C-100's shape, one module over. - C-107 (Tier 4) the doc-accuracy scan reads markdown only, so two docstrings still point at modules deleted in #151 and moved in #153. Header counts and Cluster J updated; test_register_integrity green (40 tests). Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Closes the loop the partner has now opened twice (#278, views-postprocessing#272): they ask for integer GAUL codes, we say no, and the reason we give does not survive contact with the format we actually deliver. The old reason was wrong. "Codes are always float64" was justified by an integer column being unable to carry a missing value — a property of NumPy-backed pandas, not of parquet. Arrow and parquet carry nullable integers natively, and this repo's own `data/gaul_lookup.parquet` stores all three code columns as `int64` with zero nulls; the float is introduced by `contract/wire/sidecar.py`, not by the source. The 2026-07-19 ruling weighed "int64 when complete" and rejected it for making the schema depend on the data. It never weighed nullable int64. The rule survives on a measured ground instead. Measured 2026-08-17, pyarrow 23.0.1 / pandas 3.0.5: an int64 parquet column with no null reads back `int64` under a default `pd.read_parquet`; the same column with one null reads back `float64`. So nullable int64 does not remove the float — it moves it from our writer to the consumer's reader and makes it appear only sometimes, which is the data-dependent schema §5.1 exists to prevent. Independently, faoapi's reader `reindex`es the sidecar and calls `.to_numpy()`; both yield float64 from a nullable integer column, so the change would not reach them as integers anyway. The useful half is recorded too: because the delivered region drops the GAUL-uncovered cells, no delivered code is ever missing — held in CI by `tests/test_gaul_lookup_fidelity.py::test_lookup_has_no_nulls` — so a consumer's `astype("int64")` is lossless for this product. That is a property of the region, not the contract, which is why the type stays float64. No behaviour change, no `contract_version` bump: §5.1a records a rejected alternative, it does not alter the rule. `gaul_schema.py`'s docstring now says so at the declaration and points at §5.1a, so the next person to be asked finds the answer where they are standing rather than reopening it. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
This was referenced Aug 17, 2026
Closed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Two commits, both documentation. No behaviour change, no
contract_versionbump.1. The §5.1 ruling (#278)
The partner has asked twice for integer GAUL codes. We were preparing to say no a second time, and the reason we were going to give does not survive contact with the format we deliver.
The old reason was wrong. "Codes are always float64" rested on an integer column being unable to carry a missing value. That is true of NumPy-backed pandas and false of parquet — arrow carries nullable integers natively, and this repo's own
data/gaul_lookup.parquetstores all three code columns asint64with zero nulls. The float is introduced bycontract/wire/sidecar.py, not by the data. The 2026-07-19 ruling weighed "int64 when complete" and rejected it for making the schema depend on the data; it never weighed nullable int64. #278 is right that the option was unconsidered.The rule survives on a measured ground. Measured 2026-08-17, pyarrow 23.0.1 / pandas 3.0.5:
int64float64float64So nullable int64 does not remove the float — it moves it from our writer to the consumer's reader and makes it appear only sometimes, which is precisely the data-dependent schema §5.1 exists to prevent. Independently, faoapi's
wire_readerreindexes the sidecar and calls.to_numpy(); both return float64 from a nullable integer column, so the change would not reach the consumer as integers anyway.The useful half is recorded too. Because the delivered region drops the GAUL-uncovered cells, no delivered code is ever missing — held in CI by
tests/test_gaul_lookup_fidelity.py::test_lookup_has_no_nulls— so a consumer'sastype("int64")is lossless for this product. That is a property of the region, not the contract, which is why the type stays float64.Why an ADR clarification rather than a register entry: this is a settled decision, not a risk. §5.1a sits beside the ruling it corrects, and
gaul_schema.py's docstring now states it at the declaration and points there — so the next person to be asked finds the answer where they are standing. That is what #278 asked for: "a sentence in thegaul_schema.pydocstring saying nullable int was considered and rejected, so this question is not reopened every time a consumer asks."No new test.
test_lookup_has_no_nullsalready exists and already runs in CI (the always-on half oftest_gaul_lookup_fidelity.py), and the chain to the delivered artifact is closed bybuild_sidecarfailing loud on a gid absent from the lookup. Adding a second assertion of the same fact would be duplication, not coverage.2. The register (assimilation pass)
C-103..C-107, deduplicated against all 22 open and 80 resolved entries before writing. Header counts 102→107 / 22→27 open; C-105 added to Cluster J.
datafactory_query, declared nowhere and not installed; a bareexcept Exceptionmakes a missing dependency indistinguishable from "the producer publishes no boundary", and both ship fabricated months. Carries the verification question it cannot answer from this seat.wire/header.build_headerandgaul_schema.colroware reachable only from tests. C-100's shape, one module over.Verification
ruff check .— clean.test_doc_accuracy,test_register_integrity,test_gaul_lookup_fidelity,test_falsify_adr013_s5,test_wire_naming— 78 passed.test_release_version.py:56, "HEAD carries no release tag", expected on a branch.Closes #278.