Skip to content

#278: the §5.1 dtype ruling, and the register from the assimilation pass - #279

Merged
Polichinel merged 4 commits into
developmentfrom
docs/278-gaul-dtype-ruling
Aug 17, 2026
Merged

Polichinel merged 4 commits into
developmentfrom
docs/278-gaul-dtype-ruling

Conversation

@Polichinel

Copy link
Copy Markdown
Collaborator

Two commits, both documentation. No behaviour change, no contract_version bump.

1. The §5.1 ruling (#278)

The partner has asked twice for integer GAUL codes. We were preparing to say no a second time, and the reason we were going to give does not survive contact with the format we deliver.

The old reason was wrong. "Codes are always float64" rested on an integer column being unable to carry a missing value. That is true of NumPy-backed pandas and false of parquet — arrow carries nullable integers natively, and this repo's own data/gaul_lookup.parquet stores all three code columns as int64 with zero nulls. The float is introduced by contract/wire/sidecar.py, not by the data. The 2026-07-19 ruling weighed "int64 when complete" and rejected it for making the schema depend on the data; it never weighed nullable int64. #278 is right that the option was unconsidered.

The rule survives on a measured ground. Measured 2026-08-17, pyarrow 23.0.1 / pandas 3.0.5:

written pandas default read
int64, no nulls int64
int64, one null float64
float64 float64

So nullable int64 does not remove the float — it moves it from our writer to the consumer's reader and makes it appear only sometimes, which is precisely the data-dependent schema §5.1 exists to prevent. Independently, faoapi's wire_reader reindexes the sidecar and calls .to_numpy(); both return float64 from a nullable integer column, so the change would not reach the consumer as integers anyway.

The useful half is recorded too. Because the delivered region drops the GAUL-uncovered cells, no delivered code is ever missing — held in CI by tests/test_gaul_lookup_fidelity.py::test_lookup_has_no_nulls — so a consumer's astype("int64") is lossless for this product. That is a property of the region, not the contract, which is why the type stays float64.

Why an ADR clarification rather than a register entry: this is a settled decision, not a risk. §5.1a sits beside the ruling it corrects, and gaul_schema.py's docstring now states it at the declaration and points there — so the next person to be asked finds the answer where they are standing. That is what #278 asked for: "a sentence in the gaul_schema.py docstring saying nullable int was considered and rejected, so this question is not reopened every time a consumer asks."

No new test. test_lookup_has_no_nulls already exists and already runs in CI (the always-on half of test_gaul_lookup_fidelity.py), and the chain to the delivered artifact is closed by build_sidecar failing loud on a gid absent from the lookup. Adding a second assertion of the same fact would be duplication, not coverage.

2. The register (assimilation pass)

C-103..C-107, deduplicated against all 22 open and 80 resolved entries before writing. Header counts 102→107 / 22→27 open; C-105 added to Cluster J.

  • C-103 (Tier 2) — the observed-range clip depends on datafactory_query, declared nowhere and not installed; a bare except Exception makes a missing dependency indistinguishable from "the producer publishes no boundary", and both ship fabricated months. Carries the verification question it cannot answer from this seat.
  • C-104 (Tier 3) — a stale virtualenv takes 25 tests red, 20 of them the only tests that import either manager.
  • C-105 (Tier 3) — a run uploads file-by-file with no rollback and no idempotency.
  • C-106 (Tier 4) — wire/header.build_header and gaul_schema.colrow are reachable only from tests. C-100's shape, one module over.
  • C-107 (Tier 4) — the doc-accuracy scan reads markdown only; two docstrings still point at modules deleted in S3 — Collapse the two extraction seams into one #151 and moved in S5 — Separate partner declarations from delivery machinery #153.

Verification

  • ruff check . — clean.
  • Targeted guards green: test_doc_accuracy, test_register_integrity, test_gaul_lookup_fidelity, test_falsify_adr013_s5, test_wire_naming — 78 passed.
  • Full suite: 432 passed, 25 failed, 1 skipped, 39 xfailed. The 25 are the pre-existing local venv drift now registered as C-104 (venv holds pipeline-core 2.3.0 / pyarrow 23.0.1; the lock pins 3.0.1 / 16.1.0) — the identical set before and after these commits, and CI installs from the lock. The 1 skip is test_release_version.py:56, "HEAD carries no release tag", expected on a branch.

Closes #278.

Polichinel and others added 4 commits August 14, 2026 03:14
Governance truth pass to main — no package change, no new tag
Release 1.1.1 — two delivery-path fixes from views-crafdapi, and the governance pass behind them
Five entries, none of which duplicated an existing one — checked against all 22
open and 80 resolved concerns before writing:

- C-103 (Tier 2) the observed-range clip depends on `datafactory_query`, which is
  declared nowhere and is not installed; its absence is swallowed by a bare
  `except Exception`, so a missing dependency and "the producer publishes no
  boundary" leave through the same branch and ship fabricated months either way.
  Carries the verification question it cannot answer from this seat.
- C-104 (Tier 3) a stale virtualenv takes 25 tests red, 20 of them the only tests
  that import either manager. Measured: 433 passed / 25 failed against
  pipeline-core 2.3.0 + pyarrow 23.0.1 where the lock pins 3.0.1 + 16.1.0.
- C-105 (Tier 3) a run uploads file-by-file with no rollback and no idempotency.
  Added to Cluster J.
- C-106 (Tier 4) `wire/header.build_header` is reachable only from tests; so is
  `gaul_schema.colrow`. C-100's shape, one module over.
- C-107 (Tier 4) the doc-accuracy scan reads markdown only, so two docstrings
  still point at modules deleted in #151 and moved in #153.

Header counts and Cluster J updated; test_register_integrity green (40 tests).
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Closes the loop the partner has now opened twice (#278, views-postprocessing#272):
they ask for integer GAUL codes, we say no, and the reason we give does not
survive contact with the format we actually deliver.

The old reason was wrong. "Codes are always float64" was justified by an integer
column being unable to carry a missing value — a property of NumPy-backed pandas,
not of parquet. Arrow and parquet carry nullable integers natively, and this
repo's own `data/gaul_lookup.parquet` stores all three code columns as `int64`
with zero nulls; the float is introduced by `contract/wire/sidecar.py`, not by
the source. The 2026-07-19 ruling weighed "int64 when complete" and rejected it
for making the schema depend on the data. It never weighed nullable int64.

The rule survives on a measured ground instead. Measured 2026-08-17, pyarrow
23.0.1 / pandas 3.0.5: an int64 parquet column with no null reads back `int64`
under a default `pd.read_parquet`; the same column with one null reads back
`float64`. So nullable int64 does not remove the float — it moves it from our
writer to the consumer's reader and makes it appear only sometimes, which is the
data-dependent schema §5.1 exists to prevent. Independently, faoapi's reader
`reindex`es the sidecar and calls `.to_numpy()`; both yield float64 from a
nullable integer column, so the change would not reach them as integers anyway.

The useful half is recorded too: because the delivered region drops the
GAUL-uncovered cells, no delivered code is ever missing — held in CI by
`tests/test_gaul_lookup_fidelity.py::test_lookup_has_no_nulls` — so a consumer's
`astype("int64")` is lossless for this product. That is a property of the
region, not the contract, which is why the type stays float64.

No behaviour change, no `contract_version` bump: §5.1a records a rejected
alternative, it does not alter the rule. `gaul_schema.py`'s docstring now says
so at the declaration and points at §5.1a, so the next person to be asked finds
the answer where they are standing rather than reopening it.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant