Repository navigation
C-105: a torn run says what it left behind - #288
Conversation
The contract already handles the CONSUMER's side of a mid-upload failure
correctly and by design: the manifest is uploaded last, so an attempt that dies
before it has no commit marker and is invisible rather than half-visible (§4.2).
Nothing partial is served.
Our side was the gap. The objects that did land stayed in the partner store and
nothing recorded that they had — an operator was left to diff the bucket by hand,
and at run-0 scale a retry adds ~110 more under the same names.
`deliver_run` now keeps an in-memory ledger alongside the log, and any failure in
the upload phase raises `TornRunError` naming the run, how many of how many
objects landed, and their names AND file ids. The message states three things an
operator would otherwise have to establish themselves:
- the consumer cannot see this run, so nothing partial is being served;
- the listed objects are still there and were NOT removed;
- a re-run uploads all of them again under the same names, and whether the
store supersedes or duplicates is not something this repository asserts.
DELETION IS DELIBERATELY NOT DONE. Removing objects from a partner bucket is
irreversible and an operator decision rather than a delivery-path one, and the
neighbouring delete surface is its own open question (C-58,
views-pipeline-core #333, blocked on a test key). C-105 therefore stays open with
a Partial mitigation: the mess is now legible, and it is still a mess. Closing it
needs a decision about who cleans up, which is not engineering work here.
tests/test_torn_run.py (5), mutation-proven both ways — dropping the wrapper
fails 4, dropping the ledger fails 3.
Suite: 458 passed, 3 skipped, 37 xfailed. The 26 failures are C-104 venv drift.
ruff clean.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Three findings. 1. COVERAGE GAP, recorded rather than closed. `TornRunError` wraps the upload phase inside `deliver_run`. The historical artifact uploads AFTER the wire run is committed, from the manager, so a failure there raises unwrapped — and its consequence differs rather than being smaller: the manifest already landed, so the consumer sees a complete, visible forecast run next to the PREVIOUS run's historical artifact. Not corrupt (the historical is a full snapshot, so the older one is valid, just one run stale), and the delivery does report failure — but it is the one tear where "the consumer cannot see this run" is false. The wrapper does not fire there, so nothing says anything wrong; the gap is that nothing says anything at all. Left uncovered deliberately: the manager is at 435/450 of its line budget and this belongs with whoever takes C-105's deletion decision. Recorded in C-105 and in the CIC. 2. The test module reached into `tests/test_hop_b_sink_e2e` for `FakeLease` and the PRIVATE `_synthetic_lookup`, coupling two files that should be able to change apart. Both are now local, and the lookup is derived from `gaul_schema.METADATA_COLS` rather than copied as a literal table — a hand-written table here would drift from the contract silently. 3. `_torn` named a factory that returns an exception rather than raising one. `_torn_run_error`, with the return type declared. Also bumped the CIC review date, which its own guard caught: the body cited 2026-08-19 while the header still said 2026-08-18. Suite: 458 passed, 3 skipped, 37 xfailed. ruff clean. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
…n-run refusal Six findings. The first three would each have sent an operator the wrong way, which is a bad property for a message whose only job is to direct one. 1. HIGH. The refusal omitted the object that FAILED — and that object is the likeliest orphan of the entire run. `_ContractStorePort.upload` raises precisely in the case its own comment documents: the store "RETURNS success=False with the file already uploaded". So the failing object is the C-79 shape, sitting in the bucket with no metadata document, while the message listed only the successes and called that what remains. An operator who trusted it would walk straight past the orphan this change exists to surface. Now named explicitly, as a separate thing to look for. 2. MEDIUM. File ids were truncated to five in the message and recorded nowhere else — the log ledger line carried name, doc_type and run_id but not file_id. At run-0 scale that is ~104 ids existing only in a string nobody kept. The ledger line now carries file_id and is the persistent record; the message says so. 3. MEDIUM. A failure on the FIRST upload — credentials expire, bucket unreachable, the commonest infrastructure failure there is — printed "Already in the partner store, and NOT removed: ." An empty list with a dangling period, presented as a bucket to audit. Now says nothing is confirmed landed, while keeping the caveat from (1), which still applies. 4. LOW. "The consumer cannot see this run" was asserted categorically, and the code cannot know it: a manifest upload can fail after the store committed the document, in which case the consumer IS being served this run and a re-run duplicates every object. Softened to "almost certainly", with an instruction to verify. 5. LOW. TornRunError inherited SinkError, whose family means "cannot be delivered as declared" — malformed input, do not retry. A tear is transient infrastructure with the opposite semantics, and test_hop_b_sink_e2e already uses `pytest.raises(SinkError)` as the malformed-run assertion. Re-based on RuntimeError, matching delivery.findability's two error types, with the reasoning in the docstring. 6. LOW. Missing blank lines before a module constant in the test file. Three new tests (8 total). Mutation-proven: dropping the orphan caveat fails 2, re-basing on SinkError fails 1. Suite: 461 passed, 3 skipped, 37 xfailed. ruff clean. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Review round —
|
| # | finding | fix |
|---|---|---|
| 4 | "the consumer cannot see this run" asserted categorically — the code cannot know it, since a manifest upload can fail after the store committed the document, in which case the consumer is being served and a re-run duplicates everything | softened to "almost certainly", with an instruction to verify |
| 5 | TornRunError inherited SinkError — "cannot be delivered as declared", i.e. do-not-retry. A tear is transient with the opposite semantics, and test_hop_b_sink_e2e already uses pytest.raises(SinkError) as the malformed assertion |
re-based on RuntimeError, matching delivery.findability |
| 6 | test module reached into test_hop_b_sink_e2e for a private helper; missing blank lines |
self-contained, lookup derived from gaul_schema |
Recorded, not fixed
The historical leg is outside the wrapper. It uploads after the wire run is committed, so a failure there leaves a visible, complete forecast run beside the previous run's historical artifact — the one tear where "the consumer cannot see this run" would be false. The wrapper does not fire there, so nothing says anything wrong; the gap is that nothing says anything at all. Registered as remaining scope in C-105 and the CIC rather than widening this change — the manager is at 435/450 of its line budget and it belongs with whoever takes the deletion decision.
Verification
| mutation | caught |
|---|---|
| drop the try/except | 4 tests |
| drop the ledger | 3 tests |
| drop the orphan caveat | 2 tests |
re-base on SinkError |
1 test |
CI: 485 passed, 5 skipped, 37 xfailed, 0 failed.
The gap, precisely
The contract already handles the consumer's side of a mid-upload failure correctly and by design: the manifest is uploaded last, so an attempt that dies before it has no commit marker and is invisible rather than half-visible (§4.2). Nothing partial is served.
Our side was the gap. C-105's own words: "the k uploaded objects remain, nothing records that they exist, and nothing removes them." An operator was left to diff the bucket by hand, and at run-0 scale a retry adds ~110 more under the same names.
What changed
deliver_runkeeps an in-memory ledger alongside the log line, and any failure in the upload phase raisesTornRunErrornaming the run, how many of how many objects landed, and their names and file ids.The message states three things an operator would otherwise have to establish themselves:
What this deliberately does not do
It deletes nothing. Removing objects from a partner bucket is irreversible and an operator decision rather than a delivery-path one, and the neighbouring delete surface is its own open question (C-58, views-pipeline-core #333, blocked on a test key).
So C-105 stays open with a Partial mitigation: the mess is now legible, and it is still a mess. Closing it needs a decision about who cleans up and whether the store supersedes — neither of which is engineering work here.
Verification
tests/test_torn_run.py— 5 tests, mutation-proven both ways:Suite: 458 passed, 3 skipped, 37 xfailed. The 26 failures are the C-104 venv drift, green in CI.
ruffclean.