Skip to content

C-105: a torn run says what it left behind - #288

Merged
Polichinel merged 3 commits into
developmentfrom
fix/c105-torn-run-is-diagnosable
Aug 19, 2026
Merged

Polichinel merged 3 commits into
developmentfrom
fix/c105-torn-run-is-diagnosable

Conversation

@Polichinel

Copy link
Copy Markdown
Collaborator

The gap, precisely

The contract already handles the consumer's side of a mid-upload failure correctly and by design: the manifest is uploaded last, so an attempt that dies before it has no commit marker and is invisible rather than half-visible (§4.2). Nothing partial is served.

Our side was the gap. C-105's own words: "the k uploaded objects remain, nothing records that they exist, and nothing removes them." An operator was left to diff the bucket by hand, and at run-0 scale a retry adds ~110 more under the same names.

What changed

deliver_run keeps an in-memory ledger alongside the log line, and any failure in the upload phase raises TornRunError naming the run, how many of how many objects landed, and their names and file ids.

The message states three things an operator would otherwise have to establish themselves:

  • the consumer cannot see this run — the manifest never landed, so nothing partial is being served. Usually their first question.
  • the listed objects are still there and were NOT removed. A reader who assumes cleanup happened will not go looking.
  • a re-run uploads all of them again under the same names, and whether the store supersedes or duplicates is a semantic this repository does not assert. That is a decision to take before re-running.

What this deliberately does not do

It deletes nothing. Removing objects from a partner bucket is irreversible and an operator decision rather than a delivery-path one, and the neighbouring delete surface is its own open question (C-58, views-pipeline-core #333, blocked on a test key).

So C-105 stays open with a Partial mitigation: the mess is now legible, and it is still a mess. Closing it needs a decision about who cleans up and whether the store supersedes — neither of which is engineering work here.

Verification

tests/test_torn_run.py — 5 tests, mutation-proven both ways:

mutation result
drop the try/except, let the raw error escape 4 failed
stop recording what landed 3 failed
neither 5 passed

Suite: 458 passed, 3 skipped, 37 xfailed. The 26 failures are the C-104 venv drift, green in CI. ruff clean.

Polichinel and others added 3 commits August 19, 2026 11:39
The contract already handles the CONSUMER's side of a mid-upload failure
correctly and by design: the manifest is uploaded last, so an attempt that dies
before it has no commit marker and is invisible rather than half-visible (§4.2).
Nothing partial is served.

Our side was the gap. The objects that did land stayed in the partner store and
nothing recorded that they had — an operator was left to diff the bucket by hand,
and at run-0 scale a retry adds ~110 more under the same names.

`deliver_run` now keeps an in-memory ledger alongside the log, and any failure in
the upload phase raises `TornRunError` naming the run, how many of how many
objects landed, and their names AND file ids. The message states three things an
operator would otherwise have to establish themselves:

  - the consumer cannot see this run, so nothing partial is being served;
  - the listed objects are still there and were NOT removed;
  - a re-run uploads all of them again under the same names, and whether the
    store supersedes or duplicates is not something this repository asserts.

DELETION IS DELIBERATELY NOT DONE. Removing objects from a partner bucket is
irreversible and an operator decision rather than a delivery-path one, and the
neighbouring delete surface is its own open question (C-58,
views-pipeline-core #333, blocked on a test key). C-105 therefore stays open with
a Partial mitigation: the mess is now legible, and it is still a mess. Closing it
needs a decision about who cleans up, which is not engineering work here.

tests/test_torn_run.py (5), mutation-proven both ways — dropping the wrapper
fails 4, dropping the ledger fails 3.

Suite: 458 passed, 3 skipped, 37 xfailed. The 26 failures are C-104 venv drift.
ruff clean.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Three findings.

1. COVERAGE GAP, recorded rather than closed. `TornRunError` wraps the upload
   phase inside `deliver_run`. The historical artifact uploads AFTER the wire run
   is committed, from the manager, so a failure there raises unwrapped — and its
   consequence differs rather than being smaller: the manifest already landed, so
   the consumer sees a complete, visible forecast run next to the PREVIOUS run's
   historical artifact. Not corrupt (the historical is a full snapshot, so the
   older one is valid, just one run stale), and the delivery does report failure
   — but it is the one tear where "the consumer cannot see this run" is false.
   The wrapper does not fire there, so nothing says anything wrong; the gap is
   that nothing says anything at all. Left uncovered deliberately: the manager is
   at 435/450 of its line budget and this belongs with whoever takes C-105's
   deletion decision. Recorded in C-105 and in the CIC.

2. The test module reached into `tests/test_hop_b_sink_e2e` for `FakeLease` and
   the PRIVATE `_synthetic_lookup`, coupling two files that should be able to
   change apart. Both are now local, and the lookup is derived from
   `gaul_schema.METADATA_COLS` rather than copied as a literal table — a
   hand-written table here would drift from the contract silently.

3. `_torn` named a factory that returns an exception rather than raising one.
   `_torn_run_error`, with the return type declared.

Also bumped the CIC review date, which its own guard caught: the body cited
2026-08-19 while the header still said 2026-08-18.

Suite: 458 passed, 3 skipped, 37 xfailed. ruff clean.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
…n-run refusal

Six findings. The first three would each have sent an operator the wrong way,
which is a bad property for a message whose only job is to direct one.

1. HIGH. The refusal omitted the object that FAILED — and that object is the
   likeliest orphan of the entire run. `_ContractStorePort.upload` raises
   precisely in the case its own comment documents: the store "RETURNS
   success=False with the file already uploaded". So the failing object is the
   C-79 shape, sitting in the bucket with no metadata document, while the message
   listed only the successes and called that what remains. An operator who
   trusted it would walk straight past the orphan this change exists to surface.
   Now named explicitly, as a separate thing to look for.

2. MEDIUM. File ids were truncated to five in the message and recorded nowhere
   else — the log ledger line carried name, doc_type and run_id but not file_id.
   At run-0 scale that is ~104 ids existing only in a string nobody kept. The
   ledger line now carries file_id and is the persistent record; the message says
   so.

3. MEDIUM. A failure on the FIRST upload — credentials expire, bucket
   unreachable, the commonest infrastructure failure there is — printed
   "Already in the partner store, and NOT removed: ." An empty list with a
   dangling period, presented as a bucket to audit. Now says nothing is confirmed
   landed, while keeping the caveat from (1), which still applies.

4. LOW. "The consumer cannot see this run" was asserted categorically, and the
   code cannot know it: a manifest upload can fail after the store committed the
   document, in which case the consumer IS being served this run and a re-run
   duplicates every object. Softened to "almost certainly", with an instruction
   to verify.

5. LOW. TornRunError inherited SinkError, whose family means "cannot be delivered
   as declared" — malformed input, do not retry. A tear is transient
   infrastructure with the opposite semantics, and test_hop_b_sink_e2e already
   uses `pytest.raises(SinkError)` as the malformed-run assertion. Re-based on
   RuntimeError, matching delivery.findability's two error types, with the
   reasoning in the docstring.

6. LOW. Missing blank lines before a module constant in the test file.

Three new tests (8 total). Mutation-proven: dropping the orphan caveat fails 2,
re-basing on SinkError fails 1.

Suite: 461 passed, 3 skipped, 37 xfailed. ruff clean.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
@Polichinel

Copy link
Copy Markdown
Collaborator Author

Review round — /review-diff + /code-review high

Nine findings across the two passes. Three of them would each have sent an operator the wrong way, which is a bad property for a message whose only job is to direct one.

The one that mattered — HIGH

The refusal omitted the object that failed — and that object is the likeliest orphan of the entire run.

_ContractStorePort.upload raises precisely in the case its own comment documents:

the store logs, then RETURNS success=False with the file already uploaded

So the failing object is the C-79 shape — sitting in the bucket with no metadata document — while my message listed only the successes and presented that as what remains. An operator trusting it would walk straight past the orphan this change exists to surface.

The other two that misdirect

  • File ids were truncated to five and recorded nowhere else. The log ledger carried name, doc_type and run_id but not file_id, so at run-0 scale ~104 ids existed only in a truncated string nobody kept. The ledger line now carries file_id and is the persistent record.
  • A failure on the first upload — expired credentials, unreachable bucket, the commonest infrastructure failure there is — printed Already in the partner store, and NOT removed: . An empty list and a dangling period, presented as a bucket to audit.

And three smaller

# finding fix
4 "the consumer cannot see this run" asserted categorically — the code cannot know it, since a manifest upload can fail after the store committed the document, in which case the consumer is being served and a re-run duplicates everything softened to "almost certainly", with an instruction to verify
5 TornRunError inherited SinkError — "cannot be delivered as declared", i.e. do-not-retry. A tear is transient with the opposite semantics, and test_hop_b_sink_e2e already uses pytest.raises(SinkError) as the malformed assertion re-based on RuntimeError, matching delivery.findability
6 test module reached into test_hop_b_sink_e2e for a private helper; missing blank lines self-contained, lookup derived from gaul_schema

Recorded, not fixed

The historical leg is outside the wrapper. It uploads after the wire run is committed, so a failure there leaves a visible, complete forecast run beside the previous run's historical artifact — the one tear where "the consumer cannot see this run" would be false. The wrapper does not fire there, so nothing says anything wrong; the gap is that nothing says anything at all. Registered as remaining scope in C-105 and the CIC rather than widening this change — the manager is at 435/450 of its line budget and it belongs with whoever takes the deletion decision.

Verification

mutation caught
drop the try/except 4 tests
drop the ledger 3 tests
drop the orphan caveat 2 tests
re-base on SinkError 1 test

CI: 485 passed, 5 skipped, 37 xfailed, 0 failed.

@Polichinel
Polichinel merged commit 2b29028 into development Aug 19, 2026
4 checks passed
@Polichinel
Polichinel deleted the fix/c105-torn-run-is-diagnosable branch August 19, 2026 09:48
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant