Skip to content

fix(remote): bound artifact preparation and upload phases - #4552

Merged
thymikee merged 27 commits into
mainfrom
t3/audit4441/remote-budgets
Nov 15, 2026
Merged

thymikee merged 27 commits into
mainfrom
t3/audit4441/remote-budgets

Conversation

@thymikee

@thymikee thymikee commented Nov 14, 2026 •

Copy link
Copy Markdown
Member

Summary

Remote artifact uploads share a finite 600 s lifecycle across preparation, retries, resume and fallback. Connect and post-flush headers have 30 s ceilings; body inactivity has 120 s. Headers now start at connected request finish, independently of file-close/pipeline cleanup. Typed phase expiry is terminal and caller cancellation preserves its reason.

This changes 17 files. Host-kit owns completed nonempty writes and response reading; the remote reader copy is removed. The header regression retains its progress premises and assertions while pinning the held source fd and real close completion. Duplicate lower phase comments were removed; the header contract remains. Refs #4441. Proxy, daemon and downloads remain separate.

Validation

Tested 13a4c1d9576777989ab3849cda9fb78661a33988, base d78be6c8.

Pinned pnpm 11.17.0 check:affected --run --head <sha> --json passed once: exactCommit=true, checkoutChanged=false, failOpen=false. Nine checks passed, including wire compatibility, 376 related tests and 19,841 unit tests (10 skipped). Seven CI-owned/manual checks were skipped; #4050 full-unit fallback remains.

Cold review found no findings. Focused baseline passed 121 tests. Published 77a5 production, missing finish arming and a late pipeline rearm each fail the same 30,000 ms request-destruction assertion; restored controls pass.

Linux Coverage and Integration passed at this head; all required checks are green (one matrix check skipped). Earlier failures retain their attribution; actual CI close latency remains inferred. The separate /proc fixture correction is in open #4544. Per-byte daemon ticket refresh, Node-floor, Bun and end-to-end slow-transfer evidence remain open.

thymikee and others added 9 commits November 14, 2026 12:10
…on aggregate

A single five-minute timer covered a whole upload, so an artifact that kept
moving and one that stalled after the first chunk were cut off the same way,
and nothing said which part of the hop ended the attempt.

Each attempt now runs one sequential watchdog: 30s to connect, 120s of silence
between body bytes in either direction, and 30s for response headers once the
outgoing body is flushed. Only bytes actually written or read re-arm a body
phase, and arming a later phase stops counting the earlier one. All of it stays
inside the 600s preparation aggregate from the previous commit, which every
attempt, redirect, offset resume, fallback and finalize still shares, so the
earliest deadline keeps winning and no resumed work can outlive it.

A phase expiry is typed evidence carrying the phase and its effective budget. It
is deliberately not retryable: restarting the sequence would buy a second
attempt permission the budget already refused, so neither the ticket retry nor
the legacy fallback restarts those bytes. Caller cancellation stays a separate
account and still answers with the signal's own reason.

The two control-plane calls that bracket an upload, preflight and finalize,
carry one 30s ceiling each: their payload is a few hundred bytes of JSON the
transport writes with the request, so there is no separate body phase to time.
…hases

Two corrections to the phase budgeting slice, both found in review.

An early response resolved, redirected or resumed while the attempt's own file
→ transform → request pipeline was still writing into the old hop. The answer
was read while the owned file handle stayed open and the queued body kept
flowing at a peer this client had already left behind. The attempt now reads
the accepted answer to its end, destroys its own request only when that read
completed the exchange's verdict, and joins the pipeline before resolving,
starting a redirect, or starting an offset resume. The enclosing preparation
aggregate still bounds what lingered, but no attempt now hands its body to the
next hop.

The connect ceiling ended when the request accepted its first write and body
progress counted chunks read ahead of the socket. A TLS origin that never
answers the handshake, or a peer that stops taking bytes after the headers,
was timed against the wrong budget, and bytes sitting in the write queue
re-armed a body phase that had not moved. The ceiling now ends on the events
that carry the protocol: socket `connect` for plain HTTP, `secureConnect` for
TLS, immediately for a pooled keep-alive socket. Body progress is the
transport's own `drain` and nothing else; the progress transform's byte hook,
which fired before the chunk ever reached the request, is gone.

Wire bytes are unchanged, so the ledger's acknowledged digest for the attempt
moves with the rationale that says so.

The progress-window fixture measured budget against wall-clock scheduling, so
a window could be charged before the parked peer really took a byte and the
"keeps moving" case flaked. Each release now lets one quota of bytes through
and the window waits with real event-loop turns until those bytes arrive
before it charges the budget, which removes the race and costs the whole file
a third of a second instead of two.
…gate in upload-stream

The phase watchdog and the 600-second preparation aggregate lived in two new
modules that every daemon-client entry now evaluated eagerly. upload-stream.ts
is already in that closure and is the lowest module all three consumers reach,
so it now owns both. withUploadPreparationDeadline still starts its timer
synchronously when called, and isTransferPhaseTimeoutError stays a synchronous
typed check.

Rename the transfer tests for the module they exercise (R84).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…mpt wire digest

6089d90 moved the attempt's watchdog.dispose() ahead of the redirect/resume
decision without refreshing the ledger. The digest pasted here is the one
wire-compat.test.ts prints for the current source; the rationale now names
the retired watchdog. Requests on the wire are unchanged.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…private to upload-stream

No module outside upload-stream reads them; consumers classify through
isTransferPhaseTimeoutError.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@thymikee
thymikee added this pull request to stack #4553 November 14, 2026 15:27
@github-actions

github-actions Bot commented Nov 14, 2026 •

Copy link
Copy Markdown
Contributor

Size Report

Metric Base Current Diff
Installed (including dependencies) 6.28 MB 6.28 MB +2.6 kB
Package (unpacked) 6.28 MB 6.28 MB +2.6 kB
Package (download) 1.90 MB 1.90 MB +904 B

Startup median (7 runs, lower is better):

Scenario Base Current Diff
CLI --version 20.1 ms 27.4 ms +7.3 ms
CLI --help 42.9 ms 51.4 ms +8.5 ms

@thymikee thymikee changed the title t3/audit4441/remote budgets fix(remote): bound artifact preparation and upload phases Nov 14, 2026

@cubic-dev-ai cubic-dev-ai Bot left a comment •

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

All reported issues were addressed across 10 files

You’re at about 91% of the monthly reviewed-line limit. You may want to disable incremental reviews to conserve quota. Reviews will continue until that limit is exceeded. If you need help avoiding interruptions, please contact contact@cubic.dev.

Reply with feedback, questions, or to request a fix.

View guided diff | Turn on auto-fix | Re-trigger cubic

Comment thread test/wire-compat/ledger.json Resolved
Comment thread packages/remote/src/upload-client.ts Resolved
@apex-sentinel-staging

apex-sentinel-staging Bot commented Nov 14, 2026 •

Copy link
Copy Markdown

Apex Sentinel · ✅ Approve

Since the last review: 0 new · 0 still open · 0 no longer raised

No open findings.

1 finding below minor hidden (user_settings.min_severity).

1 new finding away from what changed since 77a5e16 not shown. Comment @apex-sentinel-staging review for a full review.

Reviewed with this repository's own Sentinel configuration (d78be6c88715).

💬 Talking to Sentinel

In a finding's thread, reply with:

Command What it does
@apex-sentinel-staging note <context> Adds context; Sentinel re-checks the cited code
@apex-sentinel-staging pivot <why> Says the finding is wrong; Sentinel re-checks the cited code
@apex-sentinel-staging like Marks the finding as useful
@apex-sentinel-staging dislike <why> Marks it as not useful, so it is not raised again

On this conversation, comment @apex-sentinel-staging review to run a full review once again.

Earlier reviews

Review of 77a5e167 · 14.11.2026 20:42 UTC

Apex Sentinel · ✅ Approve

Since the last review: 0 new · 0 still open · 0 no longer raised

No open findings.

1 finding below minor hidden (user_settings.min_severity).

Reviewed with this repository's own Sentinel configuration (d78be6c88715).

Review of 22cf1d49 · 14.11.2026 18:56 UTC

Apex Sentinel · ✅ Approve

Since the last review: 0 new · 0 still open · 0 no longer raised

No open findings.

1 finding below minor hidden (user_settings.min_severity).

Reviewed with this repository's own Sentinel configuration (d78be6c88715).

Review of 09e1e760 · 14.11.2026 15:36 UTC

Apex Sentinel · ✅ Approve

No open findings.

1 finding below minor hidden (user_settings.min_severity).

Reviewed with this repository's own Sentinel configuration (d78be6c88715).

@thymikee

Copy link
Copy Markdown
Member Author

At 09e1e76, the response reader in the upload stream is a copy of the shared one, and that needs to change before merge. I ran no tests for this review.

readUploadResponseBody (upload-stream.ts#L474) matches host-kit's readNodeHttpResponseBody (transport.ts#L45-L55) line for line, except for one added onBytes() call. The stacked #4551 keeps the copy too. The issue asks to keep byte caps, so a cap added to one reader would miss the other. Please add an optional onChunk callback to readNodeHttpResponseBody, call it from upload-stream.ts#L405, and delete the copy with its comment. The rule is that the host-kit transport owns every Node HTTP response-body reader.

Not blocking, and you can take or leave these: three nested withUploadPreparationDeadline calls arm the same 600s aggregate (daemon-artifacts.ts#L49, upload-client.ts#L71, upload-stream.ts#L180), the outer one always fires first, so the inner message and hint parameters are dead, and one module should arm the aggregate while the rest receive its signal; the catch in retryDirectUpload (upload-client.ts#L217) turns a cancel or aggregate expiry into a "retrying with legacy upload" fallback event, so it should rethrow when the sequence's own cancellation or deadline ended it, as tryDirectUploadWithResume already does at :170; TransferPhaseWatchdog (upload-stream.ts#L64) moves to host-kit in #4551, so landing it there first would avoid writing 163 lines and 168 test lines twice; the connect-constant comment at upload-stream.ts#L18 mentions a standalone artifact download, but artifact-download.ts never reads that constant; and #4441 asks how the 600s budget at upload-stream.ts#L118 relates to upload-ticket expiry, which neither the PR body nor the code states.

Could one aggregate be armed at prepareRemoteRequestArtifacts with its signal passed down, with the watchdog placed in host-kit from the start? That would remove roughly 160 production lines from remote and about 170 test lines that the next layer deletes anyway, and the ~740-line fixture file could shrink to what the remaining loopback tests need. The #4551 host-kit move would have to land first, or fold into this layer.

One inline thread still needs a fix: loopback progress test. Four lower-priority threads on test cleanup, the fixture's throw inside a request listener, and the unreachable timeout hint also still apply. This thread does not apply, so please resolve it: #4552 (comment) (wire-compat acks are keyed per digest, and the cross-origin refusal is unchanged on the wire).

The Coverage check failed in its Run coverage step. Its log stops after the fuzz-envelope CLI tests and shows no test summary or failed test name, so I cannot say whether this change caused it. The diff does add four real-loopback test files that unit-ci runs, so I cannot clear the failure. I could not compare with Coverage on main. There are no conflicts. Before merge, route the response read through the host-kit reader, then rerun Coverage to get either a pass or a captured failure.

@thymikee

Copy link
Copy Markdown
Member Author

Accepted: the response reader now has an optional onChunk callback in host-kit, and the local upload copy is deleted. The local lower correction also fixes the TLS/pool cleanup and unexpected-request fixture path, pins the adopted phase expectations, and rethrows an ended sequence before retry fallback telemetry. These changes are not pushed yet.

I am retaining the two independently callable admission boundaries (prepareRemoteRequestArtifacts and standalone uploadArtifact). I will remove the attempt-level aggregate and dead timeout arguments only after checking that every production caller already carries the enclosing signal. Moving the complete owner into the lower PR or reversing the published stack is deferred; both current layers remain independently mergeable in order. I will document the independent ticket-lifetime versus client-aggregate relationship in the final reply/body.

The loopback header-wait case is being corrected to establish actual progress beyond four connect ceilings before the header wait. Fresh source review, discrimination and one exact gate per final head precede the single push. I reran the failed Coverage job once as requested and retained the original truncated log; no cause or pass is inferred yet. The wire-ack thread is not a wire defect and will receive that disposition.

@thymikee

Copy link
Copy Markdown
Member Author

The single requested Coverage rerun also failed at the unchanged published head. Its text log is still truncated, but the check-run annotations identify 3 failures in this PR's remote loopback tests: progress/stall phase cases and the pooled-connection case. The stalled-body case hits the existing 5000 ms test limit; the other failures are assertions. These are captured candidate-test failures, not a passing rerun or an infrastructure diagnosis.

The existing source owner now has the exact annotations and is reconciling real I/O with the controlled clock while making the already-assigned write-progress and fixture corrections. No additional worker, timeout increase, test deletion or CI retry is being started. No moved host-kit test is named in these annotations, but that alone does not establish its successful execution; final-head Coverage remains required. Original and rerun failures retain their heads and logs.

thymikee and others added 6 commits November 14, 2026 17:25
Request-body progress came only from `drain`, which fires after a write
returned false. A slow producer whose writes stay under the socket's
high-water mark never sees a drain, so a progressing upload expired its
120-second body budget. The attempt now wraps the request's `write` and
touches the body phase when a non-empty write completes without error:
its bytes reached the local kernel. Arguments, `this`, the backpressure
result and the caller's callback pass through unchanged, and completions
after the response, an abort, a destroy or the attempt's end do not count.

streamFileToHttpRequest no longer opens a second copy of the preparation
aggregate: its only callers run inside uploadArtifact's, so it requires
that signal and drops the timeout message and hint the outer deadline
always preempted. uploadArtifact and prepareRemoteRequestArtifacts keep
their own 600-second admission boundaries.

Tests: a real loopback slow producer with 1 KiB writes survives six
30-second windows with zero drains and then expires one ceiling after its
last completed write; the header-wait case now proves the body is still
in flight past four connect ceilings before only the header wait
expires; a cancel during the retried direct upload announces no
fallback. Loopback progress windows gate on completed writes.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…tures on owned I/O

Host-kit's transport owns observeCompletedRequestWrites: it reports each
non-empty write Node completes without an error, encoded-empty strings
excluded, and stops after the response, an abort, a destroy or disposal.
The remote upload attempt consumes it instead of a private copy, and the
generic contract has direct tests beside its owner.

The loopback fixtures charge fake time only after the owned I/O a phase
needs was observed, tie pending waits to the test lifecycle, assert the
stalled body is unfinished, and prove the early-answer join with a
producer that never ends instead of a host-speed round bound.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
… digests

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Each loopback server, held-open producer and trusted certificate is
released when its test finishes, so a failed assertion or a timeout
cannot leave one running into the next test.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
An ended transfer still gets at most 20 s of budget, less than any phase
ceiling, and then waits for its settlement event, so a slow host's real
teardown I/O cannot fail the test while the test timeout stays the bound.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@thymikee

Copy link
Copy Markdown
Member Author

Addressed the review at 22cf1d4993ffbf7d7f29f8dd4825e64b6a5f52ab (base d78be6c8).

  • readNodeHttpResponseBody now owns the optional onChunk callback; the remote response-reader copy is deleted.
  • The completed-write observer also lives in host-kit in this first layer, with its negative contract tests. It reports successful nonempty writes accepted by the local kernel, not peer acknowledgement. Empty/failed/queued writes and callbacks after response, abort, destroy or disposal do not refresh timing.
  • Removed the redundant attempt aggregate and dead hint. The independently usable remote-request and standalone-upload entries still arm finite lifecycles before preparation. The larger watchdog-first reshaping is deferred; the upper layer now moves only the phase owner and its actual observer consumer. Cancellation during a retried direct upload rethrows without a legacy-fallback event.
  • The header proof keeps a 16 MiB body unfinished through five observed-I/O windows (150 s), then proves a fresh 29,999 ms + 1 ms header ceiling. Stall cases use unfinished sources instead of assuming kernel buffer capacity. Cleanup and the unexpected-request listener are fixed. All six existing lower inline threads have replies and recorded dispositions, including the inapplicable wire rationale.

Ticket policy: the current daemon starts a five-minute timer at ticket creation and refreshes it after a completed chunk append, not per byte. A single slow chunk can still expire while the client's absolute 600 s lifecycle remains valid. Adopted per-byte refresh remains separate daemon work. This stack does not extend the server ticket or claim end-to-end success beyond that limit.

One pinned pnpm 11.17.0 check:affected --run --head 22cf1d4993ffbf7d7f29f8dd4825e64b6a5f52ab --json passed: exactCommit=true, checkoutChanged=false, failOpen=false. Nine checks passed, including 376 related tests, 19,832 unit tests (10 skipped), and wire compatibility; seven GitHub/manual lanes were skipped. The full-unit fallback remains. Isolated counterfactuals removing uppercase encoding normalization and realm-safe byte detection each failed one test; changing body inactivity to 30 s failed nine. Restored controls passed. Fresh independent review found no actionable findings. Gross size: lower 2,337; upper 741; cumulative 2,482.

The old Coverage run and its single captured rerun remain failed. The rerun annotations identify three remote loopback failures; the truncated log alone could not. Revised fixtures wait for real I/O before charging virtual time. Modeled/macOS evidence does not establish the historical Linux cause. Corrected-head Linux Coverage is now pending and will be followed through the PR watch. No additional blind retry was made. The unused download clause remains an optional comment cleanup, not a claim that downloads use these budgets. #4441 remains open for its remaining slices.

@cubic-dev-ai cubic-dev-ai Bot left a comment •

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

All reported issues were addressed across 13 files (changes from recent commits).

You’re at about 92% of the monthly reviewed-line limit. You may want to disable incremental reviews to conserve quota. Reviews will continue until that limit is exceeded. If you need help avoiding interruptions, please contact contact@cubic.dev.

Reply with feedback, questions, or to request a fix.

View guided diff | Turn on auto-fix | Re-trigger cubic

Comment thread packages/remote/src/upload-client.ts Resolved
@thymikee

Copy link
Copy Markdown
Member Author

I reviewed 22cf1d4 against the findings in my earlier review. The earlier findings are fixed, and I found no new problems that must change before merge.

Not blocking, and you can take or leave it: observeCompletedRequestWrites in transport.ts classifies string chunks by encoding and detects Uint8Array across realms, but its only caller, upload-stream.ts:280, writes only Buffers through pipeline, so those branches cannot run on a real route.

Could upload-stream.ts own the writes instead, since it owns the producer? A loop like for await (const chunk of pipeline(file, progress)) { if (!req.write(chunk, (err) => { if (!err && !responseReceived) watchdog.touchBody(); })) await once(req, "drain", { signal: watchdog.signal }); } req.end() needs no write patch, no prepended response listener and no encoding or realm checks. It would also remove observeCompletedRequestWrites, its host-kit export and transport-writes.test.ts, and the existing "writes below the high-water mark" loopback test would still cover it. Nothing has to land first, and #4551 would follow the same shape.

On the open threads: the one on control-plane timeouts falling back to legacy still applies, and it predates this PR. These look done, so please resolve them: the 16 MiB phase-budget test now covers five windows; CA trust restore is registered at creation; loopback server close is registered at creation; the unexpected-leg handler now answers 500 instead of throwing; timeoutMessage and timeoutHint are removed. The redirect and ledger thread does not apply, because the wire behavior is unchanged.

Smoke Tests (ios) failed in the swiftc compile of AgentDeviceRunnerUITests. This diff touches no Apple or Swift files, so that failure looks unrelated. The Linux Coverage (unit-ci) run is unclear: the log I had stops in the react-native companion tests and names no failed test, and this PR changes four remote loopback test files that the job runs. I did not run the new tests or your counterfactual mutants, and I did not trace the daemon ticket refresh policy.

Before merge, please share a passing Linux Coverage run at 22cf1d4, or the names of its failing tests. The control-plane timeout thread also needs your planned typed fix or an explicit decision.

@thymikee

Copy link
Copy Markdown
Member Author

The failed Linux run at 22cf1d4993ffbf7d7f29f8dd4825e64b6a5f52ab has these three unit-core failures, all Test timed out in 5000ms:

  • upload-stream-phases.test.ts:82 — “a body admitted past four connect ceilings still waits only its own header budget”.
  • upload-stream-lifecycle.test.ts:249 — “a zero-length artifact on a peer that never answers waits on its header budget”.
  • upload-stream-lifecycle.test.ts:361 — “an attempt on a reused connection is already past its connect phase”.

These are from the Coverage job annotations; the truncated text log alone does not identify them. Their Linux cause remains unproven. The upper run being green does not make this lower run a pass. No further CI rerun was requested.

For thread 4575165528, I am taking the typed fix, not adding an exception. Decision 6827686874 §6 keeps the client’s own timeout terminal. The local correction reuses the phase watchdog for preflight/finalize, preserves the earlier caller/aggregate reason, consumes successful JSON under the same 30s control lifetime and disposes it. Error statuses keep their prior immediate handling. Four initial/retried timeout cases, two identical-prose negatives and two mid-stall caller-cancellation cases cover the boundary. The shared sending owner is now wire-listed and has a precise headers/body input; POST and redirect refusal are fixed at construction, without a closure waiver.

New local heads are lower d33183d83b850b23925e77056ed3dc8156f275cc and upper 3b93a9490ed2b4ac544babb75ae8799d1bf8a62e. Focused checks passed 121/127 tests respectively, including wire checks; typecheck passed. Fresh independent recheck, isolated regression proofs and exact-head gates still precede publication. The control-timeout thread stays open until the correction is pushed. All six older lower threads are already resolved.

A producer-owned write loop is a valid alternative if it also owns write completion, close/error/abort races, early-response delivery and join-before-next-hop cleanup. I am deferring that optional ownership rewrite in this round and retaining the tested pipeline/observer composition. The route currently produces Buffers; I make no claim that its string/realm branches run on this route. The iOS compile cause also remains unproven. No merge.

@thymikee

Copy link
Copy Markdown
Member Author

Pushed the typed correction and bounded I/O diagnostics in 77a5e16; #4551 is restacked at af5c634. No timeout-policy exception: preflight/finalize expiry remains terminal, including after retry, and cancellation preserves the caller's reason. The sending helper fixes POST/redirect refusal, is included in the wire manifest, and skips parsing error-status bodies to preserve their previous handling.

Independent source review and the final count-simplification recheck found no findings. Lower/upper focused controls passed 121/127; all four control-contract counterfactuals were detected. Separate exact gates passed: 19,841/19,847 unit tests (10 skipped each), 376 related tests each, wire compatibility and all other local selected checks; seven CI-owned/manual lanes were skipped. The d331 Fallow failure remains failed; only test expectation branching was simplified, with all eight rows/assertions retained.

The three failing Linux unit-ci test names at 22cf remain those in my previous reply: header-budget-after-progress, zero-length header wait and reused connection. Their 5000 ms timeout causes and the iOS compile cause are unproven. This push schedules new-head CI after substantive corrections; I did not retry the failed jobs. Diagnostics now report the stuck I/O condition and state within each test's declared budget.

The optional producer-owned write loop remains deferred for this round, for the ownership reasons in the previous reply. The current route produces Buffers. Current daemon ticket refresh is only after completed chunk append; adopted per-byte refresh remains separate. No merge.

@thymikee

Copy link
Copy Markdown
Member Author

Coverage failed at 77a5e16 in job132579427587 (run43638146821, Linux Node24.13.1). The full captured log reports20188 tests passed,2 failed,70 skipped. The failure is in the test run; changed-line/CRAP enforcement did not run.

  1. upload-stream-phases.test.ts — “a body admitted past four connect ceilings still waits only its own header budget”: the new diagnostic reached settlement after phase expiry with all16777216 bytes received, request finish and peer body-end observed, but settled:false. One bounded isolated Claude diagnosis owns the lower pipeline-completion versus request-finish timing boundary. This is a hypothesis until reproduced; no timeout widening, CI retry or contention claim.
  2. host-process-liveness.test.ts — fleet identity: mocked PID202 was replaced by a real Linux /proc identity/zombie value. This is the fixture-isolation defect already corrected in open owned fix(apple): use typed screenshot timeouts and observed shutdown recovery #4544 (50034fe, retained at d4bc055). That fix is not on this branch; I am recording the dependency rather than duplicating its implementation. No unchanged-base run was made here, so this is not a new base-blocked claim.

The exact local77a5/af5c passes remain local evidence only. The77a5 Coverage job remains FAILED. Both PR watches stay active; no rewatch, new push or rerun. No merge.

@thymikee

Copy link
Copy Markdown
Member Author

At 77a5e16 the code looks good, but the Linux Coverage check still fails, so I cannot call it ready yet. The earlier findings from 22cf1d4 (#4552 (comment)) are fixed, and I found no new problem in the code.

The Coverage (unit-core) run fails on two tests. The first is the PR's own header-budget test, upload-stream-phases.test.ts:112. It timed out at 22cf1d4 too, so it is not a one-off. The test treats the body as flushed on request 'finish' and then advances the fake clock 30 s. Production arms the 'response-headers' phase only after the whole pipeline resolves (upload-stream.ts:288-292). That pipeline also waits for the file stream to close, which can land after the test has spent its 30 s on a loaded runner. The header timer then starts late and the test never settles, which matches the annotation (all 16 MiB received, finished and body ended, not settled). The rule to keep: the header phase must start at the same event the test and the constant's doc call flush. The smallest fix is to mark the body flushed and advance the phase on request 'finish', and keep the pipeline promise only for the join. Please do not widen any timeout. To validate, the Linux Coverage job should pass this test at the new head. I inferred this mechanism from Node's stream semantics and the CI snapshot. I did not reproduce it.

The second failure, host-process-liveness.test.ts:182, looks unrelated: this diff does not touch process liveness, and the failure is a mocked PID resolving against a real Linux /proc. I did not run the base to confirm it also fails on main, so it needs #4544 or a rebase onto it.

Not blocking, and you can take or leave it: the four fallback and retry gates in upload-client.ts each repeat "caller or aggregate aborted, or phase timeout, then rethrow". Would one exported predicate beside isTransferPhaseTimeoutError, for example isOwnUploadOutcome(error, signal), make that rule one symbol so the next fallback site cannot miss half of it? The delta is otherwise a net simplification, since one control-request owner replaces two fetch blocks and combineUploadSignals.

On the other threads: the thread on control-plane expiry reaching the legacy fallback (#4552 (comment)) is fixed at this head, and so is the thread on the 16 MiB budget test (#4552 (comment)), whose remaining flake is the Linux issue above. The minor threads on #4552 (comment), #4552 (comment), #4552 (comment) and #4552 (comment) are fixed at this head, so they can be resolved. The thread on the cross-origin redirect (#4552 (comment)) does not apply: the refusal is unchanged on the wire, so it can be resolved too.

I did not run any tests, and I did not re-review the daemon ticket-refresh policy, which you scoped out of this PR. There are no conflicts. Before merge, Linux Coverage must pass at the head.

@thymikee

thymikee commented Nov 14, 2026 •

Copy link
Copy Markdown
Member Author

Pushed the requested fix at 13a4c1d9576777989ab3849cda9fb78661a33988, with #4551 restacked at 024cb04a4d38de644148233ae79108191a28507d. Lower starts the response-header phase from connected request finish; pipeline completion only records settlement for cleanup/join. It cannot start or reset the header timer. No timeout, worker limit or gate selection changed.

The existing 16 MiB / five-progress-window case keeps every earlier assertion. It holds the upload source's actual close for 20 s on the fake clock, verifies the fd size, waits for the real close callback, and checks request destruction at 30,000 ms independently of the outcome join. With the same phase-file command:

Production counterfactual Result
Published 77a5 bytes 1 failed, 4 passed
Remove only finish arming 1 failed, 4 passed
Add a late pipeline rearm 1 failed, 4 passed

Each fails that request-destruction assertion, not the 5 s wall timeout. Restored controls pass 5/5. Cold independent review found no findings. Separate exact gates passed with full-unit fallback: lower 19,841 and upper 19,847 unit tests (10 skipped each), 376 related tests each, nine checks passed/seven CI-owned or manual checks skipped. Sizes are 2404/699/2497 gross. Duplicate lower constant/field comments were removed without changing the header contract.

The optional exported outcome predicate remains deferred; the upper attempt now centralizes its own abort-reason mapping, while the four control-request gates and eight discriminating rows remain. The process-fixture isolation fix stays canonical in open #4544; it was not duplicated. Current-head Linux Coverage now passes: job 132601152802 reports 20,190 tests passed and 70 skipped, plus 11 fuzz tests passed. All required checks are green; one matrix check is skipped. The 77a5 failure and historical receipts remain failed at their original heads; actual CI close latency is still inferred, not proven. No CI rerun or merge.

Handoff note: no review threads are open. Sentinel approves the incremental review but mentions one undisclosed off-diff candidate; the check exposes no actionable details or annotations. That remains an explicit review limitation for the maintainer, rather than a claimed full-repository clearance. No merge.

@thymikee

Copy link
Copy Markdown
Member Author

I reviewed 13a4c1d and found no problems in the code. The header phase now arms on the request finish event, and no timeout value changed. This fixes the Linux flake in the header-budget test that I raised at 77a5e16.

All 20 check runs at 13a4c1d are success or skipped. That includes Coverage, and the two Coverage failures from 77a5e16 (the header budget test and host-process-liveness.test.ts) did not recur. I did not run the tests locally. I reasoned about the new destroyed-at-30s assertion from the code and did not execute it against 77a5e16. The Coverage log shows only totals (2194 files passed, 0 failed), so I infer that upload-stream-phases.test.ts passed from the zero-failure count. There are no conflicts.

The earlier review threads need no further code change. Please resolve them: header budget test (fixed at this head), lifecycle test, lifecycle test, outgoing writes fixture, upload client and control request expiry are all fixed. The ledger rationale thread is benign, because the cross-origin redirect refusal is unchanged on the wire.

Nothing else is needed before merge except human review.

@thymikee thymikee added the ready-for-human Valid work that needs human implementation, judgment, or maintainer merge label Nov 14, 2026
@thymikee
thymikee merged commit d66323e into main Nov 15, 2026
20 checks passed
@thymikee
thymikee deleted the t3/audit4441/remote-budgets branch November 15, 2026 07:52
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

ready-for-human Valid work that needs human implementation, judgment, or maintainer merge

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant