Repository navigation
fix(remote): bound artifact preparation and upload phases - #4552
Conversation
…on aggregate A single five-minute timer covered a whole upload, so an artifact that kept moving and one that stalled after the first chunk were cut off the same way, and nothing said which part of the hop ended the attempt. Each attempt now runs one sequential watchdog: 30s to connect, 120s of silence between body bytes in either direction, and 30s for response headers once the outgoing body is flushed. Only bytes actually written or read re-arm a body phase, and arming a later phase stops counting the earlier one. All of it stays inside the 600s preparation aggregate from the previous commit, which every attempt, redirect, offset resume, fallback and finalize still shares, so the earliest deadline keeps winning and no resumed work can outlive it. A phase expiry is typed evidence carrying the phase and its effective budget. It is deliberately not retryable: restarting the sequence would buy a second attempt permission the budget already refused, so neither the ticket retry nor the legacy fallback restarts those bytes. Caller cancellation stays a separate account and still answers with the signal's own reason. The two control-plane calls that bracket an upload, preflight and finalize, carry one 30s ceiling each: their payload is a few hundred bytes of JSON the transport writes with the request, so there is no separate body phase to time.
…hases Two corrections to the phase budgeting slice, both found in review. An early response resolved, redirected or resumed while the attempt's own file → transform → request pipeline was still writing into the old hop. The answer was read while the owned file handle stayed open and the queued body kept flowing at a peer this client had already left behind. The attempt now reads the accepted answer to its end, destroys its own request only when that read completed the exchange's verdict, and joins the pipeline before resolving, starting a redirect, or starting an offset resume. The enclosing preparation aggregate still bounds what lingered, but no attempt now hands its body to the next hop. The connect ceiling ended when the request accepted its first write and body progress counted chunks read ahead of the socket. A TLS origin that never answers the handshake, or a peer that stops taking bytes after the headers, was timed against the wrong budget, and bytes sitting in the write queue re-armed a body phase that had not moved. The ceiling now ends on the events that carry the protocol: socket `connect` for plain HTTP, `secureConnect` for TLS, immediately for a pooled keep-alive socket. Body progress is the transport's own `drain` and nothing else; the progress transform's byte hook, which fired before the chunk ever reached the request, is gone. Wire bytes are unchanged, so the ledger's acknowledged digest for the attempt moves with the rationale that says so. The progress-window fixture measured budget against wall-clock scheduling, so a window could be charged before the parked peer really took a byte and the "keeps moving" case flaked. Each release now lets one quota of bytes through and the window waits with real event-loop turns until those bytes arrive before it charges the budget, which removes the race and costs the whole file a third of a second instead of two.
…gate in upload-stream The phase watchdog and the 600-second preparation aggregate lived in two new modules that every daemon-client entry now evaluated eagerly. upload-stream.ts is already in that closure and is the lowest module all three consumers reach, so it now owns both. withUploadPreparationDeadline still starts its timer synchronously when called, and isTransferPhaseTimeoutError stays a synchronous typed check. Rename the transfer tests for the module they exercise (R84). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…mpt wire digest 6089d90 moved the attempt's watchdog.dispose() ahead of the redirect/resume decision without refreshing the ledger. The digest pasted here is the one wire-compat.test.ts prints for the current source; the rationale now names the retired watchdog. Requests on the wire are unchanged. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…private to upload-stream No module outside upload-stream reads them; consumers classify through isTransferPhaseTimeoutError. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Size Report
Startup median (7 runs, lower is better):
|
There was a problem hiding this comment.
All reported issues were addressed across 10 files
You’re at about 91% of the monthly reviewed-line limit. You may want to disable incremental reviews to conserve quota. Reviews will continue until that limit is exceeded. If you need help avoiding interruptions, please contact contact@cubic.dev.
Reply with feedback, questions, or to request a fix.
View guided diff | Turn on auto-fix | Re-trigger cubic
Apex Sentinel · ✅ ApproveSince the last review: 0 new · 0 still open · 0 no longer raised No open findings. 1 finding below 💬 Talking to SentinelIn a finding's thread, reply with:
On this conversation, comment Earlier reviewsReview of
|
|
At 09e1e76, the response reader in the upload stream is a copy of the shared one, and that needs to change before merge. I ran no tests for this review.
Not blocking, and you can take or leave these: three nested Could one aggregate be armed at One inline thread still needs a fix: loopback progress test. Four lower-priority threads on test cleanup, the fixture's throw inside a request listener, and the unreachable timeout hint also still apply. This thread does not apply, so please resolve it: #4552 (comment) (wire-compat acks are keyed per digest, and the cross-origin refusal is unchanged on the wire). The Coverage check failed in its |
|
Accepted: the response reader now has an optional I am retaining the two independently callable admission boundaries ( The loopback header-wait case is being corrected to establish actual progress beyond four connect ceilings before the header wait. Fresh source review, discrimination and one exact gate per final head precede the single push. I reran the failed Coverage job once as requested and retained the original truncated log; no cause or pass is inferred yet. The wire-ack thread is not a wire defect and will receive that disposition. |
|
The single requested Coverage rerun also failed at the unchanged published head. Its text log is still truncated, but the check-run annotations identify 3 failures in this PR's remote loopback tests: progress/stall phase cases and the pooled-connection case. The stalled-body case hits the existing 5000 ms test limit; the other failures are assertions. These are captured candidate-test failures, not a passing rerun or an infrastructure diagnosis. The existing source owner now has the exact annotations and is reconciling real I/O with the controlled clock while making the already-assigned write-progress and fixture corrections. No additional worker, timeout increase, test deletion or CI retry is being started. No moved host-kit test is named in these annotations, but that alone does not establish its successful execution; final-head Coverage remains required. Original and rerun failures retain their heads and logs. |
Request-body progress came only from `drain`, which fires after a write returned false. A slow producer whose writes stay under the socket's high-water mark never sees a drain, so a progressing upload expired its 120-second body budget. The attempt now wraps the request's `write` and touches the body phase when a non-empty write completes without error: its bytes reached the local kernel. Arguments, `this`, the backpressure result and the caller's callback pass through unchanged, and completions after the response, an abort, a destroy or the attempt's end do not count. streamFileToHttpRequest no longer opens a second copy of the preparation aggregate: its only callers run inside uploadArtifact's, so it requires that signal and drops the timeout message and hint the outer deadline always preempted. uploadArtifact and prepareRemoteRequestArtifacts keep their own 600-second admission boundaries. Tests: a real loopback slow producer with 1 KiB writes survives six 30-second windows with zero drains and then expires one ceiling after its last completed write; the header-wait case now proves the body is still in flight past four connect ceilings before only the header wait expires; a cancel during the retried direct upload announces no fallback. Loopback progress windows gate on completed writes. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…tures on owned I/O Host-kit's transport owns observeCompletedRequestWrites: it reports each non-empty write Node completes without an error, encoded-empty strings excluded, and stops after the response, an abort, a destroy or disposal. The remote upload attempt consumes it instead of a private copy, and the generic contract has direct tests beside its owner. The loopback fixtures charge fake time only after the owned I/O a phase needs was observed, tie pending waits to the test lifecycle, assert the stalled body is unfinished, and prove the early-answer join with a producer that never ends instead of a host-speed round bound. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
… digests Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Each loopback server, held-open producer and trusted certificate is released when its test finishes, so a failed assertion or a timeout cannot leave one running into the next test. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
An ended transfer still gets at most 20 s of budget, less than any phase ceiling, and then waits for its settlement event, so a slow host's real teardown I/O cannot fail the test while the test timeout stays the bound. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
|
Addressed the review at
Ticket policy: the current daemon starts a five-minute timer at ticket creation and refreshes it after a completed chunk append, not per byte. A single slow chunk can still expire while the client's absolute 600 s lifecycle remains valid. Adopted per-byte refresh remains separate daemon work. This stack does not extend the server ticket or claim end-to-end success beyond that limit. One pinned pnpm 11.17.0 The old Coverage run and its single captured rerun remain failed. The rerun annotations identify three remote loopback failures; the truncated log alone could not. Revised fixtures wait for real I/O before charging virtual time. Modeled/macOS evidence does not establish the historical Linux cause. Corrected-head Linux Coverage is now pending and will be followed through the PR watch. No additional blind retry was made. The unused download clause remains an optional comment cleanup, not a claim that downloads use these budgets. #4441 remains open for its remaining slices. |
There was a problem hiding this comment.
All reported issues were addressed across 13 files (changes from recent commits).
You’re at about 92% of the monthly reviewed-line limit. You may want to disable incremental reviews to conserve quota. Reviews will continue until that limit is exceeded. If you need help avoiding interruptions, please contact contact@cubic.dev.
Reply with feedback, questions, or to request a fix.
View guided diff | Turn on auto-fix | Re-trigger cubic
|
I reviewed 22cf1d4 against the findings in my earlier review. The earlier findings are fixed, and I found no new problems that must change before merge. Not blocking, and you can take or leave it: observeCompletedRequestWrites in transport.ts classifies string chunks by encoding and detects Uint8Array across realms, but its only caller, upload-stream.ts:280, writes only Buffers through pipeline, so those branches cannot run on a real route. Could upload-stream.ts own the writes instead, since it owns the producer? A loop like On the open threads: the one on control-plane timeouts falling back to legacy still applies, and it predates this PR. These look done, so please resolve them: the 16 MiB phase-budget test now covers five windows; CA trust restore is registered at creation; loopback server close is registered at creation; the unexpected-leg handler now answers 500 instead of throwing; timeoutMessage and timeoutHint are removed. The redirect and ledger thread does not apply, because the wire behavior is unchanged. Smoke Tests (ios) failed in the swiftc compile of AgentDeviceRunnerUITests. This diff touches no Apple or Swift files, so that failure looks unrelated. The Linux Coverage (unit-ci) run is unclear: the log I had stops in the react-native companion tests and names no failed test, and this PR changes four remote loopback test files that the job runs. I did not run the new tests or your counterfactual mutants, and I did not trace the daemon ticket refresh policy. Before merge, please share a passing Linux Coverage run at 22cf1d4, or the names of its failing tests. The control-plane timeout thread also needs your planned typed fix or an explicit decision. |
|
The failed Linux run at
These are from the Coverage job annotations; the truncated text log alone does not identify them. Their Linux cause remains unproven. The upper run being green does not make this lower run a pass. No further CI rerun was requested. For thread 4575165528, I am taking the typed fix, not adding an exception. Decision 6827686874 §6 keeps the client’s own timeout terminal. The local correction reuses the phase watchdog for preflight/finalize, preserves the earlier caller/aggregate reason, consumes successful JSON under the same 30s control lifetime and disposes it. Error statuses keep their prior immediate handling. Four initial/retried timeout cases, two identical-prose negatives and two mid-stall caller-cancellation cases cover the boundary. The shared sending owner is now wire-listed and has a precise headers/body input; POST and redirect refusal are fixed at construction, without a closure waiver. New local heads are lower A producer-owned write loop is a valid alternative if it also owns write completion, close/error/abort races, early-response delivery and join-before-next-hop cleanup. I am deferring that optional ownership rewrite in this round and retaining the tested pipeline/observer composition. The route currently produces Buffers; I make no claim that its string/realm branches run on this route. The iOS compile cause also remains unproven. No merge. |
|
Pushed the typed correction and bounded I/O diagnostics in 77a5e16; #4551 is restacked at af5c634. No timeout-policy exception: preflight/finalize expiry remains terminal, including after retry, and cancellation preserves the caller's reason. The sending helper fixes POST/redirect refusal, is included in the wire manifest, and skips parsing error-status bodies to preserve their previous handling. Independent source review and the final count-simplification recheck found no findings. Lower/upper focused controls passed 121/127; all four control-contract counterfactuals were detected. Separate exact gates passed: 19,841/19,847 unit tests (10 skipped each), 376 related tests each, wire compatibility and all other local selected checks; seven CI-owned/manual lanes were skipped. The d331 Fallow failure remains failed; only test expectation branching was simplified, with all eight rows/assertions retained. The three failing Linux unit-ci test names at 22cf remain those in my previous reply: header-budget-after-progress, zero-length header wait and reused connection. Their 5000 ms timeout causes and the iOS compile cause are unproven. This push schedules new-head CI after substantive corrections; I did not retry the failed jobs. Diagnostics now report the stuck I/O condition and state within each test's declared budget. The optional producer-owned write loop remains deferred for this round, for the ownership reasons in the previous reply. The current route produces Buffers. Current daemon ticket refresh is only after completed chunk append; adopted per-byte refresh remains separate. No merge. |
|
Coverage failed at 77a5e16 in job132579427587 (run43638146821, Linux Node24.13.1). The full captured log reports20188 tests passed,2 failed,70 skipped. The failure is in the test run; changed-line/CRAP enforcement did not run.
The exact local77a5/af5c passes remain local evidence only. The77a5 Coverage job remains FAILED. Both PR watches stay active; no rewatch, new push or rerun. No merge. |
|
At 77a5e16 the code looks good, but the Linux Coverage check still fails, so I cannot call it ready yet. The earlier findings from 22cf1d4 (#4552 (comment)) are fixed, and I found no new problem in the code. The Coverage (unit-core) run fails on two tests. The first is the PR's own header-budget test, upload-stream-phases.test.ts:112. It timed out at 22cf1d4 too, so it is not a one-off. The test treats the body as flushed on request 'finish' and then advances the fake clock 30 s. Production arms the 'response-headers' phase only after the whole pipeline resolves (upload-stream.ts:288-292). That pipeline also waits for the file stream to close, which can land after the test has spent its 30 s on a loaded runner. The header timer then starts late and the test never settles, which matches the annotation (all 16 MiB received, finished and body ended, not settled). The rule to keep: the header phase must start at the same event the test and the constant's doc call flush. The smallest fix is to mark the body flushed and advance the phase on request 'finish', and keep the pipeline promise only for the join. Please do not widen any timeout. To validate, the Linux Coverage job should pass this test at the new head. I inferred this mechanism from Node's stream semantics and the CI snapshot. I did not reproduce it. The second failure, host-process-liveness.test.ts:182, looks unrelated: this diff does not touch process liveness, and the failure is a mocked PID resolving against a real Linux /proc. I did not run the base to confirm it also fails on main, so it needs #4544 or a rebase onto it. Not blocking, and you can take or leave it: the four fallback and retry gates in upload-client.ts each repeat "caller or aggregate aborted, or phase timeout, then rethrow". Would one exported predicate beside isTransferPhaseTimeoutError, for example isOwnUploadOutcome(error, signal), make that rule one symbol so the next fallback site cannot miss half of it? The delta is otherwise a net simplification, since one control-request owner replaces two fetch blocks and combineUploadSignals. On the other threads: the thread on control-plane expiry reaching the legacy fallback (#4552 (comment)) is fixed at this head, and so is the thread on the 16 MiB budget test (#4552 (comment)), whose remaining flake is the Linux issue above. The minor threads on #4552 (comment), #4552 (comment), #4552 (comment) and #4552 (comment) are fixed at this head, so they can be resolved. The thread on the cross-origin redirect (#4552 (comment)) does not apply: the refusal is unchanged on the wire, so it can be resolved too. I did not run any tests, and I did not re-review the daemon ticket-refresh policy, which you scoped out of this PR. There are no conflicts. Before merge, Linux Coverage must pass at the head. |
|
Pushed the requested fix at The existing 16 MiB / five-progress-window case keeps every earlier assertion. It holds the upload source's actual close for 20 s on the fake clock, verifies the fd size, waits for the real close callback, and checks request destruction at 30,000 ms independently of the outcome join. With the same phase-file command:
Each fails that request-destruction assertion, not the 5 s wall timeout. Restored controls pass 5/5. Cold independent review found no findings. Separate exact gates passed with full-unit fallback: lower 19,841 and upper 19,847 unit tests (10 skipped each), 376 related tests each, nine checks passed/seven CI-owned or manual checks skipped. Sizes are 2404/699/2497 gross. Duplicate lower constant/field comments were removed without changing the header contract. The optional exported outcome predicate remains deferred; the upper attempt now centralizes its own abort-reason mapping, while the four control-request gates and eight discriminating rows remain. The process-fixture isolation fix stays canonical in open #4544; it was not duplicated. Current-head Linux Coverage now passes: job 132601152802 reports 20,190 tests passed and 70 skipped, plus 11 fuzz tests passed. All required checks are green; one matrix check is skipped. The 77a5 failure and historical receipts remain failed at their original heads; actual CI close latency is still inferred, not proven. No CI rerun or merge. Handoff note: no review threads are open. Sentinel approves the incremental review but mentions one undisclosed off-diff candidate; the check exposes no actionable details or annotations. That remains an explicit review limitation for the maintainer, rather than a claimed full-repository clearance. No merge. |
|
I reviewed 13a4c1d and found no problems in the code. The header phase now arms on the request All 20 check runs at 13a4c1d are success or skipped. That includes Coverage, and the two Coverage failures from 77a5e16 (the header budget test and The earlier review threads need no further code change. Please resolve them: header budget test (fixed at this head), lifecycle test, lifecycle test, outgoing writes fixture, upload client and control request expiry are all fixed. The ledger rationale thread is benign, because the cross-origin redirect refusal is unchanged on the wire. Nothing else is needed before merge except human review. |
Summary
Remote artifact uploads share a finite 600 s lifecycle across preparation, retries, resume and fallback. Connect and post-flush headers have 30 s ceilings; body inactivity has 120 s. Headers now start at connected request finish, independently of file-close/pipeline cleanup. Typed phase expiry is terminal and caller cancellation preserves its reason.
This changes 17 files. Host-kit owns completed nonempty writes and response reading; the remote reader copy is removed. The header regression retains its progress premises and assertions while pinning the held source fd and real close completion. Duplicate lower phase comments were removed; the header contract remains. Refs #4441. Proxy, daemon and downloads remain separate.
Validation
Tested
13a4c1d9576777989ab3849cda9fb78661a33988, based78be6c8.Pinned pnpm 11.17.0
check:affected --run --head <sha> --jsonpassed once: exactCommit=true, checkoutChanged=false, failOpen=false. Nine checks passed, including wire compatibility, 376 related tests and 19,841 unit tests (10 skipped). Seven CI-owned/manual checks were skipped; #4050 full-unit fallback remains.Cold review found no findings. Focused baseline passed 121 tests. Published 77a5 production, missing finish arming and a late pipeline rearm each fail the same 30,000 ms request-destruction assertion; restored controls pass.
Linux Coverage and Integration passed at this head; all required checks are green (one matrix check skipped). Earlier failures retain their attribution; actual CI close latency remains inferred. The separate
/procfixture correction is in open #4544. Per-byte daemon ticket refresh, Node-floor, Bun and end-to-end slow-transfer evidence remain open.