Repository navigation
Commit d66323e
fix(remote): bound artifact preparation and upload phases (#4552)
* fix(remote): bound artifact preparation and propagate cancellation
* fix(remote): budget each artifact-transfer phase inside the preparation aggregate
A single five-minute timer covered a whole upload, so an artifact that kept
moving and one that stalled after the first chunk were cut off the same way,
and nothing said which part of the hop ended the attempt.
Each attempt now runs one sequential watchdog: 30s to connect, 120s of silence
between body bytes in either direction, and 30s for response headers once the
outgoing body is flushed. Only bytes actually written or read re-arm a body
phase, and arming a later phase stops counting the earlier one. All of it stays
inside the 600s preparation aggregate from the previous commit, which every
attempt, redirect, offset resume, fallback and finalize still shares, so the
earliest deadline keeps winning and no resumed work can outlive it.
A phase expiry is typed evidence carrying the phase and its effective budget. It
is deliberately not retryable: restarting the sequence would buy a second
attempt permission the budget already refused, so neither the ticket retry nor
the legacy fallback restarts those bytes. Caller cancellation stays a separate
account and still answers with the signal's own reason.
The two control-plane calls that bracket an upload, preflight and finalize,
carry one 30s ceiling each: their payload is a few hundred bytes of JSON the
transport writes with the request, so there is no separate body phase to time.
* fix(remote): end each upload attempt's owned work and time its real phases
Two corrections to the phase budgeting slice, both found in review.
An early response resolved, redirected or resumed while the attempt's own file
→ transform → request pipeline was still writing into the old hop. The answer
was read while the owned file handle stayed open and the queued body kept
flowing at a peer this client had already left behind. The attempt now reads
the accepted answer to its end, destroys its own request only when that read
completed the exchange's verdict, and joins the pipeline before resolving,
starting a redirect, or starting an offset resume. The enclosing preparation
aggregate still bounds what lingered, but no attempt now hands its body to the
next hop.
The connect ceiling ended when the request accepted its first write and body
progress counted chunks read ahead of the socket. A TLS origin that never
answers the handshake, or a peer that stops taking bytes after the headers,
was timed against the wrong budget, and bytes sitting in the write queue
re-armed a body phase that had not moved. The ceiling now ends on the events
that carry the protocol: socket `connect` for plain HTTP, `secureConnect` for
TLS, immediately for a pooled keep-alive socket. Body progress is the
transport's own `drain` and nothing else; the progress transform's byte hook,
which fired before the chunk ever reached the request, is gone.
Wire bytes are unchanged, so the ledger's acknowledged digest for the attempt
moves with the rationale that says so.
The progress-window fixture measured budget against wall-clock scheduling, so
a window could be charged before the parked peer really took a byte and the
"keeps moving" case flaked. Each release now lets one quota of bytes through
and the window waits with real event-loop turns until those bytes arrive
before it charges the budget, which removes the race and costs the whole file
a third of a second instead of two.
* fix(remote): retire completed upload hop budgets before resuming
* test(remote): preserve retryable failures across resumed upload hops
* test(remote): clarify resumed upload progress windows
* refactor(remote): keep the transfer phase owner and preparation aggregate in upload-stream
The phase watchdog and the 600-second preparation aggregate lived in two new
modules that every daemon-client entry now evaluated eagerly. upload-stream.ts
is already in that closure and is the lowest module all three consumers reach,
so it now owns both. withUploadPreparationDeadline still starts its timer
synchronously when called, and isTransferPhaseTimeoutError stays a synchronous
typed check.
Rename the transfer tests for the module they exercise (R84).
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* chore(gates): acknowledge the retired hop watchdog in the upload attempt wire digest
6089d90 moved the attempt's watchdog.dispose() ahead of the redirect/resume
decision without refreshing the ledger. The digest pasted here is the one
wire-compat.test.ts prints for the current source; the rationale now names
the retired watchdog. Requests on the wire are unchanged.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* refactor(remote): keep the phase-expiry reason and error constructor private to upload-stream
No module outside upload-stream reads them; consumers classify through
isTransferPhaseTimeoutError.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* fix(remote): reuse shared response reader and tighten review guards
* fix(remote): count completed body writes as upload progress
Request-body progress came only from `drain`, which fires after a write
returned false. A slow producer whose writes stay under the socket's
high-water mark never sees a drain, so a progressing upload expired its
120-second body budget. The attempt now wraps the request's `write` and
touches the body phase when a non-empty write completes without error:
its bytes reached the local kernel. Arguments, `this`, the backpressure
result and the caller's callback pass through unchanged, and completions
after the response, an abort, a destroy or the attempt's end do not count.
streamFileToHttpRequest no longer opens a second copy of the preparation
aggregate: its only callers run inside uploadArtifact's, so it requires
that signal and drops the timeout message and hint the outer deadline
always preempted. uploadArtifact and prepareRemoteRequestArtifacts keep
their own 600-second admission boundaries.
Tests: a real loopback slow producer with 1 KiB writes survives six
30-second windows with zero drains and then expires one ceiling after its
last completed write; the header-wait case now proves the body is still
in flight past four connect ceilings before only the header wait
expires; a cancel during the retried direct upload announces no
fallback. Loopback progress windows gate on completed writes.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* fix(transport): own completed request writes in host-kit and time fixtures on owned I/O
Host-kit's transport owns observeCompletedRequestWrites: it reports each
non-empty write Node completes without an error, encoded-empty strings
excluded, and stops after the response, an abort, a destroy or disposal.
The remote upload attempt consumes it instead of a private copy, and the
generic contract has direct tests beside its owner.
The loopback fixtures charge fake time only after the owned I/O a phase
needs was observed, tie pending waits to the test lifecycle, assert the
stalled body is unfinished, and prove the early-answer join with a
producer that never ends instead of a host-speed round bound.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* chore(gates): acknowledge the host-kit write owner in the upload wire digests
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* test(remote): tie loopback fixture cleanup to the test lifecycle
Each loopback server, held-open producer and trusted certificate is
released when its test finishes, so a failed assertion or a timeout
cannot leave one running into the next test.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* test(remote): end fixture teardown waits on settlement, not loop turns
An ended transfer still gets at most 20 s of budget, less than any phase
ceiling, and then waits for its settlement event, so a slow host's real
teardown I/O cannot fail the test while the test timeout stays the bound.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* fix(transport): make remaining progress tests independent of scheduling
* fix(remote): stop control request expiry before upload fallback
* refactor(test): simplify shared upload fixture expectations
* fix(test): preserve declared upload budgets and caller cancellation proof
* chore(gates): track the shared upload control request owner
* refactor(remote): constrain control requests at their sending owner
* chore(gates): acknowledge fixed upload control request construction
* test(remote): simplify control request count expectations
* fix(remote): start the header deadline at request finish
* chore(gates): acknowledge request-finish header timing
* test(remote): pin the upload source close before advancing time
* test(remote): narrow the held-close callback and descriptor
---------
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>1 parent d78be6c commit d66323e
17 files changed
Lines changed: 2196 additions & 208 deletions
File tree
- packages
- host-kit/src
- internal
- remote/src
- __tests__
- support
- test/wire-compat
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
| 1 | + | |
| 2 | + | |
| 3 | + | |
| 4 | + | |
| 5 | + | |
| 6 | + | |
| 7 | + | |
| 8 | + | |
| 9 | + | |
| 10 | + | |
| 11 | + | |
| 12 | + | |
| 13 | + | |
| 14 | + | |
| 15 | + | |
| 16 | + | |
| 17 | + | |
| 18 | + | |
| 19 | + | |
| 20 | + | |
| 21 | + | |
| 22 | + | |
| 23 | + | |
| 24 | + | |
| 25 | + | |
| 26 | + | |
| 27 | + | |
| 28 | + | |
| 29 | + | |
| 30 | + | |
| 31 | + | |
| 32 | + | |
| 33 | + | |
| 34 | + | |
| 35 | + | |
| 36 | + | |
| 37 | + | |
| 38 | + | |
| 39 | + | |
| 40 | + | |
| 41 | + | |
| 42 | + | |
| 43 | + | |
| 44 | + | |
| 45 | + | |
| 46 | + | |
| 47 | + | |
| 48 | + | |
| 49 | + | |
| 50 | + | |
| 51 | + | |
| 52 | + | |
| 53 | + | |
| 54 | + | |
| 55 | + | |
| 56 | + | |
| 57 | + | |
| 58 | + | |
| 59 | + | |
| 60 | + | |
| 61 | + | |
| 62 | + | |
| 63 | + | |
| 64 | + | |
| 65 | + | |
| 66 | + | |
| 67 | + | |
| 68 | + | |
| 69 | + | |
| 70 | + | |
| 71 | + | |
| 72 | + | |
| 73 | + | |
| 74 | + | |
| 75 | + | |
| 76 | + | |
| 77 | + | |
| 78 | + | |
| 79 | + | |
| 80 | + | |
| 81 | + | |
| 82 | + | |
| 83 | + | |
| 84 | + | |
| 85 | + | |
| 86 | + | |
| 87 | + | |
| 88 | + | |
| 89 | + | |
| 90 | + | |
| 91 | + | |
| 92 | + | |
| 93 | + | |
| 94 | + | |
| 95 | + | |
| 96 | + | |
| 97 | + | |
| 98 | + | |
| 99 | + | |
| 100 | + | |
| 101 | + | |
| 102 | + | |
| 103 | + | |
| 104 | + | |
| 105 | + | |
| 106 | + | |
| 107 | + | |
| 108 | + | |
| 109 | + | |
| 110 | + | |
| 111 | + | |
| 112 | + | |
| 113 | + | |
| 114 | + | |
| 115 | + | |
| 116 | + | |
| 117 | + | |
| 118 | + | |
| 119 | + | |
| 120 | + | |
| 121 | + | |
| 122 | + | |
| 123 | + | |
| 124 | + | |
| 125 | + | |
| 126 | + | |
| 127 | + | |
| 128 | + | |
| 129 | + | |
| 130 | + | |
| 131 | + | |
| 132 | + | |
| 133 | + | |
| 134 | + | |
| 135 | + | |
| 136 | + | |
| 137 | + | |
| 138 | + | |
| 139 | + | |
| 140 | + | |
| 141 | + | |
| 142 | + | |
| 143 | + | |
| 144 | + | |
| 145 | + | |
| 146 | + | |
| 147 | + | |
| 148 | + | |
| 149 | + | |
| 150 | + | |
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
44 | 44 | | |
45 | 45 | | |
46 | 46 | | |
| 47 | + | |
| 48 | + | |
| 49 | + | |
| 50 | + | |
| 51 | + | |
| 52 | + | |
| 53 | + | |
47 | 54 | | |
48 | 55 | | |
49 | 56 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
1 | 1 | | |
2 | | - | |
| 2 | + | |
| 3 | + | |
3 | 4 | | |
4 | 5 | | |
5 | 6 | | |
| |||
42 | 43 | | |
43 | 44 | | |
44 | 45 | | |
45 | | - | |
| 46 | + | |
| 47 | + | |
| 48 | + | |
| 49 | + | |
46 | 50 | | |
47 | 51 | | |
48 | 52 | | |
49 | | - | |
| 53 | + | |
| 54 | + | |
50 | 55 | | |
51 | 56 | | |
52 | 57 | | |
| |||
82 | 87 | | |
83 | 88 | | |
84 | 89 | | |
| 90 | + | |
| 91 | + | |
| 92 | + | |
| 93 | + | |
| 94 | + | |
| 95 | + | |
| 96 | + | |
| 97 | + | |
| 98 | + | |
| 99 | + | |
| 100 | + | |
| 101 | + | |
| 102 | + | |
| 103 | + | |
| 104 | + | |
| 105 | + | |
| 106 | + | |
| 107 | + | |
| 108 | + | |
| 109 | + | |
| 110 | + | |
| 111 | + | |
| 112 | + | |
| 113 | + | |
| 114 | + | |
| 115 | + | |
| 116 | + | |
| 117 | + | |
| 118 | + | |
| 119 | + | |
| 120 | + | |
| 121 | + | |
| 122 | + | |
| 123 | + | |
| 124 | + | |
| 125 | + | |
| 126 | + | |
| 127 | + | |
| 128 | + | |
| 129 | + | |
| 130 | + | |
| 131 | + | |
| 132 | + | |
| 133 | + | |
| 134 | + | |
| 135 | + | |
| 136 | + | |
| 137 | + | |
| 138 | + | |
| 139 | + | |
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
1 | 1 | | |
2 | 2 | | |
3 | 3 | | |
| 4 | + | |
4 | 5 | | |
5 | 6 | | |
6 | 7 | | |
| |||
0 commit comments