Skip to content

Commit d66323e

Browse files
thymikeeclaude
andauthored
fix(remote): bound artifact preparation and upload phases (#4552)
* fix(remote): bound artifact preparation and propagate cancellation * fix(remote): budget each artifact-transfer phase inside the preparation aggregate A single five-minute timer covered a whole upload, so an artifact that kept moving and one that stalled after the first chunk were cut off the same way, and nothing said which part of the hop ended the attempt. Each attempt now runs one sequential watchdog: 30s to connect, 120s of silence between body bytes in either direction, and 30s for response headers once the outgoing body is flushed. Only bytes actually written or read re-arm a body phase, and arming a later phase stops counting the earlier one. All of it stays inside the 600s preparation aggregate from the previous commit, which every attempt, redirect, offset resume, fallback and finalize still shares, so the earliest deadline keeps winning and no resumed work can outlive it. A phase expiry is typed evidence carrying the phase and its effective budget. It is deliberately not retryable: restarting the sequence would buy a second attempt permission the budget already refused, so neither the ticket retry nor the legacy fallback restarts those bytes. Caller cancellation stays a separate account and still answers with the signal's own reason. The two control-plane calls that bracket an upload, preflight and finalize, carry one 30s ceiling each: their payload is a few hundred bytes of JSON the transport writes with the request, so there is no separate body phase to time. * fix(remote): end each upload attempt's owned work and time its real phases Two corrections to the phase budgeting slice, both found in review. An early response resolved, redirected or resumed while the attempt's own file → transform → request pipeline was still writing into the old hop. The answer was read while the owned file handle stayed open and the queued body kept flowing at a peer this client had already left behind. The attempt now reads the accepted answer to its end, destroys its own request only when that read completed the exchange's verdict, and joins the pipeline before resolving, starting a redirect, or starting an offset resume. The enclosing preparation aggregate still bounds what lingered, but no attempt now hands its body to the next hop. The connect ceiling ended when the request accepted its first write and body progress counted chunks read ahead of the socket. A TLS origin that never answers the handshake, or a peer that stops taking bytes after the headers, was timed against the wrong budget, and bytes sitting in the write queue re-armed a body phase that had not moved. The ceiling now ends on the events that carry the protocol: socket `connect` for plain HTTP, `secureConnect` for TLS, immediately for a pooled keep-alive socket. Body progress is the transport's own `drain` and nothing else; the progress transform's byte hook, which fired before the chunk ever reached the request, is gone. Wire bytes are unchanged, so the ledger's acknowledged digest for the attempt moves with the rationale that says so. The progress-window fixture measured budget against wall-clock scheduling, so a window could be charged before the parked peer really took a byte and the "keeps moving" case flaked. Each release now lets one quota of bytes through and the window waits with real event-loop turns until those bytes arrive before it charges the budget, which removes the race and costs the whole file a third of a second instead of two. * fix(remote): retire completed upload hop budgets before resuming * test(remote): preserve retryable failures across resumed upload hops * test(remote): clarify resumed upload progress windows * refactor(remote): keep the transfer phase owner and preparation aggregate in upload-stream The phase watchdog and the 600-second preparation aggregate lived in two new modules that every daemon-client entry now evaluated eagerly. upload-stream.ts is already in that closure and is the lowest module all three consumers reach, so it now owns both. withUploadPreparationDeadline still starts its timer synchronously when called, and isTransferPhaseTimeoutError stays a synchronous typed check. Rename the transfer tests for the module they exercise (R84). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * chore(gates): acknowledge the retired hop watchdog in the upload attempt wire digest 6089d90 moved the attempt's watchdog.dispose() ahead of the redirect/resume decision without refreshing the ledger. The digest pasted here is the one wire-compat.test.ts prints for the current source; the rationale now names the retired watchdog. Requests on the wire are unchanged. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * refactor(remote): keep the phase-expiry reason and error constructor private to upload-stream No module outside upload-stream reads them; consumers classify through isTransferPhaseTimeoutError. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * fix(remote): reuse shared response reader and tighten review guards * fix(remote): count completed body writes as upload progress Request-body progress came only from `drain`, which fires after a write returned false. A slow producer whose writes stay under the socket's high-water mark never sees a drain, so a progressing upload expired its 120-second body budget. The attempt now wraps the request's `write` and touches the body phase when a non-empty write completes without error: its bytes reached the local kernel. Arguments, `this`, the backpressure result and the caller's callback pass through unchanged, and completions after the response, an abort, a destroy or the attempt's end do not count. streamFileToHttpRequest no longer opens a second copy of the preparation aggregate: its only callers run inside uploadArtifact's, so it requires that signal and drops the timeout message and hint the outer deadline always preempted. uploadArtifact and prepareRemoteRequestArtifacts keep their own 600-second admission boundaries. Tests: a real loopback slow producer with 1 KiB writes survives six 30-second windows with zero drains and then expires one ceiling after its last completed write; the header-wait case now proves the body is still in flight past four connect ceilings before only the header wait expires; a cancel during the retried direct upload announces no fallback. Loopback progress windows gate on completed writes. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * fix(transport): own completed request writes in host-kit and time fixtures on owned I/O Host-kit's transport owns observeCompletedRequestWrites: it reports each non-empty write Node completes without an error, encoded-empty strings excluded, and stops after the response, an abort, a destroy or disposal. The remote upload attempt consumes it instead of a private copy, and the generic contract has direct tests beside its owner. The loopback fixtures charge fake time only after the owned I/O a phase needs was observed, tie pending waits to the test lifecycle, assert the stalled body is unfinished, and prove the early-answer join with a producer that never ends instead of a host-speed round bound. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * chore(gates): acknowledge the host-kit write owner in the upload wire digests Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * test(remote): tie loopback fixture cleanup to the test lifecycle Each loopback server, held-open producer and trusted certificate is released when its test finishes, so a failed assertion or a timeout cannot leave one running into the next test. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * test(remote): end fixture teardown waits on settlement, not loop turns An ended transfer still gets at most 20 s of budget, less than any phase ceiling, and then waits for its settlement event, so a slow host's real teardown I/O cannot fail the test while the test timeout stays the bound. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * fix(transport): make remaining progress tests independent of scheduling * fix(remote): stop control request expiry before upload fallback * refactor(test): simplify shared upload fixture expectations * fix(test): preserve declared upload budgets and caller cancellation proof * chore(gates): track the shared upload control request owner * refactor(remote): constrain control requests at their sending owner * chore(gates): acknowledge fixed upload control request construction * test(remote): simplify control request count expectations * fix(remote): start the header deadline at request finish * chore(gates): acknowledge request-finish header timing * test(remote): pin the upload source close before advancing time * test(remote): narrow the held-close callback and descriptor --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
1 parent d78be6c commit d66323e

17 files changed

Lines changed: 2196 additions & 208 deletions
Lines changed: 150 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,150 @@
1+
import assert from 'node:assert/strict';
2+
import { EventEmitter } from 'node:events';
3+
import { runInNewContext } from 'node:vm';
4+
import type { ClientRequest } from 'node:http';
5+
import { test } from 'vitest';
6+
import { observeCompletedRequestWrites } from '../transport.ts';
7+
8+
class FakeRequest extends EventEmitter {
9+
destroyed = false;
10+
accepts = true;
11+
readonly writes: { self: unknown; args: unknown[] }[] = [];
12+
write(this: FakeRequest, ...args: unknown[]): boolean {
13+
this.writes.push({ self: this, args });
14+
return this.accepts;
15+
}
16+
complete(index: number, error: Error | null = null): void {
17+
const callback = this.writes[index]?.args.find((arg) => typeof arg === 'function');
18+
assert.ok(typeof callback === 'function', `write ${index} has no callback`);
19+
callback(error);
20+
}
21+
}
22+
23+
type Observed = ReturnType<typeof observe>;
24+
25+
function observe(prepare?: (fake: FakeRequest) => void) {
26+
const fake = new FakeRequest();
27+
prepare?.(fake);
28+
const caller = new AbortController();
29+
const completed = { count: 0 };
30+
const req = fake as unknown as ClientRequest;
31+
const dispose = observeCompletedRequestWrites(req, caller.signal, () => (completed.count += 1));
32+
return { fake, req, caller, completed, dispose };
33+
}
34+
35+
test('only a non-empty write that completes without an error is reported', () => {
36+
const { fake, req, completed, dispose } = observe();
37+
req.write(Buffer.from('queued'));
38+
req.write('', () => {});
39+
fake.complete(1);
40+
req.write(new Uint8Array(0), () => {});
41+
fake.complete(2);
42+
req.write('failed', () => {});
43+
fake.complete(3, new Error('EPIPE'));
44+
for (const [text, encoding] of [
45+
['a', 'hex'],
46+
['zz', 'hex'],
47+
['zz', 'HEX'],
48+
['=', 'base64'],
49+
['=', 'base64url'],
50+
] as const) {
51+
req.write(text, encoding as BufferEncoding, () => {});
52+
fake.complete(fake.writes.length - 1);
53+
}
54+
assert.equal(
55+
completed.count,
56+
0,
57+
'a queued, empty, failed or encoded-empty write is not progress',
58+
);
59+
fake.complete(0);
60+
assert.equal(completed.count, 1);
61+
dispose();
62+
});
63+
64+
test.each([
65+
['utf8', 'é'],
66+
['hex', 'ff'],
67+
['base64', 'YQ=='],
68+
['base64url', 'YQ'],
69+
] as const)('a successful %s string write is reported', (encoding, text) => {
70+
const { fake, req, completed, dispose } = observe();
71+
req.write(text, encoding);
72+
fake.complete(0);
73+
assert.equal(completed.count, 1);
74+
dispose();
75+
});
76+
77+
test('a Uint8Array from another realm is reported', () => {
78+
const { fake, req, completed, dispose } = observe();
79+
const chunk = runInNewContext('new Uint8Array([1])') as Uint8Array;
80+
assert.equal(chunk instanceof Uint8Array, false);
81+
req.write(chunk);
82+
fake.complete(0);
83+
assert.equal(completed.count, 1);
84+
dispose();
85+
});
86+
87+
test('the observed write keeps every argument, this, the result and one callback', () => {
88+
const { fake, req, dispose } = observe();
89+
const calls: unknown[] = [];
90+
const error = new Error('EPIPE');
91+
fake.accepts = false;
92+
assert.equal(
93+
req.write('a', 'latin1', (cause) => calls.push(cause)),
94+
false,
95+
);
96+
fake.accepts = true;
97+
assert.equal(
98+
req.write('b', (cause) => calls.push(cause)),
99+
true,
100+
);
101+
assert.equal(req.write('c'), true);
102+
assert.deepEqual(
103+
fake.writes.map(({ self, args }) => [self === fake, ...args.map((arg) => typeof arg)]),
104+
[
105+
[true, 'string', 'string', 'function'],
106+
[true, 'string', 'function'],
107+
[true, 'string', 'undefined', 'function'],
108+
],
109+
);
110+
assert.equal(fake.writes[0]?.args[1], 'latin1');
111+
fake.complete(0, error);
112+
fake.complete(1);
113+
fake.complete(2);
114+
assert.deepEqual(calls, [error, null]);
115+
dispose();
116+
});
117+
118+
const endings: [string, (observed: Observed) => unknown][] = [
119+
['the response started', ({ fake }) => fake.emit('response')],
120+
['the signal aborted', ({ caller }) => caller.abort()],
121+
['the request was destroyed', ({ fake }) => (fake.destroyed = true)],
122+
['the observer was disposed', ({ dispose }) => dispose()],
123+
];
124+
125+
test.each(endings)('a completion after %s is not reported', (_, end) => {
126+
const observed = observe();
127+
const calls: unknown[] = [];
128+
observed.req.write(Buffer.from('late'), (cause) => calls.push(cause));
129+
end(observed);
130+
observed.fake.complete(0);
131+
assert.equal(observed.completed.count, 0);
132+
assert.deepEqual(calls, [null], 'the caller still hears its own completion');
133+
observed.dispose();
134+
});
135+
136+
test('disposing removes the listener and restores only the write it wrapped', () => {
137+
const inherited = observe();
138+
inherited.dispose();
139+
assert.equal(inherited.fake.listenerCount('response'), 0);
140+
assert.equal(Object.hasOwn(inherited.fake, 'write'), false, 'the inherited write is back');
141+
const ownWrite = () => true;
142+
const owned = observe((fake) => Object.assign(fake, { write: ownWrite }));
143+
owned.dispose();
144+
assert.equal(owned.fake.write, ownWrite, 'an own write is put back');
145+
const wrapped = observe();
146+
const foreign = () => true;
147+
Object.assign(wrapped.fake, { write: foreign });
148+
wrapped.dispose();
149+
assert.equal(wrapped.fake.write, foreign, 'a later wrapper stays installed');
150+
});

‎packages/host-kit/src/internal/transport.test.ts‎

Lines changed: 7 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -44,6 +44,13 @@ test('loadNodeHttpRequester selects the protocol module and keeps request mutabl
4444
test('readNodeHttpResponseBody decodes the complete response stream', async () => {
4545
const response = Readable.from(['hello', Buffer.from(' world')]) as unknown as IncomingMessage;
4646
assert.equal(await readNodeHttpResponseBody(response), 'hello world');
47+
const observed: string[] = [];
48+
const progressing = Readable.from(['hello', Buffer.from(' world')]) as unknown as IncomingMessage;
49+
assert.equal(
50+
await readNodeHttpResponseBody(progressing, (chunk) => observed.push(chunk)),
51+
'hello world',
52+
);
53+
assert.deepEqual(observed, ['hello', ' world']);
4754
});
4855

4956
test('readNodeHttpRequestBody returns bytes and preserves the caller error message at the limit', async () => {

‎packages/host-kit/src/internal/transport.ts‎

Lines changed: 58 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -1,5 +1,6 @@
11
import crypto from 'node:crypto';
2-
import type { IncomingMessage } from 'node:http';
2+
import { types } from 'node:util';
3+
import type { ClientRequest, IncomingMessage } from 'node:http';
34
import { AppError } from '@agent-device/kernel/errors';
45

56
export function consumeTextLines(
@@ -42,11 +43,15 @@ export async function loadNodeHttpRequester(protocol: string): Promise<NodeHttpR
4243
: (await import('node:http')).default;
4344
}
4445

45-
export function readNodeHttpResponseBody(res: IncomingMessage): Promise<string> {
46+
export function readNodeHttpResponseBody(
47+
res: IncomingMessage,
48+
onChunk?: (chunk: string) => void,
49+
): Promise<string> {
4650
return new Promise((resolve, reject) => {
4751
let body = '';
4852
res.setEncoding('utf8');
49-
res.on('data', (chunk) => {
53+
res.on('data', (chunk: string) => {
54+
onChunk?.(chunk);
5055
body += chunk;
5156
});
5257
res.on('end', () => resolve(body));
@@ -82,3 +87,53 @@ export function timingSafeStringEqual(a: string, b: string): boolean {
8287
const hashB = crypto.createHash('sha256').update(b).digest();
8388
return crypto.timingSafeEqual(hashA, hashB);
8489
}
90+
91+
/**
92+
* Reports each non-empty `req.write` chunk that Node completes without an error: its bytes reached
93+
* the local kernel, not the peer. A queued write, `drain`, and a completion after the response
94+
* started, after `signal` aborted, after a destroy or after disposal are not reported.
95+
*
96+
* Attach it in the tick that creates the request, before anything writes to it. The wrapper keeps
97+
* the arguments, `this`, the backpressure result and one call of the caller's callback with its
98+
* error. Disposing removes its listener and restores `write` only while this wrapper is installed.
99+
*/
100+
export function observeCompletedRequestWrites(
101+
req: ClientRequest,
102+
signal: AbortSignal,
103+
onCompleted: () => void,
104+
): () => void {
105+
const ownWrite = Object.getOwnPropertyDescriptor(req, 'write');
106+
const write = req.write;
107+
let observing = true;
108+
const onResponse = () => {
109+
observing = false;
110+
};
111+
const observedWrite = function (this: ClientRequest, ...args: unknown[]): boolean {
112+
const [chunk, encoding] = args;
113+
const callbackIndex = typeof encoding === 'function' ? 1 : 2;
114+
const callback = args[callbackIndex];
115+
const encodingName = typeof encoding === 'string' ? encoding.toLowerCase() : undefined;
116+
const hasBytes =
117+
typeof chunk === 'string'
118+
? encodingName === 'hex' || encodingName === 'base64' || encodingName === 'base64url'
119+
? Buffer.from(chunk, encodingName).byteLength > 0
120+
: chunk.length > 0
121+
: types.isUint8Array(chunk) && chunk.byteLength > 0;
122+
if (observing && hasBytes) {
123+
args[callbackIndex] = (error?: Error | null) => {
124+
if (!error && observing && !req.destroyed && !signal.aborted) onCompleted();
125+
if (typeof callback === 'function') callback(error);
126+
};
127+
}
128+
return Reflect.apply(write, this, args);
129+
};
130+
req.write = observedWrite;
131+
req.prependListener('response', onResponse);
132+
return () => {
133+
observing = false;
134+
req.off('response', onResponse);
135+
if (req.write !== observedWrite) return;
136+
if (ownWrite) Object.defineProperty(req, 'write', ownWrite);
137+
else Reflect.deleteProperty(req, 'write');
138+
};
139+
}

‎packages/host-kit/src/transport.ts‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -1,6 +1,7 @@
11
export {
22
consumeTextLines,
33
loadNodeHttpRequester,
4+
observeCompletedRequestWrites,
45
readNodeHttpRequestBody,
56
readNodeHttpResponseBody,
67
timingSafeStringEqual,

0 commit comments

Comments
 (0)