readBlocks in the sampled-retrieval piece validation wraps the incoming piece buffer in new Uint8Array(pieceBytes) before handing it to CarReader.fromBytes:
|
private async readBlocks(pieceBytes: Buffer, signal?: AbortSignal): Promise<SampledBlock[]> { |
|
const reader = await CarReader.fromBytes(new Uint8Array(pieceBytes)); |
|
const blocks: SampledBlock[] = []; |
|
for await (const block of reader.blocks()) { |
|
signal?.throwIfAborted(); |
|
blocks.push({ cid: block.cid, bytes: block.bytes }); |
|
} |
|
return blocks; |
|
} |
pieceBytes is a Buffer, which already is a Uint8Array, so that wrapper allocates a second full copy of the entire piece. Sampled pieces run up to ~500 MiB, so peak memory for one check is roughly 2x the piece size for no reason.
Fix: pass pieceBytes straight to CarReader.fromBytes(pieceBytes). Same behavior, drops the redundant allocation.
This is pre-existing from #487 and untouched by the rename in #612, so tracking it on its own. The file moves to apps/backend/src/sampled-retrieval/piece-validation.service.ts once #612 lands.
readBlocksin the sampled-retrieval piece validation wraps the incoming piece buffer innew Uint8Array(pieceBytes)before handing it toCarReader.fromBytes:dealbot/apps/backend/src/retrieval-anon/piece-validation.service.ts
Lines 192 to 200 in 2452e7d
pieceBytesis aBuffer, which already is aUint8Array, so that wrapper allocates a second full copy of the entire piece. Sampled pieces run up to ~500 MiB, so peak memory for one check is roughly 2x the piece size for no reason.Fix: pass
pieceBytesstraight toCarReader.fromBytes(pieceBytes). Same behavior, drops the redundant allocation.This is pre-existing from #487 and untouched by the rename in #612, so tracking it on its own. The file moves to
apps/backend/src/sampled-retrieval/piece-validation.service.tsonce #612 lands.