Skip to content

Avoid duplicating the full piece buffer when parsing sampled-retrieval CARs #616

Description

@SgtPooki

readBlocks in the sampled-retrieval piece validation wraps the incoming piece buffer in new Uint8Array(pieceBytes) before handing it to CarReader.fromBytes:

private async readBlocks(pieceBytes: Buffer, signal?: AbortSignal): Promise<SampledBlock[]> {
const reader = await CarReader.fromBytes(new Uint8Array(pieceBytes));
const blocks: SampledBlock[] = [];
for await (const block of reader.blocks()) {
signal?.throwIfAborted();
blocks.push({ cid: block.cid, bytes: block.bytes });
}
return blocks;
}

pieceBytes is a Buffer, which already is a Uint8Array, so that wrapper allocates a second full copy of the entire piece. Sampled pieces run up to ~500 MiB, so peak memory for one check is roughly 2x the piece size for no reason.

Fix: pass pieceBytes straight to CarReader.fromBytes(pieceBytes). Same behavior, drops the redundant allocation.

This is pre-existing from #487 and untouched by the rename in #612, so tracking it on its own. The file moves to apps/backend/src/sampled-retrieval/piece-validation.service.ts once #612 lands.

Metadata

Metadata

Assignees

No one assigned

    Labels

    enhancementNew feature or requestgood first issueGood for newcomershelp wantedExtra attention is neededready-for-workTriaged: scope, plan, and DoD are clear; contributor can pick up

    Type

    No type

    Fields

    No fields configured for issues without a type.

    Projects

    Status
    🐱 Todo

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions