feat(payloads): streaming Payloads client + primitive payloads CLI#308
Open
poiley wants to merge 7 commits into
Open
feat(payloads): streaming Payloads client + primitive payloads CLI#308poiley wants to merge 7 commits into
primitive payloads CLI#308poiley wants to merge 7 commits into
Conversation
…push/pull (CLI) @primitivedotdev/sdk/payloads: streaming, bounded-memory push/pull for large content-addressed, end-to-end-encrypted objects. Chunk/manifest construction is byte-compatible with the server object model (64 MiB chunks, per-chunk HKDF AES-256-GCM, SHA-256 content addressing, Merkle manifest). Two-pass push keeps memory flat regardless of object size; transient 429/5xx are retried with backoff. CLI adds payloads push/pull on top. Local/unpublished: verified a 5 GiB round-trip against staging (integrity OK).
Exposes the chunk/crypto/Merkle construction as a pure function for precomputing an object's content address, convergent encoding, and conformance-testing against the server object model. A deterministic dual-encode (same cek/objectId) against payloads-core produced a byte-identical manifest — the vendored client primitives are verified byte-compatible with the reference implementation.
Golden vector generated from the server reference implementation (@primitivedotdev/payloads-core in the mono-repo) pins the SDK's vendored chunk/crypto/Merkle construction to the server's. encodeManifest must reproduce the fixture manifest byte-for-byte; any drift in HKDF/AES-GCM, content hashing, or Merkle layout fails the build — catching interop/dedup breakage before it ships. Runs with the normal SDK test suite; verified it fails on a one-char KDF change.
Fake in-memory server exercises the SDK client with no network: push/pull round-trip (multi-chunk, single-chunk, exact chunk boundary, zero-byte), the transient-503 retry path (previously unexercised), and the download integrity check (a corrupted chunk rejects). Closes the retry/edge-case coverage gaps flagged in the confidence review.
![github-advanced-security[bot]](https://avatars.githubusercontent.com/in/57789?s=60&v=4){kind=small}
Confidence Score: 5/5Safe to merge — the three issues flagged in earlier review rounds are all addressed and the crypto implementation is correct. The streaming client handles the full lifecycle correctly: deterministic re-encryption makes the two-pass upload safe, the Merkle root check closes the content-address loop on download, write-stream errors are surfaced via a raced promise rather than left as uncaught events, and failed downloads leave no plaintext on disk. The conformance vector and offline test suite cover the critical paths thoroughly. sdk-node/package.json — lint:fix script omits src/payloads/index.ts; all other files look good.
|
| Filename | Overview |
|---|---|
| sdk-node/src/payloads/index.ts | Core streaming payloads client — chunk/crypto/Merkle implementation is byte-compatible with the server, previously flagged issues (manifest root consistency check, partial-file cleanup, and write-stream error surfacing) are all addressed in this revision. |
| cli-node/src/oclif/commands/payloads.ts | New push/pull CLI commands; auth resolution, progress reporting, and JSON output look correct. |
| sdk-node/tests/payloads/client.test.ts | Offline round-trip, retry, integrity, manifest-tamper, and write-error paths all covered with an in-memory server mock. |
| sdk-node/tests/payloads/conformance.test.ts | Golden-vector test pins SDK's HKDF/AES-GCM/Merkle output byte-for-byte against the server reference implementation. |
| sdk-node/package.json | Adds ./payloads export and updates lint; lint:fix omits src/payloads/index.ts (minor inconsistency). |
| test-fixtures/payloads/conformance-vector.json | Server-generated golden vector for 200 KB input — well-documented, contains expected per-chunk hashes and Merkle root. |
Reviews (3): Last reviewed commit: "fix(payloads): robust write-stream lifec..." | Re-trigger Greptile
![greptile-apps[bot]](https://avatars.githubusercontent.com/in/867647?s=60&v=4){kind=small}
…clean partial file on failed pull - apiRoot trimmed trailing slashes with /\/+$/ on caller-supplied input (CodeQL js/polynomial-redos, high). Replace with a linear scan. - pullFile now recomputes the Merkle root from the manifest's chunk hashes and rejects a manifest that doesn't match the requested content address, and removes a partial output file if the download fails mid-stream. - Tests for the manifest-root-mismatch rejection and partial-file cleanup.
Attach an 'error' listener and race writes/end against it so an async disk-write failure (disk full, EIO, bad path) surfaces as a rejection instead of an uncaught error that crashes the process — and destroy the stream + remove the partial file on any failure. Adds a write-error test (invalid output path).
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
Adds first-class support for Primitive Payloads — large, content-addressed, end-to-end-encrypted objects (attachments up to ~1 TB) — to the Node SDK and CLI.
@primitivedotdev/sdk/payloads— a streaming client:pushFile(path, opts)chunks (64 MiB), encrypts each chunk (per-chunk HKDF-derived AES-256-GCM), content-addresses it (SHA-256), commits a Merkle manifest, and uploads — in bounded memory (a multi-GB file never loads fully into RAM). Transient429/5xxare retried with backoff.pullFile(root, out, opts)streams down, verifies each chunk against its content address, and decrypts — one chunk at a time.encodeManifest(bytes, opts)— pure, deterministic manifest computation (precompute a content address / convergent encoding / conformance testing).primitive payloads push <file>andprimitive payloads pull <root> --cek <hex> --out <file>.These routes are
openapi: falseserver-side, so this is hand-written rather than generated.Design notes
@primitivedotdev/api-core, so nothing new is published to npm and there's no low-level package next to the SDK/CLI. (For E2E encryption the client must run this code anyway.)test-fixtures/payloads/conformance-vector.json+sdk-node/tests/payloads/conformance.test.ts). It fails on any change to the crypto/hashing/Merkle construction — verified it fails on a one-character KDF change.Testing
encodeManifestreproduces the server-generated manifest byte-for-byte (per-chunk hashes + Merkle root).make node-checkandmake cli-checkgreen locally (biome incl. the new module, typecheck, full suites: SDK 715, CLI 726).shared-checknode fixtures unaffected.Depends on
The server-side Payloads datapath (schema, routes, capability-token downloads, chunk-size guard) — merged to
stagingin the API mono-repo.