feat(artifact)!: stream artifacts from sync and async readers by wolfv · Pull Request #185 · sigstore/sigstore-rust

wolfv · 2026-08-14T14:28:21Z

Summary

Adds constant-memory artifact streaming to signing and verification.

Signing

Signer::sign(...) keeps accepting raw bytes or a typed SHA-256 digest
Signer::sign_reader(...) streams a blocking std::io::Read to EOF
Signer::sign_async_reader(...) streams a runtime-independent futures_io::AsyncRead

The artifact is hashed before the Fulcio certificate is requested, so large inputs no longer eat into the certificate's lifetime. The signer configuration is validated before any hashing starts.

Verification

Verifier::verify(...) and the free verify(...) keep accepting raw bytes or a typed digest
Verifier::verify_reader(...) and Verifier::verify_async_reader(...) stream the artifact
Managed-key verification mirrors this with verify_with_key, verify_with_key_reader and verify_with_key_async_reader
The free verify_with_key(...) is now a thin wrapper over the Verifier method instead of a second copy of the verification flow

Readers are consumed exactly once. A new crate-private PreparedArtifact hashes the input in a single pass with every algorithm the bundle can ask for, and message-signature checks, DSSE subject binding, and Rekor consistency checks all draw on those digests without retaining or rereading the artifact. DSSE bundles hash only with the algorithms their in-toto subjects use; message-signature bundles hash SHA-256 for hashedrekord, the declared messageDigest algorithm, and, for streamed input, the digest the signing scheme consumes (derived from the bundle's certificate or the supplied managed key up front).

Blob input remains available for schemes such as Ed25519 that need the original message. Reader and digest input fail closed when the selected scheme does not support prehashed verification. Synchronous reader methods document that reads block the calling thread.

Shared hashing helpers

sigstore_crypto gains hash_reader, hash_reader_yielding and hash_async_reader, which feed one pass over a reader into any set of hashers. The async variants yield to the executor after every 64 KiB chunk (TOB-SIGSTORE-8), and the signer's cooperative in-memory hashing now goes through the same code. The library crates depend on futures-io only; futures remains a dev-dependency.

CLI examples stream files. Tests verify real hashedrekord and DSSE bundles through both sync and async readers, and unit tests pin the per-bundle digest selection.

Stack

fix(sign): keep unbounded hashing and signing from starving the async executor #183: incremental/prehashed signing foundation
refactor(artifact)!: make artifact digests typed #184: typed blob and digest inputs
This PR: synchronous/asynchronous reader sources

Signed-off-by: Wolf Vollprecht w.vollprecht@gmail.com

jku

Some of this code I am not really able to review... but it seems correct to me and the concept certainly makes sense.

Having so many versions of verify/sign is a bit annoying but they all seem reasonable and serve a use case that's not covered by the other variants.

jku · 2026-08-27T11:44:50Z

+
+    pub fn finalize(self) -> ArtifactDigest {
+        ArtifactDigest::new(self.algorithm, self.context.finish().as_ref())
+            .expect("cryptographic backend returned the algorithm's documented digest length")


nit: error message should say what went wrong, not the opposite

Add constant-memory reader entry points for signing, certificate verification, and managed-key verification. Synchronous APIs accept std::io::Read; runtime-independent asynchronous APIs accept futures_io::AsyncRead. Reader input is consumed once while all required digests are computed concurrently. The verification core receives prepared artifact evidence so message signatures, DSSE subject binding, and Rekor consistency checks share those digests without retaining or rereading the full artifact. The main sign and verify entry points now accept enum-based artifact sources, with existing reader-specific functions kept as forwarding compatibility wrappers. BREAKING CHANGE: verification internals consume prepared artifact evidence and reader I/O errors are exposed explicitly. Signed-off-by: Wolf Vollprecht <w.vollprecht@gmail.com>

Signed-off-by: Wolf Vollprecht <w.vollprecht@gmail.com> # Conflicts: # crates/sigstore-crypto/src/hash.rs # crates/sigstore-verify/src/verify.rs # crates/sigstore-verify/src/verify_impl/hashedrekord.rs

Signed-off-by: Wolf Vollprecht <w.vollprecht@gmail.com> # Conflicts: # crates/sigstore-verify/src/lib.rs # crates/sigstore-verify/src/verify.rs # crates/sigstore-verify/tests/verification_tests.rs

Drop the ArtifactSource enums and the reader()/async_reader() wrappers. Each input shape gets its own method, and both verification paths expose the same three: Verifier::verify / verify_reader / verify_async_reader Verifier::verify_with_key / verify_with_key_reader / verify_with_key_async_reader Signer::sign / sign_reader / sign_async_reader The free verify() and verify_with_key() stay as thin wrappers over Verifier for bytes/digest input. Move the managed-key verification body onto Verifier so it uses the same trusted root as verify(). Add sync and async reader tests for managed-key bundles. BREAKING CHANGE: ArtifactSource, AsyncArtifactSource, reader(), async_reader(), verify_async(), sign_async(), and the free verify_reader/verify_async_reader/verify_with_key_reader/ verify_with_key_async_reader functions are removed. Signed-off-by: Wolf Vollprecht <w.vollprecht@gmail.com>

PreparedArtifact::digest() now returns an ArtifactDigest instead of raw bytes, and sha256()/sha512() helpers give the typed hashes the DSSE subject check and hashedrekord path need. This removes the compute_artifact_digest wrappers and the try_from_slice fallbacks that could not fail. Blob input hashes through ArtifactHasher, so the per-algorithm match disappears. Fold the duplicated messageDigest comparison in the certificate and managed-key paths into verify_message_digest_binding(). Signed-off-by: Wolf Vollprecht <w.vollprecht@gmail.com>

Blob input was rehashed on every digest() call, so a MessageSignature bundle could hash the same bytes up to three times. PreparedArtifact now hashes blobs once at construction, through the same single multi-hasher pass the reader path uses, and digest() is a plain lookup for every input kind. The set of algorithms is derived from the bundle content instead of a fixed SHA-256 + SHA-512: MessageSignature bundles need SHA-256 plus the declared messageDigest algorithm, DSSE bundles need SHA-256 and SHA-512 for in-toto subjects. A MessageSignature bundle therefore costs exactly one SHA-256 pass over the artifact. Signed-off-by: Wolf Vollprecht <w.vollprecht@gmail.com>

The in-toto statement is in the bundle, so read it before hashing the artifact and hash only with the algorithms its subjects use. Cosign and GitHub attestations bind SHA-256 only, so the common DSSE case is now a single SHA-256 pass like MessageSignature bundles. If the payload is not a readable statement, hash SHA-256 and SHA-512 so the binding check can report the real problem. Add Statement::subject_algorithms() to sigstore-types. Signed-off-by: Wolf Vollprecht <w.vollprecht@gmail.com>

The message read like a success condition. Say what went wrong instead, as requested in review. Signed-off-by: Wolf Vollprecht <w.vollprecht@gmail.com> Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>

…-io only The read-to-EOF-into-hashers loop existed four times (sync and async, in sigstore-sign and sigstore-verify) plus a chunked in-memory variant, each with its own copy of `yield_now`. Move one implementation into sigstore-crypto next to the hashers: - `hash_reader` feeds a blocking reader into any `HashUpdate` hashers - `hash_async_reader` does the same for `futures_io::AsyncRead`, yielding to the executor after every chunk (TOB-SIGSTORE-8) - `hash_reader_yielding` adapts a blocking reader, including in-memory slices, onto the async path so signing keeps its cooperative hashing The chunk-boundary and yield tests move with the code. Only `AsyncRead` itself is needed from the futures family now, so the library crates depend on `futures-io` instead of the full `futures` crate; `futures` stays a dev-dependency for `Cursor` in tests. Signed-off-by: Wolf Vollprecht <w.vollprecht@gmail.com> Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>

`sign_sha256` validated the configuration only after the artifact had been hashed, so a misconfigured signer (Rekor v2 without a timestamp authority, for example) surfaced the error only after streaming a possibly large input. Check it first in every public entry point. Signed-off-by: Wolf Vollprecht <w.vollprecht@gmail.com> Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>

Streamed input can only be verified prehashed, so the single pass over the reader must produce the digest the signing scheme consumes. Until now the required algorithms were derived from the bundle content alone: a MessageSignature bundle without `messageDigest` whose key defaults to a non-SHA-256 scheme (P-384, RSA with SHA-512) verified from a blob but failed closed from a reader with "requires an sha2-384 artifact digest". Derive the scheme from the bundle's certificate, or the caller-supplied managed key, before reading and hash its digest alongside the binding digests. If the key material cannot be read up front, hash with every algorithm so verification proceeds far enough to report the real problem. Blob input is unchanged: the signature is verified over the bytes, so only the binding digests are computed. Signed-off-by: Wolf Vollprecht <w.vollprecht@gmail.com> Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>

The reader tests only exercised hashedrekord bundles, so the subject- algorithm selection for DSSE input had no end-to-end coverage. Verify the conda attestation (SHA-256 subject) from sync and async readers, reject a tampered streamed artifact, and bind a SHA-512-only subject from a reader. Signed-off-by: Wolf Vollprecht <w.vollprecht@gmail.com> Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>

…d digests Signed-off-by: Wolf Vollprecht <w.vollprecht@gmail.com>

wolfv · 2026-09-05T11:22:36Z

Reader follow-up is complete and all checks on this PR are green. It now rejects invalid signing/verification inputs before consuming readers, retries Interrupted reads, yields during DSSE copying/hashing, and reuses prepared digests/statements. Additional hardening starts with #198; unrelated changes are kept out of this PR.

baszalmstra

Took a quick look. Overall looks good to me.

baszalmstra · 2026-09-08T12:17:56Z

            }
        };
-        verify(&artifact_bytes, &bundle, &policy, &trusted_root)
+        verifier.verify_reader(artifact, &bundle, &policy)


The example now unconditionally streams files. The reader API intentionally rejects Ed25519 message
signatures, whereas the previous byte API could verify them. This is code-traced, not reproduced with
an Ed25519 certificate fixture.

Could we preserve a byte-input path for schemes that cannot be prehashed? Switching every file to
verify_reader drops Ed25519 message-signature support. Those bundles could previously verify here,
and supplying a digest is not an alternative.

baszalmstra · 2026-09-08T12:19:29Z

+    fn block_on<F: Future>(future: F) -> F::Output {
+        // Every future here only ever yields once per chunk and re-wakes
+        // itself, so a spin loop with a no-op waker is a sufficient executor.
+        let waker = futures_task_noop_waker();
+        let mut cx = TaskContext::from_waker(&waker);
+        let mut future = Box::pin(future);
+        loop {
+            if let Poll::Ready(output) = future.as_mut().poll(&mut cx) {
+                return output;
+            }
+        }
+    }


Why not use futures::executor::block_on?

baszalmstra · 2026-09-08T12:20:09Z

+    fn block_on<F: Future>(future: F) -> F::Output {
+        // Every future here only ever yields once per chunk and re-wakes
+        // itself, so a spin loop with a no-op waker is a sufficient executor.
+        let waker = futures_task_noop_waker();


https://doc.rust-lang.org/stable/std/task/struct.Waker.html#method.noop?

baszalmstra · 2026-09-08T12:21:31Z

+    struct CopyAndHash {
+        payload: Vec<u8>,
+        hasher: Sha256Hasher,
+    }
+    impl sigstore_crypto::HashUpdate for CopyAndHash {
+        fn update(&mut self, data: &[u8]) {
+            self.payload.extend_from_slice(data);
+            self.hasher.update(data);
+        }
+    }
+    let mut prepared = CopyAndHash {
+        payload: Vec::with_capacity(data.len()),
+        hasher: dsse_pae_hasher(payload_type, data.len()),
+    };
+    hash_reader_yielding(data, std::slice::from_mut(&mut prepared))
+        .await
+        .expect("reading from an in-memory slice cannot fail");


Could we share the yielding update logic without routing in-memory slices through a reader buffer?
This now copies each chunk into the 64 KiB buffer before copying it into payload. The previous chunk
loop hashed and copied directly from the source slice.

baszalmstra · 2026-09-08T12:23:31Z

+}
+
+/// Adapter presenting a blocking [`Read`] as an always-ready [`AsyncRead`].
+struct BlockingReader<R>(R);


Maybe we can use https://docs.rs/futures/0.3.32/futures/io/struct.AllowStdIo.html ?

baszalmstra · 2026-09-08T12:26:50Z

+///
+/// Reads block the calling thread. Inside an async task use
+/// [`hash_reader_yielding`] or [`hash_async_reader`] instead.
+pub fn hash_reader<H: HashUpdate>(mut reader: impl Read, hashers: &mut [H]) -> io::Result<()> {


Could we expose reader hashing through the existing hasher types rather than adding more free
functions? A shared trait could provide update_reader, update_reader_yielding, and update_async_reader, with a hasher collection supporting the same contract for single-pass
multi-digest hashing. That would also remove the SHA-256 reader wrappers in sign.rs, while keeping
the one-shot functions as conveniences.

Signed-off-by: Wolf Vollprecht <w.vollprecht@gmail.com>

This was referenced Aug 14, 2026

fix(sign): keep unbounded hashing and signing from starving the async executor #183

Merged

refactor(artifact)!: make artifact digests typed #184

Merged

fix(sign): keep unbounded hashing and signing from starving the async executor (TOB-SIGSTORE-8) #169

Closed

wolfv marked this pull request as ready for review August 14, 2026 14:40

wolfv force-pushed the artifact/readers branch from 603ce0a to e7d502b Compare August 17, 2026 11:40

wolfv force-pushed the artifact/readers branch 2 times, most recently from 1a7fbb4 to a2523de Compare August 17, 2026 11:59

wolfv force-pushed the artifact/readers branch from a2523de to 5160fd6 Compare August 17, 2026 12:01

wolfv force-pushed the artifact/readers branch from 5160fd6 to 1a4ec15 Compare August 17, 2026 12:18

Base automatically changed from artifact/typed-input to main August 18, 2026 10:26

wolfv force-pushed the artifact/readers branch 3 times, most recently from 0260601 to 67c4db2 Compare August 19, 2026 09:43

jku previously approved these changes Aug 27, 2026

View reviewed changes

wolfv dismissed jku’s stale review via 8601588 August 27, 2026 13:16

wolfv mentioned this pull request Aug 27, 2026

feat: use enum-based artifact sources #194

Closed

wolfv force-pushed the artifact/readers branch 2 times, most recently from 7ecbb04 to 0535811 Compare August 27, 2026 13:18

wolfv force-pushed the artifact/readers branch from 0535811 to c20edeb Compare August 27, 2026 13:34

wolfv added 6 commits September 4, 2026 06:52

Merge remote-tracking branch 'origin/main' into artifact/readers

89309b5

Signed-off-by: Wolf Vollprecht <w.vollprecht@gmail.com> # Conflicts: # crates/sigstore-crypto/src/hash.rs # crates/sigstore-verify/src/verify.rs # crates/sigstore-verify/src/verify_impl/hashedrekord.rs

Merge remote-tracking branch 'origin/main' into artifact/readers

cd31b71

Signed-off-by: Wolf Vollprecht <w.vollprecht@gmail.com> # Conflicts: # crates/sigstore-verify/src/lib.rs # crates/sigstore-verify/src/verify.rs # crates/sigstore-verify/tests/verification_tests.rs

wolfv and others added 6 commits September 4, 2026 19:52

fix(crypto): describe the failure in ArtifactHasher's expect message

fa6aa26

The message read like a success condition. Say what went wrong instead, as requested in review. Signed-off-by: Wolf Vollprecht <w.vollprecht@gmail.com> Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>

fix(artifact): reject invalid inputs before reading and reuse prepare…

55090aa

…d digests Signed-off-by: Wolf Vollprecht <w.vollprecht@gmail.com>

This was referenced Sep 5, 2026

fix(tuf): preserve rollback floors and recheck target authorization #198

Open

fix(types)!: checked log indices, unambiguous digests and DSSE verifier binding #199

Open

baszalmstra reviewed Sep 8, 2026

View reviewed changes

fix(artifact): preserve Ed25519 file verification and avoid slice copies

fed1ee2

Signed-off-by: Wolf Vollprecht <w.vollprecht@gmail.com>

Navigation Menu

feat(artifact)!: stream artifacts from sync and async readers - #185

feat(artifact)!: stream artifacts from sync and async readers#185
wolfv wants to merge 14 commits into
mainfrom
artifact/readers

wolfv commented Aug 14, 2026 •

edited

Loading

Uh oh!

jku left a comment

Uh oh!

jku Aug 27, 2026

Uh oh!

wolfv commented Sep 5, 2026

Uh oh!

baszalmstra left a comment

Uh oh!

baszalmstra Sep 8, 2026

Uh oh!

baszalmstra Sep 8, 2026

Uh oh!

baszalmstra Sep 8, 2026

Uh oh!

baszalmstra Sep 8, 2026

Uh oh!

baszalmstra Sep 8, 2026

Uh oh!

baszalmstra Sep 8, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

Conversation

wolfv commented Aug 14, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Summary

Signing

Verification

Shared hashing helpers

Stack

Uh oh!

jku left a comment

Choose a reason for hiding this comment

Uh oh!

jku Aug 27, 2026

Choose a reason for hiding this comment

Uh oh!

wolfv commented Sep 5, 2026

Uh oh!

baszalmstra left a comment

Choose a reason for hiding this comment

Uh oh!

baszalmstra Sep 8, 2026

Choose a reason for hiding this comment

Uh oh!

baszalmstra Sep 8, 2026

Choose a reason for hiding this comment

Uh oh!

baszalmstra Sep 8, 2026

Choose a reason for hiding this comment

Uh oh!

baszalmstra Sep 8, 2026

Choose a reason for hiding this comment

Uh oh!

baszalmstra Sep 8, 2026

Choose a reason for hiding this comment

Uh oh!

baszalmstra Sep 8, 2026

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

wolfv commented Aug 14, 2026 •

edited

Loading