Skip to content

Embed native assets in the integration test snapshot - #4888

Merged
mosuem merged 2 commits into
mainfrom
use-dart-build-cli
Oct 12, 2026
Merged

Embed native assets in the integration test snapshot#4888
mosuem merged 2 commits into
mainfrom
use-dart-build-cli

Conversation

@mosuem

@mosuem mosuem commented Aug 25, 2026 •
edited
Loading

Copy link
Copy Markdown
Member

Note

This PR was generated by an AI coding agent (Jetski) on behalf of @mosuem.

Prepares the integration test setup for dependencies with native assets (package:sigstore in #4875).

  • tool/test.dart passes the native assets mapping written by dart run (.dart_tool/native_assets.yaml, if present) to precompile(nativeAssets: ...), so it is embedded in the pub snapshot that the tests execute. This re-adds the nativeAssets pass-through to precompile in lib/src/dart.dart that was removed in Remove local executable precompilation and deprecate --precompile #4910.
  • CI runs the tests through dart run tool/test.dart --preset ci ..., which executes the build hooks, builds the snapshot and forwards the arguments to dart test. This replaces the separate dart --snapshot step (_PUB_TEST_SNAPSHOT is now set by tool/test.dart).

Pub keeps running under the dart VM with assertions enabled in the tests, as in production, so no changes to how pub locates the SDK are needed.

This supersedes the earlier dart build cli approach in this PR (see the review discussion): running pub as a standalone AOT executable made Platform.resolvedExecutable point at pub instead of dart, which broke precompile and required workarounds in production code.

@mosuem
mosuem force-pushed the use-dart-build-cli branch 11 times, most recently from 8966a9f to aaf5234 Compare August 28, 2026 11:14
@mosuem
mosuem requested a review from sigurdm August 28, 2026 12:49
@mosuem
mosuem marked this pull request as ready for review August 28, 2026 12:49
copybara-service Bot pushed a commit to dart-lang/sdk that referenced this pull request Sep 3, 2026
@mosuem dart-scoped@luci-project-accounts.iam.gserviceaccount.com
[frontend_server_client] Support standalone CLI executables
26b1e4c 
Needed for dart-lang/pub#4888.

In dart-lang/pub tests, pub was previously precompiled to a snapshot for
fast execution. However, snapshot compilation does not support native
assets. Switching to `dart build cli` creates a standalone binary,
causing `Platform.resolvedExecutable` to point to the standalone
executable rather than `dart`.

Previously, `FrontendServerClient.start` only used `sdkRoot` as a child
compiler argument, but resolved `dartaotruntime` and snapshot binaries
relative to `Platform.resolvedExecutable`. This caused compiler spawning
failures when running from standalone CLI tools.

Also removes dead IA32 code in the scope.

TESTED=pkg/frontend_server_client/test/frontend_server_client_test.dart
Change-Id: Ib94c15f8fc1cad71b664d0e184a4fd3b90cf96ff
Reviewed-on: https://dart-review.googlesource.com/c/sdk/+/542720
Reviewed-by: Ben Konyi <bkonyi@google.com>
Commit-Queue: Moritz Sümmermann <mosum@google.com>
sigurdm
sigurdm reviewed Sep 4, 2026
View reviewed changes
Comment thread lib/src/executable.dart Outdated
sigurdm
sigurdm reviewed Sep 4, 2026
View reviewed changes
Comment thread lib/src/global_packages.dart Outdated
sigurdm
sigurdm reviewed Sep 4, 2026
View reviewed changes
Comment thread tool/test.dart Outdated
sigurdm
sigurdm requested changes Sep 4, 2026
View reviewed changes

@sigurdm sigurdm left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for working on this! A few observations:

  • The CI test failures (Could not find an option named "--sdk-root") are because frontend_server_client resolves the SDK relative to Platform.resolvedExecutable, which points to pub instead of dart when running as an AOT binary. I saw you landed the fix in the SDK repo (commit 26b1e4c). That package will need to be published to pub.dev and pubspec.yaml updated here before CI can turn green.
  • master recently received workflow fixes (#4900), so this will need a rebase on master.
  • See inline comments regarding binstub generation in production, running tests with assertions enabled, and Windows path quoting.

Comment thread lib/src/global_packages.dart Outdated
Comment thread lib/src/io.dart Outdated
Comment thread tool/test-bin/pub.bat Outdated
Comment thread lib/src/executable.dart Outdated
@mosuem
mosuem requested a review from sigurdm October 2, 2026 13:15
@mosuem
mosuem force-pushed the use-dart-build-cli branch from aaf5234 to 65a1cea Compare October 12, 2026 07:34
@mosuem mosuem changed the title Use dart build cli for integration test executable Embed native assets in the integration test snapshot Oct 12, 2026
@mosuem
mosuem force-pushed the use-dart-build-cli branch from 65a1cea to c3fe8ce Compare October 12, 2026 07:50
mosuem added a commit to dart-lang/pub-dev that referenced this pull request Oct 12, 2026
@mosuem
Embed the native assets mapping of package:sigstore in the server sna…
08c6c0f 
…pshot

`package:sigstore` calls into `libsigstore_ffi.so` through `@Native`
bindings, which the VM resolves through a native assets mapping. `dart run`
produces that mapping by executing build hooks and embeds it in the kernel
it runs; `dart compile jit-snapshot` does not, so the server snapshot in
`Dockerfile.app` could not load the library at runtime.

- `tool/build-server-snapshot.sh` compiles the server to kernel, compiles
  a mapping (asset id -> path of the library in the image) to a kernel
  component with the frontend server, concatenates the two and creates the
  app-JIT snapshot from the result. The VM keeps the mapping in the
  snapshot, so no hooks or package sources are needed at runtime. This is
  the mechanism `dart run` uses (cf. dart-lang/pub#4888).
- `Dockerfile.app` builds the snapshot with that script, compiles the
  library from the Rust source bundled with the package in a `rust-build`
  stage (no prebuilt binaries are downloaded), and copies it - together
  with `libgcc_s`, which it needs and which is not in the Dart image's
  `/runtime` - into the final image.
- `server.jit self-check` (hidden command) calls into the native library;
  `dockerfile_app_build_test.dart` runs it in the built image, and the
  frontend entrypoint performs the same check at startup so that a broken
  image fails to boot instead of failing every attested upload.
@mosuem mosuem mentioned this pull request Oct 12, 2026
Draft
sigurdm
sigurdm approved these changes Oct 12, 2026
View reviewed changes

@sigurdm sigurdm left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM, thanks for the rewrite.

mosuem added a commit to dart-lang/pub-dev that referenced this pull request Oct 12, 2026
@mosuem
Verify uploaded package attestations using package:sigstore
da75f89 
Adds `package:sigstore` and an `AttestationVerifier` service that verifies
the Sigstore bundle uploaded with a package archive before the version is
published:

- verifies the Fulcio certificate signer identity against the trusted
  reusable publishing workflows (`defaultTrustedSignerWorkflows`), over the
  archive's sha256 digest that `publishUploadedBlob` already computed
  (`SigstoreClient.verify(..., isDigest: true)`), so the archive is neither
  read into memory nor hashed again,
- parses the in-toto SLSA v1 DSSE provenance statement from the same
  bundle JSON (`BuildProvenance.parseBundle`),
- enforces the provenance policy (`checkProvenancePolicy`): binds the
  archive sha256 and subject name, verifies `provenance.sourceRepository`
  against the `repository:` of the uploaded `pubspec.yaml`, and - when
  GitHub Actions automated publishing is enabled for the package - also
  against `publishingConfig.githubConfig.repository`,
- rejects the upload with `PackageRejectedException` when the attestation
  is invalid or fails verification; internal errors (e.g. the native
  library missing) propagate as a 500 instead of blaming the upload.

`AttestationVerifier` is a registered service (`registerAttestationVerifier`
/ `attestationVerifier`); `withFakeServices` registers a
`FakeAttestationVerifier` that replaces only the cryptographic step and
trusts the builder self-reported in the statement, so tests exercise the
real provenance parsing and policy with hand-crafted bundles. An end-to-end
test verifies a real signed bundle.

Deployment: `package:sigstore` calls into `libsigstore_ffi.so` through
`@Native` bindings, which the VM resolves through a native assets mapping
that `dart compile jit-snapshot` does not produce. `tool/build-server-
snapshot.sh` compiles the server to kernel, compiles the mapping (asset id
-> path of the library in the image) to a kernel component with the
frontend server, concatenates the two and creates the app-JIT snapshot
from the result; the VM keeps the mapping in the snapshot, so no hooks or
package sources are needed at runtime (the mechanism `dart run` uses, cf.
dart-lang/pub#4888). `Dockerfile.app` builds the library from the Rust
source bundled with the package in a `rust-build` stage (no prebuilt
binaries are downloaded) and copies it, with `libgcc_s`, into the final
image. `server.jit self-check` calls into the native library;
`dockerfile_app_build_test.dart` runs it in the built image, and the
frontend entrypoint performs the same check at startup so that a broken
image fails to boot instead of failing every attested upload.
mosuem added a commit to dart-lang/pub-dev that referenced this pull request Oct 12, 2026
@mosuem
Verify uploaded package attestations using package:sigstore
ff2b1ff 
Adds `package:sigstore` and an `AttestationVerifier` service that verifies
the Sigstore bundle uploaded with a package archive before the version is
published:

- verifies the Fulcio certificate signer identity against the trusted
  reusable publishing workflows (`defaultTrustedSignerWorkflows`), over the
  archive's sha256 digest that `publishUploadedBlob` already computed
  (`SigstoreClient.verify(..., isDigest: true)`), so the archive is neither
  read into memory nor hashed again,
- parses the in-toto SLSA v1 DSSE provenance statement from the same
  bundle JSON (`BuildProvenance.parseBundle`),
- enforces the provenance policy (`checkProvenancePolicy`): binds the
  archive sha256 and subject name, verifies `provenance.sourceRepository`
  against the `repository:` of the uploaded `pubspec.yaml`, and - when
  GitHub Actions automated publishing is enabled for the package - also
  against `publishingConfig.githubConfig.repository`,
- rejects the upload with `PackageRejectedException` when the attestation
  is invalid or fails verification; internal errors (e.g. the native
  library missing) propagate as a 500 instead of blaming the upload.

`AttestationVerifier` is a registered service (`registerAttestationVerifier`
/ `attestationVerifier`); `withFakeServices` registers a
`FakeAttestationVerifier` that replaces only the cryptographic step and
trusts the builder self-reported in the statement, so tests exercise the
real provenance parsing and policy with hand-crafted bundles. An end-to-end
test verifies a real signed bundle.

Deployment: `package:sigstore` calls into `libsigstore_ffi.so` through
`@Native` bindings, which the VM resolves through a native assets mapping
that `dart compile jit-snapshot` does not produce. `tool/build-server-
snapshot.sh` compiles the server to kernel, compiles the mapping (asset id
-> path of the library in the image) to a kernel component with the
frontend server, concatenates the two and creates the app-JIT snapshot
from the result; the VM keeps the mapping in the snapshot, so no hooks or
package sources are needed at runtime (the mechanism `dart run` uses, cf.
dart-lang/pub#4888). `Dockerfile.app` builds the library from the Rust
source bundled with the package in a `rust-build` stage (no prebuilt
binaries are downloaded) and copies it, with `libgcc_s`, into the final
image. `server.jit self-check` calls into the native library;
`dockerfile_app_build_test.dart` runs it in the built image, and the
frontend entrypoint performs the same check at startup so that a broken
image fails to boot instead of failing every attested upload.
mosuem and others added 2 commits October 12, 2026 14:00
@mosuem @sigurdm
Embed native assets in the integration test snapshot
ab17150 
Run the integration tests through `dart run tool/test.dart` in CI, and pass
the native assets mapping written by `dart run` (.dart_tool/native_assets.yaml)
to `precompile` so that it is embedded in the pub snapshot used by the tests.
This re-adds the `nativeAssets` pass-through to `precompile` that was
removed in #4910.

This prepares the test setup for dependencies with native assets
(package:sigstore in #4875) without changing how pub locates the SDK:
pub keeps running under the `dart` VM with assertions enabled, as in
production.
@sigurdm
Merge branch 'main' into use-dart-build-cli
7caaabd
@sigurdm
sigurdm force-pushed the use-dart-build-cli branch from 0bc2bd8 to 7caaabd Compare October 12, 2026 14:02
@sigurdm
sigurdm removed this pull request from stack #4904 October 12, 2026 14:16
@sigurdm
sigurdm added this pull request to stack #4944 October 12, 2026 14:16
@mosuem
mosuem merged commit 12b149e into main Oct 12, 2026
30 checks passed
@mosuem
mosuem deleted the use-dart-build-cli branch October 12, 2026 14:33
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@sigurdm sigurdm sigurdm approved these changes

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

2 participants

@mosuem @sigurdm