Conversation
|
Important Review skippedDraft detected. Please check the settings in the CodeRabbit UI or the ⚙️ Run configurationConfiguration used: Repository UI Review profile: CHILL Plan: Pro Plus Run ID: You can disable this status message by setting the Use the checkbox below for a quick retry:
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
5b966bb to
e08601e
Compare
Mic92
changed the title
8785d4d to
94b6593
Compare
Codecov Report❌ Patch coverage is Additional details and impacted files@@ Coverage Diff @@
## main #849 +/- ##
==========================================
+ Coverage 64.58% 66.25% +1.66%
==========================================
Files 146 162 +16
Lines 16822 20415 +3593
Branches 16822 20415 +3593
==========================================
+ Hits 10865 13526 +2661
- Misses 5304 6123 +819
- Partials 653 766 +113
Flags with carried forward coverage won't be shown. Click here to find out more. ☔ View full report in Codecov by Sentry. 🚀 New features to boost your workflow:
|
|
Bencher Report| Branch | harmonia-ci |
| Testbed | github-actions |
⚠️ WARNING: No Threshold found!Without a Threshold, no Alerts will ever be generated.
Click here to create a new Threshold
For more information, see the Threshold documentation.
To only post results if a Threshold exists, set the--ci-only-thresholdsflag.
Click to view all benchmark results
| Benchmark | Latency | milliseconds (ms) |
|---|---|---|
| closure/download | 📈 view plot | 1,878.00 ms |
| http/concurrent_16 | 📈 view plot | 473.51 ms |
| http/concurrent_4 | 📈 view plot | 534.47 ms |
| http/sequential | 📈 view plot | 1,110.70 ms |
|
What about Gradient? https://github.com/wavelens/gradient ? Isn't this basically the same thing you are trying, but more mature? We talked at the 38c3 and you we're like: "Another Nix CI system 🙄 , just use build-bot." But I happy to migrate to harmonia daemon sandboxes once done ❤️ |
I haven't looked your CI. Yeah. I might do the sandbox first after all. Also these days I am leaning towards stuff like NATS for really large build farms (> 1000 nodes). |
That's should also be the goal for gradient, but imo Gradient is far away from hitting that kind of performance level. While Database queries are beeing improved and the custom protocol is modified to be able to handle many builders every day, it isn't a trivial task at all. |
ad93201 to
dfdfd99
Compare
bf327c6 to
33227ca
Compare
Add design documents under docs/architecture/ci/ that describe the planned standalone daemon for CI: build protocol, scheduler/database shape, sandbox/security model, signing/upload, forge integration, HA, GC, and the webui surface. PLAN.md tracks the in-progress sandbox migration. LICENSE and harmonia-daemon/README catch up to the harmonia-nar -> harmonia-file-nar rename. Co-authored-by: Amaan Qureshi <git@amaanq.com>
Stand up the process-isolation layer the daemon will use to run builders: a Sandbox trait with NoSandbox, LinuxSandbox (CLONE_NEWUSER / CLONE_NEWMOUNT user-namespace runner), and a darwin_sandbox using sandbox-exec. build_users introduces file-locked UID allocation (auto_user_lock on Linux when root, simple_user_lock for a configured build-users-group on macOS), with supplementary GIDs from the group and guards against self-UID and system-UID collisions. pathlocks and canonicalize cover store-path locking and the canonicalisation pass that Nix runs on a builder's output before hashing. The harmonia-daemon config and NixOS module gain matching sandbox / buildUsersGroup options; when the sandbox is off the NixOS unit re-enables the standard systemd hardening that was previously unconditional. Co-authored-by: Amaan Qureshi <git@amaanq.com>
Add the three builtin builders the standalone daemon will dispatch to in place of executing a real builder. buildenv assembles the `buildEnv` output by walking the input drvs and rejecting malformed manifests instead of silently producing an empty output. fetchurl implements `builtin:fetchurl` with the same hashing and ca-rewriting semantics as Nix. unpack_channel implements `builtin:unpack-channel` by extracting a tarball through libarchive (via compress-tools-rs); nix/packages.nix and the dev shell gain pkg-config + libarchive so the crate links, and the cargo build switches to the mold linker on ELF. Co-authored-by: Amaan Qureshi <git@amaanq.com>
…d tests Wire harmonia-daemon up as a real builder. build.rs (the executor) walks a Derivation, dispatches builtin builders or spawns a sandboxed child, drains stdout+stderr into bzip2-compressed build logs, enforces the timeout via the process group, validates impureHostDeps against allowed prefixes, sets NIX_OUTPUT_CHECKED / impureEnvVars for fixed- output derivations, wires exportReferencesGraph and output constraints, and cleans up partial outputs on any failed build. passAsFile and structuredAttrs are written to the build environment in the same shape Nix uses. scheduler.rs adds a DAG-aware build scheduler with bounded concurrency. export_references_graph.rs computes the closure graph the builder hands to Nix-compatible drvs. handler.rs gains query_missing for standalone-daemon clients, an on-the-fly hashing NAR import path, and propagates repair removal errors instead of swallowing them; nar_size == 0 is documented as the "unknown" sentinel. main.rs initialises tracing-subscriber, applies the SandboxConfig, and preserves mount flags when remounting the store writable. Cargo manifests pull in the deps (bytes/bzip2/compress-tools/ libc/nix/tokio process/tracing-subscriber/...) and lib.rs publishes the new modules. Tests cover add_to_store_nar, build_derivation (1.4k lines of behaviour coverage: fixed-output env, impure host deps, builtin dispatch, timeouts using a shell busy-loop, partial-output cleanup, ...) and query_missing. TestStore provides shared fixture wiring across the suite. Three NixOS VM tests (tests/cache.nix, daemon-cache.nix, privileged-sandbox.nix) exercise the daemon end-to-end through its socket, including a real privileged-sandbox build that checks the builder's credentials. Co-authored-by: Amaan Qureshi <git@amaanq.com>
33227ca to
58a4482
Compare
|
The sandbox I am using in https://github.com/Mic92/tribuchet/ |
2 participants
No description provided.