Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
6 changes: 5 additions & 1 deletion Makefile
Original file line number Diff line number Diff line change
@@ -1,8 +1,9 @@
.PHONY: build test lint lint-spark test-client fmt-check audit
.PHONY: build test lint lint-spark test-client fmt-check audit update
build:
cargo build --locked --release --features appliance
test:
python3 tests/standalone_contract.py
python3 tests/spark_update.py
cargo test --locked --workspace --all-targets --features appliance
test-client:
cargo test --locked --no-default-features --all-targets
Expand All @@ -15,3 +16,6 @@ fmt-check:
cargo fmt --all -- --check
audit:
cargo deny --all-features check
HOST ?= dgx-spark
update:
@python3 scripts/update-spark.py "$(HOST)"
4 changes: 4 additions & 0 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -55,6 +55,10 @@ hermetic and do not contact a Spark or allocate a GPU.
- [CLI and runtime reference](docs/reference/spark.md)
- [Security and release authority](SECURITY.md)

To build and preview a signed source update, run `make update HOST=dgx-spark`.
Apply it with `python3 scripts/update-spark.py dgx-spark --apply`; see the
[workstation update workflow](docs/how-to/develop-spark.md#update-a-spark-from-this-checkout).

Inference URLs use `/openai/<instance>/v1` and `/anthropic/<instance>/v1`.
The control API uses `/api/sparkplane/v1`. Runtime paths, units, schemas,
environment variables and Docker ownership use the Sparkplane namespace.
Expand Down
3 changes: 2 additions & 1 deletion configs/sparkplane/engines/vllm-qwen38-mmap.toml
Original file line number Diff line number Diff line change
Expand Up @@ -109,6 +109,7 @@ arguments = [
"--reasoning-parser", "qwen3",
"--speculative-config", "{\"method\":\"mtp\",\"num_speculative_tokens\":3}",
]
capabilities = ["text_generation", "tool_calling"]
capabilities = ["text_generation", "tool_calling", "vision"]
vision = { processor_sha256 = "27225450ac9c6529872ee1924fcb0962ff5634834f817040f444118116f4e516", media_types = ["image/jpeg", "image/png", "image/webp"], max_bytes = 524288, max_total_bytes = 524288, max_count = 1, max_width = 4096, max_height = 4096, health_media_type = "image/png", health_image_base64 = "iVBORw0KGgoAAAANSUhEUgAAAOAAAADgAQAAAAAyVldMAAAAHUlEQVRYw+3BMQEAAADCoPVPbQZ/oAAAAAAAAHgMGWAAAct2tnAAAAAASUVORK5CYII=", health_image_sha256 = "1b8ab2918e9fb346d8ff5b7579372510b0d8b1566d590dac31aea311fff15fc6", health_prompt = "Name the single dominant color using one lowercase word.", health_expected_text = "black", health_max_tokens = 32, health_disable_thinking = true }
startup_protocol_probe = true
sampling = { temperature = 1.0, top_p = 0.95, top_k = 20, min_p = 0.0, presence_penalty = 0.0, repetition_penalty = 1.0, thinking_token_budget = 8192, reasoning_effort_map = { low = "low", medium = "medium", high = "xhigh", max = "xhigh" } }
2 changes: 1 addition & 1 deletion configs/sparkplane/models.toml
Original file line number Diff line number Diff line change
Expand Up @@ -619,7 +619,7 @@ revision = "7b719225242aacd3dbd3f9407468c2ee9a9d2594"
[models.artifact]
format = "safetensors"
quantization = "NVFP4"
capabilities = ["reasoning", "text_generation", "tool_calling"]
capabilities = ["reasoning", "text_generation", "tool_calling", "vision"]
engine_profile = "qwen3.8-flash-next-nvfp4"

[models.artifact.primary]
Expand Down
203 changes: 199 additions & 4 deletions docs/how-to/develop-spark.md
Original file line number Diff line number Diff line change
Expand Up @@ -21,15 +21,16 @@ commit revision.

The laptop is the control plane. `sparkplane <host> ...` talks to the remote
agent over its pinned TLS endpoint; it does not run Docker commands or copy
arbitrary files over SSH. SSH/SFTP is used only by the signed bootstrap and
upgrade protocol.
arbitrary files over SSH. SSH/SFTP handles signed bootstrap, upgrade and recovery;
the workstation update helper also reads the installed public release key over SSH.

## Repository map

| Area | Purpose |
| --- | --- |
| `.github/workflows/spark-release.yml` | CI contract, ARM64 build, packaging, and tagged signing |
| `scripts/package-spark-release.sh` | Creates the exact release bundle and checksum manifest |
| `scripts/update-spark.py` | Builds, signs, previews and applies a local workstation update |
| `configs/sparkplane/agent.toml`, `executor.toml` | Listener, paths, resource reserves, and pinned host policy |
| `configs/sparkplane/engines/*.toml` | Digest-pinned runtime recipes, matchers, routes, and profiles |
| `configs/sparkplane/models.toml` | Reproducible model identities and artifact metadata |
Expand Down Expand Up @@ -61,6 +62,82 @@ The tests parse and validate the real files under `configs/`; do not replace
them with a second test fixture. A new workload or protocol path also needs a
black-box test against the running daemon where practical.

## Update a Spark from this checkout

Use the workstation helper for a source update. It requires Linux x86-64 or
ARM64, Python 3.11+, the development dependencies above, Cargo Auditable,
Cargo Zigbuild, Zig, the Rust ARM64 target, Minisign, and a working OpenSSH alias
with an existing Sparkplane installation and authenticated client configuration.
The [workstation signing authority](#reuse-the-workstation-signing-credential)
must already exist; the helper does not provision or replace keys.

```bash
# Build, sign and preview the exact candidate; engines continue serving.
python3 scripts/update-spark.py dgx-spark
# Equivalent convenience target:
make update HOST=dgx-spark

# Build, preview and apply, then install the verified workstation client.
python3 scripts/update-spark.py dgx-spark --apply
```

Preparation runs `make lint`, `make test`, `make test-client` and `make audit`,
builds an auditable native client and ARM64 appliance, and packages the shipped
catalogs using the existing release packager. Each run uses fresh directories
under ignored `target/spark-updates/`; it never edits an earlier signed bundle.
The outer signed manifest covers the native client, build metadata, public key,
and signed appliance manifest. Both inventories and signatures are verified
against the independently pinned key before the bundled client executes.

By default the key is `release.key`, the pinned public key is `release.pub`, and
the encrypted user credential is `password.cred`, all under
`${XDG_CONFIG_HOME:-$HOME/.config}/sparkplane-release-signing/`. Use
`--signing-dir` and `--public-key` for other established paths. When
`password.cred` exists, the helper decrypts it with the user-scoped credential
name `sparkplane-release-password` and feeds Minisign through stdin. The password
stays in process memory and stdin; it is never an argument, environment variable,
log or release file. Without that credential, Minisign prompts in your terminal.
SSH and remote sudo authentication use their existing interactive prompts.

The helper compares the pinned authority with
`/opt/sparkplane/current/minisign.pub` over authenticated SSH and rejects a
mismatch. `--apply` performs the managed signed `upgrade` only after the exact
candidate dry-run succeeds. Afterwards it checks agent/executor versions,
semantic health, `doctor`, and the running instances' generation, model,
artifacts, engine fingerprints, resources and context. It then atomically
updates `~/.local/bin/sparkplane` to a verified client in a fresh digest-named
release directory. The preceding client remains available, and the published
client updater's `current` pointer is unchanged. An existing regular executable
at that path is a conflict; use `--no-install-client` to retain it.

The preview prints its signed release path. Reuse that exact release to apply
without rebuilding or accessing the signing secret; the helper verifies it and
performs a fresh host preview:

```bash
python3 scripts/update-spark.py dgx-spark \
--prepared target/spark-updates/release-<id> --apply
```

Use `--json` for one `sparkplane.workstation-update/v1` result on stdout; progress
and authentication prompts go to stderr. Check `state`, `appliance_applied`,
`release` and `evidence`. Every run retains private check logs, before/after
status and instance inventories, the dry-run, activation result and final
result. `--evidence-dir` changes the parent of that fresh private run directory;
`--config-dir` selects an existing Sparkplane client configuration directory.
If interrupted or if verification fails after activation, inspect the evidence
and `sparkplane <host> status --json` before choosing the documented recovery
operation. A failed result does not imply the appliance was unchanged.

This helper updates the control plane and workstation client while preserving
running engines. For changes requiring a new engine generation or refreshed
model traits, follow [capability activation](#activate-a-capability-change-on-an-existing-instance)
and use the helper's signed `appliance/` bundle at the upgrade step. After a
healthy instance already exposes its new capabilities, add
`--codex-model qwen3.8:flash-next` to an apply to regenerate the managed Codex
profile and model catalog. Ownership conflicts still fail through the normal
launch guard.

## Build the ARM64 executable

Install the target and the tools used by CI (`cargo-zigbuild` is preferred for
Expand All @@ -75,7 +152,7 @@ python3 -m pip install --user ziglang==0.16.0
Build locally with the same feature boundary as Spark:

```bash
cargo zigbuild --release \
cargo auditable zigbuild --locked --release \
--target aarch64-unknown-linux-gnu \
--no-default-features --features appliance --bin sparkplane
```
Expand Down Expand Up @@ -125,6 +202,35 @@ is pinned in the operator's local Spark configuration. A manually dispatched
workflow builds and uploads an artifact but deliberately does not sign it;
only a `v*` tag invokes the CI signing step.

### Reuse the workstation signing credential

The existing workstation authority may already be provisioned under
`~/.config/sparkplane-release-signing/`. Its encrypted `release.key` uses the
user-scoped systemd credential `password.cred`, whose embedded name is
`sparkplane-release-password`. Reuse this authority; do not run key generation
again for a local activation. Confirm its public key matches the installed,
trusted authority before signing.

Feed the decrypted credential directly to Minisign, with shell tracing disabled
and pipeline failures enabled:

```bash
set +x
set -o pipefail
sparkplane_signing_dir="${XDG_CONFIG_HOME:-$HOME/.config}/sparkplane-release-signing"
systemd-creds --user --name=sparkplane-release-password \
decrypt "$sparkplane_signing_dir/password.cred" - |
minisign -Sm target/spark-release/SHA256SUMS \
-s "$sparkplane_signing_dir/release.key"
minisign -Vm target/spark-release/SHA256SUMS \
-p "$sparkplane_signing_dir/release.pub"
```

This requires the same workstation user and credential encryption key used at
provisioning. The plaintext password stays in the pipe. SSH and remote sudo
authentication remain separate and use the existing OpenSSH transport; do not
put those credentials in command arguments, documentation, or release assets.

### Provision the release authority

Generate the dedicated encrypted key in your own terminal. Store its password
Expand Down Expand Up @@ -194,7 +300,9 @@ sparkplane dgx-spark install --yes \
For an installed host, use the side-by-side upgrade protocol:

```bash
sparkplane dgx-spark upgrade --dry-run --json
sparkplane dgx-spark upgrade --dry-run --json \
--probe release/sparkplane-aarch64 \
--release-manifest release/SHA256SUMS
sparkplane dgx-spark upgrade --yes \
--probe release/sparkplane-aarch64 \
--release-manifest release/SHA256SUMS \
Expand Down Expand Up @@ -237,6 +345,93 @@ Certificate rotation is separate from a software release:
`sparkplane dgx-spark cert rotate --dry-run --json`, followed by `--yes` after
review. Use `--ca` only when replacing the certificate authority.

### Activate a capability change on an existing instance

A control-plane-only upgrade can preserve a running engine. Changing an engine
profile or stored model capability requires a managed stop and a new serving
generation. Existing model records retain their verified artifact traits;
deploying a new `models.toml` does not update those records automatically.

For the Qwen3.8 image-input correction, build both the appliance and the
workstation client from the tested source. The client generates Codex's model
catalog, while the appliance publishes the live route's supported modalities:

```bash
cargo auditable build --locked --release --bin sparkplane
```

Use `target/release/sparkplane` for the following commands if the client on
`PATH` has not been updated. Package, sign, verify, and dry-run the ARM64 release
before stopping the model. Save `ps --json`, including the instance name,
generation, immutable model, image/profile fingerprints, resources and context.
Confirm the workload has no running or queued requests before the restart.

```bash
sparkplane dgx-spark ps --json
sparkplane dgx-spark stop qwen38-vllm --json
sparkplane dgx-spark upgrade --yes --json \
--probe release/sparkplane-aarch64 \
--release-manifest release/SHA256SUMS \
--release-signature release/SHA256SUMS.minisig \
--release-public-key "$sparkplane_signing_dir/release.pub"
sparkplane dgx-spark download qwen3.8:flash-next --dry-run --json
sparkplane dgx-spark download qwen3.8:flash-next --update-alias --detach --json
sparkplane dgx-spark operations <download-operation-id> --follow --json
sparkplane dgx-spark serve qwen3.8:flash-next --name qwen38-vllm --dry-run --json
sparkplane dgx-spark serve qwen3.8:flash-next --name qwen38-vllm --detach --json
sparkplane dgx-spark operations <serve-operation-id> --follow --json
sparkplane dgx-spark ps --json
```

The download operation resolves the signed catalog's immutable revision and
reverifies the existing native cache. With a complete cache, the dry-run reports
`unique_bytes: 0` and no model weight transfer is needed. Wait for each managed
operation to succeed before advancing. Retain the previous signed release and
image for rollback; do not edit the live database, catalogs, or container.

Hub metadata resolution can exceed the workstation host profile's default
`request_timeout_seconds = 30`, even when all weights are cached. A dry-run
timeout does not prove the cache is missing. Increase that host's request timeout
for a bounded retry of the plan. Detached refresh and serving operations return
their identifiers immediately. Following uses the same request timeout; if it
reports that the operation is still running, reconnect with the same identifier
instead of submitting another download or serve request. Advance only after
`state: "succeeded"`. A complete cache reports zero transfer bytes, but its
existing files still undergo checksum verification, which can take minutes.
Refreshing capability traits also changes the artifact fingerprint and selects
a separate compile-cache namespace. Allow the normal cold start, compilation,
and semantic probes to finish; retain the previous namespace for rollback.

For this correction, the engine image, launch arguments, resource envelope,
262,144-token context, BF16 KV and MTP settings stay unchanged. The new profile
adds a bounded image policy and semantic image health probe. After the new
generation becomes healthy, `ps --json` must advertise
`input_modalities: ["text", "image"]`. Verify an inline-image request through the
authenticated public gateway, then regenerate the owned client configuration:

```bash
sparkplane dgx-spark launch codex --model qwen3.8:flash-next --config --json
sparkplane dgx-spark launch codex --model qwen3.8:flash-next
```

The generated `sparkplane-launch-models.json` must include both `text` and
`image`. Relaunch Codex so it reads that catalog. Do not hand-edit the catalog
or its ownership receipt to make an old session accept attachments.

If regeneration reports an ownership conflict, preserve a private backup of the
affected profile and its receipt, move the indicated profile aside, and rerun
`launch ... --config --json`. Additional tables such as `[tui]` in an earlier
profile can cause this conflict. The launcher regenerates both owned files and
their receipt through the normal publication path.

Verify an actual attachment with a small test image:

```bash
sparkplane dgx-spark launch codex --model qwen3.8:flash-next --mode inherit -- \
exec --sandbox read-only --image ./test-image.png -- \
"Describe the attached image."
```

## Add an engine

Add a complete TOML file under `configs/sparkplane/engines/`. Start from the
Expand Down
19 changes: 18 additions & 1 deletion docs/how-to/install-spark.md
Original file line number Diff line number Diff line change
Expand Up @@ -73,11 +73,21 @@ You can now [serve a model](serve-a-model-on-spark.md).

## Upgrade or recover the control plane

For a workstation source checkout, the [update helper](develop-spark.md#update-a-spark-from-this-checkout)
automates the checks, build, signing, preview, activation and local client update:

```bash
python3 scripts/update-spark.py dgx-spark # build and preview
python3 scripts/update-spark.py dgx-spark --apply # build, preview and apply
```

Preview and approve the same signed ARM64 artifact transaction used at first
install:

```bash
sparkplane dgx-spark upgrade --dry-run --json
sparkplane dgx-spark upgrade --dry-run --json \
--probe release/sparkplane-aarch64 \
--release-manifest release/SHA256SUMS
sparkplane dgx-spark upgrade --yes \
--probe release/sparkplane-aarch64 \
--release-manifest release/SHA256SUMS \
Expand All @@ -91,6 +101,13 @@ and the protected DGX fingerprint. It changes only the control-plane release and
keeps healthy engine containers running. Failed semantic health requests
automatic rollback.

An engine profile or model capability change also needs a managed stop and
re-serve. Existing model records keep their original artifact traits, so refresh
them through the managed download operation using the new signed catalog.
Follow [activating a capability change](develop-spark.md#activate-a-capability-change-on-an-existing-instance)
for the complete sequence, including reuse of cached weights and regeneration
of Codex's image-capable model catalog.

Use the SSH-only recovery path even when HTTPS is unavailable:

```bash
Expand Down
23 changes: 20 additions & 3 deletions docs/reference/spark.md
Original file line number Diff line number Diff line change
Expand Up @@ -239,9 +239,12 @@ host/integration selection is reused; an interactive terminal can select from
installed models. `--config` writes only launch-owned state/config and exits.
`--dry-run` performs no local or remote mutation. `--json` is valid with either
of those non-agent modes. Generated files have a private digest-based ownership
receipt. Launch and restore verify both files before changing either; a user edit,
symlink or unowned destination is a conflict, not permission to overwrite it.
Move a conflicting file to your own backup location before regenerating it.
receipt. Launch and restore verify both files before changing either. Codex may
write its own settings, such as `approvals_reviewer`, into the profile; those
extra lines do not block the next launch, and the profile is rewritten from the
managed route. A change to a managed line, a symlink, or an unowned destination
is a conflict. The error prints the path; move that file aside before retrying.
There is no `--force`.
Interrupted publication can be retried safely. `--restore` removes only Sparkplane-owned Codex
profile/catalog files. `-y` permits the fixed Claude or OpenCode installer when
the executable is absent. Only arguments after `--` are forwarded, without a
Expand All @@ -253,6 +256,20 @@ inference-only token and pinned CA, never the administrator credential. Claude
uses the native Anthropic route, Codex uses a Sparkplane-owned Responses profile and
catalog, and OpenCode uses process-local `OPENCODE_CONFIG_CONTENT`.

Codex image attachments follow `input_modalities` reported by the healthy
instance's published gateway route. Image support requires the verified model
and engine vision policy; a vision-capable checkpoint on a text-only route
still accepts only text. Older agents that omit this field default to text.
Relaunch Codex after changing the qualified instance so its catalog is refreshed.
Existing model records retain their verified artifact traits across upgrades.
When a catalog adds vision support, refresh the record through the managed
`download --update-alias` operation and re-serve it with the updated engine
policy before relaunching Codex.
The Qwen3.8 vLLM policy accepts one inline PNG, JPEG, or WebP image per request,
at most 512 KiB of image data, with width and height at most 4096 pixels.
See the [live capability activation procedure](../how-to/develop-spark.md#activate-a-capability-change-on-an-existing-instance)
for signing, cached-model refresh, managed restart, and attachment verification.

### Internet access for development

Launches default to `--mode auto` (`SPARKPLANE_LAUNCH_MODE=auto`): full
Expand Down
Loading
Loading