Conversation
Lay the foundation for sharing a `signing` secret across a horizontally
scaled enclave fleet: the master holds the full secret and acts as a
threshold-sdk leader, and same-image joiner enclaves attest in to receive
their shares. Untagged/static secrets are unchanged.
Config:
- Add `kind` to secrets (static|signing) with validation, thread it through
the CLI build config and enclave.yaml scaffolding.
Runtime:
- ScalingEntity (scaling.go): role-gated leader/follower, attested admission
handshake (/enclave/handshake/{nonce,admit}) with mutual PCR verification,
one-shot nonces, and a NaCl-sealed reply; loads/generates the threshold
host key via KMS. Relay config is port-only (ENCLAVE_SCALING_LISTEN_PORT /
ENCLAVE_SCALING_LEADER_ADDR), bound on the TAP address.
- DkgStore (chilldkg.go): thresholdsdk.Store over encrypted Storage; persists
only public recovery blobs and returns the master's KMS secret as a0, so the
reconstructed group secret equals the normally-created secret.
- nitriding.ArePCRsIdenticalForKeys: compare only code-identity PCRs (0,1,2)
so same-image enclaves on different EC2 instances admit each other.
- KMS.decryptToBytes: attestation-gated decrypt to raw plaintext.
- StaticSecrets: wire the scaling subsystem, add the follower key-derivation
path, and skip signing secrets in ExtendPCRs.
- Build ScalingEntity in Init when ENCLAVE_THRESHOLD_ROLE is set; mount the
leader admission routes on the public listener.
Wire the scaling subsystem end to end so leader and follower enclaves hold and reload their threshold signing material across reboots. - Rename ENCLAVE_THRESHOLD_ROLE -> ENCLAVE_SCALING_ROLE (code + design doc). - Persist derived shares: both roles drain the SDK Keys() channel into StoreDerivedSecret, which bundles the master (a0) and this node's share into one KMS-encrypted envelope per secret. A later boot loads both from the single SSM slot — EnvVar=master + ShareEnvVar=share on a leader, EnvVar=share on a follower — instead of re-running a ceremony. - Pin the group key: DkgStore.GetLeaderSecret resolves a0 from the master's env var, so the group forms around the KMS-managed secret rather than a freshly generated one. Drop the no-op PutLeaderSecret (the SDK now owns a0's lifecycle through GetLeaderSecret alone). - Wire ScalingEntity.kms and move scaling Init after EnsureKeyID so the KMS-backed host key can be generated. - Build the scaling entity in New so a leader's admission handshake routes actually mount (gated on IsLeader); AWS-backed bring-up stays in Init.
…e scaling ceremony Builds on the persisted-share scaffolding to run a 1-leader + N-follower threshold ceremony across single-host enclaves, with re-randomized shares taking effect in the already-running app. Runtime: - Re-fork the child app when a reshare re-materializes its share. A share reaches the app only through the env var it inherits at fork, so a post-init reshare isn't visible live; the share handler now signals main to restart the app once it has started. Collapse the app lifecycle into one upstreamState (NotStarted -> Running -> Exited), replacing the started/exited booleans. - Initialize Storage before the scaling entity, so the leader can persist DKG recovery blobs the instant a follower joins (was initialized last, racing the handshake coming up). - Construct the scaling entity in Init from the SSM-overlaid role and mount the leader handshake routes unconditionally, gated per-request by readyLeader. - Reshare only "signing" secrets on a follower; keep the leader master (a0) in an in-memory cache so it survives once the env var holds the node's share. Drop ShareEnvVar -- one env var per secret. - Skip TLS PKI on the follower->leader handshake (trust is mutual NSM attestation) and read local PCRs without strict verification in dev. - Add a per-instance kernel-cmdline override channel (devcmdline) so same-image nodes get their deployment / role / vsock ports without changing PCRs. Test: - run-scaling.sh boots the ceremony on one host (per-node vsock CID, gvproxy port, SSM namespace); scaling-test.sh asserts three distinct valid shares and that a new join re-randomizes an incumbent. seed-scaling.sh provisions the per-deployment KMS placeholder + storage bucket (mirroring tofu). Adds `make test-scaling` + a docker-compose scaling-runner; /test/scaling exposes each node's share pubkey.
…aling CI
Replace the machine-local 'replace => /home/...' on the threshold SDK/core with versioned requires on the now-public modules github.com/ArkLabsHQ/threshold-magic/threshold_{sdk,core}@v0.1.0, so the runtime module resolves anywhere (CI, nix, fresh checkouts) instead of only on one dev's machine.
Add .github/workflows/scaling-test.yml: boots the 1-leader + 2-follower ceremony on ubuntu-latest (KVM + vhost-device-vsock), mirroring 'make test-scaling'. Validation-phase triggers (workflow_dispatch + push:[dkg]) until proven green on GitHub's kernel.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
…improve related comments - enhance metrics struct for clarity and consistency - adjust scaling logic to reference leader secret instead of master test(runtime): modify tests to reflect changes in leader secret handling docs: add Threshold Scaling Architecture documentation
…/follower system details
Add autonomous membership revocation to the threshold-scaling cluster. The leader continuously monitors follower liveness and automatically evicts failed members, triggering a resharing of all secrets among the remaining quorum. This closes the "removal requires a leave-triggered reshare" gap documented in SCALING-ARCHITECTURE.md. Mechanism: - Followers periodically sign a fresh leader nonce with their threshold host key (BIP-340 Schnorr) and POST it to /enclave/heartbeat. - The leader verifies the signature and nonce, recording each follower's last-seen timestamp. - A background monitor evicts followers that exceed ENCLAVE_SCALING_LIVENESS_TIMEOUT. - Eviction invokes Leader.Ban, removing the member, rekeying all secrets, and invalidating the follower's share while preserving the threshold public key. - Bans are persisted in encrypted storage and enforced during both authorization and attested admission, preventing re-admission across leader restarts. Heartbeat intervals and liveness timeouts are configurable via the enclavecfg.* command-line whitelist. Also simplifies the follower join path by inlining followerJoinHandshake into startFollower, and adds comprehensive unit and end-to-end tests covering heartbeat authentication, automatic eviction, durable ban enforcement, and resharing after follower failure.
|
This PR has been open for 34 days without a review decision. @aruokhai is anyone looking at this? |
arkana-ai-bot
left a comment
arkana-ai-bot
left a comment
There was a problem hiding this comment.
Review — Leader/Follower Static Secret Sharing (DKG)
Summary
Adds threshold cryptography for enclave signing: a signing secret is shared amongst N enclaves, with a quorum required to sign, while the group public key stays pinned to the original secret across resharing.
Areas requiring human expert review
-
DKG correctness — Uses
threshold_sdk@v0.1.0/threshold_core@v0.1.0. These are v0.1.0 public modules with no track record. The DKG ceremony correctness (especiallya0leader secret re-use across reshares to pin the group key) should be reviewed by a cryptographer. -
Attested admission — Followers are admitted via mutually attested handshake over TLS. The PCR check (
nitriding.ArePCRsIdenticalForKeys) must be reviewed carefully — a lenient PCR check would allow a compromised node to join the quorum. -
Share persistence atomicity — Leader writes
{leader_secret, share}atomically to SSM. If the write fails between the two fields, the node has an inconsistent state. Verify error handling covers this. -
Group restart on reshare — The lifecycle triggers
RequestUpstreamRestart(SIGTERM → 10s → re-fork) on new share receipt. Verify that in-flight signing operations are drained before restart, and that a failed restart doesn't leave the signer in a non-responsive state while still holding a valid share. -
Dev mode kernel-cmdline —
devcmdline.gomaps kernel-cmdline tokens to env vars in dev mode. Verify this path is entirely gated on the measuredENCLAVE_DEVflag and cannot be activated in production images. -
No quorum = no signing — If the threshold is t-of-n and fewer than t nodes are live, signing halts. This should be documented as an availability tradeoff — especially relevant if funds are gated on enclave signing.
This needs expert cryptographic review before merge.
|
This PR has been open for 5+ weeks without review. @aruokhai this is a security-critical cryptographic feature (threshold signing DKG). Is there a reviewer assigned? |
|
This PR has been open for 41 days without a review decision. @aruokhai is anyone looking at this? |
|
This PR has been open for 44+ days without review. @aruokhai is anyone looking at this? |
|
This PR has been open for weeks without review. @aruokhai is anyone looking at this? |
|
This PR has been open for 44+ days without review. @aruokhai is anyone looking at this? |
|
This PR has been open for 49 days without a review decision. @aruokhai is anyone looking at this? The Leader/Follower Secret Sharing design looks significant — would benefit from a dedicated reviewer. |
2 participants
Leader/Follower threshold-shared static secrets
Adds a threshold-cryptography scaling model for enclave apps. A secret tagged
kind: signingis shared amongst enclaves: each node keeps only a share, a quorum is required to sign, and thegroup public key stays pinned to the original secret as nodes join or leave. Ordinary
secrets and single-node deployments are unchanged (opt-in).
Full design:
SCALING-ARCHITECTURE.md.What's in scope
ScalingEntity(runtime/scaling.go), built inRuntime.Initonly whenENCLAVE_SCALING_ROLEis set (nilotherwise). Leader runssdk.NewLeader+ the gRPCrelay; follower runs
sdk.NewJoiningFollower. Both hold a KMS-persisted secp256k1 hostidentity key.
DkgStore(runtime/chilldkg.go) adaptssdk.Storeontoencrypted
Storage;GetLeaderSecretreturns the leader's original secret (a0) soevery ceremony reconstructs the same group key.
port:
GET /enclave/handshake/noncethenPOST /enclave/handshake/admit. The leaderverifies the COSE signature and PCRs 0/1/2 (
nitriding.ArePCRsIdenticalForKeys), thenseals
{leaderPub, relayPort}to the follower's ephemeral key. The admitted host keybacks
sdk.WithAuthorize, so the relay rejects unadmitted keys.a0; all shares re-randomize, thegroup key holds. Results arrive on
Keys(); a listener persists the sealed envelope,sets the secret's env var, and triggers a graceful upstream restart
(
RequestUpstreamRestart→ SIGTERM/10s → re-fork) so the app adopts the new share.(
sharedSecret{leader_secret, share}); leader writes both atomically, followers onlyshare.runtime/devcmdline.gomaps whitelistedenclavecfg.*kernel-cmdlinetokens to env vars (no-op unless the measured
ENCLAVE_DEVflag is set), lettingidentical-PCR QEMU enclaves take distinct roles.
Testing
.github/workflows/scaling-test.yml,test/run-scaling.sh,test/scaling-test.sh. Boots 1 leader + 2 followers of the same EIF on QEMU/KVM andasserts: nodes healthy/role-correct, handshake live, every node holds a share (not
the leader secret), the 2nd join re-randomizes the 1st follower's share, all shares
distinct valid scalars.
Notes
threshold_sdk@v0.1.0/threshold_core@v0.1.0as public versioned modules.