Skip to content

Design durable mailbox conversations and workload continuation #5126

Open

Description

@auser

Status and outcome

Proposed design for review; not an implementation, accepted ADR amendment, or security/readiness claim. Define MVM-owned durable conversations and explicit workload continuation for user, workload, and service messaging, including a minimal private encrypted context space. A single host must work without MVMD, an external account, or a separately deployed service.

This issue is the design discussion and tracking record; it does not replace the mailbox program's implementation issues or duplicate their status. Reconcile with ADR-057, the proposed ADR-047, and the external-credentials boundary in ADR-023. Propose explicit amendments or a superseding decision if needed; do not silently change accepted semantics.

Related: local mailbox service #4629, canonical envelope #4628, encrypted journal #4630, movement and fencing #4631, and optional fleet integration mvmd#254 and mvmd#266. These references do not imply those features are complete.

User scenario

A workload asks an authorized user a question; the user can answer later through an encrypted mailbox. The workload's guest may stop or restart while waiting. A newly authorized replacement can receive the still-valid reply and explicitly continue its task from retained state. Local users and multiple local microVMs must be able to communicate without a fleet coordinator or guest NIC. MVM also supports one-way cast messages; a cast carrying a trace is not automatically authoritative audit evidence.

Proposed ownership and boundaries

  • MVM owns the canonical mailbox and context contracts, local admission, identity and capability verification, key lifecycle, storage and recovery, guest/host/user APIs, and final local refusal. Standalone MVM must not depend on MVMD types, process, fleet identity, or an external storage/key service.
  • MVMD is an optional fleet layer for membership, policy administration, placement, discovery, route hints and coordination. It may carry opaque envelopes, but does not hold context/mailbox decryption keys, read payloads, mint destination receipts or override host refusal.
  • Contracts use transport-neutral MVM identities and bounded values; runtime, storage, cryptographic and coordinator implementations stay outside the DTO contract. External credentials such as API keys and passwords never enter the guest under ADR-023. Internal mailbox/context keys are sensitive but may be released narrowly to an authorized guest endpoint; they do not authorize unrelated actions.
  • Ciphertext stores, ordinary routers and MVMD do not receive payload plaintext or usable decryption keys. Plaintext can exist at authorized workload/user endpoints. Ordinary virtualization does not protect guest memory against a malicious host or hypervisor. Logs, audits and errors must not contain plaintext, credentials or keys; disclose and bound necessary metadata.

Proposed conversation and continuation semantics

  • Preserve the existing call, cast, and reply contract. Calls expect a correlated reply; casts expect no application reply and report the declared admission level. Distinguish local admission, durable destination acceptance, delivery, application processing and acknowledgement. A cast is not implicitly lossy.
  • For a durable conversation, the logical workload and mailbox survive replacement, while each boot and execution epoch is independently authenticated and authorized. A pending, valid reply can follow an explicitly authorized replacement; exact in-memory call-stack restoration is not promised. Workload or SDK persists continuation state and resumes explicitly. Invocation-pinned messages and cancellation policy remain open.
  • Each message has bounded expiry with explicit default and maximum to be decided. Calls have a separate waiting deadline; timeout does not prove non-execution or cancel an already-started effect. Expired, late, refused, exhausted and acknowledged outcomes must remain distinguishable, subject to retention bounds. Define races between expiration, delivery, processing and acknowledgement.
  • Delivery is at least once, not generally exactly once. Reuse stable message and conversation IDs across retries, with bounded recipient deduplication; application effects require their own idempotency. Use bounded backoff for retryable failures, attempt and time limits, bounded queues and explicit terminal outcomes. Refuse non-retryable authorization or malformed-input errors rather than silently retrying forever.
  • V1 target: durable accepted data survives guest replacement while its local backing storage and keys remain available. Permanent loss of that host or disk may lose it. State the actual failure domain when reporting acceptance. Preserve a future route to optional replica-based host-loss protection without claiming that P2P transport itself replicates data or creates consensus.

Proposed atomic local continuation and send intent

A continuation record and an encrypted send intent for a stable conversation/message identity should be committed atomically in the local durability domain, after required authorization and resource admission. A durable outbox can then deliver using the same identity, and recovery can reconcile a committed pending send after a crash. Remote delivery and replies are not part of that transaction. A caller that loses the commit response can query/retry with a stable operation identity and payload binding; conflicting reuse must fail.

This is a target, not an established capability. Demonstrate whether a single local transaction can preserve independent mailbox queue/credit limits, local refusal, capability/epoch verification, crash safety and opaque storage. If it cannot, compare a bounded recoverable intent protocol with explicit failure states; do not assume cross-component atomicity or bypass admission. Define safe ordering for reply consumption, continuation update and ACK; crashes can still cause duplicate processing or external effects.

Reconcile especially with ADR-057's rejection of old-epoch messages and capabilities: specify whether accepted pending ciphertext is retained, safely reauthorized, or given a terminal disposition on replacement without rewriting authenticated fields, making stale consumers valid, or creating a competing envelope. Require a current, authorized execution before key release or delivery; a previous guest may retain keys/plaintext it already saw, so revocation needs fencing and key lifecycle, not a claim that old knowledge disappears.

Minimal context space and future pool

Continuation state is the first use of a private, bounded, encrypted context space addressed to a stable task/workload identity rather than a guest boot. Mailboxes remain independently usable without a context pool. The design should allow, but V1 need not implement, separately authorized shared spaces for non-agent workloads as well as agents. Shared membership does not grant read access.

MVM owns local context identities, policy and capability enforcement; optional MVMD can administer fleet-wide roles and grant coordination without becoming a plaintext reader. Separate administrative roles from content read/write/share/delete rights. Workload read/write permissions do not imply policy administration. Capture bounded attributable provenance and trust classification without logging plaintext; remembered content is data, not authority, and cannot grant capabilities or bypass a fresh execution plan.

Search by a plaintext-blind service is not assumed. Define endpoint-side search or a separately reviewed encrypted search design, including metadata leakage, resource bounds and revocation behavior. Rich cross-workload search and replication are not prerequisites for initial private continuation.

Key custody profiles to design

  • Default trusted standalone host: key handling is built into MVM's local runtime, not a separately deployed service. Protected local key material and scoped release to a newly authorized guest allow unattended recovery. Host administrator/hypervisor confidentiality is not claimed.
  • Optional stronger profile: attest guest identity and release keys only against verified evidence and an appropriate trust root. An external authority is a configurable option, not a default dependency; explicitly requested assurance fails closed when unavailable. Attestation alone is not key custody or proof of a protected guest.
  • Define what is stored, key wrapping, owner authorization, scope, user endpoint keys, backup/recovery, key loss, rotation, revoked or previously compromised recipients, rollback, auditing and platform support. No fleet-wide master key to guests and no silent profile downgrade. Do not claim hardware-backed, FIPS, malicious-host, or unattended cross-host guarantees without their own witnesses.

Design decisions and acceptance evidence

  • Produce and review the proposed state machines and acceptance levels for local send, recipient acceptance, reply, timeout, expiry, cancellation, retry exhaustion and restart, including crash points and duplicate ACK/reply behavior.
  • Resolve the ADR-057 epoch/old-message and key-handoff conflict; compare authorized handoff against strict assignment pinning and define a fail-closed fallback. Separate pending-message continuity from exact execution resumption.
  • Specify private context scope, MVM standalone human/workload access, roles versus scoped capabilities, key custody profiles, metadata/provenance, retention and revocation; reconcile proposed ADR-047 and ADR-023. Identify the trust boundary for plaintext and applicable threat model.
  • Perform a bounded source-pinned spike or equivalent proof of feasibility for atomic continuation plus encrypted send intent, including mailbox admission and quotas. Document the alternative if atomicity cannot be provided safely.
  • Define conformance tests for standalone host/user/guest operation without MVMD; authorized replacement after restart; duplicate/lost responses; commit interruption at each boundary; stale boot/epoch and partition; wrong tenant, replay, tamper, revoked/expired grant, timeout/expiry race, oversized values, quota exhaustion, key loss and rollback; verify no plaintext in stores/logs and no external credentials in guests.
  • Define explicit retention, expiry, retry and storage failure-domain behavior and a testable policy for unknown outcomes after crash or permanent host loss.
  • Link the agreed design to the relevant existing implementation issues and propose ADR amendments or a successor decision if review changes accepted semantics. Keep completed evidence separate from planned work.

Open product decisions

Which conversations, if any, are invocation-pinned rather than transferable? What are expiry/retention defaults and the scope of cancellation? What precisely does user-facing acceptance promise? How are owner/user keys provisioned, recovered and revoked? What is the smallest private context API, and which kinds of context may safely be exposed to a workload? How will a replacement be reauthorized and old ciphertext handled after policy or key changes?

Non-goals

Transparent checkpoint restoration; general exactly-once external effects; a full shared memory/search platform; production replication or consensus; public relay/third-party dependencies; or claiming host-blind confidentiality without the applicable platform and evidence.

Activity

  1. added
    mailboxesSigned pass-by-value mailbox program
    on Nov 5, 2026
  2. auser commented on Nov 5, 2026

    @auser
    CollaboratorAuthor

    Read-only design reconciliation (proposal, not accepted amendment or readiness claim)

    Source pins: MVM e45c787a94 (local main); MVMD 50d64da (local main). Status comes from #4622/#4627–#4632, #4628/#4629 and SpiralCore#42/#43, not from ADR implementation labels or third-party component tests.

    Established. Accepted ADR-057 separates stable principal/mailbox, fresh boot and incremented execution epoch; older-epoch messages/capabilities are refused after activation. It fixes call/cast/reply, distinct ephemeral/durable destination acceptance, at-least-once stable-ID retry, finite admission, local-only operation, and tiered plaintext boundaries (delivery, isolation). Merged #4694/#4698 establish canonical mvm-contract envelope/crypto and bounded transport seam, not a live standalone mailbox service: header, open/replay seam, transport gate. #4629 is open; #4630 journal and #4631 movement remain blocked by their stated dependencies. MVMD still has its own prototype assignment-generation/boot-fenced queue (service, DTO); it is not the MVM service or transferable canonical ciphertext. #254/#266 own optional fleet integration/design, not local authority.

    Conflict requiring a reviewed ADR-057 amendment/successor. Today verify_identity_and_time compares the signed sender execution epoch against the current OpenContext epoch, and verify_capability_scope checks current-epoch capability and policy. A pending reply signed under an old epoch cannot simply be reopened under the new epoch; changing the context or rewriting signed fields would evade fencing. The signed header has a sender boot/epoch but no independent authenticated destination-acceptance epoch. Proposal for review: distinguish historical message validity/immutable acceptance from fresh delivery and key-release authority. Retain original ciphertext, signature, stable IDs, expiry, commitment, dedup and immutable journal history; before a new consumer sees it, prove explicit same-logical-mailbox replacement, durable old-owner fencing, a current receive/reply grant and policy/tenant/assurance checks, plus eligibility of that particular pending conversation. Record a separate authenticated delivery/handoff decision bound to the new consumer epoch without changing the old signed envelope; never treat old sender credentials as current. If such verification, key availability or single-owner fencing cannot be proven, strictly pin to the old assignment and return a terminal/uncertain disposition according to established receipt semantics. This is a proposed versioned extension, not a behavior the existing open_envelope supports. Decide transfer-by-default vs invocation pinning, expiry/cancellation/late-reply races, and whether changing policy revokes old queued ciphertext before amending ADR-057.

    Context/custody. Proposed private task/workload-scoped encrypted continuation records extend proposed ADR-047 beyond agent-session-only, append-only, host-stamped Observed/Derived notes. A mutable continuation checkpoint cannot silently replace its append-only audit and retention rule: specify immutable revisions + bounded collection, separate administrative vs read/write grants, and future explicit sharing (never fleet-membership access). A plaintext-blind store cannot do host-side semantic scan/search; ADR-047's proposed write-time secret-fingerprint scan would either require a trusted endpoint/broker or weaken the ciphertext-only boundary, and its efficacy is limited even there. Accepted ADR-023 forbids external credential values in guests, not narrowly scoped internal decryption keys. #4185 custody contract is architecture, not a deployed mailbox key-release mechanism: trusted standalone host can manage local scoped keys; claimed host-blind release requires separately qualified attestation/external authority and must fail closed. Old guests can retain previously seen keys/plaintext; fencing prevents new authorized effects, not erasure of knowledge. Key backup/loss/rotation, authorized user endpoint, rollback, revocation and no-plaintext logs/stores still need designs and tests. Local durability cannot promise survival of permanent host/disk/key loss.

    Next dependency-safe implementation: continue #4629's dedicated standalone bounded typed vsock call/cast/reply path and independent credits/budgets, without making it wait for #5126 continuation/outbox. Reconcile the still-open #4628 codec-roundtrip and MVMD migration criteria separately, not by copying the canonical code. #5057 now has a coordination request to the active SpiralCore#42/#43 contribution owner for exact pins and patch ownership; do not implement a competing service or adopt component evidence as upstream acceptance. #4630 follows #4629 for journal/recovery; #4631 movement follows journal/fencing.

    Source-pinned evidence to request before a continuation/atomicity claim: at e45c787a94, model OpenContext/open_envelope, ReplayProtector and authorize_outbound (currently Send-only) against two candidate rules: strict epoch-pinned terminal handling versus separate current-epoch authorized delivery of historical ciphertext. Build one tiny executable crash-injection prototype after #4629 admission and #4630 store boundaries are known: one local transaction reserves per-sender/mailbox/tenant/host bytes + dedup identity and commits encrypted versioned continuation revision plus immutable ciphertext send intent; test crashes before/after reserve, commit, response, send, destination receipt and ACK. Retry with identical operation ID+payload binding is duplicate-safe; conflicting reuse refuses, independent receive/reply grants still checked, and failed/oversized/full/revoked/stale-boot/stale-epoch/wrong-tenant/key-loss/rollback cases allocate no uncharged work. Query/reconcile uncertain commit; don't claim exactly-once handler effects. If a single store cannot atomically own all admission charges and intent, compare a bounded recoverable intent protocol with explicit unknown/terminal outcomes rather than advertise an atomic commit. Separate guest-replacement witness: old owner fenced before new activation; pending valid reply delivered only to authenticated current consumer with scoped key; expired/revoked/invocation-pinned items refused; stale guest/route and partition unable to accept effects. Verify plaintext and keys absent from ordinary ciphertext stores, MVMD and logs; do not infer malicious-host protection from ordinary virtualization.

    Open decisions for maintainer approval: transfer default vs pinning; exact new-epoch delivery proof and policy/revocation behavior; key-release/backup profile; context API/retention/trust-class handling; user-visible acceptance and expiry defaults. No issue boxes checked, source changed, or tests run in this read-only pass.

  3. auser commented on Nov 5, 2026

    @auser
    CollaboratorAuthor

    Product direction confirmed: explicit opt-in conversation transfer

    The requester approved continuing with the V1 opt-in direction discussed above. Record this as an agreed product preference, not an accepted ADR-057 amendment, a compatible V1 wire change, or proof that secure transfer works.

    • A durable root/control mailbox may outlive a guest, but durability alone does not make every pending call/reply transferable. A conversation is eligible to follow a replacement only when transfer was explicitly requested under the conversation contract and allowed by the mailbox/workload's signed plan. A sender flag alone is not authorization. Invocation-pinned conversations remain pinned. Decide the exact signed representation/version and user-facing API in the ADR review; current MailboxEnvelopeHeader has no transfer indicator.
    • An eligible pending item still requires an authenticated immutable original envelope/acceptance, unexpired lifetime, current policy and scoped capability for the specific replacement, appropriate key release, and durable old-owner fencing before the new consumer may receive or act. Retain original signed fields, IDs, dedup and history. A separate authenticated current-epoch delivery decision must not cause old send/receive capabilities or stale guests to become current. If evidence of any prerequisite is absent or uncertain, do not hand off; return the appropriate typed terminal or uncertain outcome rather than inventing success.
    • V1 continuation means the replacement workload/SDK explicitly reloads task state; no call-stack or exactly-once effect restoration. Delivery and key access do not erase plaintext or keys previously observed by an old guest. Trusted standalone-host key handling remains default; stronger host-blind claims still require their own attestation/key-custody witnesses.

    Next design slice for review under this issue: propose an ADR-057 amendment or successor that separates historical sender epoch from the current consumer delivery/authorization epoch, chooses a signed/versioned opt-in representation, specifies pre-fence/activate/release ordering and revocation/expiry/policy-change dispositions, and includes the strict-pinning fallback. At current source e45c787a94, verify_identity_and_time equates signed epoch and current OpenContext, so opt-in transfer is not implemented by changing a flag. The source-pinned replacement and crash-interruption spike in the preceding comment remains the evidence gate. #4629 standalone call/cast/reply work continues independently, coordinated through #5057; #4630/#4631 remain later dependencies. No unchecked acceptance box is complete and no security guarantee has been promoted.

  4. auser commented on Nov 10, 2026

    @auser
    CollaboratorAuthor

    Proposed queued-during-rebuild handoff rule — separate sender intent from accepted delivery

    The requested dev UX is a stable logical work session: approve a declarative dependency/configuration change (or use an explicit preapproval policy), build incrementally without mutating the running guest, launch/verify a replacement, then switch sessions while preserving history and durable workspace state. New tasks submitted during the build should look queued without a second shell/exec interface. Production fleet rebuilds are a distinct administrator-controlled MVMD orchestration use case; the MVM host/runtime remains the authority for each local launch, artifact and mailbox admission, epoch fence and refusal. A production guest must not be able to cause its own rebuild by merely accepting an application message.

    Two cases must not be conflated:

    1. Not yet dispatched: hold a bounded, authenticated/encrypted sender-owned outbox intent for the stable logical session, with logical task ID, exact payload binding, target revision/handler/schema expectation, expiry, quotas, and explicit queued_for_rebuild status. This is not a canonical mailbox delivery and not an accepted_ephemeral/accepted_durable destination receipt. Before/after the cutover, validate build result and the new admitted handler/plan, current sender capability, policy/recipient/route and epoch; only the sender may create a new signed canonical envelope for the new authorized destination. The host cannot edit an old signed envelope or impersonate the sender. A client losing the outbox-commit response can query/retry using the same task ID and payload commitment; conflicting reuse refuses. Once dispatched, the outbox records the exact envelope identity and transitions to the canonical mailbox receipt state. Failed build, missing handler, revoked capability, expired intent, full outbox, lost keys, or inaccessible sender must remain visible/pending, cancelled or refused—never silently retarget to the old VM or fabricate completion. The outbox's failure domain and key custody need a reviewed implementation design; V1 may keep it local rather than promise cross-host resilience.
    2. Already sent or accepted under the old epoch: never auto-reissue or replay on rebuild. If a handler's effect/terminal receipt is proved, report it; otherwise return unknown and require reconciliation. Existing Design durable mailbox conversations and workload continuation #5126 product direction permits transfer of accepted pending conversations only with signed opt-in in the conversation and workload plan, verified historical envelope/acceptance, unexpired/revocation-valid eligibility, durable old-owner fencing, current consumer authority and scoped key release. Keep original signed ciphertext/IDs immutable; record a separate authenticated delivery decision for the current consumer epoch. With missing evidence or an invocation-pinned conversation, stay pinned/refuse/emit a truthful terminal or uncertain disposition. No old guest gets to accept new effects after fencing.

    Why this distinction: V1 MailboxEnvelopeHeader.execution_epoch is signed, authorize_outbound binds it to the route epoch, and open_envelope refuses mismatch with OpenContext.current_execution_epoch; verify_capability_scope checks the current epoch against signed grant bounds. Rewriting an old accepted envelope or widening the context to make it open after cutover would break that gate. A sender-held intent has not crossed that boundary, so it can be authenticated and freshly sealed after the replacement is admitted. Distinguishing sender epoch, destination-consumer epoch, and the outbox's stable logical task ID may require a versioned ADR-057 extension; do not fake it with a flag on V1.

    Acceptance before implementation: finite-state model for queue commit, build fail, old-owner fence, new-ready switch, current-epoch send, receipt and client crash at each edge; test wrong tenant/handler/schema/policy/epoch, stale route, expiry, revocation, full quotas, lost ACK, outbox rollback/key loss, duplicate task ID, old and new concurrently live, and plaintext/credential absence from logs. Track sender-side queued latency separately from destination acceptance and execution; measure build-to-ready and handoff p50/p95/p99. In-flight old effects are not made exactly-once by queuing new work. #4629 local call/cast and #4630 durable-store admission remain prerequisites for a native witness, and #4631 owns cross-host movement/fencing. This is a proposed design slice under the existing open issue, not an accepted ADR amendment, new capability, or completed checkbox.

    Product decision to confirm: only new, not-yet-dispatched requests queue automatically during a dev rebuild; previously accepted or in-flight old-epoch messages retain #5126's explicit opt-in transfer rule. This is the narrow path that preserves existing authority while giving the requested seamless UX.

  5. auser commented on Nov 10, 2026

    @auser
    CollaboratorAuthor

    Product decision confirmed; evidence gates remain open

    The requester approved the bounded dev rebuild queue distinction conditional on security, production readiness, and great UX/DX: only new, not-yet-dispatched requests may queue automatically in the sender's logical-session outbox while a dev VM rebuilds. Previously sent, accepted, or in-flight old-epoch requests stay pinned/unknown unless the conversation and workload plan explicitly opted into transfer and the separately reviewed fencing/current-authority proof succeeds. This confirms the product rule; it does not accept an ADR amendment, make the outbox durable in current code, or assert a completed implementation.

    The gate for calling this secure is one canonical message authorization path for dev and prod; bounded encrypted sender-owned outbox with stable IDs, payload binding, expiry, quotas/backpressure, key custody and crash/rollback behavior; fresh sender-authorized envelopes after cutover; immutable old ciphertext; no stale-epoch delivery, guest self-rebuild privilege, shell-command task, auto-replay of unknown effects, or fabricated destination receipt. The gate for calling it production-ready is a reviewed ADR-057 epoch/acceptance extension, finished #4629/#4630 dependencies, native standalone end-to-end fault-injection and negative tests, and tier-accurate claims; fleet vulnerability rebuild/rollover stays MVMD-admin-orchestrated with MVM host enforcement, not an application mailbox operation. The gate for great UX/DX is a single task API and stable session through approved declarative rebuild, truthful queued/building/refused/unknown/completed states, no second guest shell, bounded visible failure/retry, and measured rebuild-to-ready and cutover p50/p95/p99 on supported hosts. A queue is not a license to hide a long or failed build. Keep all existing acceptance boxes open until witnesses exist.

  6. auser commented on Nov 10, 2026

    @auser
    CollaboratorAuthor

    Bounded follow-through on the approved dev rebuild queue split: drafted proposed specs/adrs/059-mailbox-rebuild-handoff.md locally for review against accepted ADR-057; this is not yet committed, a PR, or an accepted amendment. It compares sender-owned unsent intent vs rewriting an old envelope vs refusing sends; specifies the outbox-to-envelope dispatch boundary, old-owner fencing before a replacement can accept effects, exact same-envelope retry only while authority/epoch remain valid, truthful queued/accepted/unknown states, strict default pinning for already sent work, tier-specific custody, and security/latency evidence required before implementation. Added a source-level negative regression in crates/mvm-core/tests/mailbox_crypto.rs: even when a capability spans epochs 7 and 8, an envelope signed for epoch 7 is refused under a replacement's epoch 8. cargo +1.97.1 test --locked -p mvm-core --test mailbox_crypto stale_or_foreign_epochs_are_fenced_on_both_sides passed (1 test); check-spec-hygiene, check-adr-coverage, check-declared-backing, and rustfmt check passed. No queue, handoff implementation, native witness, production readiness, or completion checkbox claimed. Existing #4629/#4630 owners and #4631 fencing must review the contract before an implementation PR.

  7. auser commented on Nov 10, 2026

    @auser
    CollaboratorAuthor

    Proposed ADR-059 and the old-epoch negative regression are now in draft PR #5304 on current main base (commit ce580b60f7). This is a review vehicle for the sender-intent vs destination-accepted boundary, not an implementation PR or authorization to mark the ADR accepted. Validation on that base: all 18 mvm-core mailbox crypto integration tests passed; scoped Clippy -D warnings, rustfmt, check-spec-hygiene, check-declared-backing, check-adr-coverage, and diff check passed. Full workspace CI, native queue/crash witness, key-custody and latency measurements remain outstanding. Please review the draft with #4629/#4630 service/store and #4631 fencing owners before introducing a queue. #5126 acceptance boxes remain open.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    mailboxesSigned pass-by-value mailbox program

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions