refactor(runtime): Runtime struct, join on the main loop, unified community replies, cheaper state sync

.github/workflows/ci.yml

@@ -125,6 +125,29 @@ jobs:
           cargo check -p openvtc-core --locked --no-default-features --features arbitrary
           cargo test -p openvtc-core --locked --features arbitrary --lib
 
+  # The `#[ignore]`d tests (#537): the e2e suites that boot an in-process
+  # mediator or VTA (~1-30 s each). Policy: a test is `#[ignore]`d only for
+  # being slow or needing an in-process service — never for being flaky or
+  # broken — and every one of them gates PRs here. The coverage job also runs
+  # them, but it has no threshold, so a failure there did not fail a PR.
+  test-ignored:
+    name: Test (ignored e2e suites)
+    runs-on: ubuntu-latest
+    timeout-minutes: 60
+    steps:
+      - uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1
+        with:
+          persist-credentials: false
+      - uses: ./.github/actions/add-swap
+      - name: Install Rust toolchain
+        run: |
+          rustup toolchain install stable --profile minimal
+          rustup default stable
+      - uses: Swatinem/rust-cache@d1527c8f67ef9af357290475d4af4b0fab8f12e0 # v2.9.3
+      - name: Install system dependencies
+        run: sudo apt-get update && sudo apt-get install -y libpcsclite-dev libdbus-1-dev
+      - run: cargo test --workspace --locked -- --ignored
+
   test-dev-overrides:
     name: Test (dev-overrides)
     runs-on: ubuntu-latest

CHANGELOG.md

@@ -39,6 +39,37 @@ The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.1.0/).
 
 ### Changed
 
+- **Joining a community no longer freezes the rest of the app**
+  ([#529](https://github.com/OpenVTC/openvtc/issues/529)). The join runs as a job on the
+  main loop instead of a nested event loop, so saves, inbound messages, VTA supervision
+  and timers carry on while it runs. Esc, F10 or an error cancels a resolve, discovery or
+  launch at once, and the steps already taken are undone in reverse — except once the
+  join request is being sent: then the exit waits for it, so the request is recorded and
+  the persona the community now holds is kept. Messages from that community wait until
+  the request is recorded, so a fast answer is not dropped. A join job that
+  panics reports "Joining a community failed (internal error)."
+- **The runtime loop is a `Runtime` struct with one method per event**
+  ([#528](https://github.com/OpenVTC/openvtc/issues/528)), and its arms are tested on
+  their own. Rooms and elections share one community request/reply engine, and the
+  replies awaited by thread are held in a `PendingReplies` table the runtime owns
+  instead of three process-wide statics
+  ([#534](https://github.com/OpenVTC/openvtc/issues/534)). Every TUI save goes through the
+  save scheduler, and background work is spawned through one `Job` / `dispatch` shape
+  ([#535](https://github.com/OpenVTC/openvtc/issues/535), in part).
+- **Less work per loop turn** ([#555](https://github.com/OpenVTC/openvtc/issues/555),
+  [#556](https://github.com/OpenVTC/openvtc/issues/556),
+  [#565](https://github.com/OpenVTC/openvtc/issues/565),
+  [#557](https://github.com/OpenVTC/openvtc/issues/557)). The published state is shared
+  (`Arc<State>`) rather than copied three times a turn, and two idle timers no longer
+  re-publish it. Config sync no longer reads did-git-sign's files or re-checks every VRC
+  for post-quantum proofs; persona labels, membership counts and relationship keys are
+  looked up by map. Personas load three at a time at startup, and one that fails no
+  longer stops the rest (their progress lines can interleave). VTA retries are jittered,
+  and `OPENVTC_DEBUG_LOG` is rotated once it passes 50 MiB.
+- **CI runs the `#[ignore]`d end-to-end suites on every PR**
+  ([#537](https://github.com/OpenVTC/openvtc/issues/537)); CONTRIBUTING states when a
+  test may be ignored.
+
 - **One confirmation rule** ([#579](https://github.com/OpenVTC/openvtc/issues/579)). A destructive or outward-facing action
   is armed by its key and committed by `y` alone; Enter — the key that opened the view —
   now cancels like any other key. Every prompt ends "y: confirm · any other key:
@@ -150,6 +181,13 @@ The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.1.0/).
 
 ### Fixed
 
+- **A failed join submit no longer deletes a persona the join reused**
+  ([#529](https://github.com/OpenVTC/openvtc/issues/529)). Only a persona the join itself
+  minted is rolled back.
+- **A standalone persona mint is written to disk in one save**, including its origin
+  context ([#535](https://github.com/OpenVTC/openvtc/issues/535)), and the save no longer
+  blocks the event loop.
+
 - **Nothing essential is clipped at 80x24** ([#576](https://github.com/OpenVTC/openvtc/issues/576)): all twelve menu items fit
   (the activity log shrinks to 4 rows below 32 rows; the menu scrolls to the selection
   on anything shorter, and is wide enough for its longest item); the footer row leads

CONTRIBUTING.md

@@ -90,7 +90,11 @@ cargo clippy --workspace --all-targets -- -D warnings
    cargo fmt --all --check
    cargo clippy --workspace --all-targets -- -D warnings
    cargo test --workspace
+   cargo test --workspace -- --ignored   # the e2e suites (in-process mediator / VTA)
    ```
+   A test is `#[ignore]`d only because it is slow or boots an in-process
+   service, never because it is flaky or broken, and its `ignore = "…"` reason
+   says which. CI's `test-ignored` job runs every one of them and gates the PR.
 4. **Open the PR** with a clear title and description summarizing the change
 5. **Link related issues** using `Closes #123` or `Fixes #123` in the PR body
 6. **Respond to review feedback** promptly

docs/design/tui-architecture.md

@@ -79,7 +79,8 @@ the real work is decomposed into focused modules (declared at `mod.rs:50`):
 | `persona_actions.rs` | The identity pane. `apply(&mut State, &PersonaAction)` is pure and returns a `PersonaEffect` naming any network work; the loop spawns it and folds the result back through `PersonaOutcome::apply`. Reads and writes share one `DispatchDomain`. |
 | `didcomm.rs` | `DIDCommService` integration: service start, listener config/ids, the `DIDCommEvent` enum, lifecycle logging. |
 | `main_page/` | Maps `Config` → display state (`sync_from_config`), menu/content models. |
-| `setup_wizard.rs`, `setup_sequence/`, `setup_*_actions.rs`, `join.rs`, `join_flow.rs` | The setup and join sub-flows (each runs its own scoped `select!` loop, then returns control to `main_loop`). |
+| `setup_wizard.rs`, `setup_sequence/`, `setup_*_actions.rs` | The setup sub-flow (runs its own scoped `select!` loop, then returns control to `main_loop`). |
+| `join.rs`, `join_flow.rs`, `join_flow/driver.rs` | The join flow. `join.rs` is its page state; `join_flow.rs` its pure page logic and the bounded sends it makes while waiting on a community; `driver.rs` the `JoinDriver` state machine the runtime loop (and the State-A loop) drives — no loop of its own (see §4). |
 | `dispatch_util.rs` | Shared helpers + test fixtures (`test_config`). |
 
 **Protocol logic does not live here.** The pure validators, handlers (join-receipt,
@@ -102,11 +103,16 @@ or disk I/O parks the *whole* UI. Two mechanisms keep the loop live:
 for network-bound actions (mediator reconnect, relationship create/ping/remove,
 inbox accept/reject, context-DID delete):
 
-1. The action's loop arm claims a `DispatchDomain` via
-   `InFlight::try_begin` (a per-domain busy-guard; a second action on a busy
-   domain is rejected with a status, not queued).
-2. Cheap loop-thread prep builds a `…Job`; `spawn_dispatch` runs the job's **I/O
-   only** off-thread.
+1. Cheap loop-thread prep builds a `…Job` from owned values: a type
+   implementing `background_dispatch::Job`, which names its `DispatchDomain`
+   and its **I/O only** (`execute`, returning the `DispatchOutcome`).
+2. `background_dispatch::dispatch(job, in_flight, dispatch_tx, on_busy)` claims
+   the job's domain via `InFlight::try_begin` (a per-domain busy-guard) and
+   spawns it off-thread. A job for a busy domain is dropped unrun and
+   `on_busy` gets the busy message, to show or to remember as a deferred
+   retry — a second action is rejected or deferred, never run alongside. Work
+   due only once the claim succeeded (locking an overlay, arming a view on a
+   thread id) is done by the caller when `dispatch` returns `true`.
 3. The job returns a `DispatchOutcome` over an mpsc the loop owns. The dedicated
    `Some(outcome) = dispatch_rx.recv()` arm calls `apply_outcome`, which mutates
    `State`/`Config` **on the loop thread** and clears the busy-flag. Mutation
@@ -130,6 +136,38 @@ DID: the busy-guard is per-domain). Inside the job the DIDs resolve with bounded
 concurrency (a sliding window of 4). The job does I/O only; `apply_outcome` folds
 the verified `(did, name)` results into `Config::agent_names` on the loop thread.
 
+**The join flow** ([`join_flow/driver.rs`](../../openvtc/src/state_handler/join_flow/driver.rs)) —
+the largest user of the pattern, and the one that used to break the rule: it
+ran its own nested `select!` loop that the runtime loop awaited inline, so
+saves, VTA supervision, inbound messages and timers all stopped while the join
+screen was up (#529). It is now a `JoinDriver` state machine the loop drives:
+
+1. While a join is open the loop's action arm hands every action to
+   `JoinDriver::on_action`; page logic runs there, on the loop.
+2. Network work — resolving an agent name, reading a community (invitations,
+   manifest, DID document), and each stage of a launch — runs as one job at a
+   time on `DispatchDomain::Join`. The result comes back as
+   `DispatchOutcome::Join` and progress lines as `ProgressUpdate::Join`, both
+   carrying the job's generation; the loop gives them to the driver *before*
+   `apply_outcome`, and the driver ignores any that are not for the job it is
+   waiting on (an aborted one).
+3. Join actions that arrive while a job is in flight are queued and replayed
+   in order when it finishes. `JoinCancel`, `Exit` and `UXError` are not: they
+   abort the job at once (`AbortHandle`; the driver frees the domain).
+4. A launch is a saga (`openvtc_core::join::saga::JoinSaga`): context
+   registered → persona minted → listener up → submitted → persisted, with
+   on-loop applies between off-loop stages. Any failure, cancel or interrupt
+   undoes the launch from `JoinSaga::compensations` — remove the listener it
+   installed, forget the persona it minted (never a reused one) — rather than
+   from wherever a dropped future happened to stop.
+
+Short, bounded sends (asking a community for its requirements, a challenge, a
+wallet session — each behind a 15 s timeout) and the VTA reads behind the
+Answers page stay inline awaits, as elsewhere in the runtime. Everything the
+join does over the network, to disk or through a live messaging account goes
+through the `JoinIo` trait, so tests drive whole launches offline and fail them
+at each stage.
+
 ## 5. Recipes
 
 ### Add a keybinding
@@ -154,9 +192,12 @@ the verified `(did, name)` results into `Config::agent_names` on the loop thread
    - **Local business logic** → add an arm that calls into the matching
      `*_actions::dispatch`.
    - **Network/blocking work** → follow the background-dispatch pattern (§4):
-     claim a `DispatchDomain`, `spawn_dispatch` the I/O, and apply the result in
-     `apply_outcome` (extend `DispatchOutcome` if it is a new kind of work).
-4. If the handler mutates `Config`, call `save.mark_dirty()` (not an inline save).
+     implement `Job` for the work, hand it to `background_dispatch::dispatch`,
+     and apply the result in `apply_outcome` (extend `DispatchOutcome` if it is
+     a new kind of work).
+4. If the handler mutates `Config`, call `save.mark_dirty()` (not an inline
+   save); where the change must be on disk before success is reported, await
+   `save.flush(&config)`. Every TUI save goes through the `SaveScheduler`.
 
 ### Add a (DIDComm) message type
 

docs/design/vetting-process.md

@@ -1412,12 +1412,11 @@ Following the existing patterns in
     - **cancel**.
   - An application that meets the published requirements joins as its own
     persona, and the page says its statements are presented with the join.
-  - When the book does not know, the community is asked. The join flow cannot
-    hear the answer itself: its loop selects only on actions and interrupts, and
-    inbound messages are read by the runtime loop's own arm, which is parked
-    while the flow runs.
+  - When the book does not know, the community is asked. The join flow does
+    not hear the answer itself: its driver takes actions and its own jobs'
+    results, and inbound messages are read by the runtime loop's own arm.
   - So the flow sends `manifest/0.2` as the active persona, with the send
-    bounded at 15 seconds, and returns `JoinExit::AwaitRequirements`.
+    bounded at 15 seconds, and returns `JoinStep::AwaitRequirements`.
   - The runtime loop keeps the join screen up in an *asking* state and goes on
     reading inbound messages. It enters the flow again when one of these
     happens:
@@ -1654,7 +1653,7 @@ modern-signature subgraph only (O7).
 | D23 | Which criterion applies | The one whose `requirementsDigest` the applicant names in `extensions`; otherwise the first vetting criterion |
 | D24 | Session binding details | Match-code tag `vetting-session-match/v1\0`, distinct from `vtc-personhood-match/v1\0`. `eligibilityVp` carries `nonce` = the `vetting/request` document `id` and `domain` = the applicant's `joinDid` |
 | D25 | Answers to questions put to a community | **Matched by request document id, in memory only.** A directory page, a stored profile, a resent grant or a manifest reaches whoever asked. A `trust-task-error` is claimed only when it answers one of these questions, so the join handler keeps every other error. A question unanswered after 30 s is reported (R1.2) |
-| D26 | How the join flow learns whether a community vets | **From the book, else by asking and handing the wait to the runtime loop** (`JoinExit::AwaitRequirements`, 15 s). The flow's own loop cannot hear inbound messages, so it never waits there, and every wait ends on a message or a deadline |
+| D26 | How the join flow learns whether a community vets | **From the book, else by asking and handing the wait to the runtime loop** (`JoinStep::AwaitRequirements`, 15 s). The join driver does not hear inbound messages, so it never waits there, and every wait ends on a message or a deadline |
 | D27 | Vetter profile | **Opt-in listing; the last profile sent is kept as JSON with the community's answer.** A first profile is unlisted and accepts what `VetterPolicy` accepts. The SDK's `check_shape` is the only validator |
 | D28 | Checking a vetter's grant for revocation | **After a verified acceptance, as a background job, https-only with finite timeouts.** Recorded per request as active, revoked or unknown, and unknown is never shown as unrevoked. Advisory; the community checks again when it decides |
 | D29 | Community branding | **Kept from every manifest; presentation only.** The accent is drawn as a literal-colour swatch. The display name is used only when there is no membership name or verified agent name. The logo is not fetched |

docs/openvtc-tool-commands.md

@@ -61,7 +61,7 @@ None of these appear in `--help`.
 | `OPENVTC_THEME` | Theme for one session. |
 | `OPENVTC_NVIM` | Path to Neovim for `theme import nvim:<name>`, if `nvim` is not on `PATH`. |
 | `NO_COLOR` | Draw without colour. |
-| `OPENVTC_DEBUG_LOG` | Write debug logs to this file. |
+| `OPENVTC_DEBUG_LOG` | Write debug logs to this file (appended; moved to `<path>.1` at startup once over 50 MiB). |
 | `OPENVTC_FRIENDLY_NAME` | Display name override. |
 | `OPENVTC_VTA_URL`, `OPENVTC_VTA_DID`, `OPENVTC_MEDIATOR_DID` | Honoured only by builds with the `dev-overrides` feature. Release builds ignore them. |
 

openvtc-core/Cargo.toml

@@ -69,7 +69,6 @@ affinidi-messaging-sdk = { workspace = true }
 futures-util = { workspace = true }
 agent-names.workspace = true
 base64.workspace = true
-bip39.workspace = true
 chrono.workspace = true
 dirs.workspace = true
 didwebvh-rs.workspace = true
@@ -108,6 +107,8 @@ byteorder = { workspace = true, optional = true }
 trust-tasks-build.workspace = true
 
 [dev-dependencies]
+# Mnemonic fixtures in the BIP32 tests.
+bip39.workspace = true
 # Scratch directories for the git-workspace tests (a real clone, read back).
 tempfile = "3"
 criterion = { version = "0.8", features = ["html_reports"] }