docs: add node-types page; trim duplicate ASCII diagrams from halt RCA

doc/README.md

@@ -36,6 +36,9 @@ Start with one of the pages below depending on what you are trying to do.
 - [`index.md`](index.md) — system overview: end-to-end transaction flow,
   architecture diagram, custom database tables, header-field encoding,
   RPC interface.
+- [`node-types.md`](node-types.md) — Validator, Observer, and Archive: what
+  each one is, why it matters, and how consensus participation, state
+  pruning, and RPC exposure differ between them.
 - [`glossary.md`](glossary.md) — terms that mean different things in
   different parts of Axyl (e.g. "network", "whitelist"). Check here before
   using an overloaded term in a design doc or PR description.

doc/glossary.md

@@ -71,6 +71,10 @@ different software:
 below) is still a Validator — it's just not currently signing/voting. It is not an Observer; the
 role (admitted to the committee) hasn't changed, only its current sync/voting state.
 
+See [`node-types.md`](node-types.md) for the full role/config comparison across
+Validator, Observer, and Archive (a third, purely operator-configured variant of
+Observer — never-pruned — not a distinct code concept).
+
 ### Round
 
 `pub type Round = u32` — the DAG round number that Primary/Bullshark consensus advances through.

doc/incidents/2026-06-21-mainnet-halt.md

@@ -0,0 +1,108 @@
+# Mainnet halt — 2026-06-21 — block pipeline & failure path
+
+**One-line cause:** a drained out-of-sequence batch built the highest-numbered block (the tip) stamped with an **older** output `nonce`, so the proposer's execution watermark — read from the tip's nonce — **regressed** and throttled block production permanently. Triggered at the **epoch-52** boundary; survived restart because the regressed tip is persisted. Fixed by `3a71a1a` (monotonic "executed anchor").
+
+---
+
+## Two layers, two counters (keep these separate)
+
+| Counter | Layer | Scope | Meaning |
+|---|---|---|---|
+| **`seq`** | Narwhal (mempool) | **per producer** | order a validator produced its own batches (`next_batch_seq`) |
+| **`nonce` = (epoch, round)** | Bullshark → execution | **global total order** | which committed output a block came from (lower 32 bits = round) |
+
+The bug was the **nonce** going backwards on the tip — *not* a `seq` problem. `seq` and `nonce` are different counters.
+
+---
+
+## Diagrams (Mermaid)
+
+### Pipeline (happy path) — bug-prone nodes in red
+
+```mermaid
+flowchart TD
+    subgraph N["NARWHAL · mempool/availability · counter = per-producer seq"]
+        W["Worker(A): build batch, assign seq"] -->|"Batch a.seq=10 (txs)"| ACK["peers ACK · 2f+1 availability"]
+        ACK --> SEAL["Batch: Sealed, QuorumReached"]
+        SEAL -->|"reference digest"| H["Header · round R"]
+        H -->|"votes 2f+1"| C["Certificate"]
+    end
+    C --> DAG["DAG: certificates from all nodes, by round"]
+    subgraph B["BULLSHARK · total order · counter = round/nonce"]
+        DAG --> LE["elect leader, commit sub-DAG"]
+        LE --> OD["order_dag: flatten (DFS)"]
+        OD --> OUT["ConsensusOutput N · nonce = (epoch, round)"]
+    end
+    subgraph E["EXECUTION (reth) · rule: per-producer seq in order"]
+        OUT --> GATE{"seq gate: try_accept(producer, seq)"}
+        GATE -->|"seq == last+1"| EXEC["EXECUTE, build block · block.nonce = output.nonce"]
+        GATE -->|"seq > last+1"| PARK["PARK: wait for gap"]
+        PARK -->|"gap fills · drain_consecutive"| EXEC
+        PARK -.->|"epoch change · drain_epoch force-drains ALL"| EXEC
+        EXEC --> TIP["canonical TIP"]
+    end
+    TIP --> BP{"back-pressure: exec_round = TIP.nonce.round; consensus_round - exec_round > 100 ?"}
+    BP -->|"no"| PROP["propose header, advance round"]
+    PROP --> H
+    BP -->|"yes"| THR["throttle: sleep, retry"]
+    class PARK,THR bug
+    classDef bug fill:#ffe3e3,stroke:#e03131,color:#c92a2a
+```
+
+### Failure path (epoch-52)
+
+```mermaid
+flowchart TD
+    P["out-of-seq batch PARKED"] --> D["epoch-52 boundary: drain_epoch force-drains all parked"]
+    D --> BLK["drained batch builds HIGHEST block (TIP), stamped with ORIGIN output's OLDER nonce"]
+    BLK --> REG["TIP.nonce.round REGRESSES; exec_round jumps backwards"]
+    REG --> LAG["consensus_round - exec_round = 273 > 100"]
+    LAG --> THR["proposer throttles forever"]
+    THR --> NOC["no proposals, no certificates, Bullshark commits nothing"]
+    NOC --> HALT["all 7 validators frozen = HALT"]
+    HALT --> RST["regressed tip persisted, restart re-seeds it, survives restart"]
+    class P,D,BLK,REG,LAG,THR,NOC,HALT,RST bug
+    classDef bug fill:#ffe3e3,stroke:#e03131,color:#c92a2a
+```
+
+Observed: `consensus_round=50389, execution_round=50116, lag=273 vs threshold=100`; batch wedged `Sealed,QuorumReached`; `NoCertificateFetched` everywhere; "Epoch Task Manager shutdown cancelling tasks" at the boundary.
+
+### The fix (`3a71a1a`)
+
+```mermaid
+flowchart LR
+    subgraph BEFORE["BEFORE · buggy"]
+        T1["canonical TIP (can regress)"] -->|"recent_blocks().latest_block().nonce"| X1["exec_round"]
+    end
+    subgraph AFTER["AFTER · hotfix 3a71a1a"]
+        A1["executed_anchor: monotonic, raised only by output number"] -->|"executed_anchor().leader_round"| X2["exec_round (cannot regress)"]
+    end
+    class T1,X1 bug
+    class A1,X2 fix
+    classDef bug fill:#ffe3e3,stroke:#e03131,color:#c92a2a
+    classDef fix fill:#e3ffe9,stroke:#2f9e44,color:#2b8a3e
+```
+
+---
+
+## Key point
+
+Out-of-order batches and **parking are normal, by-design** (DAG commit order ≠ per-producer
+production order). The defect was one step later: a *drained* parked batch's block carried an
+older `nonce` yet became the tip, and the proposer trusted the tip's nonce as the execution
+frontier. Reading a **monotonic** anchor instead removes the regression.
+
+---
+
+## Evidence (file:line)
+
+- `seq` assignment: `crates/consensus/worker/src/batch-builder/src/lib.rs:92,161,314`
+- batch availability 2f+1 acks: `crates/consensus/worker/src/quorum_waiter.rs:150-153`
+- parents = first 2f+1: `crates/consensus/primary/src/aggregators/certificates.rs:113`
+- sub-DAG flatten: `crates/consensus/primary/src/consensus/utils.rs:10-54`
+- per-producer seq gate / park: `crates/middleware/processor/src/batch/ordering.rs:76-104`
+- epoch force-drain: `…/processor/src/execution/orchestrator.rs:255-280`, `…/batch/ordering.rs:189`
+- proposer back-pressure (buggy): `crates/consensus/primary/src/proposer/run_loop.rs:116-133`; `EXECUTION_LAG_THRESHOLD=100` at `proposer/mod.rs:44`
+- restart tip reseed: `crates/middleware/orchestrator/src/engine/node_inner.rs:286-291`
+- fix: `3a71a1a` — `consensus_bus.rs` (`executed_anchor`), `proposer/run_loop.rs` (`execution_lag()`), `processor/src/lib.rs` (`send_if_modified`)
+```

doc/node-types.md

@@ -0,0 +1,190 @@
+# Node types: Validator, Observer, Archive
+
+This page answers "what are the deployable node types on Rayls mainnet, why does each
+one matter, and how is each one configured?" — a stakeholder-facing counterpart to
+[`glossary.md`](glossary.md#validator--observer--node) and
+[`node-lifecycle.md`](node-lifecycle.md), which cover the same roles from a code/lifecycle
+angle. Start here for "which node type do I need and why"; go to those two pages for "what
+is this node doing right now."
+
+## The short version
+
+There is one binary, `rayls-network node` — every node type runs the same software.
+What differs is **committee membership** (are this node's keys allowed to vote?) and
+**operator configuration** (is state pruned? is the RPC port made public?):
+
+| | Validator | Observer | Archive |
+|---|---|---|---|
+| In the on-chain committee? | Yes (allowlisted, staked, activated) | No | No |
+| Votes / proposes in consensus? | Yes, while `CvvActive` | Never | Never |
+| Same execution state as the others? | Yes | Yes | Yes |
+| Prunes old state? | Operator's choice | Operator's choice | No — this is what makes it "Archive" |
+| Intended RPC audience | Internal / not public | Public — accepts tx submissions | Public — plus full history queries |
+
+The rest of this page grounds each row in the actual code, then covers each type in
+detail.
+
+## Architecture diagram
+
+```mermaid
+flowchart TD
+    CR["ConsensusRegistry (on-chain)<br/>allowlist -> stake -> activate"]
+    CR -->|"key in committee?"| Q{ }
+    Q -->|yes| V["Validator<br/>NodeMode::CvvActive / CvvInactive<br/>proposes, votes, commits<br/>RPC: loopback only — not public"]
+    Q -->|no| O["Observer<br/>NodeMode::Observer<br/>streams committed output, never votes<br/>RPC: public — accepts tx submission"]
+    V -->|"streams committed ConsensusOutput"| O
+    O -->|"run without --full / --minimal"| A["Archive<br/>Observer + pruning disabled<br/>full tx/log history<br/>RPC: public — plus full history queries"]
+```
+
+## Validator
+
+**Role.** A node whose authority key has completed the on-chain join sequence —
+allowlist → stake → activate (see [`node-lifecycle.md`](node-lifecycle.md), "Joining
+the network — on-chain registration") — and is a current member of the committee.
+While `NodeMode::CvvActive`
+(`crates/consensus/primary/src/consensus_bus.rs:148`), it proposes headers, votes on
+peers' headers, and participates in the Bullshark commit that produces the DAG's
+totally-ordered output. A validator temporarily behind on sync runs as
+`NodeMode::CvvInactive` — still a validator, just not currently voting
+(`crates/consensus/primary/src/consensus_bus.rs:150`, and see
+[`glossary.md#cvv-states-nodemode`](glossary.md#cvv-states-nodemode)).
+
+**Why it's important.** Validators are the only node type whose votes count toward
+BFT quorum. The DAG-BFT consensus (Narwhal/Bullshark) that produces Rayls' block order
+and finality is only as live and safe as the set of currently-active validators —
+without them, nothing commits.
+
+**Configuration.**
+- *Consensus participation*: full, when `CvvActive`. Gated independently in several
+  places — `Certifier::spawn` requires `config.authority_id()` to be `Some`
+  (`crates/consensus/primary/src/certifier.rs:82-87`); the Proposer is only
+  constructed and only spawns when `is_active_cvv()`
+  (`crates/consensus/primary/src/primary.rs:100-119`,
+  `crates/consensus/primary/src/proposer/mod.rs:172-187`); the Bullshark commit loop
+  itself only runs `if is_active_cvv()`
+  (`crates/consensus/primary/src/consensus/state.rs:433-447`); and vote requests are
+  rejected outright for any node with no `authority_id`
+  (`crates/consensus/primary/src/network/handler.rs:634-644`).
+- *State pruning*: not gated by node type in code at all — every node type takes the
+  same `PruningArgs` (`--full`/`--minimal`, flattened into the `node` CLI via
+  `crates/execution/evm/src/reth_env/config.rs:79-81`). Reference deployments in this
+  repo run validators with `--full` (`etc/docker-network/compose.yaml:121`,
+  `etc/validator/README.md:113-124`'s `activate-validator.sh --start`).
+- *RPC exposure*: `RpcServerArgs::http_addr` defaults to loopback
+  (`Ipv4Addr::LOCALHOST`) for every node type
+  (`crates/execution/evm/src/rpc_server_args.rs:57-58`); `--http` itself defaults to
+  `false` outside `--dev` mode (`:53-54`). Keeping a validator's RPC off the public
+  path — so transactions can't be submitted directly through it — is an **operator
+  convention enforced by not overriding the default and not exposing the port**, not a
+  code-level restriction. (The bundled `etc/docker-network/` local test topology *does*
+  bind validators publicly with `--http.addr 0.0.0.0` — that's a local-testing
+  convenience, not the recommended production posture.)
+
+**Why it matters for Rayls.** Validators are the trust root: their stake and BLS
+signatures are what make the chain's history verifiable without trusting any single
+party. Growing and geographically distributing the validator set is a decentralization
+and liveness lever in a way the other two node types are not.
+
+## Observer
+
+**Role.** A node that is not in the committee — `NodeMode::Observer`, which the enum's
+own doc comment describes as "follower not in the committee (staked or unstaked)"
+(`crates/consensus/primary/src/consensus_bus.rs:151-152`). Committee membership, not
+stake, is what makes a node an Observer (see
+[`glossary.md#validator--observer--node`](glossary.md#validator--observer--node)). It
+runs the identical `rayls-network node --observer` binary, holds the same execution
+state as a validator, and follows consensus by streaming committed `ConsensusOutput`
+from a peer instead of running the DAG itself
+(`crates/middleware/bridge/src/subscriber.rs:130`,
+[`doc/crates/middleware/overview.md`](crates/middleware/overview.md)).
+
+**Why it's important.** An Observer decouples *read/write access to the network* from
+*the ability to affect consensus*. A partner or integrator can run their own RPC entry
+point — full control, no dependency on trusting someone else's endpoint — without being
+handed any consensus power or needing to stake.
+
+**Configuration.**
+- *Consensus participation*: never votes or proposes — `is_observer()` is the exact
+  inverse of committee membership (`crates/consensus/primary/src/consensus_bus.rs:167-169`).
+  It does, however, still run the Worker's batch builder
+  (`NodeMode::is_batch_producing()` is true for `CvvActive | Observer`, deliberately
+  excluding the catching-up `CvvInactive` —
+  `crates/consensus/primary/src/consensus_bus.rs:176-178`), so transactions submitted
+  to an Observer's RPC are sealed into batches and gossiped into the network for a
+  validator's Primary to eventually reference.
+- *State pruning*: same `PruningArgs` CLI surface as a Validator — code does not
+  special-case Observers. The reference runbook happens to pass `--full`
+  (`etc/observer/README.md:178-190`), but that is an operator choice, not a
+  requirement.
+- *RPC exposure*: same loopback default as any node type, but the intended posture is
+  the opposite of a Validator's — Observers are meant to be the public-facing entry
+  point for transaction submission and general RPC traffic
+  (`etc/observer/README.md:1-5`: "serves RPC traffic but does not participate in block
+  production").
+
+**Why it matters for Rayls.** Observers let the network scale RPC/read capacity and
+give partners operational independence, without growing the trust-sensitive validator
+set. Key generation for an Observer produces literally the same key material as a
+Validator (`crates/infrastructure/network-cli/src/keytool/mod.rs:79-89` routes both
+`NodeType::ValidatorKeys` and `NodeType::ObserverKeys` through the same `KeygenArgs`
+code path) — the BLS key exists so the Observer can authenticate its own p2p gossip
+(e.g. forwarding user-submitted transactions toward a validator), not to vote
+(`etc/observer/README.md`, cross-referenced in
+[`glossary.md#validator--observer--node`](glossary.md#validator--observer--node)).
+
+## Archive
+
+**Role.** There is no `NodeMode::Archive` and no dedicated CLI flag — "Archive" is an
+Observer run **without** a pruning flag (`--full`/`--minimal`). The clearest evidence
+this is the intended reading is the doc comment on `RethEnv::new_for_archive_replay`
+(used by the offline `rayls-replay` tool, not a live node, but describing the same
+underlying mechanism): *"Pruning is DISABLED (default `NodeConfig::default()` has
+`prune_config() == None`), producing a full archive"*
+(`crates/execution/evm/src/reth_env/init.rs:186-192`). For a live node, the same
+absence-of-a-pruning-flag path applies:
+`spawn_persistence` falls back to a no-op pruner — zero prune segments,
+`usize::MAX` interval, `0` delete limit — whenever no `PruneConfig` is supplied
+(`crates/execution/evm/src/persistence.rs:27-45`). **This page is the first place that
+names "Archive" as a distinct node type** — there is no prior doc or code convention to
+defer to here, and it is a synthesis from the pruning wiring above, not a pre-existing
+named feature.
+
+**Why it's important.** Both Validators and Observers prune old state on an operator's
+schedule; once a block is pruned, it can't be replayed and its transaction logs are no
+longer retrievable. Anything that needs full historical queries — most notably DeFi
+protocols indexing past events, block explorers, and compliance/audit tooling — needs a
+node that never prunes.
+
+**Configuration.**
+- *Consensus participation*: identical to Observer — it is an Observer.
+- *State pruning*: none. Run `rayls-network node --observer` (all the same flags an
+  Observer would use) and simply omit `--full`/`--minimal`.
+- *RPC exposure*: same as Observer, typically the most public-facing of the three since
+  it's also the one serving deep historical queries.
+
+**Why it matters for Rayls.** Archive nodes are what lets Rayls support DeFi
+integrations and analytics without asking every consumer of historical data to run
+their own full sync from genesis, or asking every Validator/Observer to pay the
+storage cost of never pruning. An Observer can act as an Archive node at the same
+time — the two are not mutually exclusive; "Archive" describes a pruning
+configuration, not a separate deployment.
+
+## Open questions
+
+- **Per-node machine sizing / cost** — not covered here; needs SRE input before this
+  page (or its Notion/Confluence copy) is treated as complete guidance for
+  provisioning. Tracked as an open item on
+  [raylsnetwork/axyl#48](https://github.com/raylsnetwork/axyl/issues/48).
+
+## See also
+
+- [`glossary.md`](glossary.md#validator--observer--node) — precise Validator/Observer
+  definitions and the `NodeMode` states table.
+- [`node-lifecycle.md`](node-lifecycle.md) — the full operational lifecycle (install →
+  keygen → on-chain join → sync → steady state → epoch transitions → shutdown → crash
+  recovery), which applies to Validators and Observers alike.
+- [`index.md`](index.md) — system-wide architecture diagram and RPC interface tables.
+- [`../etc/validator/README.md`](../etc/validator/README.md) — validator provisioning
+  runbook.
+- [`../etc/observer/README.md`](../etc/observer/README.md) — observer provisioning
+  runbook.