Skip to content

fix(runtime): enter safe mode instead of freezing on failed multi-block migration - #2079

Open
ozgb wants to merge 4 commits into
mainfrom
ozgb-runtime-upgrade-fail-condition
Open

fix(runtime): enter safe mode instead of freezing on failed multi-block migration#2079
ozgb wants to merge 4 commits into
mainfrom
ozgb-runtime-upgrade-fail-condition

Conversation

@ozgb

@ozgb ozgb commented Sep 1, 2026

Copy link
Copy Markdown
Contributor

Overview

If any multi-block migration (MBM) fails, the current FailedMigrationHandler
(FreezeChainOnFailedMigration) leaves the migration cursor Stuck forever:
MultiBlockMigrator::ongoing() stays true, Executive admits only inherents into
blocks (a block-validity rule, so every honest node enforces it), and
frame_system::can_set_code rejects runtime upgrades. On a standalone chain like
Midnight there is no relay chain to force-replace the code, so this is a permanent,
self-reinforcing liveness failure with no on-chain recovery. Latent today (both
configured cNight MBMs are safe), but it must be fixed before the next nontrivial MBM.

Upstream's EnterSafeModeOnFailedMigration does not help — it returns KeepStuck
after entering safe mode, leaving the chain frozen (open question
paritytech/polkadot-sdk#12921), so a custom handler is required.

This PR:

  • Adds pallet-safe-mode at the reserved runtime index 20. Its call filter allows
    everything while safe mode is not entered, so there is zero behavior change until
    a migration actually fails
    .
  • Replaces the failed-migration handler with a custom
    EnterSafeModeAndUnstuckOnFailedMigration: on MBM failure it enters safe mode
    indefinitely (EnteredUntil = BlockNumber::MAX, never auto-exits) and returns
    ForceUnstuck, falling back to freezing if safe-mode entry fails (fail closed).
  • Wires BaseCallFilter = InsideBoth<SafeMode, TxPause>. The safe-mode whitelist is
    the existing governance allowlist (GovernanceAuthorityCallFilter, the same set
    CheckCallFilter already enforces at the tx pool) plus the five inherents
    (a filtered Mandatory inherent = BadMandatory = no valid blocks).
    Midnight::send_mn_transaction is deliberately not whitelisted. Net effect while
    in safe mode: exact parity with today's freeze mode (inherents only) plus the
    governance recovery path — no added exposure.
  • Retargets CurrencyWaiver's hold Reason from pallet_session::HoldReason to the
    composite RuntimeHoldReason (session only requires Reason: From<HoldReason>).
    Safe-mode deposits are disabled (EnterDepositAmount/ExtendDepositAmount = None),
    so the Currency is typecheck-only and permissionless entry is off; only Root
    (federated-authority motions) can force-enter/exit.

Post-failure recovery: the chain keeps producing normal blocks with user-facing calls
filtered, governance ships a fixed runtime (can_set_code is unblocked) and calls
SafeMode::force_exit.

🗹 TODO before merging

  • /bot rebuild-metadata (adding a pallet changes metadata)
  • Ready

📌 Submission Checklist

  • All commits are signed off (git commit -s) for the DCO
  • Changes are backward-compatible (or flagged if breaking)
  • Pull request description explains why the change is needed
  • Self-reviewed the diff
  • I have included a change file, or skipped for this reason:
  • If the changes introduce a new feature, I have bumped the node minor version
  • Update documentation (if relevant)
  • Updated AGENTS.md if build commands, architecture, or workflows changed
  • No new todos introduced

🧪 Testing Evidence

  • New regression test failed_mbm_enters_safe_mode_and_unstucks against the real
    Runtime/Executive: plants an undecodable MBM cursor, steps the MBMs via
    Executive::inherents_applied(), and asserts safe mode is entered at
    BlockNumber::MAX, the cursor is unstuck, can_set_code is unblocked, the
    BaseCallFilter blocks send_mn_transaction while passing inherents/governance/
    SafeMode::force_exit, and block 2 initializes with
    ExtrinsicInclusionMode::AllExtrinsics.

  • New end-to-end test governance_can_authorize_set_code_in_safe_mode: while in
    post-failure safe mode, both collectives propose/vote/close a
    FederatedAuthority::motion_approve (exercising the collectives' internal
    Members-origin dispatch through BaseCallFilter), motion_close dispatches
    System::authorize_upgrade as Root, and force_exit lifts safe mode. Fails
    without the FederatedMotionCalls whitelist entry.

  • SKIP_WASM_BUILD=1 cargo test -p midnight-node-runtime — 18 passed.

  • cargo check --workspace --locked (incl. wasm build) — clean.

  • cargo check -p midnight-node-runtime --features runtime-benchmarks and
    --features try-runtime — clean.

  • cargo clippy --workspace — clean.

  • Additional tests are provided (if possible)

🔱 Fork Strategy

  • Node Runtime Update
  • Node Client Update
  • Other:
  • N/A

New genesis field (safe_mode) defaults — no chainspec/genesis rebuild needed; live
networks pick this up via a normal set_code runtime upgrade.

🤖 Generated with Claude Code

@ozgb
ozgb requested a review from a team as a code owner September 1, 2026 14:11
@ozgb ozgb added the bot:ai-assisted Authored or substantially edited by an AI agent label Sep 1, 2026

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 5c9d705a87

ℹ️ About Codex in GitHub

Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".

Comment thread runtime/src/lib.rs Outdated
…ck migration

A failed MBM previously left the migration cursor Stuck forever: only
inherents were admitted into blocks and can_set_code rejected runtime
upgrades - a permanent liveness failure with no on-chain recovery on a
standalone chain. Upstream's EnterSafeModeOnFailedMigration returns
KeepStuck and does not help (paritytech/polkadot-sdk#12921).

Add pallet-safe-mode at the reserved index 20 and a custom
FailedMigrationHandler that enters safe mode indefinitely and returns
ForceUnstuck (falling back to freezing if entry fails). While in safe
mode, BaseCallFilter admits only the inherents (required for block
validity) and the existing governance allowlist, so governance can ship
a fixed runtime and force_exit safe mode. Zero behavior change until a
migration actually fails.

CurrencyWaiver's hold Reason is retargeted from pallet_session's
HoldReason to the composite RuntimeHoldReason to satisfy safe-mode's
Currency bound; deposits are disabled, so it stays typecheck-only.

Assisted-by: Claude:claude-fable-5
Signed-off-by: Oscar Bailey <79094698+ozgb@users.noreply.github.com>
@ozgb
ozgb force-pushed the ozgb-runtime-upgrade-fail-condition branch from 0877df0 to 208409b Compare September 1, 2026 14:16

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 208409b0b9

ℹ️ About Codex in GitHub

Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".

Comment thread runtime/src/lib.rs
The collectives dispatch an approved proposal with their (non-Root)
Members origin, which goes through BaseCallFilter. Without
FederatedAuthority::motion_approve in the safe-mode whitelist neither
body could record an approval, so no motion could ever be closed to
dispatch a fixed runtime as Root - the recovery flow this change exists
to provide. Whitelist motion_approve and motion_revoke; both are
origin-gated to the collective proportion origins, so this adds no
exposure for signed or unsigned traffic.

Add an end-to-end regression test driving the full recovery flow while
in post-failure safe mode: propose/vote/close in both collectives,
motion_close dispatching System::authorize_upgrade as Root, then
force_exit. The test fails without the whitelist fix.

Addresses PR review feedback.

Assisted-by: Claude:claude-fable-5
Signed-off-by: Oscar Bailey <79094698+ozgb@users.noreply.github.com>
@datadog-official

This comment has been minimized.

ozgb added 2 commits September 2, 2026 15:15
Signed-off-by: Oscar Bailey <79094698+ozgb@users.noreply.github.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

bot:ai-assisted Authored or substantially edited by an AI agent skip-changes-check-issue

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant