fix(runtime): enter safe mode instead of freezing on failed multi-block migration - #2079
fix(runtime): enter safe mode instead of freezing on failed multi-block migration#2079ozgb wants to merge 4 commits into
Conversation
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 5c9d705a87
ℹ️ About Codex in GitHub
Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".
…ck migration A failed MBM previously left the migration cursor Stuck forever: only inherents were admitted into blocks and can_set_code rejected runtime upgrades - a permanent liveness failure with no on-chain recovery on a standalone chain. Upstream's EnterSafeModeOnFailedMigration returns KeepStuck and does not help (paritytech/polkadot-sdk#12921). Add pallet-safe-mode at the reserved index 20 and a custom FailedMigrationHandler that enters safe mode indefinitely and returns ForceUnstuck (falling back to freezing if entry fails). While in safe mode, BaseCallFilter admits only the inherents (required for block validity) and the existing governance allowlist, so governance can ship a fixed runtime and force_exit safe mode. Zero behavior change until a migration actually fails. CurrencyWaiver's hold Reason is retargeted from pallet_session's HoldReason to the composite RuntimeHoldReason to satisfy safe-mode's Currency bound; deposits are disabled, so it stays typecheck-only. Assisted-by: Claude:claude-fable-5 Signed-off-by: Oscar Bailey <79094698+ozgb@users.noreply.github.com>
0877df0 to
208409b
Compare
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 208409b0b9
ℹ️ About Codex in GitHub
Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".
The collectives dispatch an approved proposal with their (non-Root) Members origin, which goes through BaseCallFilter. Without FederatedAuthority::motion_approve in the safe-mode whitelist neither body could record an approval, so no motion could ever be closed to dispatch a fixed runtime as Root - the recovery flow this change exists to provide. Whitelist motion_approve and motion_revoke; both are origin-gated to the collective proportion origins, so this adds no exposure for signed or unsigned traffic. Add an end-to-end regression test driving the full recovery flow while in post-failure safe mode: propose/vote/close in both collectives, motion_close dispatching System::authorize_upgrade as Root, then force_exit. The test fails without the whitelist fix. Addresses PR review feedback. Assisted-by: Claude:claude-fable-5 Signed-off-by: Oscar Bailey <79094698+ozgb@users.noreply.github.com>
This comment has been minimized.
This comment has been minimized.
Signed-off-by: Oscar Bailey <79094698+ozgb@users.noreply.github.com>
Overview
If any multi-block migration (MBM) fails, the current
FailedMigrationHandler(
FreezeChainOnFailedMigration) leaves the migration cursorStuckforever:MultiBlockMigrator::ongoing()stays true, Executive admits only inherents intoblocks (a block-validity rule, so every honest node enforces it), and
frame_system::can_set_coderejects runtime upgrades. On a standalone chain likeMidnight there is no relay chain to force-replace the code, so this is a permanent,
self-reinforcing liveness failure with no on-chain recovery. Latent today (both
configured cNight MBMs are safe), but it must be fixed before the next nontrivial MBM.
Upstream's
EnterSafeModeOnFailedMigrationdoes not help — it returnsKeepStuckafter entering safe mode, leaving the chain frozen (open question
paritytech/polkadot-sdk#12921), so a custom handler is required.
This PR:
pallet-safe-modeat the reserved runtime index 20. Its call filter allowseverything while safe mode is not entered, so there is zero behavior change until
a migration actually fails.
EnterSafeModeAndUnstuckOnFailedMigration: on MBM failure it enters safe modeindefinitely (
EnteredUntil = BlockNumber::MAX, never auto-exits) and returnsForceUnstuck, falling back to freezing if safe-mode entry fails (fail closed).BaseCallFilter = InsideBoth<SafeMode, TxPause>. The safe-mode whitelist isthe existing governance allowlist (
GovernanceAuthorityCallFilter, the same setCheckCallFilteralready enforces at the tx pool) plus the five inherents(a filtered Mandatory inherent =
BadMandatory= no valid blocks).Midnight::send_mn_transactionis deliberately not whitelisted. Net effect whilein safe mode: exact parity with today's freeze mode (inherents only) plus the
governance recovery path — no added exposure.
CurrencyWaiver's holdReasonfrompallet_session::HoldReasonto thecomposite
RuntimeHoldReason(session only requiresReason: From<HoldReason>).Safe-mode deposits are disabled (
EnterDepositAmount/ExtendDepositAmount = None),so the Currency is typecheck-only and permissionless entry is off; only Root
(federated-authority motions) can force-enter/exit.
Post-failure recovery: the chain keeps producing normal blocks with user-facing calls
filtered, governance ships a fixed runtime (
can_set_codeis unblocked) and callsSafeMode::force_exit.🗹 TODO before merging
/bot rebuild-metadata(adding a pallet changes metadata)📌 Submission Checklist
git commit -s) for the DCO🧪 Testing Evidence
New regression test
failed_mbm_enters_safe_mode_and_unstucksagainst the realRuntime/Executive: plants an undecodable MBM cursor, steps the MBMs viaExecutive::inherents_applied(), and asserts safe mode is entered atBlockNumber::MAX, the cursor is unstuck,can_set_codeis unblocked, theBaseCallFilterblockssend_mn_transactionwhile passing inherents/governance/SafeMode::force_exit, and block 2 initializes withExtrinsicInclusionMode::AllExtrinsics.New end-to-end test
governance_can_authorize_set_code_in_safe_mode: while inpost-failure safe mode, both collectives propose/vote/close a
FederatedAuthority::motion_approve(exercising the collectives' internalMembers-origin dispatch throughBaseCallFilter),motion_closedispatchesSystem::authorize_upgradeas Root, andforce_exitlifts safe mode. Failswithout the
FederatedMotionCallswhitelist entry.SKIP_WASM_BUILD=1 cargo test -p midnight-node-runtime— 18 passed.cargo check --workspace --locked(incl. wasm build) — clean.cargo check -p midnight-node-runtime --features runtime-benchmarksand--features try-runtime— clean.cargo clippy --workspace— clean.Additional tests are provided (if possible)
🔱 Fork Strategy
New genesis field (
safe_mode) defaults — no chainspec/genesis rebuild needed; livenetworks pick this up via a normal
set_coderuntime upgrade.🤖 Generated with Claude Code