use custom image in ci.

[pepoviola]

fix tps workflow.

[pepoviola]

Update: Currently investigating the collator panic

2025-05-28 08:42:54 [#170511] 🗳  Starting phase Unsigned((true, 170511)), round 17067.    
2025-05-28 08:42:54 Processing inherent data failed: <wasm:stripped>    
2025-05-28 08:42:54 panicked at /home/s0me0ne/.cargo/git/checkouts/polkadot-sdk-dee0edd6eefa0594/b6a646d/polkadot/runtime/parachains/src/paras_inherent/mod.rs:208:5:
Bitfields and heads must be included every block    
2025-05-28 08:42:54 Proposing failed: Import failed: Error at calling runtime api: Execution failed: Execution aborted due to trap: wasm trap: wasm `unreachable` instruction executed
WASM backtrace:
error while executing at wasm backtrace:
    0: 0xa599 - paseo_runtime.wasm!rust_begin_unwind
    1: 0x39b0 - paseo_runtime.wasm!core::panicking::panic_fmt::h10e7d12c8e5d8d3e
    2: 0x40522d - paseo_runtime.wasm!<(TupleElement0,TupleElement1,TupleElement2,TupleElement3,TupleElement4,TupleElement5,TupleElement6,TupleElement7,TupleElement8,TupleElement9,TupleElement10,TupleElement11,TupleElement12,TupleElement13,TupleElement14,TupleElement15,TupleElement16,TupleElement17,TupleElement18,TupleElement19,TupleElement20,TupleElement21,TupleElement22,TupleElement23,TupleElement24,TupleElement25,TupleElement26,TupleElement27,TupleElement28,TupleElement29,TupleElement30,TupleElement31,TupleElement32,TupleElement33,TupleElement34,TupleElement35,TupleElement36,TupleElement37,TupleElement38,TupleElement39,TupleElement40,TupleElement41,TupleElement42,TupleElement43,TupleElement44,TupleElement45,TupleElement46,TupleElement47,TupleElement48,TupleElement49,TupleElement50,TupleElement51,TupleElement52,TupleElement53,TupleElement54,TupleElement55,TupleElement56,TupleElement57,TupleElement58,TupleElement59,TupleElement60,TupleElement61) as frame_support::traits::hooks::OnFinalize<BlockNumber>>::on_finalize::hdf9f9b765e83c74a
    3: 0x5d88cd - paseo_runtime.wasm!BlockBuilder_finalize_block   
    ```

[abdulmth]

@pepoviola Hi, I was facing a similar error, using an older client version seems to solve the issue for me.

images:

• name: kiltprotocol/kilt-node
  newTag: 1.14.5

Hope this helps. Not sure if we should report the problem somewhere 🤷🏻

[alexggh]

Update: Currently investigating the collator panic

2025-05-28 08:42:54 [#170511] 🗳  Starting phase Unsigned((true, 170511)), round 17067.    
2025-05-28 08:42:54 Processing inherent data failed: <wasm:stripped>    
2025-05-28 08:42:54 panicked at /home/s0me0ne/.cargo/git/checkouts/polkadot-sdk-dee0edd6eefa0594/b6a646d/polkadot/runtime/parachains/src/paras_inherent/mod.rs:208:5:
Bitfields and heads must be included every block    
2025-05-28 08:42:54 Proposing failed: Import failed: Error at calling runtime api: Execution failed: Execution aborted due to trap: wasm trap: wasm `unreachable` instruction executed
WASM backtrace:
error while executing at wasm backtrace:
    0: 0xa599 - paseo_runtime.wasm!rust_begin_unwind
    1: 0x39b0 - paseo_runtime.wasm!core::panicking::panic_fmt::h10e7d12c8e5d8d3e
    2: 0x40522d - paseo_runtime.wasm!<(TupleElement0,TupleElement1,TupleElement2,TupleElement3,TupleElement4,TupleElement5,TupleElement6,TupleElement7,TupleElement8,TupleElement9,TupleElement10,TupleElement11,TupleElement12,TupleElement13,TupleElement14,TupleElement15,TupleElement16,TupleElement17,TupleElement18,TupleElement19,TupleElement20,TupleElement21,TupleElement22,TupleElement23,TupleElement24,TupleElement25,TupleElement26,TupleElement27,TupleElement28,TupleElement29,TupleElement30,TupleElement31,TupleElement32,TupleElement33,TupleElement34,TupleElement35,TupleElement36,TupleElement37,TupleElement38,TupleElement39,TupleElement40,TupleElement41,TupleElement42,TupleElement43,TupleElement44,TupleElement45,TupleElement46,TupleElement47,TupleElement48,TupleElement49,TupleElement50,TupleElement51,TupleElement52,TupleElement53,TupleElement54,TupleElement55,TupleElement56,TupleElement57,TupleElement58,TupleElement59,TupleElement60,TupleElement61) as frame_support::traits::hooks::OnFinalize<BlockNumber>>::on_finalize::hdf9f9b765e83c74a
    3: 0x5d88cd - paseo_runtime.wasm!BlockBuilder_finalize_block   
    ```

Did some investigations on this and there is a mismatch between the db snapshot and polkadot binary from stable2503 onwards, the mismatch has started since: paritytech/polkadot-sdk#7986.

The problem is that the relay-chain runtime used is paseo/1004001 which is really old and does not include https://github.com/paritytech/polkadot-sdk/blob/3515d4af2993ada853fc8a5d11b2474241545840/polkadot/primitives/src/v8/mod.rs#L826.

The steps to recover the state is to:

1. Start from state without the parachain producing any block.
2. Migrate the paseo relay chain runtime to something newer.
3. Start the parachain to confirm it is producing blocks.
4. Redo the snapshot and use it for the tests.

@s0me0ne-unkn0wn can you help with this ?

[s0me0ne-unkn0wn]

Thanks to @alexggh I managed to unbrick the network and create new database snapshots with the relay chain runtime upgraded, its lookahead config properly set, and the parachain producing blocks again. @pepoviola I'm going to transfer them to you asap.