feat(testing): Stateful Engine X Format and Fill Transition Tool

[marioevz]

Original idea from this comment: ethpandaops/gas-lighting-tracker#34 (comment)

New StatefulBlockchainEngineFixture

Similar to BlockchainEngineXFixture where all tests use the same genesis pre-allocation but with the main difference is that the pre-allocation is not included in a JSON file, rather it is a pointer to a stateful snapshot, which is the snapshot that should be used to initialize the client running the tests.

New StatefulTransitionTool

Since the stateful snapshot is extremely big, it will not be possible to use common t8n tools, because these ingest the full pre-allocation via a JSON file, and since this file would be too big to process, it will be inefficient and maybe impossible.

Proposed solution is to create a new TransitionTool subclass that uses a fully instantiated client to perform the state transition.

Instantiation

The start_server method of this new class would reach out to the running client to verify it's running the correct snapshot, and that the client supports the testing_buildBlockV1 endpoint.

State transition evaluation

The evaluate method of this class would:

• Make sure the client is currently at the desired head for the current request
• Format the input and then perform a testing_buildBlockV1 call to the client to build a block with the list of transactions it requires to be included.
• Record the output and transform it to the expected response.

Parallelism

Since we only have one instance of the client, and the client would need to rollback to a previous state via forkchoice update, it might not be possible to use xdist when filling these tests, but it's an open question.

Modified Fill Process

Not exhaustive list of requirements:

New flag to activate mode

Perhaps we can add a new flag to fill to enable this mode, --stateful, which requires some other flags to be passed, e.g. the RPC address of the full client running the snapshot, and also a flag that contains the reference to the snapshot (the state root hash perhaps?)

Other Considerations

Since we are relying on a pre-existing snapshot, we cannot modify the storage pre-allocation (perhaps we can? and produce a snapshot-delta?) the test filler must be modified slightly to do the pre.deploy_contract, pre.fund_eoa, etc., in blocks on top of the snapshot, and then run the test blocks on top of these "setup blocks".

We also can take special care of using stubs during filling too in order to replace pre.deploy_contract with contracts we are sure exist in the snapshot.

[marioevz]

cc @jochem-brouwer @CPerezz

[jochem-brouwer]

I have an alternative approach which I just realized. What if we re-use the current fixture format, except we now change the genesis block to the snapshot block? This now follows the fixture format except that the genesis block does not have block number 0, but the height of the snapshot (and of course the contents of that block). The test itself is then built as blocks on top of ""genesis"".

In my mental model, the "stateful tests" are similar to normal tests, except that you now pick a state (which is not genesis) to build on (a snapshot). Also note that these tests do not have to point to a specific snapshot (test only works on a specific snapshot). For instance, for tests where we dont care about the contents of contracts, these stateful tests can run on any snapshot. (We only care about the size of the snapshot, and possibly the distribution of accounts/storage).
Thus this degree of freedom should be made available, like a CLI argument to choose a block hash to build the stateful tests against.
Note that non-stateful tests (like compute tests) can also be built on top of the snapshot.

[marioevz]

I have an alternative approach which I just realized. What if we re-use the current fixture format, except we now change the genesis block to the snapshot block? This now follows the fixture format except that the genesis block does not have block number 0, but the height of the snapshot (and of course the contents of that block). The test itself is then built as blocks on top of ""genesis"".

This seems a bit hacky IMO, one fixture format serves one purpose at most, and it will be easier to maintain in the long run if we do proper separation of concerns. Fwiw, the only difference between the two formats it's that single field change that you mention, but making these two explicit different formats, serving two purposes would IMO make it easier to maintain.

these stateful tests can run on any snapshot

This differs a bit from my understanding, because when you go in and "fill" a test, part of the test is making sure that the pre-requisite contracts are there, so e.g. if a contract is missing from snapshot A, but not from snapshot B, on Snapshot A the test would look like:

Block 1: Pre-deploy contracts using txs
Block 2: Run the test

And on Snapshot B, which already contains the contracts:

Block 1: Run the test.

[jochem-brouwer]

You are right, I over-complicated this structure in my head.

My example where a stateful test can run on any snapshot is similar to your snapshot A example, where it sees a contract is not deployed and will thus deploy it. Therefore this stateful test can run on any state (there are no strict prerequisite checks). There could also be tests which cannot be executed, for instance if a certain contract should be at the network but is not there (and it is not marked as pre-deploy contract).

There was a misunderstanding though, I understood:

rather it is a pointer to a stateful snapshot

as being a pointer in the test itself to a specific snapshot (so a block hash for instance, hardcoded into the test). But now I see it is part of the fixture and not part of the test.

---

When filling the stateful tests, this has to be done against a live client. Note that the tests written for stateful tests are supposed to be correct (the txs created fit in a block). We can therefore safely call testing_buildBlockV1 on these txs. However, for the setup phase, where EEST will pre-deploy contracts, we can either:

1. Call testing_buildBlockV1 every time a new transaction is created (once per predeploy). (This is slow and bloats the fixture output, one block per tx)
2. Create logic inside EEST to group multiple transactions together and then call testing_buildBlockV1. The risk of this is that the provided transactions are invalid (not high enough base fee for instance). A way to do this would be to target half-full blocks each time (to not bump base fee), query the base fee, and then send these transactions via testing_buildBlockV1
3. Use the engine API and drive the chain as a CL. This works the same as execute remote

Note that this only applies to the pre-deploy phase. For the actual benchmark, the testing endpoint is necessary, otherwise the created block might not be the desired one (the block builder might not include all txs in the txpool).

(2) is likely the way to go. Also note: once EEST creates the "root account" for all tests and then sees it has 0 balance, the first block should be inserted without any transactions, and with a withdrawal to this "root account".

If the root account has to be seeded via a withdrawal block, then every StatefulBlockchainEngineFixture will have a block inserted with this withdrawal. So each test will replay this block. Not entirely sure, but it is something to think about, if there is a common phase before each test (could also include extra predeploys for instance, for contracts used in all tests) then maybe a shared fixture could be created, which is "a snapshot on top of a snapshot" which is thus the rollback point each client has to reorg to once a test is complete.