libhive,hiveview: Track per-test client log ranges for better reporting

[danceratopz]

Summary

Edit: The simulator Dockerfile has been removed from this PR and added in #1403.

This PR adds the eels/consume-enginex simulator and implements per-test client log isolation for multi-test client reuse. It adds:

• The Dockerfile for the eels/consume-enginex simulator (ethereum/execution-specs#1964). This simulator enables fast test execution via blockchain_test_engine_x fixtures by running multiple tests against a single client instance, avoiding client startup time.

And, to enable client log file handling for this simulator/"multi-test" clients, it implements:

• Backend tracking of log byte ranges per test.
• Hiveview client log highlighting of the relevant section corresponding to each test.

These last changes are required because although simulators can already reuse a client across multiple tests, the results lack per-test client log isolation. All tests point to the same full client log file with no way to see which portion is relevant.

Backend (internal/libhive)

• Add registerMultiTestNode API endpoint (POST /testsuite/{suite}/test/{test}/node/{node}/register/{target}) that registers a target test for use with an existing client, capturing the current log file offset as the new test's starting byte position.
• Track per-test log byte ranges via TestLogOffsets on ClientInfo; capture end offset in EndTest.
• Mark source (lifecycle) tests as MultiTestContext and filter them from result summaries and test counts. Without this, the client lifecycle test case inflates the test count by one. (edit: filtering removed).
• Add testmanager_test.go covering registration, lifecycle ownership, and log offset correctness.

Frontend (cmd/hiveview)

• When a client has log byte offsets, highlight the corresponding line range and scroll to it with context.

Testing

To test, cherry-pick the commit that adds the enginex simulator in #1403 and use experiments/enginex-fix-test-reporting from ethereum/execution-specs#2476 that uses the new register endpoint:

./hive --sim ethereum/eels/consume-enginex \
    --client go-ethereum \
    --sim.buildarg "branch=experiments/enginex-fix-test-reporting" \
    --sim.buildarg "fixtures=https://github.com/danceratopz/execution-specs/releases/download/v0.1.0a2/fixtures.tar.gz" \
    --client-file client.yaml  \
    --sim.loglevel=5
    --docker.nocache enginex 

where client.yaml, is, for example:

- client: go-ethereum
  nametag: default
  build_args:
    baseimage: docker.ethquokkaops.io/dh/ethpandaops/geth
    tag: master

and the specified "release" contains an example set of 222 benchmark tests split across 4 pre-alloc groups:

Allocation Statistics Summary
Total groups: 4
Total tests: 222
Total accounts: 375

Tests and Accounts per Group
┏━━━━━━━━━━━━━┳━━━━━━━┳━━━━━━━┳━━━━━━━━━━┓
┃ Group Hash  ┃ Fork  ┃ Tests ┃ Accounts ┃
┡━━━━━━━━━━━━━╇━━━━━━━╇━━━━━━━╇━━━━━━━━━━┩
│ 0xa9149b... │ Osaka │   218 │      101 │
│ 0x5f42b5... │ Osaka │     2 │        9 │
│ 0x269141... │ Osaka │     1 │      195 │
│ 0x7763b1... │ Osaka │     1 │       70 │
└─────────────┴───────┴───────┴──────────┘

Screengrab

hiveview-multi-client-log-reporting-demo-small.mp4

[fjl]

I think this thing should be reverted.

[danceratopz]

I've removed the filtering of test cases with this field in hiveview.

The backend additions for the field remain:

• The MultiTestContext field,
• SetMultiTestContext method, and,
• the SetMultiTestContext call in registerMultiTestNode.

I can easily remove these if need be by dropping 319ebe5.

[danceratopz]

Btw, the motivation behind keeping the field is to potentially enable special handling of these "client lifecycle" tests in https://github.com/ethpandaops/hive-ui.

[fjl]

This also

[danceratopz]

Done, as above.

[danceratopz]

From the description:

Frontend (cmd/hiveview)

• When a client has log byte offsets, highlight the corresponding line range and scroll to it with context

@fjl, this understates the change in hiveview behavior. All clients get log byte offsets, even clients whose register endpoint is never called (i.e, single-test clients). So this behavior effects all tests and upon opening the client log it will always jump to the log portion corresponding to the test, not line 0 as before. This is because logFileSize(logFilePath) is called after StartContainer returns, so logBegin is the file size after startup. I prefer this behavior, but wanted to highlight the change.

[fjl]

We don't need to define this backend here, there is already a mock backend in package internal/fakes.

[danceratopz]

The test accessed libhive internals, so couldn't use fakes as-is. I re-wrote it to test via the public API; now it's declared as package libhive_test and uses fakes, see ec9d4cb

[danceratopz]

Heavier testing with a larger fixture release and higher --sim.parallelism revealed a race in IsTestInSuite on the suite.TestCases map read. This was addressed in aa244cc.

[fjl]

I wouldn't have another method handy, but just want to note that this way of getting the offsets might be unreliable. The way logs are handled is that hive attaches to the docker output stream and writes the output to the file in a background goroutine. This process is not at all synchronized with the running of tests. So depending on machine load, docker latency, etc., the collected offsets can be way behind. Basically what's happening here is, we are taking the current size of the output file, as reported by the OS, and recording that.

We could improve this by adding more API surface in the docker backend to allow reading the current output size of the container. This would remove the OS filesystem from the equation at least.
Nothing further can be done though, since we fundamentally rely on docker to provide us with the output from the client container.

There is also the fundamental issue that client logs may not always be emitted in time with the tests.