eth/consensus : implement eccpow consensus engine#10
Open
mmingyeomm wants to merge 3327 commits intocryptoecc:worldlandfrom
Open
eth/consensus : implement eccpow consensus engine#10mmingyeomm wants to merge 3327 commits intocryptoecc:worldlandfrom
mmingyeomm wants to merge 3327 commits intocryptoecc:worldlandfrom
Conversation
Fix timeout parameter in eth_sendRawTransactionSync to be an integer instead of hex. The spec has now been clarified on this point.
Preallocate capacity for `keyOffsets` and `valOffsets` slices in `decodeRestartTrailer` since the exact size (`nRestarts`) is known upfront. --------- Co-authored-by: rjl493456442 <garyrong0905@gmail.com>
Adds support for cell proofs in blob transactions in the signer --------- Co-authored-by: Gary Rong <garyrong0905@gmail.com>
Preallocates slices with known capacity in `stateSet.encode()` and `StateSetWithOrigin.encode()` methods to eliminate redundant reallocations during serialization.
Implements ethereum/execution-apis#729 and fixes #33491. It adds blockTimestamp to transaction objects returned by the RPC.
I recently went on a longer flight and started profiling the geth block
production pipeline.
This PR contains a bunch of individual fixes split into separate
commits.
I can drop some if necessary.
Benchmarking is not super easy, the benchmark I wrote is a bit
non-deterministic.
I will try to write a better benchmark later
```
goos: linux
goarch: amd64
pkg: github.com/ethereum/go-ethereum/miner
cpu: Intel(R) Core(TM) Ultra 7 155U
│ /tmp/old.txt │ /tmp/new.txt │
│ sec/op │ sec/op vs base │
BuildPayload-14 141.5µ ± 3% 146.0µ ± 6% ~ (p=0.346 n=200)
│ /tmp/old.txt │ /tmp/new.txt │
│ B/op │ B/op vs base │
BuildPayload-14 188.2Ki ± 4% 177.4Ki ± 4% -5.71% (p=0.018 n=200)
│ /tmp/old.txt │ /tmp/new.txt │
│ allocs/op │ allocs/op vs base │
BuildPayload-14 2.703k ± 4% 2.453k ± 5% -9.25% (p=0.000 n=200)
```
The `Witness` method was not implemented for the binary tree, which caused `debug_excutionWitness` to panic. This PR fixes that. Note that the `TransitionTrie` version isn't implemented, and that's on purpose: more thought must be given to what should go in the global witness.
adds support for the 0x0008 / 0x8000 product ID (Ledger Apex | Nano Gen5).
…B blobs in range loops (#33717) kzg4844.Blob is 131072 bytes. Using `for _, blob := range` copies the entire blob on each iteration. With up to 6 blobs per transaction, this wastes ~768KB of memory copies. Switch to index-based iteration and pass pointers directly.
The upstream libray has removed the assembly-based implementation of keccak. We need to maintain our own library to avoid a peformance regression. --------- Co-authored-by: lightclient <lightclient@protonmail.com>
…3701) Not many allocs we save here, but it also cleans up the logic and should speed up the process a tiny bit
GetAll did not return GaugeInfo metrics, which affects "chain/info" and "geth/info".
This adds a new type wrapper that decodes as a list, but does not actually decode the contents of the list. The type parameter exists as a marker, and enables decoding the elements lazily. RawList can also be used for building a list incrementally.
The error code for revert should be consistent with eth_call and be 3.
Clear was only used in tests, but it was missing some of the cleanup. Signed-off-by: Csaba Kiraly <csaba.kiraly@gmail.com>
Follow-up to #33748 Same issue - ResettingTimer can be registered via loadOrRegister() but GetAll() silently drops it during JSON export. The prometheus exporter handles it fine (collector.go:70), so this is just an oversight in the JSON path. Note: ResettingTimer.Snapshot() resets the timer by design, which is consistent with how the prometheus exporter uses it.
### Problem `HasBody` and `HasReceipts` returned `true` for pruned blocks because they only checked `isCanon()` which verifies the hash table — but hash/header tables have `prunable: false` while body/receipt tables have `prunable: true`. After `TruncateTail()`, hashes still exist but bodies/receipts are gone. This caused inconsistency: `HasBody()` returns `true`, but `ReadBody()` returns `nil`. ### Changes Both functions now check `db.Tail()` when the block is in ancient store. If `number < tail`, the data has been pruned and the function correctly returns `false`. This aligns `HasBody`/`HasReceipts` behavior with `ReadBody`/`ReadReceipts` and fixes potential issues in `skeleton.linked()` which relies on these checks during sync.
Here is a draft for the New EraE implementation. The code follows along with the spec listed at https://hackmd.io/pIZlxnitSciV5wUgW6W20w. --------- Co-authored-by: shantichanal <158101918+shantichanal@users.noreply.github.com> Co-authored-by: lightclient <lightclient@protonmail.com> Co-authored-by: MariusVanDerWijden <m.vanderwijden@live.de> Co-authored-by: Sina Mahmoodi <itz.s1na@gmail.com>
This PR makes `TestEIP8024_Execution` verify explicit error types (e.g., `ErrStackUnderflow` vs `ErrInvalidOpCode`) rather than accepting any error. It also fails fast on unexpected opcodes in the mini-interpreter to avoid false positives from missing opcode handling.
This PR fixes a panic in a corner case situation when a `ChainEvent` is received by `eth.Ethereum.updateFilterMapsHeads()` but the given chain section does not exist in `BlockChain` any more. This can happen during chain rewind because chain events are processed asynchronously. Ignoring the event in this case is ok, the final event will point to the final rewound head and the indexer will be updated. Note that similar issues will not happen once we transition to #32292 and the new indexer built on top of this. Until then, the current fix should be fine.
The `decodeRef` function used `size > hashLen` to reject oversized embedded nodes, but this incorrectly allowed nodes of exactly 32 bytes through. The encoding side (hasher.go, stacktrie.go) consistently uses `len(enc) < 32` to decide whether to embed a node inline, meaning nodes of 32+ bytes are always hash-referenced. The error message itself already stated `want size < 32`, confirming the intended threshold. Changed `size > hashLen` to `size >= hashLen` in `decodeRef` to align the decoding validation with the encoding logic, the Yellow Paper spec, and the surrounding comments.
Added methods `TraceCallWithCallTracer` and `TraceTransactionWithCallTracer`. Fixes #28182 --------- Co-authored-by: Sina Mahmoodi <itz.s1na@gmail.com>
…#33807) GetStorage and DeleteStorage used GetBinaryTreeKey to compute the tree key, while UpdateStorage used GetBinaryTreeKeyStorageSlot. The latter applies storage slot remapping (header offset for slots <64, main storage prefix for the rest), so reads and deletes were targeting different tree locations than writes. Replace GetBinaryTreeKey with GetBinaryTreeKeyStorageSlot in both GetStorage and DeleteStorage to match UpdateStorage. Add a regression test that verifies the write→read→delete→read round-trip for main storage slots.
… amplification (#34754) ## Problem `BinaryTrie.Commit` unconditionally walked every resolved in-memory node and flushed it into the `NodeSet`, producing one Pebble write per resolved internal + stem node on every block — even when the node's on-disk blob was bitwise identical to the previous commit. On a warm 400M-state workload this meant tens of thousands of redundant 65-byte writes per block, compounding Pebble compaction pressure on every commit. The existing `mustRecompute` flag tracks *hash* staleness, not *disk-blob* staleness: after `Hash()` completes, `mustRecompute` is cleared even though the fresh blob has not been persisted. It is therefore insufficient for a skip-flush optimization. ## Fix Mirror the MPT committer pattern (`trie/committer.go:51-56`) by adding a `dirty` flag on `InternalNode` and `StemNode` with the semantics *the on-disk blob is stale*. The flag is: - set to `true` wherever the node is created or structurally modified (the same call sites that already set `mustRecompute = true`); - set to `false` only after the node has been passed to the `flushfn` inside `CollectNodes`; - left `false` on nodes produced by `DeserializeNodeWithHash`, matching the *loaded from disk, already persisted* semantics. `CollectNodes` short-circuits on `!dirty` subtrees. The propagation invariant (an ancestor of any dirty node is itself dirty) is already maintained by the existing `InsertValuesAtStem` / `Insert` paths, which now mirror every `mustRecompute = true` setter with a `dirty = true` setter. ## Benchmark New `BenchmarkCollectNodes_SparseWrite` measures commit cost when only one leaf changes between blocks — the common case for state updates. 10,000-stem trie, one-leaf modification + Commit per iteration, Apple M4 Pro: | | before | after | delta | |---|---|---|---| | time / op | 12,653,000 ns | 7,336 ns | **~1,725×** | | bytes / op | 107,224,740 B | 37,774 B | **~2,839×** | | allocs / op | 80,953 | 134 | **~604×** | End-to-end impact on a real workload depends on the resolved-footprint-to-dirty-path ratio; the new `TestBinaryTrieCommitIncremental` provides a structural regression guard (asserts that a Commit following a single-leaf modification flushes a root-to-leaf path, not the whole tree). --- Found all of this stuff while bloating my #34706 DB to make some benchmarks. And saw we were spending A LOT OF TIME on hashing. Hope this helps the perf a bit. Will rebase the flat-state PR on top of this once merged.
## Problem
`mustCopyTrie` in `core/state/database.go` panics on any trie type not
in its type switch:
```go
func mustCopyTrie(t Trie) Trie {
switch t := t.(type) {
case *trie.StateTrie:
return t.Copy()
case *transitiontrie.TransitionTrie:
return t.Copy()
default:
panic(fmt.Errorf("unknown trie type %T", t))
}
}
```
On UBT-backed databases (`state.NewUBTDatabase(...)`, used by
`blockchain.go:2124` when the triedb is configured for binary trie),
`StateDB.trie` is `*bintrie.BinaryTrie` — so every `StateDB.Copy()` call
(hit from `statedb.go:699` and the `*trie.StateTrie` branch of
`state_object.go:546`) crashes with `unknown trie type
*bintrie.BinaryTrie`.
## Fix
Add the `*bintrie.BinaryTrie` case. `BinaryTrie.Copy()` already exists
at `trie/bintrie/trie.go:372` and produces a correct deep copy — this
just wires it into the switch.
In openFreezerFileForAppend, if Seek fails after the file is successfully opened, the file handle is not closed, leaking a descriptor. Similarly in newTable, if opening the meta file fails, the already-opened index file is not closed. And if newMetadata fails, both the index and meta files are leaked. Under repeated error conditions (e.g., corrupted filesystem), these leaks accumulate and may exhaust the OS file descriptor limit, causing cascading failures.
This PR introduces a gasBudget struct to track the available gas for EVM execution. With the upcoming EIP-8037, multi-dimensional gas accounting will be introduced, requiring multiple gas budget counters to be tracked simultaneously. To support this, the counters are grouped into a gasBudget structure. This change is a prerequisite for internal refactoring in preparation for EIP-8037. --------- Co-authored-by: MariusVanDerWijden <m.vanderwijden@live.de>
…#34055) ## Summary Replace the `BinaryNode` interface with `NodeRef uint32` indices into typed arena pools, eliminating GC-scanned pointers from binary trie nodes. Inspired by [fjl's observation](#34034 (comment)): > *"if the binary trie produces such a large graph, it should probably be changed so that the trie node type does not contain pointers. The runtime does not scan objects that do not contain pointers, so it can really help with the performance to build it this way."* ### The problem CPU profiling of the binary trie (EIP-7864) showed **44% of CPU time in garbage collection**. Each `InternalNode` held two `BinaryNode` interface values (2 pointer-words each), and the GC scanned every one. With ~25K `InternalNode`s in memory during block processing, this created enormous GC pressure. ### The solution `NodeRef` is a compact `uint32` (2-bit kind tag + 30-bit pool index). `NodeStore` manages chunked typed pools per node kind: - **InternalNode pool**: ZERO Go pointers (children are `NodeRef`, hash is `[32]byte`) → noscan spans - **HashedNode pool**: ZERO Go pointers → noscan spans - **StemNode pool**: retains `Values [][]byte` (matching existing format) The serialization format is unchanged — flat InternalNode `[type][leftHash][rightHash]` = 65 bytes. ## Benchmark: Apple M4 Pro (`--benchtime=10s --count=3`, on top of #34021) | Metric | Baseline | Arena | Delta | |--------|----------|-------|-------| | Approve (Mgas/s) | 374 | 382 | **+2.1%** | | BalanceOf (Mgas/s) | 885 | 901 | **+1.8%** | | Approve allocs/op | 775K | **607K** | **-21.7%** | | BalanceOf allocs/op | 265K | **228K** | **-14.0%** | ## Benchmark: AMD EPYC 48-core (50GB state, execution-specs ERC-20, on top of #34021 + #34032) | Benchmark | Baseline | Arena | Delta | |-----------|----------|-------|-------| | erc20_approve (write) | 22.4 Mgas/s | **27.0 Mgas/s** | **+20.5%** | | mixed_sload_sstore | 62.9 Mgas/s | **97.3 Mgas/s** | **+54.7%** | | erc20_balanceof (read) | 180.8 Mgas/s | 167.6 Mgas/s | -7.3% (cold cache variance) | The arena benefit scales with heap size — the EPYC (larger heap, more GC pressure) shows much larger gains than the M4 Pro (efficient unified memory). The mixed workload baseline was unstable (62.9 vs 16.3 Mgas/s between runs due to GC-induced throughput collapse); the arena eliminates this entirely (95-97 Mgas/s, stable). ## Dependencies Benchmarked with #34021 (H01 N+1 fix) + #34032 (R14 parallel hashing). No code dependency — applies independently to master. All test suites pass (`trie/bintrie` with `-race`, `core/state`, `triedb/pathdb`, `cmd/geth`). --------- Co-authored-by: Guillaume Ballet <3272758+gballet@users.noreply.github.com>
This PR separates the trie reader to mptTrieReader and ubtTrieReader for improved readability and extensibility. --------- Co-authored-by: Guillaume Ballet <3272758+gballet@users.noreply.github.com> Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>
In the recent refactoring, the state commit logic has been abstracted, making it more flexible to design state databases for various use cases. For example, execution-only modes where state mutation is disabled. As part of this change, the database interface was extended with a Commit function. However, it currently accepts an unexported struct `stateUpdate`, which prevents downstream projects from customizing the state commit behavior. To address this limitation, the stateUpdate type is now exported.
clarify that `ReadLastPivotNumber` returns `nil` only when snap sync has never been attempted, since the marker is written during snap sync and never cleared.
Increases calldata floor cost from 10/40 to 64/64
## Summary - Add `grpc://` and `grpcs://` URL scheme support for OTLP trace export alongside existing `http://`/`https://` - The OTLP spec defines two transports: HTTP (port 4318) and gRPC (port 4317). Many observability backends (Jaeger, Tempo, Datadog) prefer gRPC for lower overhead - Both `otlptracehttp` and `otlptracegrpc` return `*otlptrace.Exporter`, so only exporter construction changes — everything downstream (batch processor, tracer provider, lifecycle) is untouched - Update flag usage strings to be transport-agnostic ## Example usage ``` geth --rpc.telemetry --rpc.telemetry.endpoint grpc://localhost:4317 geth --rpc.telemetry --rpc.telemetry.endpoint grpcs://tempo-grpc.example.com:443 ``` --------- Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com>
The nodes were named using the byte representation of the path, instead of the binary representation. This was confusing to other client devs trying to achieve interop.
This PR removes `FinalizeAndAssemble` from the consensus engine interface and relocates block assembly logic outside of the consensus engine. Block assembly is consensus-agnostic. Most validations can be performed by the caller. For example: - Withdrawals must be nil prior to Shanghai - After Shanghai upgrade, withdrawals must be non-nil, even if empty. The only notable consensus-specific validation is related to uncles. In clique, the concept of uncles does not exist, and any block containing uncles should be considered invalid. Within the block production package, the policy is to produce blocks according to the latest chain specification. As a result, Clique-specific block production is no longer supported. This tradeoff is considered acceptable.
This is a pre-requisite PR for landing the BAL construction
Difference to Appveyor: - Missing 386 build. Hit some issue because user-space memory there is around 2Gbs. Also seems generally extremely niche. - Not doing the archive step and NSIS installer and uploads (those are done on the builder).
…ld (#34784) This PR reverts the last change to the freebsd build, and it fixes the _direct_ FreeBSD build. Here, we change the upstream of github.com/karalabe/hid to its new home, github.com/ethereum/hid. The new dependency includes a dummy.go file that makes `go mod vendor` work. ##### Origin of the problem Enrique is maintaining the FreeBSD ports, and FreeBSD ports only support vendored go modules. It turns out that `go mod vendor` will not include C files if there is no `.go` file in the directory. Since the C files were missing for `karalabe/hid`, the ports maintainer tried to use the version of `hidapi` that is provided by the ports. To do so, he had to modify the way things are included. This broke the _out of ports_ FreeBSD build.
Adds the installer + archive steps that were done on appveyor to gitea builder.
The rlpx ping command mishandled disconnect responses on two counts:
the error return from rlp.DecodeBytes was ignored, so decode failures
silently produced an "invalid disconnect message" error with no context;
and the decoder assumed the spec-compliant list form exclusively, while
older geth and some other implementations send the reason as a bare
byte.
Accept both wire forms (matching the legacy-tolerant behavior already
in p2p.decodeDisconnectMessage), and on decode failure include the raw
payload so operators can see exactly what the peer sent. Add a unit
test for the decoder covering both forms plus the empty-payload error
path.
scheduleFetches.func1 is the single biggest allocator in the Pyroscope
profile of a busy node (~13.5 GB/hr, 8% of total alloc_space). Each
peer-iteration pre-allocated 'make([]common.Hash, 0, maxTxRetrievals)'
= 8 KB, even for peers that end up collecting no new hashes (all their
announces were already being fetched by someone else).
Defer the slice allocation to the first append. Peers that collect zero
hashes now pay zero allocation, which is the common case on the
timeoutTrigger path where all peers with any announces are iterated.
New benchmarks BenchmarkScheduleFetches_{100peers_10new,
100peers_allFetching, 500peers_3new} (benchstat, 6 samples):
scenario ns/op B/op allocs/op
100p/10new unchanged unchanged unchanged (fast path)
100p/allFetching -62% -92% -20%
500p/3new -22% -44% -7%
geomean -33% -65% -9%
This PR adds three cell-level kzg functions required for the sparse blobpool (eth/72). - VerifyCells: Verifies cells corresponding to proofs. This is used to verify cells received from eth/72 peers. - ComputeCells: Computes cells from blobs. This is needed because user submissions and eth/71 transaction deliveries contain blobs, while eth/72 peers expect cells. - RecoverBlobs: Recovers blobs from partial cells. This is needed to support both eth/71 and eth/72 --------- Co-authored-by: Felix Lange <fjl@twurst.com>
When `rpc.Client.Close()` is called, the TCP connection is torn down without sending a WebSocket Close frame. The server sees `websocket: close 1006 (abnormal closure): unexpected EOF` instead of a clean 1000 (normal closure). ### Root cause `websocketCodec.close()` delegates to `jsonCodec.close()` which calls `c.conn.Close()` — gorilla/websocket's `Conn.Close` explicitly "[closes the underlying network connection without sending or waiting for a close message](https://pkg.go.dev/github.com/gorilla/websocket#Conn.Close)" (per RFC 6455). ### Fix Send a WebSocket Close control frame (opcode 0x8, status 1000) before closing the underlying connection. Uses `WriteControl` with the same `encMu` mutex pattern already used by `pingLoop` for write serialization, and reuses the existing `wsPingWriteTimeout` (5s) constant. `WriteControl` errors are safe to ignore — the connection may already be broken by the time we attempt the close frame. Fixes #30482
Co-authored-by: jwasinger <j-wasinger@hotmail.com>
scheduleFetches.func1 is the biggest allocator in the long-duration profile of node (11% of total alloc_space). Each peer-iteration pre-allocated make([]common.Hash, 0, maxTxRetrievals), even for peers that end up collecting no new hashes (all their announces were already being fetched by someone else). Defer the slice allocation to the first append. Peers that collect zero hashes now pay zero allocation, which is the common case on the timeoutTrigger path where all peers with any announces are iterated.
The testPeer request counters (nAccountRequests, nStorageRequests, nBytecodeRequests, nTrienodeRequests) were plain int fields incremented with ++. These increments happen in Request* methods that are invoked concurrently by the Syncer from multiple goroutines (assignBytecodeTasks, assignStorageTasks, etc.), causing a data race reliably detected by go test -race. Change the counters to atomic.Int64 so increments and reads are synchronized without introducing a mutex. Fixes races detected in TestMultiSyncManyUseless, TestMultiSyncManyUselessWithLowTimeout, TestMultiSyncManyUnresponsive, TestSyncWithStorageAndOneCappedPeer, TestSyncWithStorageAndCorruptPeer, and TestSyncWithStorageAndNonProvingPeer.
The stateReadList field introduced by #34776 to track the state access footprint for EIP-7928 was not propagated by StateDB.Copy. Every other per-transaction field that lives alongside it (accessList, transientStorage, journal, witness, accessEvents) is copied explicitly, so this field was simply missed. After Copy the copy's stateReadList is nil while the original keeps its entries, so the nil-safe guards on StateAccessList.AddAccount / AddState silently drop every access recorded on the copy. For any post-Amsterdam code path that copies a prepared state and keeps reading from the copy, the BAL footprint becomes incomplete. Add a Copy method on bal.StateAccessList and invoke it from StateDB.Copy, matching the pattern used for accessList and accessEvents. --------- Co-authored-by: jwasinger <j-wasinger@hotmail.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
20 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
implements eccpow consensus engine for Worldland Network