Segfault in recsplit

[davecrighton]

System information

Erigon version:

Build info                               git_branch=prerelease-v3.3.3 git_tag=. git_commit=c8f94f3ac50fafc4dc4dd4da8affc634f747728f 

OS & Version: Linux

Commit hash:

c8f94f3ac50fafc4dc4dd4da8affc634f747728f

Erigon Command (with flags/config):

 Startup command                          cmd="erigon --config=/config/config.yaml --downloader.verify=false --rpc.subscription.filters.timeout=300s --log.dir.disable=true --torrent.upload.rate=16kb --nat=extip:<redacted-ip>"

Consensus Layer: Caplin

Consensus Layer Command (with flags/config):

Chain/Network: Sepolia

Expected behaviour

Not a segfault :)

Actual behaviour

Segfault, see stack backtrace below.

Steps to reproduce the behaviour

This happened during steady state operation on Sepolia, we have 3 nodes in this configuration and only one seems to have crashed.

Backtrace

2026-01-15 08:15:56.992	
kdplyz-internal/r-87mgzmceu3-erignr-0 | unexpected fault address 0x7e961c6fc342
	2026-01-15 08:15:56.992	
kdplyz-internal/r-87mgzmceu3-erignr-0 | fatal error: fault
	2026-01-15 08:15:56.995	
kdplyz-internal/r-87mgzmceu3-erignr-0 | [signal SIGSEGV: segmentation violation code=0x1 addr=0x7e961c6fc342 pc=0x12a2f96]
	2026-01-15 08:15:56.995	
kdplyz-internal/r-87mgzmceu3-erignr-0 | 
	2026-01-15 08:15:56.995	
kdplyz-internal/r-87mgzmceu3-erignr-0 | goroutine 89140 gp=0xc07720ee00 m=62 mp=0xc110b53008 [running]:
	2026-01-15 08:15:56.995	
kdplyz-internal/r-87mgzmceu3-erignr-0 | runtime.throw({0x31fd575?, 0x75?})
	2026-01-15 08:15:56.995	
kdplyz-internal/r-87mgzmceu3-erignr-0 | 	runtime/panic.go:1094 +0x48 fp=0xc0681ab788 sp=0xc0681ab758 pc=0x4b21e8
	2026-01-15 08:15:56.995	
kdplyz-internal/r-87mgzmceu3-erignr-0 | runtime.sigpanic()
	2026-01-15 08:15:56.995	
kdplyz-internal/r-87mgzmceu3-erignr-0 | 	runtime/signal_unix.go:939 +0x26c fp=0xc0681ab7e8 sp=0xc0681ab788 pc=0x4b472c
	2026-01-15 08:15:56.995	
kdplyz-internal/r-87mgzmceu3-erignr-0 | encoding/binary.bigEndian.Uint64(...)
	2026-01-15 08:15:56.995	
kdplyz-internal/r-87mgzmceu3-erignr-0 | 	encoding/binary/binary.go:204
	2026-01-15 08:15:56.995	
kdplyz-internal/r-87mgzmceu3-erignr-0 | github.com/erigontech/erigon/db/recsplit.(*Index).Lookup(0xc0cb030248, 0xb0661a5f8176c4da, 0x10f8774fca61695f)
	2026-01-15 08:15:56.995	
kdplyz-internal/r-87mgzmceu3-erignr-0 | 	github.com/erigontech/erigon/db/recsplit/index.go:425 +0x7f6 fp=0xc0681ab8f0 sp=0xc0681ab7e8 pc=0x12a2f96
	2026-01-15 08:15:56.995	
kdplyz-internal/r-87mgzmceu3-erignr-0 | github.com/erigontech/erigon/db/recsplit.(*IndexReader).twoLayerLookup(0xc0681aba30, {0xc10a3b49c0?, 0x303ed00?, 0xc13a9c4501?})
	2026-01-15 08:15:56.995	
kdplyz-internal/r-87mgzmceu3-erignr-0 | 	github.com/erigontech/erigon/db/recsplit/index_reader.go:80 +0x53 fp=0xc0681ab920 sp=0xc0681ab8f0 pc=0x12a3f13
	2026-01-15 08:15:56.995	
kdplyz-internal/r-87mgzmceu3-erignr-0 | github.com/erigontech/erigon/db/recsplit.(*IndexReader).TwoLayerLookup(0xc10a3b4540?, {0xc10a3b49c0?, 0x4af905?, 0x93c0dc?})
	2026-01-15 08:15:56.995	
kdplyz-internal/r-87mgzmceu3-erignr-0 | 	github.com/erigontech/erigon/db/recsplit/index_reader.go:111 +0x25 fp=0xc0681ab950 sp=0xc0681ab920 pc=0x12a40c5
	2026-01-15 08:15:56.995	
kdplyz-internal/r-87mgzmceu3-erignr-0 | github.com/erigontech/erigon/db/snapshotsync/freezeblocks.(*BlockReader).headerFromSnapshotByHash(0x7ecce35aa0a0?, {0x44, 0xef, 0x8, 0x75, 0x5a, 0x5, 0x24, 0xa6, 0xd0, ...}, ...)
	2026-01-15 08:15:56.995	
kdplyz-internal/r-87mgzmceu3-erignr-0 | 	github.com/erigontech/erigon/db/snapshotsync/freezeblocks/block_reader.go:1005 +0x125 fp=0xc0681abaa0 sp=0xc0681ab950 pc=0x1aad5c5
	2026-01-15 08:15:56.995	
kdplyz-internal/r-87mgzmceu3-erignr-0 | github.com/erigontech/erigon/db/snapshotsync/freezeblocks.(*BlockReader).HeaderByHash(0xc0013f4200, {0xc10a51f980?, 0x6b5ae00?}, {0x7ecce35aa0a0?, 0xc10a51f980?}, {0x44, 0xef, 0x8, 0x75, 0x5a, ...})
	2026-01-15 08:15:56.995	
kdplyz-internal/r-87mgzmceu3-erignr-0 | 	github.com/erigontech/erigon/db/snapshotsync/freezeblocks/block_reader.go:616 +0x1a9 fp=0xc0681abb50 sp=0xc0681abaa0 pc=0x1aaa669
	2026-01-15 08:15:56.995	
kdplyz-internal/r-87mgzmceu3-erignr-0 | github.com/erigontech/erigon/db/snapshotsync/freezeblocks.(*BlockReader).HeaderNumber(0xc0013f4200, {0x3b7af00, 0x6b5ae00}, {0x7ecce35aa0a0, 0xc10a51f980}, {0x44, 0xef, 0x8, 0x75, 0x5a, ...})
	2026-01-15 08:15:56.995	
kdplyz-internal/r-87mgzmceu3-erignr-0 | 	github.com/erigontech/erigon/db/snapshotsync/freezeblocks/block_reader.go:581 +0xa9 fp=0xc0681abbb0 sp=0xc0681abb50 pc=0x1aaa309
	2026-01-15 08:15:56.995	
kdplyz-internal/r-87mgzmceu3-erignr-0 | github.com/erigontech/erigon/p2p/protocols/eth.AnswerGetBlockBodiesQuery({0x7ecce35aa000, 0xc10a51f980}, {0xc0a2893400, 0x58, 0xc0876a4800?}, {0x7ecbe7cc2bd0, 0xc0013f4200})
	2026-01-15 08:15:56.995	
kdplyz-internal/r-87mgzmceu3-erignr-0 | 	github.com/erigontech/erigon/p2p/protocols/eth/handlers.go:157 +0x195 fp=0xc0681abc80 sp=0xc0681abbb0 pc=0x2520c35
	2026-01-15 08:15:56.995	
kdplyz-internal/r-87mgzmceu3-erignr-0 | github.com/erigontech/erigon/p2p/sentry/sentry_multi_client.(*MultiClient).getBlockBodies66(0xc076a903c0, {0x3b7b688, 0xc077166aa0}, 0xc121429180, {0x3ba96e0, 0xc076a29860})
	2026-01-15 08:15:56.995	
kdplyz-internal/r-87mgzmceu3-erignr-0 | 	github.com/erigontech/erigon/p2p/sentry/sentry_multi_client/sentry_multi_client.go:605 +0x209 fp=0xc0681abdd0 sp=0xc0681abc80 pc=0x256cbc9
	2026-01-15 08:15:56.995	
kdplyz-internal/r-87mgzmceu3-erignr-0 | github.com/erigontech/erigon/p2p/sentry/sentry_multi_client.(*MultiClient).handleInboundMessage(0xffffffffffffffff?, {0x3b7b688?, 0xc077166aa0?}, 0x1?, {0x3ba96e0?, 0xc076a29860?})
	2026-01-15 08:15:56.995	
kdplyz-internal/r-87mgzmceu3-erignr-0 | 	github.com/erigontech/erigon/p2p/sentry/sentry_multi_client/sentry_multi_client.go:1028 +0x70 fp=0xc0681abe28 sp=0xc0681abdd0 pc=0x2571130
	2026-01-15 08:15:56.995	
kdplyz-internal/r-87mgzmceu3-erignr-0 | github.com/erigontech/erigon/p2p/sentry/sentry_multi_client.(*MultiClient).HandleInboundMessage(0xc076a903c0, {0x3b7b688, 0xc077166aa0}, 0xc121429180, {0x3ba96e0, 0xc076a29860})
	2026-01-15 08:15:56.995	
kdplyz-internal/r-87mgzmceu3-erignr-0 | 	github.com/erigontech/erigon/p2p/sentry/sentry_multi_client/sentry_multi_client.go:999 +0x91 fp=0xc0681abec0 sp=0xc0681abe28 pc=0x2570cd1
	2026-01-15 08:15:56.995	
kdplyz-internal/r-87mgzmceu3-erignr-0 | github.com/erigontech/erigon/p2p/sentry/sentry_multi_client.(*MultiClient).HandleInboundMessage-fm({0x3b7b688?, 0xc077166aa0?}, 0x457320?, {0x3ba96e0?, 0xc076a29860?})
	2026-01-15 08:15:56.995	
kdplyz-internal/r-87mgzmceu3-erignr-0 | 	<autogenerated>:1 +0x46 fp=0xc0681abf00 sp=0xc0681abec0 pc=0x2572346
	2026-01-15 08:15:56.995	
kdplyz-internal/r-87mgzmceu3-erignr-0 | github.com/erigontech/erigon/p2p/sentry/libsentry.pumpStreamLoop[...].func2()
	2026-01-15 08:15:56.995	
kdplyz-internal/r-87mgzmceu3-erignr-0 | 	github.com/erigontech/erigon/p2p/sentry/libsentry/loop.go:143 +0x136 fp=0xc0681abfe0 sp=0xc0681abf00 pc=0x2536516
	2026-01-15 08:15:56.996	
kdplyz-internal/r-87mgzmceu3-erignr-0 | runtime.goexit({})
	2026-01-15 08:15:56.996	
kdplyz-internal/r-87mgzmceu3-erignr-0 | 	runtime/asm_amd64.s:1693 +0x1 fp=0xc0681abfe8 sp=0xc0681abfe0 pc=0x4bab41
	2026-01-15 08:15:56.996	
kdplyz-internal/r-87mgzmceu3-erignr-0 | created by github.com/erigontech/erigon/p2p/sentry/libsentry.pumpStreamLoop[...] in goroutine 89056
	2026-01-15 08:15:56.996	
kdplyz-internal/r-87mgzmceu3-erignr-0 | 	github.com/erigontech/erigon/p2p/sentry/libsentry/loop.go:137 +0x287
	2026-01-15 08:15:56.996	
kdplyz-internal/r-87mgzmceu3-erignr-0 | 
	2026-01-15 08:15:56.996	
kdplyz-internal/r-87mgzmceu3-erignr-0 | goroutine 1 gp=0xc000002380 m=nil [chan receive, 5398 minutes]:
	2026-01-15 08:15:56.996	
kdplyz-internal/r-87mgzmceu3-erignr-0 | runtime.gopark(0xc00299d6c0?, 0x4852bf?, 0xa0?, 0xd6?, 0x485300?)
	2026-01-15 08:15:56.996	
kdplyz-internal/r-87mgzmceu3-erignr-0 | 	runtime/proc.go:460 +0xce fp=0xc07f277688 sp=0xc07f277668 pc=0x4b230e
	2026-01-15 08:15:56.996	
kdplyz-internal/r-87mgzmceu3-erignr-0 | runtime.chanrecv(0xc001b96460, 0x0, 0x1)
	2026-01-15 08:15:56.996	
kdplyz-internal/r-87mgzmceu3-erignr-0 | 	runtime/chan.go:667 +0x473 fp=0xc07f277700 sp=0xc07f277688 pc=0x4447f3
	2026-01-15 08:15:56.996	
kdplyz-internal/r-87mgzmceu3-erignr-0 | runtime.chanrecv1(0xc0003b0930?, 0x3b996f0?)
	2026-01-15 08:15:56.996	
kdplyz-internal/r-87mgzmceu3-erignr-0 | 	runtime/chan.go:509 +0x12 fp=0xc07f277728 sp=0xc07f277700 pc=0x444352
	2026-01-15 08:15:56.996	
kdplyz-internal/r-87mgzmceu3-erignr-0 | github.com/erigontech/erigon/node.(*Node).Wait(...)
	2026-01-15 08:15:56.996	
kdplyz-internal/r-87mgzmceu3-erignr-0 | 	github.com/erigontech/erigon/node/node.go:268
	2026-01-15 08:15:56.996	
kdplyz-internal/r-87mgzmceu3-erignr-0 | github.com/erigontech/erigon/cmd/erigon/node.(*ErigonNode).Serve(0xc076fec050)
	2026-01-15 08:15:56.996	
kdplyz-internal/r-87mgzmceu3-erignr-0 | 	github.com/erigontech/erigon/cmd/erigon/node/node.go:58 +0x65 fp=0xc07f277778 sp=0xc07f277728 pc=0x2636b65
	2026-01-15 08:15:56.996	
kdplyz-internal/r-87mgzmceu3-erignr-0 | main.runErigon(0xc0009d7300)
	2026-01-15 08:15:56.996	
kdplyz-internal/r-87mgzmceu3-erignr-0 | 	github.com/erigontech/erigon/cmd/erigon/main.go:103 +0x558 fp=0xc07f277ad8 sp=0xc07f277778 pc=0x26b1a98
	2026-01-15 08:15:56.996	
kdplyz-internal/r-87mgzmceu3-erignr-0 | github.com/erigontech/erigon/cmd/utils/app.MakeApp.func1(0xc0009d7300)
	2026-01-15 08:15:56.996	
kdplyz-internal/r-87mgzmceu3-erignr-0 | 	github.com/erigontech/erigon/cmd/utils/app/make_app.go:70 +0x15c fp=0xc07f277bf8 sp=0xc07f277ad8 pc=0x26849dc
	2026-01-15 08:15:56.996	
kdplyz-internal/r-87mgzmceu3-erignr-0 | github.com/urfave/cli/v2.(*Command).Run(0xc00067f080, 0xc0009d7300, {0xc000052070, 0x7, 0x7})
	2026-01-15 08:15:56.996	
kdplyz-internal/r-87mgzmceu3-erignr-0 | 	github.com/urfave/cli/v2@v2.27.7/command.go:276 +0x7c2 fp=0xc07f277e68 sp=0xc07f277bf8 pc=0x896c82
	2026-01-15 08:15:56.996	
kdplyz-internal/r-87mgzmceu3-erignr-0 | github.com/urfave/cli/v2.(*App).RunContext(0xc001c3ce00, {0x3b7af00, 0x6b5ae00}, {0xc000052070, 0x7, 0x7})
	2026-01-15 08:15:56.996	
kdplyz-internal/r-87mgzmceu3-erignr-0 | 	github.com/urfave/cli/v2@v2.27.7/app.go:333 +0x5a5 fp=0xc07f277ec8 sp=0xc07f277e68 pc=0x893645
	2026-01-15 08:15:56.996	
kdplyz-internal/r-87mgzmceu3-erignr-0 | github.com/urfave/cli/v2.(*App).Run(...)
	2026-01-15 08:15:56.996	
kdplyz-internal/r-87mgzmceu3-erignr-0 | 	github.com/urfave/cli/v2@v2.27.7/app.go:307
	2026-01-15 08:15:56.996	
kdplyz-internal/r-87mgzmceu3-erignr-0 | main.main()
	2026-01-15 08:15:56.996	
kdplyz-internal/r-87mgzmceu3-erignr-0 | 	github.com/erigontech/erigon/cmd/erigon/main.go:44 +0xad fp=0xc07f277f50 sp=0xc07f277ec8 pc=0x26b144d
	2026-01-15 08:15:56.996	
kdplyz-internal/r-87mgzmceu3-erignr-0 | runtime.main()
	2026-01-15 08:15:56.996	
kdplyz-internal/r-87mgzmceu3-erignr-0 | 	runtime/proc.go:285 +0x29d fp=0xc07f277fe0 sp=0xc07f277f50 pc=0x47b11d
	2026-01-15 08:15:56.996	
kdplyz-internal/r-87mgzmceu3-erignr-0 | runtime.goexit({})
	2026-01-15 08:15:56.996	
kdplyz-internal/r-87mgzmceu3-erignr-0 | 	runtime/asm_amd64.s:1693 +0x1 fp=0xc07f277fe8 sp=0xc07f277fe0 pc=0x4bab41
	2026-01-15 08:15:56.996	
kdplyz-internal/r-87mgzmceu3-erignr-0 | 

[davecrighton]

I think the issue may be here:

erigon/db/snapshotsync/snapshots.go

Line 1474 in 649312d

segment.closeIdx()

This can happen even if the refcount on the segment is non-zero. Meaning that memory can be unmapped while it is still being used.

The timing window could be reduced significantly by checking the ref count before doing the close but of course there is s till a small window that a reader could be added between the check and the closeIdx() call.

So the some other possibilities are:

• to find a mechanism to block or fail readers when a segment is about to be deleted, then wait for the reference count to become 0.
• For closeIdx() to defer the delete until after the new segment is available then have a background process tidy up the segment once its reference count drops to zero.

I'm happy to take a stab at a PR for this but would be great but I'm not sure I have enough context to ensure I'm fully aware of the ramifications of either of my 2 suggestions so would appreciate some guidance first.

[AskAlexSharov]

In-theory - there is 2-steps algo. If file X get merged to bigger segment and not useful anymore - then:

• visibleFiles re-calculated and file X will not be there (because bigger segment available). New readers are working only with visibleFiles (they. Get pointer to visibleFiles array) - means new readers will not see file X and no refcnt.Increment() on file X anymore.
• last existing reader will close file (when refcnt is 0 - close). Existing readers holding pointer to old visibleFiles array.

Maybe we need “mark X as ready for delete” after “re-calc visibleFiles” (maybe we already do, i don’t remember)

[AskAlexSharov]

If you want work on it - you are welcome.

[davecrighton]

last existing reader will close file (when refcnt is 0 - close). Existing readers holding pointer to old visibleFiles array.

That's a nice way to avoid the need for a background process. I'll try and carve out some time to try out a code-change next week and let you know how I get on.

[VBulikov]

yperbasis added Imp1 as Importance