release: gloas (glamsterdam-devnet-6)#864
Merged
Merged
Conversation
… sets
Restructure the ClickHouse migrations from a single flat chain into per-schema
sets that target a database chosen at apply time, so the same schema can be
migrated into multiple databases (e.g. default + devnets) from one source of
truth — no per-database copies, no separate migrator.
Migrations
- Split deploy/migrations/clickhouse/{001_init,002_fast_confirmation,
003_state_metrics} into per-schema sets: xatu/ (raw event schema, formerly the
`default` tables), observoor/, admin/.
- Make every set database-agnostic:
- drop db qualifiers (default./observoor./admin. -> unqualified, resolved via
the connection's database=)
- Distributed engines target currentDatabase() instead of a hardcoded db
- remove CREATE DATABASE (databases are created out-of-band by the migrator)
- keep ON CLUSTER '{cluster}' and the {database} macro in Replicated paths so
each database gets its own ZooKeeper path
- Strip the one-time generator header.
Local docker
- clickhouse-migrate.sh: discover each set directory and apply it to its target
database(s). Convention: directory name == database name; remap/fan-out via
MIGRATE_DB_OVERRIDES (default: xatu=default,devnets).
- clickhouse-db-init.sh: discover + CREATE DATABASE the targets before migrations
run (golang-migrate cannot bootstrap the db it connects to).
- docker-compose: db-init + migrator wired to the scripts via a shared
x-ch-migrate-env anchor; drop the obsolete schema_migrations bootstrap from the
clickhouse-02 init (each set now tracks its own schema_migrations_<set>).
Other
- correctness_test: drop the now-dead "schema_migrations" skip entry.
- cannon-smoke-test: bump the migrator-wait ceiling 120s -> 240s to cover the
multi-database fan-out (loop still early-exits on success).
- create-new-sentry-event command: point new event migrations at the xatu/ set
and document the database-agnostic rules.
BREAKING CHANGE: ClickHouse migrations now live under per-set subdirectories
(deploy/migrations/clickhouse/<set>/) and are database-agnostic. The flat layout
and the global schema_migrations table are gone (hard cutover, no backwards
compatibility) — wipe local volumes before bringing the stack back up. Anything
that runs these migrations must apply each set with database= per target and a
per-set x-migrations-table=schema_migrations_<set>.
Validated locally against a 2-node ClickHouse cluster: the matrix builds 217
tables in both `default` and `devnets`, 77 in `observoor`, 5 in `admin`, with
ON CLUSTER replication intact across both nodes.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…migrations feat(clickhouse)!: database-agnostic per-schema migration sets
PR #847 split the ClickHouse migrations into per-schema sets that are applied into a database chosen at apply time (golang-migrate `database=`), so the SQL must never pin a database itself. Nothing currently guards that invariant, so a new migration with a hardcoded `default.` qualifier or a Distributed engine pointing at a literal db would pass CI and only surface as data landing in the wrong database. Add a dedicated workflow that enforces it on every PR touching the migrations. New workflow `.github/workflows/clickhouse-migrations-lint.yaml` runs two jobs in parallel (no inter-job `needs`), path-filtered to `deploy/migrations/clickhouse/**`: - database-agnostic lint (deploy/migrations/clickhouse/lint.sh) Pure bash/awk, no dependencies. Enforces three rules over every *.sql: 1. no `CREATE DATABASE` (databases are provisioned by the migrator) 2. no database-qualified identifiers (e.g. `default.foo`) 3. every `Distributed(...)` uses currentDatabase() as its database (2nd) arg Rules 1-2 run against a sanitized view of each file in which `--` / `/* */` comments and single-quoted string contents are blanked (line numbers preserved), so legitimate dotted text inside COMMENT '...' (e.g. 'ethseer.io', '(e.g., V1, V2)') and the {database} macro inside Replicated ZooKeeper paths never trip the qualifier check. Rule 3 reconstructs statements by splitting the sanitized text on `;`. Emits GitHub `::error` inline annotations in CI. - clickhouse syntax check (deploy/migrations/clickhouse/check-syntax.sh) Validates every migration parses with the real ClickHouse parser via `clickhouse format --multiquery --quiet`, run inside the clickhouse/clickhouse-server image (version pinned to CHVER, matching docker-compose). Catches malformed SQL before it reaches the migrator. Raises --max_query_size past the 256 KiB parser default, which otherwise rejects the valid 364 KiB xatu/001_init.up.sql. Both scripts accept an optional MIGRATIONS_DIR arg and run locally (clickhouse on PATH, override with CLICKHOUSE_BIN) so devs can check before pushing. Validated: lint + syntax pass on all 10 current migration files (locally and via the exact workflow docker invocation); negative tests confirm the lint catches all three violations without false-positives on comment/string dots, and the syntax check rejects malformed SQL with an annotation. shellcheck clean. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
ci(clickhouse): lint migrations for database-agnosticism + syntax
* fix(libp2p): use local_peer_id for join/leave routing and dedup libp2p_join and libp2p_leave events come from go-libp2p-pubsub's RawTracer Join(topic)/Leave(topic) callbacks, which carry no remote peer. The clickhouse route validator required a non-empty peer id and dropped every join/leave event (visible as 'nil PeerId: invalid event' in consumoor), so neither table has received data since the clickhouse-raw cutover. Add a local_peer_id field to the join/leave client metadata, populated from the producer's own host ID, and key routing/dedup off it (peer_id_unique_key -> local_peer_id_unique_key). Migration 004 recreates the two tables with the renamed sorting-key column, preserving existing rows via a temporary backup. Pairs with the tysm tracer change that stamps the local host ID onto join/leave. Co-authored-by: original branch feat/libp2p-join-leave-local-peer-id * fix(migration,test): comment-safe migration splitting + goconst constants - Migration 004: remove semicolons from SQL comments. golang-migrate's x-multi-statement splitter is not comment-aware, so a ';' inside the leading comment produced a comment-only chunk and failed with 'code 62, Empty query'. - Tests: extract repeated column/topic literals into shared constants to satisfy goconst. * fix(migration): make 004 database-agnostic Drop hardcoded 'default.' table prefixes and use currentDatabase() in the Distributed engine database arg, matching the db-agnostic migration convention from the per-set migrations refactor. Unqualified table names resolve to the migrator connection's database.
Adds an execution-layer dimension to cannon that extracts canonical execution-layer datasets with the third-party `cryo` binary and writes them straight to ClickHouse, mirroring the per-deriver/per-cursor design of the consensus-layer cannon. Every cryo dataset flows proto DecoratedEvent -> ClickHouse route -> sink, reusing the existing sink layer (no bespoke pipeline). Datasets (16, all cryo-fed): block, transaction, logs, traces, native_transfers, erc20_transfers, erc721_transfers, contracts, balance_diffs, storage_diffs, nonce_diffs, balance_reads, storage_reads, nonce_reads, four_byte_counts, address_appearances. Key pieces: - pkg/cannon/execution/cryo: dataset-agnostic cryo runner (--hex, zstd, column projection) + typed parquet reader. - pkg/cannon/deriver/execution: one deriver per dataset, chunked repeated-payload events, shared internal_index stamping (legacy cumcount parity). - pkg/cannon/iterator/backfilling_block_iterator: CL-finality-gated block iterator (never iterates past the execution block of the CL-finalized beacon block); head walks up, backfill walks down to a floor. State-read datasets are floored to block >=1 (genesis is untraceable by cryo's state tracer). - pkg/clickhouse/route/execution: 16 columnar ClickHouse routes. - proto: 16 ExecutionCanonical* event types + CannonType + CannonLocation wrappers + BackfillingBlockMarker. - Dockerfile: build cryo from a pinned git commit (master exposes gas_limit; pinning gives reproducible builds + stable state-read semantics). Schema fix: canonical_execution_four_byte_counts ORDER BY now includes `signature` -- the prior key collapsed a tx's multiple selectors into one row under ReplacingMergeTree. Validated against the legacy (old-script) prod data across genesis->24M and all fork boundaries: 12/16 datasets byte-identical; the 3 state-reads capture more (newer cryo, desired); four_byte_counts fixed; remaining diffs are prod's known-missing data (the reason for this work) and a single DAO-fork balance_diffs.to_value edge. End-to-end run validated on mainnet (beacon + EL + coordinator + ClickHouse): CL-gating, per-dataset Postgres cursors, and the direct-to-ClickHouse sink all confirmed. Out of scope: transaction structlog (not cryo-fed the same way); extra_data_string for non-UTF8 blocks (proto3 string limit); range-sharded backfill (future). BREAKING CHANGE: the xatu-server output (type: xatu) is now rejected by cannon at config validation, for both CL and EL data -- cannon writes directly to ClickHouse. Existing cannon configs using a `type: xatu` output must switch to a `type: clickhouse` output. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Folds the standalone `execution:` config silo into the established cannon conventions and groups derivers by layer: - `derivers:` becomes `derivers.consensus.*` (the 15 beacon derivers, moved) and `derivers.execution.*` (the 16 cryo derivers, with short names -- block, transaction, logs, ...). This also resolves the key clash with the existing consensus `executionTransaction` deriver. - The shared EL connection moves to `ethereum.executionNodeAddress`. - The cryo runner config is now a top-level `cryo:` block. - The cryo package moves to `pkg/cryo` -- it has zero cannon coupling, so it is a reusable top-level package alongside pkg/clickhouse, pkg/output, etc. - EL derivers are wired whenever any is enabled (per-deriver, like the consensus side); the old `execution.enabled` master toggle is removed. Config plumbing only -- no logic or data-path changes. Adds a regression test that loads example_cannon.yaml through the real config loader and validates it, so the shipped example can't drift from the structs. BREAKING CHANGE: all consensus deriver config paths move from `derivers.X` to `derivers.consensus.X`, and the execution config moves from the top-level `execution:` block to `derivers.execution.*` + `ethereum.executionNodeAddress` + a top-level `cryo:` block. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…l_ tables Cannon writes canonical_* events to ClickHouse and advances a per-dataset Postgres cursor as soon as an insert succeeds. With the Distributed engine's default async forwarding (distributed_foreground_insert=0), an INSERT is acknowledged once the rows are buffered on the coordinating node and queued for best-effort background delivery to the peer shards. If that forward later fails (e.g. a shard is briefly unreachable and the send times out), the rows are silently dropped -- but the cursor has already advanced, so cannon never revisits the range and the gap is permanent. This is invisible on dense tables (every block still has rows on some shard, so queries by block_number look complete) and only surfaces on sparse, one-row-per-block tables like canonical_execution_block, which lose roughly half their rows -- the ones whose sharding key hashes to the shard whose forward failed. Fix: default distributed_foreground_insert=1 in applyCanonicalTableDefaults so a successful insert means the data is durable on the target shard before the cursor advances. Scope matches the existing insert_quorum=auto default (canonical_ tables only), it is overridable via insertSettings, and it is a no-op on single-node / non-Distributed setups. Also: - document both canonical_ defaults on the InsertSettings field and in example_cannon.yaml, incl. the single-replica-per-shard override (insert_quorum: 0 -- no peer to form a quorum with, so auto just adds contention) - replace two manual map-copy loops with maps.Copy (golangci-lint mapsloop) Verified end-to-end against a 2-shard/1-replica local cluster: with the new default, canonical_execution_block backfills land evenly across both shards with no loss, and the rendered statement is `INSERT ... SETTINGS distributed_foreground_insert = 1, insert_quorum = ...`. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The flat ethereum config conflated three concerns at one level: connection
addresses (beaconNodeAddress, executionNodeAddress), beacon auth
(beaconNodeHeaders), and beacon-block cache/preload tuning (blockCacheSize,
blockCacheTtl, blockPreloadWorkers, blockPreloadQueueSize). The cache/preload
knobs in particular read as if they applied to the whole ethereum config when
they are beacon-block specific.
Group by layer instead:
ethereum:
overrideNetworkName: ... # network identity (derived from beacon if unset)
beacon:
address: ...
headers: { ... }
blockCacheSize / blockCacheTtl / blockPreloadWorkers / blockPreloadQueueSize
execution:
address: ... # creds may be embedded (https://user:pass@host)
Notes:
- overrideNetworkName stays at the ethereum root: it is a network-identity
override for the whole context, not a beacon connection knob.
- execution has no headers field on purpose: cryo only takes --rpc <url> with
embedded credentials and has no header-injection path. Documented inline.
- cryo stays a top-level config block (reusable, EL-extraction tool config),
unchanged by this refactor.
- The conditional requirement is unchanged and now reads naturally:
ethereum.beacon.address is always required; ethereum.execution.address (and a
valid cryo config) is required only when any execution deriver is enabled, so
ethereum.execution can be omitted entirely for a consensus-only cannon.
- Hardened the beacon-headers override path with a nil-map guard (this was the
panic hit when the authorization-header override fired with no headers set).
BREAKING CHANGE: ethereum config keys move beaconNodeAddress -> beacon.address,
beaconNodeHeaders -> beacon.headers, block{CacheSize,CacheTtl,PreloadWorkers,
PreloadQueueSize} -> beacon.*, executionNodeAddress -> execution.address.
overrideNetworkName is unchanged. cannon-only; other components' ethereum
configs are separate packages and untouched.
go build ./... and the cannon/clickhouse test suites pass, including the
example_cannon.yaml loader regression test.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…c G115) The block-preload worker loop converted BlockPreloadWorkers (uint64) to int, which gosec flags as a potential integer-overflow conversion (G115). The loop index isn't used as an int, so iterate in uint64 directly and drop the cast. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
feat(cannon)!: add execution-layer (EL) dimension via cryo
v0.16 removed the ADD_PEER / REMOVE_PEER gossipsub tracer events (replaced by per-stream ON_NEW_OUTBOUND_STREAM / ON_CLOSED_OUTBOUND_STREAM). Bump the dependency and remove the now-dead producer-side handling for those two events; peer connect/disconnect remains covered by the host notifee (CONNECTED / DISCONNECTED). The xatu proto event types are kept for schema / historical-data backward compatibility.
EL cannon shells out to the third-party `cryo` binary at runtime (pkg/cryo/cryo.go, BinaryPath defaults to "cryo" on $PATH), but the goreleaser-published images never contained it. The cannon PR only added cryo to the local Dockerfile; goreleaser builds from goreleaser-scratch.Dockerfile (distroless) and goreleaser-debian.Dockerfile, neither of which bundles cryo. So EL cannon would fail in every released image the moment it execs cryo. Add a third release image variant — xatu + cryo — built release-only: :VERSION-cryo :cryo-latest (or :SUFFIX-cryo-latest for suffix/prerelease tags) alongside the existing scratch (:VERSION / :latest) and debian (:VERSION-debian) images, which stay lean and unchanged. Why a separate build-push-action step instead of a .goreleaser.yaml entry: - distroless `cc-debian12` (the scratch image behind :latest) lacks the libssl/libcrypto cryo links against, so it can't carry cryo without hauling per-arch openssl libs. The debian base can. - `dockers.skip` is not a valid goreleaser field, so a cryo entry could not be gated and would build on every PR via test-build.yaml, compiling cryo (Rust, incl. arm64-under-QEMU) and adding 30-60 min to PR CI. So the cryo image is built in goreleaser.yaml (release/tag only), reusing ./Dockerfile (which already does Go-build + cryo-build). The pinned cryo layer is cache-stable, so GHA layer cache compiles cryo once and reuses it across releases until the pinned ref changes. Single source of truth for the cryo ref: ./Dockerfile is the only Dockerfile that builds cryo, and every path (docker build, docker compose, `make docker`, the smoke tests, and the release image) builds it — so the `ARG CRYO_GIT_REF` default in that one file is the single place to bump cryo. (An earlier revision moved this to a .cryo-version file injected as a build-arg, but docker-compose builds the xatu image without that arg, so a no-default Dockerfile broke `docker compose up`. The ARG default is the robust single source.) Also: ./Dockerfile now stamps version ldflags (Release/GitCommit/GOOS/GOARCH) via VERSION/GIT_COMMIT/TARGET* build-args so the released cryo image reports its tag like the other images (defaults dev/unknown for local/CI/compose builds). Added a `make docker` convenience target. Operators must point EL cannon at the -cryo tag (:cryo-latest / :VERSION-cryo); :latest deliberately stays cryo-free and lean. Verified locally: `make docker` (no build-arg, the docker-compose path) builds (exit 0, 248MB); cryo runs at the pinned ref (`cryo 559b654`); libssl/libcrypto resolve in-image; version ldflags inject; xatu `cannon` and `version` subcommands work. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
ci(cannon): ship a cryo-bundled release image for EL cannon
The xatu+cryo image step compiled cryo from Rust source for linux/arm64 under QEMU emulation, which hung the v1.18.1 release for 35+ min on a cold cache (the scratch/debian images and the GitHub release had already been published by goreleaser; only the cryo image was still grinding). EL cannon is a server-side data pipeline that runs on amd64, so the arm64 cryo image is dead weight. Drop linux/arm64 from the cryo build: native amd64 compiles cryo in ~2-3 min, no emulation, no hang. The other two image variants (scratch, debian) remain multi-arch via goreleaser; only the cryo variant is amd64-only. Add linux/arm64 back if cannon ever needs it. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
ci(cannon): build the cryo release image amd64-only
The cryo image step used `cache-to/from: type=gha`, intending to reuse the expensive cryo-builder layer across releases. But GHA cache is scoped per git ref, and releases are tag-triggered: a v1.18.2 run can only restore caches from its own ref or the default branch, never from refs/tags/v1.18.1. So every release got a cache miss and recompiled cryo from Rust source (~10 min), defeating the point. Switch to registry cache (ethpandaops/xatu:cryo-buildcache), which is ref-agnostic — any release can pull it. Now each release reuses the prior cryo-builder layer and skips the Rust compile; a release becomes roughly the Go build + image push (~2-4 min). Bumping CRYO_GIT_REF in ./Dockerfile changes the cryo-builder RUN instruction, which invalidates exactly that cache layer: cryo recompiles once with the new ref, then cache-to re-warms the buildcache for subsequent releases. cache-to uses ignore-error=true so a cache-export hiccup can never fail an actual release. A `cryo-buildcache` tag will appear in the DockerHub repo — that is the cache manifest, not a runnable image. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
ci(cannon): cache the cryo build in a registry image, not GHA cache
The cannon and sentry smoke tests build ./Dockerfile (which compiles cryo) with only gha cache, so a cold PR branch recompiled cryo from Rust source (~10 min) on its first smoke-test run — independent of the release pipeline's registry buildcache. Add the release pipeline's registry cache as a read-only cache-from source on both xatu image builds. Now a smoke test pulls the prebuilt cryo-builder layer from ethpandaops/xatu:cryo-buildcache and skips the compile, as long as the PR doesn't change CRYO_GIT_REF (in which case it correctly recompiles). Read-only by design: PRs still write only their own gha cache (cache-to unchanged), never the shared release buildcache. The sentry-logs image build is untouched (no cryo). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
ci: reuse the cryo registry buildcache in the smoke tests
The breaking cannon refactors on master — `refactor(cannon)!: nest ethereum
config into beacon/execution` and `refactor(cannon)!: group derivers by
layer` — renamed config keys, but cannon-smoke-test.yaml's inline config still
used the old flat keys. cannon fataled at startup:
{"level":"fatal","msg":"ethereum.beacon.address is required"}
so 0 rows ever reached ClickHouse and "Verify Clickhouse has data" timed out.
master doesn't run the smoke test (pull_request only), so nothing caught it —
every PR's cannon-smoke-test has failed since those merged.
Migrate the embedded config:
- derivers.* -> derivers.consensus.* (layer grouping). Consensus derivers
default to enabled:true, so every deriver the test does NOT assert is now
explicitly disabled — including the new ones (elaboratedAttestation,
beaconValidators, beaconSyncCommittee, beaconBlockSyncAggregate). The six
enabled derivers still match seeding.yaml exactly.
- derivers.execution is omitted entirely: those derivers default to
enabled:false, so AnyEnabled() stays false and no ethereum.execution.address
/ cryo / EL RPC is required (this test is consensus-only).
- ethereum.beaconNodeAddress -> ethereum.beacon.address and
ethereum.beaconNodeHeaders -> ethereum.beacon.headers.
Verified locally against the built image: cannon now loads the config,
overrides the network name, and starts cannon mode (then fails only on the
absent ClickHouse, as expected when run standalone) — no config fatal.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
ci(cannon): migrate smoke-test config to nested derivers/ethereum schema
cryo writes its failure message to stdout, not stderr (confirmed: a failing run prints "Failed to get block: ..." on stdout with an empty stderr, exit 1). Collect only captured cmd.Stderr, so every failed cryo invocation surfaced as the opaque "cryo <dataset> [from:to] failed: exit status 1:" with no cause — which made the EL cannon's storage_reads / balance_diffs / nonce_reads errors undiagnosable in production. Capture stdout alongside stderr and include both in the wrapped error via a small cryoOutput helper (stdout first, stderr appended when non-empty). cryo's parquet data goes to --output-dir files, so its stdout is just the bounded progress/error text — safe to buffer. Added a table-driven test for cryoOutput. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
fix(cryo): capture cryo's stdout so failures report the actual error
…through Fulu Adds shared/generated artifacts for 11 new cannon event types (Electra execution requests, rewards endpoints, state metadata, Electra state queues): - event_ingester.proto: payload + Additional*Data messages, Event.Name enum values, ClientMeta.additional_data and DecoratedEvent.data oneof fields. - coordinator.proto: CannonType enum values, CannonLocation* messages, CannonLocation.Data oneof fields. - 11 ClickHouse migration pairs (005-015) under deploy/migrations/clickhouse/xatu. - 11 generated canonical_*.gen.go route batches plus compiling route + test scaffolds (empty event names; Phase B wires them in). Also fixes the route generator to read migrations from the xatu/ subdir (broken by the per-schema migration split) and fixes the route scaffold template's package-shadowing bug.
…+ clickhouse route
…er + clickhouse route
…ndler + clickhouse route
… + clickhouse route
…data, Electra queues)
All three reward derivers now default enabled (block_reward was the only one defaulting off). Verified end-to-end against a mainnet archive beacon node: block_reward ~0.3 GB/yr, sync_committee_reward ~11 GB/yr, attestation_reward ~127 GB/yr (compressed). Update example_cannon.yaml to reflect the enabled-by-default rewards and document their storage cost.
…ches The backfilling checkpoint iterator's GetMarker and setMarker switches were missing all 11 new cannon types, so UpdateLocation failed with 'unknown cannon type' — the derivers fetched and inserted correct rows but could never advance their backfill cursor, leaving them stuck reprocessing the head epoch instead of backfilling. Verified: cursor now advances (next_epoch decrements toward the activation-fork target) and consolidations land across the full epoch range.
Schema audit against the existing canonical_* tables surfaced type inconsistencies in the new cannon tables. Align them with the established conventions (verified live against the beacon API after the change): - gwei amount columns UInt64 -> UInt128, matching canonical_beacon_block_deposit / canonical_beacon_block_withdrawal (005, 006, 013, 014) - deposit withdrawal_credentials String -> FixedString(66), matching canonical_beacon_block_deposit.deposit_data_withdrawal_credentials (005, 013) - canonical_beacon_state_pending_deposit.slot UInt64 -> UInt32, matching the repo-wide slot convention (013) - position_in_block / position_in_queue CODEC ZSTD(1) -> DoubleDelta,ZSTD(1), matching existing monotonic position columns (005-007, 013-015) Regenerated route batch columns (ColUInt128 / SafeColFixedStr) and updated the hand-written FlattenTo + snapshot expectations accordingly. Not changed (deliberate): state tables omit state_id from ORDER BY (state_id is functionally determined by epoch; matches canonical_beacon_validators / proposer_duty precedent), and block_reward/sync_committee_reward omit block_version (the rewards endpoints don't return it).
- Collapse migrations 005-015 into a single 005_cannon_fulu_data pair (none were applied anywhere yet, so renumbering is safe). Verified the combined migration applies cleanly and the route generator discovers all 11 tables. - Fix nil-deref risk in state/reward deriver fetch paths: finality_checkpoint, pending_deposit/partial_withdrawal/consolidation now guard rsp==nil || rsp.Data==nil (matching beacon_sync_committee); block_reward and sync_committee_reward extended to also check .Data. An empty Beacon API response is now a retriable error instead of a panic that strands the cursor. - Add nil-field validation to the consolidation route (source_address / source_pubkey / target_pubkey), matching the deposit/withdrawal routes. - gosec G115 nolint on the pending_deposit slot uint64->uint32 cast, matching the repo convention.
govulncheck v1.4.0 panics ("ForEachElement called on type containing
*types.TypeParam") when scanning this codebase's generics under Go 1.26 — a
bug in its bundled x/tools v0.46.0 (golang/go#80055, fixed by CL 786280 but
not yet tagged; tracked in golang/go#80139). Pin the installer to v1.3.0,
which ships the unaffected x/tools v0.44.0, until x/vuln releases a version
> v1.4.0 with the fix.
With scanning restored, fix the reachable dependency advisories it surfaces:
- golang.org/x/crypto v0.50.0 -> v0.52.0 (GO-2026-5013/5017/5018/5019/5020)
- otlptrace + otlptracehttp v1.40.0 -> v1.43.0 (GO-2026-4985), matching the
otel core already at v1.43.0
Allowlist the two new stdlib advisories that remain (GO-2026-5037 crypto/x509,
GO-2026-5039 net/textproto), both fixed only in go1.26.4 while the toolchain
stays pinned at 1.26.2 to match goreleaser-cross:v1.26.
Several new route FlattenTo paths fell back to a zero/empty value when a
required proto field was absent (e.g. validator_index -> 0, reward -> 0,
proposer_index -> 0). Zero is a valid value for these fields, so the fallback
silently fabricated data indistinguishable from a real zero.
Fix all 11 new cannon tables to the established deposit/withdrawal pattern:
validate() rejects the event (route.ErrInvalidEvent) when a required field is
nil, and appendPayload/appendAdditionalData append directly with no zero
fallback. Covers attestation_reward, sync_committee_reward, block_reward,
finality_checkpoint, the three pending_* queues, and the position_in_block on
the execution_request_{deposit,withdrawal,consolidation} routes.
attestation_reward.inclusion_delay is genuinely optional (absent post-Altair,
the deriver's only activation range) so it becomes Nullable(UInt64) and is
written as NULL when absent rather than a fabricated 0.
Note: this anti-pattern is systemic across the route layer (~378 sites in 57
files, pre-existing); only the tables this PR adds are corrected here.
cannon: add 11 derivers for all Beacon-API data through Fulu
When no DLQ is configured, invalid (permanently unflattenable) events hit the same fail-the-message path as recoverable decode/write errors, forcing Kafka to redeliver them forever — one malformed event halts the whole partition on data that can never become storable. The route layer's own contract (route.ErrInvalidEvent) already states these events are permanently unflattenable and should be dropped rather than retried. Align rejectMessage with that contract: invalid events (like already-handled route rejections) drop and ack, tracked by the MessagesRejected metric. Recoverable failures (decode, write) still fail-fast for redelivery.
… row) (#862) * clickhouse: halt on invalid canonical events instead of dropping Canonical (cannon-backfilled) data is authoritative — a dropped row is a permanent gap in history. For canonical_* tables, route.ErrInvalidEvent now returns a non-permanent invalidEventError so the caller halts and retries (cannon: no checkpoint advance; consumoor: NAK/redeliver) rather than dropping. Non-canonical (live/sentry) invalid events still drop — they are point-in-time and cannot be re-fetched. isCanonical() keys off the canonical_ table prefix. * canonical_beacon_block: strict validate (error-if-absent on required fields) * canonical_beacon_block_attester_slashing: strict validate (error-if-absent on required fields) * canonical_beacon_block_proposer_slashing: strict validate (error-if-absent on required fields) * canonical_beacon_block_deposit: strict validate (error-if-absent on required fields) * canonical_beacon_block_voluntary_exit: strict validate (error-if-absent on required fields) * canonical_beacon_block_bls_to_execution_change: strict validate (error-if-absent on required fields) * canonical_beacon_block_sync_aggregate: strict validate (error-if-absent on required fields) * canonical_beacon_block_execution_transaction: strict validate (error-if-absent on required fields) * canonical_beacon_elaborated_attestation: strict validate (error-if-absent on required fields) * canonical_beacon_blob_sidecar: strict validate (error-if-absent on required fields) * canonical_beacon_committee: strict validate (error-if-absent on required fields) * canonical_beacon_proposer_duty: strict validate (error-if-absent on required fields) * canonical_beacon_state_finality_checkpoint: strict validate (error-if-absent on required fields) * canonical_beacon_state_randao: strict validate (error-if-absent on required fields) * canonical: strict validate for consolidation + attestation_reward; fix aliasing test fixture Apply the audit's missing guards onto the existing (#851) validate()s rather than the workflow agents' from-scratch versions (their pre-#851 base conflicted): consolidation gains block_root/block_version/meta_network_name; attestation_reward gains epoch_start_date_time/meta_network_name. Enrich the elaborated-attestation aliasing test fixture to satisfy the new strict validate (AttestationData + block identity + meta). * consumoor: fix stale canonical-defaults tests (distributed_foreground_insert) b826b35 added distributed_foreground_insert=1 as a canonical_ table default but left TestTableConfigForAppliesCanonicalDefaults / MergesInsertSettings asserting the pre-feature settings, leaving master red. Update expectations to the intended behaviour.
The route's setOptionalEpoch mapped both 0 and FAR_FUTURE_EPOCH (2^64-1) to NULL and nulled a zero balance — discarding values the beacon API returns verbatim (the Validator spec marks these required and documents 'FAR_FUTURE_EPOCH if not exited/activated'). Conflating real data with absence made NULL ambiguous. Keep the columns Nullable(UInt64) (NO type migration — the table is ~350B rows; a type-change mutation would be huge and pointless) and instead always write the exact uint64 the API reports as a set value: balance 0, genesis epoch 0, and FAR_FUTURE_EPOCH are all preserved. Halt (route.ErrInvalidEvent) on a missing index/status/data/slashed, and on empty pubkey/withdrawal_credentials in the fanout tables, rather than skipping or empty-filling. New rows are unambiguous immediately; historical NULLs stay losslessly recoverable by status and can be backfilled later if desired.
Resolves the 48-commit drift between release/gloas and master: - proto enum/field renumbering: master's cannon-cryo and Fulu event IDs are frozen (deployed); all gloas ePBS/BAL event IDs, AdditionalData oneof fields, DecoratedEvent data oneof fields, CannonTypes and CannonLocation oneof fields move above master's ranges. Regenerated all .pb.go via buf. - gloas cannon derivers (block access list, payload attestation, execution payload bid) adapted to master's layer-grouped deriver config (Derivers.Consensus.*) and re-registered in cannon.go. - gloas migrations re-authored into the per-schema layout as xatu/006_gloas_bals_support and xatu/007_gloas_epbs_support, converted to the database-agnostic convention (no default. prefix, Distributed(..., currentDatabase(), ...)). - event categorizer test counts updated for the 7 gloas events on top of master's 27. - envelope cache wired to the nested Beacon.BlockCacheTTL config.
…evnet-6) Bump ethpandaops/go-eth2-client to v0.1.5 and adapt to the spec changes between devnet-4 (alpha.8) and devnet-6 (alpha.11): - ProposerPreferences.gas_limit renamed to target_gas_limit (consensus-specs #5235), through proto, conversions, ClickHouse columns and routes. - EIP-8282 builder execution requests (#5359): execution requests proto gains builder_deposits/builder_exits, converted from the gloas-local ExecutionRequests container now carried by ExecutionPayloadEnvelope and gloas blocks. - Execution request derivers adapted to the versioned VersionedExecutionRequests accessor API. - canonical_beacon_block_withdrawal route now writes withdrawal_type (validator|builder), which the deriver already populated. - canonical_beacon_block_access_list gains the standard meta_consensus_version_major/minor/patch columns.
Replaces the TODO(epbs) placeholders with realistic Gloas payloads (devnet-6 flavored values) and payload-specific column assertions for the sentry SSE, cannon-derived, gossipsub and synthetic event routes.
Found by running sentry + cannon + the full local pipeline against the live devnet (the first devnet run for the cannon side): - canonical_beacon_block route: accept Gloas blocks in the identity field dispatch (strict-validate halted every canonical block flush). - eth_v3_validator_block route: accept Gloas proposed blocks (no inline execution payload; payload columns legitimately empty). - cannon + sentry beacon block events: Gloas bodies carry no transactions (they live in the envelope) — record zero body-tx stats instead of erroring/warning per block. - execution transaction deriver: blob sidecars are best-effort from Gloas (reconstruction from columns is unreliable across clients) — never block the transaction dataset on them. - bump go-eth2-client to the envelope JSON-unwrap fix (ethpandaops/go-eth2-client#36); without it cannon could not fetch any execution payload envelope over JSON.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Brings
release/gloas(#795) up to date with master and glamsterdam-devnet-6. Merges master into the branch (gloas proto IDs renumbered above master's deployed ranges, the two gloas migrations re-authored into the per-schemadeploy/migrations/clickhouse/xatu/layout as 006/007, cannon derivers adapted to the layer-grouped config), and bumps go-eth2-client to the consensus-specs v1.7.0-alpha.11 gloas types (target_gas_limitrename, EIP-8282 builder execution requests,BuilderPendingPayment.proposer_index).Also fixes what live verification against devnet-6 surfaced: canonical_beacon_block and eth_v3_validator_block routes rejecting gloas blocks, body-transaction stats erroring on gloas (transactions live in the envelope), blob sidecar fetches blocking the transaction dataset from gloas onwards, and the 12 ePBS/BAL snapshot fixtures are now seeded. Verified end to end against the live devnet: sentry v2 block capture, cannon's payload-attestation/bid/block derivers writing to ClickHouse, and the full sentry → server → kafka → consumoor → ClickHouse pipeline.
Targets
release/gloasso the branch build picks it up; #795 remains the PR to master. Depends on ethpandaops/go-eth2-client#36, currently pinned as a pseudo-version of that branch — repin after it merges.