feat: add native indexer mode - #2051
Conversation
Signed-off-by: Skylar Simoncelli <skylar@skylar.email>
Signed-off-by: Skylar Simoncelli <skylar@skylar.email>
This comment has been minimized.
This comment has been minimized.
Signed-off-by: Skylar Simoncelli <skylar@skylar.email>
Signed-off-by: Skylar Simoncelli <skylar@skylar.email>
Signed-off-by: Skylar Simoncelli <skylar@skylar.email>
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 918c258eac
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
| telemetry_config: telemetry::Config { tracing_config, metrics_config }, | ||
| } = Config::load().context("load indexer configuration")?; | ||
|
|
||
| log::info!(application_config:?, infra_config:?; "starting embedded indexer"); |
There was a problem hiding this comment.
Redact the Blockfrost credential from startup logs
Whenever --indexer is enabled, this info-level log formats the entire InfraConfig; its derived Debug output includes SpoNodeConfig.blockfrost_id, which is stored as a plain String until it is converted to SecretString later. This exposes the Blockfrost API credential to routine production logs, so log only non-sensitive fields or redact the credential before formatting the configuration.
Useful? React with 👍 / 👎.
| @@ -0,0 +1,10 @@ | |||
| #indexer #api | |||
There was a problem hiding this comment.
Move the change file into the node component
This node feature's change file is outside changes/node/added/, even though the release workflow packages and filters component notes under changes/node (.github/workflows/release-image.yml lines 473-476) and the archive workflow likewise processes only that directory. Consequently this feature will be omitted from the node release changelog; move the file into the node component directory.
AGENTS.md reference: AGENTS.md:L195-L201
Useful? React with 👍 / 👎.
| async move { | ||
| if let Err(error) = crate::indexer::run() { |
There was a problem hiding this comment.
Make the embedded runner cancellable during node shutdown
When shutdown is initiated by SIGINT or by another essential node task failing, rather than by SIGTERM, this synchronous run() call remains inside a single poll of the TaskManager's blocking future and cannot observe the TaskManager exit signal; the nested indexer runtime itself only installs SIGTERM handlers. The Substrate runner therefore waits its full 60-second runtime shutdown timeout before leaking the blocked task, making Ctrl-C and internal-failure shutdowns stall; propagate node cancellation into the nested runtime or otherwise ensure this call can return for every node shutdown path.
Useful? React with 👍 / 👎.
Signed-off-by: Skylar Simoncelli <skylar@skylar.email>
Signed-off-by: Skylar Simoncelli <skylar@skylar.email>
Signed-off-by: Skylar Simoncelli <skylar@skylar.email>
Signed-off-by: Skylar Simoncelli <skylar@skylar.email>
Signed-off-by: Skylar Simoncelli <skylar@skylar.email>
Signed-off-by: Skylar Simoncelli <skylar@skylar.email>
Signed-off-by: Skylar Simoncelli <skylar@skylar.email>
There was a problem hiding this comment.
Opengrep OSS found more than 20 potential problems in the proposed changes. Check the Files changed tab for more details.
There was a problem hiding this comment.
zizmor found more than 20 potential problems in the proposed changes. Check the Files changed tab for more details.
| ARG RUST_VERSION | ||
| FROM lukemathwalker/cargo-chef:0.1.77-rust-$RUST_VERSION-trixie AS chef | ||
| WORKDIR /build | ||
|
|
||
| FROM chef AS planner | ||
| COPY . . | ||
| RUN cargo chef prepare --recipe-path recipe.json | ||
|
|
||
| FROM chef AS builder | ||
| SHELL ["/bin/bash", "-c"] | ||
| ARG PROFILE=release | ||
| COPY --from=planner /build/recipe.json recipe.json | ||
| RUN cargo chef cook --profile $PROFILE --recipe-path recipe.json | ||
| COPY . . | ||
| RUN cargo build -p chain-indexer --locked --features cloud --profile $PROFILE && \ | ||
| mkdir -p /runtime/usr/local/bin && \ | ||
| mv "./target/${PROFILE/dev/debug}/chain-indexer" /runtime/usr/local/bin && \ | ||
| mv /build/chain-indexer/bin/entrypoint.sh /runtime/usr/local/bin && \ | ||
| mkdir -p /runtime/opt/chain-indexer && \ | ||
| mv /build/chain-indexer/config.yaml /runtime/opt/chain-indexer | ||
|
|
||
| FROM debian:trixie-slim@sha256:f6e2cfac5cf956ea044b4bd75e6397b4372ad88fe00908045e9a0d21712ae3ba AS runtime | ||
| RUN apt-get update && \ | ||
| apt-get install -y --no-install-recommends ca-certificates=20250419 && \ | ||
| rm -rf /var/lib/apt/lists/* | ||
| RUN useradd -u 10001 -d /nonexistent -s /usr/sbin/nologin -M -c "" appuser && \ | ||
| passwd -l appuser && \ | ||
| mkdir /var/run/chain-indexer && \ | ||
| chown appuser:appuser /var/run/chain-indexer | ||
| COPY --from=builder --chown=appuser:appuser /runtime / | ||
| USER appuser | ||
| WORKDIR /opt/chain-indexer | ||
| ENTRYPOINT ["entrypoint.sh"] |
| ARG RUST_VERSION | ||
| FROM lukemathwalker/cargo-chef:0.1.77-rust-$RUST_VERSION-trixie AS chef | ||
| WORKDIR /build | ||
|
|
||
| FROM chef AS planner | ||
| COPY . . | ||
| RUN cargo chef prepare --recipe-path recipe.json | ||
|
|
||
| FROM chef AS builder | ||
| SHELL ["/bin/bash", "-c"] | ||
| ARG PROFILE=release | ||
| COPY --from=planner /build/recipe.json recipe.json | ||
| RUN cargo chef cook --profile $PROFILE --recipe-path recipe.json | ||
| COPY . . | ||
| RUN cargo build -p indexer-api --locked --features cloud --profile $PROFILE && \ | ||
| mkdir -p /runtime/usr/local/bin && \ | ||
| mv "./target/${PROFILE/dev/debug}/indexer-api" /runtime/usr/local/bin && \ | ||
| mv /build/indexer-api/bin/entrypoint.sh /runtime/usr/local/bin && \ | ||
| mkdir -p /runtime/opt/indexer-api && \ | ||
| mv /build/indexer-api/config.yaml /runtime/opt/indexer-api | ||
|
|
||
| FROM debian:trixie-slim@sha256:f6e2cfac5cf956ea044b4bd75e6397b4372ad88fe00908045e9a0d21712ae3ba AS runtime | ||
| RUN useradd -u 10001 -d /nonexistent -s /usr/sbin/nologin -M -c "" appuser && \ | ||
| passwd -l appuser && \ | ||
| mkdir /var/run/indexer-api && \ | ||
| chown appuser:appuser /var/run/indexer-api | ||
| COPY --from=builder --chown=appuser:appuser /runtime / | ||
| USER appuser | ||
| WORKDIR /opt/indexer-api | ||
| ENTRYPOINT ["entrypoint.sh"] | ||
| EXPOSE 8088 |
| ARG RUST_VERSION | ||
| FROM lukemathwalker/cargo-chef:0.1.77-rust-$RUST_VERSION-trixie AS chef | ||
| WORKDIR /build | ||
|
|
||
| FROM chef AS planner | ||
| COPY . . | ||
| RUN cargo chef prepare --recipe-path recipe.json | ||
|
|
||
| FROM chef AS builder | ||
| SHELL ["/bin/bash", "-c"] | ||
| ARG PROFILE=release | ||
| COPY --from=planner /build/recipe.json recipe.json | ||
| RUN cargo chef cook --profile $PROFILE --recipe-path recipe.json | ||
| COPY . . | ||
| RUN cargo build -p indexer-standalone --locked --features standalone --profile $PROFILE && \ | ||
| mkdir -p /runtime/usr/local/bin && \ | ||
| mv "./target/${PROFILE/dev/debug}/indexer-standalone" /runtime/usr/local/bin && \ | ||
| mv /build/indexer-standalone/bin/entrypoint.sh /runtime/usr/local/bin && \ | ||
| mkdir -p /runtime/opt/indexer-standalone && \ | ||
| mv /build/indexer-standalone/config.yaml /runtime/opt/indexer-standalone | ||
|
|
||
| FROM debian:trixie-slim@sha256:f6e2cfac5cf956ea044b4bd75e6397b4372ad88fe00908045e9a0d21712ae3ba AS runtime | ||
| RUN apt-get update && \ | ||
| apt-get install -y --no-install-recommends ca-certificates=20250419 && \ | ||
| rm -rf /var/lib/apt/lists/* | ||
| RUN useradd -u 10001 -d /nonexistent -s /usr/sbin/nologin -M -c "" appuser && \ | ||
| passwd -l appuser && \ | ||
| mkdir /var/run/indexer-standalone && \ | ||
| chown appuser:appuser /var/run/indexer-standalone && \ | ||
| mkdir /data && \ | ||
| chown appuser:appuser /data | ||
| COPY --from=builder --chown=appuser:appuser /runtime / | ||
| USER appuser | ||
| WORKDIR /opt/indexer-standalone | ||
| ENTRYPOINT ["entrypoint.sh"] | ||
| EXPOSE 8088 |
| ARG RUST_VERSION | ||
| FROM lukemathwalker/cargo-chef:0.1.77-rust-$RUST_VERSION-trixie AS chef | ||
| WORKDIR /build | ||
|
|
||
| FROM chef AS planner | ||
| COPY . . | ||
| RUN cargo chef prepare --recipe-path recipe.json | ||
|
|
||
| FROM chef AS builder | ||
| SHELL ["/bin/bash", "-c"] | ||
| ARG PROFILE=release | ||
| COPY --from=planner /build/recipe.json recipe.json | ||
| RUN cargo chef cook --profile $PROFILE --recipe-path recipe.json | ||
| COPY . . | ||
| RUN cargo build -p indexer-tests --locked --profile $PROFILE && \ | ||
| mkdir -p /runtime/usr/local/bin && \ | ||
| mv "./target/${PROFILE/dev/debug}/indexer-tests" /runtime/usr/local/bin | ||
|
|
||
| FROM debian:trixie-slim@sha256:66b37a5078a77098bfc80175fb5eb881a3196809242fd295b25502854e12cbec AS runtime | ||
| RUN useradd -u 10001 -d /nonexistent -s /usr/sbin/nologin -M -c "" appuser && \ | ||
| passwd -l appuser && \ | ||
| mkdir /var/run/indexer-tests && \ | ||
| chown appuser:appuser /var/run/indexer-tests | ||
| COPY --from=builder --chown=appuser:appuser /runtime / | ||
| USER appuser | ||
| WORKDIR /opt/indexer-tests | ||
| ENTRYPOINT ["indexer-tests"] |
| ARG RUST_VERSION | ||
| FROM lukemathwalker/cargo-chef:0.1.77-rust-$RUST_VERSION-trixie AS chef | ||
| WORKDIR /build | ||
|
|
||
| FROM chef AS planner | ||
| COPY . . | ||
| RUN cargo chef prepare --recipe-path recipe.json | ||
|
|
||
| FROM chef AS builder | ||
| SHELL ["/bin/bash", "-c"] | ||
| ARG PROFILE=release | ||
| COPY --from=planner /build/recipe.json recipe.json | ||
| RUN cargo chef cook --profile $PROFILE --recipe-path recipe.json | ||
| COPY . . | ||
| RUN cargo build -p spo-indexer --locked --features cloud --profile $PROFILE && \ | ||
| mkdir -p /runtime/usr/local/bin && \ | ||
| mv "./target/${PROFILE/dev/debug}/spo-indexer" /runtime/usr/local/bin && \ | ||
| mv /build/spo-indexer/bin/entrypoint.sh /runtime/usr/local/bin && \ | ||
| mkdir -p /runtime/opt/spo-indexer && \ | ||
| mv /build/spo-indexer/config.yaml /runtime/opt/spo-indexer | ||
|
|
||
| FROM debian:trixie-slim@sha256:f6e2cfac5cf956ea044b4bd75e6397b4372ad88fe00908045e9a0d21712ae3ba AS runtime | ||
| RUN apt-get update && \ | ||
| apt-get install -y --no-install-recommends ca-certificates=20250419 && \ | ||
| rm -rf /var/lib/apt/lists/* | ||
| RUN useradd -u 10001 -d /nonexistent -s /usr/sbin/nologin -M -c "" appuser && \ | ||
| passwd -l appuser && \ | ||
| mkdir /var/run/spo-indexer && \ | ||
| chown appuser:appuser /var/run/spo-indexer | ||
| COPY --from=builder --chown=appuser:appuser /runtime / | ||
| USER appuser | ||
| WORKDIR /opt/spo-indexer | ||
| ENTRYPOINT ["entrypoint.sh"] |
| ARG RUST_VERSION | ||
| FROM lukemathwalker/cargo-chef:0.1.77-rust-$RUST_VERSION-trixie AS chef | ||
| WORKDIR /build | ||
|
|
||
| FROM chef AS planner | ||
| COPY . . | ||
| RUN cargo chef prepare --recipe-path recipe.json | ||
|
|
||
| FROM chef AS builder | ||
| SHELL ["/bin/bash", "-c"] | ||
| ARG PROFILE=release | ||
| COPY --from=planner /build/recipe.json recipe.json | ||
| RUN cargo chef cook --profile $PROFILE --recipe-path recipe.json | ||
| COPY . . | ||
| RUN cargo build -p wallet-indexer --locked --features cloud --profile $PROFILE && \ | ||
| mkdir -p /runtime/usr/local/bin && \ | ||
| mv "./target/${PROFILE/dev/debug}/wallet-indexer" /runtime/usr/local/bin && \ | ||
| mv /build/wallet-indexer/bin/entrypoint.sh /runtime/usr/local/bin && \ | ||
| mkdir -p /runtime/opt/wallet-indexer && \ | ||
| mv /build/wallet-indexer/config.yaml /runtime/opt/wallet-indexer | ||
|
|
||
| FROM debian:trixie-slim@sha256:f6e2cfac5cf956ea044b4bd75e6397b4372ad88fe00908045e9a0d21712ae3ba AS runtime | ||
| RUN useradd -u 10001 -d /nonexistent -s /usr/sbin/nologin -M -c "" appuser && \ | ||
| passwd -l appuser && \ | ||
| mkdir /var/run/wallet-indexer && \ | ||
| chown appuser:appuser /var/run/wallet-indexer | ||
| COPY --from=builder --chown=appuser:appuser /runtime / | ||
| USER appuser | ||
| WORKDIR /opt/wallet-indexer | ||
| ENTRYPOINT ["entrypoint.sh"] |
Signed-off-by: Skylar Simoncelli <skylar@skylar.email>
Signed-off-by: Skylar Simoncelli <skylar@skylar.email>
There was a problem hiding this comment.
gitleaks found more than 20 potential problems in the proposed changes. Check the Files changed tab for more details.
There was a problem hiding this comment.
Checkov found more than 20 potential problems in the proposed changes. Check the Files changed tab for more details.
Signed-off-by: Skylar Simoncelli <skylar@skylar.email>
Signed-off-by: Skylar Simoncelli <skylar@skylar.email>
Signed-off-by: Skylar Simoncelli <skylar@skylar.email>
Signed-off-by: Skylar Simoncelli <skylar@skylar.email>
Signed-off-by: Skylar Simoncelli <skylar@skylar.email>
Signed-off-by: Skylar Simoncelli <skylar@skylar.email>
Signed-off-by: Skylar Simoncelli <skylar@skylar.email>
2f204ed to
f71324a
Compare
Signed-off-by: Skylar Simoncelli <skylar@skylar.email>
The ported schema-v4.graphql lost its tab-indented blank lines and trailing newline, so the byte-exact schema check in the indexer test recipe failed. Restore the file to the generator's exact output. Assisted-by: Claude <noreply@anthropic.com> Signed-off-by: Skylar Simoncelli <skylar@skylar.email>
The in-memory pub-sub had no channel for BridgeEventIndexed, so the standalone and native flavors panicked the chain-indexer task on the first indexed c2m-bridge event. The block was already saved, so the worker resumed past it on restart, but every bridge event cost a worker restart and subscription clients silently missed the event. Add the missing broadcast channel with publisher/subscriber arms and a drain task, plus a publish/subscribe regression test. Assisted-by: Claude <noreply@anthropic.com> Signed-off-by: Skylar Simoncelli <skylar@skylar.email>
Phase timing on devnet attributed roughly half of the 125 ms per-block cycle to awaiting the next block from the node stream. The existing `.map(ready).buffered(blocks_buffer)` adapter cannot fetch ahead: stream combinators only run while the consumer polls them, so fetching and indexing were strictly serial. Run the block stream on its own task feeding a bounded channel sized by `blocks_buffer`, so block fetching overlaps block processing while preserving ordering and the existing gap/duplicate handling. Assisted-by: Claude <noreply@anthropic.com> Signed-off-by: Skylar Simoncelli <skylar@skylar.email>
Fetching one block takes about 11 sequential RPC awaits. During by-height catch-up these are independent across heights except for the authority cache, which is only needed to resolve block authors and is invalidated by NewSession events. Split make_block into make_raw_block, which fetches and assembles everything except the author and can run for several heights at once, and finish_block, which resolves the author against the sequential authority cache in block order. The catch-up path now drives make_raw_block through a bounded buffered stream (new node config fetch_concurrency, default 8) while author resolution and parent-hash verification stay sequential, so results are byte-identical to the serial implementation. Assisted-by: Claude:claude-fable-5 Signed-off-by: Skylar Simoncelli <skylar@skylar.email>
Phase timing on devnet showed determine_system_parameters_change spending 48 ms per block: a redundant at-block client construction plus two runtime-API calls on the sequential indexing path, every block. Fetch the D-parameter and terms and conditions in make_raw_block instead, where the at-block client already exists and the lookups ride the concurrent block prefetch, and carry the values on the node block. Change detection compares the same values fetched at the same block state, so detected changes and their recorded blocks are identical. The Node trait loses fetch_system_parameters, which had no other callers. Assisted-by: Claude:claude-fable-5 Signed-off-by: Skylar Simoncelli <skylar@skylar.email>
The system-parameters change removed the node parameter from index_block, but the genesis branch still fetches the genesis ledger state from the node. Restore the parameter. Assisted-by: Claude:claude-fable-5 Signed-off-by: Skylar Simoncelli <skylar@skylar.email>
The per-block gc pass consistently consumes its full time budget while culling almost nothing during catch-up (2.6 nodes per 12 ms pass measured on devnet), costing about 11% of replay wall time. Add gc_block_interval (default 1, i.e. unchanged behavior) so deployments can amortize the pass over several blocks with the same per-pass budget. Assisted-by: Claude:claude-fable-5 Signed-off-by: Skylar Simoncelli <skylar@skylar.email>
Summary
--indexernode mode that starts and supervises the native indexer workermidnight-node, with no submodule and no separately released or deployed indexer componentmidnight-nodeimage and preserve the existing GraphQL API, SQL schema, configuration, metrics, and indexer behavior--indexerimply archive retention for both historical state and block bodiesfetch_concurrency), fetch system parameters with the block instead of per-block on the sequential path, and make gc cadence configurable (gc_block_interval)Architecture
The externally visible primitive is
midnight-node --indexer: one node image, one deployment, and one node-managed lifecycle. Internally, the node supervises a siblingmidnight-indexerworker built from the source now owned by this repository.The worker is compiled separately with
indexer/Cargo.tomland its checked-in lockfile. This is intentional: the original Ledger 8 indexer depends on older Base Crypto, Serialize, Storage, and transient-crypto revisions. Linking those crates into the Ledger 9 node process allows root Cargo patches to rewrite the historical verifier graph. The process boundary preserves the exact dependency closure used by the current standalone indexer without changing node consensus dependencies.This is source and release consolidation, not an external runtime dependency. The standalone
midnight-indexerrepository and deployment can be retired once parity, replay, and performance validation are complete.The indexer performs historical runtime-state queries in addition to reading block bodies.
--indexertherefore forces both state and block archive modes. An existing database created with pruned state must be rebuilt because pruned historical trie state cannot be recovered in place.Performance
Measured on devnet (683k blocks) against a fully synced colocated archive node, 4/8 CPU 10/14 GiB pod, one variable changed per deployment:
fetch_concurrency8gc_block_interval16fetch_concurrency32A fresh from-genesis replay of the full devnet chain (685,016 blocks) on the final build completed in 1 h 42 m with zero worker exits, indexing the chain's c2m-bridge event and re-detecting the D-parameter change at exactly the same block as the dedicated indexer. Remaining per-block cost at 111 blocks/s is dominated by the ledger SQLite FULL-fsync persist (5.7 ms). Steady-state GraphQL block queries measure p50 1.50 ms / p99 2.09 ms against the embedded service versus p50 3.45 ms / p99 4.70 ms against the dedicated internal service from the same in-cluster client. ledger-db.sqlite reaches ~32 GB for devnet; disk sizing must budget for it. The SPO indexer's historical epoch catch-up is separately ~148x faster after rewriting a Cardano db-sync query whose generic plan scanned 1.6M rows to return one.
Validation
Devnet validation completed so far:
systemParameters_getAriadneParametersresponses match validator-01 byte-for-byte at multiple epochsindexer_*_duration_seconds) expose fetch/convert/ledger/persist/system-parameters/storage/publish timingsRemaining validation