Skip to content

Documentation gap: api re-genesis runbook (Railway Postgres wipe + indexer cursor reset) #16

@proofmancer

Description

@proofmancer

Context

When ligate-chain went through the v0.2.0 re-genesis on 2026-05-17, we discovered that the existing chain re-genesis runbook (ligate-chain/docs/development/runbooks/chain-id-bump.md) doesn't cover the api / indexer Postgres reset.

The chain wipe → restart → bootstrap-cli sequence worked, but the api kept serving stale state because:

  1. The api's attestations table TRUNCATE migration only covers ONE table; the other 6 (slots, transactions, attestor_sets, schemas, address_summaries, indexer_state) keep their pre-wipe rows.
  2. The api's indexer cursor (indexer_state.last_indexed_height) cached the pre-wipe height. After the chain re-genesised to height 0, the api still thought it was at height 30018 and stopped ingesting.
  3. Required intervention: manual TRUNCATE on all 7 tables + Railway redeploy to clear the in-memory cursor.

What's needed in docs.ligate.io

A new operator runbook section (probably under nodes.mdx or a new operators/re-genesis.mdx) covering:

  1. When re-genesis happens (wire-format break, chain_hash change, state corruption recovery).
  2. What the chain side does (link to ligate-chain runbook).
  3. What the operator must ALSO do on the api side:
    • psql $DATABASE_URL -c "TRUNCATE TABLE slots, transactions, attestor_sets, schemas, attestations, address_summaries, indexer_state CASCADE;"
    • railway redeploy --service ligate-api to clear the in-memory cursor
    • Verify /v1/info shows indexer_height: 0 then growing
  4. Optional: if you want a TRULY clean restart (not re-process historical Celestia blobs), advance genesis_da_height in devnet-1/celestia.toml past the prior chain's blob range, OR rotate to a new DA namespace.

Related

  • chain re-genesis 2026-05-17 (we're tracking this in the ligate-chain repo's chain-id-bump.md runbook as a sibling section)
  • Audit-fix PR for docs: ligate-io/docs#13

Priority

Important but not launch-blocking. File now so a future operator (or v0.3.0 re-genesis) doesn't relive this debugging session.

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type
    No fields configured for issues without a type.

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions