Hecate 🔮

The magical gateway between Ogmios and Exura for Cardano blockchain synchronization

Overview

Hecate is an independent data relay service that connects through Ogmios and efficiently fetches both historical and real-time on-chain data.
Named after the Greek goddess of magic, crossroads, and keeper of keys, Hecate serves as a bridge between the chain and downstream processing systems via standardized interfaces, focusing exclusively on reliable data acquisition and transmission through well-defined API boundaries, enabling integration with any system that needs to track on-chain data. What it relays lands in a sink, and the sink decides what downstream sees: Hecate ships one that prints to the command line (CLI) for debugging or testing, and Redis sinks that write streams or a list; a message queue or a database would be a sink of its own.

NOTE: This project is in early development and is not yet ready for production use. Please use at your own risk.

Architecture Overview

┌───────────────────────────────────────────────────────────────────────────────┐
│                                   HECATE                                      │
│                                                                               │
│  ┌─────────────┐     ┌────────────────────────────┐     ┌──────────────────┐  │
│  │             │     │                            │     │                  │  │
│  │   Async     │     │  Core Processing           │     │  Data Sinks      │  │
│  │   Client    │◄───►│                            │◄───►│                  │  │
│  │             │     │  ┌──────────┐ ┌─────────┐  │     │ ┌────────┐       │  │
│  └─────────────┘     │  │ Backfill │ │ Follow  │  │     │ │Redis   │       │  │
│         ▲            │  │(historic)│ │ (tip)   │  │     │ │Sinks   │       │  │
│         │            │  └──────────┘ └─────────┘  │     │ └────────┘       │  │
│         │            │                            │     │ ┌────────┐       │  │
│         │            │             ▲              │     │ │CLI     │       │  │
│         │            │             │              │     │ │Sink    │       │  │
│         │            │        ┌────┴────┐         │     │ └────────┘       │  │
│         │            │        │   CLI   │         │     │                  │  │
│         │            └────────┴─────────┴─────────┘     └──────────────────┘  │
│         │                                                      │              │
└─────────┼──────────────────────────────────────────────────────┼──────────────┘
          │                                                      │
          ▼                                                      ▼
┌────────────────┐                                    ┌──────────────────────┐
│                │                                    │                      │
│    Cardano     │                                    │    Downstream        │
│    Node +      │                                    │    Applications      │
│    Ogmios      │                                    │    (e.g. Exura)      │
└────────────────┘                                    └──────────────────────┘

Hecate consists of:

1. Ogmios Client - Asynchronous client for the Ogmios WebSocket API
2. Data Relay - Efficiently forward blockchain data with minimal transformation
3. Backfill Core - A plain asyncio coroutine that fetches epochs concurrently, driven by a thin CLI
4. Live Follow - A plain asyncio coroutine that stays at the chain tip, holding the newest blocks back so shallow rollbacks never reach its sink, driven by the same CLI
5. Sinks - Where relayed blocks land: the command line, per-epoch Redis streams or a Redis list for the backfill, and the Redis live stream for the live follow

Features

• ⚡ Parallel Historical Fetching - Efficiently fetch the entire blockchain history in batches
• 🔄 Real-time Data Relay - Follow the chain tip block by block, rollbacks included
• 🧩 Pluggable Sinks - The backfill relays into anything that implements send_batch; CLI and Redis sinks ship with Hecate
• 🛡️ Reorg Detection - Catch chain reorganizations early and handle them gracefully
• 🪶 No Orchestrator - Just asyncio and a CLI; progress lives in the sink, not an engine's database
• 🔁 Resumable - Rerun after a failure and it continues from the last completed epoch
• 🧰 Flexible Deployment - Run as a standalone service with simple configuration
• 🔌 Optional Dependencies - Use only what you need - Redis is optional and can be installed separately

Demo

Hecate includes a demo script showcasing of some of the async client capabilities:

# Run the demo (assumes Ogmios on localhost)
uv run python -m demo

Usage 🔮

# Backfill from the first Shelley epoch to the chain's last finalized epoch
uv run python -m cli backfill

# A bounded range, six epochs at a time
uv run python -m cli backfill --start-epoch 208 --end-epoch 320 --concurrency 6

# No Redis needed — print what would be relayed
uv run python -m cli backfill --start-epoch 208 --end-epoch 208 --sink cli

# How far along is the Redis sink?
uv run python -m cli status

# Follow the chain tip into the Redis live stream, starting an empty one at an epoch
uv run python -m cli follow --from-epoch 650 --kupo-url http://localhost:1442

Epochs are fetched concurrently — each in its own process, since block parsing is GIL-bound — then committed in ascending order so a consumer reading epoch-by-epoch never sees a gap. Progress lives in the sink, so a rerun resumes after the last epoch that completed, and a run whose window would leave a gap in what consumers can read is refused rather than published.

backfill() is also usable directly as a library: give it something that implements send_batch and it will relay blocks into it, no Redis and no CLI involved. See the detailed backfill documentation.

follow stays at the tip instead: it relays every block as the node adopts it, holding the newest few back so that shallow rollbacks never reach its sink and deeper ones reach it as rollbacks. Its sink is the Redis live stream, one stream any number of consumers read; the lease that lets a second follower wait as a warm standby, and the tail a restarted follower resumes from, are that sink's. See the live follow documentation, and the Redis live stream's contract for its consumers.

Epoch boundaries and block counts are derived directly from the chain over the same Ogmios connection used for streaming (see epoch_derivation.py), with an optional kupo endpoint to accelerate boundary lookups. The committed data/*.csv are only a frozen bootstrap anchor index; epochs past it are derived on demand and cached locally.

Installation

Prerequisites

• Python 3.12+
• uv (Python package manager)
• Redis (Optional)
• Ogmios node access

Setup

# Clone the repository
git clone https://github.com/ExuraLabs/hecate.git
cd hecate

# Install dependencies
uv venv -p 3.12

# Install one of the following:
# 1) Base installation (CLI sink only, no Redis support)
uv sync

# 2) With the Redis sink
uv sync --group redis

# 3) Complete installation (Redis + development tools)
uv sync --all-groups

Configuration

Settings are read from the environment at import time, optionally seeded from a dotenv file — .env at the repo root by default, or point HECATE_ENV_FILE at whichever one this environment should use. See the backfill docs for the full table; the two that matter most are OGMIOS_ENDPOINTS and REDIS_URL:

OGMIOS_ENDPOINTS='["ws://your-ogmios-host:1337"]' REDIS_URL='redis://localhost:6379/0' \
  uv run python -m cli backfill --end-epoch 210

Project Structure

hecate/
├── cli               # Command line entry point (python -m cli)
├── backfill          # Historical backfill core — plain asyncio, no orchestrator
├── client/           # Ogmios WebSocket client
├── config/           # Environment-backed settings and logging setup
├── data/             # Frozen bootstrap epoch-boundary data
├── docs/             # Longer-form documentation
├── epoch_derivation  # Derive epoch boundaries from the chain
├── sinks/            # Data sinks
│   ├── base          # Sink protocols: BlockRelay, EpochCoordinator
│   ├── redis         # Redis sink for downstream service(s)
│   └── cli           # CLI sink for command line output
├── constants         # Constant values and configurations
├── models            # Data models and type definitions
└── tests/            # Test suite (see Development, below)

Development

Tests

# The suite that needs no chain access
uv run pytest

# All of it, including the tests that relay real blocks
HECATE_TEST_OGMIOS=ws://your-server:1337 uv run pytest

The tests exercise the real Redis sink, so they need a Redis they are allowed to empty. Supply one however suits the machine:

# bring your own — a container, a remote instance, anything
docker run --rm -d -p 6390:6379 redis:8-alpine
HECATE_TEST_REDIS_URL=redis://localhost:6390/0 uv run pytest

Without HECATE_TEST_REDIS_URL the suite starts and owns a private redis-server on a free port, if that binary happens to be installed. If a given database is not empty the run stops before touching it, and one the suite borrowed is handed back empty.

Everything degrades to skips rather than errors: no Redis to be had, or an install without the redis dependency group, and the suite reports why and passes. The backfill tests that must see blocks move ask for an Ogmios endpoint the same way; the rest — window refusals, purge safety, status — run against faked sink state and finish in seconds. The follow tests need no Ogmios at all: they drive it against an in-process fake (tests/fake_ogmios.py) that serves a chain each test controls, forks included.

Type Checking and Linting

# Run mypy for type checking
uv run mypy

# Run ruff for linting
uv run ruff check

# Run ruff for formatting
uv run ruff format

# Set up pre-commit hooks
uv run pre-commit install

License

This project builds on the ogmios-python client, which is distributed under GPL-3.0-or-later. As a result, this project is also shares the same license terms - see the LICENSE file for details.

Acknowledgements

• Ogmios - WebSocket bridge that wraps Ouroboros' mini-protocols
• ogmios-python - Original Python SDK for Ogmios
• kupo - Fast, lightweight chain-index used to accelerate epoch-boundary lookups