hyperi-rustlib

There's plenty of sage advice out there about how to run Rust services in production at scale — config cascades, structured logging, masking secrets, multi-backend secrets management, Prometheus, OpenTelemetry, Kafka transports, tiered disk-spillover sinks, adaptive worker pools, graceful shutdown — but almost none of it as code you can just install and use.

This is that code.

Opinionated, drop-in, working out of the box. The patterns from blog posts, watercooler chats and beers with your Google mates as actual library — not a framework you assemble from twenty crates and 8 weeks of munging.

Built as the foundation for HyperI's PB/hr data services. Generic enough that you don't need to be at HyperI to use it.

This module exists because of this — https://www.youtube.com/watch?v=xE9W9Ghe4Jk — but for the backend. And of course, no microservices.

Quick Start

[dependencies]
hyperi-rustlib = "2"

Default features: config, logger. Add the others you want explicitly.

use hyperi_rustlib::{config, logger, env};

fn main() -> anyhow::Result<()> {
    let environment = env::Environment::detect();
    logger::setup_default()?;
    config::setup(config::ConfigOptions {
        env_prefix: "MYAPP".into(),
        ..Default::default()
    })?;

    tracing::info!("Running in {environment:?}");
    Ok(())
}

Features

Pick the slice you need; pay only for what you use.

Feature	Description
env	Environment detection (K8s, Docker, Container, BareMetal)
runtime	Runtime path resolution (XDG/container-aware)
config	8-layer config cascade (figment-based)
config-reload	SharedConfig<T> + ConfigReloader hot-reload
config-postgres	PostgreSQL config source
logger	Structured logging, JSON/text auto-detect, sensitive-field masking
metrics	Prometheus metrics + process/container metrics
otel-metrics	OpenTelemetry metrics export (OTLP)
otel-tracing	OpenTelemetry distributed tracing
http	HTTP client with retry middleware (reqwest)
http-server	Axum HTTP server with health probe trinity (/healthz/{startup,live,ready})
transport-kafka	Kafka transport (rdkafka, dynamic-linking)
transport-grpc	gRPC transport (tonic/prost)
transport-memory	In-memory transport (testing/dev)
transport-grpc-vector-compat	Vector wire-protocol compatibility
spool	Disk-backed async FIFO queue (yaque + zstd)
tiered-sink	Resilient delivery: hot buffer + circuit breaker + disk spillover
secrets	Secrets management core (file backend)
secrets-vault	OpenBao / HashiCorp Vault provider
secrets-aws	AWS Secrets Manager provider
directory-config	YAML directory-backed config store
directory-config-git	Git integration for directory-config (git2)
scaling	Back-pressure / scaling-pressure primitives
cli	Standard CLI framework (clap)
top	TUI metrics dashboard (ratatui)
io	File rotation, NDJSON writer
dlq	Dead-letter queue (file backend)
dlq-kafka	DLQ Kafka backend
output-file	File output sink
expression	CEL expression evaluation
deployment	Deployment-contract validation
version-check	Optional startup version check
resilience	Circuit breaker, retry, bulkhead (tower-resilience)
full	Everything

Native System Dependencies

This crate dynamically links against system C libraries for several features. Both build hosts and deployment targets need the appropriate packages.

Build Host (CI / Development)

Feature	Crate	Build Package	Notes
transport-kafka	rdkafka-sys	librdkafka-dev (>= 2.12.1)	Requires Confluent APT repo — Ubuntu's default is too old
directory-config-git	libgit2-sys	libgit2-dev, libssh2-1-dev	System lib avoids vendored C build
spool, tiered-sink	zstd-sys	libzstd-dev	System lib avoids vendored C build
(transitive)	libz-sys	zlib1g-dev	Used by multiple deps
(transitive)	openssl-sys	libssl-dev	Dynamic linking via pkg-config
secrets-aws	aws-lc-sys	—	C/C++ compiled from source (no system lib available); ~20–30s first build, cached by sccache

For librdkafka-dev >= 2.12.1, add the Confluent APT repo:

curl -fsSL https://packages.confluent.io/clients/deb/archive.key \
  | sudo gpg --dearmor -o /usr/share/keyrings/confluent-clients.gpg
echo "deb [signed-by=/usr/share/keyrings/confluent-clients.gpg] \
  https://packages.confluent.io/clients/deb noble main" \
  | sudo tee /etc/apt/sources.list.d/confluent-clients.list
sudo apt-get update
sudo apt-get install -y librdkafka-dev libssl-dev libsasl2-dev pkg-config

Deployment Host (Runtime)

The compiled binary links against .so files at runtime. Install the runtime packages (not -dev) on deployment hosts or in Docker images.

Feature	Runtime Package	Shared Object
transport-kafka	librdkafka1 (from Confluent repo)	librdkafka.so.1
directory-config-git	libgit2-1.7 (or matching version)	libgit2.so
spool, tiered-sink	libzstd1	libzstd.so.1
(transitive)	zlib1g	libz.so.1
(transitive)	libssl3	libssl.so.3

Only install what you use. Check the features your binary enables to determine which runtime packages are needed.

Docker Example

# Build stage
FROM rust:1 AS builder
RUN apt-get update && apt-get install -y \
    pkg-config libssl-dev librdkafka-dev libgit2-dev libzstd-dev
COPY . .
RUN cargo build --release

# Runtime stage
FROM ubuntu:24.04
RUN apt-get update && apt-get install -y --no-install-recommends \
    librdkafka1 libssl3 libgit2-1.7 libzstd1 ca-certificates \
    && rm -rf /var/lib/apt/lists/*
COPY --from=builder /app/target/release/myapp /usr/local/bin/

For librdkafka1, add the Confluent APT repo to both build and runtime stages.

Health Check Endpoints — The Probe Trinity

For services deployed to Kubernetes, the http-server feature provides the three K8s probe types:

Probe	Path	Checks	On failure
Startup	/healthz/startup	Init complete	K8s waits, then restarts
Liveness	/healthz/live	Process not deadlocked	Restart pod
Readiness	/healthz/ready	Deps healthy + ready flag set	Stop routing traffic

Liveness MUST NEVER check downstream dependencies (a DB outage shouldn't restart your replicas). Readiness checks dependencies AND requires an explicit set_ready() call — cleared during graceful shutdown.

Self-regulation (default vertical scaling)

A rustlib data-plane app regulates its own intake. Sized for steady state, a pod slows down or speeds up WITHIN itself first -- the default, fast, local response to a burst, a stalled upstream, or a transform that balloons memory. Only when that vertical headroom is exhausted does it escalate to horizontal scale (KEDA adding pods), driven by the same pressure signal. Memory is the hard, never-OOM authority; CPU is left to the kernel scheduler (CFS), which the byte-budget loop reads through longer process times. It is ON by default and opt-out via self_regulation.enabled = false. See docs/SELF-REGULATION.md.

Architecture

See docs/ for the full documentation index — docs/ARCHITECTURE.md for the module map and layering, and docs/core-pillars/CONFIG.md for the 8-layer config cascade reference.

License

BUSL-1.1 — Business Source License 1.1, transitions to Apache 2.0 after 3 years.

Related

• hyperi-pylib — sister library for Python services. Same opinions, same patterns, expressive Python ergonomics for control planes, APIs, and integration layers.