Skip to content

feat: add Litestream replication config and disaster recovery runbook #137

Description

@dennisonbertram

Problem

The platform stores all state in SQLite databases (ah.db, ah-metering.db). There is no continuous backup or replication strategy. If the server disk fails, all tenant data is lost. WAL-safe backup exists but is point-in-time only.

Proposed Solution

Scaffold Litestream continuous replication to S3-compatible storage (without enabling it in systemd yet — phase 1 is config + docs).

Deliverables

  1. deploy/litestream.yml — Litestream config file for both SQLite databases:

    dbs:
      - path: /var/lib/ah/ah.db
        replicas:
          - type: s3
            bucket: ${LITESTREAM_BUCKET}
            path: ah.db
            endpoint: ${LITESTREAM_ENDPOINT}
            region: ${LITESTREAM_REGION}
      - path: /var/lib/ah/ah-metering.db
        replicas:
          - type: s3
            bucket: ${LITESTREAM_BUCKET}
            path: ah-metering.db
            endpoint: ${LITESTREAM_ENDPOINT}
            region: ${LITESTREAM_REGION}
  2. docs/ops/litestream-replication.md — Runbook covering:

    • Prerequisites (S3 bucket, Litestream binary)
    • How to enable replication
    • How to verify replication is working
    • How to restore from a replica (full disaster recovery procedure)
    • How to monitor replication lag
    • Integration with existing backup strategy
  3. deploy/litestream.env.example — Example environment variables

Acceptance Criteria

  • Litestream config file covers both databases
  • Config uses environment variables for all credentials
  • Runbook covers setup, verification, and restore procedures
  • Example env file provided
  • No changes to systemd unit or production config (phase 1 only)

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions