Allium Metrics - Deployment Template

Automated Tor relay metrics generation and deployment to Cloudflare Pages with multi-storage support.

Live example: https://metrics.1aeo.com
Allium source: https://github.com/1aeo/allium

Storage Options

Provider	Best For
Cloudflare R2	Native integration, zero egress
DigitalOcean Spaces	Simple flat pricing
Both	Redundancy with failover

Quick Start

# 1. Clone this deployment repo
git clone https://github.com/1aeo/allium-deploy.git ~/allium-deploy
cd ~/allium-deploy

# 2. Configure
cp config.env.example config.env
nano config.env  # Edit values (see below)

# 3. Install
./scripts/allium-deploy-install.sh

# 4. Install cron (uses /etc/cron.d/ drop-in for isolation)
sudo cp ~/allium-deploy/allium.cron /etc/cron.d/allium && sudo chmod 644 /etc/cron.d/allium

# Done! Metrics update every 30 minutes

Configuration (config.env)

Storage Selection

# Storage fetch order: comma-separated list of backends to try
# Options: r2, do, failover
# Examples:
#   "r2,do,failover"  - Try R2 first, then DO Spaces, then failover
#   "do,r2,failover"  - Try DO Spaces first, then R2, then failover
#   "r2,failover"     - R2 only with failover
#   "do,failover"     - DO Spaces only with failover
STORAGE_ORDER=r2,do,failover

Cloudflare R2 (if using R2)

# Cloudflare credentials
CLOUDFLARE_ACCOUNT_ID=your_account_id
R2_ENABLED=true
R2_ACCESS_KEY_ID=your_access_key
R2_SECRET_ACCESS_KEY=your_secret_key
R2_BUCKET=my-metrics-content

Get R2 credentials:

Account ID: Found in dashboard URL or R2 overview
R2 API Token: R2 → Manage R2 API Tokens → Create token with "Object Read & Write"

DigitalOcean Spaces (if using DO)

DO_ENABLED=true
DO_SPACES_KEY=your_spaces_key
DO_SPACES_SECRET=your_spaces_secret
DO_SPACES_REGION=nyc3
DO_SPACES_BUCKET=my-metrics-content
DO_SPACES_URL=https://my-metrics-content.nyc3.digitaloceanspaces.com

Get DO Spaces credentials: See DigitalOcean Spaces Setup below.

Pages Configuration

PAGES_PROJECT_NAME=my-metrics
SITE_URL=https://metrics.example.com

Workers Static Assets shadow (Stages 1–2)

The migration plan is tracked in docs/WORKERS_STATIC_ASSETS_MIGRATION_PLAN.md. During Stages 1–2, Workers Static Assets receives and verifies an immutable candidate after every build, while metrics.1aeo.com continues using the existing Pages/DO/R2 path.

CF_ASSETS_ENABLED=true
CF_ASSETS_REQUIRED=false
CF_ASSETS_ALLOW_PROMOTION=false
CF_ASSETS_WORKER_NAME=1aeo-metrics-assets-stage2
CF_ASSETS_PREVIEW_ALIAS=allium-stage2
CF_ASSETS_VERIFY_ATTEMPTS=3
CF_ASSETS_VERIFY_MAX_ATTEMPTS=12
CF_ASSETS_VERIFY_RETRY_DELAY=10
CF_ASSETS_VERIFY_CURL_TIMEOUT=60

Safety properties in shadow mode:

wrangler.assets.toml is generated separately from the Pages configuration.
The generated Worker configuration contains no zone route or custom domain.
Normal scheduled runs use wrangler versions upload; they do not promote a version or change production traffic.
_headers and 404.html are installed before any publisher reads the output tree. _headers is excluded from DO and R2 synchronization.
Candidate root, search index, a nested page, the largest file, headers, redirects, static 404, and dynamic search are checked three times.
A shadow failure is recorded but remains non-blocking while CF_ASSETS_REQUIRED=false.

Workers Static Assets production promotion (Stage 4)

After the production route and two-hour gate pass, enable automatic promotion only with the complete Stage 4 controls:

CF_ASSETS_ENABLED=true
CF_ASSETS_REQUIRED=true
CF_ASSETS_ALLOW_PROMOTION=true
PAGES_PURGE_URL=https://1aeo-metrics.pages.dev

Each scheduled job uploads an immutable version, requires the complete preview verifier to pass, records that exact version ID, and then runs:

wrangler versions deploy VERSION_ID@100% --yes --config wrangler.assets.toml

The version file is accepted only when it contains one valid Worker version UUID. Missing or malformed state, upload failure, preview-verification failure, or Wrangler deployment failure leaves the previous healthy production version active and fails the required Workers publication. Attempted promotions append one bounded timestamp/version/status row to logs/cfassets-promotion-summary.tsv; normal request logging remains disabled.

While the Worker route fronts SITE_URL, the legacy Pages purge endpoint is not reachable through that hostname. PAGES_PURGE_URL therefore targets the canonical Pages deployment directly, while every purge body uses absolute SITE_URL cache keys. This keeps Pages usable as a fresh route-removal rollback without purging Workers Static Assets or making purge part of Worker version activation.

Run pnpm test before activation. It includes the Node tests and every shell behavior suite, including enabled promotion, failure propagation, and the guarantee that an unverified candidate never reaches the promoter. Full activation and rollback details are in docs/WORKERS_STATIC_ASSETS_STAGE4_READINESS.md.

A brand-new shadow Worker needs one route-free bootstrap deployment before its version preview URLs can serve the assets binding. Generate and inspect the configuration first, enable Workers Preview URLs for that Worker, and bootstrap only the dedicated *-stage2/*-shadow service on workers.dev. Do not attach a zone route, custom domain, or production DNS record. The normal scheduled path never repeats this bootstrap. Cloudflare otherwise returns error 1042 for asset-binding preview requests when the Worker has versions but no deployment record.

Deployment evidence is bounded in:

logs/cfassets-shadow-summary.tsv
logs/cfassets-shadow-consecutive-successes
logs/cfassets-last-version

These are deployment summaries, not request logs. Stage 3 production routing remains disabled until explicit review.

DigitalOcean Spaces Setup

1. Create a Space

Log in to DigitalOcean
Go to Spaces → Create a Space
Configure:
- Region: Choose closest to your users (nyc3, sfo3, ams3, sgp1, fra1)
- Name: your-metrics-content (must be globally unique)
- File Listing: Enable "Restrict File Listing" = OFF (public read)
Click Create a Space

2. Enable CDN (Optional)

DO Spaces CDN is optional. It's faster but has a limitation:

Mode	Setting	Speed	Freshness
Origin	`DO_SPACES_CDN=false`	Slower	Always fresh
CDN	`DO_SPACES_CDN=true`	Faster	Up to ~1hr stale

Important: DO Spaces CDN does NOT support cache invalidation/purging. If using CDN, content may be up to 1 hour stale.

To enable CDN:

Go to your Space → Settings → CDN → Enable CDN
Set in config.env:
```
DO_SPACES_CDN=true
```

3. Create Spaces Access Keys

Go to API → Spaces Keys
Click Generate New Key
Name it (e.g., "allium-deploy")
Copy both the Key and Secret (secret shown only once!)

Add to config.env:

DO_SPACES_KEY=XXXXXXXXXXXXXXXXXXXX
DO_SPACES_SECRET=XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX

4. Make Bucket Public

For the Pages function to fetch content, the bucket must be publicly readable:

Go to your Space → Settings
Under File Listing, ensure it's set to allow public access
Alternatively, use a CDN endpoint which handles this automatically

5. Configure rclone (Automatic)

The upload script auto-configures rclone on first run. Manual setup:

rclone config create spaces s3 \
    provider=DigitalOcean \
    access_key_id="YOUR_KEY" \
    secret_access_key="YOUR_SECRET" \
    endpoint="nyc3.digitaloceanspaces.com" \
    acl=public-read

6. View Metrics

DigitalOcean does not provide built-in Spaces metrics via the dashboard or API. To monitor usage:

Bandwidth: Check your monthly bill/usage in Billing → Usage
Storage size: Run rclone size spaces:your-bucket
Object count: Run rclone ls spaces:your-bucket | wc -l
Third-party tools: Consider MetricFire or custom scripts using the S3-compatible API

Search Functionality

Server-side search for Tor relay metrics without client-side JavaScript.

How It Works

Search Form (on any page) → /search?q=... GET request
Pages Function (functions/search.js) → Loads search-index.json
302 Redirect → Direct to matching relay/family/AS/country page

Supported Query Types

Query Type	Example	Result
Fingerprint (full)	`ABCD1234...` (40 hex)	Direct to relay page
Fingerprint (partial)	`ABCD12` (6-39 hex)	Direct or disambiguation
Nickname	`MyTorRelay`	Direct to relay page
AS Number	`AS24940` or `24940`	Direct to AS page
Country	`Germany` or `de`	Direct to country page
IP Address	`1.2.3.4`	Direct to relay page
Contact/AROI	`example.org`	Direct to contact page
Platform	`linux`, `freebsd`	Direct to platform page
Flag	`exit`, `guard`	Direct to flag page

CIISS Version Indicators (v1.6+ index)

Search disambiguation pages display CIISS v2 / v3 indicators next to relay names when the operator's ContactInfo declares a ciissversion. v3 indicators are green; v2 indicators are grey; relays without an AROI declaration show no indicator. This requires search-index.json schema v1.6+ (Allium build with PR #207 or later, which adds the per-relay vn field). Older indexes (≤ v1.5) continue to work unchanged with no indicators shown.

Security Features

Input Validation: Query length limit (100 chars), character allowlist
XSS Prevention: All output HTML-escaped
Open Redirect Prevention: Path allowlist validation
Security Headers: CSP, X-Frame-Options, X-Content-Type-Options
ReDoS-Safe Patterns: All regex patterns are O(n) complexity
Error Handling: Generic messages to users, detailed server logs

Requirements

The search function requires search-index.json to be generated by allium and deployed with the site. This is handled automatically when running allium with search index generation enabled.

Architecture

The diagram below is the production request path retained through Stage 2. Workers Static Assets publishes in parallel to it, but only its workers.dev candidate and non-production rehearsal hostname are reachable.

┌─────────────────────────┐
│  Allium Generator       │
│  (Python + Jinja2)      │
│  ~4 min / ~21k files    │
│  + search-index.json    │
└───────────┬─────────────┘
            │
            ▼
┌─────────────────────────┐
│  Local Output           │
│  ~/metrics-output       │
│  ~3GB                   │
└───────────┬─────────────┘
            │ rclone sync (128 parallel)
            ▼
┌─────────────────────────┐  ┌─────────────────────────┐
│  Cloudflare R2          │  │  DigitalOcean Spaces    │
│  (native binding)       │  │  (HTTP fetch via CDN)   │
└───────────┬─────────────┘  └───────────┬─────────────┘
            │                            │
            └──────────┬─────────────────┘
                       │ STORAGE_ORDER determines try order
                       ▼
         ┌─────────────────────────┐
         │  Cloudflare Pages       │
         │  ├─ [[path]].js (content)
         │  └─ search.js (search)  │
         └─────────────────────────┘

Commands

# === Upload Commands ===

# Upload to R2
~/allium-deploy/scripts/allium-deploy-upload-r2.sh

# Upload to DO Spaces
~/allium-deploy/scripts/allium-deploy-upload-do.sh

# List backups (R2)
~/allium-deploy/scripts/allium-deploy-upload-r2.sh --list-backups

# List backups (DO Spaces)
~/allium-deploy/scripts/allium-deploy-upload-do.sh --list-backups

# Force backup even if done today
~/allium-deploy/scripts/allium-deploy-upload-r2.sh --force-backup

# === Other Commands ===

# Manual update (runs allium + upload)
~/allium-deploy/scripts/allium-deploy-update.sh

# View logs
tail -f ~/allium-deploy/logs/update.log

# Deploy Pages function
~/allium-deploy/scripts/allium-deploy-cfpages.sh

# Generate and inspect the route-free Workers Assets configuration only
CF_ASSETS_DIRECTORY="$HOME/metrics-output" \
  ~/allium-deploy/scripts/allium-deploy-cfassets.sh --generate-only

# Upload and verify one immutable shadow version (never promotes)
~/allium-deploy/scripts/allium-deploy-cfassets.sh --upload-only

# Inspect the bounded Stage 1/2 soak result
tail -n 5 ~/allium-deploy/logs/cfassets-shadow-summary.tsv
cat ~/allium-deploy/logs/cfassets-shadow-consecutive-successes

# Check cron status
~/allium-deploy/scripts/allium-cron-check.sh

Directory Structure

allium/                 # Allium source (git clone)
allium-deploy/          # This deployment repo
├── config.env          # Your settings (gitignored)
├── config.env.example  # Template
├── wrangler.toml.template  # Wrangler config template
├── wrangler.toml       # Generated at deploy time (gitignored)
├── wrangler.assets.toml.template # Route-free Workers Assets template
├── wrangler.assets.toml # Generated shadow config (gitignored)
├── cloudflare-assets/  # Workers _headers and static 404 overlay
├── functions/
│   ├── [[path]].js     # Pages function (multi-storage + failover)
│   └── search.js       # Search function (query → redirect)
├── workers/
│   └── search.js       # Workers /search entry point + ALLIUM_ASSETS binding
├── scripts/
│   ├── allium-deploy-install.sh      # One-time setup
│   ├── allium-deploy-update.sh       # Runs every 30 min via cron
│   ├── allium-cron-check.sh          # Verify cron is installed
│   ├── allium-deploy-upload-r2.sh    # R2 upload with backups
│   ├── allium-deploy-upload-do.sh    # DO Spaces upload with backups
│   ├── allium-deploy-upload-common.sh # Shared upload functions
│   ├── allium-deploy-prune.sh        # Remove old backups
│   ├── allium-deploy-cfpages.sh      # Deploy Pages function
│   ├── allium-deploy-cfassets.sh     # Upload route-free Worker versions
│   └── allium-deploy-verify-cfassets.sh # Candidate health gate
├── logs/
│   ├── update.log
│   ├── last-local-backup-date
│   ├── last-r2-backup-date
│   └── last-do-backup-date
└── public/             # Empty dir for Pages build output

Performance

Operation	Time	Notes
Allium generation	~4 min	9,800 relays → 21,000 pages
R2 upload (full)	~3 min	128 parallel transfers
DO Spaces upload (full)	~3 min	128 parallel transfers
Upload (incremental)	<1 min	Only changed files
Local backup	~1.5 min	Download from storage
Remote backup	~3.5 min	Server-side copy

Backup & Rollback

Automatic backups (once daily):

Local: ~/metrics-backups/backup-YYYY-MM-DD_HHMMSS/
R2: r2:bucket/_backups/YYYY-MM-DD_HHMMSS/
DO Spaces: spaces:bucket/_backups/YYYY-MM-DD_HHMMSS/

Rollback from local backup:

./scripts/allium-deploy-upload-r2.sh ~/metrics-backups/backup-2025-12-01_063000
# or
./scripts/allium-deploy-upload-do.sh ~/metrics-backups/backup-2025-12-01_063000

Rollback from remote backup:

# From R2
~/bin/rclone sync r2:bucket/_backups/2025-12-01_063000 ~/metrics-output
./scripts/allium-deploy-upload-r2.sh

# From DO Spaces
~/bin/rclone sync spaces:bucket/_backups/2025-12-01_063000 ~/metrics-output
./scripts/allium-deploy-upload-do.sh

Multi-Storage Modes

R2 Only (`STORAGE_ORDER=r2,failover`)

Native R2 binding (fastest)
Best for: Maximum performance

DO Spaces Only (`STORAGE_ORDER=do,failover`)

HTTP fetch from Spaces CDN
Best for: Simplicity

Both (`STORAGE_ORDER=do,r2,failover` or `r2,do,failover`)

Tries storage backends in order
Best for: Redundancy

Response headers show which source served the request:

X-Served-From: cloudflare-r2
X-Served-From: digitalocean-spaces
X-Served-From: failover-origin

Origin Failover

Independent of storage selection, you can configure a failover origin:

FAILOVER_ENABLED=true
FAILOVER_ORIGIN_URL=https://backup.example.com/metrics

The Pages function tries sources in order defined by STORAGE_ORDER:

First storage backend (R2 or DO Spaces)
Second storage backend (if configured)
Failover origin (if enabled)
Return 404

CDN Caching

Cloudflare CDN (via Pages)

HTML: 30 min cache, purged after each upload
Static assets: 24 hour cache
Purge: Automatic via /_purge endpoint after uploads

DigitalOcean Spaces

Mode	`DO_SPACES_CDN`	Behavior
Origin	`false`	Always fresh (default)
CDN	`true`	Faster, up to ~1hr stale (no purge available)

Troubleshooting

502 errors during upload:

Normal with 128 parallel connections
Retries handle it automatically

DO Spaces "Access Denied":

Check bucket is public or CDN is enabled
Verify DO_SPACES_URL matches your bucket

R2 connection failed:

Verify credentials in config.env
Test: ~/bin/rclone ls r2:bucket --max-depth 1

Pages function not using correct storage:

Check STORAGE_ORDER in config.env
Redeploy: ./scripts/allium-deploy-cfpages.sh
Check X-Served-From response header

DO Spaces serving stale content:

If using CDN (DO_SPACES_CDN=true), switch to origin (DO_SPACES_CDN=false)
DO Spaces CDN has no cache purge - origin mode always serves fresh content

Search not working:

Verify search-index.json exists in your output directory
Enable search index generation in allium (see allium docs)
Check browser console for 404 errors on /search-index.json
Search requires allium to generate the index file during build

Requirements

Debian/Ubuntu Linux
Python 3.8+
python3-jinja2
Cloudflare account (for Pages, optionally R2)
DigitalOcean account (optional, for Spaces)
~4GB RAM (for allium processing)
~10GB disk space

Credits

Allium: https://github.com/1aeo/allium
Tor Project: https://www.torproject.org
Onionoo API: https://onionoo.torproject.org

License

Apache 2.0

Name		Name	Last commit message	Last commit date
.github/workflows		.github/workflows
cloudflare-assets		cloudflare-assets
docs		docs
functions		functions
public		public
scripts		scripts
tests		tests
workers		workers
.gitignore		.gitignore
LICENSE		LICENSE
README.md		README.md
config.env.example		config.env.example
package.json		package.json
pnpm-lock.yaml		pnpm-lock.yaml
pnpm-workspace.yaml		pnpm-workspace.yaml
wrangler.assets.toml.template		wrangler.assets.toml.template
wrangler.toml.template		wrangler.toml.template

Latest commit

History

Folders and files

Repository files navigation

Allium Metrics - Deployment Template

Storage Options

Quick Start

Configuration (config.env)

Storage Selection

Cloudflare R2 (if using R2)

DigitalOcean Spaces (if using DO)

Pages Configuration

Workers Static Assets shadow (Stages 1–2)

Workers Static Assets production promotion (Stage 4)

DigitalOcean Spaces Setup

1. Create a Space

2. Enable CDN (Optional)

3. Create Spaces Access Keys

4. Make Bucket Public

5. Configure rclone (Automatic)

6. View Metrics

Search Functionality

How It Works

Supported Query Types

CIISS Version Indicators (v1.6+ index)

Security Features

Requirements

Architecture

Commands

Directory Structure

Performance

Backup & Rollback

Multi-Storage Modes

R2 Only (STORAGE_ORDER=r2,failover)

DO Spaces Only (STORAGE_ORDER=do,failover)

Both (STORAGE_ORDER=do,r2,failover or r2,do,failover)

Origin Failover

CDN Caching

Cloudflare CDN (via Pages)

DigitalOcean Spaces

Troubleshooting

Requirements

Credits

License

About

Resources

Stars

Watchers

Forks

Releases

Packages

Contributors

Languages

R2 Only (`STORAGE_ORDER=r2,failover`)

DO Spaces Only (`STORAGE_ORDER=do,failover`)

Both (`STORAGE_ORDER=do,r2,failover` or `r2,do,failover`)