Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
82 changes: 2 additions & 80 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -81,87 +81,9 @@ Everything above works natively on Windows. Differences:

Other standard `OTEL_*` variables (`OTEL_TRACES_SAMPLER`, `OTEL_METRIC_EXPORT_INTERVAL`, …) are honored. The default sampler is `parentbased_always_on`: requests follow the caller's sampling decision, scheduler jobs are always traced.

## Deploying to Kubernetes
## Deployment

WynnSourceServer uses [Kustomize](https://kustomize.io/) overlays for deployment and [ArgoCD](https://argo-cd.readthedocs.io/) with [Image Updater](https://argocd-image-updater.readthedocs.io/) for GitOps.

### Directory Structure

```
deploy/
base/ # Shared manifests
deployment.yaml # App deployment with health probes
service.yaml # ClusterIP service on port 8000
postgres.yaml # CNPG PostgreSQL cluster
redis.yaml # Redis instance (via Redis Operator)
migration-job.yaml # Alembic migration (ArgoCD sync hook)
kustomization.yaml
overlays/
dev/ # Dev environment (namespace: wynnsource-dev)
kustomization.yaml
prod/ # Prod environment (namespace: wynnsource, replicas: 2)
kustomization.yaml
```

### Cluster Prerequisites

The following operators must be installed in the cluster:

- [CloudNativePG](https://cloudnative-pg.io/) — PostgreSQL operator
- [Redis Operator](https://github.com/OT-CONTAINER-KIT/redis-operator) — Redis operator
- [ArgoCD Image Updater](https://argocd-image-updater.readthedocs.io/) — Automatic image updates

### Step 1: Create Secrets

Create the application secrets in the target namespace. Using [Sealed Secrets](https://sealed-secrets.netlify.app/) is recommended.

```bash
kubectl -n <namespace> create secret generic wynnsource-secrets \
--from-literal=WCS_ADMIN_TOKEN='<your-admin-token>'
```

### Step 2: Create ConfigMap

```bash
kubectl -n <namespace> create configmap wynnsource-config \
--from-literal=LEVEL='INFO' \
--from-literal=BETA_ALLOWED_VERSIONS='0.2.6, 0.2.7'
```

### Step 3: Create ArgoCD Application

```yaml
apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
name: wynnsource
namespace: argocd
annotations:
argocd-image-updater.argoproj.io/image-list: app=ghcr.io/wynnsource/wynnsource-server
argocd-image-updater.argoproj.io/app.update-strategy: newest-build
argocd-image-updater.argoproj.io/app.allow-tags: "regexp:^dev-"
argocd-image-updater.argoproj.io/app.ignore-tags: "latest,dev-latest"
spec:
project: default
source:
repoURL: https://github.com/WynnSource/WynnSourceServer.git
targetRevision: dev # or master for production
path: deploy/overlays/dev # or deploy/overlays/prod
destination:
server: https://kubernetes.default.svc
namespace: wynnsource-dev # or wynnsource
syncPolicy:
automated:
prune: true
selfHeal: true
syncOptions:
- CreateNamespace=true
- ServerSideApply=true
```

### Step 4: Configure Ingress

Add an IngressRoute (or Ingress) pointing to the `wynnsource-server` service on port 8000. Make sure the `X-Real-IP` header is forwarded from the ingress controller for proper client IP logging and rate limiting.
See [`deploy/`](deploy/README.md): a single-instance Docker Compose setup using the published image, and notes for running several replicas (for example on Kubernetes).

## Using the Schema (Protobuf)

Expand Down
2 changes: 1 addition & 1 deletion app/core/log.py
Original file line number Diff line number Diff line change
Expand Up @@ -14,7 +14,7 @@


def _add_trace_context(record: "loguru.Record") -> None:
"""Attach the active span, so Grafana can jump from a trace to its log lines."""
"""Attach the active span, so log viewers can jump from a trace to its log lines."""
ctx = trace.get_current_span().get_span_context()
if ctx.is_valid:
trace_id, span_id = trace.format_trace_id(ctx.trace_id), trace.format_span_id(ctx.span_id)
Expand Down
4 changes: 2 additions & 2 deletions app/core/telemetry.py
Original file line number Diff line number Diff line change
Expand Up @@ -76,7 +76,7 @@ def setup_telemetry(app: FastAPI) -> bool:
if not TELEMETRY_CONFIG.enabled:
return False

# Stable HTTP semantic conventions (http.route, url.path, ...), matching Traefik's spans.
# Stable HTTP semantic conventions (http.route, url.path, ...), matching current proxies' spans.
os.environ.setdefault("OTEL_SEMCONV_STABILITY_OPT_IN", "http")

resource = _resource()
Expand All @@ -88,7 +88,7 @@ def setup_telemetry(app: FastAPI) -> bool:
_meter_provider = MeterProvider(
resource=resource,
metric_readers=[PeriodicExportingMetricReader(OTLPMetricExporter())],
# HTTP RED metrics come from Tempo span metrics and Traefik.
# HTTP RED metrics are expected from span metrics in the tracing backend or from the proxy.
views=[View(instrument_name="http.server.*", aggregation=DropAggregation())],
)
metrics.set_meter_provider(_meter_provider)
Expand Down
34 changes: 34 additions & 0 deletions deploy/.env.example
Original file line number Diff line number Diff line change
@@ -0,0 +1,34 @@
# Copy to .env next to docker-compose.yaml and fill in the required values.
# .env is git-ignored. Every variable is documented in the root README.

# --- Required -------------------------------------------------------------
POSTGRES_PASSWORD=change-me
WCS_ADMIN_TOKEN=change-me

# --- Image and listener ---------------------------------------------------
# Release tag (e.g. 0.1.8) or latest. Pin a version in production.
WCS_VERSION=latest
# Host address and port the server is published on. Keep 127.0.0.1 when a
# reverse proxy on the same host terminates TLS.
WCS_BIND=127.0.0.1
WCS_PORT=8000

# --- Database --------------------------------------------------------------
POSTGRES_DB=wcs_db
POSTGRES_USER=wynnsource

# --- Server ---------------------------------------------------------------
LEVEL=INFO
LOG_JSON=false
MAX_IP_RECORDS=10
# Comma-separated mod versions allowed to submit while in beta.
BETA_ALLOWED_VERSIONS=

# --- Optional: error reporting (Sentry) -----------------------------------
# WCS_SENTRY_DSN=
# WCS_SENTRY_ENVIRONMENT=production
# WCS_SENTRY_PROFILE_SESSION_SAMPLE_RATE=0

# --- Optional: OpenTelemetry traces and metrics (OTLP/HTTP) -----------------
# OTEL_EXPORTER_OTLP_ENDPOINT=http://otel-collector:4318
# OTEL_RESOURCE_ATTRIBUTES=deployment.environment.name=production
142 changes: 142 additions & 0 deletions deploy/README.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,142 @@
# Deploying WynnSourceServer

Two starting points:

- **[Single instance with Docker Compose](#single-instance-with-docker-compose)**: one host, everything in this folder.
- **[Larger deployments](#larger-deployments)**: what to keep in mind when running several replicas, for example on Kubernetes.

All configuration is environment variables, documented in the [root README](../README.md#environment-variables).

## Single instance with Docker Compose

`docker-compose.yaml` runs four services from the published image `ghcr.io/wynnsource/wynnsource-server`:

| Service | Role |
|---|---|
| `server` | The API, on port 8000 inside the container |
| `migration` | Runs `alembic upgrade head` once, then exits; `server` starts only after it succeeds |
| `db` | PostgreSQL 18, data in the `db-data` volume |
| `redis` | Redis (cache, rate limits, scheduler job locks), data in the `redis-data` volume |

### Start

```bash
cd deploy
cp .env.example .env
# Set at least POSTGRES_PASSWORD and WCS_ADMIN_TOKEN, and pin WCS_VERSION to a release.
docker compose up -d
docker compose ps # server should become healthy
curl http://127.0.0.1:8000/readyz
```

To run your own checkout instead of the published image, uncomment `build: ..` in `docker-compose.yaml` and use `docker compose up -d --build`.

### Put a reverse proxy in front

The server is published on `127.0.0.1:8000` only. Expose it through a reverse proxy on the same host that:

- terminates TLS;
- sets **`X-Real-IP`** to the client address. The server uses it for rate limiting and per-user IP records, and **rejects token-authenticated requests that arrive without it** (except from `127.0.0.1`);
- forwards `Host` and `X-Forwarded-Proto`.

Example for nginx:

```nginx
location / {
proxy_pass http://127.0.0.1:8000;
proxy_set_header Host $host;
proxy_set_header X-Real-IP $remote_addr;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
proxy_set_header X-Forwarded-Proto $scheme;
}
```

### Upgrade

```bash
# set WCS_VERSION in .env to the new release
docker compose pull
docker compose up -d # migration runs first, then the server is recreated
```

Releases are tagged `X.Y.Z`; `latest` follows the newest release. `dev-<sha>` and `dev-latest` are pre-release builds from the `dev` branch.

### Back up

The database is the only state worth keeping. Redis holds cache and locks that rebuild themselves.

```bash
docker compose exec -T db sh -c 'pg_dump -U "$POSTGRES_USER" -Fc "$POSTGRES_DB"' > wcs_db-$(date +%F).dump
# restore into an empty database:
docker compose exec -T db sh -c 'pg_restore -U "$POSTGRES_USER" -d "$POSTGRES_DB" --clean --if-exists' < wcs_db-YYYY-MM-DD.dump
```

Keep copies off the host, and try a restore once before you need it.

### Stop

`docker compose down` keeps the volumes; `docker compose down -v` deletes the database.

## Larger deployments

The server is stateless, so it scales horizontally. State lives in PostgreSQL; Redis is shared by all replicas.

```mermaid
flowchart LR
client([Clients]) --> proxy[TLS reverse proxy / ingress<br/>sets X-Real-IP]
proxy --> s1[server replica]
proxy --> s2[server replica]
s1 & s2 --> pg[(PostgreSQL)]
s1 & s2 --> redis[(Redis)]
mig[migration job<br/>alembic upgrade head] --> pg
s1 & s2 -. OTLP/HTTP .-> otel[OpenTelemetry collector]
```

### Rollout order

1. PostgreSQL and Redis are available.
2. Run the migration **once** per release (`uv run --no-dev alembic upgrade head` in the release image), as a one-shot job, not from every replica.
3. Roll the server replicas to the same image.

### Requirements for several replicas

- **Distinct identity per replica.** Scheduled jobs take a lock in Redis held under `K8S_POD_NAME` (defaults to the hostname). Telemetry needs a distinct `service.instance.id` per replica, e.g. `OTEL_RESOURCE_ATTRIBUTES=service.instance.id=<pod or host name>`, so replicas never share a metric series.
- **Probes.** Liveness on `GET /healthz`, readiness on `GET /readyz` (port 8000). Give startup up to a minute before liveness checks begin.
- **Client IP.** The proxy or ingress in front must set `X-Real-IP` (see above). Uvicorn trusts forwarded headers only from `127.0.0.1`, `::1` and `10.0.0.0/8`.
- **Shutdown.** The container runs uvicorn as PID 1 and handles `SIGTERM` (finishes requests, flushes telemetry); give it a termination grace period of a few seconds.
- **Resources.** About 100m CPU / 256 MiB per replica at rest is a reasonable request; limit memory at around 1 GiB.

### Configuration and secrets

- Keep `WCS_ADMIN_TOKEN`, database credentials and `WCS_SENTRY_DSN` in a secret store and inject them as environment variables. Never commit them.
- Non-secret settings (`LEVEL`, `LOG_JSON`, `MAX_IP_RECORDS`, `BETA_ALLOWED_VERSIONS`) can live in plain configuration. Restart or roll the replicas when they change.

### Database

- Use a managed PostgreSQL or a PostgreSQL operator rather than a hand-run container.
- Enable point-in-time recovery (continuous WAL archiving plus base backups), store backups off the cluster, and test restores.
- Size `max_connections` for all replicas plus the migration job.

### Observability

- `LOG_JSON=true` writes one JSON object per line, with `trace_id`/`span_id`, for log pipelines.
- `OTEL_EXPORTER_OTLP_ENDPOINT` enables OpenTelemetry traces and metrics over OTLP/HTTP.
- `WCS_SENTRY_DSN` enables error reporting; `WCS_SENTRY_PROFILE_SESSION_SAMPLE_RATE` adds continuous profiling.

### Image updates

Pin a release tag and update it deliberately, or let an image-update tool follow the `X.Y.Z` tags (production) or `dev-<sha>` tags (staging). Avoid deploying `latest`, since a restart could then change the version.

### Kubernetes objects

A typical Kubernetes setup uses these objects. Write them in whatever tooling you use (plain manifests, Kustomize, Helm):

| Object | Purpose |
|---|---|
| `Deployment` | Server replicas with the probes and env above |
| `Service` | ClusterIP on port 8000 |
| `ConfigMap` / `Secret` | Settings and credentials, consumed with `envFrom` |
| `Job` | Migration, run before each rollout (e.g. as a sync or pre-upgrade hook) |
| PostgreSQL | Managed database or an operator-managed cluster |
| Redis | A small instance; persistence is optional |
| `Ingress` (or equivalent) | TLS and the `X-Real-IP` header |
6 changes: 0 additions & 6 deletions deploy/chart/Chart.yaml

This file was deleted.

10 changes: 0 additions & 10 deletions deploy/chart/templates/configmap.yaml

This file was deleted.

Loading
Loading