From 525df5aff3c207acff05cc827e40d293526f697f Mon Sep 17 00:00:00 2001 From: Claude Date: Wed, 14 Oct 2026 15:20:51 +0000 Subject: [PATCH 1/8] =?UTF-8?q?docs:=20projects=20model=20=E2=80=94=20a=20?= =?UTF-8?q?project=20is=20one=20load=20balancer=20with=20targets,=20servic?= =?UTF-8?q?es,=20domains?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Rewrite the spec and README around projects (Overview, Graphs, Targets, Services, Domains). Each project gets its own port; attached domains are routed on the shared :80/:443 with automatic HTTPS. ADR 0006 supersedes ADR 0005. Record the newly verified HAProxy behaviour (domain routing on sys_http, maintenance inside a backend). Co-Authored-By: Claude Opus 5.5 Claude-Session: https://claude.ai/code/session_0159fEExV6P6AdgP7HbsJNBc --- .env.example | 2 +- CLAUDE.md | 2 +- INSTRUCTIONS.md | 832 +++++++++++++---------- README.md | 67 +- docs/adr/0005-product-direction.md | 4 +- docs/adr/0006-projects.md | 110 +++ docs/haproxy-notes.md | 7 +- docs/operations.md | 11 +- docs/screenshots/backend-graphs-dark.png | Bin 75093 -> 0 bytes docs/screenshots/backend-servers.png | Bin 81413 -> 0 bytes docs/screenshots/backends.png | Bin 53882 -> 0 bytes docs/screenshots/dashboard.png | Bin 107602 -> 0 bytes docs/screenshots/domains.png | Bin 53948 -> 0 bytes docs/screenshots/frontend-new.png | Bin 73188 -> 0 bytes docs/screenshots/frontend.png | Bin 74949 -> 0 bytes docs/screenshots/frontends-dark.png | Bin 47682 -> 0 bytes docs/screenshots/frontends.png | Bin 50305 -> 0 bytes docs/screenshots/system.png | Bin 112050 -> 0 bytes internal/api/backends.go | 523 -------------- internal/api/binds_test.go | 31 - internal/api/frontends.go | 620 ----------------- 21 files changed, 631 insertions(+), 1578 deletions(-) create mode 100644 docs/adr/0006-projects.md delete mode 100644 docs/screenshots/backend-graphs-dark.png delete mode 100644 docs/screenshots/backend-servers.png delete mode 100644 docs/screenshots/backends.png delete mode 100644 docs/screenshots/dashboard.png delete mode 100644 docs/screenshots/domains.png delete mode 100644 docs/screenshots/frontend-new.png delete mode 100644 docs/screenshots/frontend.png delete mode 100644 docs/screenshots/frontends-dark.png delete mode 100644 docs/screenshots/frontends.png delete mode 100644 docs/screenshots/system.png delete mode 100644 internal/api/backends.go delete mode 100644 internal/api/binds_test.go delete mode 100644 internal/api/frontends.go diff --git a/.env.example b/.env.example index ac1baa1..5ffcd2c 100644 --- a/.env.example +++ b/.env.example @@ -29,7 +29,7 @@ LOXY_HAPROXY_RUN_DIR=/run/haproxy # haproxy binary used for `haproxy -c` validation. LOXY_HAPROXY_BIN=haproxy -# IPv6 listeners for "*" binds: auto (probe the host), on or off. +# IPv6 listeners for service ports and domains (:80/:443): auto (probe the host), on or off. LOXY_IPV6=auto # Stats poll interval and log level (debug|info|warn|error). diff --git a/CLAUDE.md b/CLAUDE.md index 2be933a..af6640d 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -1,6 +1,6 @@ # Loxy — notes for coding agents -Loxy is a lightweight, self-hosted management panel for HAProxy 3.4 LTS (frontends, backends, servers, certificates, DNS). +Loxy is a lightweight, self-hosted load-balancer panel for HAProxy 3.4 LTS (projects with targets, services and domains). **The full build spec is `INSTRUCTIONS.md`. Read all of it before writing code, and treat it as the source of truth.** diff --git a/INSTRUCTIONS.md b/INSTRUCTIONS.md index a44d173..9dde331 100644 --- a/INSTRUCTIONS.md +++ b/INSTRUCTIONS.md @@ -1,4 +1,4 @@ -# Loxy — a self‑hosted management panel for HAProxy +# Loxy — a self‑hosted load‑balancer panel for HAProxy **Build instructions for the implementing agent** @@ -7,8 +7,8 @@ | Name | **Loxy** — repo `github.com/2shiraz/loxy`, Go module `github.com/2shiraz/loxy`, binary `loxy`, image `ghcr.io/2shiraz/loxy`, env prefix `LOXY_`, DB file `loxy.db` | | Data plane | **HAProxy 3.4 LTS** (official Docker image `haproxy:3.4.x-alpine`, patch version pinned) | | Control plane | Single Go binary with embedded web UI, SQLite, no other services | -| Purpose | Manage the **frontends, backends, servers, certificates and DNS** of HAProxy safely: every change is validated (`haproxy -c`), versioned as a revision, applied without dropping traffic (runtime API or hitless reload) and observable (health, graphs, logs, activity). A replacement for hand‑editing `haproxy.cfg` and for heavier HAProxy GUIs. | -| Spec date | 2026‑10‑14 (product direction: ADR 0005) | +| Purpose | Run simple **load balancers** on your own servers. A **project** is one load balancer: it sends traffic to **targets**, listens on **services** (ports) and can be reached on **domains** with automatic HTTPS. Every change is validated (`haproxy -c`), versioned as a revision, applied without dropping traffic (runtime API or hitless reload) and observable (health, graphs, logs, activity). | +| Spec date | 2026‑10‑14 (product model: ADR 0006, which supersedes ADR 0005) | --- @@ -20,70 +20,71 @@ 4. Items tagged **VERIFY** are HAProxy behaviours that must be proven by an automated integration test against the pinned HAProxy image *before* code relies on them. If the test disproves the statement, the observed behaviour wins: change the code, keep the test, and note it in `docs/haproxy-notes.md`. 5. Do not add runtime dependencies beyond §4 without an ADR. "Lightweight and fast" is a hard requirement, not a preference. 6. Never let a rendered HAProxy config that has not passed `haproxy -c` become the live config (§7.7). +7. The product model is **projects → targets, services, domains** (ADR 0006). HAProxy frontends and backends are an *internal* rendering detail (§2, §7) and never appear as user objects in the UI or the API. --- ## 1. Product summary ### 1.1 What we are building -A self‑hosted control panel + REST API for one or more Linux servers running HAProxy. The operator works with HAProxy's own concepts, presented safely: - -- **Frontends** — where traffic enters: a mode (`http | tcp`), one or more **binds** (`address:port`, address `*` = all interfaces), optional **TLS termination** with certificates, an optional HTTP→HTTPS redirect listener, a **default backend** and (http) **host/path routing rules** to other backends, timeouts, `maxconn`, forwarded headers, enable/maintenance switches and labels. -- **Backends** — where traffic goes: a mode, a load‑balancing **algorithm**, a default server port, **health checks**, sticky sessions, PROXY protocol, timeouts and labels. A backend can be used by many frontends. -- **Servers** of a backend — a raw **IP**, a **host** from the inventory, or a **label selector** that matches many hosts; weight, backup flag and admin state (ready / drain / maint), added and removed at runtime. -- **Hosts** (inventory of machines), **Certificates** (uploaded or managed via ACME), **Domains & DNS** (records at external DNS providers). -- The panel shows **health** per server and backend, **live and historical graphs** per frontend and backend (open connections, connections/s, requests/s, bandwidth, sessions, queue, response time, HTTP status classes, errors), live logs and an activity/audit trail. +A self‑hosted control panel + REST API for one or more Linux servers running HAProxy. The owner's guiding requirement: **it must be simple.** The user thinks in load balancers, not in `haproxy.cfg` sections. + +- **Projects** — one project is one load balancer. A user can have many. Each project has an **Overview**, **Graphs**, **Targets**, **Services**, **Domains**, **Logs** and **Settings**, plus a load‑balancing **algorithm** (`roundrobin | leastconn | source | random`), a **maintenance** switch (http → 503 page, tcp → connection rejected) and **labels**. +- **Targets** — where traffic goes: an **IP** (optionally `ip:port`), a **host** from the inventory, or a **label selector** over host labels. Each target has a weight, a backup flag and an admin state (ready / drain / maintenance), is added and removed at runtime, and is health‑checked by every service of the project. +- **Services** — the ports a project listens on: protocol `http | https | tcp`, **listen port → destination port**, a health check (`tcp | http | https`), sticky sessions (http/https), PROXY protocol v1/v2, a timeout and, for https, certificates. A new project automatically gets an **http service on its own free port** (from 20000 up); the listen port can also be chosen explicitly. Listen ports are unique across all projects. Ports **80 and 443 are reserved for domains**. +- **Domains** — attach a hostname to a project (optionally to one of its services; default: the project's first http/https service). Traffic for that hostname on the shared ports 80/443 goes to the project. **HTTPS is automatic:** once the name resolves to the server, Loxy obtains a certificate (Let's Encrypt HTTP‑01 through :80, or DNS‑01 through a DNS provider) and redirects :80 → :443. Loxy shows the required DNS records (the node's addresses) and can manage A/AAAA records automatically through a DNS provider (libdns). +- **Hosts** (inventory of machines), **Certificates** (uploaded or managed via ACME), **DNS providers**. +- The panel shows **health** per target, service and project, **live and historical graphs** per project and per service (open connections, connections/s, requests/s, bandwidth, sessions, queue, response time, HTTP status classes, errors), live logs per project and an activity/audit trail. - Every change is applied to HAProxy automatically, validated, versioned, and **without dropping traffic** (runtime API where possible, hitless reload otherwise). ### 1.2 Goals -1. **Cover the everyday HAProxy job without touching `haproxy.cfg`:** frontends, backends, servers, TLS, routing, health checks, certificates, DNS (§8), plus the extras in §20. +1. **Simple:** "create project → add targets → reach it on its port or attach a domain" covers the everyday job. No HAProxy knowledge needed; no `haproxy.cfg` editing. 2. **Minimal install:** `docker compose up -d` with two containers (official HAProxy + `loxy`). No external database, queue, Prometheus or Redis required. 3. **Fast and light:** budgets in §1.4 are acceptance criteria. 4. **Safe:** invalid config can never reach the live path; every applied state is a numbered revision that can be diffed and rolled back. 5. **API‑first:** the UI uses only the public REST API, so everything is scriptable. ### 1.3 Non‑goals (v1) -Being a cloud load‑balancer *product* (no "create load balancer" objects, sizes/plans, rescaling, prices, billing or traffic quotas); managing NGINX/Apache; Kubernetes ingress; a WAF rule engine; anycast/multi‑region traffic steering; UDP load balancing; provider‑specific cloud integrations (inventory sync, floating‑IP APIs); **running an authoritative DNS server** (Loxy manages records at external DNS providers through their APIs, §8.9; it never answers DNS queries itself); exposing every HAProxy keyword (Loxy is opinionated; raw config import is a backlog item, §20). +Sizes/plans, rescaling, prices, billing or traffic quotas; managing NGINX/Apache; Kubernetes ingress; a WAF rule engine; anycast/multi‑region traffic steering; UDP load balancing; provider‑specific cloud integrations (inventory sync, floating‑IP APIs); **running an authoritative DNS server** (Loxy manages records at external DNS providers through their APIs, §8.12; it never answers DNS queries itself); exposing HAProxy frontends/backends or arbitrary HAProxy keywords (Loxy is opinionated; raw config import is a backlog item, §20); per‑address binds (services listen on all addresses of the node). ### 1.4 Performance budgets (acceptance criteria) | Item | Budget | |---|---| -| Controller resident memory, idle, 50 frontends / 200 backends / 1 000 servers | ≤ 60 MB | +| Controller resident memory, idle, 50 projects / 200 services / 1 000 targets | ≤ 60 MB | | Controller CPU, idle (1 s stats polling) | < 1 % of one core | | API read p95 (served from in‑memory snapshot) | < 10 ms | -| Write that is applied via Runtime API (e.g. add server), request → active in HAProxy | < 200 ms | -| Write that needs a reload (validate + swap + reload), 50 frontends / 200 backends | < 1.5 s | -| UI initial load, all JS + CSS gzip | ≤ 150 KB (charts lazy‑loaded on the Graphs tabs) | +| Write that is applied via Runtime API (e.g. add target), request → active in HAProxy | < 200 ms | +| Write that needs a reload (validate + swap + reload), 50 projects / 200 services | < 1.5 s | +| UI initial load, all JS + CSS gzip | ≤ 150 KB (charts lazy‑loaded on the Graphs tab) | | Live graph latency (counter change → pixel) | ≤ 2 s | | Failed client requests during 100 consecutive reloads under a 2 000 rps load test | **0** | -| Failed client requests while adding/removing 50 servers under load | **0** | +| Failed client requests while adding/removing 50 targets under load | **0** | --- ## 2. Concept mapping (Loxy → HAProxy) +Users see projects, targets, services and domains. The renderer compiles them into the frontend/backend sections that were verified in Phases 0–1 (ADR 0004, ADR 0006). + | Loxy object | Table(s) | HAProxy construct | |---|---|---| -| Frontend (name, mode, binds, TLS, options) | `frontends`, `frontend_binds`, `frontend_certificates` | `frontend fe_ from loxy_http|loxy_tcp` with one `bind` per address:port (IPv6 as `ipv6@addr:port`, `*` → `:port` / `ipv6@:::port`) | -| HTTP→HTTPS redirect (tls only) | `frontends.http_redirect` | Extra `frontend fe__rd` on the same addresses:80 with the ACME HTTP‑01 rule and `http-request redirect scheme https code 301` | -| TLS termination + certificates | `frontends.tls`, `frontend_certificates` | `bind … ssl crt-list fe.lst alpn h2,http/1.1` (`crt-base` + relative entries, ADR 0004); certs hot‑swapped through the Runtime API | -| Default backend | `frontends.default_backend_id` | `default_backend be_` | -| Routing rule (hosts, path prefix → backend; http only) | `routes` (ordered by `position`) | `use_backend be_ if { req.hdr(host),field(1,:),lower -m str|-m end … } { path_beg … }` in route order | -| Maintenance switch | `frontends.maintenance` | `opt@maint.map` keyed by frontend id: http → `http-request return status 503`, tcp → `tcp-request connection reject` | -| Enabled switch | `frontends.enabled` | Frontend not rendered when disabled | -| Backend (mode, algorithm, port, options) | `backends` | `backend be_ from loxy_http|loxy_tcp`, `balance` **always explicit** (§7.9) | -| Health check (protocol, port, interval, timeout, retries, domain, path, response, status codes, TLS verify) | columns on `backends` | `option httpchk`, `http-check send/expect`, `check port inter rise fall`, `timeout check`, `check-ssl` | -| Sticky sessions (cookie name, lifetime) | backend columns | `cookie insert indirect nocache httponly [secure] maxlife s` + `cookie t[x]` per server | -| PROXY protocol | backend column | `send-proxy`/`send-proxy-v2` **and** `check-send-proxy` on server lines | -| Server: IP / host / label selector | `backend_servers`, resolved into `server_slots` | `server t` (IP) or `server tx` (host / selector match) with `id `; added/removed at runtime as dynamic servers | -| Host (inventory) | `hosts` | Address source for host/selector servers only | -| Labels | JSON column on frontends, backends, hosts, certificates | Selectors, list filters and alert‑rule filters only; never rendered | -| Node addresses | `node_addresses` | Not rendered; bind‑address suggestions and DNS required records for `*` binds | -| Metrics | collector + rollup tables (scope frontend / backend) | Deltas of `show stat` counters | -| System | — | `sys_panel` (+ `sys_panel_be`), `sys_stats`, `sys_acme_fe` (ACME HTTP‑01 :80 listener for a TLS frontend without redirect) | - -The metric names of the API (`open_connections`, `connections_per_second`, `requests_per_second`, `bandwidth` with `in`/`out` series, plus backend `sessions`, `queue`, `response_time_ms`, `http_*xx`, `errors`) are stable contract (§10.5). +| Project (name, algorithm, maintenance, labels) | `balancers` (§6 naming) | No section of its own. `algorithm` → `balance` in every `be_` of the project (**always explicit**, §7.9); `maintenance` → one `opt@maint.map` entry per service of the project | +| Service (protocol, listen → destination port, options) | `services` | `frontend fe_ from loxy_http|loxy_tcp` with `bind 0.0.0.0:` (+ `ipv6@::: v6only` when the host has IPv6, ADR 0004 §9) and `default_backend be_`; `backend be_ from loxy_http|loxy_tcp` | +| https service + certificates | `services.protocol = https`, `service_certificates` | `bind … ssl crt-list svc.lst alpn h2,http/1.1` (`crt-base` + relative entries, ADR 0004); targets get plain HTTP (TLS termination); certs hot‑swapped through the Runtime API | +| Health check (protocol, port, interval, timeout, retries, domain, path, response, status codes, TLS verify) | columns on `services` | In `be_`: `option httpchk`, `http-check send/expect`, `check port inter rise fall`, `timeout check`, `check-ssl` | +| Sticky sessions (cookie name, lifetime; http/https) | service columns | `cookie insert indirect nocache httponly [secure] maxlife s` in `be_` (`secure` on https services) + `cookie ` per server | +| PROXY protocol | service column | `send-proxy`/`send-proxy-v2` **and** `check-send-proxy` on the server lines of `be_` | +| Target: IP / host / label selector | `targets`, resolved into `target_slots` | In **every** `be_` of the project: `server s` (IP) or `server sx` (host / selector match) with `id `, port = target port override or the service's destination port; added/removed at runtime as dynamic servers | +| Domain (hostname → project / service, https) | `domains` | Shared `frontend sys_http` (:80) and `frontend sys_https` (:443): `use_backend be_ if { req.hdr(host),field(1,:),lower -m str }`; ACME HTTP‑01 answers and 301 → https in `sys_http`; domain certificates in `sys_https.lst` | +| Maintenance | `balancers.maintenance` | `opt@maint.map` keyed by service id: http/https → `http-request return status 503` in `fe_`, and also in `be_` when a domain routes to that backend (so domain traffic is covered); tcp → `tcp-request connection reject` in `fe_` | +| Host (inventory) | `hosts` | Address source for host/selector targets only | +| Labels | JSON column on projects, hosts, certificates | Selectors, list filters and alert‑rule filters only; never rendered | +| Node addresses | `node_addresses` | Not rendered; endpoint URLs and DNS required records for domains | +| Metrics | collector + rollup tables (scopes `fe`, `be`, `pr`) | Deltas of `show stat` counters of `fe_`/`be_`; project (`pr`) = sum of its service backends | +| System | — | `sys_panel` (+ `sys_panel_be`), `sys_stats`, `sys_http`, `sys_https` | + +The metric names of the API (`open_connections`, `connections_per_second`, `requests_per_second`, `bandwidth` with `in`/`out` series, plus `sessions`, `queue`, `response_time_ms`, `http_*xx`, `errors`) are stable contract (§10.5). --- @@ -105,7 +106,7 @@ The metric names of the API (`open_connections`, `connections_per_second`, `requ ▼ ▼ HAProxy 3.4 official image, master‑worker, network_mode: host │ - Clients ⇄ bind IP:port ⇄ HAProxy (frontends → backends) ⇄ Servers + Clients ⇄ : or :80/:443 (domains) ⇄ HAProxy (project services) ⇄ Targets ``` ### 3.2 Process modes (one binary) @@ -130,7 +131,7 @@ type NodeTransport interface { 4. It **always** renders the full config tree for the new version into a new revision directory and validates it with `haproxy -c` (§7.2, §7.7). 5. If the plan contains any `reload` op → atomically swap `live` → master CLI `reload`. Otherwise → swap `live` (no reload) and execute the runtime ops. If a runtime op fails → fall back to a reload of the same revision. 6. Store a `config_revisions` row (result, duration, HAProxy output), set `applied_version`, emit events. -7. On any failure the previous good revision stays (or is put back) as `live`, the affected frontends/backends and System show an `apply_error` banner with the HAProxy output, and a notification fires. +7. On any failure the previous good revision stays (or is put back) as `live`, the affected projects and System show an `apply_error` banner with the HAProxy output, and a notification fires. **Invariant:** the live config tree on disk always fully describes the state HAProxy is running, so a cold restart of the HAProxy container reproduces exactly what runtime operations created. Runtime API calls are an optimisation to avoid reloads, never the only record of a change. @@ -147,7 +148,7 @@ type NodeTransport interface { | HTTP server | stdlib `net/http` (method+path patterns) with small middleware; handler interfaces generated by `oapi-codegen` from `api/openapi.yaml` (OpenAPI 3.0.3) | Contract‑first, typed, no framework weight | | Database | SQLite in WAL mode via `modernc.org/sqlite` (pure Go, no CGO); typed queries with `sqlc`; migrations embedded and run at start | Zero ops, microsecond reads | | ACME | `github.com/go-acme/lego/v5` used as a library | HTTP‑01 and DNS‑01, custom challenge providers, renewal info (ARI) | -| DNS providers | `github.com/libdns/libdns` interfaces + provider modules (the first ones are listed in §21 Q4; ADR 0003) | One small, uniform record API (`GetRecords`/`AppendRecords`/`SetRecords`/`DeleteRecords`, `ListZones`) for many providers; the same credentials drive ACME DNS‑01 through a lego `challenge.Provider` adapter (§8.9) | +| DNS providers | `github.com/libdns/libdns` interfaces + provider modules (ADR 0003) | One small, uniform record API (`GetRecords`/`AppendRecords`/`SetRecords`/`DeleteRecords`, `ListZones`) for many providers; the same credentials drive ACME DNS‑01 through a lego `challenge.Provider` adapter (§8.12) | | Crypto | `golang.org/x/crypto/argon2` (passwords), AES‑256‑GCM (secrets at rest), TOTP lib for optional 2FA | | | Logging | `log/slog` JSON to stdout | | | Frontend | Svelte 5 + SvelteKit with `adapter-static` (SPA mode, `ssr = false`), TypeScript, Vite | Smallest runtime of the mainstream frameworks | @@ -184,10 +185,10 @@ loxy/ │ ├── config/ # env + flag parsing, defaults, validation │ ├── model/ # domain types, enums, validation rules (§14.4) │ ├── store/ -│ │ ├── migrations/ # 0001_init.sql, ... +│ │ ├── migrations/ # 0001_init.sql, 0002_projects.sql, ... │ │ ├── queries/ # sqlc input │ │ └── db/ # sqlc output (generated, committed) -│ ├── state/ # immutable NodeState snapshots, label-selector resolution, server slots +│ ├── state/ # immutable NodeState snapshots, label-selector resolution, target slots, port allocation │ ├── reconcile/ # diff, op classification, apply plan, retries │ ├── haproxy/ │ │ ├── render/ # text/template renderer + templates/*.tmpl + golden tests @@ -198,13 +199,13 @@ loxy/ │ ├── metrics/ # collector, delta + reset logic, ring buffers, rollups, queries │ ├── health/ # per-check status, aggregation, transitions → events │ ├── certs/ # PEM parsing/validation, storage, lego integration, renew scheduler -│ ├── dns/ # libdns provider registry, lego DNS-01 adapter, domain verifier, record sync (§8.9) +│ ├── dns/ # libdns provider registry, lego DNS-01 adapter, domain verifier, record sync (§8.12) │ ├── api/ # handler implementations, middleware (auth, CSRF, rate-limit, request-id) │ ├── auth/ # sessions, argon2id, API tokens, RBAC, TOTP │ ├── events/ # in-process pub/sub + SSE hub │ ├── audit/ │ ├── notify/ # webhook, email (SMTP), Telegram, Slack -│ ├── logrecv/ # UDP syslog receiver + per-frontend ring buffers (live logs) +│ ├── logrecv/ # UDP syslog receiver + per-project ring buffers (live logs) │ ├── agent/ # node-side applier + NodeTransport (in-process now, network in Phase 3) │ ├── cluster/ # panel-side node registry, enrollment, transport server (Phase 3) │ └── webui/ # go:embed of web/build (precompressed .br/.gz) @@ -213,11 +214,11 @@ loxy/ │ ├── src/lib/components/ # StatBar, StatCard, Donut, Chart, Tabs, Table, Modal, Dropdown, Toast, LabelChips, PortField, ... │ ├── src/lib/stores/ # session, sse, snapshot cache │ ├── src/lib/styles/tokens.css -│ └── src/routes/ # /login, /dashboard, /frontends/[id]/{overview,routing,graphs,logs,config}, /backends/[id]/{servers,settings,graphs}, /hosts, /certificates, /domains/{,providers,zones}, /activity, /system, /settings/* +│ └── src/routes/ # /login, /dashboard, /projects, /projects/new, /projects/[id]/{overview,graphs,targets,services,domains,logs,settings}, /hosts, /certificates, /domains/{,providers,zones}, /activity, /system, /settings/* ├── deploy/ │ ├── docker/Dockerfile │ ├── compose/compose.yml # production all-in-one -│ ├── compose/compose.dev.yml # dev: haproxy + whoami servers + pebble ACME + loxy +│ ├── compose/compose.dev.yml # dev: haproxy + whoami targets + pebble ACME + loxy │ ├── compose/compose.agent.yml # Phase 3: node agent only │ ├── haproxy/bootstrap.cfg # minimal config written on first start │ ├── keepalived/ # Phase 3 templates + notify scripts @@ -228,7 +229,7 @@ loxy/ │ ├── e2e/ # Playwright │ └── load/ # oha / k6 scripts + reload-under-load harness └── docs/ - ├── adr/ # architecture decision records + ├── adr/ # architecture decision records (0006: projects model) ├── haproxy-notes.md # verified behaviours, version quirks └── operations.md # backup, restore, upgrade, break-glass ``` @@ -237,46 +238,50 @@ loxy/ ## 6. Data model (SQLite) -Conventions: `INTEGER PRIMARY KEY` ids (shown in UI as `#42`), timestamps as unix milliseconds, JSON stored as `TEXT CHECK(json_valid(col))`, `ON DELETE` rules explicit, `PRAGMA foreign_keys=ON`, `journal_mode=WAL`, `synchronous=NORMAL`, `busy_timeout=5000`. All resources belong to a `project` (one default project in v1; keeps the door open for multi‑tenant use). +Conventions: `INTEGER PRIMARY KEY` ids (shown in UI as `#42`), timestamps as unix milliseconds, JSON stored as `TEXT CHECK(json_valid(col))`, `ON DELETE` rules explicit, `PRAGMA foreign_keys=ON`, `journal_mode=WAL`, `synchronous=NORMAL`, `busy_timeout=5000`. + +**Naming:** the user‑facing *project* (one load balancer) is stored in the table **`balancers`**, because `projects` is the pre‑existing **tenancy** table (one row, id 1, kept for possible multi‑tenant use). Foreign keys to a load balancer are therefore called `balancer_id`; the API and UI call it `project` / `project_id`. Tenancy‑scoped tables (`hosts`, `certificates`, `dns_providers`, `api_tokens`, `balancers`, `domains`, `events`) keep their `project_id`, which is always 1 in v1 — that is why hosts, certificates and DNS providers are usable by every load‑balancer project. | Table | Key columns | |---|---| -| `projects` | id, name, created_at | | `users` | id, email (unique), password_hash (argon2id), role (`owner|admin|operator|viewer`), totp_secret_enc, disabled, created_at, last_login_at | -| `project_members` | project_id, user_id, role | -| `sessions` | id_hash, user_id, created_at, expires_at, ip, user_agent | -| `api_tokens` | id, project_id, name, prefix, secret_hash, role, expires_at, last_used_at | +| `sessions` | id_hash, user_id, created_at, last_seen_at, expires_at, ip, user_agent | +| `projects` (tenancy) | id, name, created_at — one row (id 1) in v1 | +| `api_tokens` | id, project_id (tenancy), name, prefix, secret_hash, role, expires_at, last_used_at, created_at | | `nodes` | id, name, mode (`local|remote`), status, haproxy_version, loxy_version, features JSON (`QUIC`, `PROMEX`, …), last_seen_at, desired_version, applied_version | -| `node_addresses` | id, node_id, address, family (4/6), label — the node's own addresses: bind‑address suggestions and DNS required records for frontends that bind `*` | +| `node_addresses` | id, node_id, address, family (4/6), label — the node's own addresses: endpoint URLs (`http://:`) and DNS required records for domains | | `node_groups` | id, name, kind (`single|ha_pair`), vip_mode (`none|vrrp`), virtual_ips JSON — Phase 3 | -| `frontends` | id, project_id, name (unique per project), description, node_id / node_group_id, mode (`http|tcp`), tls, http_redirect, default_backend_id (nullable, `ON DELETE RESTRICT`), timeout_client_s, maxconn (0 = unlimited), forwarded_headers, enabled, maintenance, labels JSON, status (`active|applying|apply_error`), last_error, options JSON (extras, §20), created_at, updated_at, version | -| `frontend_binds` | frontend_id, address (IP or `*`), port — UNIQUE(frontend_id, address, port); cross‑frontend collisions are checked in the API (§8.3) | -| `frontend_certificates` | frontend_id, certificate_id, position | -| `routes` | id, frontend_id, position, hosts JSON (exact or `*.example.com`), path_prefix, backend_id (`ON DELETE RESTRICT`) | -| `backends` | id, project_id, name (unique per project), description, mode (`http|tcp`), algorithm (`roundrobin|leastconn|source|random`), port, sticky_enabled, sticky_cookie_name, sticky_cookie_lifetime_s, proxy_protocol (`none|v1|v2`), timeout_server_s, hc_protocol (`tcp|http|https`), hc_port, hc_interval_s, hc_timeout_s, hc_retries, hc_domain, hc_path, hc_response, hc_status_codes JSON, hc_tls_verify, labels JSON, options JSON, created_at, updated_at, version | -| `backend_servers` | id, backend_id, type (`ip|host|label_selector`), ip, host_id, label_selector, port (nullable override), use_private_ip, weight (1–256, default 100), backup, admin_state (`ready|drain|maint`), removing, created_at | -| `server_slots` | id (stable HAProxy server `id`), backend_id, backend_server_id, host_id NULL, address, port — one row per *resolved* server, so ids survive reloads | -| `hosts` (inventory) | id, project_id, name, ipv4, ipv6, private_ipv4, labels JSON, notes | -| `certificates` | id, project_id, name, type (`uploaded|managed`), domains JSON, key_type (`ecdsa256|rsa2048|rsa4096`), challenge (`http01|dns01`), dns_provider_id (→ `dns_providers`), cert_pem, chain_pem, key_pem_enc, fingerprint_sha256, not_before, not_after, status (`pending|issued|renewing|failed`), last_error, renew_after, labels JSON | +| `balancers` (API: project) | id, project_id (tenancy, default 1), node_id, name (UNIQUE(project_id, name)), description, algorithm (`roundrobin|leastconn|source|random`, default roundrobin), labels JSON, maintenance, status (`applying|ready|error`), last_error, version, created_at, updated_at | +| `services` | id, balancer_id (`ON DELETE CASCADE`), protocol (`http|https|tcp`), listen_port (**UNIQUE** — across all projects), destination_port, proxy_protocol (`none|v1|v2`), timeout_s (default 60), sticky_enabled, sticky_cookie_name (default `LOXYSTICKY`), sticky_cookie_lifetime_s (default 300), hc_protocol (`tcp|http|https`), hc_port (0 = target port), hc_interval_s, hc_timeout_s, hc_retries, hc_domain, hc_path, hc_response, hc_status_codes JSON, hc_tls_verify, version, created_at. The project's first http/https service (lowest id) is the domain default | +| `service_certificates` | service_id (`ON DELETE CASCADE`), certificate_id (`ON DELETE RESTRICT`), position — PK(service_id, certificate_id) | +| `targets` | id, balancer_id (`ON DELETE CASCADE`), type (`ip|host|label_selector`), ip, host_id (`ON DELETE CASCADE`; the API refuses deleting a directly referenced host, §8.5), label_selector, port (nullable override), use_private_ip, weight (1–256, default 100), backup, admin_state (`ready|drain|maint`), removing, created_at | +| `target_slots` | id (stable HAProxy server `id`), balancer_id, target_id (`ON DELETE CASCADE`), host_id NULL, address, UNIQUE(target_id, host_id, address) — one row per *resolved* target (an IP target has one slot, a host one, a selector one per matching host), so ids survive reloads. The same slot id is used in every `be_` of the project (server ids are unique per backend) | +| `hosts` (inventory) | id, project_id (tenancy), name, ipv4, ipv6, private_ipv4, labels JSON, notes | +| `certificates` | id, project_id (tenancy), name, type (`uploaded|managed`), domains JSON, key_type (`ecdsa256|rsa2048|rsa4096`), challenge (`http01|dns01`), dns_provider_id (→ `dns_providers`), cert_pem, chain_pem, key_pem_enc, fingerprint_sha256, not_before, not_after, status (`pending|issued|renewing|failed`), last_error, renew_after, labels JSON | | `acme_accounts` | id, directory_url, email, key_pem_enc, registration JSON, eab_kid, eab_hmac_enc | -| `dns_providers` | id, project_id, name, kind (registry key, e.g. `cloudflare`), credentials_enc (AES‑256‑GCM JSON), status (`ok|error|unknown`), last_error, last_checked_at, created_at | -| `domains` | id, project_id, fqdn (unique per project, RFC 1123, lowercase, `*.` allowed only with a DNS provider), frontend_id (nullable, `ON DELETE SET NULL`), dns_provider_id (nullable), zone (detected apex, e.g. `example.com`), auto_records (bool, default false), ttl (default 300), status (`unknown|ok|wrong_ip|missing|error`), observed JSON (last A/AAAA answers), checked_at, certificate_id (nullable), managed_record_ids JSON, created_at, updated_at | +| `dns_providers` | id, project_id (tenancy), name, kind (registry key), credentials_enc (AES‑256‑GCM JSON), status (`ok|error|unknown`), last_error, last_checked_at, created_at | +| `domains` | id, project_id (tenancy), fqdn (UNIQUE(project_id, fqdn) — i.e. unique in the installation; RFC 1123, lowercase, `*.` allowed only with a DNS provider), balancer_id (nullable, `ON DELETE SET NULL` — a detached domain is kept but not rendered), service_id (nullable, `ON DELETE SET NULL`; NULL = the project's first http/https service), https (default 1), dns_provider_id (nullable), zone (detected apex, e.g. `example.com`), auto_records (default false), ttl (default 300), status (`unknown|ok|wrong_ip|missing|error`), status_detail, observed JSON (last A/AAAA answers), checked_at, certificate_id (nullable; managed automatically or an uploaded cert that covers the fqdn), managed_record_ids JSON, created_at, updated_at | | `config_revisions` | id, node_id, revision (monotonic), desired_version, sha256, kind (`runtime|reload|rollback|bootstrap`), result (`ok|failed`), duration_ms, haproxy_output, created_by, created_at, pinned | -| `metrics_1m`, `metrics_5m`, `metrics_1h` | scope (`frontend|backend`), scope_id, ts, open_conn_avg, open_conn_max, cps, rps, bytes_in, bytes_out, sessions_avg, queue_max, resp_time_ms_avg, http_2xx, http_3xx, http_4xx, http_5xx, errors — PK(scope, scope_id, ts) | -| `events` | id, ts, project_id, frontend_id NULL, backend_id NULL, kind, severity, message, data JSON | +| `metrics_1m`, `metrics_5m`, `metrics_1h` | scope (`fe` = `fe_` frontend, id 0 = node total; `be` = `be_` service backend; `pr` = project), scope_id, ts, open_conn_avg, open_conn_max, cps, rps, bytes_in, bytes_out, sessions_avg, queue_max, resp_time_ms_avg, http_2xx, http_3xx, http_4xx, http_5xx, errors — PK(scope, scope_id, ts) | +| `events` | id, ts, project_id (tenancy, default 1), balancer_id NULL, kind, severity, message, data JSON — indexes on `ts` and `(balancer_id, ts)` | | `audit_log` | id, ts, actor_user_id, actor_token_id, ip, action, resource_type, resource_id, before JSON, after JSON | | `notification_channels` | id, kind (`webhook|email|telegram|slack`), config_enc, enabled | -| `alert_rules` | id, event_kinds JSON, label_filter (label selector matched against frontend/backend labels), channel_id, min_severity, cooldown_s | +| `alert_rules` | id, event_kinds JSON, label_filter (label selector matched against **project** labels; node‑wide events such as apply and backup are not filtered by it), channel_id, min_severity, cooldown_s | | `settings` | key, value JSON | Indexes on every foreign key, on `events(ts)`, `audit_log(ts)`, metrics PKs. Keep the whole desired state (minus metrics/events/audit) in an in‑memory snapshot rebuilt on change; API reads hit the snapshot, not SQLite. +**Migrations.** +- `0001_init.sql` — the frontend/backend schema of the previous model (ADR 0005). Kept unchanged so existing databases migrate forward. +- `0002_projects.sql` — moves to this model (ADR 0006). The tenancy `projects` table is left unchanged, and hosts, certificates, DNS providers, users, API tokens, ACME accounts, revisions, audit log and settings keep their rows (and their `project_id`). It creates `balancers`, `services`, `service_certificates`, `targets` and `target_slots`; re‑creates `domains` with `balancer_id`, `service_id` and `https` and copies every domain **detached** (`balancer_id`/`service_id` NULL, `https` 1); re‑creates `events` with `balancer_id` and copies the rows without their frontend/backend ids; drops `routes`, `frontend_certificates`, `frontends`, `server_slots`, `backend_servers` and `backends` (frontend/backend definitions are **not** converted — they don't map 1:1 onto projects); and clears `metrics_1m`. Runs after the automatic pre‑migration backup (§16.6). +- Only a database from the **pre‑release cloud‑load‑balancer model** (before ADR 0005, recognised by its `load_balancers` table) is **refused** at start with a clear error telling the operator to start from an empty data directory — never migrated silently. + --- ## 7. HAProxy integration (the core) ### 7.1 Process model, sockets, container -- HAProxy runs in its **own container** from the **official image**, `network_mode: host` (frontends add listeners on arbitrary addresses and ports, which Docker port publishing cannot follow, and host networking avoids the userland proxy/NAT hop). +- HAProxy runs in its **own container** from the **official image**, `network_mode: host` (services add listeners on arbitrary ports, which Docker port publishing cannot follow, and host networking avoids the userland proxy/NAT hop). - Start command (override the image entrypoint so the flags are explicit): `haproxy -W -db -S /run/haproxy/master.sock,mode,660,gid,1500 -f /usr/local/etc/haproxy/live/haproxy.cfg` - `-W` master‑worker; `-db` foreground; `-S` master CLI socket. **VERIFY** the `-S` bind‑option syntax and resulting socket permissions. @@ -303,8 +308,9 @@ Indexes on every foreign key, on `events(ts)`, `audit_log(ts)`, metrics PKs. Kee ├── haproxy.check.cfg # same content rendered with Root=, used only for `haproxy -c` ├── certs/.pem # leaf + chain + key, mode 0640 ├── certs/_placeholder.pem # self-signed, served until a managed cert is issued - ├── crt-lists/fe.lst # entries relative to crt-base (ADR 0004) - ├── maps/maint.map # loaded with opt@, keyed by frontend id, updated at runtime too + ├── crt-lists/svc.lst # https services; entries relative to crt-base (ADR 0004) + ├── crt-lists/sys_https.lst # issued domain certificates, entries relative to crt-base + ├── maps/maint.map # loaded with opt@, keyed by service id, updated at runtime too ├── maps/panel_allow.lst └── errors/503.http, maint.http /var/lib/haproxy/server-state # dump of `show servers state`, written before each reload @@ -315,18 +321,22 @@ Indexes on every foreign key, on `events(ts)`, `audit_log(ts)`, metrics PKs. Kee - **Atomic swap:** create `live.tmp` → `revs/N`, then `rename(live.tmp, live)`, then fsync the directory. - Unchanged files are **hard‑linked** from the previous revision (cheap, fast). - Garbage collection keeps the last 20 revisions plus any `pinned` (rollback) ones. -- Runtime‑only changes (server add, cert renewal) still create a new revision so disk always equals memory. +- Runtime‑only changes (target add, cert renewal) still create a new revision so disk always equals memory. ### 7.3 Naming and stable ids -- Frontend `fe_{F}`; redirect frontend `fe_{F}_rd`; backend `be_{B}`; system sections `sys_panel`, `sys_panel_be`, `sys_stats`, `sys_acme_fe{F}`. +- Per service `S`: frontend `fe_{S}`, backend `be_{S}` (`S` = `services.id`). System sections: `sys_panel`, `sys_panel_be`, `sys_stats`, `sys_http`, `sys_https`. - Frontend and backend names MUST differ (HAProxy 3.3+ rejects a frontend and backend sharing a name) — guaranteed by the `fe_`/`be_` prefixes. -- Server names: `t{T}` for an IP server, `t{T}x{H}` for a host or label‑selector expansion (`T` = `backend_servers.id`, `H` = `hosts.id`). Server `id` = `server_slots.id` (stable), so `load-server-state-from-file` re‑applies weight/admin/health state after reloads. -- Because a server belongs to exactly one backend, a runtime op is one command per resolved server — it never multiplies across frontends. -- Names in HAProxy use ids only; display names, descriptions and labels never appear in the config. +- Server names: `s{T}` for an IP target, `s{T}x{H}` for a host or label‑selector expansion (`T` = `targets.id`, `H` = `hosts.id`). Server `id` = `target_slots.id` (stable), so `load-server-state-from-file` re‑applies weight/admin/health state after reloads. +- A target is rendered in **every** `be_` of its project, so a target runtime op is one command per resolved slot **per service** of the project. +- Names in HAProxy use ids only; project names, descriptions, labels and domains as display text never appear in the config (domain FQDNs appear only as validated ACL patterns, §14.4). ### 7.4 Rendered config (reference output) -Templates live in `internal/haproxy/render/templates/`. Output MUST be deterministic (sorted by id; routes by position) so revisions diff cleanly. Reference rendering for a TLS http frontend with redirect and host/path routing to two backends, plus a tcp frontend/backend pair (deviations recorded in ADR 0004 are already applied): +Templates live in `internal/haproxy/render/templates/`. Output MUST be deterministic (projects, services and targets sorted by id, domains by FQDN with exact before wildcard) so revisions diff cleanly. Reference rendering (deviations recorded in ADR 0004 are already applied; the host has IPv6): + +- Project **#1 "shop"** (roundrobin): targets #101 IP `10.0.0.11`, #102 label selector `role=web` matching hosts #7 (`10.0.0.12`) and #8 (`10.0.0.13`), #103 IP `10.0.0.14` (backup). Service **#1** http `:20001 → 8080` with an http health check; service **#2** https `:20002 → 8080` with certificate #12, tcp health check and sticky sessions. +- Project **#2 "db"** (leastconn, in maintenance): target #120 IP `10.0.0.41`; service **#3** tcp `:20003 → 5432` with PROXY protocol v2. +- Domains: `shop.example.com` → project #1 (default service #1), https on, certificate #21 issued; `beta.example.com` → project #1, https on, certificate still pending. ```haproxy # Generated by loxy. DO NOT EDIT. node=1 revision=123 desired_version=456 @@ -393,96 +403,132 @@ frontend sys_stats from loxy_http stats uri /stats stats refresh 5s -# ---------- frontend 3 "web" (http, tls, redirect on) ---------- -frontend fe_3 from loxy_http - bind 0.0.0.0:443 ssl crt-list /usr/local/etc/haproxy/live/crt-lists/fe3.lst alpn h2,http/1.1 - bind ipv6@:::443 ssl crt-list /usr/local/etc/haproxy/live/crt-lists/fe3.lst alpn h2,http/1.1 - maxconn 10000 # omitted when maxconn = 0 - timeout client 50s +# ---------- domains (shared :80 / :443) ---------- +frontend sys_http from loxy_http + bind 0.0.0.0:80 + bind ipv6@:::80 v6only + http-request return status 200 content-type text/plain lf-string "%[path,field(-1,/),map(virt@acme_http01)]" if { path_beg /.well-known/acme-challenge/ } { path,field(-1,/),map(virt@acme_http01) -m found } + http-request redirect scheme https code 301 if { req.hdr(host),field(1,:),lower -m str shop.example.com } + option forwardfor + http-request set-header X-Forwarded-Proto http + http-request return status 404 content-type text/plain string "Unknown host" unless { req.hdr(host),field(1,:),lower -m str beta.example.com shop.example.com } + use_backend be_1 if { req.hdr(host),field(1,:),lower -m str beta.example.com shop.example.com } + +frontend sys_https from loxy_http + bind 0.0.0.0:443 ssl crt-list /usr/local/etc/haproxy/live/crt-lists/sys_https.lst alpn h2,http/1.1 + bind ipv6@:::443 v6only ssl crt-list /usr/local/etc/haproxy/live/crt-lists/sys_https.lst alpn h2,http/1.1 option forwardfor http-request set-header X-Forwarded-Proto https - http-request set-header X-Forwarded-Port %[dst_port] - http-request return status 503 content-type text/html file /usr/local/etc/haproxy/live/errors/maint.http if { str(3),map(opt@/usr/local/etc/haproxy/live/maps/maint.map) -m found } - use_backend be_5 if { req.hdr(host),field(1,:),lower -m str app.example.com } { path_beg /api } - use_backend be_6 if { req.hdr(host),field(1,:),lower -m end .static.example.com } - default_backend be_4 + http-request return status 404 content-type text/plain string "Unknown host" unless { req.hdr(host),field(1,:),lower -m str beta.example.com shop.example.com } + use_backend be_1 if { req.hdr(host),field(1,:),lower -m str beta.example.com shop.example.com } -frontend fe_3_rd from loxy_http - bind 0.0.0.0:80 - bind ipv6@:::80 - http-request return status 200 content-type text/plain lf-string "%[path,field(-1,/),map(virt@acme_http01)]" if { path_beg /.well-known/acme-challenge/ } { path,field(-1,/),map(virt@acme_http01) -m found } - http-request redirect scheme https code 301 +# ---------- project 1 "shop" · service 1 (http :20001 -> 8080) ---------- +frontend fe_1 from loxy_http + bind 0.0.0.0:20001 + bind ipv6@:::20001 v6only + timeout client 60s + option forwardfor + http-request set-header X-Forwarded-Proto http + http-request set-header X-Forwarded-Port %[dst_port] + http-request return status 503 content-type text/html file /usr/local/etc/haproxy/live/errors/maint.http if { str(1),map(opt@/usr/local/etc/haproxy/live/maps/maint.map) -m found } + default_backend be_1 -backend be_4 from loxy_http +backend be_1 from loxy_http balance roundrobin - timeout server 50s + timeout server 60s + http-request return status 503 content-type text/html file /usr/local/etc/haproxy/live/errors/maint.http if { str(1),map(opt@/usr/local/etc/haproxy/live/maps/maint.map) -m found } # only because domains route to be_1 option httpchk - http-check send meth GET uri /health ver HTTP/1.1 hdr Host app.example.com + http-check send meth GET uri /health ver HTTP/1.1 hdr Host shop.example.com http-check expect status 200-399 - http-check expect string 'OK' # VERIFY AND-semantics, see §7.5 - timeout check 13s - server t101 10.0.0.11:9000 id 101 check inter 15s rise 3 fall 3 - server t102x7 10.0.0.12:9000 id 102 check inter 15s rise 3 fall 3 - server t102x8 10.0.0.13:9000 id 103 check inter 15s rise 3 fall 3 backup - -backend be_5 from loxy_http - balance leastconn - server t110 10.0.0.21:8080 id 110 check inter 15s rise 3 fall 3 + http-check expect string 'OK' + timeout check 10s + server s101 10.0.0.11:8080 id 101 check inter 15s rise 3 fall 3 + server s102x7 10.0.0.12:8080 id 102 check inter 15s rise 3 fall 3 + server s102x8 10.0.0.13:8080 id 103 check inter 15s rise 3 fall 3 + server s103 10.0.0.14:8080 id 104 check inter 15s rise 3 fall 3 backup + +# ---------- project 1 "shop" · service 2 (https :20002 -> 8080) ---------- +frontend fe_2 from loxy_http + bind 0.0.0.0:20002 ssl crt-list /usr/local/etc/haproxy/live/crt-lists/svc2.lst alpn h2,http/1.1 + bind ipv6@:::20002 v6only ssl crt-list /usr/local/etc/haproxy/live/crt-lists/svc2.lst alpn h2,http/1.1 + timeout client 60s + option forwardfor + http-request set-header X-Forwarded-Proto https + http-request set-header X-Forwarded-Port %[dst_port] + http-request return status 503 content-type text/html file /usr/local/etc/haproxy/live/errors/maint.http if { str(2),map(opt@/usr/local/etc/haproxy/live/maps/maint.map) -m found } + default_backend be_2 -backend be_6 from loxy_http +backend be_2 from loxy_http balance roundrobin - server t111 10.0.0.31:80 id 111 check inter 15s rise 3 fall 3 - -# ---------- frontend 9 "postgres" (tcp) ---------- -frontend fe_9 from loxy_tcp - bind 10.0.0.5:5432 - maxconn 2000 - tcp-request connection reject if { str(9),map(opt@/usr/local/etc/haproxy/live/maps/maint.map) -m found } - default_backend be_12 + timeout server 60s + cookie LOXYSTICKY insert indirect nocache httponly secure maxlife 300s + server s101 10.0.0.11:8080 id 101 check inter 15s rise 3 fall 3 cookie s101 + server s102x7 10.0.0.12:8080 id 102 check inter 15s rise 3 fall 3 cookie s102x7 + server s102x8 10.0.0.13:8080 id 103 check inter 15s rise 3 fall 3 cookie s102x8 + server s103 10.0.0.14:8080 id 104 check inter 15s rise 3 fall 3 cookie s103 backup + +# ---------- project 2 "db" · service 3 (tcp :20003 -> 5432) ---------- +frontend fe_3 from loxy_tcp + bind 0.0.0.0:20003 + bind ipv6@:::20003 v6only + timeout client 60s + tcp-request connection reject if { str(3),map(opt@/usr/local/etc/haproxy/live/maps/maint.map) -m found } + default_backend be_3 -backend be_12 from loxy_tcp +backend be_3 from loxy_tcp balance leastconn - server t120 10.0.0.41:5432 id 120 check inter 15s rise 3 fall 3 send-proxy-v2 check-send-proxy + timeout server 60s + server s120 10.0.0.41:5432 id 120 check inter 15s rise 3 fall 3 send-proxy-v2 check-send-proxy ``` +`crt-lists/svc2.lst` contains `12.pem [alpn h2,http/1.1]`; `crt-lists/sys_https.lst` contains `21.pem [alpn h2,http/1.1]`; `maps/maint.map` contains `3 1` (project #2 is in maintenance). + Notes: - No `master-worker` keyword (deprecated since 3.3; `-W` on the command line is enough). `crt-base` + relative crt‑list entries make each revision validate against its own files (ADR 0004). -- Bind address `*` renders as `0.0.0.0:` + `ipv6@:::`; explicit IPv6 addresses use the `ipv6@addr:port` prefix form (unambiguous). -- Route conditions: each host becomes one `-m str` (exact) or `-m end` (`*.example.com` → `.example.com`) pattern on the lower‑cased Host without port; multiple hosts in a route are OR‑ed (one ACL with several patterns); `path_prefix` adds `{ path_beg }` (AND). A route with only a path prefix matches any host. Routes are evaluated in `position` order; first match wins, then `default_backend`. A frontend without a default backend answers unmatched requests with 503. -- The ACME HTTP‑01 rule is rendered in every `fe__rd` frontend. If a TLS frontend has a managed HTTP‑01 certificate but no redirect, render `sys_acme_fe` on that frontend's addresses:80 that answers challenges and returns 404 otherwise (port collisions with other frontends are rejected by the API, §8.3). -- `maint.map` is an `opt@` map keyed by frontend id: present on disk in the revision **and** updated at runtime with `add map`/`del map`, so maintenance mode is instant and survives reloads. HTTP frontends return the maintenance page with 503; TCP frontends reject the connection. `virt@acme_http01` is a virtual (memory‑only) map because challenge tokens are ephemeral; the reconciler re‑pushes pending tokens after every reload. -- There is no global/per‑object connection cap other than the frontend `maxconn` (0 = not rendered, the global `maxconn` applies). +- Every service listens on all addresses: `0.0.0.0:` plus `ipv6@::: v6only` only when the host supports IPv6 (startup probe or `LOXY_IPV6`, ADR 0004 §9). The same applies to `sys_http` / `sys_https`. +- `sys_http` is rendered whenever at least one domain is attached to a project **or** any managed certificate uses HTTP‑01 (incl. the panel certificate for `LOXY_PANEL_DOMAIN`). Rule order: ACME answer, 301 for https domains whose certificate is issued, `http-request return status 404` for any Host that is not an attached domain, then `use_backend be_` by Host (a redirected domain never reaches it; http‑only domains and https domains still waiting for their certificate do). +- `sys_https` is rendered once at least one attached domain has an issued certificate (a bind with an empty crt‑list is invalid, so no placeholder is needed). Its crt‑list holds the issued domain certificates; clients without SNI get the first entry. Same 404 rule and `use_backend` by Host as `sys_http`. +- Unknown hosts get 404 from `http-request return` (evaluated before `use_backend`), so no extra backend section is needed. +- Host conditions: each FQDN becomes one `-m str` pattern on the lower‑cased Host without port; `*.example.com` becomes `-m end .example.com`. Domains of the same service share one `use_backend` condition (patterns OR‑ed, sorted by FQDN). `use_backend` rules with only exact patterns MUST come before rules with wildcard patterns, so an exact domain of one project wins over a wildcard of another. +- A domain routes to `be_` of its service (default: the project's first http/https service by `position`). Domains of a project without an http/https service, and detached domains, are not rendered. +- `maint.map` is an `opt@` map keyed by **service id**: present on disk in the revision **and** updated at runtime with `add map`/`del map`, so maintenance is instant and survives reloads. Turning a project's maintenance on adds one entry per service. http/https services answer 503 from `fe_` (the service port) and, when at least one domain routes to the service, also from `be_` (traffic arriving through `sys_http`/`sys_https`; **VERIFY** `http-request return` with an `opt@` map lookup in a backend section); tcp services reject the connection in `fe_`. `virt@acme_http01` is a virtual (memory‑only) map because challenge tokens are ephemeral; the reconciler re‑pushes pending tokens after every reload. +- `timeout client` / `timeout server` are always rendered from the service's `timeout_s` (default 60). +- Targets are rendered in every service backend of their project, with port = the target's port override, else the service's `destination_port`. ### 7.5 Field → directive mapping -**Frontend fields** +**Project fields** -| Field | http | tcp | -|---|---|---| -| mode | `mode http` (via `from loxy_http`) | `mode tcp` (via `from loxy_tcp`); no header injection; use for TLS passthrough | -| binds [{address, port}] | one `bind` per entry; `*` → all IPv4 + IPv6 | same | -| tls + certificates | `bind … ssl crt-list fe.lst alpn h2,http/1.1`; backends get plain HTTP (TLS termination) | not allowed | -| http_redirect (tls only) | extra `fe__rd` on the same addresses:80 with ACME rule + `http-request redirect scheme https code 301` | not allowed | -| default_backend_id | `default_backend be_` | same | -| routes | `use_backend be_ if …` in position order (§7.4) | not allowed | -| timeout_client_s | `timeout client s` | same | -| maxconn | `maxconn ` (omitted when 0) | same | -| forwarded_headers | `option forwardfor`, `X-Forwarded-Proto` (`https` when tls, else `http`), `X-Forwarded-Port` = `%[dst_port]` | not allowed | -| enabled | frontend (and its `_rd` / `sys_acme_fe`) not rendered when false | same | -| maintenance | `http-request return status 503 … maint.http` if id in `maint.map` | `tcp-request connection reject` if id in `maint.map` | - -**Backend fields** - -| Field | http | tcp | -|---|---|---| -| algorithm | `balance roundrobin|leastconn|source|random` — rendered **explicitly** | same | -| port | default server port (a server's `port` overrides it) | same | -| sticky (cookie name, lifetime) | `cookie insert indirect nocache httponly maxlife s` (+ `secure` when every frontend using the backend has tls), each server `cookie ` | not allowed | -| proxy_protocol v1/v2 | `send-proxy` / `send-proxy-v2` **and** `check-send-proxy` on every server line (otherwise servers that require PROXY reject the health check) | same | -| timeout_server_s | `timeout server s` | same | -| server weight / backup / admin_state | `weight `, `backup`; admin state applied at runtime and kept by server‑state (ADR 0004) | same | +| Field | Rendering | +|---|---| +| algorithm | `balance roundrobin|leastconn|source|random` in every `be_` of the project — rendered **explicitly** | +| maintenance | one `maint.map` entry per service; 503 rule in `fe_` and, for services with domains, in `be_` (§7.4) | +| name, description, labels | never rendered | + +**Service fields** + +| Field | http | https | tcp | +|---|---|---|---| +| protocol | `fe_`/`be_` `from loxy_http` | as http + TLS termination | `from loxy_tcp`; no header injection; use for TLS passthrough | +| listen_port | `bind 0.0.0.0:

` (+ `ipv6@:::

v6only`) | `… ssl crt-list svc.lst alpn h2,http/1.1` | as http | +| destination_port | default port of every server line | same | same | +| certificates | not allowed | crt‑list `svc.lst` (≥ 1) | not allowed | +| forwarded headers | always: `option forwardfor`, `X-Forwarded-Proto http`, `X-Forwarded-Port %[dst_port]` | same with `X-Forwarded-Proto https` | none | +| timeout_s | `timeout client` in `fe_`, `timeout server` in `be_` | same | same | +| sticky (cookie name, lifetime) | `cookie insert indirect nocache httponly maxlife s`, each server `cookie ` | same + `secure` | not allowed | +| proxy_protocol v1/v2 | `send-proxy` / `send-proxy-v2` **and** `check-send-proxy` on every server line (otherwise targets that require PROXY reject the health check) | same | same | + +**Target fields** (rendered once per service of the project) + +| Field | Rendering | +|---|---| +| ip / host / label selector | `server s` / `sx` `

: id ` — address: IP, host `ipv4` (or `private_ipv4` with `use_private_ip`, else `ipv6`), one server per matching host for selectors | +| port | overrides the service destination port | +| weight | `weight ` (omitted when 100) | +| backup | `backup` | +| admin_state | applied at runtime (`set server … state`) and kept across reloads by server‑state (ADR 0004 §6, §8). `render.ServerArgs` (shared with runtime `add server`) never carries it; the backend renderer appends `disabled` to the server line of a target in `maint`, so a cold start keeps maintenance. `drain` is runtime‑only | -**Health check (backend)** +**Health check (service)** | Field | Rendering | |---|---| @@ -500,36 +546,45 @@ Status‑code conversion (pure function, table‑tested): each item matches `^[1 **VERIFY (important):** that two consecutive `http-check expect` rules after one `send` are both evaluated against the same response (status AND body). Test: server returns 200 without the string → must be DOWN; returns 500 with the string → must be DOWN; 200 with string → UP. If HAProxy evaluates only one, fall back to a raw `option tcp-check` sequence for that backend (send an HTTP/1.1 request, `tcp-check expect rstring ^HTTP/1\.[01] (2|3)[0-9][0-9]`, then `tcp-check expect string `), and document it in `docs/haproxy-notes.md`. +**Domain fields** + +| Field | Rendering | +|---|---| +| fqdn + project/service | Host condition in `sys_http` (until redirected) and `sys_https` → `use_backend be_` | +| https | `sys_https` routing; once the certificate is issued: certificate in `sys_https.lst` and 301 in `sys_http` | +| certificate (managed HTTP‑01) | ACME answers through `virt@acme_http01` in `sys_http` | +| dns_provider, auto_records, ttl | not rendered (DNS only, §8.12) | + ### 7.6 Change classification (runtime vs reload) | Change | Path | Commands / notes | |---|---|---| -| Add server (any type) when algorithm ∈ {roundrobin, leastconn, random} | runtime | `add server be_4/t105 1.2.3.4:9000 id 105 check inter 15s rise 3 fall 3 [weight n] [backup] [cookie t105] [send-proxy-v2 check-send-proxy]` → `enable server …` → `enable health …` (dynamic servers start in maintenance with checks off; `default-server` settings are **not** inherited, so render every keyword explicitly; the `enabled` keyword is rejected) | -| Remove server | runtime | `set server / state drain` → wait for `scur=0` or drain timeout (default 30 s) → `set server … state maint` → `wait 5s srv-removable /; del server /` | -| Weight change | runtime | `set server / weight ` | -| Drain / maintenance / ready | runtime | `set server / state drain|maint|ready` | +| Add target (any type) when algorithm ∈ {roundrobin, leastconn, random} | runtime | for every service `S` of the project and every resolved slot: `add server be_/s105 1.2.3.4:8080 id 105 check inter 15s rise 3 fall 3 [weight n] [backup] [cookie s105] [send-proxy-v2 check-send-proxy]` → `enable server …` → `enable health …` (dynamic servers start in maintenance with checks off; `default-server` settings are **not** inherited, so render every keyword explicitly; the `enabled` keyword is rejected) | +| Remove target | runtime | per service: `set server / state drain` → wait for `scur=0` on all services or drain timeout (default 30 s) → `set server … state maint` → `wait 5s srv-removable /; del server /` | +| Weight change | runtime | `set server be_/ weight ` for every service | +| Drain / maintenance / ready | runtime | `set server be_/ state drain|maint|ready` for every service | | Label‑selector match set changes (host labels or addresses edited) | runtime | treated as add/remove | -| Server changes in `source` (hash) backends | **reload** | dynamic servers only work with roundrobin, leastconn, first, random | -| Frontend maintenance on/off | runtime | `add map /usr/local/etc/haproxy/live/maps/maint.map 1` / `del map … ` (+ file in revision). The `opt@` prefix is not part of a map's name, so runtime commands use the plain path; `virt@` *is* part of the name | +| Target changes in a `source` (hash) project | **reload** | dynamic servers only work with roundrobin, leastconn, first, random | +| Project maintenance on/off | runtime | `add map /usr/local/etc/haproxy/live/maps/maint.map 1` / `del map … ` per service (+ file in revision). The `opt@` prefix is not part of a map's name, so runtime commands use the plain path; `virt@` *is* part of the name | | Certificate renewed / replaced (same file) | runtime | `set ssl cert <<\n\n` → `commit ssl cert ` | | ACME HTTP‑01 token | runtime | `add map virt@acme_http01 ` / `del map …` | -| Add / edit / delete / enable / disable frontend | reload | new or changed listeners need a new worker | -| Binds, TLS, redirect, routes, default backend, timeouts, maxconn, forwarded headers | reload | | -| Certificate attached to / detached from a frontend | reload | any crt‑list change is reload‑class (ADR 0004); MAY later use `add/del ssl crt-list` | -| Add / edit / delete backend (algorithm, health check, sticky, proxy protocol, timeouts, port) | reload | | +| Create / delete project, algorithm change | reload | | +| Add / edit / delete service (any field) | reload | new or changed listeners need a new worker | +| Certificate attached to / detached from an https service | reload | any crt‑list change is reload‑class (ADR 0004); MAY later use `add/del ssl crt-list` | +| Add / delete / move domain, toggle `https`, domain certificate issued (adds the cert to `sys_https.lst` and turns on the 301) | reload | `sys_http`/`sys_https` rules change | | Panel allowlist | reload (rare) | MAY use `add acl` at runtime | Every runtime op also produces the new on‑disk revision. If any runtime command fails, the reconciler falls back to a reload of that revision. -HAProxy 3.4 also allows creating, publishing and deleting whole **backends** at runtime (`add backend … from `, `publish backend`, `unpublish backend`, `del backend`). v1 doesn't need it (backends become reachable only through a frontend change, which is a reload anyway); MAY be used later to make "add backend + route" reload‑free once `use_backend` rules can also change at runtime. +HAProxy 3.4 also allows creating, publishing and deleting whole **backends** at runtime (`add backend … from `, `publish backend`, `unpublish backend`, `del backend`). v1 doesn't need it (a new service always needs a new listener, which is a reload anyway); MAY be used later. ### 7.7 Reload procedure 1. `show servers state` → write `/var/lib/haproxy/server-state` atomically. 2. `haproxy -c -f revs/N/haproxy.check.cfg` (inside the loxy container, same HAProxy binary). Non‑zero exit → abort, keep current `live`, record the output. 3. Swap `live` → `revs/N`. 4. Master CLI `reload`; read until EOF; require `Success=1`. -5. On `Success=0`: swap `live` back to the previous revision (so a future cold start is safe), record the startup log, mark the affected frontends/backends `apply_error`, notify. Old workers keep serving the previous config. -6. Post‑checks: `show proc` lists a new worker; `show stat` contains every expected frontend/backend; otherwise treat as failed. +5. On `Success=0`: swap `live` back to the previous revision (so a future cold start is safe), record the startup log, mark the affected projects `error`, notify. Old workers keep serving the previous config. +6. Post‑checks: `show proc` lists a new worker; `show stat` contains every expected frontend/backend section; otherwise treat as failed. 7. Re‑push ephemeral runtime state (pending ACME tokens in `virt@acme_http01`). 8. Rate limits: coalesce via the reconciler debounce; at most one reload per second per node; a reload storm queues. 9. Old workers are bounded by `hard-stop-after` (setting, default 15 m) and `mworker-max-reloads` (default 50), so long‑lived connections cannot pile up memory forever. @@ -547,110 +602,143 @@ On first start loxy writes `revs/000001/` from `deploy/haproxy/bootstrap.cfg` (g - 3.4 changed compression syntax: `filter comp-res` / `filter comp-req`; `compression direction` is deprecated. - 3.4 stats page hides the version unless `stats show-version`. - Native ACME inside HAProxy is still marked experimental and keeps renewed certs in memory only; loxy uses lego instead (§11) and MAY switch later. +- `ipv6@:::` makes the worker exit at startup on hosts without IPv6 although `haproxy -c` passes (ADR 0004 §9). --- ## 8. Feature specification (screens, behaviour, validation, acceptance) -Global navigation: **Dashboard · Frontends · Backends · Hosts · Certificates · Domains · Activity · System · Settings** (Domains has sub‑pages *Domains*, *DNS providers*, *Zones*). Top bar: project switcher (hidden while only one project exists), global apply status (applying / last error with "Retry", `POST /apply/retry`), user menu, live connection indicator (SSE state). +Global navigation: **Dashboard · Projects · Hosts · Certificates · Domains · Activity · System · Settings** (Domains has sub‑pages *Domains*, *DNS providers*, *Zones*). Top bar: global apply status (applying / last error with "Retry", `POST /apply/retry`), user menu, live connection indicator (SSE state). + +All destructive actions use a confirm modal; deleting a project requires typing its name. Every mutation returns an **action** object (§12.4) whose progress is shown as a toast and in Activity. An apply error shows a banner on the affected project and on System with the HAProxy output and "Retry" / "Roll back to last good revision". -All destructive actions use a confirm modal; deleting a frontend or backend requires typing its name. Every mutation returns an **action** object (§12.4) whose progress is shown as a toast and in Activity. An apply error shows a banner on the affected frontend/backend and on System with the HAProxy output and "Retry" / "Roll back to last good revision". +**Reserved ports** (used by every port validation below): **80 and 443** (domains), the panel port (`LOXY_PANEL_PORT`, default 8443), **8405** (stats), **5514** (log receiver) and the port of `LOXY_LISTEN` (default 9090). `LOXY_PANEL_PORT` set to 80 or 443 is refused at start. ### 8.1 Dashboard (`/dashboard`) -- Stat row (live via SSE): frontends (enabled / total, in maintenance), backends by health (healthy / degraded / unhealthy / unknown), servers (healthy / total, draining, in maintenance), certificates expiring < 21 d, current open connections and requests/s across all frontends, bandwidth in/out. -- **Needs attention** list: unhealthy/degraded backends with the first failing checks ("10.0.0.12:9000 · Layer7 wrong status 503"), frontends with apply errors, expiring or failed certificates, domains not resolving correctly. -- Traffic sparkline per top‑5 frontend (last 5 min), last 10 events, node status (HAProxy version, uptime, last reload result). +- Stat row (live via SSE): projects (total, by health: healthy / degraded / unhealthy, in maintenance, with apply errors), targets (healthy / total, draining, in maintenance), certificates expiring < 21 d or failed, current open connections and requests/s across all projects, bandwidth in/out. +- **Needs attention** list: unhealthy/degraded projects with the first failing checks ("shop · 10.0.0.12:8080 on :20001 · Layer7 wrong status 503"), projects with apply errors, expiring or failed certificates, domains not resolving correctly. +- Traffic sparkline per top‑5 project (last 5 min), last 10 events, node status (HAProxy version, uptime, last reload result). - Acceptance: renders from `GET /overview` + one SSE stream in < 100 ms after data arrives; no per‑card polling. -### 8.2 Frontends list (`/frontends`) -- Table: status dot (apply status + maintenance + disabled), name, `#id`, mode badge (`HTTP`, `HTTPS` when tls, `TCP`), binds (`*:443`, `10.0.0.5:5432`, "+2"), default backend, routes count, backend health aggregate, live open connections sparkline (last 5 min), req/s, labels. -- Filters: text search, label selector (`env=prod,team!=ops`), mode, status. Sort by name/created/connections. -- **Create frontend** button → form (§8.3). Row ⋯ menu: Edit · Maintenance on/off · Enable/Disable · Duplicate · Delete. -- Empty state explains the model in one line ("A frontend listens on addresses and ports and sends traffic to backends.") with "Create frontend" and "Create backend" actions. -- Acceptance: 100 frontends render in < 100 ms after data arrives; sparklines update via one SSE stream, not per‑row polling. - -### 8.3 Frontend create / edit form -Single page with sections; the same form edits an existing frontend (PUT with `If-Match`). -1. **Name, description, labels.** -2. **Mode & TLS** — mode `http | tcp` (segmented control); for http a **TLS termination** toggle; when on: **certificates** picker (chips; existing certificates or create a managed certificate inline from the hostnames of the routes, or upload) and **HTTP → HTTPS redirect (301)** checkbox (default on). -3. **Binds** editor — rows of address + port; the address field is a combobox with `*` (all addresses) and the node addresses (§8.10) as suggestions, any IPv4/IPv6 literal allowed; add/remove rows; templates for the port: 80, 443, custom. Inline collision warnings while typing. -4. **Default backend** — select (filtered to backends of the same mode) or "Create backend…" (opens the backend form in a modal, returns its id). -5. **Routing rules** (http only) — table of rules, drag handle to reorder (order = evaluation order, first match wins): hosts (chip input, exact or `*.example.com`), path prefix (`/api`), backend select; "Add rule". A preview line under each rule reads like the rendered condition ("Host is app.example.com and path starts with /api → api‑backend"). -6. **Options** — client timeout (s, default 60), max connections (0 = unlimited), forwarded headers (http, default on), enabled (default on). -- Footer: "Show config preview" (renders the would‑be `fe_` sections via `loxy render --dry-run` logic, read‑only `CodeView`), Cancel, Save. Creation of a frontend with a new default backend happens in one transaction → one reload. +### 8.2 Projects list (`/projects`) +- Cards or table (toggle, remembered per browser): health badge, name, `#id`, **endpoints** (`:20001`, `https://shop.example.com`, "+2", copy on click), protocols (`HTTP`, `HTTPS`, `TCP` badges), targets healthy/total, live open connections sparkline (last 5 min), req/s, maintenance badge, labels. +- Filters: text search, label selector (`env=prod,team!=ops`), health. Sort by name/created/connections. +- **Create project** button → wizard (§8.3). Row ⋯ menu: Open · Maintenance on/off · Copy ID · Delete. +- Empty state explains the model in one line ("A project is a load balancer: it listens on a port or a domain and spreads traffic over your targets.") with "Create project". +- Acceptance: 50 projects render in < 100 ms after data arrives; sparklines update via one SSE stream, not per‑row polling. + +### 8.3 Create project wizard (`/projects/new`) +One page, three short steps with a summary; only the name is required. Submitting creates everything in **one transaction → one reload** (`POST /projects` with `targets` and `service`). +1. **Name** (required), description, labels, algorithm (Round Robin default; Least Connections, Source, Random). +2. **Targets** (optional, "Skip for now"): the same add‑targets control as §8.4 Targets (IP list / hosts / label selector). +3. **Service:** protocol `http` (default) | `https` | `tcp` (segmented control); **listen port**: "Automatic" (default; shows the port that will be taken, e.g. "next free: 20003") or a custom value; **destination port** (default 80 for http/https, required for tcp); certificates for https (picker; upload or create managed). Advanced (collapsed): health check, sticky sessions, PROXY protocol, timeout — with the defaults of §8.4 Services. +4. Optional: **Attach a domain** (FQDN; shown only for http/https). +- On success the project Overview opens with its endpoints ("Reachable at `http://203.0.113.10:20003`") and, if a domain was added, its required DNS records. +- Validation: as §8.4 Services/Targets and §8.4 Domains. +- Acceptance: creating a project with only a name gives an http service on the lowest free port in 20000–29999; with 2 IP targets it is reachable and both targets are healthy within 3 s of submit on a dev box; exactly one reload. + +### 8.4 Project (`/projects/[id]/…`) +**Header (all tabs):** editable name (inline), health badge, `#id`, endpoints (copy on click), maintenance toggle (confirm), ⋯ menu: Export JSON · Copy ID · Delete. + +Tabs: **Overview · Graphs · Targets · Services · Domains · Logs · Settings**. Every tab is a route (`/projects/1/targets`). + +**Overview** +- Endpoints card: each service as `protocol listen → destination` with its URL (`http://:`), each domain as `https://fqdn` (or `http://fqdn` until the certificate is issued) with DNS and certificate status. +- Stat cards (live): open connections, conn/s, req/s, bandwidth in/out. +- **Health** card: donut of healthy targets / total, badge `Healthy | Degraded | Unhealthy | Unknown`, first 3 failing checks with service and reason. +- Targets and services summaries (counts, link to tabs); **Labels** row (chips + editor, §14.4 rules); getting‑started hints while the project has no targets or no domain. + +**Graphs** +- Range selector **Live** (rolling 5 min, 1 s points, auto‑updating), 30 min, 1 h, 6 h, 24 h, 7 d, 30 d; filter: whole project / one service. +- Charts: **Open connections**, **Connections per second**, **HTTP requests per second** (http/https), **Bandwidth** (in/out; units auto‑scale B/s → GB/s); "Details" (collapsed): **Sessions**, **Queue**, **Average response time (ms)**, **HTTP responses per second by class** (2xx/3xx/4xx/5xx), **Errors per second** (connection, response, retries/redispatches, denied). +- uPlot with synced cursors; tooltip with exact values and timestamp in the browser's timezone; Live pauses while hovering; no false spikes or dips across reloads (§10.3). +- Acceptance: switching ranges renders in < 150 ms for 30 d; Live updates every second with ≤ 2 s lag; browser CPU < 5 % on a laptop. + +**Targets** +- **Add targets** dropdown: **IP** — one or many (paste, one per line, optional `:port`, IPv6 as `[addr]:port`); rejects loopback (unless `LOXY_DEV=true`), link‑local, multicast, unspecified. **Host** — multi‑select from the inventory with search and label filter; "Use private IP" toggle. **Label selector** — input with syntax `key=value`, `key!=value`, `key in (a,b)`, `key` (exists), comma = AND; live preview "matches N hosts" (`GET /label_selector/preview`). Each with optional port override, weight (default 100) and backup flag. +- **Grouped / List** toggle. Grouped: each label selector is a group (tag icon, "N hosts", aggregate health, ⋯); host targets grouped as "Hosts"; IP targets as "IP addresses". List: flat table of resolved targets. +- Row columns: name (host name or "—"), address (port override shown when set), **status** per service (one chip per service: `:20001 healthy`, with check detail tooltip `L7OK/200 in 3 ms` or failure reason; the row status is healthy if healthy on all services, degraded if mixed, unhealthy if on none), weight, backup, sessions, ⋯ menu: Drain · Maintenance · Ready · Set weight · Remove. +- Removal is graceful (drain → maint → delete, §7.6) and shows progress (`removing`). +- Empty state: "No targets yet — add the servers that should receive traffic." with the three add options. +- Validation: ≤ 1 000 targets per installation budget (§1.4); duplicate IP targets (same address and port) within a project refused; a host may be referenced directly only once per project; weight 1–256; port 1–65535. +- Acceptance: adding 25 IP targets to a project with one service results in 25 runtime `add server` ops, **no reload**, all active in < 1 s, zero failed requests under load. + +**Services** +- List: protocol badge, `listen → destination`, URL, health check summary (`HTTP GET /health · 15 s`), sticky / PROXY badges, certificates (https), domains routed to it, health (healthy targets / total for this service). Row ⋯: Edit · Delete. **Add service** button. +- Form: protocol `http | https | tcp`; listen port ("Automatic" or custom); destination port; certificates (https; ≥ 1, pending managed allowed — the placeholder is served until issuance); **health check**: protocol (tcp/http/https), port (default = target port), interval (s), timeout (s), retries, domain, path, response, status codes, TLS verify (https checks); **sticky sessions** (http/https): checkbox + cookie name + lifetime (s); **PROXY protocol**: none/v1/v2 with a red warning that enabling it for targets that don't accept PROXY makes the service unreachable; timeout (s). "Show config preview" renders the would‑be `fe_`/`be_` (read‑only `CodeView`). +- Defaults: http, automatic port, destination 80; health check tcp for tcp services and http (`GET /`, status codes `2??, 3??`) for http/https, interval 15 s, timeout 10 s, retries 3; timeout 60 s; sticky cookie `LOXYSTICKY`, lifetime 300 s; PROXY none. - Validation (server side, mirrored client side): - - ≥ 1 bind; port 1–65535; address `*` or a valid IP literal (`net/netip`); no duplicate binds within the frontend. - - **Port collisions across frontends:** a bind collides with another frontend's bind on the same port when the addresses are equal or either is `*`; the check includes every frontend's redirect listener (:80 on its addresses when `http_redirect`), every `sys_acme_fe` :80 listener, and reserved system ports (panel port, `127.0.0.1:8405`, `127.0.0.1:5514`, `127.0.0.1:9090`). Disabled frontends still reserve their ports (so enabling never fails on a collision). - - `tls` requires mode http and ≥ 1 certificate (a pending managed cert is allowed; a placeholder self‑signed cert is served until issuance). - - `http_redirect` only with tls, and only when :80 is free on all of the frontend's addresses. - - Routes require mode http; every route needs ≥ 1 host or a path prefix; hosts are RFC 1123 (`*.` prefix allowed), lowercase, ≤ 20 per route; path prefix matches the path rule of §14.4; ≤ 200 routes per frontend. - - The default backend and every route's backend must exist and have the **same mode** as the frontend. - - `timeout_client_s` 1–3600; `maxconn` 0–1 000 000. -- Acceptance: creating an http frontend on `*:8080` with a default backend of 2 IP servers → reachable, both servers healthy, within 3 s of submit on a dev box; a TLS frontend with a managed cert on a domain pointing to the node yields a valid certificate (Pebble in CI) and a 301 from :80 within 60 s, with only one reload. - -### 8.4 Frontend detail (`/frontends/[id]/…`) -**Header (all tabs):** editable name (inline), status, `#id`, mode badge, binds (copy on click), maintenance toggle (confirm), ⋯ menu: Edit · Enable/Disable · Duplicate · Export JSON · Copy ID · Delete. - -Tabs: **Overview · Routing · Graphs · Logs · Config**. -- **Overview:** stat cards (open connections, conn/s, req/s, bandwidth in/out, live); **Health** card: donut of healthy servers / total across the backends this frontend uses, badge `Healthy | Degraded | Unhealthy | Unknown`, first 3 failing checks with reason; **Binds** card; **TLS** card (certificates with expiry, redirect on/off); **Labels** row (chips + editor, §14.4 rules); **Domains** attached to this frontend with their DNS status (§8.9). -- **Routing:** read view of the default backend and the ordered rules, each with the target backend's health; "Edit routing" opens the routing section of §8.3. Backends link to §8.6. -- **Graphs:** range selector **Live** (rolling 5 min, 1 s points, auto‑updating), 30 min, 1 h, 6 h, 24 h, 7 d, 30 d. Charts: **Open connections**, **Connections per second**, **HTTP requests per second** (http only), **Bandwidth** (in/out; units auto‑scale B/s → GB/s); "Details" (collapsed): HTTP responses per second by class (2xx/3xx/4xx/5xx), denied/errored requests. uPlot with synced cursors; tooltip with exact values and timestamp in the browser's timezone; Live pauses while hovering; no false spikes or dips across reloads (§10.3). Acceptance: switching ranges renders in < 150 ms for 30 d; Live updates every second with ≤ 2 s lag; browser CPU < 5 % on a laptop. -- **Logs:** live log stream of this frontend (sampled, off by default, §20): pause, filter text, status class filter; columns time, client, method + path (http) / bytes, backend/server, status, timings. -- **Config:** read‑only rendered sections of this frontend (`fe_`, `fe__rd`, `sys_acme_fe`, and the backends it uses) from `GET /frontends/{id}/config`, with "Copy" and "Download". -- Delete: confirm by typing the name; removes the frontend sections (reload), detaches its domains (`frontend_id` set NULL) and certificates (certificates are kept), keeps metrics for 7 days then purges, audit entry. Backends are untouched. - -### 8.5 Backends list (`/backends`) -- Table: health badge, name, `#id`, mode, algorithm, servers healthy/total (with draining/maint counts), used by (frontend names), sessions, queue, avg response time, labels. -- Filters: text, label selector, mode, health. **Create backend** → form (name, description, mode, algorithm, default port, health check, then add servers right away). -- Acceptance: 200 backends with 1 000 servers render in < 150 ms after data arrives. - -### 8.6 Backend detail (`/backends/[id]/…`) -Header: name, `#id`, mode, algorithm, health badge, "Used by" chips linking to frontends, ⋯ menu: Duplicate · Export JSON · Copy ID · Delete. - -Tabs: **Servers · Settings · Graphs**. -- **Servers:** - - **Add servers** dropdown: **IP** — one or many (paste, one per line, optional `:port`); IPv4/IPv6; rejects loopback (unless `LOXY_DEV=true`), link‑local, multicast, unspecified. **Host** — multi‑select from the inventory with search and label filter; "Use private IP" toggle. **Label selector** — input with syntax `key=value`, `key!=value`, `key in (a,b)`, `key` (exists), comma = AND; live preview "matches N hosts" (`GET /label_selector/preview`). Each with optional port override, weight (default 100) and backup flag. - - **Grouped / List** toggle. Grouped: each label selector is a group (tag icon, "N hosts", aggregate health, ⋯); host servers grouped as "Hosts"; IP servers as "IP servers". List: flat table of resolved servers. - - Row columns: name (host name or "—"), address:port, **status** (healthy / unhealthy / draining / maintenance / unknown, with check detail tooltip `L7OK/200 in 3 ms` or failure reason), weight, backup, sessions, ⋯ menu: Drain · Maintenance · Ready · Set weight · Show in graphs · Remove. - - Removal is graceful (drain → maint → delete, §7.6) and shows progress (`removing`). - - Empty state: "No servers yet." with the three add options. - - Acceptance: adding 25 IP servers results in 25 runtime `add server` ops, **no reload**, all active in < 1 s, zero failed requests under load. -- **Settings:** algorithm (Round Robin default, Least Connections, Source, Random; note that Source makes server changes reload‑class), default port; **health check**: protocol (tcp/http/https), port, interval (s), timeout (s), retries, domain, path, response, status codes, TLS verify (https checks); **sticky sessions**: checkbox + cookie name + lifetime (s); **PROXY protocol**: none/v1/v2 with a red warning that enabling it for servers that don't accept PROXY makes the backend unreachable; server timeout (s); labels. - - Defaults: algorithm roundrobin; hc tcp for tcp backends and http for http backends, interval 15 s, timeout 10 s, retries 3, path `/`, status codes `2??, 3??`; server timeout 60 s; sticky cookie `LOXYSTICKY`, lifetime 300 s; health‑check port = server port. - - Validation: port 1–65535; `hc_timeout < hc_interval`; interval 3–3600; timeout 1–3600; retries 1–10; http/https checks require a path and are allowed on tcp backends too; HTTP fields are returned as null for tcp checks; sticky only for http mode; changing `mode` is refused while the backend is used by a frontend of the other mode. -- **Graphs:** same range selector as §8.4; charts **Sessions** (current), **Queue**, **Average response time (ms)**, **HTTP responses per second by class** (http), **Errors per second** (connection, response, retries/redispatches); filter: whole backend / one server. -- Delete: refused with `409 conflict` while the backend is a default backend or a route target of any frontend (message lists them); otherwise confirm by typing the name → reload, server slots released, metrics kept 7 days. - -### 8.7 Hosts (inventory, `/hosts`) + - listen port 1–65535, **not reserved** (§8 header) and **unique across all projects** (409 names the project using it: "port 20001 is used by project shop (#1)"); automatic allocation picks the lowest free port in **20000–29999** inside the same transaction ("no free port left" error when exhausted; explicit ports outside the range are allowed). + - destination port 1–65535. + - certificates only for https and ≥ 1 there; sticky only for http/https; `hc_timeout < hc_interval`; interval 3–3600; timeout 1–3600; retries 1–10; http/https checks require a path and are allowed on tcp services too; HTTP fields are returned as null for tcp checks; `timeout_s` 1–3600. + - ≤ 20 services per project. + - Changing a service to `tcp` or deleting it is refused with `409 conflict` while it is the **only** http/https service of a project that has domains (message lists the domains); domains pinned to a deleted service fall back to the default service. +- Acceptance: adding an https service with a managed certificate (Pebble in CI) serves a valid certificate within 60 s with only one reload; changing the health check path takes effect after one reload. + +**Domains** +- The project's domains: FQDN, service (default or pinned), HTTPS on/off, DNS status (**Resolves correctly** / **Wrong IP** / **Missing** / **Unknown**), certificate (issued / pending / failed / none), URL. ⋯: Check now · Edit · Detach · Delete. +- **Add domain**: FQDN (§14.4), service (only http/https services; default "First HTTP service"), **HTTPS** toggle (default on), optional DNS provider (auto‑detected when a connected provider owns a matching zone), "Manage A/AAAA records automatically" (only with a provider; Phase 2). Shows the **required records** immediately (§8.12). +- Validation: FQDN unique across the installation (409 names the project using it); wildcard `*.example.com` only with a DNS provider (DNS‑01 required); the project needs ≥ 1 http/https service; ≤ 100 domains per project. +- Acceptance: see §8.12. + +**Logs** — live log stream of this project (sampled, off by default, §20): pause, filter text, service filter, status class filter; columns time, client, service, method + path (http) / bytes, target, status, timings. + +**Settings** +- Name, description, labels, algorithm (note that Source makes target changes reload‑class), maintenance mode (explains: http 503 page, tcp connections rejected; applies to ports and domains). +- **Config** (read‑only): rendered sections of this project (`fe_`, `be_` of all services, plus the `sys_http`/`sys_https` lines for its domains) from `GET /projects/{id}/config`, with "Copy" and "Download". +- **Danger zone:** Delete project — confirm by typing the name; removes all services and targets (reload), detaches its domains through `ON DELETE SET NULL` on `domains.balancer_id` (the domain stays, with no project, on the global Domains page; auto‑managed records are removed), keeps metrics for 7 days then purges, audit entry. Certificates are kept. + +### 8.5 Hosts (inventory, `/hosts`) - CRUD of hosts: name, IPv4, IPv6, private IPv4, labels, notes. Bulk import from CSV (`name,ipv4,ipv6,private_ipv4,labels`). -- Shows where each host is used (backends, directly or via a label selector) and its current health there. -- Editing a host's labels or addresses updates every label‑selector / host server at runtime (§7.6). Deleting a host that is referenced directly by a server is refused; selector matches simply drop out (graceful removal). +- Shows where each host is used (projects, directly or via a label selector) and its current health there. +- Editing a host's labels or addresses updates every label‑selector / host target at runtime (§7.6). Deleting a host that is referenced directly by a target is refused; selector matches simply drop out (graceful removal). -### 8.8 Certificates (`/certificates`) -- List: name, type (Uploaded/Managed), domains, status (Pending/Issued/Renewing/Failed), expires in (amber < 21 d, red < 7 d), used by N frontends, labels. +### 8.6 Certificates (`/certificates`) +- List: name, type (Uploaded/Managed), domains, status (Pending/Issued/Renewing/Failed), expires in (amber < 21 d, red < 7 d), used by (services and domains), labels. - **Upload**: paste or file for certificate, chain, private key. Validate: key matches cert, not expired, chain completes to a root in the system bundle (warn only), reorder chain if needed. Key stored encrypted at rest. -- **Create managed**: domains (≤ 100), challenge HTTP‑01 (default; domains must resolve to a frontend that serves the challenge on :80 — its redirect frontend or `sys_acme_fe`) or DNS‑01 (pick a connected DNS provider from §8.9 — credentials are not entered again; required for wildcards), key type ECDSA P‑256 (default) / RSA 2048/4096. -- Detail drawer: SANs, issuer, serial, fingerprint, validity, last issuance log, renewal schedule, frontends using it. Actions: Renew now · Download chain (never the key unless owner role + re‑auth) · Delete (blocked while attached to a frontend). +- **Create managed**: domains (≤ 100), challenge HTTP‑01 (default; the names must resolve to the node — Loxy answers on :80 through `sys_http`) or DNS‑01 (pick a connected DNS provider from §8.12 — credentials are not entered again; required for wildcards), key type ECDSA P‑256 (default) / RSA 2048/4096. +- Certificates created automatically for domains appear here with a "Domain" badge and are deleted with their domain. +- Detail drawer: SANs, issuer, serial, fingerprint, validity, last issuance log, renewal schedule, services/domains using it. Actions: Renew now · Download chain (never the key unless owner role + re‑auth) · Delete (blocked while used by a service or domain). + +### 8.7 Domains — global list (`/domains`) +- All domains of the installation: FQDN, project (or **Detached**), service, HTTPS, DNS provider (or "manual"), DNS status, last checked, certificate, auto‑managed records on/off. +- Filters: project, status, detached only. **Add domain** (as §8.4 Domains, with a project picker). Detached domains (e.g. after a project was deleted or after migration `0002`) offer **Attach to project**. +- Sub‑pages *DNS providers* and *Zones* (§8.12). + +### 8.8 System (`/system`) +- **Node** card (all‑in‑one shows the local node): status, HAProxy version and feature flags (`haproxy -vv`: QUIC, PROMEX, OpenSSL version), loxy version, HAProxy uptime, reload count and last reload result/duration, current connections, CPU/memory of both containers, IPv6 listener status; warning when the bundled `haproxy -v` differs from the running one. +- **Node addresses**: list of the node's IPv4/IPv6 addresses with an optional label (`POST/DELETE /nodes/{id}/ips`); detected interface addresses are offered for one‑click adding. Used for endpoint URLs and as DNS required records for domains (§8.12). Changing them never changes the HAProxy config by itself. +- **Ports in use**: every listen port with its project and service, plus the reserved ports. +- **Config revisions**: list (revision, kind runtime/reload/rollback/bootstrap, result, duration, author, time, pinned), diff between any two (`CodeView`), "Roll back to this revision" (break‑glass swap + reload of the on‑disk revision, see ADR 0004 §7; the next mutation re‑applies desired state, which the UI states), pin/unpin, HAProxy output of failed applies. +- Phase 3 adds enrollment, multiple nodes and HA groups (§15). -### 8.9 Domains & DNS (`/domains`) +### 8.9 Activity (`/activity`) +- **Events** feed (target/project health transitions, apply/reload results, cert issued/failed/expiring, domain status changes, node offline) with filters by project/kind/severity. +- **Audit log** (who, what, when, before/after diff, IP, token) — read‑only, exportable CSV. + +### 8.10 Settings (`/settings/*`) +Users & roles · API tokens · ACME (account email, directory: Let's Encrypt production / staging / custom, EAB) · Notification channels & alert rules (event kinds, label filter on **project labels** — node‑wide events like apply and backup are not filtered, min severity, cooldown, "Send test") · Two‑factor authentication (own TOTP; enforcement for all users) · Panel (domain, port, allowlist, session lifetime) · System (hard‑stop‑after, drain timeout, reload min interval, metrics retention, live‑log sampling) · Backup (schedule, retention, optional S3‑compatible target). + +### 8.11 Automatic HTTPS for domains +- A domain with `https` on gets a managed certificate automatically: each verification (`VerifyDomainNow` — on add, "Check now" and the periodic checks of §8.12) that finds the name resolving to the node's addresses issues a managed certificate for the FQDN, if the domain has none yet — HTTP‑01 through `sys_http`, or DNS‑01 when the domain has a DNS provider (required for wildcards) — and links it via `domains.certificate_id`. A domain that already has a certificate is left as is. +- Until the certificate is issued, the domain is served over http on :80 (no redirect); only once it is issued the certificate is added to `sys_https.lst` (rendering `sys_https` if it is the first) and :80 answers `301` to `https://` (one reload). Renewals are runtime (§11.2). +- Failures (rate limits, wrong DNS) show on the domain with the ACME error and retry with the backoff of §11.2; the domain keeps working over http meanwhile. +- `https` off: the domain is served over http on :80 only and no certificate is requested. -Loxy helps operators point names at their frontends and keeps DNS in sync, **without running its own DNS server** (non‑goal, §1.3). All record changes go through the APIs of external DNS providers using the **libdns** ecosystem (`github.com/libdns/*`). +### 8.12 DNS: required records, verification, providers, zones -**Domains page** (`/domains`) -- List: FQDN, attached frontend, DNS provider (or "manual"), status badge (**Resolves correctly** / **Wrong IP** / **Missing** / **Unknown**), last checked, certificate (issued / pending / none), auto‑managed records on/off. -- **Add domain**: FQDN (validated per §14.4), choose a frontend (required), optional DNS provider (auto‑detected when a connected provider owns a matching zone), "Manage A/AAAA records automatically" checkbox (only with a provider; Phase 2). -- **Required records** are derived from the frontend's binds: each explicit bind address gives `A → ` / `AAAA → `; when the frontend binds `*`, the node's addresses (§8.10, public ones only — loopback/private are skipped unless they are the only ones) are used instead. TTL 300, with copy buttons. -- Detail drawer: required records table; **live verification**: Loxy resolves the name (A and AAAA, via the system resolver and, when a provider is attached, also by reading the provider's records) and shows per family: ✓ resolves correctly, ✗ wrong IP (shows the observed addresses), ✗ missing. -- Verification runs on add, on "Check now", every 5 minutes for unverified domains and every hour for verified ones, and whenever the frontend's binds or the node addresses change. Status changes emit `domain.status.changed` events. -- **Issue certificate** (one click) is enabled once the domain is verified: creates a managed certificate for the FQDN (HTTP‑01 if the name resolves to the frontend and it has a :80 listener, or DNS‑01 if a provider is attached — required for wildcards) and attaches it to the frontend when it terminates TLS. +Loxy helps operators point names at their projects and keeps DNS in sync, **without running its own DNS server** (non‑goal, §1.3). All record changes go through the APIs of external DNS providers using the **libdns** ecosystem (`github.com/libdns/*`). + +- **Required records** of a domain are the node's addresses (§8.8; public ones only — loopback/private are skipped unless they are the only ones): `A → ` / `AAAA → `, TTL 300, with copy buttons. (Phase 3: the HA group's virtual IPs.) +- **Live verification**: Loxy resolves the name (A and AAAA, via the system resolver and, when a provider is attached, also by reading the provider's records) and shows per family: ✓ resolves correctly, ✗ wrong IP (shows the observed addresses), ✗ missing. Runs on add, on "Check now", every 5 minutes for unverified domains and every hour for verified ones, and whenever the node addresses change. Status changes emit `domain.status.changed` events. **DNS providers** (`/domains/providers`) -- Connect provider accounts through libdns provider modules (first: the providers listed in §21 Q4; each with its own token fields, e.g. a scoped API token with zone‑read and DNS‑edit rights). The registry is a map `kind → {fields schema, constructor}` so new libdns providers are added in one file; `GET /dns_providers` also returns the available kinds and their fields. +- Connect provider accounts through libdns provider modules (the registry's kinds, §21 Q4; each with its own token fields, e.g. a scoped API token with zone‑read and DNS‑edit rights). The registry is a map `kind → {fields schema, constructor}` so new libdns providers are added in one file; `GET /dns_providers` also returns the available kinds and their fields. - Credentials are encrypted at rest with AES‑256‑GCM (`LOXY_SECRET_KEY`, §14.5), never returned by the API (only `"configured": true`), and tested on save with a `ListZones` call (status `ok|error`). -- **ACME DNS‑01 reuses the same credentials**: `internal/dns` implements lego's `challenge.Provider` (`Present`/`CleanUp`) on top of the libdns `RecordAppender`/`RecordDeleter` interfaces, writing `_acme-challenge. TXT ` (TTL 60) into the right zone and deleting exactly that record afterwards. Certificates reference a `dns_provider_id` instead of carrying their own credentials, so credentials live in one place. +- **ACME DNS‑01 reuses the same credentials**: `internal/dns` implements lego's `challenge.Provider` (`Present`/`CleanUp`) on top of the libdns `RecordAppender`/`RecordDeleter` interfaces, writing `_acme-challenge. TXT ` (TTL 60) into the right zone and deleting exactly that record afterwards. Certificates reference a `dns_provider_id` instead of carrying their own credentials. **Auto‑managed records** (Phase 2) -- Opt‑in per domain. When on and a provider is connected, Loxy creates the A/AAAA records for the FQDN pointing at the required addresses, **updates** them when they change (frontend binds edited, node addresses edited, domain moved to another frontend), and **removes** them when the domain is detached or deleted or its frontend is deleted. +- Opt‑in per domain. When on and a provider is connected, Loxy creates the A/AAAA records for the FQDN pointing at the required addresses, **updates** them when the node addresses change, and **removes** them when the domain is detached or deleted or its project is deleted. - Loxy only touches records it created (their provider IDs are tracked in `domains.managed_record_ids`); pre‑existing records with the same name are reported as a conflict and never overwritten without explicit confirmation. - Sync runs through the reconciler's event stream (debounced) and is retried with backoff; failures surface as `domain.sync.failed` events. @@ -658,25 +746,12 @@ Loxy helps operators point names at their frontends and keeps DNS in sync, **wit - List zones from all connected providers; open a zone to list and edit records: types **A, AAAA, CNAME, TXT, MX, CAA, SRV, NS**; columns type, name (relative, `@` for apex), value, TTL; add / edit / delete with confirm. - Server‑side validation per type: A/AAAA parsed with `net/netip`; CNAME/NS/MX targets are RFC 1123 hostnames; MX priority 0–65535; SRV `priority weight port target`; CAA `flags tag "value"` with tag ∈ `issue|issuewild|iodef`; TXT ≤ 255 chars per string (longer values are split); TTL 60–86400; CNAME cannot coexist with other records of the same name. Records managed by Loxy are marked and edits to them warn. -**API** (§12.2): `/domains`, `/domains/{id}/actions/{verify,issue_certificate}`, `/dns_providers`, `/dns_providers/{id}/zones`, `/dns_providers/{id}/zones/{zone}/records`. - **Acceptance** -- Adding a domain whose A record points to the frontend's bind address shows "Resolves correctly" within one check; a wrong A record shows "Wrong IP" with the observed address; for a `*` frontend the node addresses are the expected values. +- Adding a domain whose A record points to the node shows "Resolves correctly" within one check; a wrong A record shows "Wrong IP" with the observed address. +- A verified https domain gets a certificate (Pebble in CI) and `http://fqdn` answers 301 → `https://fqdn`, which reaches the project's targets, within 60 s, with at most two reloads (attach + issued). +- Two projects with different domains on the shared :443 each receive only their own traffic; an unknown Host gets 404. - With a DNS provider connected, a managed DNS‑01 certificate for `*.example.com` is issued (Pebble + a libdns test double in CI) without entering credentials again. -- With auto‑records on, changing the frontend's bind IPv4 updates the A record within 10 s; deleting the domain removes exactly the records Loxy created. - -### 8.10 System (`/system`) -- **Node** card (all‑in‑one shows the local node): status, HAProxy version and feature flags (`haproxy -vv`: QUIC, PROMEX, OpenSSL version), loxy version, HAProxy uptime, reload count and last reload result/duration, current connections, CPU/memory of both containers; warning when the bundled `haproxy -v` differs from the running one. -- **Node addresses**: list of the node's IPv4/IPv6 addresses with an optional label (`POST/DELETE /nodes/{id}/ips`); detected interface addresses are offered for one‑click adding. Used as bind‑address suggestions (§8.3) and as DNS required records for `*` binds (§8.9). Changing them never changes the HAProxy config by itself. -- **Config revisions**: list (revision, kind runtime/reload/rollback/bootstrap, result, duration, author, time, pinned), diff between any two (`CodeView`), "Roll back to this revision" (break‑glass swap + reload of the on‑disk revision, see ADR 0004 §7; the next mutation re‑applies desired state, which the UI states), pin/unpin, HAProxy output of failed applies. -- Phase 3 adds enrollment, multiple nodes and HA groups (§15). - -### 8.11 Activity (`/activity`) -- **Events** feed (server/backend health transitions, apply/reload results, cert issued/failed/expiring, domain status changes, node offline) with filters by frontend/backend/kind/severity. -- **Audit log** (who, what, when, before/after diff, IP, token) — read‑only, exportable CSV. - -### 8.12 Settings (`/settings/*`) -Users & roles · API tokens · ACME (account email, directory: Let's Encrypt production / staging / custom, EAB) · Notification channels & alert rules (event kinds, label filter on frontend/backend labels, min severity, cooldown, "Send test") · Two‑factor authentication (own TOTP; enforcement per project) · Panel (domain, port, allowlist, session lifetime) · System (hard‑stop‑after, drain timeout, reload min interval, metrics retention, live‑log sampling) · Backup (schedule, retention, optional S3‑compatible target). +- With auto‑records on, changing a node IPv4 updates the A record within 10 s; deleting the domain removes exactly the records Loxy created. --- @@ -684,7 +759,7 @@ Users & roles · API tokens · ACME (account email, directory: Let's Encrypt pro Source: `show stat` server rows (`status`, `check_status`, `check_code`, `check_duration`, `check_desc`/`last_chk`, `lastchg`). -| HAProxy server `status` | loxy server status | +| HAProxy server `status` | loxy check status | |---|---| | `UP` | healthy | | `UP n/m` (going down) | healthy (transitioning) | @@ -695,45 +770,36 @@ Source: `show stat` server rows (`status`, `check_status`, `check_code`, `check_ | `DRAIN` | draining | | `no check` | unknown | -Aggregation: health is tracked **per resolved server** (one check each, since a server belongs to one backend). Backend status = `Healthy` if all active (non‑maintenance) servers are healthy, `Degraded` if some are unhealthy but ≥ 1 is healthy, `Unhealthy` if none is healthy, `Unknown` without servers or checks. A frontend's health summary aggregates the backends it uses (default + routes). Transitions produce `server.health.changed` / `backend.health.changed` events (with check reason) used by notifications; debounce flapping with the alert‑rule cooldown. Optionally enable `option log-health-checks` and parse health transitions from the UDP log stream for sub‑second detection. +Aggregation: health is checked **per resolved target per service** (one server line per `be_`). A resolved target is `healthy` if healthy on all services, `degraded` if healthy on some, `unhealthy` if on none, `unknown` without checks. A service's health counts its servers; a project is `Healthy` if all active (non‑maintenance) checks are healthy, `Degraded` if some are unhealthy but every service has ≥ 1 healthy target, `Unhealthy` if any service has no healthy target, `Unknown` without targets or services. Transitions produce `target.health.changed` events carrying the `project_id` (stored as `events.balancer_id`), service and check reason, used by notifications; debounce flapping with the alert‑rule cooldown. Optionally enable `option log-health-checks` and parse health transitions from the UDP log stream for sub‑second detection. --- ## 10. Metrics pipeline ### 10.1 Collection -Every 1 s per node: `show stat` (CSV; parse the header line to map columns by name, never by position) and `show info`. Frontend rows (`fe_`, plus `fe__rd` summed into its frontend) → frontend metrics; backend rows (`be_`) → backend metrics; server rows → per‑server metrics and health. One `show stat` call returns everything; it is cheap even with thousands of servers. - -### 10.2 Derived series +Every 1 s per node: `show stat` (CSV; parse the header line to map columns by name, never by position) and `show info`. `be_` rows carry all traffic of a service (from its own port **and** from domains through `sys_http`/`sys_https`) → service metrics; `fe_` rows add frontend‑only counters (denied/errored requests on the port); server rows → per‑target metrics and health. Project metrics are a separate scope `pr`: the sum of the project's service backends (`be_`), so domain traffic is included. The node total (scope `fe`, id 0) also counts `sys_http`/`sys_https`. One `show stat` call returns everything; it is cheap even with thousands of servers. -**Per frontend** +### 10.2 Derived series (per service backend; project scope `pr` = sum over its services) | Series | Derivation | |---|---| -| open_connections | frontend `scur` | -| connections_per_second | Δ frontend `conn_tot` / Δt (fallback `conn_rate`) | -| requests_per_second | Δ frontend `req_tot` / Δt (http frontends only) | -| bandwidth_in / bandwidth_out | Δ frontend `bin` / Δt, Δ `bout` / Δt (bytes/s) | -| http_1xx…5xx per second | Δ frontend `hrsp_*` / Δt | -| denied / errors per second | Δ frontend `dreq` + `ereq` / Δt | - -**Per backend (and per server)** - -| Series | Derivation | -|---|---| -| sessions | backend/server `scur` | -| queue | backend `qcur` (server `qcur`) | -| response_time_ms | backend `rtime` (moving average HAProxy maintains), `ttime` | -| http_1xx…5xx per second | Δ backend `hrsp_*` / Δt | -| errors per second | Δ backend `econ` + `eresp` / Δt; retries `wretr`, redispatches `wredis` | -| bandwidth_in / bandwidth_out | Δ backend `bin` / `bout` / Δt | +| open_connections | `be_` `scur` | +| connections_per_second | Δ `be_` `stot` / Δt | +| requests_per_second | Δ `be_` `req_tot` / Δt (http/https only) | +| bandwidth_in / bandwidth_out | Δ `be_` `bin` / Δt, Δ `bout` / Δt (bytes/s) | +| sessions | `be_` / server `scur` | +| queue | `be_` `qcur` (server `qcur`) | +| response_time_ms | `be_` `rtime` (moving average HAProxy maintains), `ttime` | +| http_1xx…5xx per second | Δ `be_` `hrsp_*` / Δt | +| errors per second | Δ `be_` `econ` + `eresp` / Δt; retries `wretr`, redispatches `wredis` | +| denied per second | Δ `fe_` `dreq` + `ereq` / Δt | ### 10.3 Counter resets New workers start counters at zero. Detect a new worker via `show info` `Pid`/`Uptime_sec` change or any counter decreasing; for that sample use `delta = current` (not negative, not a spike). Connections still served by old workers during drain are not counted — accepted. HAProxy 3.3+ has an experimental `shm-stats-file` that can preserve counters across reloads; MAY be enabled behind a setting after a **VERIFY** test. ### 10.4 Storage and retention -- In memory: ring buffer of 1 s points for the last 60 min per frontend and per backend (≈ 3 600 × ~12 series × 8 B ≈ 350 KB per object at worst; keep per‑server series only for the last 15 min). -- SQLite rollups written in one transaction per minute: `metrics_1m` (scope `frontend|backend`, scope_id), with 5 m / 1 h resolutions aggregated on read or written by a later rollup job (ADR 0001 §5). Each rollup stores avg and max where meaningful. +- In memory: ring buffer of 1 s points for the last 60 min per service and per project (≈ 3 600 × ~12 series × 8 B ≈ 350 KB per object at worst; keep per‑target series only for the last 15 min). +- SQLite rollups written in one transaction per minute: `metrics_1m` (scope `fe|be|pr`, scope_id), with 5 m / 1 h resolutions aggregated on read or written by a later rollup job (ADR 0001 §5). Each rollup stores avg and max where meaningful. - Range → resolution: Live/30 min → 1 s (downsample to ≤ 1 800 points), 1 h → 10 s (from memory), 6 h/24 h → 1 m, 7 d → 5 m, 30 d → 1 h. ### 10.5 API format @@ -744,10 +810,10 @@ Columnar, uPlot‑native: "series": { "open_connections": [2210, 2215, ...], "connections_per_second": [41, 18, ...], "requests_per_second": [8820, 8790, ...], "bandwidth.in": [4.1e6, ...], "bandwidth.out": [4.3e6, ...] } } ``` -History: `GET /api/v1/frontends/{id}/metrics?type=…` and `GET /api/v1/backends/{id}/metrics?type=sessions,queue,response_time_ms,…`. Live: `GET /api/v1/frontends/{id}/metrics/stream` (SSE) pushes one compact point per second. +History: `GET /api/v1/projects/{id}/metrics?type=…` returns the project (`pr`) series; `&service_id=` returns that service's backend (`be_`) series. Live: `GET /api/v1/projects/{id}/metrics/stream` (SSE) pushes one compact point per second. ### 10.6 Optional exports -- `sys_stats` frontend on `127.0.0.1:8405`: HAProxy's built‑in Prometheus exporter at `/metrics` (render only if `haproxy -vv` shows PROMEX) and the native stats page at `/stats`. Because names are `fe_`/`be_`, loxy's own `/metrics` exposes an info series mapping ids to display names and labels. +- `sys_stats` frontend on `127.0.0.1:8405`: HAProxy's built‑in Prometheus exporter at `/metrics` (render only if `haproxy -vv` shows PROMEX) and the native stats page at `/stats`. Because names are `fe_`/`be_`, loxy's own `/metrics` exposes an info series mapping service ids to project ids, names and labels. - loxy's own `/metrics` (controller health: reload durations, apply errors, poll latency, ACME results). --- @@ -755,14 +821,14 @@ History: `GET /api/v1/frontends/{id}/metrics?type=…` and `GET /api/v1/backends ## 11. Certificates and ACME ### 11.1 Storage -Certificates are project‑level and reusable across frontends. Private keys are encrypted at rest (AES‑256‑GCM, key from `LOXY_SECRET_KEY`). The rendered revision necessarily contains plaintext PEMs (HAProxy reads them); the config volume is mode 0750, files 0640, group `loxy` (§16). +Certificates belong to the single tenancy project and are reusable across all load‑balancer projects, services and domains. Private keys are encrypted at rest (AES‑256‑GCM, key from `LOXY_SECRET_KEY`). The rendered revision necessarily contains plaintext PEMs (HAProxy reads them); the config volume is mode 0750, files 0640, group `loxy` (§16). ### 11.2 Managed certificates with lego - One ACME account per directory URL (`acme_accounts`), created lazily with the configured email (and EAB when required by the CA). -- **HTTP‑01**: implement lego's `challenge.Provider` (`Present(ctx, domain, token, keyAuth)` / `CleanUp(...)`). `Present` pushes `add map virt@acme_http01 ` to **every node** that hosts a frontend serving that domain, waits for acknowledgement, then returns; `CleanUp` deletes the entry. HAProxy answers the challenge itself with the `http-request return … map(virt@acme_http01)` rule in `fe__rd` / `sys_acme_fe` (§7.4) — no extra hop, no port juggling. -- **DNS‑01**: a lego `challenge.Provider` implemented on top of the libdns provider of a connected DNS provider account (§8.9), so DNS credentials are stored once (encrypted) and shared by record management and ACME; required for wildcard domains. +- **HTTP‑01**: implement lego's `challenge.Provider` (`Present(ctx, domain, token, keyAuth)` / `CleanUp(...)`). `Present` pushes `add map virt@acme_http01 ` to **every node** that serves the domain, waits for acknowledgement, then returns; `CleanUp` deletes the entry. HAProxy answers the challenge itself with the `http-request return … map(virt@acme_http01)` rule in `sys_http` (§7.4) — no extra hop, no port juggling. The virtual map only exists once a loaded config references it, so issuance waits until `sys_http` is live. +- **DNS‑01**: a lego `challenge.Provider` implemented on top of the libdns provider of a connected DNS provider account (§8.12), so DNS credentials are stored once (encrypted) and shared by record management and ACME; required for wildcard domains. - Key type default ECDSA P‑256; RSA optional. -- Issuance runs as an async action; failures record the lego error and retry with backoff (1 m, 5 m, 30 m, 2 h, then every 6 h) without blocking the frontend (placeholder cert keeps serving). +- Issuance runs as an async action; failures record the lego error and retry with backoff (1 m, 5 m, 30 m, 2 h, then every 6 h) without blocking the service (placeholder cert keeps serving) or the domain (stays on http). - **Renewal** scheduler: honour ACME Renewal Information (ARI) when the CA offers it; otherwise renew at 2/3 of the lifetime (≈ 30 days before expiry for 90‑day certs). Jitter start times. On success: new revision with the new PEM + runtime `set ssl cert` / `commit ssl cert` (no reload). - Use Let's Encrypt **staging** automatically in dev and CI (Pebble in tests). - HAProxy's native ACME (`acme` section, 3.2+) MAY replace lego later once it's no longer experimental; keep the `CertIssuer` interface abstract so that's a swap, not a rewrite. @@ -777,11 +843,11 @@ Same runtime hot‑swap path when replaced. Expiry warnings at 21/7/1 days. The contract is `api/openapi.yaml` (OpenAPI 3.0.3); this section summarises it. When they disagree, fix one of them in the same PR. ### 12.1 Conventions -- Base path `/api/v1`, JSON, `snake_case`, plural resource names (`frontends`, `backends`, `servers`, `hosts`, `certificates`, `domains`, `actions`, `metrics`). +- Base path `/api/v1`, JSON, `snake_case`, plural resource names (`projects`, `services`, `targets`, `hosts`, `certificates`, `domains`, `actions`, `metrics`). - Auth: session cookie (UI) or `Authorization: Bearer loxy__` (API tokens). - Pagination: `?page=&per_page=` (max 100) with `meta.pagination`. Filtering: `label_selector=`, `name=`. Sorting: `sort=name:asc`. - Optimistic concurrency: every mutable resource has `version`; PUT/PATCH honour `If-Match` (ADR 0001 §6); mismatch → `409 conflict`. -- Errors: `{ "error": { "code": "invalid_input", "message": "...", "details": { "fields": [{ "name": "binds[0].port", "messages": ["already used by frontend web (#3)"] }] } } }`. Codes: `invalid_input`, `conflict` (incl. in‑use deletes), `protected`, `not_found`, `forbidden`, `unauthorized`, `rate_limited`, `apply_failed`. +- Errors: `{ "error": { "code": "invalid_input", "message": "...", "details": { "fields": [{ "name": "service.listen_port", "messages": ["port 20001 is used by project shop (#1)"] }] } } }`. Codes: `invalid_input`, `conflict` (incl. port/domain in use and in‑use deletes), `protected`, `not_found`, `forbidden`, `unauthorized`, `rate_limited`, `apply_failed`. - Idempotency: POSTs accept `Idempotency-Key`. ### 12.2 Endpoints @@ -789,25 +855,23 @@ The contract is `api/openapi.yaml` (OpenAPI 3.0.3); this section summarises it. | Method & path | Purpose | |---|---| | `POST /auth/login`, `POST /auth/logout`, `GET /auth/me`, `POST /auth/password`, `POST /auth/totp/{setup,enable,disable}` | Session auth, password change, 2FA | -| `GET /overview` | Dashboard counts: frontends, backends, servers, certificates, node, traffic | -| `GET/POST /frontends`, `GET/PUT/PATCH/DELETE /frontends/{id}` | Frontend CRUD (PUT: full object incl. binds, certificates, routes; PATCH: name, description, labels, enabled, maintenance) | -| `GET /frontends/{id}/config` | Rendered HAProxy sections for this frontend and the backends it uses (read‑only) | -| `GET /frontends/{id}/metrics?type=…&start=&end=&step=`, `GET /frontends/{id}/metrics/stream` | History (§10.5), SSE live points | -| `GET /frontends/{id}/logs/stream` | SSE live logs (when enabled) | -| `GET/POST /backends`, `GET/PUT/DELETE /backends/{id}` | Backend CRUD (DELETE → `409` while in use) | -| `GET /backends/{id}/status` | Health: summary + resolved servers with check details | -| `GET /backends/{id}/metrics?type=…` | Backend history | -| `GET/POST /backends/{id}/servers`, `PATCH/DELETE /backends/{id}/servers/{sid}` | Servers (POST takes a list; PATCH: weight, admin_state, backup; DELETE is graceful) | +| `GET /overview` | Dashboard counts: projects, targets, certificates, node, traffic | +| `GET/POST /projects`, `GET/PATCH/DELETE /projects/{id}` | Project CRUD. POST takes `name` plus optional `targets[]` and `service` (default: http, automatic port, destination 80) in one transaction; PATCH: name, description, algorithm, labels, maintenance | +| `GET /projects/{id}/config` | Rendered HAProxy sections of this project (read‑only, text) | +| `GET /projects/{id}/metrics?type=…&start=&end=&step=&service_id=`, `GET /projects/{id}/metrics/stream` | History (§10.5), SSE live points | +| `GET /projects/{id}/logs/stream` | SSE live logs (when enabled) | +| `GET/POST /projects/{id}/services`, `PUT/DELETE /projects/{id}/services/{sid}` | Services (omit `listen_port` for automatic allocation; DELETE → `409` when it is the last http/https service with domains) | +| `GET/POST /projects/{id}/targets`, `PATCH/DELETE /projects/{id}/targets/{tid}` | Targets (POST takes a list; PATCH: weight, admin_state, backup; DELETE is graceful) with resolved slots and per‑service checks | | `GET/POST /hosts`, `PATCH/DELETE /hosts/{id}` | Inventory | | `GET /label_selector/preview?selector=` | Hosts matched by a selector | | `GET/POST /certificates`, `GET/PATCH/DELETE /certificates/{id}`, `POST /certificates/{id}/actions/renew` | Certificates | -| `GET/POST /domains`, `GET/PATCH/DELETE /domains/{id}`, `POST /domains/{id}/actions/verify`, `POST /domains/{id}/actions/issue_certificate` | Domains (§8.9) | -| `GET/POST /dns_providers`, `DELETE /dns_providers/{id}`, `GET /dns_providers/{id}/zones`, `GET/POST/DELETE /dns_providers/{id}/zones/{zone}/records` | DNS providers (incl. available kinds) and zone editor (§8.9; records Phase 2) | +| `GET/POST /domains`, `GET/PATCH/DELETE /domains/{id}`, `POST /domains/{id}/actions/verify`, `POST /domains/{id}/actions/issue_certificate` | Domains (`project_id`, `service_id`, `https`; filter `?project_id=`; §8.11–8.12) | +| `GET/POST /dns_providers`, `DELETE /dns_providers/{id}`, `GET /dns_providers/{id}/zones`, `GET/POST/DELETE /dns_providers/{id}/zones/{zone}/records` | DNS providers (incl. available kinds) and zone editor (§8.12; records Phase 2) | | `GET /nodes`, `PATCH /nodes/{id}`, `POST /nodes/{id}/ips`, `DELETE /nodes/{id}/ips/{ipid}` | Node info and node addresses | | `GET /nodes/{id}/revisions`, `GET /nodes/{id}/revisions/{rev}/diff?against=`, `POST /nodes/{id}/revisions/{rev}/actions/rollback` | Config revisions | | `POST /apply/retry` | Re‑run the reconciler after an apply error | | `GET /actions`, `GET /actions/{id}` | Async action status | -| `GET /events` (SSE), `GET /events/history` | Event stream & history | +| `GET /events` (SSE), `GET /events/history` | Event stream & history (filter `?project_id=`) | | `GET /audit` | Audit log | | `GET/POST/DELETE /users`, `/api_tokens`, `/notification_channels` (+ `POST …/{id}/actions/test`), `/alert_rules`, `GET/PATCH /settings` | Admin | | `GET /healthz`, `GET /readyz`, `GET /metrics` | Ops (no auth on healthz; readyz checks DB + HAProxy socket) | @@ -815,34 +879,55 @@ The contract is `api/openapi.yaml` (OpenAPI 3.0.3); this section summarises it. ### 12.3 Example resources ```json { - "frontend": { - "id": 3, "name": "web", "description": "public site", "version": 14, - "mode": "http", "tls": true, "http_redirect": true, "certificates": [12], - "binds": [ { "address": "*", "port": 443 } ], - "default_backend_id": 4, - "routes": [ - { "id": 1, "hosts": ["app.example.com"], "path_prefix": "/api", "backend_id": 5 }, - { "id": 2, "hosts": ["*.static.example.com"], "path_prefix": "", "backend_id": 6 } + "project": { + "id": 1, "name": "shop", "description": "web shop", "version": 9, + "algorithm": "roundrobin", "maintenance": false, "labels": { "env": "prod" }, + "status": "ready", "last_error": null, "created_at": 1760412300000, + "endpoints": [ + { "kind": "port", "url": "http://203.0.113.10:20001", "service_id": 1 }, + { "kind": "port", "url": "https://203.0.113.10:20002", "service_id": 2 }, + { "kind": "domain", "url": "https://shop.example.com", "service_id": 1, "domain_id": 4 } ], - "timeout_client_s": 50, "maxconn": 10000, "forwarded_headers": true, - "enabled": true, "maintenance": false, "labels": { "env": "prod" }, - "status": "active", "last_error": null, "created_at": 1760412300000, - "stats": { "open_connections": 2210, "requests_per_second": 8820, "bytes_out_per_second": 4.3e6 }, - "health": { "status": "healthy", "healthy": 4, "total": 4, "messages": [] } + "counts": { "targets": 3, "services": 2, "domains": 2 }, + "health": { "status": "healthy", "healthy": 4, "total": 4, "messages": [] }, + "stats": { "open_connections": 2210, "requests_per_second": 8820, "bytes_out_per_second": 4.3e6 } } } ``` ```json { - "backend": { - "id": 4, "name": "app", "version": 7, "mode": "http", "algorithm": "roundrobin", "port": 9000, - "health_check": { "protocol": "http", "port": 9000, "interval_s": 15, "timeout_s": 13, "retries": 3, - "http": { "domain": "app.example.com", "path": "/health", "response": "OK", "status_codes": ["2??", "3??"], "tls_verify": false } }, + "service": { + "id": 1, "project_id": 1, "protocol": "http", "listen_port": 20001, "destination_port": 8080, + "proxy_protocol": "none", "timeout_s": 60, "certificates": [], "sticky_sessions": false, "cookie_name": "LOXYSTICKY", "cookie_lifetime_s": 300, - "proxy_protocol": "none", "timeout_server_s": 50, "labels": { "env": "prod" }, - "used_by": [3], "servers_count": 3, - "health": { "status": "healthy", "healthy": 3, "total": 3, "messages": [] }, - "stats": { "sessions": 180, "queue": 0, "response_time_ms": 12 } + "health_check": { "protocol": "http", "port": null, "interval_s": 15, "timeout_s": 10, "retries": 3, + "http": { "domain": "shop.example.com", "path": "/health", "response": "OK", "status_codes": ["2??", "3??"], "tls_verify": false } }, + "health": { "status": "healthy", "healthy": 4, "total": 4, "messages": [] }, + "domains": ["shop.example.com", "beta.example.com"] + } +} +``` +```json +{ + "target": { + "id": 102, "project_id": 1, "type": "label_selector", "label_selector": "role=web", + "port": null, "use_private_ip": false, "weight": 100, "backup": false, "admin_state": "ready", "removing": false, + "resolved": [ + { "slot_id": 102, "host_id": 7, "name": "web-1", "address": "10.0.0.12", "status": "healthy", + "checks": [ { "service_id": 1, "status": "healthy", "check": "L7OK/200 in 3 ms" }, + { "service_id": 2, "status": "healthy", "check": "L4OK in 1 ms" } ] } + ] + } +} +``` +```json +{ + "domain": { + "id": 4, "fqdn": "shop.example.com", "project_id": 1, "service_id": null, "https": true, + "dns_provider_id": null, "auto_records": false, "status": "ok", + "required_records": [ { "type": "A", "name": "shop.example.com", "value": "203.0.113.10", "ttl": 300 } ], + "observed": { "a": ["203.0.113.10"], "aaaa": [] }, "certificate_id": 21, + "url": "https://shop.example.com" } } ``` @@ -851,7 +936,7 @@ The contract is `api/openapi.yaml` (OpenAPI 3.0.3); this section summarises it. Long or multi‑step operations return `202` with `{ "action": { "id", "command", "status": "running|success|error", "progress": 0-100, "resources": [...], "error": null, "started", "finished" } }`. The UI follows actions over the `/events` SSE stream (no polling). ### 12.5 SSE event kinds -`frontend.updated`, `backend.updated`, `apply.started|succeeded|failed`, `server.health.changed`, `backend.health.changed`, `certificate.updated`, `domain.status.changed`, `domain.sync.failed`, `action.updated`, `node.status.changed`, `metrics.tick` (only on metric streams). Reconnect with `Last-Event-ID`; server keeps a 2‑minute replay buffer. +`project.updated`, `apply.started|succeeded|failed`, `target.health.changed` (with `project_id`), `certificate.updated`, `domain.status.changed`, `domain.sync.failed`, `action.updated`, `node.status.changed`, `metrics.tick` (only on metric streams). Reconnect with `Last-Event-ID`; server keeps a 2‑minute replay buffer. ### 12.6 Codegen `make gen` runs `oapi-codegen` (Go server interfaces + models) and `openapi-typescript` (web types). CI fails if generated files are stale. @@ -861,10 +946,11 @@ Long or multi‑step operations return `202` with `{ "action": { "id", "command" ## 13. Frontend implementation - SvelteKit SPA (`adapter-static`, fallback `index.html`, `ssr = false`), served by loxy from `go:embed` with precompressed `.br`/`.gz` assets, `Cache-Control: public, max-age=31536000, immutable` for hashed files and `no-cache` for `index.html`. -- Routes mirror the tabs so every tab is linkable (`/frontends/3/routing`, `/backends/4/servers`). +- Routes mirror the tabs so every tab is linkable (`/projects/1/targets`, `/projects/1/domains`). - Data: one session‑scoped snapshot store fed by REST on navigation and kept fresh by the single `/events` SSE connection; mutations are optimistic where safe (labels, names) and action‑driven otherwise. -- Charts: `Chart.svelte` wraps uPlot, dynamically imported only on the Graphs tabs (sparklines on the Dashboard and lists use a 1 KB SVG polyline component, not uPlot). -- Components: `StatBar`, `StatCard`, `Donut`, `HealthBadge`, `Tabs`, `Table` (virtualised above 200 rows), `Modal`, `Dropdown`, `Toast`, `LabelChips`, `LabelSelectorInput`, `PortField`, `BindsEditor`, `RouteTable` (drag to reorder), `BackendSelect`, `CertPicker`, `HostPicker`, `CopyText`, `ConfirmTyped`, `CodeView` (read‑only config/diff viewer, no heavy editor). +- Charts: `Chart.svelte` wraps uPlot, dynamically imported only on the Graphs tab (sparklines on the Dashboard and lists use a 1 KB SVG polyline component, not uPlot). +- Components: `StatBar`, `StatCard`, `Donut`, `HealthBadge`, `Tabs`, `Table` (virtualised above 200 rows), `Modal`, `Dropdown`, `Toast`, `LabelChips`, `LabelSelectorInput`, `PortField` (automatic / custom), `EndpointList`, `TargetAdder`, `ServiceForm`, `DomainForm`, `RequiredRecords`, `CertPicker`, `HostPicker`, `CopyText`, `ConfirmTyped`, `CodeView` (read‑only config/diff viewer, no heavy editor). +- Wording: the UI says project, target, service, domain, port — never frontend/backend/server (except in the read‑only Config view, which shows HAProxy text). - Design tokens in `tokens.css`; dark theme default, light theme via `prefers-color-scheme` + toggle; accent colour configurable in Settings: ```css :root { --bg:#141414; --surface:#1f1f1f; --surface-2:#272727; --border:#343434; @@ -884,11 +970,11 @@ Long or multi‑step operations return `202` with `{ "action": { "id", "command" - First start: create the owner account with a random password printed once to the logs (`loxy admin reset-password` regenerates it). - Passwords: argon2id (memory 64 MB, iterations 3, parallelism 2), minimum length 12. - Sessions: 256‑bit random id, stored hashed; cookie `__Host-loxy_session; Secure; HttpOnly; SameSite=Strict; Path=/`; idle timeout 12 h, absolute 7 d; rotate on login. -- Optional TOTP 2FA, enforceable per project. +- Optional TOTP 2FA, enforceable for all users. - Login rate limit: 5 attempts / 15 min per IP + per account, exponential backoff. ### 14.2 Authorization -Roles: **owner** (everything incl. users, secrets), **admin** (all resources, settings except owner transfer), **operator** (frontends, backends, servers, hosts, certificates, domains; no users/settings), **viewer** (read‑only). API tokens carry a role and optional expiry; shown once at creation; stored as SHA‑256 hash with a visible prefix. +Roles: **owner** (everything incl. users, secrets), **admin** (all resources, settings except owner transfer), **operator** (projects, services, targets, hosts, certificates, domains; no users/settings), **viewer** (read‑only). API tokens carry a role and optional expiry; shown once at creation; stored as SHA‑256 hash with a visible prefix. ### 14.3 Web hardening CSRF: SameSite=Strict cookies **and** a required `X-LOXY-CSRF` header token on mutating requests. Headers: strict CSP (`default-src 'self'`), `X-Content-Type-Options: nosniff`, `Referrer-Policy: same-origin`, `Frame-Options: DENY`, HSTS on the panel. Request body limit 1 MB (5 MB for cert upload). @@ -901,17 +987,16 @@ User values end up inside `haproxy.cfg`. **Never** interpolate raw text. Validat | Resource display names | 1–64 chars, any printable; **never rendered** into config | | Label keys | `^([a-z0-9]([-a-z0-9]*[a-z0-9])?(\.[a-z0-9]([-a-z0-9]*[a-z0-9])?)*/)?[a-zA-Z0-9]([-a-zA-Z0-9_.]{0,61}[a-zA-Z0-9])?$` | | Label values | `^([a-zA-Z0-9]([-a-zA-Z0-9_.]{0,61}[a-zA-Z0-9])?)?$` | -| Domains (health check Host, cert SANs, route hosts) | RFC 1123 hostname, lowercase, ≤ 253; `*.` prefix only for DNS‑01 certificates and route hosts | -| Health check path, route path prefix | `^/[A-Za-z0-9._~!&()*+,;=:@%/?-]{0,254}$` (no spaces, quotes, `#`, `$`, `\`) | +| Domains (domain FQDNs, health check Host, cert SANs) | RFC 1123 hostname, lowercase, ≤ 253; `*.` prefix only for domains/certificates using DNS‑01 | +| Health check path | `^/[A-Za-z0-9._~!&()*+,;=:@%/?-]{0,254}$` (no spaces, quotes, `#`, `$`, `\`) | | Health check response string | printable ASCII except `'"\#$`, 1–128 chars | | Cookie name | `^[A-Za-z0-9_-]{1,64}$` | -| Ports | integers 1–65535 | -| Bind address | `*` or an IP literal parsed with `net/netip` (zone ids rejected) | -| IPs / CIDRs | parsed with `net/netip`; servers reject loopback, link‑local, multicast, unspecified | +| Ports | integers 1–65535; listen ports additionally not reserved (§8) | +| IPs / CIDRs, target `ip[:port]` | parsed with `net/netip` (`netip.ParseAddrPort` for the port form, IPv6 as `[addr]:port`); zone ids rejected; targets reject loopback (unless dev), link‑local, multicast, unspecified | | Status code items | `^[1-5][0-9?]{2}$`, ≤ 20 items | | Label selector | parsed by our own small parser into an AST; never passed to HAProxy | -Tests: a Go fuzz target generates random valid models and asserts the rendered config (a) passes `haproxy -c`, (b) contains no line not produced by a known template block, (c) never contains a newline inside a value. +Tests: a Go fuzz target generates random valid models (projects, services, targets, domains) and asserts the rendered config (a) passes `haproxy -c`, (b) contains no line not produced by a known template block, (c) never contains a newline inside a value. ### 14.5 Secrets at rest `LOXY_SECRET_KEY` (32 random bytes, base64) in `.env` (mode 0600). Encrypts private keys, DNS credentials, TOTP secrets, notification credentials, ACME account keys. Key rotation command: `loxy admin rotate-secret`. @@ -931,13 +1016,13 @@ loxy listens on `127.0.0.1:9090` only. Public access goes through the `sys_panel ### 15.1 Panel ↔ agent - `loxy agent` runs next to HAProxy on each HAProxy node (same compose as all‑in‑one minus the panel). It dials **out** to the panel over HTTPS WebSocket, so nodes need no inbound management port. - Enrollment: panel creates a node and shows a one‑time join token; `loxy agent join --panel https://panel.example.com --token …` stores a long‑lived node credential and pins the panel's TLS certificate fingerprint. -- Frontends are placed on a node or an HA group; backends are rendered on every node that has a frontend using them. +- Projects are placed on a node or an HA group; their services, targets and domains are rendered on those nodes. Listen ports stay unique installation‑wide (simple, and lets a project move between nodes). - Messages: panel → agent `DesiredState{version, haproxy config inputs, certs, acme_tokens}`; agent → panel `Applied{version, result, output}`, `StatsBatch` (1 s samples, sent every 1 s), `HealthDelta`, `NodeInfo`. - The agent persists the last applied desired state locally and keeps running autonomously if the panel is down (HAProxy keeps serving; ACME HTTP‑01 tokens pushed before disconnect stay valid). ### 15.2 HA pairs (active/passive, generic) -- Both nodes of an HA group receive identical desired state for the group's frontends; both may bind the group's **virtual IPs** at all times (`net.ipv4.ip_nonlocal_bind=1` and `net.ipv6.ip_nonlocal_bind=1` on the host), so failover needs no reload. -- Leader election and address movement: **keepalived** VRRP (in **unicast** mode between the nodes' private addresses by default; multicast optional) owns the virtual IP(s) and moves them to the backup node; a `track_script` checks HAProxy (`/healthz` of the agent + a TCP probe of a frontend port). `notify_*` scripts report state changes to loxy (`loxy ha notify `), which records events. No provider API is called; environments that need provider‑specific address routing can hook their own script into `notify_master`. +- Both nodes of an HA group receive identical desired state for the group's projects; both may bind the group's **virtual IPs** at all times (`net.ipv4.ip_nonlocal_bind=1` and `net.ipv6.ip_nonlocal_bind=1` on the host), so failover needs no reload. Domains of projects on an HA group require the virtual IPs as their DNS records. +- Leader election and address movement: **keepalived** VRRP (in **unicast** mode between the nodes' private addresses by default; multicast optional) owns the virtual IP(s) and moves them to the backup node; a `track_script` checks HAProxy (`/healthz` of the agent + a TCP probe of a service port). `notify_*` scripts report state changes to loxy (`loxy ha notify `), which records events. No provider API is called; environments that need provider‑specific address routing can hook their own script into `notify_master`. - Loxy renders the keepalived config from the HA group (virtual IPs, peer addresses, priorities, auth) into `deploy/keepalived/` templates; keepalived runs as its own container (`network_mode: host`, `NET_ADMIN`) — an ADR is required before adding it. - Stick tables (sticky/rate‑limit state) are synced with an HAProxy `peers` section between the two nodes. - UI: HA group page on System shows active/backup, virtual IPs, last failover, "Fail over now" button (lowers the local VRRP priority temporarily). @@ -961,7 +1046,9 @@ net.ipv4.tcp_tw_reuse = 1 fs.file-max = 2097152 fs.nr_open = 2097152 ``` -(If `nf_conntrack` is loaded, raise `net.netfilter.nf_conntrack_max` accordingly or exclude frontend ports from tracking.) +(If `nf_conntrack` is loaded, raise `net.netfilter.nf_conntrack_max` accordingly or exclude service ports from tracking.) + +Note: the ephemeral port range (`1024 65000`) overlaps the automatic service ports (20000+). A listening socket always wins over later outgoing connections, but an outgoing connection that already holds a port blocks a new listener on it until it closes; the API reports the bind error from `haproxy -c`/reload as `apply_failed` with the HAProxy text. Operators who want strict separation MAY reserve the automatic range with `net.ipv4.ip_local_reserved_ports = 20000-29999`. ### 16.2 Dockerfile (`deploy/docker/Dockerfile`) ```dockerfile @@ -1050,8 +1137,8 @@ services: | `HAPROXY_VERSION` | `3.4.6` | Pinned HAProxy image version (same for both containers) | | `LOXY_VERSION` | — | loxy release | | `LOXY_SECRET_KEY` | generated by installer | 32 bytes base64, encrypts secrets at rest | -| `LOXY_LISTEN` | `127.0.0.1:9090` | Internal HTTP listener | -| `LOXY_PANEL_PORT` | `8443` | Port of the `sys_panel` frontend | +| `LOXY_LISTEN` | `127.0.0.1:9090` | Internal HTTP listener (its port is reserved, §8) | +| `LOXY_PANEL_PORT` | `8443` | Port of the `sys_panel` frontend (must not be 80/443) | | `LOXY_PANEL_DOMAIN` | empty | If set, managed cert for the panel | | `LOXY_PANEL_ALLOW` | empty | Comma‑separated CIDRs allowed to reach the panel | | `LOXY_ACME_EMAIL` | empty | ACME account contact | @@ -1059,11 +1146,12 @@ services: | `LOXY_DATA_DIR` | `/var/lib/loxy` | DB, backups | | `LOXY_HAPROXY_CONF_DIR` | `/usr/local/etc/haproxy` | Revision tree | | `LOXY_HAPROXY_RUN_DIR` | `/run/haproxy` | Sockets | +| `LOXY_IPV6` | `auto` | IPv6 listeners for services and domains: `auto` (probe), `on`, `off` (ADR 0004 §9) | | `LOXY_STATS_INTERVAL` | `1s` | Poll interval | | `LOXY_LOG_LEVEL` | `info` | | ### 16.5 Installer (`deploy/install.sh`) -1. Check root, Docker ≥ 24, Compose v2, free ports (8443, 80/443 if they will be used by frontends). +1. Check root, Docker ≥ 24, Compose v2, free ports (8443; 80/443 if domains will be used). 2. Write `/etc/sysctl.d/99-loxy.conf`, `sysctl --system`. 3. Create `/opt/loxy/{data/haproxy-conf,data/haproxy-state,data/loxy,run/haproxy}`; `chown -R 1500:1500`; `chmod 0750` on data dirs and `chmod 0770` on `run/haproxy` (the capability‑dropped HAProxy master creates its sockets there through group 1500). 4. Write `compose.yml` and `.env` (generate `LOXY_SECRET_KEY`, chmod 0600). @@ -1071,19 +1159,19 @@ services: 6. Print panel URL (`https://:8443`) and the one‑time owner password (`docker compose logs loxy | grep 'initial password'`), remind to restrict access (allowlist/firewall). ### 16.6 Operations -- **Upgrade loxy:** `docker compose pull loxy && docker compose up -d loxy` — HAProxy keeps serving, no traffic impact. DB migrations run at start after an automatic pre‑migration backup. +- **Upgrade loxy:** `docker compose pull loxy && docker compose up -d loxy` — HAProxy keeps serving, no traffic impact. DB migrations run at start after an automatic pre‑migration backup. Upgrading a database from the frontend/backend version runs `0002_projects.sql` (§6): frontend/backend definitions are dropped, so recreate them as projects; domains are kept detached and can be attached to the new projects. - **Upgrade HAProxy:** bump `HAPROXY_VERSION` (and the matching loxy image tag) → `docker compose up -d`. Replacing the HAProxy container restarts it (brief interruption) — schedule it, or in an HA pair fail over first, upgrade passive, fail back. - **Backup:** `loxy backup` (SQLite online backup via `VACUUM INTO` + current live revision tarball), nightly by default, keep 14; optional upload to S3‑compatible storage. **Restore:** `loxy restore ` then the reconciler re‑applies. - **Break‑glass:** `docker compose exec loxy loxy admin reset-password`; `loxy render --dry-run` prints the config that would be applied; `loxy admin rollback --revision N`. -- **Firewall:** allow frontend bind ports publicly; restrict 8443 to admin IPs; never expose 9090, 8405 or 5514. +- **Firewall:** open the project ports you use, plus 80/443 when using domains; restrict 8443 to admin IPs; never expose 9090, 8405 or 5514. --- ## 17. Development workflow -- `make dev` → `docker compose -f deploy/compose/compose.dev.yml up` with: HAProxy (same image), three `traefik/whoami` servers on different ports (one can be toggled unhealthy via an env flag), **Pebble** (Let's Encrypt's ACME test server) + its challenge DNS, and loxy in dev mode with live reload; Vite dev server proxies `/api` to loxy. +- `make dev` → `docker compose -f deploy/compose/compose.dev.yml up` with: HAProxy (same image), three `traefik/whoami` targets on different ports (one can be toggled unhealthy via an env flag), **Pebble** (Let's Encrypt's ACME test server) + its challenge DNS, and loxy in dev mode with live reload; Vite dev server proxies `/api` to loxy. - `make gen` (OpenAPI → Go + TS, sqlc), `make lint` (golangci‑lint, `svelte-check`, eslint, prettier), `make test`, `make test-integration`, `make e2e`, `make load`, `make build`, `make image`. -- `make seed` creates a demo project: an http, a TLS (redirect + routes) and a tcp frontend, 3 backends, 6 servers (IP, host and label selector), 3 hosts, an uploaded cert, a managed cert against Pebble, and synthetic traffic (`oha` loop) so graphs have data. +- `make seed` creates demo projects: "shop" (http service on an automatic port + an https service, 3 targets of all three types, two domains), "db" (tcp service), 3 hosts, an uploaded cert, a managed cert against Pebble, and synthetic traffic (`oha` loop) so graphs have data. - Conventional commits; CI on every PR: gen‑check, lint, unit, integration (real HAProxy via testcontainers), e2e (headless), bundle‑size check, image build. --- @@ -1092,13 +1180,14 @@ services: | Layer | What | |---|---| -| Unit | Validation rules (incl. cross‑frontend port collisions with `*`, redirect :80 and `sys_acme_fe` :80; frontend/backend mode match; in‑use backend delete refusal); status‑code conversion; label‑selector parser; route → ACL rendering; metric delta/reset logic; health aggregation; reconciler classification table (§7.6) | -| Renderer golden tests | Fixture models → expected `haproxy.cfg` (byte‑exact): tls frontend with redirect and routes, tcp pair, `*` binds, maintenance, disabled frontend, `sys_acme_fe`; every golden passes `haproxy -c` in CI | +| Unit | Validation rules (listen‑port uniqueness across projects, reserved ports, automatic port allocation incl. gaps and concurrent creates, domain uniqueness, default‑service resolution, refusal to remove the last http/https service with domains); status‑code conversion; label‑selector parser; target `ip[:port]` parsing; Host‑condition rendering (exact before wildcard); metric delta/reset logic and project = Σ services; health aggregation (per target per service); reconciler classification table (§7.6) | +| Migration | `0002_projects.sql` on a fixture database of the frontend/backend version: tenancy `projects`, hosts, certificates, users, settings, DNS providers survive; domains are copied detached; events keep their rows; frontend/backend tables are gone and `metrics_1m` is empty; a database with a `load_balancers` table is refused with the documented error | +| Renderer golden tests | Fixture models → expected `haproxy.cfg` (byte‑exact): the §7.4 reference, a project without targets, maintenance (http in `fe_` + `be_` only with domains, tcp in `fe_`), IPv6 off, `sys_http` only (no certificate issued yet), exact vs. wildcard domains of two projects, HTTP‑01 cert without domains; every golden passes `haproxy -c` in CI | | Fuzz | Random valid models → config passes `haproxy -c`, no injected lines (§14.4) | -| Integration (real HAProxy 3.4) | Runtime client matchers for every command; reload success and failure paths (`Success=0` keeps old workers, symlink restored); server‑state survives reload; dynamic server add/remove/drain; host/path routing reaches the right backend; maintenance via map (http 503, tcp reject); cert hot‑swap; ACME HTTP‑01 + DNS‑01 against Pebble; metrics deltas across reloads; **all VERIFY items**: `-S` socket options, double `http-check expect` semantics, `@system-ca`, `del ssl cert`, minimal capability set, `chroot auto`, prompt mode, `shm-stats-file` (if enabled) | -| E2E (Playwright) | Login, create backend + servers of all three types, create http / TLS (with redirect) / tcp frontends, add routing rules and reorder them, toggle sticky/proxy protocol, drain/maint/weight a server, maintenance mode, delete refused for an in‑use backend, delete frontend, certificate upload/managed flow, domain verification, graphs render, rollback | -| Load / zero‑downtime | `oha` at 2 000 rps against a frontend while (a) 100 reloads, (b) 50 server add/removes, (c) cert renewals — **0 non‑2xx/connection errors** allowed | -| Resource | loxy RSS and CPU under the §1.4 dataset (50 frontends / 200 backends / 1 000 servers); UI bundle size budget | +| Integration (real HAProxy 3.4) | Runtime client matchers for every command; reload success and failure paths (`Success=0` keeps old workers, symlink restored); server‑state survives reload; dynamic target add/remove/drain across two services; two projects with domains on shared :80/:443 route by Host, unknown Host → 404; 301 only after issuance; maintenance via map (http 503 on port and domain, tcp reject); cert hot‑swap; ACME HTTP‑01 + DNS‑01 against Pebble; metrics deltas across reloads; **all VERIFY items**: `-S` socket options, double `http-check expect` semantics, `@system-ca`, `del ssl cert`, minimal capability set, `http-request return` with `opt@` map in a backend, `chroot auto`, prompt mode, `shm-stats-file` (if enabled) | +| E2E (Playwright) | Login, create project with the wizard (automatic port), add targets of all three types, reach the project on its port, add https and tcp services, port conflict error, toggle sticky/PROXY protocol, drain/maint/weight a target, maintenance mode, attach a domain → verification → certificate → 301, detach domain, delete project, certificate upload/managed flow, graphs render (project and per service), rollback | +| Load / zero‑downtime | `oha` at 2 000 rps against a service port and a domain while (a) 100 reloads, (b) 50 target add/removes, (c) cert renewals — **0 non‑2xx/connection errors** allowed | +| Resource | loxy RSS and CPU under the §1.4 dataset (50 projects / 200 services / 1 000 targets); UI bundle size budget | --- @@ -1108,49 +1197,51 @@ services: Repo skeleton, Makefile, CI, OpenAPI skeleton + codegen, SQLite + migrations, auth (owner login, sessions), runtime client + parsers, renderer + golden tests, revision FS + validator, bootstrap config, reload procedure. *DoD:* `docker compose up` on a clean VM brings up HAProxy with the panel reachable at :8443; `loxy render --dry-run` works; reload path integration tests pass. -**Phase 1 — HAProxy management MVP on a single node** -Frontends (http/tcp, binds incl. `*`, TLS termination, HTTP→HTTPS redirect, default backend, host/path routes, timeouts, maxconn, forwarded headers, enable, maintenance, labels); backends (algorithm, full health checks, sticky, PROXY protocol, timeouts, labels); servers (IP, host, label selector) with runtime apply, weight/backup/drain/maint and graceful removal; hosts inventory; certificates (upload + managed HTTP‑01/DNS‑01); **Domains & DNS part 1** (§8.9): domains attached to frontends with required records and live verification, DNS provider accounts with encrypted credentials, DNS‑01 via the libdns‑backed lego provider, one‑click certificate issuance; node addresses; Dashboard, frontend Overview/Routing/Graphs/Config, backend Servers/Settings/Graphs (live + history); Activity (events + audit); users/roles. -*DoD:* every acceptance criterion in §8 for these screens passes in E2E; zero‑downtime load tests pass; §1.4 budgets met. +**Phase 1 — Load‑balancer MVP on a single node (projects model, ADR 0006)** +Migration `0002_projects.sql`; projects (algorithm, maintenance, labels) with the create wizard and automatic ports; services (http/https/tcp, listen → destination, health checks, sticky, PROXY protocol, timeout, certificates); targets (IP, host, label selector) with runtime apply, weight/backup/drain/maint and graceful removal; domains on shared `sys_http`/`sys_https` with automatic HTTPS (HTTP‑01 and DNS‑01), required records and live verification; hosts inventory; certificates (upload + managed); DNS provider accounts with encrypted credentials; node addresses; Dashboard, Projects list, project Overview/Graphs/Targets/Services/Domains/Settings (live + history graphs), global Domains page; Activity (events + audit); users/roles. +*DoD:* every acceptance criterion in §8 for these screens passes in E2E; zero‑downtime load tests pass; §1.4 budgets met; migration test green. **Phase 2 — Operations polish** -Config revisions UI (list, diff, rollback, pin), notifications (webhook, email, Telegram, Slack) and alert rules with label filters, API tokens, live logs (sampled) on the frontend Logs tab, Prometheus endpoints, backups (+ S3), 2FA, `loxy` CLI client for the API, **Domains & DNS part 2** (§8.9): auto‑managed A/AAAA records and the zone editor. -*DoD:* alert on server down arrives within 5 s; rollback restores previous behaviour verified by integration test; nightly backup + restore round‑trip test. +Config revisions UI (list, diff, rollback, pin), notifications (webhook, email, Telegram, Slack) and alert rules with project label filters, API tokens, live logs (sampled) on the project Logs tab, Prometheus endpoints, backups (+ S3), 2FA, `loxy` CLI client for the API, **DNS part 2** (§8.12): auto‑managed A/AAAA records and the zone editor. +*DoD:* alert on target down arrives within 5 s; rollback restores previous behaviour verified by integration test; nightly backup + restore round‑trip test. **Phase 3 — Multi‑node and generic HA** `loxy panel` / `loxy agent`, enrollment, node pages, HA groups with keepalived VRRP and a virtual IP (no provider API), `peers` sync of stick tables. *DoD:* killing the active node moves the virtual IP and traffic to the passive node automatically; the panel going down doesn't affect traffic; agent reconnects and reconciles. **Phase 4 — Advanced L7 (optional)** -Rate limiting per client IP (stick tables), IP allow/deny lists per frontend (runtime ACLs), request/response header rules, HSTS, response compression (3.4 `filter comp-res`), custom error pages, HTTP/3 (QUIC) when the image reports QUIC support, backend TLS re‑encryption to servers, OIDC SSO for the panel. +Path routing inside a project (path prefix → service), rate limiting per client IP (stick tables), IP allow/deny lists per project (runtime ACLs), request/response header rules, HSTS for domains, response compression (3.4 `filter comp-res`), custom error pages, HTTP/3 (QUIC) when the image reports QUIC support, TLS re‑encryption to targets, OIDC SSO for the panel. --- ## 20. Backlog of useful extras -- **Config import** from an existing `haproxy.cfg`: parse frontends/backends/servers that map onto the Loxy model, show what can't be imported (kept out, listed), create the objects in one transaction, then let Loxy own the config. -- **Maintenance page** per frontend with custom text (instant, runtime map). -- **Graceful drain** with configurable timeout, and **slow start** (`slowstart`) for servers returning to service. -- **Live logs**: HAProxy logs to `127.0.0.1:5514` UDP → loxy ring buffer per frontend, sampling (e.g. 1:10), off by default (privacy and performance first). +- **Config import** from an existing `haproxy.cfg`: map frontend/backend pairs onto projects and services, show what can't be imported (kept out, listed), create the objects in one transaction, then let Loxy own the config. +- **Maintenance page** per project with custom text (instant, runtime map). +- **Graceful drain** with configurable timeout, and **slow start** (`slowstart`) for targets returning to service. +- **Live logs**: HAProxy logs to `127.0.0.1:5514` UDP → loxy ring buffer per project (by `be_` name), sampling (e.g. 1:10), off by default (privacy and performance first). - **Config preview & diff** before applying risky changes; "Download HAProxy config". -- **Per‑server graphs** and response‑time / status‑code breakdowns. -- **Bandwidth limits** per frontend using HAProxy's `bwlim-in`/`bwlim-out` filters. -- **Duplicate frontend/backend**, **Export/Import JSON**, Terraform provider later (the API is resource‑shaped, which makes this easy). +- **Per‑target graphs** and response‑time / status‑code breakdowns. +- **Bandwidth limits** per service using HAProxy's `bwlim-in`/`bwlim-out` filters. +- **Duplicate project**, **Export/Import JSON**, Terraform provider later (the API is resource‑shaped, which makes this easy). - **Webhooks** for every event kind. - **Rate limiting & IP lists** via stick tables and runtime ACLs (Phase 4). - **HTTP/3** advertisement (`alt-svc`) when QUIC is available. -- **Runtime backends** (3.4 `add backend` / `publish backend`) to make some backend changes reload‑free. -- **Own authoritative DNS server** — explicitly *not* planned (non‑goal); revisit only if operators need GeoDNS/health‑based DNS failover. Today Loxy manages records at external providers (§8.9). +- **Runtime backends** (3.4 `add backend` / `publish backend`) to make some service changes reload‑free. +- **Own authoritative DNS server** — explicitly *not* planned (non‑goal); revisit only if operators need GeoDNS/health‑based DNS failover. Today Loxy manages records at external providers (§8.12). - **More DNS providers** via libdns (deSEC, Route 53, DigitalOcean, OVH, …) — one registry entry each. -- **Health‑check templates** using the 3.4 `healthcheck` section (reusable checks shared by backends). +- **Health‑check templates** using the 3.4 `healthcheck` section (reusable checks shared by services). --- ## 21. Open questions for the owner (ask before Phase 1 ends; defaults in brackets) 1. Accent colour and logo [neutral red, simple wordmark "Loxy"]. (Name is decided: **Loxy**.) -2. Will people other than you log in (multi‑tenant projects)? [single owner + team, one project] +2. Will people other than you log in? [single owner + team; one tenancy project (`projects` id 1) — the load‑balancer projects are `balancers` in the DB; per‑project permissions are backlog] 3. Is HA in scope for the first production rollout? [no — Phase 3, generic VRRP] -4. DNS providers for DNS‑01 and record management [libdns providers: Hetzner DNS, Cloudflare first, §8.9]. +4. DNS providers for DNS‑01 and record management [the libdns providers already in the `internal/dns` registry; more are one registry entry each, §8.12]. +5. Automatic port range [lowest free port in 20000–29999; explicit ports may be anywhere else that is free]. +6. Should a domain default to HTTPS? [yes — `https` on, :80 redirects once the certificate is issued]. --- @@ -1171,3 +1262,4 @@ Rate limiting per client IP (stick tables), IP allow/deny lists per frontend (ru - libdns (provider‑agnostic DNS record APIs for Go) — https://github.com/libdns/libdns - lego ACME library (custom challenge providers, DNS providers) — https://go-acme.github.io/lego/ - Roxy‑WI (for contrast) — https://github.com/roxy-wi/roxy-wi +- ADR 0006 — projects model (`docs/adr/0006-projects.md`) diff --git a/README.md b/README.md index be8ed21..9225710 100644 --- a/README.md +++ b/README.md @@ -1,29 +1,39 @@ # Loxy -A lightweight, self-hosted management panel for -[HAProxy 3.4 LTS](https://www.haproxy.org/) — a safer alternative to -hand-editing `haproxy.cfg`. - -Manage HAProxy's own building blocks from a small web UI or REST API: - -- **Frontends** — HTTP or TCP listeners on any addresses/ports (`*` = all), - TLS termination with certificates, HTTP→HTTPS redirect, a default backend - and host/path routing rules, timeouts, `maxconn`, forwarded headers, - one-click maintenance mode. -- **Backends** — algorithm, health checks (TCP/HTTP/HTTPS), sticky sessions, - PROXY protocol, timeouts; shared by as many frontends as you like. -- **Servers** — IPs, hosts from the inventory or label selectors; weight, - backup, drain/maintenance and graceful removal, applied at runtime. -- **Hosts**, **certificates** (uploaded or Let's Encrypt via HTTP-01/DNS-01) - and **domains & DNS** (required records, live verification, DNS providers - via libdns). -- **Observability** — per-server health, live and historical graphs per - frontend and backend, live logs, activity and audit log. - -Every change is rendered, validated with `haproxy -c`, stored as a numbered -config revision (diff and rollback), and applied through the runtime API when -possible or a hitless reload otherwise. One Go binary, SQLite, no other -services. +Simple, self-hosted load balancers on your own servers, powered by +[HAProxy 3.4 LTS](https://www.haproxy.org/). One Go binary, SQLite and the +official HAProxy image — no other services. + +## How it works + +A **project** is one load balancer. Create as many as you like. Each one has: + +- **Targets** — where traffic goes: IP addresses (optionally `ip:port`), hosts + from your inventory, or a label selector such as `role=web`. Set weights, + mark backups, drain or pause a target; changes apply instantly without a + reload. +- **Services** — the ports it listens on: `http`, `https` or `tcp`, listen + port → destination port, health checks, sticky sessions, PROXY protocol. + Every new project gets its own free port automatically (from 20000 up). +- **Domains** — attach `app.example.com` and Loxy routes it to the project on + ports 80/443. HTTPS is automatic: as soon as the name points at your server, + Loxy gets a Let's Encrypt certificate and redirects HTTP to HTTPS. A DNS + provider (via libdns) can manage the records for you. + +Plus live and historical graphs per project and service, live logs, health per +target, maintenance mode, certificates, users and roles, API tokens, 2FA, +notifications and backups. Every change is validated with `haproxy -c`, stored +as a config revision you can diff and roll back, and applied without dropping +traffic. + +## Quick start + +1. **Create a project** — give it a name. It is reachable right away on its own + port, e.g. `http://:20001`. +2. **Add targets** — paste the IPs of your app servers (or pick hosts / a label + selector). Loxy health-checks them and starts sending traffic. +3. **Reach it** — use the project's port, or attach a domain: create the A/AAAA + records Loxy shows you, and HTTPS turns on by itself. > Status: under active development. The build spec is > [INSTRUCTIONS.md](INSTRUCTIONS.md). @@ -99,8 +109,9 @@ Open `https://:8443` and log in as `admin@loxy.local` (override with docker compose logs loxy | grep 'initial password' ``` -Restrict panel access with `LOXY_PANEL_ALLOW` or a firewall. Never expose -ports 9090, 8405 or 5514. See [docs/operations.md](docs/operations.md) for +Restrict panel access with `LOXY_PANEL_ALLOW` or a firewall. Open the project +ports you use, plus 80/443 when you use domains. Never expose ports 9090, 8405 +or 5514. See [docs/operations.md](docs/operations.md) for upgrades, backup/restore and break-glass procedures. ## Configuration @@ -113,7 +124,7 @@ All settings are environment variables in `.env`; see Prerequisites: Go (see `go.mod`), Node 22+, Docker. ```sh -make dev # haproxy + 3 whoami servers + Pebble ACME + loxy (dev mode) +make dev # haproxy + 3 whoami targets + Pebble ACME + loxy (dev mode) make gen # OpenAPI -> Go + TS, sqlc make lint # gofmt, go vet, golangci-lint, svelte-check make test # unit tests @@ -123,7 +134,7 @@ make build # bin/loxy make image # ghcr.io/2shiraz/loxy:-haproxy ``` -In dev the whoami servers listen on `127.0.0.1:9001-9003` (loopback servers are +In dev the whoami targets listen on `127.0.0.1:9001-9003` (loopback targets are accepted only when `LOXY_DEV=true`), Pebble on `https://127.0.0.1:14000/dir`; state is kept in `.dev/`. diff --git a/docs/adr/0005-product-direction.md b/docs/adr/0005-product-direction.md index d3ecceb..183b81d 100644 --- a/docs/adr/0005-product-direction.md +++ b/docs/adr/0005-product-direction.md @@ -1,6 +1,8 @@ # ADR 0005 — From cloud-LB clone to HAProxy management panel -Status: accepted · 2026-10-14 +> **Superseded by [ADR 0006](0006-projects.md)** (projects with targets, services and domains). Kept for history; frontends/backends are no longer user objects. + +Status: superseded · 2026-10-14 ## Context Phase 0/1 modelled Loxy on a cloud load-balancer console: "load balancer" diff --git a/docs/adr/0006-projects.md b/docs/adr/0006-projects.md new file mode 100644 index 0000000..54ec599 --- /dev/null +++ b/docs/adr/0006-projects.md @@ -0,0 +1,110 @@ +# ADR 0006 — Projects: simple load balancers with targets, services and domains + +Status: accepted · 2026-10-14 · supersedes ADR 0005 + +## Context +ADR 0005 turned Loxy into an HAProxy management panel whose user objects were +HAProxy's own: frontends (binds, TLS, redirect, routes), backends (algorithm, +health check, sticky, PROXY protocol) and servers. Working with it, the owner +found that model too complex for the everyday job: to publish one app a user +had to understand binds, `*` addresses, default backends, routing rules, +redirect listeners and mode matching. The owner's requirement: **"I want this +to be simple."** + +What users want is a load balancer they can create in one step, point at their +servers, and reach either on its own port or on a domain with HTTPS that just +works. + +## Decision +- **Project = one load balancer.** A user can have many. Each project has + Overview, Graphs, Targets, Services, Domains, Logs and Settings, plus an + algorithm (roundrobin/leastconn/source/random), a maintenance switch + (http 503 / tcp reject) and labels. +- **Targets** — where traffic goes: an IP (optionally `ip:port`), an inventory + host, or a label selector over host labels; weight, backup, drain/maintenance. + Each target is health-checked per service. +- **Services** — ports the project listens on: protocol `http | https | tcp`, + listen port → destination port, health check (tcp/http/https), sticky + sessions (http/https), PROXY protocol v1/v2, timeout, certificates (https). + A new project automatically gets an http service on **its own free port, + allocated from 20000 upward** (20000–29999); the listen port can be set explicitly. Listen + ports are unique across all projects. **Ports 80 and 443 are reserved for + domains**; the panel port, 8405, 5514 and loxy's internal listener port are + reserved too. +- **Domains** — attach a hostname to a project (optionally to one service; + default: the project's first http/https service). Traffic for the hostname on + the shared :80/:443 is routed to the project. **HTTPS is automatic**: once the + name resolves to the node, Loxy issues a certificate (Let's Encrypt HTTP-01 + through the shared :80, or DNS-01 through a DNS provider) and redirects + :80 → :443. Required DNS records are the node's addresses; optional automatic + A/AAAA records via a libdns DNS provider. +- **Compiled to the verified rendering engine.** Users and the API never see + frontends/backends. Internally each service `S` renders `frontend fe_` + (`bind 0.0.0.0:` + `ipv6@::: v6only` when the host has IPv6; + `ssl crt-list` for https) with `default_backend be_`, and `backend be_` + holds the project's targets as servers `s` / `sx` + (port = target override or service destination port) with the service's + health check, sticky cookie (`secure` on https) and PROXY protocol. Domains + render two shared frontends: `sys_http` on :80 (ACME HTTP-01 from + `virt@acme_http01`, 301 to https for domains whose certificate is issued, + `use_backend be_` by Host otherwise, 404 for unknown hosts) and `sys_https` + on :443 once a domain certificate is issued (`ssl crt-list` of domain + certificates, `use_backend` by Host, `X-Forwarded-Proto https`, 404 for + unknown hosts). HTTPS is automatic: when a domain verification finds the + name resolving to the node and `https` is on and no certificate exists yet, + a managed certificate is issued (HTTP-01, or DNS-01 when the domain has a DNS + provider); :80 redirects only once it is issued. Maintenance uses the `opt@` + maint map keyed by service id. Everything verified in ADR 0004 / `docs/haproxy-notes.md` + is kept: `sys_panel`, `sys_stats`, master-worker `-W`, `crt-base` relative + crt-lists, named defaults with `from`, runtime ops for target + add/remove/weight/state, reload otherwise, `haproxy -c` before going live. +- Metrics and graphs per project and per service (from `be_`, so domain + traffic is included), live logs per project. Alert rules filter by project + labels; node-wide events (apply, backup) are not filtered. +- Revisions + rollback, users/roles/API tokens/TOTP, notifications, + backup/restore and the Prometheus endpoint are unchanged. +- API: `/projects`, `/projects/{id}/services`, `/projects/{id}/targets`, + `/projects/{id}/metrics`, `/domains` (with `project_id`, `service_id`, + `https`) — see spec §12 and `api/openapi.yaml`. + +## Consequences +- Storage: the load-balancer project lives in a new table **`balancers`** + (foreign keys `balancer_id` on services, targets, target_slots, domains and + events), because `projects` is the existing tenancy table. The tenancy table + is unchanged; hosts, certificates, DNS providers, users and API tokens keep + their `project_id` (always 1), which is why they are usable by every + load-balancer project. The API and UI call a balancer a "project". +- Migration `0002_projects.sql`: frontend/backend definitions are dropped (they + do not map 1:1 onto projects and Loxy is pre-1.0) — `routes`, + `frontend_certificates`, `frontends`, `server_slots`, `backend_servers` and + `backends` are removed and `metrics_1m` is cleared; domains are copied + detached (`balancer_id` NULL, `https` 1; attach them to a project + afterwards); events are copied without their frontend/backend ids. Only + pre-release databases with a `load_balancers` table are refused, with a clear + error. +- Deleting a project detaches its domains (`ON DELETE SET NULL`); the domain is + kept with no project. +- Frontends, backends, binds, routes and default backends are no longer + user-visible. The read-only Config view still shows the rendered HAProxy text. +- No per-address binds: services listen on all addresses of the node. Port + collisions reduce to "listen port unique + not reserved". +- The per-frontend HTTP→HTTPS redirect option and `sys_acme_fe` are + removed: **:80 redirects only for domains**, through `sys_http`. A service + port is served exactly as configured (http or https). +- Host/path routing rules are removed from v1; host routing happens only + through domains. Path routing inside a project is a Phase 4 item. +- A target is rendered in every service backend of its project, so target + runtime ops multiply by the number of services (still one command per + backend; no reload). +- Maintenance for http/https services is rendered in `fe_` and, when a + domain routes to the service, also in `be_`, so domain traffic is covered + (new VERIFY item); tcp keeps `tcp-request connection reject` in `fe_`. +- Exact-host `use_backend` rules are rendered before wildcard ones. Deleting the + last http/https service of a project with domains, or switching it to tcp, + returns 409. `LOXY_PANEL_PORT` 80/443 is refused at start. Automatic ports + come from 20000–29999. +- Project metrics are a new `pr` scope (sum of the project's service backends, + so domain traffic is included); `?service_id=` returns that service's backend + series. Health events are `target.health.changed` with the project id. +- Performance budgets are restated as 50 projects / 200 services / + 1 000 targets. diff --git a/docs/haproxy-notes.md b/docs/haproxy-notes.md index 87fd78e..0eab1b7 100644 --- a/docs/haproxy-notes.md +++ b/docs/haproxy-notes.md @@ -14,11 +14,13 @@ All items below are covered by `test/integration` (`make test-integration`, real | §7.6 dynamic servers | **Confirmed.** `add server` with the exact `render.ServerArgs` (incl. `cookie`) → `New server registered.`; `enable server`/`enable health` → empty output; `set server … weight/state` → empty; `del server` after maint → `Server deleted.` | `TestDynamicServers` | | §7.6 SSL runtime | **Confirmed.** `set ssl cert` → "Transaction created/updated", `commit ssl cert` → `Success!`, `new ssl cert` → "New empty certificate store", `add ssl crt-list` (payload) → `Success!`, `del ssl crt-list` → "deleted in crtlist", `del ssl cert` → "deleted!" (command available). | `TestVerifySSLRuntimeAndMaps` | | §7.6 maps | **Confirmed.** `opt@` is not part of the map name (runtime uses the plain path); `virt@acme_http01` works with `add map` and the `http-request return … lf-string` rule answers the challenge. | `TestVerifySSLRuntimeAndMaps` | +| §7.4 domains on shared `sys_http` (ADR 0006) | **Confirmed.** Host routing via `use_backend … if { req.hdr(host),field(1,:),lower -m str … }` routes a known domain to its service backend; `http-request return status 404 … unless ` answers unknown hosts; ACME HTTP-01 answers precede it. | `TestVerifySSLRuntimeAndMaps`, `TestACMEHTTP01Pebble` | +| §7.4 maintenance inside a backend | **Confirmed.** `http-request return status 503 … if { str(),map(opt@maint.map) -m found }` in `be_` (rendered when a domain routes to the service) answers 503 for domain traffic; the same map entry makes `fe_` answer 503 on the service port. | `TestVerifySSLRuntimeAndMaps` | | §14.7 minimal capability set | **Confirmed.** `--user 0:0 --group-add 1500 --cap-drop ALL --cap-add NET_BIND_SERVICE,SETUID,SETGID,KILL,CHOWN` starts, binds :80 and reloads. | `TestVerifyMinimalCapabilities` | ## Other findings -- **`master-worker` in `global` is deprecated since 3.3** (warning: "will be removed in 3.5"). Loxy relies on `-W` only and does not render the keyword (deviation from the §7.4 reference text). +- **`master-worker` in `global` is deprecated since 3.3** (warning: "will be removed in 3.5"). Loxy relies on `-W` only and does not render the keyword (ADR 0004 §1). - **Global `maxconn 200000` requires a matching `nofile` ulimit** (≈ 2×maxconn). With a lower hard limit the worker exits ("Cannot raise FD limit"). Compose sets `nofile: 1048576`; the value is configurable (`LOXY_HAPROXY_MAXCONN`), tests use 4000. - Starting as root without `chroot` logs a warning recommending `chroot auto`; still off by default (§7.4), to be enabled after a dedicated test. - `server-state-file` missing on first start only logs a NOTICE. @@ -37,4 +39,5 @@ All items below are covered by `test/integration` (`make test-integration`, real - `chroot auto` (§7.4) — not enabled. ## Retired -- The cross-frontend connection cap (`track-sc0 … table lb_cap`) was verified for the earlier load-balancer model and is no longer rendered: limits are per frontend (`maxconn`) since the frontend/backend rework (ADR 0005). Its test was removed. +- The cross-frontend connection cap (`track-sc0 … table lb_cap`) was verified for the earlier load-balancer model and is no longer rendered (ADR 0005). Its test was removed. +- Per-frontend HTTP→HTTPS redirect listeners (`fe__rd`) and `sys_acme_fe` are no longer rendered: since the projects model (ADR 0006) :80 is the shared `sys_http` frontend, which answers ACME HTTP-01 and redirects only domains. The verified building blocks (`virt@acme_http01`, crt-lists via `crt-base`, dynamic servers, maps) are unchanged; frontends/backends are now internal sections (`fe_`/`be_` per service) rather than user objects. diff --git a/docs/operations.md b/docs/operations.md index 0cc5405..ae79229 100644 --- a/docs/operations.md +++ b/docs/operations.md @@ -23,6 +23,14 @@ docker compose pull loxy && docker compose up -d loxy HAProxy keeps serving; no traffic impact. DB migrations run at start after an automatic pre-migration backup in `data/loxy/backups/`. +Upgrading from the frontend/backend version runs migration `0002_projects` +(ADR 0006): frontend and backend definitions are dropped and the 1-minute +metrics are cleared, so recreate them as projects (services + targets); domains +are kept but detached — attach them to the new projects on the Domains page. +Hosts, certificates, users, API tokens, DNS providers, events and settings are +kept. Only a pre-release database with a `load_balancers` table is refused at +start with an error; start from an empty data directory for that one. + ## Upgrade HAProxy Bump `HAPROXY_VERSION` **and** `LOXY_VERSION` to a loxy image built for that @@ -69,6 +77,7 @@ The reconciler then re-applies the restored state to HAProxy (validated with ## Firewall -Allow the ports your frontends bind publicly. Restrict the panel port (8443) to admin IPs +Open the project ports you use (the service listen ports, 20000+ by default), +plus 80/443 when you use domains. Restrict the panel port (8443) to admin IPs (`LOXY_PANEL_ALLOW` and/or host firewall). Never expose 9090 (internal API), 8405 (stats/metrics) or 5514 (syslog receiver). diff --git a/docs/screenshots/backend-graphs-dark.png b/docs/screenshots/backend-graphs-dark.png deleted file mode 100644 index 4fe9c5b9abc86ffbe8fdba7909677e7ac5ca4e87..0000000000000000000000000000000000000000 GIT binary patch literal 0 HcmV?d00001 literal 75093 zcmcfpRa}+p7e9*9ARyfx5+V(X)FcHYBm_aaTTmJ#1nCrMQ5r-62?3>%E~P{1E|Koq z16{v;aW2lqIeY)-#af@wT&#D#^L?IYjBkw@ruIPoIyMzH5)#t&`wFreNJ!{vNJyyq zSSaw$9Xw(~BqRc)`?6A+9&dk7qPvhRkE3r1{h=dbU{qlET2rI|DcUZ_DRMQFgi?suifxIoyJNvCzk%&(A=5w?(fTB3DMhASo-<% zX9lm_iOI?HvokcpzpvGpQU<2KuPl@h|MbwmuY4ra8Mc4#a=f583j6miHUoz5|Nb=e z|M@@7LTT;m>zkOE7#kaFX+dkIh79c4Sjj0V;(ixzi!M-l z)d>DwiTiVTx$L~b0@dEuySH!oRZlsTgTuB7q|kMvMb5?yONUKw^^54=FvzE4feYWe z*V@*G{qO73Qc_arCS#k)pfYF2=iG=*eoI5l^V&s(iB>(=Ykw_?+d!IvuvoLf_gq6m z*~ zQ|F}~n}M9L+A51~vfJ2qp8VQ7-E44ob|yY~Q4y;yVmDIwb2;5Fy0wo}+`6o?@gX;KBK{`4*tmC@phxKk7zbc=MA8by?IPgt<8!6Jm>8iG;8;B7}=|GN; zotvLW|F^huk+`@;8lpirnOe&%uSKlUwc#b+pXK9AqLY7DXT_kgsS>FwsPN-va6tp{P$3EZtmYd9&a{a~JosK?e!qd#itg2%%|9wYAf zsjZ$AfnxotGV6XOGcz->?(@?_Vj`jk>4>Oczo$=29+hT1oo(8z-NiGcO?=XuCY~zf z*rwR_kVf2#Bmw2&!-u%(qV9M>!M^7wnb8dEh$yPtw$t_EX;{t=PQ41W3huz?W!;M~ zRnqxvd7VrzdULAo`FQ1%(rc5|t~m4b2=CBX@#hCE5%(lc`?KYZ{LU-rgy?)aEko$- zjCS)vN}oN9;H1IMnoqd#x2~*2Sr|KTr1n0lQ2JF-V7_K+XKib*(a~1vie7M%`;d)S z^T$T7H!B84=w4`rf=`9rXrdH`(E8Aamwb45cv$vqs%iSQ9@{T5@p&WT+4nHFk9s98 z?fWv2ndYR?F`g2mW8J>Lq%Vi5HaC$Lqxn1P$sjjc6OcDr6-5ewJBWr@o_K=e6%X?GwU;KBA?mITTGR zda}2gU1g7zpu-_HsVZ^qvec6z^%BD*xWB(YUkaC2ghXko!Ozd#{jfO%f4bI_xB_(u zv4F})iA}SbcTU8j=)Kd)W@2J8MyK{U8Q~++))8WS;nBVHJvHP_lRUjo;QkBO@mwYH z+DwnAu&}UfxhSH?{_gVavBI(68;f11E*SLMZ617D5~0V=>TbR9;GKx;>A{n_2IIe_ z#*2%B;*wf~)7Hy+(&BM;@`xZ0DFML?2JL!pPh93FOq3_b$6EPn$h0sekJ6}DPdR?Ah+%NM)764tHLHSVg;_X$S%wlify2o?!O)$Jrv8{mN1 zSk=3(KO!N7s|MWhT^;y9`c^T4!{ofBt!?i9;h)KfmT0H2+5u#yIw*jactsky;ROY^ z!cjavk=}YVvk@0dCm`G4^Lwl`pPAvye3YCP$|_Nm&qPFsF?HkX_orU!WKthc&?C&+ zqkWDSlRS=fLZpS!Q8yV3smL5VU$aPDo;f=?@f%WoM5=MyXhIr!3UA(f7bt+2V^z*uy<2ZduaI`_FTB!r_4}&!kyK$apX6&*72}<5zGTKJ+I?>{G+%Sa zea4-MCwskwVZ(f@hARf22%y>o6~4H8_wI~%*+b(f6TH)d4VfeIuH#E~WotuIf_s>I z!;6{wFBU&Q6``4eP7z5dz+2Fr#M8}(iBBgRDPNKGY`)E-8cKA;o-uNom?x88A98Or z7xE@aFfA?Z@MG~uSh;SzPiG{>ZYGl5Vk?k9-9BDU?|pL{r`PH1=vn4l>poa>*#DX< z3BD*2cZj}I=kutzJw=cZyWY9e*3)SGwr;R9Ie)N}PukD^$&>Ayt*lP#BV1LM;$mW5 z@i&R7s5oMxzF8HTGzU{O-if+SLZ3;@_8=cW$(5>hJ>YL;*`d$rix0ekIylt$`iyC$ z@4}B;F^PnZ3zIjhkCLBSw<6SvMKNBor;VZo?S`Llp*$9I**dp#;S%zX9n9{nHNp3t zQAXVIN)6IM*dDen9~UY1;uiAYA5+FU}DXSKFW^SHSTFR%0RQA>%d4CFuXEviz zk#0pe=`F(sUs|(I1~p5Yn=yep?;pH_5>U{J#ig#U-jT4T((^DaF-F+Tb11fq`^72=f}{>?G3;%+6r-sYnx_xz$;7)ZdUq_Ees<&sC45(A8*L*?{QORT zT=%D7Nyie)`QUAnCbW3-NASU(yGwvZW}`^DPEM9m9Um1NauvG@J1un3`<<2Eiy#~0 ztUvuzb9IWozrt@1j;`5i!}?n+y1u-=P_Zc0PP3jQb{AE%qcUKiny7NdWIVVyKjnWs zS@)b=_>tjj8mUju5C32#GPO4~HC?D>$WTH>S(d)bsawu2(u0bD+on{@3k>WCM z2Z!rb;1&_WjKFx?}TG#hxGF`r{P&7WP0_!V;IFrnkGv z$T+(;TmTK=lVP2R2K7lzs4G;pYR}!grIxQ@(w%yabu95e#5B_Eo7`6ia$)x?C2~sn zxleq3wswrC5WSL>J@VOc8X;^6phrjP&93b3SM$!-J1@}Vm1=8i7x)Nu(%>1-?VD0K zE&@6Vc8y4QsTG0}o}Hatiy`%>vu9d0mfFT>L1X4nRZ;g$tfixIhL~-XV~Gr(cuqS= zg4`=Wwc}`eo?gs@*Q_nl$oClCabqA?iEv3qTKcMTi~P*FANT4Q8a)vO#nJw#ks0$? zjXRH`tU--ig0nSr--|zEr80k!tGaIjz|GTHo*xs>pY1q!tI0Y^Ye*}Y5B+@3^I(Eh z@e9qMwYcrCa+|@+`9FU?x$^LvR5>jUHGc-s57jvT-_dohMMteQ#*QL64H47hqmUEt4?oIDam!J=N zQKa`Ji}GxxpV`IFz`%eo&4SVb_AmP!LwZ3mx^unHX{8tgeczUR?W)T9oZaKQS3r>|THA9od_v*W_*ES$ zFE%-Mq313RTT(UIqq;P)NQL9eIko3X>xVRN=~-`hQ~WL#Hf(B3I(|6wGGsVJ2q~B6 z`}SDc79!LBN}++CoLdwdRZQdEYcvW1u>4ys0)(u{18d$~={t=|~0aN*I* z$;qj<*yYI;wTEIvcJz_-Z{@%2Zft5N%0)wUl$9+;Dxs1d_ zdl?A2?w{)PA`pl*ioN4frxGH5Va2)O0qqc@Q-1t(;VNYpPLkw!ou(m|CxnMitS86M zaB2?~PV>1s%X#9da4)<(J1*lwZOY=qTrT|E(rSF)|`T30Q)f{9UI7tU<&T z{8mRK5yictpkzKhyA(jMu(;MD2I|%D9{Zi_eV7_VNMq04hZ?Acmnt8A?b@~2P6FnB zIk`t9CojIHZ8U~G*jFlMm|0&>b+~-oU}MB?XVVr*xw5PJPGlb_0<@Z)zDyZ%9z(IG zGmVjS0f%@$w{mS7oR@z`jh)T|xqGxp+ps(-wB(XTY@{x$Vrvb?6TzQzuAmYer)u)e43Opl-TWB9#2-%js) zl<9T*^a?{i@Smzs!z=OInnA8UmsWlC;IsR_e|~;``F^}Mmb%0K+VE9Vfo;|uf78sy zrpj$YcX5%r{C`@&ccY7Lx`z)30f^EdyzdHqib+miRGh(AP^6(hS}>88u2m2wBKDE! z+pJ2nr5mGxV-5_zJK>f)w98DIPcaSdeRuJO`dnNzNOnoJ!m1*;WSJ-s}=~nfXrg*o$Pq zmHVAYnv71#E$mFTj+b*$ImL~|2E^5ITbJa$9`P^D%zObsSIq4F{`na=K=zG;pH}JL zQ-$z9)B_pxIX$?3{d&TvBebvJprBSDL2kdt`k^%ZnXECByr80&zWDI)y&7#vW#z@{ z`}BX0T%GP$nT67)gMsFzCQt?zT=!jLfOfrIc_kJx z&h>Nss?9Q46Gz^dVVp={D=K!G%lazzf$$jC@yV&WTe#9;he&=ZJv=UQ>{&V~GaMb7M)(WQo7*8Grq=%X z&pwMJ<3HUfbc*7qM2G|v6H5PNS6A22&`?LmO70*V3rnUl0V!!h@=K#t=D&b~iOFDL zCCexD52y$KT{(1z5}xgUx>V>6>Hq%{)O+u5lamJjM4;FU-mQoAGt0|t|87;m@BftA z#gv8e|AL10|MOJ`WM=3^T&(-E?hU8<`uYY32kV-R6h3DDZw1+lR~Wa(fBpJ1-5|ls zOVIr3)2IFQ(RRsSeAO$HHD9W#gfU*n#WgoK1HyRGx3g)# z%k#5i=UBU*o*v-8&N~a8HSU}L$!S^IQW7D1%;OT?M`{Z?a0|@L%sxIoQBhHR`nigv zJUqo9!Gv*^m6WhQDohRwyI(!8qM~xofuEnh=`)Ch|G+3ymi7(pAG7jCb)b^0sYpmj zOwv;m5IjBFG7-Ka;drznnnk+%JB!_4ZH$f8K3l5T*{y&5de_l$6G*IZNWLZUBo)NeK!kZ z8@^lB4Ye}!&Y2lwkEF>nP*SPy*b^v5 zwjaT>>geczYR|N6_9&T44&ifB4ojYoFviR0oLSAv&VHz&!E2~?!TW4ZR@0pK#g|jS zTc?Mcqa5e8L%;&B#Dd)=l}vn{G>MBNW*#MJKz@xl5^$Xkzr7lVze2G>1RaXsFW>sQ zI-i3dv#Qp`68}s`Tq1vz3DNlHSqWO>e)(3+?hIkc`DR7C+WT?*D@!&y9>07 z4LzpdpCY27pdL5^m%V@g{>ttbyHN)KOA0Z`VY#`B_!4I?+Ow2jQ2EEi^yaCg@{YrU z!RZnrC|TU5${&SBMVuP$F*)}0=e5O`tpB#XOI}E0x`Tv~k=1^|tFuc$nJ#wyTbn~4 z;`$V%D1L$x1uZ0Vv%uxhdG|NrB%0dyk$p?`wjer?mt-q?fIw&Kwv zBFlN`LlI9uXyksn0C_Q2IfehvpFcaEKmr1tkWc_4MWaYANj#eie9J|3vDlqN9^ac$ zHuzENR_(*>?QM@iQMcbKGczwZdg0zJ_+Txua&OhTKYRAfeRE12)XBo%3YM0ZV?96K z3cvi`+17?Zotc_S-KDys-W}q;(uZ8P2jASZYkjmhn>Ub&uuaIiF9R4LMO*yZM9KFI zg}@@bdG!a?&dU#2m7=3+L2NzB01@=j3Q{V!VJKipvgW=%HQJam@+C6|hiL7P*EE%G zwP@ZM%F4>19|s1tzn<}DQA(68wTCuR*772#MM3jt$CzpRFjKy12-W(xza> z&Hx4JxZxBK$EIp}W8f}R9q?%B$sg}T7Y#Z)JJ()VYDtS9OK@-~vL_))&PZ`f_dI+r zg(r(jj|zv%4p`J{RZ~+Yl+2~F&bw^!acA4wK6ho55P^+9v+m&xIV{vne|~p&lxnw4-j<#m8j`Q&t9+(-!RiI%)S&5U7Mt<_Eyb2Z#M91jDap=XDucyF4X};6MUwKMFx4_q9&JZ@#B&zT zFxJ=CQx@?l>jMc{@E#i7+JYTbtoP-?-?kW=MiEXUlbCq!z5m8jSNzgGeZWYcu{aJ6 zns^IFU=&EYq_eWl{4FKJ1=_tS0+?Lx3z&j;1k$;f5=H#Z?oAVQqEX7x#<+yIvG7N# zJvfp#eO%?lk|va0$f87~z`v6h5)zW(o$WYX519Q5bf}YR6w`+phn2_pgUYIDyPy~2 z5^hWQTA1oo?~8-D+wWwxXaXt5Yxj?uf^ZZjrEPv>@4bAk&bza{Efu}Lv-A3M0y?@F z%`loWYkacLT+20YK{eU23cDm-fhI9p8k&;aOq1v$-6Zk=tM(@AG(;Tt-dbtvoD3l` zvGYO)VXdUA*he#{N=+2;sAA&c7Q6H~Zh)R~Gd5zh0}&Ew=+2U}k_haeYeW4di6t0n zqK&$*=k3lgi&5ykYi+&J)L5I9nwuBTwh5bi1Rk3K+m%1Zn=f6OoSZx!C;JPhyy9foh#xsAoO(ly7^giDwF2Une{5w}#+S~F{7I_EyI z)N!y_2>o>{d-avsd0P21IdZBBM!_uT&!q;l8*O<(HhrFnF&bZ(;tTrl(GGPzw1lGP?Q zglwd-+|!Oz&3feh)Gtj@tc3Unt=o6L_6lhr!X_T?$6!7M`)iv9UX zA$#-a@aF!>{`wo@?w<5(wK~TFm}^R}h6BO8u~YGcrUw6QzBQ{hfHWfJbuCH2+xv2M z$_7Y7@h`)apBq)$KX!L_*9`6ZB=6LQ!5lv*^7h@kpfY5JygRl- z*03&L!#*LVZ_Rm3KA)i-yZ zoNUd?XvPXkkbqn!3`GB(sJmLQHMoRINe!m0;nuC)*$?S zAdN<$`O{?{H4GJ=o=Y!Cq;I*=! z7q9?WQMc57SBPn>LR$&n)zA8m9Bq=g?N4gSN@1N5jB z$O?NOGqyjSnlPN0T3F`)(a&B5qCVAoH&verr@Mp1O7I z>_4`tS6`#}&*Ud0T%`HW?N}U4BB|B?oXdm+_sJP;tx(wpMG=O?!Qb&-8Yas!1+Z#=P5rS%3+49bQFVzRsbyQ!kWO+61qF580sTfx zPW;!v_^+~r>Fim54SjWAGkg2DuU|8*_?d*bBwZmkEf`S6*x=^?c zuGiJI2Fyz9qvjR70&|ydpu4-fwUtFsa1U_e*l0Ti6rG%B5)uty8Kk~>1N8z89o?iT zfn8ntM-GTnJEaQD7G3e*#MBc!b{4|C6PfOX8zU5MCOa!AbO7Ue{rdHSL4E~vb8c2v z4CdS9xV>|;v;M6fCwn(7TdyD9=7*96rOeD24mp@*BspmNAl(_yfX?q&x1ZQS=FLt+ zLnAvSZShUnpC|Ch!@%rQY3$R|+q^su7xYyS0=hzhrPA|y&15A zcV98e0DG$hpis1u+MIY7#T#%fVFO`p|2LU?#t;JNIQaJM+k*!WpdmC|oZRX{2Bzq9 zvM04A*C8Y%Y!slBh*fHUbtMOeCoU%k_!qST&c zcdlL3%Ej-FGZ7V=#5wzy(o#|gUCatxcT7T}lryGiOqQ!|RhQ{Rb<{sCcc_QQY@l7J zLug|pXRVc}J3VH>^SCUBM(!c)HXUlmto3R7-DfDx3OgPHfHkv=qYarp;({Ci3e?t!paSzWm_ z9&;VD4bEa#5dVe@Ad1cD1|DXSlQkj7Db9y^kta{->FKL*^AR4Np1JS*fGBbs)`r&C zsd00MenPZ$REc};$g;Y55+dvA>5<^^ve;f%IHi~er^V^zTU8H_yBAEC(6KPPt2}ot zrH(14blOwH{s_gW$J@bU3A-*Fq&7e;@(^Rg@{C519vh0@>! z2`LLn81DR zI7=t)H4~(#qoY}DXy8mma)d{77aU!yqIO7Aq#wP7Ekxd|GXs3xVy}M1X&A|jdDc8` zF#5~O3igfHy8H^tM827w-(stLMO34Ri|XCZLH^PE6?tUq6c>p(y$MnOe+a z3yx}~ME;HQeLj<$V8EkYLiZWS{#ZN$_EiBCtT@z zs>`8vR#w)o)Sohr1y+dox(UK@Ef$NXHxr z|0TFw1Mi4()9=TzW`>%X?>U|xetU^2VA*pQw2|5mX^Zt>fnLe{;L8E$mlj^7#g`3* z8nz6!U3gRS`ZW_+CBnTuDub`C9kFzyeNbmXxa)!FI(KFL0T_QRFi(T0O6ER zYMB9QT~^`S^kS^-mv`&))h4Upt4mmxR(;-%wD- zyGW6!o;ca{%=hU>o*NyMCD(Bs1ge@Y#U-$t?-@uGxJ}Eo*eHjNuA4Ha1+SKSKv)V7 z2S-Ukq497bMZjukTW9x5a&Bia0QD@3Q>v%sL4}&zi~c{u=l)}Ca9_8X#vxb3Hf8K3 zdQk}v7s|cLVV2Lyq!fhCTDd=X5x>)$bjF$!84%Si1~p>P!VAq~?;wJVD_oK({!e~g zUELeyv$L}Wi?ffK9|9`P!GIa|mOjSB=JR+tlzOzlZ2bP*1o41w*7xM3_SB7R51$`S_?o>^?yd=*P zi$@x?L%6c;_FCI~S9#{BsGqlpvf$~hu|?P@I#u{sz1XAW)TLR? zt9Isz^t@rE)Wk6doP;&i2CEz}TLi3r(7@ILCf2tsBk?;|{x_KiAdNBdp>3zf!huJD zUkEnpK_?w4(8jOHv5C3a7wA@3doTt1L4TsWKqbf{SR42N0`#l+5A6D9TSi{>H}e?P z>q9wfxuYD@m`!K*BE>khnIE!Z>W>7|#GkXWr%TD`<@waggFd=-a3DwR*OwvXv00uc zTVt5#LxS?fVKV%W(3C@Y3l|p`;e7u=k^}~$NnGQ26js#Tp#p85A#@-d!7py8N=pT{ zDNK+kGAd*E0>~plH~$5+h>{G0>k!;8kdjHYZjhjWQFA_!^s+Fbm=_dov3E3PmX_eV zu?XvfkL^v`Riyslg#~24BrxP6DHsP49JD(25H*A>Vzy#Rf<2oPRoMs}MeU;@ zV{!c&x9@|4#0A}8stga42H}^Vw-y9TryfmDBM-kz=~QC1e%ybwlWi`8+Fu*6ok`7O z^IoZc8)Mh4{Yjej!?IVi=Q9i3!zxox?EX;4FBsc=eQNakS-R}4huc1W-LJWWP5N(Y z)3a54H2-DuK$Z|wrhHCyTBU@K+}XW#|AXVNvf681$Uz&|{SYKy=1tey?o`Biu6MDj zYA7LfuI-}_{-AeG9-J=w8fPM_jBSg_G(SrBz8Qx@GoN6DxAK0x{9@^Y7mm6^&K6Z6 z4N8hv8{-4CVXTcYFl**lzo%sHkB4l0w6&eqsN(b8>$Ci=zbO4*}yjnf4%8 zp#J`1roFs1+w7!D^c{CkjDVGjR3XctZBT z-F#|LZ)Vhs4{K}(MR!NvLR{M65~|~tLlmtj+2%B00{+bc=1Ye1?Bg(s-AVUp#&Kh# zN+>VVAJ7>-S-n1(?TBUW&{VLXXfxyDT_^=A(ZrEIbPWO;p?-Qg3m62vjamoAc^~-zM+NXdyaGgrjeR*YBapFoTs{cTZ#%5f zoxaOT_dcBkx_8$CuVKa+dN_2Dr5jsU2Zz>o^_B|GrZxl!0j@%Q3M}6I?K5HcYl?nq z=yzO$X^D;Z)Sz^GBYtnTM%q2rBp%!N+nXS>_GKY4pwI0=xXw} zk^HiC10O2FO`|bvmZxhzrBtMt^adsSbuAn$rPR70L+Vo;8aY@8GJSUh1O?xUxN;mgEv_zXb+V@`30fjDu^}Gotwd&}5Rr=qartTtf^CMT?{sxNR@(|yZyn*v{ti)ta zt2;r-mO0$m;)8ZiBCW6ir%nmOM#oF+?|caCcp|kQ>hf90xt@~yh#;D&qF2oxqF2p_ z8vW#K&GYn)#K(~BBE|oEMgXb!2R1bU-SgjLe6`=k@F3vBcbzWaZpbxW`U+CH<8Uz? zaRkE9@LXLw0q?$!c&s72d_p&Jk<2+h&0PjNnS@)~BNJHtok-oz`}gk!iv3k{SIO5BHvVHg@z}YM-$M0lco0H9CoYbTj>1;5{5dWpSiHHv z$Bn9-&~51FEoMM(uJ?`?b4ADDS$ELYUTxMM0Ay6M03!6LH)wHC-#-eezP&rUz8>*N zqW(yYSgW_Lm&5ycS3)H~nOIjHwxdVicIN9IMi4%>NL(eKD#^<$h;Gy3iS5--BGLGvP{hd)K2)Y*I4s(M~GL3m)0ro75rNS8AmNDb<59)YzJl zF)O+GW$3{qU4O6#>T!3!9uKNYm?-|{W%kc{7R2s8-*)|{eH=k;DSgDN81egp8{3jD zo0dJvjPW?6nL1BFd3?>H1Qm#^=mJau(_WhMV?t)RjEYE~)vJuKyYBj%hi_PwlV2<` z1!Rg<0}~v9Fch_bWn>4Y6U&yFJMVbX(7-#(U%ej{g8mE^galYiO)9AQ2H{uX@^;MX zHY9soUid%`m@b50Dc0@Lj*|sPQN(z6S?p3S%e^;B+8JH!m}*ojU^{kl1s z7c?04vRgq+@h=Yuj}|>o6)f0|8Pl|ibhX%*Q}c=Ei)+g=)S~&-@ICtivEMDJ#Pe2= z(J>lvlVWH6#5)rX92WK3vyK%=p^5r4Ypzon+TX>zjWzYPYSXsK-G?h&mvNFbEPI=U|?w`J(8%thRS9?aGds`RY#_)EaN^}8QLB&9++yp6W2{$tr+i21&? zRDRfWHGN&e*WTj6jH?`VJu+VAB<>;Zzz+$=o2^qj(?50JP?^tH6l~F_)!x9q<8d6p zCM`_VCBuj+@$vT&V(2`C`iioZ6#=?l84d~z#L8_ALImQTRoD(U;dO#SuxJkTuge=T z=3MWwoh6oj2{bA6a3=&RbApg(|1GpYh@W@Fd0SMbLof+KD_gaI*oJ=%j|LR_dlb{D zK}tz-qK8f5=LLtKU@E)zhtjiLE7}5j!Kj_7|3ZHAD%gBvJbi) z4Kt7R+xhIR5DsXAT;8`V->T03#1AArs6yR1WF4`P=!p#x`(Qf7Q>C4ZsTYFxUI~MP!@a2H9L+-WXGPkuKefmsNjb3+b6gLzrGkL zG){kbcPVbO(lviWaH~`^wU?##bUF-a^~Ytf(ZI^1rKsA-O>{OB|9fr42IgzfDgl{- zlc)<+m1BCx!*{IQCXS{tPZYFI(4Tg!j$r}V@#BG$i?ikUcyhXBN(a4aBixT2FI|r? z@R=;W9wV1gNTpzX{*jjFJ97mg_Coy$91cqe=Q;RI*ZY*l_#FN*Ks$o#D5ptV`aXV1 z!EdgBxPKbppN`GzZK{6UJc?YsURhcBhF*Pv^83x<^}8_U!abS(p<6iqngw3cGTw*z z;qJ78Sl>SrRVlrYqV;dkuX2KfgOGp#4mNhR_c4Ec>Xzxo@85k3NyQHgQH`wMK78rO zeN60=B1298dHC#h7bNpLkdJG;ii?k0hLk4W59b#?GnOq=m9*W?EiW+9DDxnA?L96b z%+5sVPTTPA$2HNVyx@?$aaNYxqY$=Y1li$o%Fw48KaZ|}NxoeS9ZE{dmNvFaH+}?P z=SxPF))zry*hnRPGOdooqUUDIsAiEhxQulSxTyEuu zCND%THLN>Hk8`ko_Ik|9ns@e$^^+`PcBJ8{5-E?NH@K0Yv@ew^0q)xCa$V_@Vo#dZ zNOguak&C1Er$6NY$wWhS6*Qx3C(a;N^jJT#);2EItE<25mNRUJY`y@;W2bNnbO7+N zb3t!`F_Ir=j9AAzIp=3_2k590r1mQzV4DuYxRc-I1sI}n%!IYWMn!Q$4-ctt+vZRn z-Fj3Qdb5AG7Nnyk?IrIIkn-$W-^vd2nrTE9CNdI%XlPusEyC}ni>bPu>^mYJMRq4575M%YM`8&2T->8$Skm8hHdQw(Gw}F43nl zOg*ODST!Y}i6tVu^@t|A_o$*_L5=*eeb>tC@Ub?*;y7torTBu9(~$G0-z_PA>5fgF zLsA>`4Fq^2n-I?iPJVBk-k+xC9A*a+M61VqR31DS5o3MU9DE&2r*DNY>Lq5}_yR)H z=DrwSpk_c9De*pb+8p|>qMrEgh7!!Lt_p0hTH7%N~9TU{@CoM4RLXUkQIbtdIm}cfNg{I zn~0+sR83VlbZW@oq`P8uQc}`u<1brpP_(>vH$ROzH6tE;Zb1=NBTL5E%lxIlroo=D z`TG15#ES7jr{2;-(bQrQqNG3Bf6!U%@_cRzDl%+XA^bXp*l%xGp%sg;rb)cN8qp|@ z0%J{7R5ZJ;Sl9`i{IQsOpJ~5y{8n#oX)#_th~TNmx_0dt0?&NkK$o}*K$*5+O%xnA zf~?k?%!jIS0pX@P=+2?2DPPu0r`G}6V58_-gFrI{kY}gbTv%M+;)D1hOu{uUOH3NY>VJKjSzA&I>YVPmbk&cBrQyv!#xEr$ zlE_3+&s-M(KdIer6nX$D`2`BuXFVJnHd_!XkO{_>($ZM5hj=jm7_&cfWB0??;bdGN z-S(kTgz*+Xa5Of4(F3w=UHNAbuf3I({%k(_{o&d5o^Riz)_?f}(_NcxxKzSdm1dNw z)kx77*W$lT=75Lcd1khw6;3&_FBlf7gjkI`T1olDmb!`~vYStJqjsu-3d9)>hOQ)@ z^yA(vtFWXb@}^6rLnJK#mu-gC+$x;EJF{0_y?S+(2|Jb6;jd&IHak?AW)$D}=Ne9m3g%9lH26zbH*+T4Kzk8zBT3{8_v)inX5Ndv zD39skF(l0KG54iqyK{#m$o@K~GLAh(beQUui{Ot!Wx?V z)%;9F+0yJPlDVRCHEnxB@}8jhTFxB#v$=b3#VTe=-x{ZjQtU$*j6GdZSy_;m*JD+< z(vhmXtHe^4yYKc^Dtly9)NY}jd@+xUerX*m`#;|*7UdlyW+X5@&V$J$PRQK{H+dq8$aQ< z3glx>CFlPo0~g&tT$z+bMd;_4cFYLG1*iv1Xqolbn zxRwF}pM{PIoWPAkE({3_L@OkY6*VMUGMZJjedzCB6w0>y&vPe%NtO&dU_*9xr)_tV z)%QcAqI@pSJh}G|a+3=6)5=wa+l&5PTkswS+G1G}nlETSQt<*4;s4&=q3iAa|2}H1 zfbT#3y``n~?&oO(9AyQMHysh>|Jk66wBl$N^xw|Az4{A^S0gGUx)kpa@GBL!}9C{Vy{tTZ{^~ zL}JqSqwUk1AdBilvO9O~-zTiD0W%P^=!GrtuOQ8M6Sd*P2O9uyv6tCdS-h-FOwtd` z%*}^DYzG`Q;}4@xO?A2gF7EE~8;y{hW?(=P_=d|@T3oydV}V3O8mfZq5A$#>PZv7l zxDBdd2Jp{R-T3sh5dvb=SF8wL;JppKeGhR_IG7Ml+k@Z-_<@i-!uci=0V3vI zhJq#Tu_gI890LY9wb#cZ_sRd|05cnS?)NN7Os`Jd-o_#6;TBm8EgVb`3^#~69$bKZI}l@di;MdSf=Be z>T1&^@XTT4v$?RTNyF%-v@Z-4Z#BYq%bdHK?a5Yd{GY>r96sfni0^MCuz~9EhMKC4 zl_jiDfS30w|2HK!ByW27h(Y^sZCJ_hPVaZF{c5C+S@6nbf3-hVP{4X~_6A>GRzAfT zvc=$#g{6t>5V}n&Moo5#=R8uf9If9wmb#=$_ zTwVTHZYCdma&W`@T2Pe`)t4d`;s?)>SuXM}Og6kv_YLP3`E_wMXbk+G;08D>84n_B z2pR_8;d>onEx;7wt5w>k#IzTTmtheR4|&jDB`4=2cyHdkwjWRqBjH2xwSE97O5R>Y zm(~i3VJ1I(;6;XaAy}cnH}73npnZAx3G{R(D={%KDj^5XcpPOxyAkDRSuJb8$xN6f z{SN27@YPBe^1Z|Kd4YX_1a2Fp|JgFidks7c)Kqt%n*1rvFu=D+)u)-SmI;x-b)OW- z28f?a08nmzZVOX8^JcfF8Mr+BL9>4{lpnjKP#Z=8!)wGcQB0P|c$%fosL}A~=!P-dC`pgVN{0YuTa#fY*qow%o@)kCDI*u>~L+sfB zr`M>#SDJpy(uBGFAVb)4%TKU<9jDUo>`Sh29-u5?Zzg+J-Eo~C4fV1ZMx0(DQC4tJ zY})6kKS0XG=_}-yj#P~nlt(`qNSyu_i0y{N`BqVcil&{In{UZDbI5HYD#(}q*&{fb z=33AlJG%5s+;rK)*+=uu)Gv3;QW|m7(}H0u$T%qchTUHY1tcBe13#~D0DHB7IO8~m zmL#PYnbtS(x^Sc<0=&Wa_;{#;X72Q60RaJh!}NK`!-2|av_-yWMz^e@8`fr#2%t^) z`~-4=Pr-}yyxg|W(A)e=Nh?Rq_P449i!jPvTR;53D4t8DBr!_stI{2Lc&K6z! zPA2@n1IcR;C!Rz1-TH<(M*MxCdAk&b&hyjL)BA+HZr)TW;;!_ZW95IZMC`>u3kC%Bn=9 ztM;b<>~*iX`J-70tcr_N;*M7!Bim0_SG;M-09XS6ihS>5CVy7qjQa3*+hNXzZ*I9{ zxBmB3pDFk$cBOS}Df=cCrzKSMQT{3^3kkj)Q+wP{%=>7D+hsCVW+<%xz^K-P?^h(4 zanikHMPX127+-c4oL^0h{zSTl>=Sv#Y0~{Jqy}M-UkcE!G>R;_DM($AYOs@2DuZqU zbJ1RK)K@&2NR1n$=m=l5Uw{t>IXS&*mzzyGrlxZM2+8xvQBnAbI{dyu>2>2&&@WjlZ}m{_ ze#>(``2E^6)zo!RPrsI@?Nz$m(k57(67dK^2Mdh#hP zNGiW%hX~633XMeWKs~YBhM}BE8N>+hqo=a8nm^%pMZ!G0I>RAfUT0e_#3yICENzej zSOwSkT?_>u6DzCb?$URl`NYD(S|M8D_jY!6nm#^X6%C51nlMXr>;r>*=&^}~sW8lJ zyf~h@cMd6h9VK?R-&1wejx>)55vO2Ovm54!YFwmY8D@8=jP(HJdkAqqdtL2M`Y<4j zB{xT127J@Q)#Lam_zX-GLDh^#jIa&L&3BNp4x@(d1r z5S=#@0!Fl9sMDk5i8cJ@$E}fM7zO|kwq((zu~ZVkG8vRKFNXi~{uZG*Tm$wTIiw4Khc3(2i*Nc$Pq zxD}~Ie~=o+jR?71DJ?%5kld!doSbj20b%=?SlFgvgWuHF_j9}C^Zlegi;WFHTO!}5lGaS6zc1a$05jQ=A z;JlW8_D3IFTr}TXO!@PsJum2zeBlerBLiS^TO7cR^t|(il7gkTYBuX5@EN95zHb^< zv1F74v9ynbzDu8AuO%Yvx0A=Gy|}?h_LWpZquRUr@~JlbSj~8dPqwi0(oBQUnP1Fh zgH=zme=L|-QsIpkOm6S{43d@?P!FW|kPjrbriCwLT75rGUaI#}30UH$8PHjT&xE{loWJyCMU^7&<&IxhGbL_q@o~bMawPcX<{Wp!-xS$x)M=;L$ zP>}pSp7CKhoNxGn7<4!D!B3a=GI9yis60OJw?^#hq&Q?%G?@aO((o`879}dn|DwA*`$@FbtNs~Fk)rWmApDo*C_}M$k#asbtV6HM|U2~x$LhF zE<;z&f5EU}7B?vNntt;E2n1jN<|Q0tLV;ZgpC=9vuKK@dd+Vqw*S=j;MH=apQcyub z5hSETNO8hFmhxlz?ip@W4Z@aU8kbI#^xLfzFY#lM9Z36mXKrYC!7kfUe`JPzAIZUK z8}Xz{`RN_H8VcPa`S*K`U(Fu5Dhv$e62-I9;<(kHdU8-+TGR5-hRkL*U-pg&Y3t}P zi_&X6go)MX*3HerPQzRkG5?8UMDcaJK=h3uQX`~xugmCMr90I=TMJ*z)H-m}8G?hk zbMm*rQ%Vyxx1gywa>^tRhOPil@I$Nfz!VgJzNr5jnfNU{>FI zbfxg#PeM=X^CrpKpYY)z6jf@zdWMDaictcL{c^q