diff --git a/content/configuration/listeners.md b/content/configuration/listeners.md index e4db811..b4a0238 100644 --- a/content/configuration/listeners.md +++ b/content/configuration/listeners.md @@ -436,7 +436,7 @@ Connections in progress continue with old certificates. New connections use upda ## ACME (Automatic Certificate Management) -Zentinel supports automatic TLS certificate management using the ACME protocol (RFC 8555). This eliminates manual certificate management by automatically requesting, validating, and renewing certificates from Let's Encrypt. +Zentinel supports automatic TLS certificate management using the ACME protocol (RFC 8555). This eliminates manual certificate management by automatically requesting, validating, and renewing certificates from ACME-compatible CAs (Let's Encrypt, ZeroSSL, Step-ca, etc.). ### Basic ACME Configuration @@ -454,9 +454,9 @@ listener "https" { ``` With ACME enabled, Zentinel will: -1. Create or restore a Let's Encrypt account +1. Create or restore an ACME account (Let's Encrypt by default, or a custom CA) 2. Request certificates for configured domains -3. Complete HTTP-01 domain validation automatically +3. Complete HTTP-01 or DNS-01 domain validation automatically 4. Store certificates securely on disk 5. Renew certificates before expiration 6. Hot-reload certificates without proxy restart @@ -477,6 +477,16 @@ listener "https" { staging #false // Use staging environment for testing storage "/var/lib/zentinel/acme" // Certificate storage directory renew-before-days 30 // Days before expiry to renew + key-type "ecdsa-p256" // Key type: ecdsa-p256, ecdsa-p384 + + // Custom ACME server (e.g., ZeroSSL, Step-ca) + // server-url "https://acme.zerossl.com/v2/DV90" + + // External Account Binding (required by some CAs) + // eab { + // kid "your-eab-kid" + // hmac-key "your-base64url-encoded-hmac-key" + // } } } } @@ -484,12 +494,15 @@ listener "https" { | Option | Type | Default | Description | |--------|------|---------|-------------| -| `email` | string | **required** | Contact email for Let's Encrypt account | +| `email` | string | **required** | Contact email for ACME account | | `domains` | string[] | **required** | Domains to include in certificate | -| `staging` | bool | `false` | Use Let's Encrypt staging environment | +| `server-url` | string | - | Custom ACME directory URL (e.g., ZeroSSL, Step-ca) | +| `staging` | bool | `false` | Use Let's Encrypt staging environment (ignored if `server-url` is set) | +| `eab` | block | - | External Account Binding credentials (required by some CAs) | | `storage` | path | `/var/lib/zentinel/acme` | Directory for certificates and credentials | | `renew-before-days` | u32 | `30` | Days before expiry to trigger renewal | | `challenge-type` | string | `"http-01"` | Challenge type: `http-01` or `dns-01` | +| `key-type` | string | `"ecdsa-p256"` | Certificate key type: `ecdsa-p256`, `ecdsa-p384` | | `dns-provider` | block | - | DNS provider config (required for `dns-01`) | ### HTTP-01 Challenge (Default) @@ -566,9 +579,31 @@ listener "https" { | Provider | Type | Description | |----------|------|-------------| +| Cloudflare | `cloudflare` | Cloudflare DNS API v4 | | Hetzner | `hetzner` | Hetzner DNS API | | Webhook | `webhook` | Generic webhook for custom integrations | +#### Cloudflare DNS Provider + +```kdl +dns-provider { + type "cloudflare" + credentials-file "/etc/zentinel/secrets/cloudflare-token.txt" + api-timeout-secs 30 + + propagation { + initial-delay-secs 20 + check-interval-secs 10 + timeout-secs 300 + nameservers "1.1.1.1" "8.8.8.8" + } +} +``` + +The token needs **Zone.DNS:Edit** and **Zone.Zone:Read** permissions. Credential file is plain text (the token itself) or JSON `{"token": "..."}`. + +Zone IDs are resolved and cached automatically from the domain name. + #### Hetzner DNS Provider ```kdl @@ -607,7 +642,7 @@ The webhook provider makes HTTP calls: | Option | Type | Default | Description | |--------|------|---------|-------------| -| `type` | string | **required** | Provider type: `hetzner`, `webhook` | +| `type` | string | **required** | Provider type: `cloudflare`, `hetzner`, `webhook` | | `credentials-file` | path | - | Path to credentials JSON file | | `credentials-env` | string | - | Environment variable with credentials | | `api-timeout-secs` | u64 | `30` | API request timeout | @@ -642,6 +677,56 @@ your-api-token Security: Credential files should have mode `0600` or `0400`. +### Custom ACME Server and EAB + +By default, Zentinel uses Let's Encrypt. To use a different ACME-compatible CA (ZeroSSL, BuyPass, Step-ca), set `server-url` to the CA's directory URL. Some CAs also require External Account Binding (EAB) credentials. + +```kdl +tls { + acme { + email "admin@example.com" + domains "example.com" "www.example.com" + + // Custom ACME directory URL + server-url "https://acme.zerossl.com/v2/DV90" + + // EAB credentials (obtain from your CA's dashboard) + eab { + kid "your-eab-kid" + hmac-key "your-base64url-encoded-hmac-key" + } + } +} +``` + +| EAB Option | Type | Description | +|------------|------|-------------| +| `kid` | string | Key ID provided by the ACME CA | +| `hmac-key` | string | HMAC key (base64url-encoded) provided by the ACME CA | + +When `server-url` is set, the `staging` option is ignored. + +### Certificate Key Type + +Zentinel allows configuring the key algorithm for ACME certificates: + +```kdl +tls { + acme { + email "admin@example.com" + domains "example.com" + key-type "ecdsa-p384" + } +} +``` + +| Value | Description | +|-------|-------------| +| `ecdsa-p256` | ECDSA with NIST P-256 curve (default, fast and widely supported) | +| `ecdsa-p384` | ECDSA with NIST P-384 curve (higher security strength) | + +Invalid values produce a config parse error. + ### Staging Environment Use Let's Encrypt's staging environment for testing to avoid rate limits: diff --git a/content/reference/config-schema.md b/content/reference/config-schema.md index d13271b..77b7e0d 100644 --- a/content/reference/config-schema.md +++ b/content/reference/config-schema.md @@ -141,6 +141,47 @@ upstreams { | `session-resumption` | bool | `true` | Enable session tickets | | `cipher-suites` | list | - | Allowed cipher suites | +### ACME Options + +| Option | Type | Default | Description | +|--------|------|---------|-------------| +| `email` | string | **required** | Contact email for ACME account | +| `domains` | string[] | **required** | Domains to include in certificate | +| `server-url` | string | - | Custom ACME directory URL (e.g., ZeroSSL) | +| `staging` | bool | `false` | Use Let's Encrypt staging environment | +| `eab` | block | - | External Account Binding credentials | +| `storage` | path | `/var/lib/zentinel/acme` | Certificate storage directory | +| `renew-before-days` | u32 | `30` | Days before expiry to trigger renewal | +| `challenge-type` | string | `"http-01"` | Challenge type: `http-01` or `dns-01` | +| `key-type` | string | `"ecdsa-p256"` | Key type: `ecdsa-p256`, `ecdsa-p384` | +| `dns-provider` | block | - | DNS provider config (required for dns-01) | + +### EAB Options + +| Option | Type | Description | +|--------|------|-------------| +| `kid` | string | Key ID provided by the ACME CA | +| `hmac-key` | string | HMAC key (base64url-encoded) from the CA | + +### DNS Provider Options + +| Option | Type | Default | Description | +|--------|------|---------|-------------| +| `type` | string | **required** | Provider: `cloudflare`, `hetzner`, `webhook` | +| `credentials-file` | path | - | Path to credentials file | +| `credentials-env` | string | - | Environment variable with credentials | +| `api-timeout-secs` | u64 | `30` | API request timeout | +| `propagation` | block | - | DNS propagation check settings | + +### Propagation Options + +| Option | Type | Default | Description | +|--------|------|---------|-------------| +| `initial-delay-secs` | u64 | `10` | Wait before first propagation check | +| `check-interval-secs` | u64 | `5` | Interval between checks | +| `timeout-secs` | u64 | `120` | Max time to wait for propagation | +| `nameservers` | string[] | public DNS | DNS servers to query | + ## Routes Block ```kdl diff --git a/content/v/26.04/_index.md b/content/v/26.04/_index.md new file mode 100644 index 0000000..a0dfa5f --- /dev/null +++ b/content/v/26.04/_index.md @@ -0,0 +1,160 @@ ++++ +title = "Introduction" +weight = 0 +sort_by = "weight" +template = "section.html" ++++ +Welcome to **Zentinel** documentation version **26.04**. + +> **Note**: You are viewing an archived version of the documentation. For the latest documentation, visit the [current version](/). + +--- + +Welcome to **Zentinel**, a high-performance **reverse proxy** platform built on [Cloudflare's Pingora](https://github.com/cloudflare/pingora) framework. Zentinel extends Pingora's robust foundation with enterprise-grade features designed for modern web infrastructure. + +## Key Features + +### High Performance +- Built on Pingora's async Rust foundation +- Memory-safe architecture with zero-copy operations +- Efficient connection pooling and keep-alive management +- Optimized for both throughput and latency + +### Service-Type Awareness +Zentinel understands different service types and optimizes behavior accordingly: + +- **Web Applications**: HTML error pages, session handling, SPA support +- **REST APIs**: JSON schema validation, structured error responses, OpenAPI integration +- **Static Files**: Direct file serving, automatic MIME types, caching headers + +### Advanced Routing +- Flexible path-based and host-based routing +- Route priorities and groups +- Path variables and pattern matching +- Per-route configuration overrides + +### Comprehensive Error Handling +- Service-type-specific error formats (HTML, JSON, XML, Text) +- Custom error page templates with variable substitution +- Graceful fallbacks for connection failures +- Detailed error tracking with request IDs + +### Observability +- Structured logging with configurable levels +- Prometheus-compatible metrics +- Distributed tracing support +- Health check endpoints + +### Security Features +- Path traversal protection for static files +- Request validation and sanitization +- Rate limiting capabilities +- TLS/SSL with modern cipher suites +- HTTP/3 preparation with QUIC support (ready for activation) + +### Configuration +- Human-friendly KDL configuration format +- Hot reload without downtime +- Environment variable substitution +- Comprehensive validation on startup + +## Why Zentinel? + +### Production-Ready +Zentinel is designed for production use from the ground up. Every feature is implemented with reliability, performance, and operational excellence in mind. + +### Type-Safe and Memory-Safe +Written in Rust, Zentinel eliminates entire classes of bugs common in traditional proxies, including buffer overflows, use-after-free errors, and data races. + +### Cloud-Native +Built for modern cloud environments with support for: +- Container deployments (Docker, Kubernetes) +- Horizontal scaling +- Service mesh integration +- Dynamic configuration + +### Developer-Friendly +- Clear, expressive configuration +- Comprehensive error messages +- Extensive documentation +- Example configurations for common use cases + +## Use Cases + +Zentinel excels in various deployment scenarios: + +- **API Gateway**: Validate requests, transform responses, implement rate limiting +- **Static Content Delivery**: Serve files directly with optimal caching headers +- **Load Balancer**: Distribute traffic across multiple upstream servers +- **Web Application Proxy**: Handle sessions, provide custom error pages, support SPAs +- **Microservices Router**: Route requests to different services based on paths +- **Edge Proxy**: Terminate SSL, implement security policies, cache responses + +## Architecture Highlights + +Zentinel leverages Pingora's battle-tested architecture while adding its own innovations: + +```text +Client Request + ↓ +[TLS Termination] + ↓ +[Route Matching] ← Service Type Detection + ↓ +[Request Processing] + ├─→ Static File Serving (no upstream) + ├─→ API Validation → Upstream + └─→ Web App Processing → Upstream + ↓ +[Response Processing] + ├─→ Error Page Generation + ├─→ Header Manipulation + └─→ Caching Headers + ↓ +Client Response +``` + +## Getting Started + +This documentation will guide you through: + +1. **[Installation](./getting-started/installation.md)** — Get Zentinel up and running +2. **[Quick Start](./getting-started/quick-start.md)** — Your first proxy configuration +3. **[Core Concepts](./concepts/architecture.md)** — Understand how Zentinel works +4. **[Configuration](./configuration/file-format.md)** — Master the configuration system +5. **[Features](./features/)** — Explore all capabilities +6. **[Deployment](./deployment/docker.md)** — Deploy to production + +## Documentation Structure + +- **[Getting Started](./getting-started/)** — Installation and basic setup +- **[Core Concepts](./concepts/)** — Fundamental architecture and design principles +- **[Configuration](./configuration/)** — Detailed configuration reference +- **[Features](./features/)** — Complete feature list with code references +- **[Agents](./agents/)** — External agent system for extensibility +- **[Operations](./operations/)** — Production management and troubleshooting +- **[Deployment](./deployment/)** — Container and cloud deployment guides +- **[Control Plane](./control-plane/)** — Fleet management, rollouts, and observability +- **[Examples](./examples/)** — Real-world configuration examples +- **[Reference](./reference/)** — Metrics, CLI, and API documentation +- **[Development](./development/)** — Contributing to Zentinel + +## Community + +- 💬 **[Discussions](https://github.com/zentinelproxy/zentinel/discussions)** — Questions, ideas, show & tell +- 🐛 **[Issues](https://github.com/zentinelproxy/zentinel/issues)** — Bug reports and feature requests +- 📦 **[GitHub](https://github.com/zentinelproxy/zentinel)** — Source code and releases + +Contributions are welcome! See our [Contributing Guide](./development/contributing.md) to get started. + +## Version Information + +This documentation covers Zentinel release **26.04** (archived). For the latest updates and changes, see the [Changelog](./appendix/changelog.md). For details on the versioning scheme, see [Versioning](./appendix/versioning.md). + +## License + +Zentinel is open-source software licensed under the Apache License, Version 2.0. See the [License](./appendix/license.md) page for details. + +--- + +Ready to get started? Head to the [Installation Guide](./getting-started/installation.md) to begin your journey with Zentinel! diff --git a/content/v/26.04/agents/_index.md b/content/v/26.04/agents/_index.md new file mode 100644 index 0000000..16cfeba --- /dev/null +++ b/content/v/26.04/agents/_index.md @@ -0,0 +1,154 @@ ++++ +title = "Agents" +weight = 8 +sort_by = "weight" +template = "section.html" ++++ + +Agents are the primary extension mechanism for Zentinel. They allow you to add custom logic, security policies, and integrations without modifying the core proxy. + +## Protocol + +Zentinel uses the **v2 agent protocol** for all agent communication: + +| Feature | Details | +|---------|---------| +| **Transports** | UDS (binary), gRPC, Reverse Connections | +| **Connection Pooling** | Multiple connections per agent with load balancing (4 strategies) | +| **Streaming** | Full bidirectional streaming support | +| **Observability** | Built-in metrics export in Prometheus format | +| **Config Push** | Dynamic configuration updates | +| **Health Tracking** | Comprehensive health checks | +| **Flow Control** | Backpressure support | +| **Request Cancellation** | Cancel in-flight requests when clients disconnect | + +See the [v2 protocol documentation](v2/) for full details. + +--- + +## What Are Agents? + +Agents are **external processes** that communicate with Zentinel over a well-defined protocol. When a request flows through Zentinel, configured agents receive events at key lifecycle points and can: + +- **Inspect** request/response headers and bodies +- **Modify** headers, routing metadata, and more +- **Decide** to allow, block, redirect, or challenge requests +- **Log** audit information for observability + +``` +┌───────────────────────────────────────────────────────────────────────────┐ +│ Zentinel Proxy │ +│ ┌─────────────────────────────────────────────────────────────────────┐ │ +│ │ Agent Manager │ │ +│ │ ┌───────────┐ ┌───────────┐ ┌───────────┐ ┌───────────┐ │ │ +│ │ │ Auth │ │ RateLimit │ │ WAF │ │ Policy │ │ │ +│ │ │ Client │ │ Client │ │ Client │ │ Client │ │ │ +│ │ └─────┬─────┘ └─────┬─────┘ └─────┬─────┘ └─────┬─────┘ │ │ +│ └────────┼──────────────┼──────────────┼──────────────┼──────────────┘ │ +└───────────┼──────────────┼──────────────┼──────────────┼─────────────────┘ + │ │ │ │ + ▼ ▼ ▼ ▼ + ┌────────────┐ ┌────────────┐ ┌────────────┐ ┌────────────┐ + │ Auth Agent │ │ RateLimit │ │ WAF Agent │ │ Policy │ + │ (local) │ │ Agent │ │ (remote) │ │ Agent │ + └────────────┘ └────────────┘ └────────────┘ └────────────┘ + UDS gRPC gRPC Reverse +``` + +## Why External Agents? + +Zentinel's architecture keeps the dataplane minimal and predictable: + +| Benefit | Description | +|---------|-------------| +| **Isolation** | A buggy or crashing agent cannot take down the proxy | +| **Independent Deployment** | Update agents without restarting Zentinel | +| **Language Flexibility** | Write agents in any language with gRPC or Unix socket support | +| **Circuit Breakers** | Zentinel protects itself from slow or failing agents | +| **Horizontal Scaling** | Run agents as separate services for high availability | + +## Transport Options + +Agents can communicate with Zentinel via multiple transports: + +| Transport | Protocol | Best For | +|-----------|----------|----------| +| **Unix Socket** | Binary + JSON | Local agents, lowest latency | +| **gRPC** | Protocol Buffers over HTTP/2 | High throughput, streaming, remote | +| **Reverse Connection** | Binary | NAT traversal, dynamic scaling | + +## Quick Configuration Example + +```kdl +agents { + // Unix socket agent with pooling + agent "auth-agent" type="auth" { + unix-socket "/var/run/zentinel/auth.sock" + connections 4 + events "request_headers" + timeout-ms 100 + failure-mode "closed" + } + + // gRPC agent + agent "waf-agent" type="waf" { + grpc "http://localhost:50051" + connections 4 + events "request_headers" "request_body" + timeout-ms 200 + failure-mode "open" + circuit-breaker { + failure-threshold 5 + timeout-seconds 30 + } + } +} + +// Reverse connection listener +reverse-listener { + path "/var/run/zentinel/agents.sock" + max-connections-per-agent 4 + handshake-timeout "10s" +} + +routes { + route "api" { + matches { path-prefix "/api/" } + upstream "backend" + agents "auth-agent" "waf-agent" + } +} +``` + +## Building Your Own Agent + +The easiest way to build a custom agent is with the **Zentinel Agent SDK**: + +```rust +use zentinel_agent_protocol::v2::{AgentPool, AgentPoolConfig}; + +let pool = AgentPool::new(); +pool.add_agent("my-agent", "/var/run/my-agent.sock").await?; + +let response = pool.send_request_headers("my-agent", &headers).await?; +``` + +The SDK provides ergonomic wrappers around the protocol, handling connection management, health tracking, and metrics automatically. + +## Documentation + +### Protocol Reference + +| Page | Description | +|------|-------------| +| [Protocol Specification](v2/protocol/) | Wire protocol, message types, streaming | +| [API Reference](v2/api/) | AgentPool, client, and server APIs | +| [Connection Pooling](v2/pooling/) | Load balancing and circuit breakers | +| [Transport Options](v2/transports/) | gRPC, UDS, and Reverse comparison | +| [Reverse Connections](v2/reverse-connections/) | NAT traversal setup | + +### Legacy (Removed) + +| Page | Description | +|------|-------------| +| [Protocol v1](v1/) | Historical v1 documentation (removed in 26.02_18) | diff --git a/content/v/26.04/agents/v1/_index.md b/content/v/26.04/agents/v1/_index.md new file mode 100644 index 0000000..0d1e91d --- /dev/null +++ b/content/v/26.04/agents/v1/_index.md @@ -0,0 +1,22 @@ ++++ +title = "Protocol v1 (Removed)" +weight = 20 +sort_by = "weight" ++++ + +{% callout(type="warning", title="V1 Protocol Removed") %} +Agent Protocol v1 was **removed** in Zentinel release 26.02_18 (February 2026). All agents must use [Protocol v2](../v2/). The documentation below is preserved for historical reference only. +{% end %} + +## Documentation (Historical) + +| Page | Description | +|------|-------------| +| [Protocol Specification](protocol/) | Wire protocol and message formats | +| [Events & Hooks](events/) | Request lifecycle events agents can handle | +| [Building Agents](building/) | How to create your own agent | +| [Transport Protocols](transports/) | Unix sockets and gRPC connectivity | + +## Migrating to v2 + +All agents must use [Protocol v2](../v2/). See the [v2 documentation](../v2/) for the current protocol specification, API reference, and transport options. diff --git a/content/v/26.04/agents/v1/building.md b/content/v/26.04/agents/v1/building.md new file mode 100644 index 0000000..e633349 --- /dev/null +++ b/content/v/26.04/agents/v1/building.md @@ -0,0 +1,712 @@ ++++ +title = "Building Agents (v1 - Removed)" +weight = 3 +updated = 2026-02-26 ++++ + +{% callout(type="warning", title="V1 Protocol Removed") %} +This page documents the **removed** v1 protocol. For current documentation, see [Building Agents (v2)](../../v2/building/). +{% end %} + +This guide covers two approaches to building Zentinel agents: + +1. **SDK (Recommended)** - High-level, ergonomic API with less boilerplate +2. **Low-level Protocol** - Direct protocol access for maximum control + +## Using the SDK (Recommended) + +The [Zentinel Agent SDK](https://github.com/zentinelproxy/zentinel-agent-rust-sdk) provides a high-level API that handles protocol details, connection management, CLI parsing, and logging automatically. + +### Add Dependency + +```toml +[dependencies] +zentinel-agent-sdk = { git = "https://github.com/zentinelproxy/zentinel-agent-rust-sdk" } +``` + +### Implement Your Agent + +```rust +use zentinel_agent_sdk::prelude::*; + +struct MyAgent; + +#[async_trait] +impl Agent for MyAgent { + async fn on_request(&self, request: &Request) -> Decision { + // Block admin paths without token + if request.path_starts_with("/admin") && request.header("x-admin-token").is_none() { + return Decision::deny().with_body("Admin access required"); + } + + // Add headers to allowed requests + Decision::allow() + .add_request_header("X-Processed-By", "my-agent") + .add_request_header("X-Client-IP", request.client_ip()) + } + + async fn on_response(&self, _request: &Request, response: &Response) -> Decision { + // Add security headers to HTML responses + if response.is_html() { + Decision::allow() + .add_response_header("X-Frame-Options", "DENY") + } else { + Decision::allow() + } + } +} + +#[tokio::main] +async fn main() -> anyhow::Result<()> { + AgentRunner::new(MyAgent) + .with_name("my-agent") + .with_socket("/tmp/my-agent.sock") + .run() + .await +} +``` + +### SDK Features + +| Type | Methods | +|------|---------| +| `Request` | `method()`, `path()`, `query("key")`, `header("name")`, `client_ip()`, `body_json::()` | +| `Response` | `status_code()`, `is_success()`, `is_html()`, `header("name")`, `body_json::()` | +| `Decision` | `allow()`, `block(status)`, `deny()`, `redirect(url)`, `add_request_header()`, `with_tag()` | +| `AgentRunner` | `with_name()`, `with_socket()`, `with_json_logs()`, `run()` | + +### Handling Configuration + +Receive configuration from the proxy's KDL config block: + +```rust +#[async_trait] +impl Agent for MyAgent { + async fn on_configure(&self, config: serde_json::Value) -> Result<(), String> { + let my_config: MyConfig = serde_json::from_value(config) + .map_err(|e| format!("Invalid config: {}", e))?; + // Store config... + Ok(()) + } +} +``` + +--- + +## Using cargo-generate (Low-level) + +For more control, use the low-level protocol directly. The fastest way to start is using `cargo-generate`: + +```bash +# Install cargo-generate +cargo install cargo-generate + +# Generate from template +cargo generate --git https://github.com/zentinelproxy/zentinel --path agent-template + +# Follow prompts for project name and description +``` + +## Manual Setup + +### 1. Create Project + +```bash +cargo new my-agent +cd my-agent +``` + +### 2. Add Dependencies + +```toml +# Cargo.toml +[package] +name = "my-agent" +version = "0.1.0" +edition = "2021" + +[dependencies] +# Zentinel agent protocol +zentinel-agent-protocol = "0.1" + +# Async runtime +tokio = { version = "1", features = ["full"] } +async-trait = "0.1" + +# CLI and configuration +clap = { version = "4", features = ["derive", "env"] } +anyhow = "1" + +# Logging +tracing = "0.1" +tracing-subscriber = { version = "0.3", features = ["json", "env-filter"] } + +# Serialization (for custom config) +serde = { version = "1", features = ["derive"] } +serde_json = "1" +``` + +### 3. Implement AgentHandler + +The core of every agent is the `AgentHandler` trait: + +```rust +use async_trait::async_trait; +use zentinel_agent_protocol::{ + AgentHandler, AgentResponse, AuditMetadata, HeaderOp, + ConfigureEvent, RequestHeadersEvent, RequestBodyChunkEvent, + ResponseHeadersEvent, ResponseBodyChunkEvent, + RequestCompleteEvent, +}; + +pub struct MyAgent { + // Your agent's state +} + +#[async_trait] +impl AgentHandler for MyAgent { + /// Called once when agent connects (optional) + async fn on_configure(&self, event: ConfigureEvent) -> AgentResponse { + // Handle configuration from KDL config block + AgentResponse::default_allow() + } + + /// Called when request headers are received + async fn on_request_headers(&self, event: RequestHeadersEvent) -> AgentResponse { + // Your logic here + AgentResponse::default_allow() + } + + /// Called for each request body chunk (optional) + async fn on_request_body_chunk(&self, event: RequestBodyChunkEvent) -> AgentResponse { + AgentResponse::default_allow() + } + + /// Called when response headers are received (optional) + async fn on_response_headers(&self, event: ResponseHeadersEvent) -> AgentResponse { + AgentResponse::default_allow() + } + + /// Called for each response body chunk (optional) + async fn on_response_body_chunk(&self, event: ResponseBodyChunkEvent) -> AgentResponse { + AgentResponse::default_allow() + } + + /// Called after request completes (optional, for logging) + async fn on_request_complete(&self, event: RequestCompleteEvent) -> AgentResponse { + AgentResponse::default_allow() + } +} +``` + +### 4. Create Main Entry Point + +```rust +use std::path::PathBuf; +use anyhow::{Context, Result}; +use clap::Parser; +use tracing::info; +use zentinel_agent_protocol::{AgentServer, GrpcAgentServer}; + +mod handler; +use handler::MyAgent; + +#[derive(Parser)] +#[command(author, version, about)] +struct Args { + /// Unix socket path + #[arg(short, long, conflicts_with = "grpc")] + socket: Option, + + /// gRPC address (e.g., "0.0.0.0:50051") + #[arg(short, long, conflicts_with = "socket")] + grpc: Option, + + /// Log level + #[arg(short, long, default_value = "info")] + log_level: String, +} + +#[tokio::main] +async fn main() -> Result<()> { + let args = Args::parse(); + + // Initialize logging + tracing_subscriber::fmt() + .with_env_filter(&args.log_level) + .json() + .init(); + + let agent = Box::new(MyAgent::new()); + + match (&args.socket, &args.grpc) { + (Some(socket), None) => { + info!("Starting agent on Unix socket: {:?}", socket); + let server = AgentServer::new("my-agent", socket, agent); + server.run().await.context("Server failed")?; + } + (None, Some(addr)) => { + info!("Starting agent on gRPC: {}", addr); + let server = GrpcAgentServer::new("my-agent", agent); + server.run(addr.parse()?).await.context("gRPC server failed")?; + } + _ => { + // Default to Unix socket + let socket = PathBuf::from("/tmp/my-agent.sock"); + info!("Starting agent on default socket: {:?}", socket); + let server = AgentServer::new("my-agent", socket, agent); + server.run().await.context("Server failed")?; + } + } + + Ok(()) +} +``` + +## Echo Agent Deep Dive + +The Echo Agent is a complete reference implementation. Let's examine its key components. + +### Agent Structure + +```rust +pub struct EchoAgent { + /// Header prefix for echo headers + prefix: String, + /// Verbose mode flag + verbose: bool, + /// Request counter for tracking + request_count: std::sync::atomic::AtomicU64, +} + +impl EchoAgent { + pub fn new(prefix: String, verbose: bool) -> Self { + Self { + prefix, + verbose, + request_count: std::sync::atomic::AtomicU64::new(0), + } + } +} +``` + +### Handling Request Headers + +```rust +#[async_trait] +impl AgentHandler for EchoAgent { + async fn on_request_headers(&self, event: RequestHeadersEvent) -> AgentResponse { + // Increment request counter + let request_num = self.request_count + .fetch_add(1, std::sync::atomic::Ordering::Relaxed) + 1; + + // Log the event + tracing::debug!( + correlation_id = %event.metadata.correlation_id, + method = %event.method, + uri = %event.uri, + "Processing request" + ); + + // Build response with header mutations + let mut response = AgentResponse::default_allow(); + + // Add echo headers + response = response + .add_request_header(HeaderOp::Set { + name: format!("{}Agent", self.prefix), + value: "echo-agent/1.0".to_string(), + }) + .add_request_header(HeaderOp::Set { + name: format!("{}Correlation-Id", self.prefix), + value: event.metadata.correlation_id.clone(), + }) + .add_request_header(HeaderOp::Set { + name: format!("{}Method", self.prefix), + value: event.method.clone(), + }) + .add_request_header(HeaderOp::Set { + name: format!("{}Path", self.prefix), + value: event.uri.clone(), + }); + + // Add audit metadata + let mut audit = AuditMetadata::default(); + audit.tags = vec!["echo".to_string()]; + audit.custom.insert( + "request_num".to_string(), + serde_json::Value::Number(request_num.into()), + ); + + response.with_audit(audit) + } +} +``` + +### Blocking Requests + +To block a request, return a block decision: + +```rust +async fn on_request_headers(&self, event: RequestHeadersEvent) -> AgentResponse { + // Check for blocked paths + if event.uri.starts_with("/admin") { + return AgentResponse::block(403, Some("Forbidden".to_string())) + .with_audit(AuditMetadata { + tags: vec!["blocked".to_string()], + reason_codes: vec!["ADMIN_PATH".to_string()], + ..Default::default() + }); + } + + // Check for blocked IPs + if self.blocked_ips.contains(&event.metadata.client_ip) { + return AgentResponse::block(403, Some("IP Blocked".to_string())); + } + + AgentResponse::default_allow() +} +``` + +### Redirecting Requests + +```rust +async fn on_request_headers(&self, event: RequestHeadersEvent) -> AgentResponse { + // Redirect unauthenticated users + if !event.headers.contains_key("authorization") { + return AgentResponse::redirect( + "https://login.example.com/auth".to_string(), + 302, + ); + } + + AgentResponse::default_allow() +} +``` + +### Handling Configuration + +Implement `on_configure()` to receive configuration from the proxy's KDL config: + +```rust +use std::sync::RwLock; +use serde::{Deserialize, Serialize}; + +// Define your config struct with kebab-case for KDL compatibility +#[derive(Debug, Clone, Serialize, Deserialize, Default)] +#[serde(rename_all = "kebab-case")] +pub struct MyAgentConfig { + #[serde(default)] + pub enabled: bool, + pub threshold: Option, + #[serde(default)] + pub allowed_paths: Vec, +} + +pub struct MyAgent { + config: RwLock, +} + +impl MyAgent { + pub fn new() -> Self { + Self { + config: RwLock::new(MyAgentConfig::default()), + } + } +} + +#[async_trait] +impl AgentHandler for MyAgent { + async fn on_configure(&self, event: ConfigureEvent) -> AgentResponse { + // Parse the JSON config into your struct + match serde_json::from_value::(event.config) { + Ok(new_config) => { + // Update the agent's configuration + if let Ok(mut config) = self.config.write() { + *config = new_config; + tracing::info!("Agent configured successfully"); + } + AgentResponse::default_allow() + } + Err(e) => { + tracing::error!("Invalid configuration: {}", e); + // Reject invalid config - proxy won't route to this agent + AgentResponse::block(500, Some(format!("Invalid config: {}", e))) + } + } + } + + async fn on_request_headers(&self, event: RequestHeadersEvent) -> AgentResponse { + // Read the current config + let config = self.config.read().unwrap(); + + if !config.enabled { + return AgentResponse::default_allow(); + } + + // Use config values in your logic + if config.allowed_paths.iter().any(|p| event.uri.starts_with(p)) { + return AgentResponse::default_allow(); + } + + // ... rest of your logic + AgentResponse::default_allow() + } +} +``` + +The corresponding KDL configuration: + +```kdl +agent "my-agent" type="custom" { + unix-socket "/tmp/my-agent.sock" + events "request_headers" + config { + enabled #true + threshold 100 + allowed-paths "/health" "/metrics" "/api/public" + } +} +``` + +**Key points:** + +- Use `#[serde(rename_all = "kebab-case")]` to match KDL naming conventions +- Use `#[serde(default)]` for optional fields with defaults +- Wrap mutable config in `RwLock` for thread-safe updates +- Return a block decision to reject invalid configurations +- CLI args can serve as fallback when no config block is present + +## Running the Agent + +### Unix Socket Mode + +```bash +# Build +cargo build --release + +# Run +./target/release/my-agent --socket /tmp/my-agent.sock +``` + +### gRPC Mode + +```bash +./target/release/my-agent --grpc 0.0.0.0:50051 +``` + +### Docker Deployment + +```dockerfile +FROM rust:1.75-slim AS builder +WORKDIR /app +COPY . . +RUN cargo build --release + +FROM debian:bookworm-slim +RUN apt-get update && apt-get install -y ca-certificates && rm -rf /var/lib/apt/lists/* +COPY --from=builder /app/target/release/my-agent /usr/local/bin/ +USER nobody +ENTRYPOINT ["my-agent"] +CMD ["--grpc", "0.0.0.0:50051"] +``` + +### Systemd Service + +```ini +# /etc/systemd/system/my-agent.service +[Unit] +Description=My Zentinel Agent +After=network.target + +[Service] +Type=simple +User=zentinel +ExecStart=/usr/local/bin/my-agent --socket /var/run/zentinel/my-agent.sock +Restart=always +RestartSec=5 + +[Install] +WantedBy=multi-user.target +``` + +## Proxy Configuration + +Configure Zentinel to use your agent: + +```kdl +agents { + agent "my-agent" type="custom" { + unix-socket "/var/run/zentinel/my-agent.sock" + // Or for gRPC: + // grpc "http://localhost:50051" + + events "request_headers" + timeout-ms 100 + failure-mode "open" + } +} + +routes { + route "api" { + matches { path-prefix "/api/" } + upstream "backend" + agents "my-agent" + } +} +``` + +## Testing Your Agent + +### Unit Tests + +```rust +#[cfg(test)] +mod tests { + use super::*; + use zentinel_agent_protocol::Decision; + + #[tokio::test] + async fn test_allows_normal_requests() { + let agent = MyAgent::new(); + let event = RequestHeadersEvent { + metadata: RequestMetadata { + correlation_id: "test-123".to_string(), + client_ip: "127.0.0.1".to_string(), + ..Default::default() + }, + method: "GET".to_string(), + uri: "/api/users".to_string(), + headers: HashMap::new(), + }; + + let response = agent.on_request_headers(event).await; + assert_eq!(response.decision, Decision::Allow); + } + + #[tokio::test] + async fn test_blocks_admin_path() { + let agent = MyAgent::new(); + let event = RequestHeadersEvent { + method: "GET".to_string(), + uri: "/admin/secret".to_string(), + ..Default::default() + }; + + let response = agent.on_request_headers(event).await; + match response.decision { + Decision::Block { status, .. } => assert_eq!(status, 403), + _ => panic!("Expected block decision"), + } + } +} +``` + +### Integration Testing + +Test with the actual protocol using grpcurl: + +```bash +# Start your agent +./my-agent --grpc 127.0.0.1:50051 & + +# Test with grpcurl +grpcurl -plaintext \ + -import-path ./proto -proto agent.proto \ + -d '{ + "version": 1, + "event_type": "EVENT_TYPE_REQUEST_HEADERS", + "request_headers": { + "metadata": {"correlation_id": "test-123", "client_ip": "127.0.0.1"}, + "method": "GET", + "uri": "/api/test" + } + }' \ + 127.0.0.1:50051 zentinel.agent.v1.AgentProcessor/ProcessEvent +``` + +## Best Practices + +### Performance + +1. **Keep handlers fast** - Agents add latency to every request +2. **Use async I/O** - Never block the event loop +3. **Pre-compile patterns** - Compile regexes at startup +4. **Limit body inspection** - Only inspect when necessary + +### Reliability + +1. **Handle errors gracefully** - Return allow/block, don't panic +2. **Configure timeouts** - The proxy will timeout slow agents +3. **Use structured logging** - Include correlation IDs +4. **Export metrics** - Prometheus metrics for observability + +### Security + +1. **Validate all input** - Don't trust data from the proxy +2. **Minimize dependencies** - Fewer deps = smaller attack surface +3. **Keep secrets secure** - Use environment variables +4. **Audit regularly** - Run `cargo audit` in CI + +## Building Agents in Other Languages + +With gRPC support, you can build agents in any language. See the [Protocol Specification](protocol/) for the protobuf definitions. + +### Python Example + +```python +import grpc +from concurrent import futures +import agent_pb2 +import agent_pb2_grpc + +class MyAgent(agent_pb2_grpc.AgentProcessorServicer): + def ProcessEvent(self, request, context): + if request.event_type == agent_pb2.EVENT_TYPE_REQUEST_HEADERS: + headers = request.request_headers + # Your logic here + return agent_pb2.AgentResponse( + version=1, + allow=agent_pb2.AllowDecision() + ) + return agent_pb2.AgentResponse(version=1, allow=agent_pb2.AllowDecision()) + +server = grpc.server(futures.ThreadPoolExecutor(max_workers=10)) +agent_pb2_grpc.add_AgentProcessorServicer_to_server(MyAgent(), server) +server.add_insecure_port('[::]:50051') +server.start() +server.wait_for_termination() +``` + +### Go Example + +```go +package main + +import ( + "context" + "net" + pb "github.com/your-org/zentinel-proto" + "google.golang.org/grpc" +) + +type myAgent struct { + pb.UnimplementedAgentProcessorServer +} + +func (a *myAgent) ProcessEvent(ctx context.Context, req *pb.AgentRequest) (*pb.AgentResponse, error) { + return &pb.AgentResponse{ + Version: 1, + Decision: &pb.AgentResponse_Allow{ + Allow: &pb.AllowDecision{}, + }, + }, nil +} + +func main() { + lis, _ := net.Listen("tcp", ":50051") + s := grpc.NewServer() + pb.RegisterAgentProcessorServer(s, &myAgent{}) + s.Serve(lis) +} +``` diff --git a/content/v/26.04/agents/v1/events.md b/content/v/26.04/agents/v1/events.md new file mode 100644 index 0000000..463bc30 --- /dev/null +++ b/content/v/26.04/agents/v1/events.md @@ -0,0 +1,465 @@ ++++ +title = "Events & Hooks (v1 - Removed)" +weight = 2 +updated = 2026-02-26 ++++ + +{% callout(type="warning", title="V1 Protocol Removed") %} +This page documents the **removed** v1 protocol. For current documentation, see [Events & Hooks (v2)](../../v2/events/). +{% end %} + +Agents receive events at key points in the request/response lifecycle. Each event carries relevant data and expects a response with a decision and optional mutations. + +## Event Overview + +| Event | Phase | Can Block | Can Mutate | Use Cases | +|-------|-------|-----------|------------|-----------| +| `configure` | Startup | Yes | None | Agent configuration | +| `request_headers` | Request | Yes | Request headers | Auth, routing, early blocking | +| `request_body` | Request | Yes | Request headers | WAF inspection, content validation | +| `response_headers` | Response | No | Response headers | Header injection, caching hints | +| `response_body` | Response | No | Response headers | Content filtering, transformation | +| `request_complete` | Logging | No | None | Audit logging, metrics | + +## Event Lifecycle + +``` +┌─────────────────────────────────────────────────────────────────────┐ +│ REQUEST PHASE │ +├─────────────────────────────────────────────────────────────────────┤ +│ │ +│ Client ──▶ [request_headers] ──▶ [request_body] ──▶ Upstream │ +│ │ │ │ +│ Decision: Decision: │ +│ ALLOW/BLOCK/REDIRECT ALLOW/BLOCK │ +│ │ +├─────────────────────────────────────────────────────────────────────┤ +│ RESPONSE PHASE │ +├─────────────────────────────────────────────────────────────────────┤ +│ │ +│ Client ◀── [response_headers] ◀── [response_body] ◀── Upstream │ +│ │ │ │ +│ Mutations: Mutations: │ +│ Add/Set/Remove headers Add/Set/Remove headers │ +│ │ +├─────────────────────────────────────────────────────────────────────┤ +│ LOGGING PHASE │ +├─────────────────────────────────────────────────────────────────────┤ +│ │ +│ [request_complete] ──▶ Audit Log │ +│ │ +└─────────────────────────────────────────────────────────────────────┘ +``` + +## Configure Event + +**Event Type:** `configure` + +Sent once when the agent connects to the proxy, before any request events. This allows agents to receive configuration from the KDL config file instead of relying solely on CLI arguments. + +### Payload + +```rust +struct ConfigureEvent { + agent_id: String, // Agent identifier from config + config: serde_json::Value, // Configuration as JSON object +} +``` + +### Configuration Source + +The configuration comes from the `config` block in KDL: + +```kdl +agent "waf" type="waf" { + unix-socket "/var/run/zentinel/waf.sock" + events "request_headers" "request_body" + config { + paranoia-level 2 + sqli #true + xss #true + exclude-paths "/health" "/metrics" + } +} +``` + +This becomes: + +```json +{ + "agent_id": "waf", + "config": { + "paranoia-level": 2, + "sqli": true, + "xss": true, + "exclude-paths": ["/health", "/metrics"] + } +} +``` + +### Use Cases + +- **Dynamic Configuration:** Apply settings without restarting the agent +- **Centralized Config:** Keep all configuration in one KDL file +- **Environment-Specific Settings:** Different configs for dev/staging/prod + +### Example Response + +```json +{ + "version": 1, + "decision": {"allow": {}}, + "audit": { + "tags": ["configured"], + "custom": {"paranoia_level": "2"} + } +} +``` + +### Rejecting Configuration + +If the configuration is invalid, the agent can reject it: + +```json +{ + "version": 1, + "decision": { + "block": { + "status": 500, + "body": "Invalid config: paranoia-level must be 1-4" + } + } +} +``` + +When configuration is rejected, the proxy will not start routing traffic to that agent. + +### KDL to JSON Conversion + +| KDL | JSON | +|-----|------| +| `paranoia-level 2` | `{"paranoia-level": 2}` | +| `sqli #true` | `{"sqli": true}` | +| `paths "/a" "/b"` | `{"paths": ["/a", "/b"]}` | +| `nested { key "val" }` | `{"nested": {"key": "val"}}` | + +## Request Headers Event + +**Event Type:** `request_headers` + +The most commonly used event. Sent when HTTP headers are received from the client, before the body is read. + +### Payload + +```rust +struct RequestHeadersEvent { + metadata: RequestMetadata, + method: String, // "GET", "POST", etc. + uri: String, // "/api/users?page=1" + headers: HashMap>, +} + +struct RequestMetadata { + correlation_id: String, // Unique request identifier + request_id: String, // Internal request ID + client_ip: String, // Client IP address + client_port: u16, // Client port + server_name: Option, // SNI or Host header + protocol: String, // "HTTP/1.1", "HTTP/2" + tls_version: Option, // "TLSv1.3" + tls_cipher: Option, // Cipher suite + route_id: Option, // Matched route ID + upstream_id: Option, // Target upstream + timestamp: String, // RFC3339 timestamp +} +``` + +### Use Cases + +- **Authentication:** Validate JWT tokens, API keys, session cookies +- **Authorization:** Check permissions based on path and headers +- **Rate Limiting:** Count requests per client/route +- **Routing Decisions:** Modify routing metadata +- **Early Blocking:** Reject malformed or suspicious requests + +### Example Response + +```json +{ + "version": 1, + "decision": {"allow": {}}, + "request_headers": [ + {"set": {"name": "X-User-Id", "value": "user-123"}}, + {"set": {"name": "X-Authenticated", "value": "true"}} + ], + "audit": { + "tags": ["auth", "jwt"], + "custom": {"user_id": "user-123"} + } +} +``` + +## Request Body Event + +**Event Type:** `request_body` + +Sent when request body chunks are received. Requires `request_body` in the agent's event list and appropriate body limits configured. + +### Payload + +```rust +struct RequestBodyChunkEvent { + correlation_id: String, + data: String, // Body chunk (base64 for binary) + is_last: bool, // True if final chunk + total_size: Option, // Total body size if known +} +``` + +### Configuration + +```kdl +agent "waf" type="waf" { + grpc "http://localhost:50051" + events "request_headers" "request_body" + max-request-body-bytes 1048576 // Limit to 1MB +} +``` + +### Use Cases + +- **WAF Inspection:** Scan for SQL injection, XSS, command injection +- **Content Validation:** Verify JSON schema, file types +- **Size Limits:** Enforce body size restrictions +- **Malware Scanning:** Check uploaded files + +### Body Decompression + +When `decompress: true` is set in the WAF `body-inspection` config, Zentinel automatically decompresses request bodies before sending to agents: + +```kdl +waf { + body-inspection { + inspect-request-body #true + decompress #true + max-decompression-ratio 100.0 // Zip bomb protection + } +} +``` + +Supported encodings: `gzip`, `deflate`, `br` (Brotli) + +The decompression ratio limit protects against zip bombs by rejecting payloads where the decompressed size exceeds the compressed size by more than the configured ratio. + +### Important Notes + +- Body inspection adds latency - use only when necessary +- Set `max-request-body-bytes` to limit memory usage +- Streaming bodies may arrive in multiple chunks +- Enable `decompress` to inspect compressed payloads (e.g., gzipped JSON) +- Use `max-decompression-ratio` to protect against zip bomb attacks + +## Response Headers Event + +**Event Type:** `response_headers` + +Sent when response headers are received from the upstream, before the body. + +### Payload + +```rust +struct ResponseHeadersEvent { + correlation_id: String, + status: u16, // HTTP status code + headers: HashMap>, +} +``` + +### Use Cases + +- **Header Injection:** Add security headers, CORS headers +- **Caching Hints:** Modify cache-control headers +- **Response Logging:** Record upstream response status +- **Header Removal:** Strip internal headers + +### Example Response + +```json +{ + "version": 1, + "decision": {"allow": {}}, + "response_headers": [ + {"set": {"name": "X-Frame-Options", "value": "DENY"}}, + {"set": {"name": "X-Content-Type-Options", "value": "nosniff"}}, + {"remove": {"name": "X-Powered-By"}} + ] +} +``` + +## Response Body Event + +**Event Type:** `response_body` + +Sent when response body chunks are received from the upstream. + +### Payload + +```rust +struct ResponseBodyChunkEvent { + correlation_id: String, + data: String, // Body chunk (base64 for binary) + is_last: bool, + total_size: Option, +} +``` + +### Configuration + +```kdl +agent "content-filter" type="custom" { + unix-socket "/tmp/filter.sock" + events "response_body" + max-response-body-bytes 5242880 // Limit to 5MB +} +``` + +### Use Cases + +- **Content Filtering:** Redact sensitive data +- **Response Transformation:** Modify response content +- **DLP (Data Loss Prevention):** Detect sensitive data leakage +- **Logging:** Record response content for audit + +## Request Complete Event + +**Event Type:** `request_complete` (also known as `log`) + +Sent after the response has been sent to the client. This is a **fire-and-forget** event for logging and audit purposes. + +### Payload + +```rust +struct RequestCompleteEvent { + correlation_id: String, + status: u16, // Final HTTP status + duration_ms: u64, // Total request duration + request_body_size: usize, // Bytes received + response_body_size: usize, // Bytes sent + upstream_attempts: u32, // Retry count + error: Option, // Error message if failed +} +``` + +### Use Cases + +- **Audit Logging:** Record all requests for compliance +- **Metrics Collection:** Track latency, status codes, sizes +- **Alerting:** Trigger alerts on errors or anomalies +- **Analytics:** Feed data to analytics systems + +### Example Response + +The response decision is ignored for this event, but audit metadata is still collected: + +```json +{ + "version": 1, + "decision": {"allow": {}}, + "audit": { + "tags": ["api", "success"], + "rule_ids": [], + "custom": { + "response_time_bucket": "fast", + "cache_hit": "false" + } + } +} +``` + +## Agent Decisions + +Agents return one of these decisions: + +| Decision | Description | Applicable Events | +|----------|-------------|-------------------| +| `allow` | Continue processing | All | +| `block` | Reject with status code and optional body | `request_headers`, `request_body` | +| `redirect` | Redirect to URL | `request_headers` | +| `challenge` | Present challenge (CAPTCHA, etc.) | `request_headers` | + +### Block Response + +```json +{ + "decision": { + "block": { + "status": 403, + "body": "Access Denied", + "headers": {"X-Block-Reason": "rate-limit"} + } + } +} +``` + +### Redirect Response + +```json +{ + "decision": { + "redirect": { + "url": "https://login.example.com/auth", + "status": 302 + } + } +} +``` + +## Header Mutations + +Agents can mutate headers using these operations: + +| Operation | Description | +|-----------|-------------| +| `set` | Set header value (replaces if exists) | +| `add` | Add header value (appends if exists) | +| `remove` | Remove header entirely | + +```json +{ + "request_headers": [ + {"set": {"name": "X-Forwarded-User", "value": "alice"}}, + {"add": {"name": "X-Request-Tag", "value": "processed"}}, + {"remove": {"name": "X-Internal-Token"}} + ], + "response_headers": [ + {"set": {"name": "Cache-Control", "value": "no-store"}} + ] +} +``` + +## Audit Metadata + +Every response can include audit metadata for logging and observability: + +```json +{ + "audit": { + "tags": ["waf", "blocked", "sqli"], + "rule_ids": ["942100", "942110"], + "confidence": 0.95, + "reason_codes": ["SQL_INJECTION_DETECTED"], + "custom": { + "matched_pattern": "' OR 1=1", + "source_field": "query_param:id" + } + } +} +``` + +| Field | Description | +|-------|-------------| +| `tags` | Searchable tags for filtering logs | +| `rule_ids` | IDs of rules that matched (e.g., CRS rules) | +| `confidence` | Confidence score (0.0 - 1.0) | +| `reason_codes` | Machine-readable reason codes | +| `custom` | Arbitrary key-value metadata | diff --git a/content/v/26.04/agents/v1/protocol.md b/content/v/26.04/agents/v1/protocol.md new file mode 100644 index 0000000..a3eef2c --- /dev/null +++ b/content/v/26.04/agents/v1/protocol.md @@ -0,0 +1,604 @@ ++++ +title = "Protocol Specification (v1 - Removed)" +weight = 5 +updated = 2026-02-26 ++++ + +{% callout(type="warning", title="V1 Protocol Removed") %} +This page documents the **removed** v1 protocol. For the current protocol specification, see [Protocol v2](../../v2/protocol/). +{% end %} + +This document defines the Zentinel Agent Protocol v1—the wire format for communication between Zentinel and external agents. + +## Overview + +The protocol supports two encodings: + +| Transport | Encoding | Schema | +|-----------|----------|--------| +| Unix Socket | JSON | Informal (see below) | +| gRPC | Protocol Buffers | `zentinel.agent.v1` | + +Both encodings represent the same logical protocol. Agents can implement either or both. + +--- + +## Protocol Buffers Definition + +```protobuf +// Zentinel Agent Protocol - gRPC Definition +// Package: zentinel.agent.v1 + +syntax = "proto3"; +package zentinel.agent.v1; + +// ============================================================================ +// Event Types +// ============================================================================ + +enum EventType { + EVENT_TYPE_UNSPECIFIED = 0; + EVENT_TYPE_CONFIGURE = 1; // Agent configuration + EVENT_TYPE_REQUEST_HEADERS = 2; + EVENT_TYPE_REQUEST_BODY_CHUNK = 3; + EVENT_TYPE_RESPONSE_HEADERS = 4; + EVENT_TYPE_RESPONSE_BODY_CHUNK = 5; + EVENT_TYPE_REQUEST_COMPLETE = 6; +} + +// ============================================================================ +// Request Metadata +// ============================================================================ + +message RequestMetadata { + string correlation_id = 1; // Unique ID for request correlation + string request_id = 2; // Internal request ID + string client_ip = 3; // Client IP address + uint32 client_port = 4; // Client port + optional string server_name = 5; // SNI or Host header + string protocol = 6; // "HTTP/1.1", "HTTP/2", etc. + optional string tls_version = 7; // "TLSv1.3", etc. + optional string tls_cipher = 8; // Cipher suite + optional string route_id = 9; // Matched route ID + optional string upstream_id = 10; // Target upstream + string timestamp = 11; // RFC3339 timestamp + optional string traceparent = 12; // W3C Trace Context header +} + +// ============================================================================ +// Event Messages +// ============================================================================ + +// Sent once when agent connects, before any request events +message ConfigureEvent { + string agent_id = 1; // Agent identifier from config + string config_json = 2; // Configuration as JSON string +} + +// Header values (supports multiple values per header name) +message HeaderValues { + repeated string values = 1; +} + +// Sent when HTTP request headers are received +message RequestHeadersEvent { + RequestMetadata metadata = 1; + string method = 2; // GET, POST, etc. + string uri = 3; // /path?query + map headers = 4; +} + +// Sent for each request body chunk +message RequestBodyChunkEvent { + string correlation_id = 1; + bytes data = 2; // Raw bytes + bool is_last = 3; // True if final chunk + optional uint64 total_size = 4; // Total body size if known +} + +// Sent when upstream response headers are received +message ResponseHeadersEvent { + string correlation_id = 1; + uint32 status = 2; // HTTP status code + map headers = 3; +} + +// Sent for each response body chunk +message ResponseBodyChunkEvent { + string correlation_id = 1; + bytes data = 2; // Raw bytes + bool is_last = 3; + optional uint64 total_size = 4; +} + +// Sent after response completes (for logging) +message RequestCompleteEvent { + string correlation_id = 1; + uint32 status = 2; // Final HTTP status + uint64 duration_ms = 3; // Total request duration + uint64 request_body_size = 4; // Bytes received + uint64 response_body_size = 5; // Bytes sent + uint32 upstream_attempts = 6; // Retry count + optional string error = 7; // Error message if failed +} + +// ============================================================================ +// Header Operations +// ============================================================================ + +message HeaderOp { + oneof operation { + SetHeader set = 1; + AddHeader add = 2; + RemoveHeader remove = 3; + } +} + +message SetHeader { + string name = 1; + string value = 2; +} + +message AddHeader { + string name = 1; + string value = 2; +} + +message RemoveHeader { + string name = 1; +} + +// ============================================================================ +// Audit Metadata +// ============================================================================ + +message AuditMetadata { + repeated string tags = 1; // Searchable tags + repeated string rule_ids = 2; // Matched rule IDs + optional float confidence = 3; // Confidence score (0.0-1.0) + repeated string reason_codes = 4; // Machine-readable codes + map custom = 5; // Arbitrary key-value data +} + +// ============================================================================ +// Decision Types +// ============================================================================ + +message AllowDecision { + // Empty - request proceeds +} + +message BlockDecision { + uint32 status = 1; // HTTP status code (e.g., 403) + optional string body = 2; // Response body + map headers = 3; // Response headers +} + +message RedirectDecision { + string url = 1; // Target URL + uint32 status = 2; // 301, 302, 307, or 308 +} + +message ChallengeDecision { + string challenge_type = 1; // "captcha", "javascript", etc. + map params = 2; // Challenge parameters +} + +// ============================================================================ +// Request/Response Wrappers +// ============================================================================ + +message AgentRequest { + uint32 version = 1; // Protocol version (1) + EventType event_type = 2; + + oneof event { + ConfigureEvent configure = 9; + RequestHeadersEvent request_headers = 10; + RequestBodyChunkEvent request_body_chunk = 11; + ResponseHeadersEvent response_headers = 12; + ResponseBodyChunkEvent response_body_chunk = 13; + RequestCompleteEvent request_complete = 14; + } +} + +message AgentResponse { + uint32 version = 1; // Protocol version (1) + + oneof decision { + AllowDecision allow = 2; + BlockDecision block = 3; + RedirectDecision redirect = 4; + ChallengeDecision challenge = 5; + } + + repeated HeaderOp request_headers = 10; // Request header mutations + repeated HeaderOp response_headers = 11; // Response header mutations + map routing_metadata = 12; // Routing hints + optional AuditMetadata audit = 13; // Logging metadata +} + +// ============================================================================ +// Service Definition +// ============================================================================ + +service AgentProcessor { + // Process a single event + rpc ProcessEvent(AgentRequest) returns (AgentResponse); + + // Bidirectional streaming for body inspection + rpc ProcessEventStream(stream AgentRequest) returns (AgentResponse); +} +``` + +--- + +## JSON Schema (Unix Socket) + +For Unix socket transport, messages use JSON with the following schema: + +### AgentRequest + +```json +{ + "$schema": "http://json-schema.org/draft-07/schema#", + "type": "object", + "required": ["version", "event_type", "payload"], + "properties": { + "version": { + "type": "integer", + "const": 1 + }, + "event_type": { + "type": "string", + "enum": [ + "configure", + "request_headers", + "request_body_chunk", + "response_headers", + "response_body_chunk", + "request_complete" + ] + }, + "payload": { + "type": "object", + "description": "Event-specific payload" + } + } +} +``` + +### Event Payloads + +**ConfigureEvent:** + +```json +{ + "agent_id": "waf-agent", + "config": { + "paranoia-level": 2, + "sqli": true, + "xss": true, + "exclude-paths": ["/health", "/metrics"] + } +} +``` + +**RequestHeadersEvent:** + +```json +{ + "metadata": { + "correlation_id": "string", + "request_id": "string", + "client_ip": "string", + "client_port": 12345, + "server_name": "string|null", + "protocol": "HTTP/1.1|HTTP/2", + "tls_version": "string|null", + "tls_cipher": "string|null", + "route_id": "string|null", + "upstream_id": "string|null", + "timestamp": "2025-12-29T08:00:00Z", + "traceparent": "00-0af7651916cd43dd8448eb211c80319c-b7ad6b7169203331-01|null" + }, + "method": "GET|POST|...", + "uri": "/path?query", + "headers": { + "header-name": ["value1", "value2"] + } +} +``` + +> **Note**: The `traceparent` field contains the W3C Trace Context header when distributed tracing is enabled. Agents can use this to create child spans for their processing. Format: `{version}-{trace-id}-{span-id}-{flags}`. + +**RequestBodyChunkEvent:** + +```json +{ + "correlation_id": "string", + "data": "base64-encoded-data", + "is_last": true, + "total_size": 1234 +} +``` + +**ResponseHeadersEvent:** + +```json +{ + "correlation_id": "string", + "status": 200, + "headers": { + "content-type": ["application/json"] + } +} +``` + +**ResponseBodyChunkEvent:** + +```json +{ + "correlation_id": "string", + "data": "base64-encoded-data", + "is_last": true, + "total_size": 5678 +} +``` + +**RequestCompleteEvent:** + +```json +{ + "correlation_id": "string", + "status": 200, + "duration_ms": 150, + "request_body_size": 1024, + "response_body_size": 2048, + "upstream_attempts": 1, + "error": #null +} +``` + +### AgentResponse + +```json +{ + "version": 1, + "decision": { + "allow": {} + }, + "request_headers": [ + {"set": {"name": "X-Header", "value": "value"}}, + {"add": {"name": "X-Tag", "value": "processed"}}, + {"remove": {"name": "X-Internal"}} + ], + "response_headers": [], + "routing_metadata": {}, + "audit": { + "tags": ["auth", "success"], + "rule_ids": [], + "confidence": 0.95, + "reason_codes": ["AUTH_SUCCESS"], + "custom": { + "user_id": "user-123" + } + } +} +``` + +### Decision Types (JSON) + +**Allow:** +```json +{"decision": {"allow": {}}} +``` + +**Block:** +```json +{ + "decision": { + "block": { + "status": 403, + "body": "Access Denied", + "headers": {"X-Block-Reason": "rate-limit"} + } + } +} +``` + +**Redirect:** +```json +{ + "decision": { + "redirect": { + "url": "https://login.example.com/auth", + "status": 302 + } + } +} +``` + +**Challenge:** +```json +{ + "decision": { + "challenge": { + "challenge_type": "captcha", + "params": { + "site_key": "abc123", + "action": "login" + } + } + } +} +``` + +--- + +## Protocol Version + +Current version: **1** + +The `version` field in requests and responses allows for future protocol evolution: + +```json +{"version": 1, ...} +``` + +Agents should reject requests with unsupported versions. + +--- + +## Header Operations + +Three operations are supported for header mutation: + +| Operation | Description | Example | +|-----------|-------------|---------| +| `set` | Set value (replaces existing) | `{"set": {"name": "X-User", "value": "alice"}}` | +| `add` | Add value (appends) | `{"add": {"name": "X-Tag", "value": "processed"}}` | +| `remove` | Remove header entirely | `{"remove": {"name": "X-Internal"}}` | + +### Mutation Ordering + +1. All `remove` operations execute first +2. Then all `set` operations +3. Finally all `add` operations + +This ensures predictable behavior regardless of the order in the array. + +--- + +## Error Handling + +### Protocol Errors + +| Error | Response | +|-------|----------| +| Malformed JSON | Connection closed | +| Unknown event_type | 400 Bad Request (gRPC: INVALID_ARGUMENT) | +| Missing required field | 400 Bad Request (gRPC: INVALID_ARGUMENT) | +| Message too large | Connection closed | +| Version mismatch | 400 Bad Request (gRPC: INVALID_ARGUMENT) | + +### Timeout Behavior + +When an agent times out, Zentinel applies the configured `failure-mode`: + +- `failure-mode "open"` → Allow request +- `failure-mode "closed"` → Block request (503) + +--- + +## Correlation ID + +The `correlation_id` field links all events for a single HTTP request: + +``` +request_headers ─┐ +request_body[0] ─┤ +request_body[1] ─┼── Same correlation_id +response_headers ─┤ +request_complete ─┘ +``` + +Use this ID for: +- Log correlation across events +- Stateful body inspection (accumulating chunks) +- Request tracing + +--- + +## Body Inspection + +Body chunks are sent incrementally with `is_last` indicating the final chunk: + +``` +┌──────────────┐ ┌──────────────┐ ┌──────────────┐ +│ chunk 1 │ → │ chunk 2 │ → │ chunk 3 │ +│ is_last: F │ │ is_last: F │ │ is_last: T │ +└──────────────┘ └──────────────┘ └──────────────┘ +``` + +For streaming inspection, use `ProcessEventStream`: + +1. Headers event sent first +2. Body chunks streamed +3. Single response returned after all chunks processed + +--- + +## Limits + +| Limit | Value | Notes | +|-------|-------|-------| +| Max message size | 16 MB | Per individual message | +| Max header name | 8 KB | | +| Max header value | 64 KB | Per value | +| Max headers per request | 100 | | +| Max body chunk size | 1 MB | Recommended | + +--- + +## Versioning Strategy + +Future protocol changes will follow semantic versioning: + +- **Patch** (1.0.x): Bug fixes, no schema changes +- **Minor** (1.x.0): Additive changes (new optional fields) +- **Major** (x.0.0): Breaking changes (new required fields, removed fields) + +Agents should: +1. Accept unknown fields gracefully +2. Reject requests with major version mismatch +3. Handle missing optional fields with defaults + +--- + +## Generating Code + +### Rust (tonic) + +```bash +# Build script (build.rs) +tonic_build::compile_protos("proto/agent.proto")?; +``` + +### Go + +```bash +protoc --go_out=. --go-grpc_out=. agent.proto +``` + +### Python + +```bash +python -m grpc_tools.protoc \ + -I. \ + --python_out=. \ + --grpc_python_out=. \ + agent.proto +``` + +### TypeScript (Node.js) + +```bash +npm install @grpc/grpc-js @grpc/proto-loader +``` + +```typescript +import * as grpc from '@grpc/grpc-js'; +import * as protoLoader from '@grpc/proto-loader'; + +const packageDefinition = protoLoader.loadSync('agent.proto'); +const proto = grpc.loadPackageDefinition(packageDefinition); +``` + +--- + +## Reference + +- **Proto file:** [`crates/agent-protocol/proto/agent.proto`](https://github.com/zentinelproxy/zentinel/tree/main/crates/agent-protocol/proto/agent.proto) +- **Rust SDK:** [`zentinel-agent-protocol`](https://crates.io/crates/zentinel-agent-protocol) +- **Example agents:** [`agents/`](https://github.com/zentinelproxy/zentinel/tree/main/agents) diff --git a/content/v/26.04/agents/v1/registry.md b/content/v/26.04/agents/v1/registry.md new file mode 100644 index 0000000..849a9cf --- /dev/null +++ b/content/v/26.04/agents/v1/registry.md @@ -0,0 +1,198 @@ ++++ +title = "Agent Registry" +weight = 1 +updated = 2026-02-24 ++++ + +Zentinel has a growing ecosystem of agents for security, traffic management, and custom logic. This page catalogs official agents maintained by the Zentinel team and community-contributed agents. + +> **Browse the full registry at [zentinelproxy.io/agents](https://zentinelproxy.io/agents/)** + +## Official Agents + +Official agents are maintained by the Zentinel Core Team and follow strict quality, security, and compatibility standards. All bundled agents are distributed via `zentinel bundle install`. + +### Core + +| Agent | Version | Status | Description | +|-------|---------|--------|-------------| +| **WAF** | v0.3.0 | Stable | Pure Rust WAF with 285 detection rules, anomaly scoring, and API security | +| **Denylist** | v0.3.0 | Stable | Block requests based on IP addresses, CIDR ranges, or custom patterns with real-time updates | +| **Rate Limiter** | v0.3.0 | Deprecated | Token bucket rate limiting with configurable windows per route, IP, or custom keys | + +### Security + +| Agent | Version | Status | Description | +|-------|---------|--------|-------------| +| **ZentinelSec** | v0.3.0 | Stable | Pure Rust ModSecurity-compatible WAF with full OWASP CRS support — no C dependencies | +| **ModSecurity** | v0.3.0 | Stable | Full OWASP Core Rule Set (CRS) support via libmodsecurity with 800+ detection rules | +| **IP Reputation** | v0.4.0 | Stable | IP threat intelligence with AbuseIPDB integration, file-based blocklists, and Tor exit node detection | +| **Bot Management** | v0.4.0 | Stable | Comprehensive bot detection with multi-signal analysis, known bot verification, and behavioral tracking | +| **Content Scanner** | v0.4.0 | Stable | Malware scanning agent using ClamAV daemon for file upload protection | + +### API Security + +| Agent | Version | Status | Description | +|-------|---------|--------|-------------| +| **GraphQL Security** | v0.4.0 | Stable | Query depth limiting, complexity analysis, introspection control, and field-level authorization | +| **gRPC Inspector** | v0.4.0 | Stable | Method authorization, rate limiting, metadata inspection, and reflection control for gRPC services | +| **SOAP** | v0.4.0 | Stable | Envelope validation, WS-Security verification, operation control, and XXE prevention | +| **API Deprecation** | v0.4.0 | Stable | API lifecycle management with RFC 8594 Sunset headers, usage tracking, and automatic redirects | + +### Protocol + +| Agent | Version | Status | Description | +|-------|---------|--------|-------------| +| **WebSocket Inspector** | v0.4.0 | Stable | Content filtering, schema validation, and attack detection for WebSocket frames | +| **MQTT Gateway** | v0.4.0 | Stable | IoT protocol security with topic-based ACLs, client authentication, payload inspection, and QoS enforcement | + +### Scripting + +| Agent | Version | Status | Description | +|-------|---------|--------|-------------| +| **Lua** | v0.3.0 | Stable | Embed custom Lua scripts for flexible request/response processing | +| **JS** | v0.3.0 | Stable | JavaScript-based custom logic using the QuickJS engine | +| **WASM** | v0.3.0 | Stable | Execute custom Wasm modules for high-performance request/response processing in any language | + +### Utility + +| Agent | Version | Status | Description | +|-------|---------|--------|-------------| +| **Transform** | v0.4.0 | Stable | URL rewriting, header manipulation, and JSON body transforms | +| **Audit Logger** | v0.4.0 | Stable | Structured audit logging with PII redaction, multiple formats (JSON, CEF, LEEF), and compliance templates | +| **Mock Server** | v0.4.0 | Stable | Configurable stub responses with templating, latency simulation, and fault injection | +| **Chaos** | v0.4.0 | Stable | Controlled fault injection for resilience testing with flexible targeting and safety controls | +| **Image Optimization** | v0.1.0 | Stable | On-the-fly JPEG/PNG to WebP/AVIF conversion with content negotiation and filesystem caching | + +### Identity + +| Agent | Version | Status | Description | +|-------|---------|--------|-------------| +| **SPIFFE** | v0.3.0 | Stable | SPIFFE/SPIRE workload identity authentication for zero-trust service-to-service communication | + +### Planned + +On the roadmap for future development. + +| Agent | Description | +|-------|-------------| +| **Adaptive Shield** | Self-learning threat detection using edge ML | +| **Geo Filter** | Geographic IP-based request filtering | +| **LLM Guardian** | AI-powered threat analysis for intelligent traffic decisions | +| **Request Hold** | Pause suspicious requests for async verification | +| **Response Cache** | High-performance caching with TTL controls | +| **Telemetry** | Observability agent for analytics and logging | + +## Built-in Reference Agents + +The Zentinel repository includes reference implementations for testing and as templates: + +### Echo Agent + +A simple agent that echoes request metadata back as headers. Useful for testing and debugging. + +```bash +# Run with Unix socket +zentinel-echo-agent --socket /tmp/echo.sock + +# Run with gRPC +zentinel-echo-agent --grpc 0.0.0.0:50051 +``` + +**Source:** [`agents/echo/`](https://github.com/zentinelproxy/zentinel/tree/main/agents/echo) + +### Features + +- Adds `X-Echo-*` headers with request metadata +- Returns correlation ID, method, path, client IP +- Supports verbose mode for additional debugging headers +- Works with both Unix socket and gRPC transports + +## Community Agents + +Community agents are created and maintained by the Zentinel community. They follow the agent protocol specification but are not officially supported. + +> **No community agents registered yet.** +> +> Want to contribute? [Submit your agent](https://github.com/zentinelproxy/zentinel/issues/new?template=community-agent.md) to the registry! + +### Submission Requirements + +To submit a community agent: + +1. Implement the [Agent Protocol](protocol/) +2. Include a `zentinel-agent.toml` manifest +3. Provide documentation and examples +4. Open an issue with the `community-agent` template + +### Agent Manifest + +Every agent should include a manifest file: + +```toml +# zentinel-agent.toml +[agent] +name = "my-awesome-agent" +version = "0.1.0" +description = "Does awesome things with requests" +authors = ["Your Name "] +license = "MIT OR Apache-2.0" +repository = "https://github.com/yourname/my-awesome-agent" + +[protocol] +version = "1" +events = ["request_headers", "response_headers"] + +[compatibility] +zentinel-proxy = ">=0.1.0" +zentinel-agent-protocol = "0.1" + +[registry] +homepage = "https://example.com/my-agent" +documentation = "https://docs.example.com/my-agent" +keywords = ["zentinel", "agent", "awesome"] +categories = ["security"] # security, traffic, observability, custom +``` + +## Agent Configuration + +Configure agents in your `zentinel.kdl`: + +```kdl +agents { + // Official auth agent + agent "auth" type="auth" { + unix-socket "/var/run/zentinel/auth.sock" + events "request_headers" + timeout-ms 100 + failure-mode "closed" // Block if agent fails + } + + // Official WAF agent (gRPC) + agent "waf" type="waf" { + grpc "http://waf-service:50051" + events "request_headers" "request_body" + timeout-ms 200 + failure-mode "open" // Allow if agent fails + max-request-body-bytes 1048576 // 1MB + } + + // Community or custom agent + agent "custom-logic" type="custom" { + grpc "http://localhost:50052" + events "request_headers" "response_headers" + timeout-ms 50 + } +} +``` + +## Agent Types + +| Type | Description | +|------|-------------| +| `auth` | Authentication and authorization | +| `waf` | Web Application Firewall | +| `rate_limit` | Rate limiting and throttling | +| `custom` | Custom business logic | + +The type is informational and used for metrics/logging. All agents use the same protocol. diff --git a/content/v/26.04/agents/v1/transports.md b/content/v/26.04/agents/v1/transports.md new file mode 100644 index 0000000..c11ea42 --- /dev/null +++ b/content/v/26.04/agents/v1/transports.md @@ -0,0 +1,508 @@ ++++ +title = "Transport Protocols (v1 - Removed)" +weight = 4 +updated = 2026-02-26 ++++ + +{% callout(type="warning", title="V1 Protocol Removed") %} +This page documents the **removed** v1 protocol. For current transport documentation, see [Transports (v2)](../../v2/transports/). +{% end %} + +Zentinel agents communicate with the proxy over two transport mechanisms: Unix domain sockets (UDS) and gRPC. Both transports use the same logical protocol—only the wire encoding differs. + +## Transport Comparison + +| Feature | Unix Socket | gRPC | +|---------|-------------|------| +| **Encoding** | Length-prefixed JSON | Protocol Buffers | +| **Location** | Local only | Local or remote | +| **Latency** | ~50-100µs | ~100-500µs | +| **Throughput** | High | Very high | +| **Streaming** | Manual chunking | Native support | +| **Tooling** | Any JSON library | Protobuf + gRPC toolchain | +| **Language Support** | Universal | Most languages | + +## Unix Domain Sockets + +Unix sockets provide the lowest-latency option for agents running on the same host as Zentinel. + +### Wire Format + +Messages are length-prefixed JSON: + +``` +┌──────────────────┬─────────────────────────────────────┐ +│ Length (4 bytes) │ JSON Message (variable length) │ +│ Big-endian u32 │ UTF-8 encoded │ +└──────────────────┴─────────────────────────────────────┘ +``` + +**Example:** + +``` +00 00 00 1A {"event_type":"request_headers"...} +└─────────┘ └──────────────────────────────────┘ + 26 bytes JSON payload +``` + +### Configuration + +```kdl +agent "my-agent" type="custom" { + unix-socket "/var/run/zentinel/my-agent.sock" + events "request_headers" + timeout-ms 100 +} +``` + +### Message Flow + +``` +Zentinel Proxy Agent Process + │ │ + │ ──── [4 bytes: length] ────────────────▶ │ + │ ──── [N bytes: JSON request] ──────────▶ │ + │ │ + │ (process) + │ │ + │ ◀──── [4 bytes: length] ─────────────── │ + │ ◀──── [N bytes: JSON response] ──────── │ + │ │ +``` + +### Rust Implementation + +**Server Side (Agent):** + +```rust +use tokio::net::UnixListener; +use tokio::io::{AsyncReadExt, AsyncWriteExt}; + +async fn run_server(socket_path: &str) -> Result<(), Box> { + // Remove existing socket + let _ = std::fs::remove_file(socket_path); + + let listener = UnixListener::bind(socket_path)?; + + loop { + let (mut stream, _) = listener.accept().await?; + + tokio::spawn(async move { + loop { + // Read length prefix (4 bytes, big-endian) + let mut len_bytes = [0u8; 4]; + if stream.read_exact(&mut len_bytes).await.is_err() { + break; // Client disconnected + } + let msg_len = u32::from_be_bytes(len_bytes) as usize; + + // Read JSON message + let mut buffer = vec![0u8; msg_len]; + stream.read_exact(&mut buffer).await?; + + let request: AgentRequest = serde_json::from_slice(&buffer)?; + + // Process and respond + let response = process_request(request); + let response_bytes = serde_json::to_vec(&response)?; + + // Write length prefix + let len = (response_bytes.len() as u32).to_be_bytes(); + stream.write_all(&len).await?; + + // Write response + stream.write_all(&response_bytes).await?; + stream.flush().await?; + } + Ok::<_, Box>(()) + }); + } +} +``` + +**Client Side (Proxy):** + +```rust +use tokio::net::UnixStream; + +async fn call_agent( + socket_path: &str, + request: &AgentRequest, +) -> Result> { + let mut stream = UnixStream::connect(socket_path).await?; + + // Send request + let request_bytes = serde_json::to_vec(request)?; + let len = (request_bytes.len() as u32).to_be_bytes(); + stream.write_all(&len).await?; + stream.write_all(&request_bytes).await?; + stream.flush().await?; + + // Read response + let mut len_bytes = [0u8; 4]; + stream.read_exact(&mut len_bytes).await?; + let msg_len = u32::from_be_bytes(len_bytes) as usize; + + let mut buffer = vec![0u8; msg_len]; + stream.read_exact(&mut buffer).await?; + + let response: AgentResponse = serde_json::from_slice(&buffer)?; + Ok(response) +} +``` + +### JSON Message Format + +**Request:** + +```json +{ + "version": 1, + "event_type": "request_headers", + "payload": { + "metadata": { + "correlation_id": "abc-123", + "client_ip": "192.168.1.100", + "client_port": 54321, + "protocol": "HTTP/1.1", + "timestamp": "2025-12-29T08:00:00Z" + }, + "method": "POST", + "uri": "/api/users", + "headers": { + "content-type": ["application/json"], + "authorization": ["Bearer token123"] + } + } +} +``` + +**Response:** + +```json +{ + "version": 1, + "decision": {"allow": {}}, + "request_headers": [ + {"set": {"name": "X-User-Id", "value": "user-123"}} + ], + "audit": { + "tags": ["auth", "success"] + } +} +``` + +### Socket Path Conventions + +| Pattern | Use Case | +|---------|----------| +| `/var/run/zentinel/.sock` | Production (systemd) | +| `/tmp/.sock` | Development/testing | +| `~/.zentinel/.sock` | User-space development | + +### Message Size Limits + +The protocol enforces a maximum message size of **16 MB** (16,777,216 bytes). Messages exceeding this limit are rejected: + +```rust +const MAX_MESSAGE_SIZE: usize = 16 * 1024 * 1024; +``` + +--- + +## gRPC Transport + +gRPC provides higher throughput and native streaming support, ideal for remote agents or high-volume scenarios. + +### Configuration + +```kdl +agent "waf-agent" type="waf" { + grpc "http://localhost:50051" + events "request_headers" "request_body" + timeout-ms 200 +} + +// Remote agent (Kubernetes sidecar, etc.) +agent "ml-scorer" type="custom" { + grpc "http://ml-service.default.svc.cluster.local:50051" + timeout-ms 500 +} +``` + +### Service Definition + +Agents implement the `AgentProcessor` service: + +```protobuf +service AgentProcessor { + // Process a single event + rpc ProcessEvent(AgentRequest) returns (AgentResponse); + + // Stream body chunks for inspection + rpc ProcessEventStream(stream AgentRequest) returns (AgentResponse); +} +``` + +### Rust Implementation (Server) + +Using the `zentinel-agent-protocol` crate: + +```rust +use zentinel_agent_protocol::{GrpcAgentServer, AgentHandler, AgentResponse}; + +struct MyAgent; + +#[async_trait::async_trait] +impl AgentHandler for MyAgent { + async fn on_request_headers(&self, event: RequestHeadersEvent) -> AgentResponse { + // Your logic here + AgentResponse::default_allow() + } +} + +#[tokio::main] +async fn main() -> Result<(), Box> { + let agent = Box::new(MyAgent); + let server = GrpcAgentServer::new("my-agent", agent); + + server.run("0.0.0.0:50051".parse()?).await?; + Ok(()) +} +``` + +### Go Implementation (Server) + +```go +package main + +import ( + "context" + "net" + + pb "github.com/zentinelproxy/zentinel-proto/go" + "google.golang.org/grpc" +) + +type myAgent struct { + pb.UnimplementedAgentProcessorServer +} + +func (a *myAgent) ProcessEvent( + ctx context.Context, + req *pb.AgentRequest, +) (*pb.AgentResponse, error) { + // Handle different event types + switch e := req.Event.(type) { + case *pb.AgentRequest_RequestHeaders: + return a.handleRequestHeaders(e.RequestHeaders) + case *pb.AgentRequest_RequestBodyChunk: + return a.handleRequestBody(e.RequestBodyChunk) + } + + return &pb.AgentResponse{ + Version: 1, + Decision: &pb.AgentResponse_Allow{ + Allow: &pb.AllowDecision{}, + }, + }, nil +} + +func main() { + lis, _ := net.Listen("tcp", ":50051") + s := grpc.NewServer() + pb.RegisterAgentProcessorServer(s, &myAgent{}) + s.Serve(lis) +} +``` + +### Python Implementation (Server) + +```python +import grpc +from concurrent import futures +import agent_pb2 +import agent_pb2_grpc + +class MyAgent(agent_pb2_grpc.AgentProcessorServicer): + def ProcessEvent(self, request, context): + if request.event_type == agent_pb2.EVENT_TYPE_REQUEST_HEADERS: + headers = request.request_headers + # Your logic here + + return agent_pb2.AgentResponse( + version=1, + allow=agent_pb2.AllowDecision() + ) + +def serve(): + server = grpc.server(futures.ThreadPoolExecutor(max_workers=10)) + agent_pb2_grpc.add_AgentProcessorServicer_to_server(MyAgent(), server) + server.add_insecure_port('[::]:50051') + server.start() + server.wait_for_termination() + +if __name__ == '__main__': + serve() +``` + +### Testing with grpcurl + +```bash +# List available services +grpcurl -plaintext localhost:50051 list + +# Test request headers event +grpcurl -plaintext -d '{ + "version": 1, + "event_type": "EVENT_TYPE_REQUEST_HEADERS", + "request_headers": { + "metadata": { + "correlation_id": "test-123", + "client_ip": "127.0.0.1" + }, + "method": "GET", + "uri": "/api/test" + } +}' localhost:50051 zentinel.agent.v1.AgentProcessor/ProcessEvent +``` + +### Streaming for Body Inspection + +For large request/response bodies, use the streaming RPC: + +```rust +// Client-side (proxy sending body chunks) +let mut stream = client.process_event_stream().await?; + +// Send headers first +stream.send(AgentRequest { + event_type: EventType::RequestHeaders, + request_headers: Some(headers_event), + ..Default::default() +}).await?; + +// Stream body chunks +for chunk in body_chunks { + stream.send(AgentRequest { + event_type: EventType::RequestBodyChunk, + request_body_chunk: Some(RequestBodyChunkEvent { + correlation_id: correlation_id.clone(), + data: chunk.data, + is_last: chunk.is_last, + total_size: chunk.total_size, + }), + ..Default::default() + }).await?; +} + +// Get final response +let response = stream.finish().await?; +``` + +--- + +## Choosing a Transport + +### Use Unix Sockets When: + +- Agent runs on the same host as Zentinel +- Latency is critical (< 100µs per call) +- Simplicity is preferred (no protobuf toolchain) +- Deploying as systemd services + +### Use gRPC When: + +- Agent runs on a different host +- Building agents in languages with strong gRPC support +- Need streaming for large body inspection +- Deploying in Kubernetes (service mesh integration) +- Higher throughput requirements + +--- + +## Connection Management + +### Unix Socket Considerations + +```kdl +agent "local-auth" type="auth" { + unix-socket "/var/run/zentinel/auth.sock" + + // Connection pool settings + pool { + min-connections 2 + max-connections 10 + idle-timeout-ms 30000 + } +} +``` + +### gRPC Considerations + +```kdl +agent "remote-waf" type="waf" { + grpc "http://waf-service:50051" + + // HTTP/2 connection settings + http2 { + keep-alive-interval-ms 10000 + keep-alive-timeout-ms 5000 + max-concurrent-streams 100 + } +} +``` + +--- + +## Security + +### Unix Socket Security + +Unix sockets rely on filesystem permissions: + +```bash +# Restrict socket access +chmod 0600 /var/run/zentinel/auth.sock +chown zentinel:zentinel /var/run/zentinel/auth.sock +``` + +### gRPC Security + +For production gRPC agents, use TLS: + +```kdl +agent "secure-agent" type="custom" { + grpc "https://agent.internal:50051" + + tls { + ca-cert "/etc/zentinel/ca.crt" + client-cert "/etc/zentinel/client.crt" + client-key "/etc/zentinel/client.key" + } +} +``` + +--- + +## Failure Handling + +Both transports support the same failure policies: + +```kdl +agent "auth" type="auth" { + unix-socket "/var/run/zentinel/auth.sock" + timeout-ms 100 + + // What to do when agent fails + failure-mode "closed" // Block requests (secure default) + // failure-mode "open" // Allow requests (availability) + + // Circuit breaker + circuit-breaker { + failure-threshold 5 // Open after 5 failures + success-threshold 3 // Close after 3 successes + timeout-seconds 30 // Half-open after 30s + } +} +``` diff --git a/content/v/26.04/agents/v2/_index.md b/content/v/26.04/agents/v2/_index.md new file mode 100644 index 0000000..36d4d7d --- /dev/null +++ b/content/v/26.04/agents/v2/_index.md @@ -0,0 +1,63 @@ ++++ +title = "Protocol v2 (Current)" +weight = 10 +sort_by = "weight" ++++ + +Agent Protocol v2 is the recommended protocol for new agent deployments. It provides enhanced features for production environments. + +## Key Features + +| Feature | Description | +|---------|-------------| +| **Connection Pooling** | Maintain multiple connections per agent with load balancing | +| **Multiple Transports** | gRPC, Binary UDS, and Reverse Connections | +| **Request Cancellation** | Cancel in-flight requests when clients disconnect | +| **Reverse Connections** | Agents connect to proxy (NAT traversal) | +| **Enhanced Observability** | Built-in metrics export in Prometheus format | +| **Config Push** | Push configuration updates to capable agents | + +## Documentation + +| Page | Description | +|------|-------------| +| [Protocol Specification](protocol/) | Wire protocol, message types, and streaming | +| [API Reference](api/) | AgentPool, client, and server APIs | +| [Connection Pooling](pooling/) | Load balancing and circuit breakers | +| [Transport Options](transports/) | gRPC, UDS, and Reverse comparison | +| [Reverse Connections](reverse-connections/) | NAT traversal and agent-initiated connections | +| [Performance Benchmarks](performance/) | Latency, throughput, and optimization results | +| [Migration Guide](migration/) | Historical v1 → v2 migration notes (v1 removed in 26.02_18) | + +## Quick Start + +```rust +use zentinel_agent_protocol::v2::{AgentPool, AgentPoolConfig, LoadBalanceStrategy}; +use std::time::Duration; + +let config = AgentPoolConfig { + connections_per_agent: 4, + load_balance_strategy: LoadBalanceStrategy::LeastConnections, + request_timeout: Duration::from_secs(30), + ..Default::default() +}; + +let pool = AgentPool::with_config(config); + +// Add agents (transport auto-detected) +pool.add_agent("waf", "localhost:50051").await?; // gRPC +pool.add_agent("auth", "/var/run/auth.sock").await?; // UDS +``` + +## Features + +| Feature | Details | +|---------|---------| +| Transport | UDS (binary), gRPC, Reverse Connections | +| Connection pooling | Yes (4 strategies) | +| Bidirectional streaming | Full support | +| Metrics export | Prometheus format | +| Config push | Yes | +| Health tracking | Comprehensive | +| Flow control | Yes | +| Request cancellation | Yes | diff --git a/content/v/26.04/agents/v2/api.md b/content/v/26.04/agents/v2/api.md new file mode 100644 index 0000000..c268a9f --- /dev/null +++ b/content/v/26.04/agents/v2/api.md @@ -0,0 +1,572 @@ ++++ +title = "API Reference" +weight = 2 +updated = 2026-02-27 ++++ + +This document covers the v2 APIs for building agent integrations with connection pooling, multiple transports, and reverse connections. + +## Quick Start + +```rust +use zentinel_agent_protocol::v2::{AgentPool, AgentPoolConfig, LoadBalanceStrategy}; +use std::time::Duration; + +#[tokio::main] +async fn main() -> Result<(), Box> { + // Create a connection pool + let config = AgentPoolConfig { + connections_per_agent: 4, + load_balance_strategy: LoadBalanceStrategy::LeastConnections, + request_timeout: Duration::from_secs(30), + ..Default::default() + }; + + let pool = AgentPool::with_config(config); + + // Add agents (transport auto-detected from endpoint) + pool.add_agent("waf", "localhost:50051").await?; // gRPC + pool.add_agent("auth", "/var/run/auth.sock").await?; // UDS + + // Send requests through the pool + let response = pool.send_request_headers("waf", &headers).await?; + + Ok(()) +} +``` + +--- + +## AgentPool + +The `AgentPool` is the primary interface for v2 agent communication. It manages connections, load balancing, health tracking, and metrics. + +### Creating a Pool + +```rust +use zentinel_agent_protocol::v2::{AgentPool, AgentPoolConfig, LoadBalanceStrategy}; + +// Default configuration +let pool = AgentPool::new(); + +// Custom configuration +let config = AgentPoolConfig { + connections_per_agent: 4, + load_balance_strategy: LoadBalanceStrategy::LeastConnections, + request_timeout: Duration::from_secs(30), + connect_timeout: Duration::from_secs(5), + health_check_interval: Duration::from_secs(10), + circuit_breaker_threshold: 5, + circuit_breaker_reset_timeout: Duration::from_secs(30), +}; + +let pool = AgentPool::with_config(config); +``` + +### Adding Agents + +```rust +// gRPC agent (detected by host:port format) +pool.add_agent("waf", "localhost:50051").await?; +pool.add_agent("remote-waf", "waf.internal:50051").await?; + +// UDS agent (detected by path format) +pool.add_agent("auth", "/var/run/zentinel/auth.sock").await?; + +// Explicit transport selection +pool.add_grpc_agent("waf", "localhost:50051", tls_config).await?; +pool.add_uds_agent("auth", "/var/run/auth.sock").await?; +``` + +### Sending Requests + +```rust +use zentinel_agent_protocol::v2::RequestHeaders; + +let headers = RequestHeaders { + request_id: 1, + method: "POST".to_string(), + uri: "/api/users".to_string(), + headers: vec![ + ("content-type".to_string(), "application/json".to_string()), + ], + has_body: true, + metadata: request_metadata, +}; + +// Send to specific agent +let response = pool.send_request_headers("waf", &headers).await?; + +// Send body chunks +let chunk = RequestBodyChunk { + request_id: 1, + chunk_index: 0, + data: base64::encode(&body_bytes), + is_last: true, +}; +let response = pool.send_request_body_chunk("waf", &chunk).await?; +``` + +### Sending Response Events + +For agents that subscribe to response-phase events, the proxy sends upstream response headers and body chunks through the pool: + +```rust +use zentinel_agent_protocol::v2::{ResponseHeaders, ResponseBodyChunk}; + +// Send upstream response headers to agent +let response_headers = ResponseHeaders { + request_id: 1, + status: 200, + headers: vec![ + ("content-type".to_string(), "image/png".to_string()), + ("content-length".to_string(), "1024".to_string()), + ], + metadata: request_metadata, +}; +let decision = pool.send_response_headers("image-optimizer", &response_headers).await?; + +// Apply response header modifications from the agent's decision +for op in &decision.response_headers { + match op { + HeaderOp::Set { name, value } => { /* set header */ } + HeaderOp::Add { name, value } => { /* add header */ } + HeaderOp::Remove { name } => { /* remove header */ } + } +} + +// Send response body chunk to agent +let chunk = ResponseBodyChunk { + request_id: 1, + chunk_index: 0, + data: base64::encode(&body_bytes), + is_last: true, + total_size: Some(body_bytes.len()), +}; +let decision = pool.send_response_body_chunk("image-optimizer", &chunk).await?; + +// Apply body mutation if present +if let Some(mutation) = &decision.response_body_mutation { + if let Some(data) = &mutation.data { + let new_body = base64::decode(data)?; + // Replace response body with new_body + } +} +``` + +### Cancelling Requests + +```rust +// Cancel specific request +pool.cancel_request("waf", request_id).await?; + +// Cancel all requests for an agent +pool.cancel_all("waf").await?; +``` + +### Pool Methods + +| Method | Description | +|--------|-------------| +| `new()` | Create pool with default config | +| `with_config(config)` | Create pool with custom config | +| `add_agent(name, endpoint)` | Add agent with auto-detected transport | +| `add_grpc_agent(name, endpoint, tls)` | Add gRPC agent explicitly | +| `add_uds_agent(name, path)` | Add UDS agent explicitly | +| `add_reverse_connection(name, client, caps)` | Add reverse-connected agent | +| `remove_agent(name)` | Remove agent from pool | +| `send_request_headers(agent, headers)` | Send request headers | +| `send_request_body_chunk(agent, chunk)` | Send request body chunk | +| `send_response_headers(agent, status, headers)` | Send response headers | +| `send_response_body_chunk(agent, chunk)` | Send response body chunk | +| `cancel_request(agent, request_id)` | Cancel specific request | +| `cancel_all(agent)` | Cancel all requests | +| `get_health(agent)` | Get agent health status | +| `metrics_collector()` | Get metrics collector reference | + +--- + +## AgentClientV2 (gRPC) + +Low-level gRPC client for direct use without pooling. + +### Creating a Client + +```rust +use zentinel_agent_protocol::v2::AgentClientV2; + +let client = AgentClientV2::connect( + "waf-agent", + "http://localhost:50051", + Duration::from_secs(30), +).await?; + +// With TLS +let client = AgentClientV2::connect_with_tls( + "waf-agent", + "https://waf.internal:50051", + tls_config, + Duration::from_secs(30), +).await?; +``` + +### Sending Messages + +```rust +// Send request headers +let response = client.send_request_headers(&headers).await?; + +// Send body chunk +let response = client.send_request_body_chunk(&chunk).await?; + +// Cancel request +client.cancel_request(request_id).await?; +``` + +--- + +## AgentClientV2Uds (Unix Domain Socket) + +Low-level UDS client for direct use without pooling. + +### Creating a Client + +```rust +use zentinel_agent_protocol::v2::AgentClientV2Uds; + +let client = AgentClientV2Uds::connect( + "auth-agent", + "/var/run/zentinel/auth.sock", + Duration::from_secs(30), +).await?; +``` + +### Handshake + +The UDS client performs automatic handshake on connection: + +```rust +// Handshake is automatic, but you can query capabilities +let capabilities = client.capabilities(); + +println!("Agent: {}", capabilities.agent_name); +println!("Handles body: {}", capabilities.handles_request_body); +println!("Max concurrent: {:?}", capabilities.max_concurrent_requests); +``` + +--- + +## ReverseConnectionListener + +Accepts inbound connections from agents. + +### Creating a Listener + +```rust +use zentinel_agent_protocol::v2::{ReverseConnectionListener, ReverseConnectionConfig}; + +let config = ReverseConnectionConfig { + handshake_timeout: Duration::from_secs(10), + max_connections_per_agent: 4, + require_auth: false, + allowed_agents: None, +}; + +let listener = ReverseConnectionListener::bind_uds( + "/var/run/zentinel/agents.sock", + config, +).await?; +``` + +### Accepting Connections + +```rust +// Accept a single connection +let (client, registration) = listener.accept().await?; +println!("Agent connected: {}", registration.agent_id); + +// Add to pool +pool.add_reverse_connection( + ®istration.agent_id, + client, + registration.capabilities, +).await?; +``` + +--- + +## Configuration Types + +### AgentPoolConfig + +```rust +pub struct AgentPoolConfig { + /// Number of connections to maintain per agent + pub connections_per_agent: usize, // Default: 4 + + /// Load balancing strategy + pub load_balance_strategy: LoadBalanceStrategy, // Default: LeastConnections + + /// Timeout for individual requests + pub request_timeout: Duration, // Default: 30s + + /// Timeout for establishing connections + pub connect_timeout: Duration, // Default: 5s + + /// Interval between health checks + pub health_check_interval: Duration, // Default: 10s + + /// Failures before opening circuit breaker + pub circuit_breaker_threshold: u32, // Default: 5 + + /// Time before circuit breaker resets + pub circuit_breaker_reset_timeout: Duration, // Default: 30s +} +``` + +### LoadBalanceStrategy + +```rust +pub enum LoadBalanceStrategy { + /// Distribute requests evenly across connections + RoundRobin, + + /// Route to connection with fewest in-flight requests + LeastConnections, + + /// Prefer healthier connections based on error rates + HealthBased, + + /// Random selection + Random, +} +``` + +--- + +## Metrics + +### MetricsCollector + +```rust +let metrics = pool.metrics_collector(); + +// Get metrics snapshot +let snapshot = metrics.snapshot(); +println!("Total requests: {}", snapshot.total_requests); +println!("Active connections: {}", snapshot.active_connections); + +// Export in Prometheus format +let prometheus_output = metrics.export_prometheus(); +``` + +### Available Metrics + +| Metric | Type | Description | +|--------|------|-------------| +| `agent_requests_total` | Counter | Total requests by agent and decision | +| `agent_request_duration_seconds` | Histogram | Request latency distribution | +| `agent_connections_active` | Gauge | Current active connections | +| `agent_errors_total` | Counter | Error counts by type | +| `agent_circuit_breaker_state` | Gauge | Circuit breaker state (0=closed, 1=open) | + +--- + +## Error Handling + +### V2-Specific Errors + +```rust +pub enum AgentProtocolError { + // ... existing errors ... + + /// Connection was closed unexpectedly (v2) + #[error("Connection closed")] + ConnectionClosed, +} +``` + +### Pool Error Handling + +```rust +match pool.send_request_headers("waf", &headers).await { + Ok(decision) => { + // Handle decision + } + Err(AgentProtocolError::Timeout) => { + // Request timed out - apply fallback policy + } + Err(AgentProtocolError::ConnectionClosed) => { + // Connection lost - pool will reconnect automatically + } + Err(e) => { + tracing::error!("Agent error: {}", e); + } +} +``` + +--- + +## Header Utilities + +The library provides zero-allocation header handling for common HTTP headers. + +### Common Header Names + +Pre-defined static strings for standard headers avoid allocations: + +```rust +use zentinel_agent_protocol::headers::names; + +// Use static strings directly +let content_type = names::CONTENT_TYPE; // "content-type" +let authorization = names::AUTHORIZATION; // "authorization" +let x_request_id = names::X_REQUEST_ID; // "x-request-id" +``` + +### Header Name Interning + +The `intern_header_name` function returns borrowed references for known headers: + +```rust +use zentinel_agent_protocol::headers::intern_header_name; +use std::borrow::Cow; + +let name = intern_header_name("Content-Type"); +// Returns Cow::Borrowed("content-type") - no allocation + +let custom = intern_header_name("X-Custom-Header"); +// Returns Cow::Owned("x-custom-header") - allocates only for unknown headers +``` + +### Cow Header Maps + +For high-throughput scenarios, use `CowHeaderMap` to minimize allocations: + +```rust +use zentinel_agent_protocol::headers::{CowHeaderMap, CowHeaderName, intern_header_name}; + +let mut headers: CowHeaderMap = CowHeaderMap::new(); +headers.insert(intern_header_name("content-type"), vec!["application/json".into()]); + +// Convert from standard HashMap +let cow_headers = to_cow_optimized(&standard_headers); + +// Convert back when needed +let standard = from_cow_optimized(&cow_headers); +``` + +### Available Header Constants + +| Constant | Value | +|----------|-------| +| `CONTENT_TYPE` | "content-type" | +| `CONTENT_LENGTH` | "content-length" | +| `AUTHORIZATION` | "authorization" | +| `ACCEPT` | "accept" | +| `HOST` | "host" | +| `USER_AGENT` | "user-agent" | +| `X_REQUEST_ID` | "x-request-id" | +| `X_FORWARDED_FOR` | "x-forwarded-for" | +| ... | (32 total) | + +--- + +## Memory-Mapped Buffers + +For large request/response bodies, memory-mapped buffers minimize heap allocation. + +**Feature flag required:** +```toml +[dependencies] +zentinel-agent-protocol = { version = "0.3", features = ["mmap-buffers"] } +``` + +### Basic Usage + +```rust +use zentinel_agent_protocol::mmap_buffer::{LargeBodyBuffer, LargeBodyBufferConfig}; + +// Configure threshold for switching to mmap +let config = LargeBodyBufferConfig { + mmap_threshold: 1024 * 1024, // 1MB - use mmap above this size + max_body_size: 100 * 1024 * 1024, // 100MB - maximum allowed + temp_dir: None, // Use system temp directory +}; + +let mut buffer = LargeBodyBuffer::with_config(config); + +// Write chunks - automatically switches to mmap when needed +buffer.write_chunk(b"request body data...")?; + +// Read back - seamless regardless of storage type +let data = buffer.as_slice()?; +``` + +### Configuration Options + +| Option | Default | Description | +|--------|---------|-------------| +| `mmap_threshold` | 1MB | Size above which to use memory-mapped files | +| `max_body_size` | 100MB | Maximum allowed body size | +| `temp_dir` | None | Custom temp directory for mmap files | + +### Storage Behavior + +| Body Size | Storage | Allocation | +|-----------|---------|------------| +| < threshold | `Vec` | Heap memory | +| >= threshold | mmap'd file | OS page cache | + +### Methods + +| Method | Description | +|--------|-------------| +| `new()` | Create buffer with default config | +| `with_config(config)` | Create buffer with custom config | +| `write_chunk(data)` | Write data (auto-transitions to mmap) | +| `as_slice()` | Get immutable slice of data | +| `as_mut_slice()` | Get mutable slice (forces to memory) | +| `into_vec()` | Take ownership as Vec | +| `clear()` | Reset buffer | +| `len()` | Current data size | +| `is_empty()` | Check if empty | +| `is_mmap()` | Check if using mmap storage | + +### When to Use + +| Scenario | Recommendation | +|----------|----------------| +| File uploads | Use with 1MB threshold | +| API responses | Use with default config | +| Streaming bodies | Accumulate chunks, then read | +| Memory-constrained | Lower threshold (e.g., 256KB) | + +--- + +## Migration from v1 + +### Before (v1) + +```rust +use zentinel_agent_protocol::AgentClient; + +let client = AgentClient::unix_socket( + "proxy", + "/tmp/agent.sock", + Duration::from_secs(5), +).await?; + +let response = client.send_event(EventType::RequestHeaders, &event).await?; +``` + +### After (v2 with pooling) + +```rust +use zentinel_agent_protocol::v2::AgentPool; + +let pool = AgentPool::new(); +pool.add_agent("agent", "/tmp/agent.sock").await?; + +let response = pool.send_request_headers("agent", &headers).await?; +``` diff --git a/content/v/26.04/agents/v2/migration.md b/content/v/26.04/agents/v2/migration.md new file mode 100644 index 0000000..2b6be39 --- /dev/null +++ b/content/v/26.04/agents/v2/migration.md @@ -0,0 +1,512 @@ ++++ +title = "Migration Guide (v1 to v2)" +weight = 6 +updated = 2026-02-26 ++++ + +{% callout(type="warning", title="V1 Protocol Removed") %} +V1 was **removed** in Zentinel release 26.02_18 (February 2026). All agents must use v2. This guide is preserved for teams completing their migration. +{% end %} + +This guide helps you migrate from Agent Protocol v1 to v2. The v2 protocol offers significant improvements in performance, reliability, and observability while maintaining conceptual compatibility. + +## Why Migrate? + +| Improvement | v1 | v2 | +|-------------|----|----| +| **Latency** | ~50μs per request | ~10-20μs per request | +| **Throughput** | Single connection | Pooled connections (4x+ throughput) | +| **Reliability** | Basic timeouts | Circuit breakers, health tracking | +| **Streaming** | Limited | Full bidirectional streaming | +| **Observability** | Manual | Built-in Prometheus metrics | +| **NAT Traversal** | Not supported | Reverse connections | + +--- + +## Quick Migration + +### Minimal Change (Drop-in) + +If you just want pooling benefits without code changes: + +**Before (v1):** +```rust +use zentinel_agent_protocol::AgentClient; + +let client = AgentClient::unix_socket( + "proxy", + "/var/run/agent.sock", + Duration::from_secs(5), +).await?; + +let response = client.send_event(EventType::RequestHeaders, &event).await?; +``` + +**After (v2):** +```rust +use zentinel_agent_protocol::v2::AgentPool; + +let pool = AgentPool::new(); +pool.add_agent("agent", "/var/run/agent.sock").await?; + +let response = pool.send_request_headers("agent", &headers).await?; +``` + +The `AgentPool` automatically: +- Maintains 4 connections per agent +- Load balances requests +- Tracks health and circuit breaker state +- Exports Prometheus metrics + +--- + +## Step-by-Step Migration + +### 1. Update Dependencies + +```toml +# Cargo.toml +[dependencies] +zentinel-agent-protocol = "0.3" # v2 included +``` + +### 2. Import v2 Types + +```rust +// Before +use zentinel_agent_protocol::{AgentClient, EventType, AgentEvent}; + +// After +use zentinel_agent_protocol::v2::{ + AgentPool, + AgentPoolConfig, + LoadBalanceStrategy, + Decision, +}; +``` + +### 3. Replace Client with Pool + +**Before:** +```rust +// Create individual clients +let waf_client = AgentClient::unix_socket("proxy", "/run/waf.sock", timeout).await?; +let auth_client = AgentClient::grpc("http://localhost:50051", timeout).await?; + +// Store clients somewhere +struct Clients { + waf: AgentClient, + auth: AgentClient, +} +``` + +**After:** +```rust +// Create single pool for all agents +let pool = AgentPool::new(); + +// Add agents (transport auto-detected) +pool.add_agent("waf", "/run/waf.sock").await?; +pool.add_agent("auth", "localhost:50051").await?; + +// Pool is Clone + Send + Sync +let pool = Arc::new(pool); +``` + +### 4. Update Request Sending + +**Before:** +```rust +let event = AgentEvent { + event_type: EventType::RequestHeaders, + request_id: req_id, + method: method.to_string(), + uri: uri.to_string(), + headers: headers.clone(), + // ... +}; + +let response = client.send_event(EventType::RequestHeaders, &event).await?; +``` + +**After:** +```rust +use zentinel_agent_protocol::v2::RequestHeadersEvent; + +let event = RequestHeadersEvent { + correlation_id: correlation_id.clone(), + method: method.to_string(), + uri: uri.to_string(), + headers: headers.clone(), + client_ip: client_ip.clone(), + // ... +}; + +let response = pool.send_request_headers("waf", &event).await?; +``` + +### 5. Update Response Handling + +**Before:** +```rust +match response.action { + Action::Allow => { /* continue */ } + Action::Block => { + return Err(blocked_response(response.status_code)); + } + Action::Redirect(url) => { + return Ok(redirect_response(url)); + } +} +``` + +**After:** +```rust +match response.decision { + Decision::Allow => { /* continue */ } + Decision::Block { status, body, headers } => { + return Err(blocked_response(status, body, headers)); + } + Decision::Redirect { location, status } => { + return Ok(redirect_response(location, status)); + } + Decision::Modify { headers_to_add, headers_to_remove } => { + apply_modifications(&mut request, headers_to_add, headers_to_remove); + } +} +``` + +### 6. Add Error Handling for New Error Types + +```rust +use zentinel_agent_protocol::v2::AgentProtocolError; + +match pool.send_request_headers("waf", &event).await { + Ok(response) => handle_response(response), + + // New in v2: Circuit breaker open + Err(AgentProtocolError::CircuitBreakerOpen { agent_id }) => { + tracing::warn!("Circuit open for {}, applying fallback", agent_id); + apply_fallback_policy() + } + + // New in v2: Flow control + Err(AgentProtocolError::FlowControlPaused { agent_id }) => { + tracing::warn!("Agent {} paused, request rejected", agent_id); + apply_fallback_policy() + } + + // Existing errors still work + Err(AgentProtocolError::Timeout) => { + apply_fallback_policy() + } + + Err(e) => { + tracing::error!("Agent error: {}", e); + apply_fallback_policy() + } +} +``` + +--- + +## Configuration Migration + +### KDL Configuration + +**Before (v1):** +```kdl +agents { + agent "waf" type="waf" { + unix-socket "/var/run/waf.sock" + timeout-ms 100 + failure-mode "open" + } +} +``` + +**After (v2):** +```kdl +agents { + agent "waf" type="waf" { + unix-socket "/var/run/waf.sock" + protocol-version 2 // Enable v2 + connections 4 // Connection pool size + timeout-ms 100 + failure-mode "open" + + // New v2 options + circuit-breaker { + failure-threshold 5 + reset-timeout-seconds 30 + } + } +} +``` + +### Rust Configuration + +**Before:** +```rust +let client = AgentClient::unix_socket( + "proxy", + socket_path, + Duration::from_millis(100), +).await?; +``` + +**After:** +```rust +let config = AgentPoolConfig { + connections_per_agent: 4, + load_balance_strategy: LoadBalanceStrategy::LeastConnections, + request_timeout: Duration::from_millis(100), + circuit_breaker_threshold: 5, + circuit_breaker_reset_timeout: Duration::from_secs(30), + ..Default::default() +}; + +let pool = AgentPool::with_config(config); +pool.add_agent("waf", socket_path).await?; +``` + +--- + +## Feature-by-Feature Migration + +### Body Streaming + +**Before (v1):** +```rust +// Send body as single event +let body_event = AgentEvent { + event_type: EventType::RequestBody, + body: Some(full_body), + .. +}; +client.send_event(EventType::RequestBody, &body_event).await?; +``` + +**After (v2):** +```rust +// Stream body in chunks +for (i, chunk) in body_chunks.enumerate() { + let is_last = i == body_chunks.len() - 1; + + let chunk_event = RequestBodyChunkEvent { + correlation_id: correlation_id.clone(), + data: chunk, + chunk_index: i as u32, + is_last, + ..Default::default() + }; + + pool.send_request_body_chunk("waf", &chunk_event).await?; +} +``` + +### Health Checks + +**Before (v1):** +```rust +// Manual health check +match client.ping().await { + Ok(_) => { /* healthy */ } + Err(_) => { /* unhealthy, handle manually */ } +} +``` + +**After (v2):** +```rust +// Automatic health tracking +let health = pool.get_health("waf")?; + +println!("Healthy connections: {}/{}", + health.healthy_connections, + health.total_connections); +println!("Success rate: {:.1}%", health.success_rate * 100.0); +println!("Circuit breaker: {:?}", health.circuit_breaker_state); +``` + +### Metrics + +**Before (v1):** +```rust +// Manual metrics collection +metrics::counter!("agent_requests_total").increment(1); +let start = Instant::now(); +let result = client.send_event(...).await; +metrics::histogram!("agent_request_duration").record(start.elapsed()); +``` + +**After (v2):** +```rust +// Automatic metrics export +let prometheus_output = pool.metrics_collector().export_prometheus(); +// Expose via /metrics endpoint + +// Or get snapshot for custom handling +let snapshot = pool.protocol_metrics().snapshot(); +println!("Total requests: {}", snapshot.requests_total); +println!("In-flight: {}", snapshot.in_flight_requests); +``` + +--- + +## Agent-Side Migration + +If you maintain custom agents, update the server implementation: + +### gRPC Agents + +The protobuf definitions are compatible. Update to support new message types: + +```protobuf +// New message types in v2 +message RequestHeadersEvent { + string correlation_id = 1; + string method = 2; + string uri = 3; + map headers = 4; + // ... +} + +message RequestBodyChunkEvent { + string correlation_id = 1; + bytes data = 2; + bool is_last = 3; + uint32 chunk_index = 4; + // ... +} +``` + +### UDS Agents + +V2 UDS uses binary MessagePack encoding for better performance: + +```rust +// Server handshake response includes encoding negotiation +let handshake = HandshakeResponse { + agent_id: "my-agent".to_string(), + supported_encodings: vec!["msgpack", "json"], + capabilities: Capabilities { + handles_request_body: true, + handles_response_body: false, + supports_streaming: true, + max_concurrent_requests: Some(100), + }, +}; +``` + +--- + +## Rollback Plan + +If you need to rollback to v1: + +1. **Keep v1 client code** in a feature flag during migration +2. **Monitor metrics** during rollout +3. **Gradual rollout** using traffic splitting + +```rust +#[cfg(feature = "agent-v2")] +async fn send_to_agent(event: &Event) -> Result { + pool.send_request_headers("waf", event).await +} + +#[cfg(not(feature = "agent-v2"))] +async fn send_to_agent(event: &Event) -> Result { + client.send_event(EventType::RequestHeaders, event).await +} +``` + +--- + +## Compatibility Notes + +### Wire Protocol + +- v2 UDS uses length-prefixed MessagePack (or JSON with negotiation) +- v2 gRPC uses updated protobuf messages +- v1 agents cannot connect to v2 pool (and vice versa) + +### Breaking Changes + +| Change | Migration | +|--------|-----------| +| `AgentClient` → `AgentPool` | Use pool pattern | +| `send_event()` → `send_request_headers()` | Update method calls | +| `Action` → `Decision` | Update response handling | +| `EventType` enum removed | Use typed methods | +| Request ID → Correlation ID | Use string correlation IDs | + +### Deprecated (Still Working) + +| Deprecated | Replacement | +|------------|-------------| +| `AgentClient` (v1) | `AgentPool` (v2) | +| JSON-only UDS | MessagePack UDS | +| Manual health checks | Automatic health tracking | + +--- + +## Troubleshooting + +### "Connection refused" after migration + +Ensure the agent supports v2 protocol. Check handshake: + +```bash +# Test UDS connection +echo '{"type":"handshake","version":2}' | nc -U /var/run/agent.sock +``` + +### Circuit breaker keeps opening + +Tune thresholds for your error rates: + +```rust +let config = AgentPoolConfig { + circuit_breaker_threshold: 10, // More tolerant + circuit_breaker_reset_timeout: Duration::from_secs(10), // Faster recovery + ..Default::default() +}; +``` + +### Higher latency than expected + +Check connection pool size and load balancing: + +```rust +// For low-latency workloads +let config = AgentPoolConfig { + connections_per_agent: 2, // Fewer connections + load_balance_strategy: LoadBalanceStrategy::LeastConnections, + ..Default::default() +}; +``` + +### Memory usage increased + +Large bodies may need mmap buffers: + +```toml +[dependencies] +zentinel-agent-protocol = { version = "0.3", features = ["mmap-buffers"] } +``` + +--- + +## Next Steps + +After migration: + +1. **Enable metrics export** - Add `/metrics` endpoint for Prometheus +2. **Configure alerts** - Set up alerts for circuit breaker state +3. **Tune pool size** - Adjust `connections_per_agent` based on load testing +4. **Consider reverse connections** - For agents behind NAT/firewalls + +See also: +- [API Reference](../api/) +- [Connection Pooling](../pooling/) +- [Transport Options](../transports/) diff --git a/content/v/26.04/agents/v2/performance.md b/content/v/26.04/agents/v2/performance.md new file mode 100644 index 0000000..5e95ccf --- /dev/null +++ b/content/v/26.04/agents/v2/performance.md @@ -0,0 +1,253 @@ ++++ +title = "Performance Benchmarks" +weight = 7 +updated = 2026-02-19 ++++ + +This page presents benchmark results for Agent Protocol v2 optimizations, helping you understand expected performance characteristics and make informed configuration decisions. + +## Executive Summary + +Agent Protocol v2 achieves significant performance improvements through lock-free data structures, optimized serialization, and efficient memory management. + +| Optimization | Improvement | Impact | +|--------------|-------------|--------| +| Atomic health cache | **10x faster** | Hot-path health checks | +| MessagePack serialization | **24-32% faster** | Large payload processing | +| SmallVec headers | **40% faster** | Single-value header allocation | +| Body chunk streaming | **4.7-11x faster** | WAF body inspection | +| Protocol metrics | **<3ns overhead** | Zero-cost observability | + +**Full request path latency:** ~230ns (excluding network I/O) + +--- + +## Lock-Free Operations + +### Health State Caching + +Health checks are on every request's hot path. Using atomic operations instead of locks provides 10x improvement: + +| Operation | Atomic | RwLock | Speedup | +|-----------|--------|--------|---------| +| Read | **0.46ns** | 4.6ns | **10x** | +| Write | **0.46ns** | 1.8ns | **4x** | + +### Timestamp Tracking + +Atomic timestamp reads for "last seen" tracking: + +| Operation | AtomicU64 | RwLock | Speedup | +|-----------|-----------|-----------------|---------| +| Read | **0.78ns** | 4.7ns | **6x** | +| Write | 18.7ns | 16.7ns | ~Equal | + +Reads dominate in production, making the 6x improvement significant. + +### Connection Affinity Lookup + +DashMap provides O(1) lookup regardless of concurrent request count: + +| Entries | Lookup (hit) | Lookup (miss) | +|---------|--------------|---------------| +| 10 | 12.3ns | 8.8ns | +| 100 | 13.5ns | 8.3ns | +| 1,000 | 14.0ns | 8.9ns | +| 10,000 | 12.9ns | 9.5ns | + +--- + +## Serialization Performance + +### MessagePack vs JSON + +MessagePack provides significant wins for larger payloads with many headers: + +**Serialization:** + +| Payload | JSON | MessagePack | Speedup | +|---------|------|-------------|---------| +| Small (204B) | 153ns | 150ns | 2% | +| Large (1080B) | 745ns | **562ns** | **25%** | + +**Deserialization:** + +| Payload | JSON | MessagePack | Speedup | +|---------|------|-------------|---------| +| Small (204B) | 403ns | 297ns | 26% | +| Large (894B) | 2.46us | **1.68us** | **32%** | + +**Wire size reduction:** + +``` +JSON small: 204 bytes +MessagePack small: 110 bytes (46% smaller) + +JSON large: 1080 bytes +MessagePack large: 894 bytes (17% smaller) +``` + +### When to Use MessagePack + +Enable the `binary-uds` feature for: + +- Processing request/response bodies (8-10x improvement) +- High header volume (25-32% improvement for large headers) +- Bandwidth-constrained environments (17-46% smaller payloads) + +Use JSON for: + +- Debugging and observability (human-readable) +- Interop with non-Rust agents lacking MessagePack support +- Small payloads where simplicity matters more + +--- + +## Body Chunk Streaming + +The most dramatic improvement is in body chunk handling, critical for WAF agents inspecting request bodies. + +### Serialization Throughput + +| Size | JSON + Base64 | MessagePack Binary | Speedup | +|------|---------------|-------------------|---------| +| 1KB | 1.97 GiB/s | **9.25 GiB/s** | **4.7x** | +| 4KB | 2.46 GiB/s | **27.2 GiB/s** | **11x** | +| 16KB | 2.51 GiB/s | **31.4 GiB/s** | **12.5x** | +| 64KB | 1.75 GiB/s | **4.62 GiB/s** | **2.6x** | + +### Deserialization Throughput + +| Size | JSON + Base64 | MessagePack Binary | Speedup | +|------|---------------|-------------------|---------| +| 1KB | 4.4 GiB/s | **20.4 GiB/s** | **4.6x** | +| 4KB | 6.6 GiB/s | **49.5 GiB/s** | **7.5x** | +| 16KB | 7.2 GiB/s | **48.7 GiB/s** | **6.8x** | +| 64KB | 7.5 GiB/s | **62.4 GiB/s** | **8.3x** | + +MessagePack with `serde_bytes` achieves **8-10x better throughput** by avoiding base64 encoding overhead. + +--- + +## Header Optimization (SmallVec) + +Most HTTP headers have a single value. SmallVec stores these inline, avoiding heap allocation. + +### Single-Value Headers (Common Case) + +| Container | Time | Notes | +|-----------|------|-------| +| Vec | 18.9ns | Heap allocation | +| SmallVec<[String; 1]> | **11.5ns** | Inline storage | + +**Speedup: 40%** for the most common case. + +### Header Map Creation (20 headers) + +| Container | Time | Speedup | +|-----------|------|---------| +| Vec-based map | 1.29us | - | +| SmallVec-based map | **1.07us** | **17%** | + +Multi-value headers (3+ values) spill to heap with ~5% overhead, but this case is rare in practice. + +--- + +## Protocol Metrics Overhead + +Built-in metrics add negligible overhead: + +| Operation | Time | +|-----------|------| +| Counter increment | **1.65ns** | +| Counter read | **0.31ns** | +| Histogram record | **2.61ns** | + +A typical request with 5 metric updates adds ~15ns total. + +--- + +## Full Request Path + +The complete hot path (excluding network I/O): + +1. Agent lookup (DashMap) +2. Affinity check (DashMap) +3. Health check (AtomicBool) +4. Counter increments (2x AtomicU64) +5. Serialization +6. Affinity store/clear (DashMap insert/remove) + +| Path | Time | +|------|------| +| JSON path | ~226ns | +| MessagePack path | ~226ns | + +**Total: ~230ns** for the complete hot path. + +--- + +## Comparison with Targets + +| Metric | Target | Achieved | Result | +|--------|--------|----------|--------| +| Connection selection | <1us | ~15ns | **67x better** | +| Health check | O(1) | 0.46ns | **Achieved** | +| Body throughput | >1 GiB/s | 62 GiB/s | **62x better** | +| Metrics overhead | Negligible | 2.6ns | **Achieved** | +| Affinity lookup | O(1) | ~13ns | **Achieved** | + +--- + +## Configuration Recommendations + +Based on benchmarks, these configurations work well for most deployments: + +### High Throughput + +```rust +let config = AgentPoolConfig { + connections_per_agent: 8, + load_balance_strategy: LoadBalanceStrategy::LeastConnections, + channel_buffer_size: 128, + ..Default::default() +}; +``` + +### Low Latency + +```rust +let config = AgentPoolConfig { + connections_per_agent: 4, + load_balance_strategy: LoadBalanceStrategy::LeastConnections, + request_timeout: Duration::from_millis(100), + channel_buffer_size: 32, + ..Default::default() +}; +``` + +### Body Inspection (WAF) + +Enable binary transport for body-heavy workloads: + +```kdl +agents { + agent "waf" type="waf" { + binary-uds "/var/run/zentinel/waf.sock" + events "request_headers" "request_body" + buffer-size 65536 + } +} +``` + +--- + +## Test Environment + +These benchmarks were collected on: + +- **Platform:** macOS Darwin 24.6.0 +- **Rust:** 1.92.0 (release build, LTO enabled) +- **Tool:** Criterion with 100 samples per benchmark + +For detailed methodology and raw data, see the [benchmark source code](https://github.com/zentinelproxy/zentinel/tree/main/crates/agent-protocol/benches). diff --git a/content/v/26.04/agents/v2/pooling.md b/content/v/26.04/agents/v2/pooling.md new file mode 100644 index 0000000..e4110e6 --- /dev/null +++ b/content/v/26.04/agents/v2/pooling.md @@ -0,0 +1,553 @@ ++++ +title = "Connection Pooling" +weight = 3 +updated = 2026-02-19 ++++ + +This document covers the AgentPool connection pooling system, including load balancing strategies, health tracking, and circuit breakers. + +## Overview + +The `AgentPool` maintains multiple connections per agent for: + +- **Higher throughput**: Parallel request processing +- **Lower latency**: Reduced connection overhead +- **Better reliability**: Automatic failover between connections +- **Smart routing**: Load-balanced request distribution + +``` +┌─────────────────────────────────────────────────────────────┐ +│ AgentPool │ +│ │ +│ ┌─────────────────┐ ┌─────────────────┐ │ +│ │ Agent: waf │ │ Agent: auth │ │ +│ │ │ │ │ │ +│ │ ┌───────────┐ │ │ ┌───────────┐ │ │ +│ │ │ Conn 1 │ │ │ │ Conn 1 │ │ │ +│ │ │ (gRPC) │ │ │ │ (UDS) │ │ │ +│ │ ├───────────┤ │ │ ├───────────┤ │ │ +│ │ │ Conn 2 │ │ │ │ Conn 2 │ │ │ +│ │ ├───────────┤ │ │ ├───────────┤ │ │ +│ │ │ Conn 3 │ │ │ │ Conn 3 │ │ │ +│ │ ├───────────┤ │ │ ├───────────┤ │ │ +│ │ │ Conn 4 │ │ │ │ Conn 4 │ │ │ +│ │ └───────────┘ │ │ └───────────┘ │ │ +│ │ │ │ │ │ +│ │ Health: OK │ │ Health: OK │ │ +│ │ In-flight: 12 │ │ In-flight: 8 │ │ +│ └─────────────────┘ └─────────────────┘ │ +│ │ +│ Load Balancer: LeastConnections │ +│ Circuit Breaker: Enabled │ +└─────────────────────────────────────────────────────────────┘ +``` + +--- + +## Configuration + +### Basic Setup + +```rust +use zentinel_agent_protocol::v2::{AgentPool, AgentPoolConfig, LoadBalanceStrategy}; +use std::time::Duration; + +let config = AgentPoolConfig { + connections_per_agent: 4, + load_balance_strategy: LoadBalanceStrategy::LeastConnections, + request_timeout: Duration::from_secs(30), + connect_timeout: Duration::from_secs(5), + health_check_interval: Duration::from_secs(10), + circuit_breaker_threshold: 5, + circuit_breaker_reset_timeout: Duration::from_secs(30), +}; + +let pool = AgentPool::with_config(config); +``` + +### Configuration Options + +| Option | Default | Description | +|--------|---------|-------------| +| `connections_per_agent` | 4 | Number of connections maintained per agent | +| `load_balance_strategy` | LeastConnections | How requests are distributed | +| `request_timeout` | 30s | Timeout for individual requests | +| `connect_timeout` | 5s | Timeout for establishing connections | +| `health_check_interval` | 10s | Interval between health checks | +| `circuit_breaker_threshold` | 5 | Failures before opening circuit | +| `circuit_breaker_reset_timeout` | 30s | Time before circuit resets | + +--- + +## Load Balancing Strategies + +### RoundRobin + +Distributes requests evenly across all connections in rotation. + +```rust +let config = AgentPoolConfig { + load_balance_strategy: LoadBalanceStrategy::RoundRobin, + ..Default::default() +}; +``` + +**Behavior**: +``` +Request 1 → Connection 1 +Request 2 → Connection 2 +Request 3 → Connection 3 +Request 4 → Connection 4 +Request 5 → Connection 1 (wraps around) +``` + +**Best for**: Uniform request processing times, simple distribution. + +### LeastConnections + +Routes to the connection with the fewest in-flight requests. + +```rust +let config = AgentPoolConfig { + load_balance_strategy: LoadBalanceStrategy::LeastConnections, + ..Default::default() +}; +``` + +**Behavior**: +``` +Connection 1: 3 in-flight +Connection 2: 1 in-flight ← Next request goes here +Connection 3: 4 in-flight +Connection 4: 2 in-flight +``` + +**Best for**: Variable request processing times, optimal latency. + +### HealthBased + +Prefers healthier connections based on recent error rates. + +```rust +let config = AgentPoolConfig { + load_balance_strategy: LoadBalanceStrategy::HealthBased, + ..Default::default() +}; +``` + +**Behavior**: +``` +Connection 1: Health 100%, Weight 1.0 +Connection 2: Health 95%, Weight 0.95 +Connection 3: Health 80%, Weight 0.80 (recent errors) +Connection 4: Health 100%, Weight 1.0 + +Weighted random selection favors healthy connections +``` + +**Best for**: Unreliable networks, degraded agent instances. + +### Random + +Random selection for simple distribution. + +**Best for**: Testing, simple deployments. + +--- + +## Health Tracking + +### Connection Health + +Each connection tracks: + +- **Success rate**: Percentage of successful requests +- **Average latency**: Recent request latencies +- **Last error**: Most recent error and timestamp +- **State**: Healthy, Degraded, or Unhealthy + +```rust +let health = pool.get_health("waf")?; + +println!("Agent: {}", health.agent_name); +println!("Connections: {}", health.total_connections); +println!("Healthy: {}", health.healthy_connections); +println!("Success rate: {:.2}%", health.success_rate * 100.0); +println!("Avg latency: {:?}", health.average_latency); +``` + +### Health States + +| State | Criteria | Behavior | +|-------|----------|----------| +| Healthy | Success rate > 95% | Normal routing | +| Degraded | Success rate 80-95% | Reduced weight in HealthBased | +| Unhealthy | Success rate < 80% | Minimal traffic, recovery checks | + +--- + +## Circuit Breaker + +### Overview + +The circuit breaker prevents cascading failures by temporarily disabling unhealthy agents. + +``` + ┌─────────┐ + │ Closed │ Normal operation + │ (Pass) │ + └────┬────┘ + │ threshold failures + ▼ + ┌─────────┐ + │ Open │ Fail fast, no requests sent + │ (Fail) │ + └────┬────┘ + │ reset_timeout elapsed + ▼ + ┌──────────┐ + │Half-Open │ Allow one test request + │ (Test) │ + └────┬─────┘ + │ + ┌────────┴────────┐ + │ │ + ▼ success ▼ failure +┌─────────┐ ┌─────────┐ +│ Closed │ │ Open │ +└─────────┘ └─────────┘ +``` + +### States + +| State | Behavior | +|-------|----------| +| **Closed** | Requests pass through normally | +| **Open** | Requests fail immediately with error | +| **Half-Open** | One request allowed to test recovery | + +### Monitoring + +```rust +let health = pool.get_health("waf")?; + +match health.circuit_breaker_state { + CircuitBreakerState::Closed => { + // Normal operation + } + CircuitBreakerState::Open { opened_at } => { + tracing::warn!("Circuit open since {:?}", opened_at); + } + CircuitBreakerState::HalfOpen => { + tracing::info!("Circuit testing recovery"); + } +} +``` + +--- + +## Metrics + +### Prometheus Export + +```rust +let prometheus_output = pool.metrics_collector().export_prometheus(); +``` + +Output: +```prometheus +# HELP agent_requests_total Total number of requests to agents +# TYPE agent_requests_total counter +agent_requests_total{agent="waf",decision="allow"} 15234 +agent_requests_total{agent="waf",decision="block"} 423 + +# HELP agent_request_duration_seconds Request duration histogram +# TYPE agent_request_duration_seconds histogram +agent_request_duration_seconds_bucket{agent="waf",le="0.001"} 5234 +agent_request_duration_seconds_bucket{agent="waf",le="0.005"} 12453 + +# HELP agent_connections_active Current number of active connections +# TYPE agent_connections_active gauge +agent_connections_active{agent="waf"} 4 + +# HELP agent_circuit_breaker_state Circuit breaker state (0=closed, 1=open) +# TYPE agent_circuit_breaker_state gauge +agent_circuit_breaker_state{agent="waf"} 0 +``` + +--- + +## Best Practices + +### 1. Size Your Pool Appropriately + +```rust +// For high-throughput: more connections +let high_throughput = AgentPoolConfig { + connections_per_agent: 8, + ..Default::default() +}; + +// For low-latency: fewer connections, faster timeouts +let low_latency = AgentPoolConfig { + connections_per_agent: 2, + request_timeout: Duration::from_millis(100), + ..Default::default() +}; +``` + +### 2. Choose the Right Load Balancer + +| Scenario | Recommended Strategy | +|----------|---------------------| +| Uniform workload | RoundRobin | +| Variable latency | LeastConnections | +| Unreliable agents | HealthBased | +| Testing | Random | + +### 3. Graceful Shutdown + +```rust +async fn shutdown(pool: &AgentPool) { + // Cancel all in-flight requests + for agent_name in pool.agent_names() { + if let Err(e) = pool.cancel_all(&agent_name).await { + tracing::error!("Failed to cancel requests for {}: {}", agent_name, e); + } + } + + // Wait for connections to drain + tokio::time::sleep(Duration::from_secs(5)).await; +} +``` + +--- + +## Protocol Metrics + +The `AgentPool` includes built-in protocol-level metrics for detailed monitoring. + +### Accessing Metrics + +```rust +// Get metrics instance +let metrics = pool.protocol_metrics(); + +// Get point-in-time snapshot +let snapshot = metrics.snapshot(); + +// Export to Prometheus format +let prometheus_text = metrics.to_prometheus("agent_protocol"); +``` + +### Available Metrics + +| Type | Metric | Description | +|------|--------|-------------| +| Counter | `requests_total` | Total requests sent | +| Counter | `responses_total` | Total responses received | +| Counter | `timeouts_total` | Requests that timed out | +| Counter | `connection_errors_total` | Connection failures | +| Counter | `flow_control_rejections_total` | Requests rejected due to flow control | +| Gauge | `in_flight_requests` | Current in-flight requests | +| Gauge | `healthy_connections` | Number of healthy connections | +| Gauge | `paused_connections` | Number of paused connections | +| Histogram | `serialization_time_us` | Serialization latency (μs) | +| Histogram | `request_duration_us` | End-to-end request latency (μs) | + +### Prometheus Export + +```rust +let prometheus = pool.protocol_metrics().to_prometheus("agent_protocol"); +``` + +Output: +```prometheus +# HELP agent_protocol_requests_total Total requests sent +# TYPE agent_protocol_requests_total counter +agent_protocol_requests_total 12345 + +# HELP agent_protocol_request_duration_us Request duration histogram +# TYPE agent_protocol_request_duration_us histogram +agent_protocol_request_duration_us_bucket{le="100"} 5234 +agent_protocol_request_duration_us_bucket{le="500"} 10453 +agent_protocol_request_duration_us_bucket{le="+Inf"} 12345 +``` + +--- + +## Connection Affinity + +For streaming requests, body chunks should be routed to the same connection as the initial headers. + +### Automatic Affinity + +When `send_request_headers` is called, the pool stores the selected connection for the correlation_id: + +```rust +// Headers sent to connection A +let response = pool.send_request_headers("waf", &headers).await?; + +// Body chunks automatically routed to connection A +pool.send_request_body_chunk("waf", &chunk1).await?; +pool.send_request_body_chunk("waf", &chunk2).await?; +``` + +### Manual Cleanup + +After a request completes, clear the affinity mapping: + +```rust +// Clear affinity for a specific correlation_id +pool.clear_correlation_affinity("correlation-123"); + +// Check current affinity count +let count = pool.correlation_affinity_count(); +``` + +--- + +## Flow Control Modes + +Configure how the pool behaves when an agent signals it cannot accept requests. + +### Configuration + +```rust +use zentinel_agent_protocol::v2::{AgentPoolConfig, FlowControlMode}; + +let config = AgentPoolConfig { + flow_control_mode: FlowControlMode::FailClosed, // Default + flow_control_wait_timeout: Duration::from_millis(100), + ..Default::default() +}; +``` + +### Available Modes + +| Mode | Behavior | Use Case | +|------|----------|----------| +| `FailClosed` | Returns error immediately | Strict backpressure | +| `FailOpen` | Skips agent, returns allow | Optional processing | +| `WaitAndRetry` | Waits up to timeout, then fails | Transient pauses | + +### Example: FailOpen for Analytics + +```rust +// Analytics agent is optional - don't fail requests if it's busy +let config = AgentPoolConfig { + flow_control_mode: FlowControlMode::FailOpen, + ..Default::default() +}; + +// If agent is paused, request proceeds without analytics +let response = pool.send_request_headers("analytics", &event).await?; +``` + +--- + +## Buffer Size Configuration + +Tune the internal channel buffer size for backpressure behavior: + +```rust +let config = AgentPoolConfig { + channel_buffer_size: 64, // Default + ..Default::default() +}; +``` + +| Scenario | Buffer Size | Trade-off | +|----------|-------------|-----------| +| Low latency | 16-32 | Tighter backpressure | +| High throughput | 64-128 | Better burst handling | +| Memory constrained | 8-16 | Lower memory use | + +--- + +## Sticky Sessions + +Ensure long-lived streaming connections (WebSocket, SSE) use the same agent connection. + +### Creating a Session + +```rust +// When WebSocket connects +pool.create_sticky_session("ws-12345", "waf-agent")?; +``` + +### Using Sticky Sessions + +```rust +// All messages use the same connection +let (response, used_sticky) = pool + .send_request_headers_with_sticky_session( + "ws-12345", + "waf-agent", + "corr-123", + &event, + ) + .await?; +``` + +### Session Management + +```rust +// Check if session exists +pool.has_sticky_session("ws-12345"); + +// Refresh session (updates last-accessed time) +pool.refresh_sticky_session("ws-12345"); + +// Clear when stream ends +pool.clear_sticky_session("ws-12345"); + +// Get active session count +let count = pool.sticky_session_count(); +``` + +### Automatic Expiry + +Sessions expire after `sticky_session_timeout` (default: 5 minutes): + +```rust +let config = AgentPoolConfig { + sticky_session_timeout: Some(Duration::from_secs(300)), + ..Default::default() +}; + +// Disable automatic expiry +let config = AgentPoolConfig { + sticky_session_timeout: None, + ..Default::default() +}; +``` + +| Scenario | Use Sticky Sessions? | +|----------|---------------------| +| WebSocket | Yes | +| Server-Sent Events | Yes | +| Long-polling | Yes | +| Regular HTTP | No (use correlation affinity) | + +--- + +## Performance Optimizations + +The `AgentPool` is optimized for high-throughput, low-latency operation: + +- **Lock-free agent lookup**: Uses `DashMap` for O(1) concurrent reads +- **Cached health state**: Atomic reads avoid async I/O in hot path +- **Synchronous connection selection**: No `.await` during selection +- **Atomic timestamp tracking**: `AtomicU64` instead of `RwLock` +- **Configurable flow control**: Choose fail-open or fail-closed behavior +- **Sticky session support**: Session affinity for streaming connections + +| Operation | Latency | Sync Points | +|-----------|---------|-------------| +| Agent lookup | ~100ns | 0 (lock-free) | +| Connection selection | ~1μs | 1 (try_read) | +| Health check (cached) | ~10ns | 0 (atomic) | +| Sticky session lookup | ~13ns | 0 (lock-free) | + +**Total hot-path sync points per request:** 2 diff --git a/content/v/26.04/agents/v2/protocol.md b/content/v/26.04/agents/v2/protocol.md new file mode 100644 index 0000000..e495b96 --- /dev/null +++ b/content/v/26.04/agents/v2/protocol.md @@ -0,0 +1,413 @@ ++++ +title = "Protocol Specification" +weight = 1 +updated = 2026-02-27 ++++ + +This document describes the v2 wire protocol for communication between the Zentinel proxy dataplane and external processing agents. + +## Protocol Constants + +| Constant | Value | Description | +|----------|-------|-------------| +| `PROTOCOL_VERSION` | `2` | Current protocol version | +| `MAX_MESSAGE_SIZE_GRPC` | `10,485,760` (10 MB) | Maximum message size for gRPC | +| `MAX_MESSAGE_SIZE_UDS` | `16,777,216` (16 MB) | Maximum message size for UDS binary | + +## Transport Options + +Protocol v2 supports three transport mechanisms: + +| Transport | Use Case | Latency | Features | +|-----------|----------|---------|----------| +| **gRPC over HTTP/2** | Remote agents, cross-network | ~1.2ms | TLS, flow control, streaming | +| **Binary over UDS** | Co-located agents | ~0.4ms | Lowest latency, simple format | +| **Reverse Connections** | NAT traversal, dynamic scaling | Varies | Agent-initiated connections | + +--- + +## gRPC Transport + +### Service Definition + +```protobuf +syntax = "proto3"; + +package zentinel.agent.v2; + +service AgentProcessorV2 { + // Bidirectional streaming for request/response lifecycle + rpc ProcessStream(stream AgentMessage) returns (stream AgentMessage); + + // Health check + rpc HealthCheck(HealthRequest) returns (HealthResponse); + + // Capability query + rpc GetCapabilities(CapabilityRequest) returns (CapabilityResponse); +} + +message AgentMessage { + uint64 request_id = 1; + oneof payload { + RequestHeaders request_headers = 2; + RequestBodyChunk request_body_chunk = 3; + ResponseHeaders response_headers = 4; + ResponseBodyChunk response_body_chunk = 5; + AgentDecision decision = 6; + CancelRequest cancel = 7; + } +} +``` + +### Streaming Semantics + +Unlike v1's request-response model, v2 uses bidirectional streaming: + +``` +Proxy Agent + │ │ + │ ──── RequestHeaders (id=1) ──────────► │ + │ ──── RequestBodyChunk (id=1) ────────► │ + │ │ + │ ◄──── Decision (id=1) ──────────────── │ + │ │ + │ ──── RequestHeaders (id=2) ──────────► │ (pipelined) + │ ──── CancelRequest (id=1) ───────────► │ (cancellation) + │ │ +``` + +### Message Ordering + +- Messages for a single `request_id` are ordered +- Messages for different `request_id`s may be interleaved +- `CancelRequest` terminates processing for a `request_id` + +--- + +## Binary UDS Transport + +### Wire Format + +``` +┌──────────────────┬──────────────────┬─────────────────────────────────┐ +│ Length (4 bytes) │ Type (1 byte) │ JSON Payload (variable length) │ +│ Big-endian u32 │ Message type ID │ UTF-8 encoded │ +└──────────────────┴──────────────────┴─────────────────────────────────┘ +``` + +- **Length prefix**: 4-byte unsigned integer in big-endian byte order (includes type byte) +- **Type byte**: Message type identifier (see table below) +- **Payload**: JSON-encoded message body +- **Maximum size**: 16 MB total + +### Message Types + +| Type ID | Name | Direction | Description | +|---------|------|-----------|-------------| +| `0x01` | `HandshakeRequest` | Proxy → Agent | Initial capability negotiation | +| `0x02` | `HandshakeResponse` | Agent → Proxy | Capability confirmation | +| `0x10` | `RequestHeaders` | Proxy → Agent | HTTP request headers | +| `0x11` | `RequestBodyChunk` | Proxy → Agent | Request body chunk | +| `0x12` | `ResponseHeaders` | Proxy → Agent | HTTP response headers | +| `0x13` | `ResponseBodyChunk` | Proxy → Agent | Response body chunk | +| `0x20` | `Decision` | Agent → Proxy | Processing decision | +| `0x21` | `BodyMutation` | Agent → Proxy | Body chunk mutation | +| `0x30` | `CancelRequest` | Proxy → Agent | Cancel in-flight request | +| `0x31` | `CancelAll` | Proxy → Agent | Cancel all requests | +| `0xF0` | `Ping` | Either | Keep-alive ping | +| `0xF1` | `Pong` | Either | Keep-alive response | + +### Example Frame + +``` +00 00 00 4A 10 {"request_id":1,"method":"GET","uri":"/api/users"...} +└────┬─────┘ └┘ └──────────────────────┬────────────────────────┘ + 74 bytes │ JSON payload (RequestHeaders) + │ + Type: RequestHeaders (0x10) +``` + +### Handshake Protocol + +Connection establishment requires a handshake: + +```rust +pub struct UdsHandshakeRequest { + pub protocol_version: u32, // Must be 2 + pub client_name: String, // Proxy identifier + pub supported_features: Vec, +} + +pub struct UdsHandshakeResponse { + pub protocol_version: u32, + pub agent_name: String, + pub capabilities: UdsCapabilities, +} + +pub struct UdsCapabilities { + pub handles_request_headers: bool, + pub handles_request_body: bool, + pub handles_response_headers: bool, + pub handles_response_body: bool, + pub supports_streaming: bool, + pub supports_cancellation: bool, + pub max_concurrent_requests: Option, +} +``` + +--- + +## Reverse Connections + +Reverse connections allow agents to connect to the proxy instead of the proxy connecting to agents. This enables: + +- Agents behind NAT/firewalls +- Dynamic agent scaling +- Load-based connection management + +### Registration Protocol + +When an agent connects via reverse connection: + +``` +Agent Proxy + │ │ + │ ──── Connect to listener socket ─────► │ + │ │ + │ ──── RegistrationRequest ────────────► │ + │ │ + │ ◄──── RegistrationResponse ─────────── │ + │ │ + │ (normal v2 protocol) │ + │ │ +``` + +### Registration Messages + +```rust +pub struct RegistrationRequest { + pub protocol_version: u32, // Must be 2 + pub agent_id: String, // Unique agent identifier + pub capabilities: UdsCapabilities, + pub auth_token: Option, // Optional authentication + pub metadata: Option, // Additional agent metadata +} + +pub struct RegistrationResponse { + pub accepted: bool, + pub error: Option, + pub assigned_id: Option, // Proxy-assigned connection ID + pub config: Option, // Optional pushed configuration +} +``` + +--- + +## Message Types (Detailed) + +### RequestHeaders + +Sent when HTTP request headers are received. + +```rust +pub struct RequestHeadersMessage { + pub request_id: u64, // Unique ID for this request + pub metadata: RequestMetadata, + pub method: String, + pub uri: String, + pub headers: Vec<(String, String)>, + pub has_body: bool, // Whether body chunks will follow +} +``` + +### RequestBodyChunk + +Sent for each chunk of the request body. + +```rust +pub struct RequestBodyChunkMessage { + pub request_id: u64, + pub chunk_index: u32, + pub data: String, // Base64-encoded bytes + pub is_last: bool, +} +``` + +### Decision + +Agent's processing decision for a request. + +```rust +pub struct DecisionMessage { + pub request_id: u64, + pub decision: Decision, + pub request_headers: Vec, + pub response_headers: Vec, + pub response_body_mutation: Option, + pub needs_more: bool, // Agent needs more events (e.g. body chunks) + pub audit: Option, +} + +pub struct BodyMutation { + pub data: Option, // None = pass through, Some("") = drop, Some(base64) = replace +} + +pub enum Decision { + Allow, + Block { status: u16, body: Option, headers: HashMap }, + Redirect { url: String, status: u16 }, +} +``` + +### ResponseHeaders + +Sent when upstream response headers are received. The agent can inspect the status code and headers, and return a Decision with `response_headers` operations to modify them before they are sent to the client. + +```rust +pub struct ResponseHeadersMessage { + pub request_id: u64, + pub metadata: RequestMetadata, + pub status: u16, // HTTP status code + pub headers: Vec<(String, String)>, +} +``` + +### ResponseBodyChunk + +Sent for each chunk of the upstream response body. The agent can accumulate chunks and return a `BodyMutation` in its Decision to replace the body content. + +```rust +pub struct ResponseBodyChunkMessage { + pub request_id: u64, + pub chunk_index: u32, + pub data: String, // Base64-encoded bytes + pub is_last: bool, + pub total_size: Option, // Total body size if known +} +``` + +### CancelRequest + +Cancels processing for a specific request. + +```rust +pub struct CancelRequestMessage { + pub request_id: u64, + pub reason: Option, +} +``` + +--- + +## Request Lifecycle + +### Request-Phase Flow + +``` +┌─────────┐ RequestHeaders ┌─────────┐ +│ Proxy │ ───────────────────► │ Agent │ +│ │ │ │ +│ │ RequestBodyChunk │ │ +│ │ ───────────────────► │ │ +│ │ (repeat) │ │ +│ │ │ │ +│ │ Decision │ │ +│ │ ◄─────────────────── │ │ +└─────────┘ └─────────┘ +``` + +### Response-Phase Flow + +Agents that subscribe to `response_headers` and `response_body` events can inspect and modify upstream responses. This enables use cases like image optimization, content transformation, and response body inspection. + +``` +┌─────────┐ RequestHeaders ┌─────────┐ +│ Proxy │ ─────────────────────► │ Agent │ +│ │ Decision (allow) │ │ +│ │ ◄───────────────────── │ │ +│ │ │ │ +│ │ (proxy forwards to upstream, │ +│ │ receives response) │ +│ │ │ │ +│ │ ResponseHeaders │ │ +│ │ ─────────────────────► │ │ +│ │ Decision │ │ +│ │ (+ header mods) │ │ +│ │ ◄───────────────────── │ │ +│ │ │ │ +│ │ ResponseBodyChunk │ │ +│ │ ─────────────────────► │ │ +│ │ Decision │ │ +│ │ (+ body mutation) │ │ +│ │ ◄───────────────────── │ │ +└─────────┘ └─────────┘ +``` + +**Response header modifications** are applied before headers are sent to the client. The agent can set, add, or remove headers using `HeaderOp` operations in the Decision message. + +**Response body mutations** replace the original body. The agent receives body chunks (base64-encoded), processes them, and returns a `BodyMutation` in the Decision: + +| `BodyMutation.data` | Behavior | +|----------------------|----------| +| `None` | Pass through original body unchanged | +| `Some("")` | Drop the body chunk | +| `Some("")` | Replace body with decoded base64 data | + +When response body processing is active, the proxy sets `Connection: close` and removes `Content-Length` since the body size may change. + +### Cancellation Flow + +``` +┌─────────┐ RequestHeaders ┌─────────┐ +│ Proxy │ ───────────────────► │ Agent │ +│ │ │ │ +│ │ CancelRequest │ │ +│ │ ───────────────────► │ │ +│ │ │ │ +│ │ (agent cleans up) │ │ +└─────────┘ └─────────┘ +``` + +--- + +## Protocol Guarantees + +### Ordering + +1. Messages for a single `request_id` are delivered in order +2. Messages for different requests may be interleaved +3. `CancelRequest` is processed immediately, discarding pending messages + +### Reliability + +1. Each message must be acknowledged (Decision for requests) +2. Timeouts are enforced per-message and per-request +3. Connection failures trigger reconnection with backoff + +### Concurrency + +1. Multiple requests can be in-flight simultaneously +2. `max_concurrent_requests` in capabilities limits concurrency +3. Backpressure via flow control (gRPC) or queue bounds (UDS) + +--- + +## Compatibility + +### v1 to v2 Migration + +| v1 Feature | v2 Equivalent | +|------------|---------------| +| Length-prefixed JSON | Binary UDS (type byte added) | +| Unary gRPC calls | Bidirectional streaming | +| Per-request connections | Multiplexed connections | +| N/A | Request cancellation | +| N/A | Reverse connections | + +### Version Negotiation + +- gRPC: Service name includes version (`AgentProcessorV2`) +- UDS: `protocol_version` field in handshake +- Reverse: `protocol_version` field in registration + +Agents should reject connections with incompatible versions. diff --git a/content/v/26.04/agents/v2/reverse-connections.md b/content/v/26.04/agents/v2/reverse-connections.md new file mode 100644 index 0000000..9785959 --- /dev/null +++ b/content/v/26.04/agents/v2/reverse-connections.md @@ -0,0 +1,474 @@ ++++ +title = "Reverse Connections" +weight = 5 +updated = 2026-02-19 ++++ + +This document provides detailed coverage of the reverse connection feature in Agent Protocol v2, which allows agents to connect to the proxy instead of the proxy connecting to agents. + +## Overview + +Traditional agent deployment requires the proxy to initiate connections to agents: + +``` +┌─────────┐ ┌─────────┐ +│ Proxy │ ──── Connect ────► │ Agent │ +└─────────┘ └─────────┘ +``` + +This model has limitations: +- Agents behind NAT cannot be reached +- Firewall rules must allow inbound connections to agents +- Static agent discovery required +- Scaling requires configuration changes + +**Reverse connections** flip this model: + +``` +┌─────────┐ ┌─────────┐ +│ Proxy │ ◄──── Connect ──── │ Agent │ +│ │ │ (NAT) │ +│ Listener│ │ │ +└─────────┘ └─────────┘ +``` + +Benefits: +- **NAT Traversal**: Agents behind NAT/firewalls can connect out +- **Dynamic Scaling**: Agents register on startup, no config changes +- **Zero-Config Discovery**: Agents announce their capabilities +- **Load-Based Pooling**: Agents can open multiple connections + +--- + +## Architecture + +### Component Overview + +``` +┌─────────────────────────────────────────────────────────────┐ +│ Proxy │ +│ │ +│ ┌─────────────────────────────────────────────────────┐ │ +│ │ ReverseConnectionListener │ │ +│ │ │ │ +│ │ ┌─────────────┐ ┌─────────────┐ │ │ +│ │ │ UDS Socket │ │ TCP Socket │ │ │ +│ │ │ (local) │ │ (remote) │ │ │ +│ │ └──────┬──────┘ └──────┬──────┘ │ │ +│ │ │ │ │ │ +│ │ └────────────────┘ │ │ +│ │ │ │ │ +│ │ ▼ │ │ +│ │ ┌───────────────┐ │ │ +│ │ │ Registration │ │ │ +│ │ │ Validator │ │ │ +│ │ └───────┬───────┘ │ │ +│ │ │ │ │ +│ └─────────────────┼────────────────────────────────────┘ │ +│ │ │ +│ ▼ │ +│ ┌─────────────────────────────────────────────────────┐ │ +│ │ AgentPool │ │ +│ │ ┌──────────┐ ┌──────────┐ ┌──────────┐ │ │ +│ │ │ waf-1 │ │ waf-2 │ │ auth-1 │ │ │ +│ │ │(reverse) │ │(reverse) │ │(reverse) │ │ │ +│ │ └──────────┘ └──────────┘ └──────────┘ │ │ +│ └─────────────────────────────────────────────────────┘ │ +│ │ +└──────────────────────────────────────────────────────────────┘ +``` + +### Registration Flow + +``` +Agent Proxy + │ │ + │ 1. TCP/UDS Connect │ + │ ───────────────────────────────────────────────────────►│ + │ │ + │ 2. RegistrationRequest │ + │ { │ + │ protocol_version: 2, │ + │ agent_id: "waf-worker-3", │ + │ capabilities: { │ + │ handles_request_headers: true, │ + │ handles_request_body: true, │ + │ supports_cancellation: true, │ + │ max_concurrent_requests: 100 │ + │ }, │ + │ auth_token: "secret-token", │ + │ metadata: { "version": "1.2.0" } │ + │ } │ + │ ───────────────────────────────────────────────────────►│ + │ │ + │ 3. Validate │ + │ - Auth │ + │ - Allowlist │ + │ │ + │ 4. RegistrationResponse │ + │ { │ + │ accepted: true, │ + │ assigned_id: "waf-worker-3-conn-7", │ + │ config: { "rules_version": "3.4.0" } │ + │ } │ + │ ◄───────────────────────────────────────────────────────│ + │ │ + │ 5. Normal v2 protocol │ + │ ◄──────────────────────────────────────────────────────►│ + │ │ +``` + +--- + +## Listener Configuration + +### Basic Setup + +```rust +use zentinel_agent_protocol::v2::{ + ReverseConnectionListener, + ReverseConnectionConfig, +}; +use std::time::Duration; + +let config = ReverseConnectionConfig { + handshake_timeout: Duration::from_secs(10), + max_connections_per_agent: 4, + require_auth: false, + allowed_agents: None, +}; + +// UDS listener for local agents +let listener = ReverseConnectionListener::bind_uds( + "/var/run/zentinel/agents.sock", + config.clone(), +).await?; + +// TCP listener for remote agents +let listener = ReverseConnectionListener::bind_tcp( + "0.0.0.0:9090", + config, +).await?; +``` + +### Configuration Options + +| Option | Default | Description | +|--------|---------|-------------| +| `handshake_timeout` | 10s | Time allowed for registration handshake | +| `max_connections_per_agent` | 4 | Max connections from same agent_id | +| `require_auth` | false | Require auth_token in registration | +| `allowed_agents` | None | Allowlist of agent IDs (supports wildcards) | + +### Security Configuration + +```rust +let config = ReverseConnectionConfig { + // Require authentication + require_auth: true, + + // Only allow specific agents + allowed_agents: Some(vec![ + "waf-*".to_string(), // Wildcard: any waf-prefixed agent + "auth-primary".to_string(), // Exact match + "auth-secondary".to_string(), + ]), + + // Shorter timeout for faster failure detection + handshake_timeout: Duration::from_secs(5), + + ..Default::default() +}; +``` + +--- + +## Accepting Connections + +### Simple Accept Loop + +```rust +let pool = AgentPool::new(); +let listener = ReverseConnectionListener::bind_uds( + "/var/run/zentinel/agents.sock", + ReverseConnectionConfig::default(), +).await?; + +// Accept loop +loop { + match listener.accept().await { + Ok((client, registration)) => { + tracing::info!( + agent_id = %registration.agent_id, + capabilities = ?registration.capabilities, + "Agent connected" + ); + + // Add to pool + if let Err(e) = pool.add_reverse_connection( + ®istration.agent_id, + client, + registration.capabilities, + ).await { + tracing::error!("Failed to add agent: {}", e); + } + } + Err(e) => { + tracing::error!("Accept error: {}", e); + } + } +} +``` + +### Production Accept Loop + +```rust +use tokio::select; +use tokio::sync::broadcast; + +async fn run_accept_loop( + listener: ReverseConnectionListener, + pool: AgentPool, + mut shutdown: broadcast::Receiver<()>, +) { + loop { + select! { + result = listener.accept() => { + match result { + Ok((client, registration)) => { + handle_new_connection(&pool, client, registration).await; + } + Err(e) => { + tracing::error!("Accept error: {}", e); + tokio::time::sleep(Duration::from_millis(100)).await; + } + } + } + _ = shutdown.recv() => { + tracing::info!("Shutting down accept loop"); + break; + } + } + } +} +``` + +--- + +## Agent-Side Implementation + +### Connecting to Proxy + +```rust +use tokio::net::UnixStream; +use zentinel_agent_protocol::v2::reverse::{ + RegistrationRequest, + RegistrationResponse, + write_registration_request, + read_registration_response, +}; + +async fn connect_to_proxy( + socket_path: &str, + agent_id: &str, + auth_token: Option, +) -> Result> { + // Connect to proxy listener + let mut stream = UnixStream::connect(socket_path).await?; + + // Build registration request + let request = RegistrationRequest { + protocol_version: 2, + agent_id: agent_id.to_string(), + capabilities: UdsCapabilities { + handles_request_headers: true, + handles_request_body: true, + handles_response_headers: true, + handles_response_body: false, + supports_streaming: true, + supports_cancellation: true, + max_concurrent_requests: Some(100), + }, + auth_token, + metadata: Some(serde_json::json!({ + "version": env!("CARGO_PKG_VERSION"), + })), + }; + + // Send registration + write_registration_request(&mut stream, &request).await?; + + // Read response + let response = read_registration_response(&mut stream).await?; + + if !response.accepted { + return Err(format!( + "Registration rejected: {}", + response.error.unwrap_or_default() + ).into()); + } + + tracing::info!( + assigned_id = ?response.assigned_id, + "Registered with proxy" + ); + + Ok(stream) +} +``` + +### Connection Pool on Agent Side + +For high availability, agents should maintain multiple connections: + +```rust +struct AgentConnectionManager { + socket_path: String, + agent_id: String, + auth_token: Option, + target_connections: usize, +} + +impl AgentConnectionManager { + pub async fn run(&self) { + loop { + // Maintain target number of connections + while active_connections() < self.target_connections { + match self.establish_connection().await { + Ok(stream) => { + tokio::spawn(async move { + handle_connection(stream).await; + }); + } + Err(e) => { + tracing::error!("Connection failed: {}", e); + tokio::time::sleep(Duration::from_secs(5)).await; + } + } + } + tokio::time::sleep(Duration::from_secs(1)).await; + } + } +} +``` + +--- + +## Error Handling + +### Registration Errors + +| Error | Cause | Resolution | +|-------|-------|------------| +| Version mismatch | Protocol version != 2 | Update agent to v2 | +| Auth failed | Invalid or missing token | Check auth configuration | +| Not allowed | Agent ID not in allowlist | Add to allowed_agents | +| Connection limit | Too many connections | Wait or reduce connections | +| Timeout | Handshake took too long | Check network/agent health | + +### Handling Disconnects + +```rust +// Agent side: reconnect loop with exponential backoff +let mut backoff = Duration::from_secs(1); + +loop { + match connect_and_handle().await { + Ok(()) => { + tracing::info!("Connection closed normally"); + backoff = Duration::from_secs(1); // Reset on success + } + Err(e) => { + tracing::error!("Connection error: {}", e); + } + } + + tokio::time::sleep(backoff).await; + backoff = std::cmp::min(backoff * 2, Duration::from_secs(60)); +} +``` + +--- + +## Best Practices + +### 1. Use Multiple Connections Per Agent + +```rust +// Agent side: maintain 4 connections for load distribution +let manager = AgentConnectionManager::new( + "/var/run/zentinel/agents.sock", + "waf-worker-1", + Some("auth-token".to_string()), + 4, // target connections +); +``` + +### 2. Include Useful Metadata + +```rust +let request = RegistrationRequest { + // ... + metadata: Some(serde_json::json!({ + "version": env!("CARGO_PKG_VERSION"), + "hostname": hostname::get()?.to_string_lossy(), + "pid": std::process::id(), + "started_at": chrono::Utc::now().to_rfc3339(), + "features": ["waf", "rate-limiting"], + })), +}; +``` + +### 3. Handle Configuration Pushes + +```rust +if let Some(config) = response.config { + // Hot-reload configuration + if let Some(rules_version) = config.get("rules_version") { + reload_rules(rules_version.as_str().unwrap())?; + } +} +``` + +### 4. Implement Health Monitoring + +```rust +// Agent side: track connection health +let mut consecutive_errors = 0; + +loop { + match handle_next_request(&mut stream).await { + Ok(()) => { + consecutive_errors = 0; + } + Err(e) => { + consecutive_errors += 1; + if consecutive_errors > 5 { + tracing::warn!("Too many errors, reconnecting"); + break; + } + } + } +} +``` + +--- + +## KDL Configuration + +Configure reverse connection listener in your Zentinel config: + +```kdl +reverse-listener { + path "/var/run/zentinel/agents.sock" + max-connections-per-agent 4 + handshake-timeout "10s" + + // Optional: TCP listener for remote agents + // tcp-address "0.0.0.0:9090" + + // Security settings + require-auth true + allowed-agents "waf-*" "auth-agent" +} +``` diff --git a/content/v/26.04/agents/v2/transports.md b/content/v/26.04/agents/v2/transports.md new file mode 100644 index 0000000..ded28bf --- /dev/null +++ b/content/v/26.04/agents/v2/transports.md @@ -0,0 +1,335 @@ ++++ +title = "Transport Options" +weight = 4 +updated = 2026-02-19 ++++ + +This document covers the three transport mechanisms available in Agent Protocol v2: gRPC, Unix Domain Sockets (UDS), and Reverse Connections. + +## Transport Comparison + +| Feature | gRPC | UDS Binary | Reverse Connection | +|---------|------|------------|-------------------| +| **Latency** | ~1.2ms | ~0.4ms | ~0.5ms | +| **Throughput** | 28K req/s | 45K req/s | 40K req/s | +| **TLS Support** | Yes | N/A (local) | Yes | +| **Cross-network** | Yes | No | Yes | +| **NAT Traversal** | No | No | Yes | +| **Max Message** | 10 MB | 16 MB | 16 MB | +| **Flow Control** | HTTP/2 | Manual | Manual | + +--- + +## gRPC Transport + +### Overview + +gRPC over HTTP/2 is the best choice for: +- Remote agents across networks +- Agents requiring TLS encryption +- Language-agnostic implementations +- Complex streaming scenarios + +### Client Setup + +```rust +use zentinel_agent_protocol::v2::AgentClientV2; +use std::time::Duration; + +// Basic connection +let client = AgentClientV2::connect( + "waf-agent", + "http://localhost:50051", + Duration::from_secs(30), +).await?; + +// With TLS +use zentinel_agent_protocol::v2::TlsConfig; + +let tls_config = TlsConfig { + ca_cert: Some("/path/to/ca.crt".into()), + client_cert: Some("/path/to/client.crt".into()), + client_key: Some("/path/to/client.key".into()), + verify_server: true, +}; + +let client = AgentClientV2::connect_with_tls( + "waf-agent", + "https://waf.internal:50051", + tls_config, + Duration::from_secs(30), +).await?; +``` + +### Streaming Semantics + +gRPC v2 uses bidirectional streaming for efficient request handling: + +``` +Proxy Agent + │ │ + │ ──── RequestHeaders (id=1) ──────────► │ + │ ──── RequestBodyChunk (id=1) ────────► │ + │ │ + │ ◄──── Decision (id=1) ──────────────── │ + │ │ + │ ──── RequestHeaders (id=2) ──────────► │ (pipelined) + │ ──── CancelRequest (id=1) ───────────► │ (cancellation) + │ │ +``` + +--- + +## Unix Domain Socket (UDS) Transport + +### Overview + +UDS binary transport is the best choice for: +- Co-located agents on the same host +- Lowest possible latency requirements +- High-throughput local processing +- Simple deployment without TLS + +### Wire Format + +``` +┌──────────────────┬──────────────────┬─────────────────────────────────┐ +│ Length (4 bytes) │ Type (1 byte) │ JSON Payload (variable length) │ +│ Big-endian u32 │ Message type ID │ UTF-8 encoded │ +└──────────────────┴──────────────────┴─────────────────────────────────┘ +``` + +### Client Setup + +```rust +use zentinel_agent_protocol::v2::AgentClientV2Uds; +use std::time::Duration; + +let client = AgentClientV2Uds::connect( + "auth-agent", + "/var/run/zentinel/auth.sock", + Duration::from_secs(30), +).await?; + +// Query capabilities after handshake +let caps = client.capabilities(); +println!("Agent: {}", caps.agent_name); +println!("Streaming: {}", caps.supports_streaming); +``` + +### Handshake Protocol + +UDS connections begin with a handshake: + +``` +Proxy Agent + │ │ + │ ──── Connect ────────────────────────────────► │ + │ │ + │ ──── HandshakeRequest ─────────────────────────► │ + │ { │ + │ protocol_version: 2, │ + │ client_name: "zentinel-proxy", │ + │ supported_features: ["streaming", ...] │ + │ } │ + │ │ + │ ◄──────────────────────── HandshakeResponse ─── │ + │ { │ + │ protocol_version: 2, │ + │ agent_name: "auth-agent", │ + │ capabilities: { ... } │ + │ } │ + │ │ + │ (normal message flow) │ + │ │ +``` + +### Message Types + +| Type ID | Name | Direction | +|---------|------|-----------| +| `0x01` | HandshakeRequest | Proxy → Agent | +| `0x02` | HandshakeResponse | Agent → Proxy | +| `0x10` | RequestHeaders | Proxy → Agent | +| `0x11` | RequestBodyChunk | Proxy → Agent | +| `0x12` | ResponseHeaders | Proxy → Agent | +| `0x13` | ResponseBodyChunk | Proxy → Agent | +| `0x20` | Decision | Agent → Proxy | +| `0x30` | CancelRequest | Proxy → Agent | +| `0x31` | CancelAll | Proxy → Agent | +| `0xF0` | Ping | Either | +| `0xF1` | Pong | Either | + +### Binary Encoding (MessagePack) + +UDS supports MessagePack encoding for improved performance over JSON. Encoding is negotiated during the handshake. + +**Enable in Cargo.toml:** + +```toml +zentinel-agent-protocol = { version = "0.3", features = ["binary-uds"] } +``` + +**Handshake with encoding negotiation:** + +``` +Proxy Agent + │ │ + │ ──── HandshakeRequest ─────────────────────────► │ + │ { supported_encodings: ["msgpack", "json"] }│ + │ │ + │ ◄──────────────────────── HandshakeResponse ─── │ + │ { encoding: "msgpack" } │ + │ │ + │ (subsequent messages use msgpack) │ +``` + +**Available encodings:** + +| Encoding | Pros | Cons | +|----------|------|------| +| `json` | Human readable, always available | Larger payloads, slower | +| `msgpack` | Compact, fast serialization | Requires `binary-uds` feature | + +### Zero-Copy Body Streaming + +For large request/response bodies, use binary body chunk methods to avoid base64 encoding overhead: + +```rust +use zentinel_agent_protocol::{BinaryRequestBodyChunkEvent, Bytes}; + +// Create binary body chunk (no base64) +let chunk = BinaryRequestBodyChunkEvent::new( + "correlation-123", + Bytes::from_static(b"raw binary data"), + 0, // chunk_index + false, // is_last +); + +// Send via UDS client +// - With MessagePack: raw bytes (most efficient) +// - With JSON: falls back to base64 +client.send_request_body_chunk_binary(&chunk).await?; +``` + +**Performance comparison (1KB body chunk):** + +| Method | Encoding | Serialized Size | +|--------|----------|-----------------| +| `send_request_body_chunk` | JSON + base64 | ~1,450 bytes | +| `send_request_body_chunk_binary` | MessagePack | ~1,050 bytes | + +--- + +## Reverse Connections + +### Overview + +Reverse connections allow agents to connect to the proxy instead of the proxy connecting to agents. This enables: + +- Agents behind NAT/firewalls +- Dynamic agent scaling +- Cloud-native deployments +- Zero-config agent discovery + +### Architecture + +``` +Agent Proxy + │ │ + │ ──── TCP/UDS Connect ───────►│ + │ │ + │ ──── RegistrationRequest ──►│ + │ - agent_id │ + │ - capabilities │ + │ - auth_token │ + │ │ + │ ◄── RegistrationResponse ───│ + │ - accepted │ + │ - config │ + │ │ + │ (bidirectional protocol) │ + │ │ +``` + +### Listener Setup + +```rust +use zentinel_agent_protocol::v2::{ + ReverseConnectionListener, + ReverseConnectionConfig, +}; + +let config = ReverseConnectionConfig { + handshake_timeout: Duration::from_secs(10), + max_connections_per_agent: 4, + require_auth: true, + allowed_agents: Some(vec!["waf-*".to_string(), "auth-agent".to_string()]), +}; + +// UDS listener for local agents +let listener = ReverseConnectionListener::bind_uds( + "/var/run/zentinel/agents.sock", + config, +).await?; + +// TCP listener for remote agents +let listener = ReverseConnectionListener::bind_tcp( + "0.0.0.0:9090", + config, +).await?; +``` + +See [Reverse Connections](reverse-connections/) for detailed setup instructions. + +--- + +## V2Transport Abstraction + +The `V2Transport` enum provides a unified interface across all transport types: + +```rust +use zentinel_agent_protocol::v2::V2Transport; + +pub enum V2Transport { + Grpc(AgentClientV2), + Uds(AgentClientV2Uds), + Reverse(ReverseConnectionClient), +} + +// All transports support the same operations +impl V2Transport { + pub async fn send_request_headers(&mut self, headers: &RequestHeaders) + -> Result; + pub async fn send_request_body_chunk(&mut self, chunk: &RequestBodyChunk) + -> Result; + pub async fn cancel_request(&mut self, request_id: u64) + -> Result<(), AgentProtocolError>; + pub fn is_healthy(&self) -> bool; +} +``` + +--- + +## Choosing a Transport + +| Scenario | Recommended Transport | +|----------|----------------------| +| Same host, lowest latency | UDS Binary | +| Remote agent, needs TLS | gRPC | +| Agent behind NAT/firewall | Reverse Connection | +| Cloud-native, dynamic scaling | Reverse Connection | +| Cross-language agent | gRPC | +| Simple local deployment | UDS Binary | +| Mixed environment | AgentPool (auto-detect) | + +### Auto-Detection in AgentPool + +```rust +let pool = AgentPool::new(); + +// Transport is auto-detected from endpoint format +pool.add_agent("local", "/var/run/agent.sock").await?; // → UDS +pool.add_agent("remote", "waf.internal:50051").await?; // → gRPC +pool.add_agent("https", "https://waf.example.com").await?; // → gRPC+TLS +``` diff --git a/content/v/26.04/appendix/_index.md b/content/v/26.04/appendix/_index.md new file mode 100644 index 0000000..6054dda --- /dev/null +++ b/content/v/26.04/appendix/_index.md @@ -0,0 +1,19 @@ ++++ +title = "Appendices" +weight = 11 +sort_by = "weight" +template = "section.html" ++++ + +Supplementary reference material for Zentinel. + +## Contents + +| Page | Description | +|------|-------------| +| [Changelog](changelog/) | Version history and changes | +| [Versioning](versioning/) | Release versions, crate versions, and upgrade guides | +| [FAQ](faq/) | Frequently asked questions | +| [Glossary](glossary/) | Term definitions | +| [License](license/) | Apache 2.0 license | + diff --git a/content/v/26.04/appendix/changelog.md b/content/v/26.04/appendix/changelog.md new file mode 100644 index 0000000..8399695 --- /dev/null +++ b/content/v/26.04/appendix/changelog.md @@ -0,0 +1,333 @@ ++++ +title = "Changelog" +weight = 1 +updated = 2026-03-01 ++++ + +All notable changes to Zentinel are documented here. + +The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.1.0/). +Zentinel uses [CalVer](https://calver.org/) (`YY.MM_PATCH`) for releases and +[SemVer](https://semver.org/) for crate versions on crates.io. CalVer is the +primary, operator-facing version. See [Versioning](../versioning/) for details. + +## Release Overview + +| CalVer | Crate Version | Date | Highlights | +|--------|---------------|------|------------| +| [26.03_1](#26-03-1) | 0.5.12 | 2026-03-01 | March release, image optimization agent v0.2.0 | +| [26.02_5](#26-02-5) | 0.5.11 | 2026-02-27 | `include` directive support in single-file config loading | +| [26.02_4](#26-02-4) | 0.4.10 | 2026-02-04 | Install script fix, CI workflows, Pingora fork security fix | +| [26.02_3](#26-02-3) | 0.4.9 | 2026-02-03 | First-time user smoke tests, protocol-version config, docs refresh | +| [26.02_1](#26-02-1) | 0.4.7 | 2026-02-02 | Pingora 0.7 upgrade, drop fork, major dependency sweep | +| [26.02_0](#26-02-0) | 0.4.5 | 2026-01-29 | Supply chain security: SBOM, cosign signing, SLSA provenance | +| [26.01_11](#26-01-11) | 0.4.5 | 2026-01-29 | Per-request allocation reduction in hot path | +| [26.01_10](#26-01-10) | 0.4.3 | 2026-01-27 | Security fixes, dependency updates | +| [26.01_9](#26-01-9) | 0.4.2 | 2026-01-21 | Sticky load balancing, install script UX | +| [26.01_8](#26-01-8) | 0.4.1 | 2026-01-21 | Dependency updates (prost, tonic, tungstenite, sysinfo) | +| [26.01_7](#26-01-7) | 0.4.0 | 2026-01-21 | DNS-01 ACME challenge support | +| [26.01_6](#26-01-6) | 0.3.1 | 2026-01-14 | Agent Protocol v2 connection pooling | +| [26.01_4](#26-01-4) | 0.3.0 | 2026-01-11 | Agent Protocol v2, WASM runtime | +| [26.01_3](#26-01-3) | 0.2.3 | 2026-01-05 | Bug fixes | +| [26.01_0](#26-01-0) | 0.2.0 | 2026-01-01 | First CalVer release | +| [25.12](#25-12) | 0.1.x | 2025-12 | Initial public releases | + +--- + +## 26.03_1 + +**Date:** 2026-03-01 +**Crate version:** 0.5.12 + +### Changed +- **Image optimization agent v0.2.0** — Content-Type header is now set correctly during response header phase (proxy commits headers before body filtering). Conversion fallback paths restore original Content-Type. Cache directory defaults to `~/.cache/zentinel/image-optimization` instead of requiring root access. Fixed event name `response_body` → `response_body_chunk` in agent manifest. + +--- + +## 26.02_5 + +**Date:** 2026-02-27 +**Crate version:** 0.5.11 + +### Added +- **`include` directive in single-file config** — `include "routes/*.kdl"` now works directly in `zentinel.kdl` when loaded via `Config::from_file()` or `zentinel --config`. Previously, include directives only worked through the multi-file loader (`--config-dir`). Includes support glob patterns, relative path resolution, recursive expansion, and circular include detection. + +### Changed +- **Improved error message for `include` in raw KDL** — When `include` is encountered via `Config::from_kdl()` (raw string parsing), the error now explains to use `Config::from_file()` instead of showing the generic "unknown block" message. + +--- + +## 26.02_4 + +**Date:** 2026-02-04 +**Crate version:** 0.4.10 + +### Fixed +- **Install script** — `get_latest_version()` now queries `/releases` and selects the first release with actual binary assets, instead of relying on `/releases/latest` which could point to a release without binaries ([#67](https://github.com/zentinelproxy/zentinel/issues/67)). +- **Release workflow** — Version bump push to `main` now falls back to creating a PR when blocked by branch protection. +- **16 rustdoc warnings** — Fixed bare URLs, unclosed HTML tags, unresolved type references, and private module links across 10 files. +- **Clippy warnings** — Resolved warnings and migrated to updated dependency APIs. +- **`_build.yml` header comment** — Fixed misleading "Called by" reference. + +### Changed +- **Pingora switched to fork** — All Pingora dependencies now point to `raskell-io/pingora` fork (rev `5847d5e`) which disables the prometheus protobuf default feature, removing the RUSTSEC-2024-0437 vulnerability. +- **Dependency updates:** + - `cargo update` — 61 packages updated to latest compatible versions + - reqwest 0.12 → 0.13 (feature renames: `rustls-tls` → `rustls`, `query` now opt-in) + - jsonschema 0.40 → 0.41 (performance improvements) + - bytes 1.9 → 1.11.1 (integer overflow fix) + +### Added +- **CI workflow** (`.github/workflows/ci.yml`) — Formatting, clippy, tests, and docs checks on PRs and pushes to main. +- **Weekly audit workflow** (`.github/workflows/audit.yml`) — Runs `cargo audit` weekly, creates/updates GitHub issues on vulnerabilities. +- **Cargo audit ignore list** (`.cargo/audit.toml`) — Documented ignores for upstream-only advisories (daemonize, derivative, fxhash, rustls-pemfile). +- **Branch protection** — Required status checks (Formatting, Clippy, Tests, Documentation) on main. + +--- + +## 26.02_3 + +**Date:** 2026-02-03 +**Crate version:** 0.4.9 + +### Added +- **First-time user smoke tests** — Self-contained integration tests (`test_first_time_waf.sh`, `test_first_time_lua.sh`) that validate building Zentinel + an agent from source, wiring them together, and verifying end-to-end behavior. WAF test covers 8 scenarios (SQLi, XSS, path traversal, fail-open, recovery); Lua test covers 4 (header injection, blocking, fail-open). +- **`protocol-version` KDL config** — Agent blocks now accept `protocol-version "v2"` to explicitly select Protocol v2 for gRPC agents, instead of always defaulting to v1. +- **Makefile targets** — `test-first-time`, `test-first-time-waf`, `test-first-time-lua` for running smoke tests. + +### Fixed +- **Example configs** — All configs in `config/examples/` now pass `zentinel test` validation. +- **Install script** — Removed stale linux-arm64 block, fixed sudo fallback. + +### Changed +- **README** — Replaced Inference Gateway section with Use Cases overview; updated feature table with caching, WebSocket, hot reload details; linked to full features page. + +--- + +## 26.02_1 + +**Date:** 2026-02-02 +**Crate version:** 0.4.7 + +### Changed +- **Pingora 0.6 → 0.7** — Upgraded to upstream Pingora 0.7.0, removing the `raskell-io/pingora` security fork and all 16 `[patch.crates-io]` overrides. Zentinel now builds against upstream Pingora with zero patches. + - `ForcedInvalidationKind` renamed to `ForcedFreshness` in cache layer + - `range_header_filter` now accepts `max_multipart_ranges` parameter (defaults to 200) +- **Major dependency updates:** + - thiserror 1.x → 2.0 + - redis 0.27 → 1.0 (distributed rate limiting) + - criterion 0.6 → 0.8 (benchmarking) + - instant-acme 0.7 → 0.8 (ACME client rewritten for new builder/stream API) + - jsonschema 0.18 → 0.40 (validation module rewritten for new API: `JSONSchema` → `Validator`, `compile` → `draft7::new`) + - quick-xml 0.37 → 0.39 (data masking agent: `unescape()` → `decode()`) + - async-memcached 0.5 → 0.6 + - tiktoken-rs 0.6 → 0.9 + - sysinfo 0.37 → 0.38 + +### Security +- **Resolved all three security issues** previously requiring a Pingora fork: + - [RUSTSEC-2026-0002](https://rustsec.org/advisories/RUSTSEC-2026-0002.html): `lru` crate vulnerability (fixed in upstream Pingora 0.7) + - `atty` unmaintained dependency removed (fixed in upstream Pingora 0.7) + - `protobuf` uncontrolled recursion bounded (fixed in upstream Pingora 0.7) + +### Removed +- `[patch.crates-io]` section with 16 git overrides pointing to `raskell-io/pingora` fork + +See the [blog post](/blog/pingora-0-7-upgrade/) for a detailed writeup. + +--- + +## 26.02_0 + +**Date:** 2026-01-29 +**Crate version:** 0.4.5 + +### Added +- **Supply chain security for release pipeline** + - SBOM generation in CycloneDX 1.5 and SPDX 2.3 formats via `cargo-sbom` + - Binary signing with Sigstore cosign (keyless, GitHub Actions OIDC) + - Container image signing with cosign and SBOM attestation via syft + - SLSA v1.0 provenance via `slsa-github-generator` (Build Level 3) + - Sigstore bundles (`.bundle`), SBOMs (`.cdx.json`, `.spdx.json`), and SLSA provenance (`.intoto.jsonl`) attached to every GitHub release + - Supply chain verification commands in release notes + +See [Supply Chain Security](/docs/operations/supply-chain/) for verification procedures. + +--- + +## 26.01_11 + +**Date:** 2026-01-29 +**Crate version:** 0.4.5 + +### Changed +- **Performance:** Reduce per-request allocations in hot path +- **Performance:** Avoid cloning header modification maps per request +- **Performance:** Optimize agent header map construction + +--- + +## 26.01_10 + +**Date:** 2026-01-27 +**Crate version:** 0.4.3 + +### Fixed +- Prevent single connection failure from permanently marking upstream target unhealthy +- Update code for rand 0.9 and hickory-resolver 0.25 API changes +- Use pingora fork to resolve remaining security vulnerabilities + +### Security +- Resolve dependabot security alerts + +### Changed +- **Dependency updates:** + - opentelemetry_sdk 0.27 → 0.31 + - opentelemetry-otlp 0.27 → 0.31 + - hickory-resolver 0.24 → 0.25 + - rand 0.8 → 0.9 + - wasmtime 40.0 → 41.0 + - notify 6.1 → 8.2 + - validator 0.18 → 0.20 + - nix 0.29 → 0.31 + - webpki-roots 0.26 → 1.0 + +--- + +## 26.01_9 + +**Date:** 2026-01-21 +**Crate version:** 0.4.2 + +### Added +- Sticky load balancing algorithm support in simulation framework + +### Changed +- Improved install script user experience + +--- + +## 26.01_8 + +**Date:** 2026-01-21 +**Crate version:** 0.4.1 + +### Changed +- **Dependency updates** with breaking change fixes: + - prost 0.13 → 0.14 (with tonic ecosystem upgrade to 0.14) + - tonic 0.12 → 0.14 (TLS features renamed: `tls` → `tls-ring`, `tls-roots` → `tls-native-roots`) + - tungstenite 0.24 → 0.28 (`Message::Text` now uses `Utf8Bytes`) + - sysinfo 0.31 → 0.37 (`RefreshKind::new()` → `RefreshKind::nothing()`) + - toml 0.8 → 0.9 + - brotli 7.0 → 8.0 + - directories 5.0 → 6.0 + - signal-hook 0.3 → 0.4 + - jsonschema 0.17 → 0.18 + - ip2location 0.5 → 0.6 + - tokio-tungstenite 0.24 → 0.28 +- GitHub Actions updates: checkout v6, github-script v8, docker/build-push-action v6 + +### Fixed +- WebSocket test compatibility with tungstenite 0.28 API changes +- System metrics collection with sysinfo 0.37 API changes + +--- + +## 26.01_7 + +**Date:** 2026-01-21 +**Crate version:** 0.4.0 + +### Added +- **DNS-01 ACME challenge support** for wildcard certificate issuance + - Modular DNS provider system with `DnsProvider` trait + - Hetzner DNS provider implementation + - Generic webhook provider for custom DNS integrations + - DNS propagation checking with configurable nameservers + - Secure credential loading from files or environment variables +- New configuration options for DNS-01 challenges: + - `challenge-type` option in ACME config (`http-01` or `dns-01`) + - `dns-provider` block with provider-specific settings + - `propagation` block for DNS propagation check tuning +- Integration tests for DNS providers using wiremock + +### Changed +- ACME scheduler now supports both HTTP-01 and DNS-01 renewal flows +- ACME client extended with `create_order_dns01()` method + +--- + +## 26.01_6 + +**Date:** 2026-01-14 +**Crate version:** 0.3.1 + +### Added +- Agent Protocol v2 with connection pooling and load balancing +- Reverse connection support for NAT traversal +- gRPC transport with bidirectional streaming +- Request cancellation support +- Prometheus metrics export for agent pools + +### Changed +- Improved agent health tracking with circuit breakers +- Better error messages for configuration validation + +### Fixed +- Connection leak in agent pool under high load +- Race condition in route matching cache + +--- + +## 26.01_4 + +**Date:** 2026-01-11 +**Crate version:** 0.3.0 + +### Added +- Initial Agent Protocol v2 implementation +- Binary UDS transport for lower latency +- Connection pooling with multiple strategies (RoundRobin, LeastConnections, HealthBased) +- WASM agent runtime using Wasmtime + +### Changed +- Agent protocol documentation reorganized into v1/ and v2/ + +--- + +## 26.01_3 + +**Date:** 2026-01-05 +**Crate version:** 0.2.3 + +See [GitHub Release](https://github.com/zentinelproxy/zentinel/releases/tag/26.01_3). + +--- + +## 26.01_0 + +**Date:** 2026-01-01 +**Crate version:** 0.2.0 + +First release using CalVer tagging. + +See [GitHub Release](https://github.com/zentinelproxy/zentinel/releases/tag/26.01_0). + +--- + +## 25.12 + +**Crate versions:** 0.1.0 -- 0.1.8 +**Releases:** 25.12_0 through 25.12_19 + +Initial public release series. Core proxy, routing, upstreams, agent system, observability, and KDL configuration. + +See [GitHub Releases](https://github.com/zentinelproxy/zentinel/releases?q=25.12) for individual release notes. + +--- + +## Links + +- [GitHub Releases](https://github.com/zentinelproxy/zentinel/releases) +- [Versioning](../versioning/) -- CalVer/SemVer scheme, LTS windows, version mapping +- [Supply Chain Security](/docs/operations/supply-chain/) -- Verify binary and container authenticity diff --git a/content/v/26.04/appendix/faq.md b/content/v/26.04/appendix/faq.md new file mode 100644 index 0000000..3cc502e --- /dev/null +++ b/content/v/26.04/appendix/faq.md @@ -0,0 +1,360 @@ ++++ +title = "FAQ" +weight = 2 +updated = 2026-02-19 ++++ + +Frequently asked questions about Zentinel. + +## General + +### What is Zentinel? + +Zentinel is a high-performance reverse proxy and load balancer built on [Pingora](https://github.com/cloudflare/pingora), Cloudflare's Rust-based proxy framework. It provides HTTP/1.1, HTTP/2, and HTTP/3 support with features like load balancing, health checks, TLS termination, and extensible request processing through agents. + +### How does Zentinel compare to nginx? + +| Aspect | Zentinel | nginx | +|--------|----------|-------| +| Language | Rust | C | +| Config format | KDL | nginx.conf | +| Hot reload | SIGHUP | `nginx -s reload` | +| Memory safety | Guaranteed | Manual | +| HTTP/3 | Native | Requires patch | +| Extensibility | Agents (external) | Modules (compiled) | + +Zentinel offers memory safety guarantees and a more modern configuration format, while nginx has a larger ecosystem and longer track record. + +### How does Zentinel compare to Envoy? + +Both are modern proxies with similar capabilities. Zentinel uses KDL configuration files while Envoy uses YAML/JSON and xDS APIs. Zentinel is lighter weight and simpler to configure for common use cases, while Envoy offers more extensive observability and service mesh features. + +### Is Zentinel production-ready? + +Zentinel is under active development. Check the [changelog](../changelog/) for the current version and stability status. For production deployments, thoroughly test your specific use case and monitor the GitHub repository for updates. + +## Configuration + +### Where should I put the configuration file? + +The default location is `/etc/zentinel/zentinel.kdl`. You can specify a different path with `--config`: + +```bash +zentinel --config /path/to/zentinel.kdl +``` + +Or set the `ZENTINEL_CONFIG` environment variable. + +### How do I validate my configuration? + +Use the `--test` flag to validate without starting the server: + +```bash +zentinel --test --config zentinel.kdl +``` + +Add `--verbose` for detailed validation output. + +### How do I reload configuration without downtime? + +Send a `SIGHUP` signal to the Zentinel process: + +```bash +kill -HUP $(cat /var/run/zentinel.pid) +# or +systemctl reload zentinel +``` + +Zentinel validates the new configuration before applying it. If validation fails, the old configuration remains active. + +### Can I use environment variables in configuration? + +Currently, environment variables are not interpolated in KDL configuration files. Use separate configuration files for different environments or generate configuration programmatically. + +## Networking + +### What ports does Zentinel use? + +By default: +- **8080** - HTTP traffic (configurable) +- **8443** - HTTPS traffic (configurable) +- **9090** - Admin/metrics endpoint (configurable) + +All ports are configurable in the `listeners` block. + +### How do I bind to port 80 or 443? + +Ports below 1024 require elevated privileges. Options: + +1. **Linux capabilities** (recommended): + ```bash + sudo setcap cap_net_bind_service=+ep /usr/local/bin/zentinel + ``` + +2. **iptables redirect**: + ```bash + iptables -t nat -A PREROUTING -p tcp --dport 80 -j REDIRECT --to-port 8080 + ``` + +3. **Run as root** (not recommended for production) + +### Does Zentinel support WebSockets? + +Yes. WebSocket connections are proxied transparently when using HTTP/1.1. Ensure your route doesn't have policies that buffer the request body. + +### Does Zentinel support gRPC? + +Yes. Zentinel can proxy gRPC traffic over HTTP/2. Configure your listener with `protocol "h2"` or `protocol "https"` and ensure the upstream supports HTTP/2. + +## TLS + +### What TLS versions are supported? + +Zentinel supports TLS 1.2 and TLS 1.3. Configure minimum version in the listener: + +```kdl +listeners { + listener "https" { + tls { + min-version "1.2" // or "1.3" + } + } +} +``` + +### How do I configure mTLS (client certificates)? + +Enable client authentication in the TLS block: + +```kdl +listeners { + listener "https" { + tls { + cert-file "/path/to/server.crt" + key-file "/path/to/server.key" + ca-file "/path/to/client-ca.crt" + client-auth #true + } + } +} +``` + +### How do I rotate certificates without downtime? + +Update the certificate files on disk and send `SIGHUP` to reload: + +```bash +cp new-cert.crt /etc/zentinel/certs/server.crt +cp new-key.key /etc/zentinel/certs/server.key +kill -HUP $(cat /var/run/zentinel.pid) +``` + +## Load Balancing + +### What load balancing algorithms are available? + +- `round_robin` - Sequential rotation (default) +- `least_connections` - Server with fewest active connections +- `random` - Random selection +- `ip_hash` - Consistent hashing by client IP +- `weighted` - Weighted random selection +- `consistent_hash` - Consistent hashing for cache affinity +- `power_of_two_choices` - Best of two random choices +- `adaptive` - Response-time based selection + +### How do I configure sticky sessions? + +Use `ip_hash` or `consistent_hash` load balancing: + +```kdl +upstreams { + upstream "backend" { + load-balancing "ip_hash" + targets { + target { address "10.0.1.1:8080" } + target { address "10.0.1.2:8080" } + } + } +} +``` + +### How do I drain a server for maintenance? + +Set its weight to 0 or remove it from the configuration and reload: + +```kdl +// Before +target { address "10.0.1.1:8080" weight=1 } + +// Draining +target { address "10.0.1.1:8080" weight=0 } +``` + +Existing connections will complete; new requests go to other servers. + +## Health Checks + +### Why is my upstream marked unhealthy? + +Check the admin endpoint for details: + +```bash +curl http://localhost:9090/admin/upstreams +``` + +Common causes: +- Health endpoint returning non-200 status +- Health check timeout too short +- Network/firewall blocking health checks +- Upstream server overloaded + +### How do I disable health checks? + +Remove the `health-check` block from the upstream configuration. Without health checks, Zentinel assumes all targets are healthy. + +### Can I use different health check paths for different servers? + +Health check configuration applies to all targets in an upstream. If servers need different health endpoints, create separate upstreams. + +## Performance + +### How many worker threads should I use? + +Set `worker-threads 0` (the default) to auto-detect based on CPU cores. For most workloads, one thread per core is optimal. Reduce threads if memory is constrained. + +### How do I increase connection limits? + +Adjust limits in the configuration: + +```kdl +system { + max-connections 50000 +} + +limits { + max-connections-per-client 200 + max-total-connections 50000 +} +``` + +Also increase OS limits: +```bash +ulimit -n 65535 +``` + +### Why am I seeing high latency? + +Common causes: +1. **Upstream slow** - Check upstream response times directly +2. **DNS resolution** - Use IP addresses or local DNS cache +3. **TLS handshakes** - Enable session resumption +4. **Connection establishment** - Increase connection pool size +5. **Request buffering** - Disable if not needed + +See [Troubleshooting](../../operations/troubleshooting/) for diagnostics. + +## Agents + +### What are agents used for? + +Agents handle request processing tasks that require external logic or state: +- **Authentication** - Validate tokens, check sessions +- **Rate limiting** - Distributed rate limiting with shared state +- **WAF** - Web application firewall inspection +- **Custom logic** - Any request/response transformation + +### How do agents communicate with Zentinel? + +Agents connect via: +- **Unix sockets** (recommended for local agents) +- **gRPC** (for remote or containerized agents) + +### What happens if an agent fails? + +Depends on the `failure-mode` setting: +- `closed` (default) - Reject the request (fail-safe) +- `open` - Allow the request to proceed (fail-open) + +Circuit breakers prevent repeated failures from overwhelming agents. + +## Troubleshooting + +### Why am I getting a redirect loop? + +This usually happens when your backend expects HTTPS but Zentinel connects with plaintext HTTP. The backend sees an HTTP request and redirects to HTTPS, creating an infinite loop. + +**Fix:** Add a `tls` block to your upstream when the backend serves HTTPS: + +```kdl +upstreams { + upstream "backend" { + targets { + target { address "api.example.com:443" } + } + tls { + sni "api.example.com" + } + } +} +``` + +Setting the target port to `443` is **not** enough. You must explicitly add the `tls` block to tell Zentinel to connect over TLS. Without it, Zentinel always uses plaintext HTTP. + +A redirect loop can also happen if the `Host` header doesn't match what the backend expects (e.g., `www.example.com` vs `example.com`). See [Troubleshooting: Redirect Loops](../../operations/troubleshooting/#redirect-loops) for all causes and solutions. + +### How do I enable debug logging? + +Set the `RUST_LOG` environment variable: + +```bash +RUST_LOG=debug zentinel --config zentinel.kdl + +# Module-specific debugging +RUST_LOG=zentinel::proxy=debug zentinel --config zentinel.kdl +``` + +### Where are the logs? + +| Deployment | Location | +|------------|----------| +| systemd | `journalctl -u zentinel` | +| Docker | `docker logs zentinel` | +| Kubernetes | `kubectl logs -l app=zentinel` | + +### How do I trace a specific request? + +Every request has a correlation ID in the `X-Correlation-Id` response header. Search logs by this ID: + +```bash +curl -i http://localhost:8080/api/endpoint +# Note the X-Correlation-Id header + +grep "abc123xyz" /var/log/zentinel/*.log +``` + +## Migration + +### How do I migrate from nginx? + +See the [Migration Guide](../../operations/migration/#from-nginx) for detailed configuration mapping and examples. + +### How do I migrate from HAProxy? + +See the [Migration Guide](../../operations/migration/#from-haproxy) for detailed configuration mapping and examples. + +### Can I run Zentinel alongside my existing proxy? + +Yes. Run Zentinel on a different port and gradually shift traffic: + +```bash +# Zentinel on 8080, nginx on 80 +# Test: curl http://localhost:8080/api/endpoint +# Compare: diff <(curl -s nginx/api) <(curl -s zentinel/api) +``` + +## See Also + +- [Glossary](../glossary/) - Term definitions +- [Troubleshooting](../../operations/troubleshooting/) - Diagnostic guides +- [Configuration](../../configuration/) - Full configuration reference + diff --git a/content/v/26.04/appendix/glossary.md b/content/v/26.04/appendix/glossary.md new file mode 100644 index 0000000..3fdb34e --- /dev/null +++ b/content/v/26.04/appendix/glossary.md @@ -0,0 +1,169 @@ ++++ +title = "Glossary" +weight = 3 +updated = 2026-02-19 ++++ + +Definitions of key terms used throughout Zentinel documentation. + +## A + +### Agent +An external process that handles request processing tasks like authentication, rate limiting, or WAF inspection. Agents communicate with Zentinel via Unix sockets or gRPC. + +### ALPN (Application-Layer Protocol Negotiation) +A TLS extension that allows the application layer to negotiate which protocol should be performed over a secure connection, enabling HTTP/2 or HTTP/3 negotiation. + +## B + +### Backend +See [Upstream](#upstream). + +### Backoff +A strategy for retrying failed requests with increasing delays between attempts. Zentinel supports exponential backoff with configurable base and maximum delays. + +## C + +### Circuit Breaker +A fault tolerance pattern that prevents cascading failures by temporarily stopping requests to an unhealthy upstream. States include closed (normal), open (blocking), and half-open (testing). + +### Connection Pool +A cache of reusable connections to upstream servers, reducing the overhead of establishing new connections for each request. + +### Correlation ID +A unique identifier assigned to each request for tracing through logs and distributed systems. Zentinel uses the `X-Correlation-Id` header. + +## D + +### Downstream +The client side of the proxy - the entity making requests to Zentinel. Opposite of [upstream](#upstream). + +## F + +### Failover +The process of automatically switching to a backup upstream server when the primary server fails. + +### Filter +A processing component that modifies requests or responses as they pass through Zentinel. Filters can add headers, transform bodies, or enforce policies. + +## G + +### Graceful Shutdown +A shutdown process that allows in-flight requests to complete before the server stops, preventing dropped connections. + +### gRPC +A high-performance RPC framework using HTTP/2 and Protocol Buffers. Zentinel can proxy gRPC traffic and use gRPC for agent communication. + +## H + +### Health Check +A periodic probe to determine if an upstream server is healthy and able to receive traffic. Supports HTTP, TCP, and gRPC protocols. + +### Hot Reload +The ability to reload configuration without restarting the server or dropping connections. Triggered by `SIGHUP` signal. + +## K + +### KDL (KDL Document Language) +The configuration file format used by Zentinel. A human-friendly, document-oriented configuration language. + +### Keepalive +A mechanism to maintain persistent connections between client and server, reducing connection establishment overhead. + +## L + +### Listener +A network endpoint where Zentinel accepts incoming connections. Defined by address, port, and protocol (HTTP, HTTPS, H2, H3). + +### Load Balancing +The distribution of incoming requests across multiple upstream servers. Algorithms include round robin, least connections, IP hash, and weighted random. + +## M + +### Middleware +See [Filter](#filter). + +### mTLS (Mutual TLS) +TLS authentication where both client and server present certificates to verify each other's identity. + +## O + +### OCSP Stapling +A method for checking certificate revocation status where the server periodically obtains a signed OCSP response and includes it in the TLS handshake. + +## P + +### Pingora +The Rust-based proxy framework developed by Cloudflare that powers Zentinel's core networking capabilities. + +### Policy +Configuration that defines how requests are processed for a specific route, including timeouts, rate limits, header modifications, and retry behavior. + +### Proxy +A server that acts as an intermediary between clients and backend servers. Zentinel is a reverse proxy. + +## Q + +### QUIC +A UDP-based transport protocol that provides the foundation for HTTP/3, offering reduced latency and improved connection migration. + +## R + +### Rate Limiting +Controlling the number of requests a client can make within a time period. Configurable per client IP, route, or globally. + +### Retry Policy +Configuration defining how Zentinel handles failed upstream requests, including maximum attempts, retryable status codes, and backoff strategy. + +### Reverse Proxy +A proxy server that sits in front of backend servers and forwards client requests to them. Zentinel is a reverse proxy. + +### Route +A rule that matches incoming requests based on criteria (path, host, headers) and directs them to an upstream or handler. + +### Round Robin +A load balancing algorithm that distributes requests sequentially across upstream servers in rotation. + +## S + +### Session Resumption +A TLS optimization that allows clients to resume previous sessions without a full handshake, reducing latency. + +### SNI (Server Name Indication) +A TLS extension that allows a client to indicate which hostname it's connecting to, enabling multiple TLS certificates on a single IP address. + +### Static Files +Files served directly from disk without backend processing. Zentinel can serve static content with caching and compression. + +## T + +### Target +An individual server within an upstream group, identified by address and optional weight. + +### TLS (Transport Layer Security) +A cryptographic protocol for secure communication. Zentinel supports TLS 1.2 and 1.3 for both client connections and upstream connections. + +### Tinyflake +A compact, time-ordered unique ID format used by Zentinel for correlation IDs. More compact than UUIDs. + +## U + +### Upstream +A backend server or group of servers that Zentinel forwards requests to. Also called "backend" in other proxy software. + +## W + +### WAF (Web Application Firewall) +A security layer that filters and monitors HTTP traffic to protect against common web exploits like SQL injection and XSS. + +### Weight +A value assigned to upstream targets that influences load balancing distribution. Higher weights receive proportionally more traffic. + +### Worker Thread +An OS thread dedicated to processing requests. Zentinel uses multiple worker threads for parallel request handling. + +## Numbers + +### 0-RTT (Zero Round Trip Time) +A TLS 1.3 feature allowing data to be sent on the first flight of the handshake, reducing latency for repeat connections. + diff --git a/content/v/26.04/appendix/license.md b/content/v/26.04/appendix/license.md new file mode 100644 index 0000000..f144ff5 --- /dev/null +++ b/content/v/26.04/appendix/license.md @@ -0,0 +1,100 @@ ++++ +title = "License" +weight = 4 +updated = 2026-02-19 ++++ + +Zentinel is licensed under the Apache License, Version 2.0. + +## Apache License + +Version 2.0, January 2004 + + + +### Terms and Conditions for Use, Reproduction, and Distribution + +#### 1. Definitions + +"License" shall mean the terms and conditions for use, reproduction, and distribution as defined by Sections 1 through 9 of this document. + +"Licensor" shall mean the copyright owner or entity authorized by the copyright owner that is granting the License. + +"Legal Entity" shall mean the union of the acting entity and all other entities that control, are controlled by, or are under common control with that entity. For the purposes of this definition, "control" means (i) the power, direct or indirect, to cause the direction or management of such entity, whether by contract or otherwise, or (ii) ownership of fifty percent (50%) or more of the outstanding shares, or (iii) beneficial ownership of such entity. + +"You" (or "Your") shall mean an individual or Legal Entity exercising permissions granted by this License. + +"Source" form shall mean the preferred form for making modifications, including but not limited to software source code, documentation source, and configuration files. + +"Object" form shall mean any form resulting from mechanical transformation or translation of a Source form, including but not limited to compiled object code, generated documentation, and conversions to other media types. + +"Work" shall mean the work of authorship, whether in Source or Object form, made available under the License, as indicated by a copyright notice that is included in or attached to the work (an example is provided in the Appendix below). + +"Derivative Works" shall mean any work, whether in Source or Object form, that is based on (or derived from) the Work and for which the editorial revisions, annotations, elaborations, or other modifications represent, as a whole, an original work of authorship. For the purposes of this License, Derivative Works shall not include works that remain separable from, or merely link (or bind by name) to the interfaces of, the Work and Derivative Works thereof. + +"Contribution" shall mean any work of authorship, including the original version of the Work and any modifications or additions to that Work or Derivative Works thereof, that is intentionally submitted to the Licensor for inclusion in the Work by the copyright owner or by an individual or Legal Entity authorized to submit on behalf of the copyright owner. For the purposes of this definition, "submitted" means any form of electronic, verbal, or written communication sent to the Licensor or its representatives, including but not limited to communication on electronic mailing lists, source code control systems, and issue tracking systems that are managed by, or on behalf of, the Licensor for the purpose of discussing and improving the Work, but excluding communication that is conspicuously marked or otherwise designated in writing by the copyright owner as "Not a Contribution." + +"Contributor" shall mean Licensor and any individual or Legal Entity on behalf of whom a Contribution has been received by Licensor and subsequently incorporated within the Work. + +#### 2. Grant of Copyright License + +Subject to the terms and conditions of this License, each Contributor hereby grants to You a perpetual, worldwide, non-exclusive, no-charge, royalty-free, irrevocable copyright license to reproduce, prepare Derivative Works of, publicly display, publicly perform, sublicense, and distribute the Work and such Derivative Works in Source or Object form. + +#### 3. Grant of Patent License + +Subject to the terms and conditions of this License, each Contributor hereby grants to You a perpetual, worldwide, non-exclusive, no-charge, royalty-free, irrevocable (except as stated in this section) patent license to make, have made, use, offer to sell, sell, import, and otherwise transfer the Work, where such license applies only to those patent claims licensable by such Contributor that are necessarily infringed by their Contribution(s) alone or by combination of their Contribution(s) with the Work to which such Contribution(s) was submitted. If You institute patent litigation against any entity (including a cross-claim or counterclaim in a lawsuit) alleging that the Work or a Contribution incorporated within the Work constitutes direct or contributory patent infringement, then any patent licenses granted to You under this License for that Work shall terminate as of the date such litigation is filed. + +#### 4. Redistribution + +You may reproduce and distribute copies of the Work or Derivative Works thereof in any medium, with or without modifications, and in Source or Object form, provided that You meet the following conditions: + +1. You must give any other recipients of the Work or Derivative Works a copy of this License; and + +2. You must cause any modified files to carry prominent notices stating that You changed the files; and + +3. You must retain, in the Source form of any Derivative Works that You distribute, all copyright, patent, trademark, and attribution notices from the Source form of the Work, excluding those notices that do not pertain to any part of the Derivative Works; and + +4. If the Work includes a "NOTICE" text file as part of its distribution, then any Derivative Works that You distribute must include a readable copy of the attribution notices contained within such NOTICE file, excluding those notices that do not pertain to any part of the Derivative Works, in at least one of the following places: within a NOTICE text file distributed as part of the Derivative Works; within the Source form or documentation, if provided along with the Derivative Works; or, within a display generated by the Derivative Works, if and wherever such third-party notices normally appear. The contents of the NOTICE file are for informational purposes only and do not modify the License. You may add Your own attribution notices within Derivative Works that You distribute, alongside or as an addendum to the NOTICE text from the Work, provided that such additional attribution notices cannot be construed as modifying the License. + +You may add Your own copyright statement to Your modifications and may provide additional or different license terms and conditions for use, reproduction, or distribution of Your modifications, or for any such Derivative Works as a whole, provided Your use, reproduction, and distribution of the Work otherwise complies with the conditions stated in this License. + +#### 5. Submission of Contributions + +Unless You explicitly state otherwise, any Contribution intentionally submitted for inclusion in the Work by You to the Licensor shall be under the terms and conditions of this License, without any additional terms or conditions. Notwithstanding the above, nothing herein shall supersede or modify the terms of any separate license agreement you may have executed with Licensor regarding such Contributions. + +#### 6. Trademarks + +This License does not grant permission to use the trade names, trademarks, service marks, or product names of the Licensor, except as required for reasonable and customary use in describing the origin of the Work and reproducing the content of the NOTICE file. + +#### 7. Disclaimer of Warranty + +Unless required by applicable law or agreed to in writing, Licensor provides the Work (and each Contributor provides its Contributions) on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied, including, without limitation, any warranties or conditions of TITLE, NON-INFRINGEMENT, MERCHANTABILITY, or FITNESS FOR A PARTICULAR PURPOSE. You are solely responsible for determining the appropriateness of using or redistributing the Work and assume any risks associated with Your exercise of permissions under this License. + +#### 8. Limitation of Liability + +In no event and under no legal theory, whether in tort (including negligence), contract, or otherwise, unless required by applicable law (such as deliberate and grossly negligent acts) or agreed to in writing, shall any Contributor be liable to You for damages, including any direct, indirect, special, incidental, or consequential damages of any character arising as a result of this License or out of the use or inability to use the Work (including but not limited to damages for loss of goodwill, work stoppage, computer failure or malfunction, or any and all other commercial damages or losses), even if such Contributor has been advised of the possibility of such damages. + +#### 9. Accepting Warranty or Additional Liability + +While redistributing the Work or Derivative Works thereof, You may choose to offer, and charge a fee for, acceptance of support, warranty, indemnity, or other liability obligations and/or rights consistent with this License. However, in accepting such obligations, You may act only on Your own behalf and on Your sole responsibility, not on behalf of any other Contributor, and only if You agree to indemnify, defend, and hold each Contributor harmless for any liability incurred by, or claims asserted against, such Contributor by reason of your accepting any such warranty or additional liability. + +--- + +## Third-Party Licenses + +Zentinel uses the following open source components: + +### Pingora + +Zentinel is built on [Pingora](https://github.com/cloudflare/pingora), licensed under the Apache License 2.0. + +Copyright 2024 Cloudflare, Inc. + +### Other Dependencies + +For a complete list of dependencies and their licenses, see the `Cargo.lock` file in the Zentinel repository or run: + +```bash +cargo license +``` + diff --git a/content/v/26.04/appendix/versioning.md b/content/v/26.04/appendix/versioning.md new file mode 100644 index 0000000..8651ccc --- /dev/null +++ b/content/v/26.04/appendix/versioning.md @@ -0,0 +1,339 @@ ++++ +title = "Versioning" +weight = 2 +updated = 2026-03-01 ++++ + +How Zentinel versions work, mapping between release and crate versions, and changelogs. + +## Dual Versioning Scheme + +Zentinel uses two versioning systems for different audiences: + +| System | Format | Example | Audience | Used For | +|--------|--------|---------|----------|----------| +| **Release Version** | CalVer (`YY.MM_PATCH`) | `26.01_0` | Operators, enterprise, docs | Downloads, release tags, LTS windows, support contracts | +| **Crate Version** | SemVer (`MAJOR.MINOR.PATCH`) | `0.3.0` | Library consumers | Cargo.toml, crates.io, dependency management | + +**CalVer is the primary version.** When you deploy Zentinel, report issues, reference documentation, or verify supply chain artifacts, use the CalVer release version. SemVer exists solely for Rust's package ecosystem. + +### Release Version (CalVer) + +All public-facing releases use [Calendar Versioning](https://calver.org/) in `YY.MM_PATCH` format: + +``` +YY.MM_PATCH + +26.01_0 - January 2026, first release +26.01_1 - January 2026, first patch +26.02_0 - February 2026, first release +``` + +- **`YY.MM`** identifies the release series (e.g., `26.01` = January 2026) +- **`_PATCH`** increments for bug fixes and security patches within a series + +This provides: +- **Age at a glance** — `25.06_3` tells you the release is from June 2025 +- **LTS windows tied to the calendar** — an LTS branch like `26.01 LTS` receives security backports for 12 months, through January 2027 +- **Upgrade urgency** — if you're running `25.06_3` and the current release is `26.01_0`, you're 7 months behind + +### Crate Version (SemVer) + +Rust crates published to crates.io use [Semantic Versioning](https://semver.org/). This is an implementation detail for library consumers using Zentinel crates as dependencies. Operators do not need to track SemVer. + +``` +MAJOR.MINOR.PATCH + +0.1.0 - Initial development +0.2.0 - New features (pre-1.0, may have breaking changes) +1.0.0 - First stable release +1.1.0 - New features, backwards compatible +1.1.1 - Bug fixes only +2.0.0 - Breaking changes +``` + +### Which Version Do I Use? + +| Context | Use | +|---------|-----| +| Downloading binaries | CalVer (`26.01_0`) | +| Docker image tags | CalVer (`ghcr.io/zentinelproxy/zentinel:26.01_0`) | +| Filing issues / support tickets | CalVer | +| Verifying supply chain signatures | CalVer (matches release tag) | +| LTS / support contracts | CalVer series (`26.01 LTS`) | +| Cargo.toml dependencies | SemVer (`zentinel = "0.3"`) | +| crates.io | SemVer | + +## Version Mapping + +This table maps CalVer release versions to their corresponding crate versions: + +| Release (CalVer) | Crate Version (SemVer) | Protocol | Release Date | Status | +|---------|---------------|----------|--------------|--------| +| **26.03_1** | `0.5.12` | `0.2.0` | 2026-03-01 | Current | +| **26.02_5** | `0.5.11` | `0.2.0` | 2026-02-27 | Previous | +| **26.02_4** | `0.4.10` | `0.2.0` | 2026-02-04 | Previous | +| **26.01_0** | `0.2.0` | `0.2.0` | 2026-01-01 | Previous | +| **25.12_0** | `0.1.0` | `0.1.0` | 2025-12-15 | Archive | +| — | `0.1.0` | `0.1.0` | 2025-11-01 | Internal | + +### Finding Your Version + +**From the binary:** + +```bash +zentinel --version +# zentinel 26.03_1 (0.5.12) +``` + +The CalVer release version is shown first, with the crate SemVer in parentheses. + +**From Docker:** + +```bash +docker inspect ghcr.io/zentinelproxy/zentinel:26.03_1 --format '{{ index .Config.Labels "org.opencontainers.image.version" }}' +# 26.03_1 +``` + +**From the documentation URL:** + +- `/docs/` — Current release (26.03) +- `/docs/v/26.02/` — Previous release +- `/docs/v/25.12/` — Archive + +--- + +## Changelogs + +For the full changelog with all patch releases, see [Changelog](../changelog/). + +### Release 26.03 + +**Crate version:** `0.5.12` +**Release date:** March 2026 + +#### Changed + +- Image optimization agent v0.2.0 with Content-Type fix and improved defaults + +--- + +### Release 26.02 + +**Crate version:** `0.5.11` +**Release date:** January -- February 2026 + +#### Added + +- **`include` directive in single-file config** — `include "routes/*.kdl"` works directly in config files loaded via `zentinel --config`, with glob patterns, recursive expansion, and circular include detection +- **Supply chain security for release pipeline** + - SBOM generation in CycloneDX 1.5 and SPDX 2.3 formats + - Binary signing with Sigstore cosign (keyless, GitHub Actions OIDC) + - Container image signing with cosign and SBOM attestation + - SLSA v1.0 provenance (Build Level 3) +- Per-request allocation reduction in hot path +- DNS-01 ACME challenge support for wildcard certificates +- Agent Protocol v2 connection pooling, load balancing, reverse connections +- Sticky load balancing algorithm +- CI workflow (formatting, clippy, tests, docs checks) +- Weekly `cargo audit` workflow with automatic issue creation +- First-time user smoke tests (WAF, Lua) +- `protocol-version` KDL config for agent blocks + +#### Changed + +- Major dependency updates (opentelemetry, prost, tonic, tungstenite, sysinfo, reqwest, jsonschema, and more) +- Pingora switched to `raskell-io/pingora` fork to remove protobuf vulnerability (RUSTSEC-2024-0437) + +#### Fixed + +- Prevent single connection failure from permanently marking upstream target unhealthy +- 16 rustdoc warnings across 10 files +- Example configs now pass `zentinel test` validation + +--- + +### Release 26.01 + +**Crate version:** `0.2.0` -- `0.3.0` +**Release date:** January 2026 + +#### Added + +- **Traffic Mirroring / Shadow Traffic** + - Fire-and-forget async request duplication to shadow upstreams + - Percentage-based sampling (0-100%) for controlled traffic mirroring + +- **API Schema Validation** + - JSON Schema validation for API routes (requests and responses) + - OpenAPI 3.0 and Swagger 2.0 specification support + +- WebSocket frame inspection support in agent protocol +- Graceful shutdown improvements +- Connection draining during rolling updates + +#### Changed + +- Improved upstream health check reliability +- Reduced memory usage for idle connections + +#### Security + +- Removed archived agents with unsafe FFI code (Lua, WAF, auth, denylist, ratelimit) +- Replaced `unreachable!()` panics with proper error handling in agent-protocol + +--- + +### Release 25.12 + +**Crate version:** `0.2.0` +**Protocol version:** `0.1.0` +**Release date:** December 2025 + +#### Added + +- **Core Proxy** + - HTTP/1.1 and HTTP/2 support + - HTTPS with TLS 1.2/1.3 + - Configurable listeners (multiple ports, protocols) + - Request/response header manipulation + +- **Routing** + - Path-based routing with prefix, exact, and regex matching + - Host-based virtual hosting + - Method-based routing + - Header-based routing conditions + +- **Upstreams** + - Multiple backend targets with load balancing + - Round-robin and random load balancing strategies + - Active health checks (HTTP, TCP) + - Passive health monitoring with circuit breaker + - Connection pooling + +- **Agent System** + - Unix socket transport for local agents + - gRPC transport for remote agents + - Request/response lifecycle hooks + - WebSocket frame inspection hooks + - Fail-open mode for agent failures + - Agent timeout configuration + +- **Observability** + - Prometheus metrics endpoint + - Structured JSON logging + - Request tracing with correlation IDs + - OpenTelemetry integration + +- **Configuration** + - KDL configuration format + - Environment variable substitution + - Configuration validation + - Hot reload via SIGHUP + +--- + +## Upgrade Guides + +### From 26.01 to 26.02 + +No breaking changes. Direct upgrade supported. + +```bash +# Stop current version +systemctl stop zentinel + +# Install new version +VERSION="26.03_1" +curl -Lo /usr/local/bin/zentinel \ + "https://github.com/zentinelproxy/zentinel/releases/download/${VERSION}/zentinel-${VERSION}-linux-amd64.tar.gz" +tar -xzf "zentinel-${VERSION}-linux-amd64.tar.gz" +chmod +x zentinel && sudo mv zentinel /usr/local/bin/ + +# Validate configuration +zentinel validate -c /etc/zentinel/zentinel.kdl + +# Start new version +systemctl start zentinel +``` + +**New features to consider:** + +- [Supply Chain Security](/docs/operations/supply-chain/) -- verify binary and container authenticity +- DNS-01 ACME challenges for wildcard certificates +- Agent Protocol v2 connection pooling + +### From 25.12 to 26.01 + +No breaking changes. Direct upgrade supported. + +**New features to consider:** + +- [Traffic Mirroring](/configuration/routes/#shadow) for canary deployments +- [Schema Validation](/configuration/routes/#schema-validation) for API routes + +--- + +## Compatibility Matrix + +### Agent Compatibility + +| Zentinel Release | Protocol | Compatible Agent Versions | +|------------------|----------|---------------------------| +| 26.03 | `0.2.0` | Agents built with protocol `0.2.x` | +| 26.02 | `0.2.0` | Agents built with protocol `0.2.x` | +| 26.01 | `0.2.0` | Agents built with protocol `0.2.x` | +| 25.12 | `0.1.0` | Agents built with protocol `0.1.x` | + +### Rust Toolchain + +| Zentinel Release | Minimum Rust Version | Recommended | +|------------------|----------------------|-------------| +| 26.03 | 1.85.0 | 1.92.0+ | +| 26.02 | 1.85.0 | 1.85.0+ | +| 26.01 | 1.75.0 | 1.83.0+ | +| 25.12 | 1.70.0 | 1.75.0+ | + +--- + +## Release Schedule + +Zentinel follows a monthly release cadence: + +- **Feature releases:** First week of each month (e.g., `26.02_0`) +- **Patch releases:** As needed for security or critical bugs (e.g., `26.01_1`, `26.01_2`) + +### Community Support + +| Release | Status | Security Fixes Until | +|---------|--------|----------------------| +| 26.03 | Current | Active development | +| 26.02 | Previous | 26.06 (3 months) | +| 26.01 | Previous | 26.04 (3 months) | +| 25.12 | EOL | No support | + +Community releases receive security patches for **3 months** after the next release series ships. + +### Enterprise LTS + +Enterprise customers receive long-term support branches designated by their CalVer series: + +| LTS Branch | Based On | Security Fixes Until | Config Stability | +|------------|----------|----------------------|------------------| +| 26.01 LTS | `26.01_0` | January 2027 (12 months) | Guaranteed | + +LTS branches receive: +- **Security backports** for 12 months from the initial release +- **Configuration compatibility** — no breaking config changes within the LTS window +- **Patch releases** on the same CalVer series (e.g., `26.01_1`, `26.01_2`, ...) +- **Early security advisories** before public disclosure + +LTS is available through the [Enterprise Builds](/support/) offering. See [Supply Chain Security](/docs/operations/supply-chain/) for verification procedures. + +--- + +## See Also + +- [Release Process](/development/releases/) — How releases are made +- [GitHub Releases](https://github.com/zentinelproxy/zentinel/releases) — Download binaries +- [crates.io](https://crates.io/crates/zentinel) — Rust crate registry diff --git a/content/v/26.04/concepts/_index.md b/content/v/26.04/concepts/_index.md new file mode 100644 index 0000000..b2a5b77 --- /dev/null +++ b/content/v/26.04/concepts/_index.md @@ -0,0 +1,49 @@ ++++ +title = "Core Concepts" +weight = 2 +sort_by = "weight" +template = "section.html" ++++ + +Understanding Zentinel's architecture and design principles. + +## Overview + +Zentinel is a high-performance reverse proxy built on [Cloudflare's Pingora](https://github.com/cloudflare/pingora) framework. It provides a flexible agent-based architecture for implementing security controls, traffic management, and custom request processing. + +## Key Concepts + +| Concept | Description | +|---------|-------------| +| **Proxy** | The core Zentinel process that handles incoming requests | +| **Listener** | A network endpoint (IP:port) that accepts connections | +| **Route** | Rules that match requests and direct them to upstreams | +| **Upstream** | A group of backend servers that handle requests | +| **Agent** | An external process that inspects/modifies requests | + +## Architecture Principles + +1. **Performance First** - Built on Pingora for minimal latency overhead +2. **Agent Isolation** - Security logic runs in separate processes +3. **Fail-Safe Defaults** - Configurable fail-open behavior for resilience +4. **Observable** - Built-in metrics, logging, and tracing + +## In This Section + +| Page | Description | +|------|-------------| +| [Architecture](architecture/) | System design and component interaction | +| [Components](components/) | Detailed breakdown of each component | +| [Pingora Foundation](pingora/) | Understanding the Pingora framework | +| [Request Flow](request-flow/) | How requests traverse the proxy | +| [Routing](routing/) | Request matching and forwarding rules | +| [Comparison](comparison/) | How Zentinel compares to Envoy, HAProxy, and Nginx | + +## Recommended Reading Order + +1. Start with [Architecture](architecture/) for the big picture +2. Read [Components](components/) to understand each part +3. Review [Request Flow](request-flow/) to see how they work together +4. Dive into [Routing](routing/) for traffic management details +5. See [Comparison](comparison/) to understand trade-offs with alternatives + diff --git a/content/v/26.04/concepts/architecture.md b/content/v/26.04/concepts/architecture.md new file mode 100644 index 0000000..a07ac3b --- /dev/null +++ b/content/v/26.04/concepts/architecture.md @@ -0,0 +1,303 @@ ++++ +title = "Architecture Overview" +weight = 1 +updated = 2026-02-19 ++++ + +Zentinel is a security-first reverse proxy built on [Cloudflare's Pingora](https://github.com/cloudflare/pingora) framework. This page explains the high-level architecture and design philosophy. + +## Design Philosophy + +Zentinel follows four core principles: + +### Sleepable Operations + +No operational surprises at 3 AM: + +- **Bounded resources** - Hard limits on memory, queues, connections +- **Deterministic timeouts** - Every operation has an explicit timeout +- **Graceful degradation** - Clear failure modes (fail-open/fail-closed) +- **Hot reload** - Configuration changes without restarts + +### Security-First + +Security decisions are explicit, not magical: + +- **No hidden behavior** - All limits and policies are in configuration +- **Isolation by default** - Complex logic runs in external agents +- **Observable decisions** - Every security action is logged and traceable + +### Minimal Dataplane + +The proxy core stays boring and predictable: + +- **Small surface area** - Core proxy does routing, load balancing, forwarding +- **Stable behavior** - No surprises under load or failure +- **Innovation at the edges** - Advanced features live in agents + +### Production Correctness + +Features ship only when they're production-ready: + +- **Bounded and observable** - Every feature has limits and metrics +- **Testable** - Load tests, soak tests, regression gates +- **Rollback-safe** - Safe deployment and quick recovery + +## High-Level Architecture + +``` + ┌─────────────────────┐ + │ External Agents │ + │ ┌───┐ ┌───┐ ┌───┐ │ + │ │WAF│ │Auth│ │...│ │ + │ └─┬─┘ └─┬─┘ └─┬─┘ │ + └────┼─────┼─────┼───┘ + │ UDS │ │ +┌──────────┐ ┌──────────────────────┼─────┼─────┼────────────────┐ +│ │ │ │ │ │ │ +│ Client │────▶│ Zentinel Proxy ▼ ▼ ▼ │ +│ │ │ ┌─────────────────────────────────────────────┐ │ +└──────────┘ │ │ Agent Manager │ │ + │ └─────────────────────────────────────────────┘ │ + │ │ │ + │ ┌──────────┐ ┌─────┴─────┐ ┌────────────┐ │ + │ │ Route │───▶│ Request │───▶│ Upstream │ │ + │ │ Matcher │ │ Handler │ │ Pool │ │ + │ └──────────┘ └───────────┘ └─────┬──────┘ │ + │ │ │ + └──────────────────────────────────────────┼───────┘ + │ + ▼ + ┌────────────────┐ + │ Upstream │ + │ Servers │ + └────────────────┘ +``` + +## Core Components + +### 1. Proxy Dataplane (Pingora) + +The foundation is Cloudflare's Pingora library, providing: + +- High-performance async HTTP handling +- Connection pooling to upstreams +- TLS termination +- HTTP/1.1 and HTTP/2 support +- Zero-copy buffer management + +Zentinel extends Pingora with routing, load balancing, and agent coordination. + +### 2. Route Matcher + +Matches incoming requests to configured routes based on: + +- Path (exact, prefix, regex) +- Host header +- HTTP method +- Request headers +- Query parameters + +Routes are compiled at startup and cached for efficient matching. + +### 3. Upstream Pool + +Manages backend server connections: + +- Multiple load balancing algorithms +- Active and passive health checking +- Circuit breakers for failure isolation +- Connection pooling and reuse + +### 4. Agent Manager + +Coordinates external agent processes: + +- Connection pooling per agent +- Timeout enforcement +- Circuit breakers for agent failures +- Decision aggregation from multiple agents + +### 5. Configuration System + +Declarative configuration with: + +- KDL format (human-friendly) +- Schema validation +- Hot reload without downtime +- Multi-file support + +## Request Flow + +``` +1. Client Connection + └─▶ Pingora accepts TCP connection + └─▶ TLS handshake (if HTTPS) + +2. Request Received + └─▶ Parse HTTP request + └─▶ Generate trace ID + +3. Route Matching + └─▶ Match against compiled routes + └─▶ Select highest priority match + +4. Agent Processing (if configured) + └─▶ Send request to agents + └─▶ Collect decisions (allow/block/redirect) + └─▶ Apply header mutations + +5. Request Handling + ├─▶ Static: Serve file from disk + ├─▶ Builtin: Health check, metrics + └─▶ Proxy: Forward to upstream + +6. Upstream Selection + └─▶ Load balancer selects target + └─▶ Health check filters unhealthy + └─▶ Circuit breaker filters failing + +7. Upstream Request + └─▶ Connect to upstream (pooled) + └─▶ Send request with modified headers + └─▶ Retry on failure (if configured) + +8. Response Processing + └─▶ Add security headers + └─▶ Agent response processing (optional) + └─▶ Stream to client + +9. Logging + └─▶ Access log entry + └─▶ Metrics update + └─▶ Audit log (if security event) +``` + +## Extension Model + +Zentinel's extension model keeps complexity out of the dataplane: + +``` +┌────────────────────────────────────────────────────────────────┐ +│ Dataplane (Rust) │ +│ Fast, bounded, predictable. Handles 99% of requests quickly. │ +└────────────────────────────────────────────────────────────────┘ + │ + Unix Domain Sockets + or gRPC + │ +┌──────────┐ ┌──────────┐ ┌──────────┐ ┌──────────┐ +│ WAF │ │ Auth │ │ Rate │ │ Custom │ +│ Agent │ │ Agent │ │ Limit │ │ Logic │ +│ │ │ │ │ Agent │ │ Agent │ +│ (CRS) │ │ (JWT) │ │ (Redis) │ │ (Lua) │ +└──────────┘ └──────────┘ └──────────┘ └──────────┘ + Any language, independent deployment, isolated failures +``` + +**Why external agents?** + +| Concern | Dataplane | Agents | +|---------|-----------|--------| +| Crash isolation | Must not crash | Can crash safely | +| Deployment | Full restart | Independent update | +| Language | Rust only | Any language | +| Complexity | Minimal | Unlimited | +| Resources | Shared, bounded | Isolated | + +## Failure Handling + +### Circuit Breakers + +Protect against cascading failures: + +``` + Closed Open + ┌─────────┐ failures ┌─────────┐ + │ Normal │─────────────▶│ Failing │ + │ traffic │ exceed │ fast- │ + └─────────┘ threshold │ fail │ + ▲ └────┬────┘ + │ │ + │ Half-Open │ timeout + │ ┌─────────┐ │ + └───│ Testing │◀─────────┘ + │ traffic │ + └─────────┘ +``` + +### Failure Modes + +Per-route configuration: + +- **fail-closed** (default): Block request if agent fails +- **fail-open**: Allow request if agent fails + +### Health Checking + +- **Active**: Periodic HTTP/TCP probes +- **Passive**: Learn from real traffic failures +- **Recovery**: Gradual reintroduction after failures + +## Observability + +### Metrics (Prometheus) + +- Request latency histograms (per route) +- Status code counters +- Upstream health status +- Agent latency and timeouts +- Circuit breaker state + +### Logging (Structured JSON) + +- **Access logs**: Request/response metadata +- **Error logs**: Failures with stack traces +- **Audit logs**: Security decisions + +### Tracing + +- Correlation IDs on all requests +- Distributed tracing support (OpenTelemetry) + +## Configuration Lifecycle + +``` +┌──────────────┐ +│ Config File │ +│ (zentinel. │ +│ kdl) │ +└──────┬───────┘ + │ + ▼ +┌──────────────┐ ┌──────────────┐ +│ Parse │────▶│ Validate │ +│ (KDL) │ │ (Schema) │ +└──────────────┘ └──────┬───────┘ + │ + ┌────────────────────┴────────────────────┐ + │ │ + ▼ ▼ +┌──────────────┐ ┌──────────────┐ +│ Invalid │ │ Valid │ +│ (error + │ │ (apply) │ +│ keep old) │ └──────┬───────┘ +└──────────────┘ │ + ▼ + ┌──────────────┐ + │ Atomic Swap │ + │ (ArcSwap) │ + └──────┬───────┘ + │ + ▼ + ┌──────────────┐ + │ Drain Old │ + │ (60s grace) │ + └──────────────┘ +``` + +## Next Steps + +- [Component Design](../components/) - Deep dive into each component +- [Request Flow](../request-flow/) - Detailed request lifecycle +- [Pingora Foundation](../pingora/) - How Zentinel uses Pingora diff --git a/content/v/26.04/concepts/comparison.md b/content/v/26.04/concepts/comparison.md new file mode 100644 index 0000000..164c3a4 --- /dev/null +++ b/content/v/26.04/concepts/comparison.md @@ -0,0 +1,832 @@ ++++ +title = "Comparison with Alternatives" +weight = 6 +updated = 2026-02-19 ++++ + +How Zentinel compares to other popular reverse proxies and load balancers. + +## Overview + +Zentinel occupies a unique position in the reverse proxy landscape. Rather than competing directly with established proxies on feature breadth, it focuses on security-first design, operational predictability, and an extensible agent architecture. + +| Feature | Zentinel | Envoy | HAProxy | Nginx | Traefik | Caddy | +|---------|----------|-------|---------|-------|---------|-------| +| **Language** | Rust | C++ | C | C | Go | Go | +| **Memory Safety** | Yes | No | No | No | Yes | Yes | +| **Configuration** | KDL | YAML/xDS | Config file | Config file | YAML/Labels | Caddyfile/JSON | +| **Hot Reload** | Yes | Yes (xDS) | Yes | Yes (SIGHUP) | Yes (auto) | Yes (API) | +| **Extension Model** | External agents | Filters (C++/Wasm) | Lua/SPOE | Modules/Lua | Plugins (Go) | Modules (Go) | +| **Auto HTTPS** | Planned | No | No | No | Yes | Yes | +| **Primary Use Case** | Security gateway | Service mesh | Load balancing | Web server/proxy | Cloud-native edge | Simple web server | + +## Zentinel vs Envoy + +### Architecture Philosophy + +**Envoy** is designed as a universal data plane for service mesh architectures. It provides extensive protocol support, advanced traffic management, and deep observability through a filter chain architecture. + +**Zentinel** is designed as a security-focused edge proxy with an external agent model. Rather than embedding security logic in filters, agents run as isolated processes that can be updated, rate-limited, or disabled independently. + +### When to Choose Envoy + +- Building a service mesh with Istio, Consul, or similar +- Need extensive protocol support (gRPC, MongoDB, Redis, etc.) +- Require xDS-based dynamic configuration from a control plane +- Want a mature, battle-tested proxy at massive scale + +### When to Choose Zentinel + +- Need a security gateway with WAF, auth, and rate limiting +- Want isolated security agents that can fail independently +- Prefer explicit configuration over dynamic control planes +- Value memory safety and predictable resource usage +- Building custom security controls with the agent protocol + +### Configuration Comparison + +**Envoy** (YAML): +```yaml +static_resources: + listeners: + - name: listener_0 + address: + socket_address: + address: 0.0.0.0 + port_value: 8080 + filter_chains: + - filters: + - name: envoy.filters.network.http_connection_manager + typed_config: + "@type": type.googleapis.com/envoy.extensions.filters.network.http_connection_manager.v3.HttpConnectionManager + stat_prefix: ingress_http + route_config: + name: local_route + virtual_hosts: + - name: backend + domains: ["*"] + routes: + - match: + prefix: "/" + route: + cluster: backend_cluster + clusters: + - name: backend_cluster + type: STRICT_DNS + load_assignment: + cluster_name: backend_cluster + endpoints: + - lb_endpoints: + - endpoint: + address: + socket_address: + address: backend + port_value: 3000 +``` + +**Zentinel** (KDL): +```kdl +listeners { + listener "http" { + address "0.0.0.0:8080" + protocol "http" + } +} + +routes { + route "default" { + matches { + path-prefix "/" + } + upstream "backend" + } +} + +upstreams { + upstream "backend" { + targets { + target { address "backend:3000" } + } + } +} +``` + +### Extension Model + +**Envoy filters** are compiled into the binary (C++) or loaded as Wasm modules. They run in-process and have access to the full request/response lifecycle. + +**Zentinel agents** are external processes that communicate via Unix sockets or gRPC. This provides: +- Process isolation (agent crash doesn't crash proxy) +- Independent deployment and updates +- Language flexibility (any language that speaks the protocol) +- Resource limits per agent + +## Zentinel vs HAProxy + +### Architecture Philosophy + +**HAProxy** is the gold standard for high-performance TCP/HTTP load balancing. It's known for reliability, performance, and a powerful ACL system for traffic management. + +**Zentinel** shares HAProxy's focus on reliability but adds a security-first architecture with external agents for policy enforcement. + +### When to Choose HAProxy + +- Pure load balancing with extreme performance requirements +- Need advanced health checking and connection management +- TCP-level proxying (databases, message queues) +- Established operational expertise with HAProxy + +### When to Choose Zentinel + +- Security controls are a primary requirement +- Want to implement custom policies without Lua +- Need process isolation for security components +- Prefer Rust's memory safety guarantees + +### Configuration Comparison + +**HAProxy**: +``` +frontend http_front + bind *:8080 + default_backend http_back + +backend http_back + balance roundrobin + server backend1 127.0.0.1:3000 check + server backend2 127.0.0.1:3001 check +``` + +**Zentinel** (KDL): +```kdl +listeners { + listener "http" { + address "0.0.0.0:8080" + protocol "http" + } +} + +routes { + route "default" { + matches { path-prefix "/" } + upstream "backend" + } +} + +upstreams { + upstream "backend" { + targets { + target { address "127.0.0.1:3000" } + target { address "127.0.0.1:3001" } + } + load-balancing "round_robin" + health-check { + path "/health" + interval-secs 10 + } + } +} +``` + +### Extension Comparison + +| Aspect | HAProxy | Zentinel | +|--------|---------|----------| +| Scripting | Lua (embedded) | External agents | +| External calls | SPOE protocol | Agent protocol | +| Isolation | In-process | Process-level | +| Hot reload | Requires restart | Independent | + +## Zentinel vs Nginx + +### Architecture Philosophy + +**Nginx** started as a high-performance web server and evolved into a versatile reverse proxy. It excels at serving static content, SSL termination, and basic proxying with an extensive module ecosystem. + +**Zentinel** is purpose-built as a security-focused reverse proxy without web server capabilities. It focuses on the proxy use case with deep integration for security agents. + +### When to Choose Nginx + +- Serving static files alongside proxying +- Need extensive third-party module ecosystem +- Using OpenResty for Lua-based customization +- Established Nginx operational expertise + +### When to Choose Zentinel + +- Security controls are the primary requirement +- Want isolated, updateable security components +- Prefer explicit configuration over complex conditionals +- Need static file serving with SPA support (`fallback` for try_files equivalent) + +### Configuration Comparison + +**Nginx**: +```nginx +upstream backend { + server 127.0.0.1:3000; + server 127.0.0.1:3001; +} + +server { + listen 8080; + + location / { + proxy_pass http://backend; + proxy_set_header Host $host; + proxy_set_header X-Real-IP $remote_addr; + } +} +``` + +**Zentinel** (KDL): +```kdl +listeners { + listener "http" { + address "0.0.0.0:8080" + protocol "http" + } +} + +routes { + route "default" { + matches { path-prefix "/" } + upstream "backend" + } +} + +upstreams { + upstream "backend" { + targets { + target { address "127.0.0.1:3000" } + target { address "127.0.0.1:3001" } + } + } +} +``` + +### Security Features + +| Feature | Nginx | Zentinel | +|---------|-------|----------| +| WAF | ModSecurity module | Native WAF agent | +| Rate limiting | ngx_http_limit_req | Rate limit agent | +| Authentication | Third-party modules | Auth agent | +| Custom logic | Lua/njs | Any language via agents | + +## Zentinel vs Traefik + +### Architecture Philosophy + +**Traefik** is a modern, cloud-native edge router designed for automatic service discovery and configuration. It excels in dynamic environments like Docker and Kubernetes where services come and go frequently. + +**Zentinel** focuses on explicit configuration and security-first design. While it supports service discovery (Consul, Kubernetes), it emphasizes predictable behavior over automatic configuration. + +### When to Choose Traefik + +- Heavy use of Docker labels for configuration +- Need automatic Let's Encrypt certificate provisioning +- Kubernetes Ingress controller use case +- Prefer dynamic, auto-discovered configuration + +### When to Choose Zentinel + +- Security agents are a primary requirement +- Want explicit, auditable configuration +- Need process isolation for security components +- Building custom security policies with agents +- Require token-aware rate limiting for LLM/inference workloads + +### Configuration Comparison + +**Traefik** (Docker labels): +```yaml +services: + app: + labels: + - "traefik.enable=true" + - "traefik.http.routers.app.rule=Host(`app.example.com`)" + - "traefik.http.services.app.loadbalancer.server.port=3000" +``` + +**Traefik** (File): +```yaml +http: + routers: + app: + rule: "Host(`app.example.com`)" + service: app + services: + app: + loadBalancer: + servers: + - url: "http://127.0.0.1:3000" +``` + +**Zentinel** (KDL): +```kdl +listeners { + listener "http" { + address "0.0.0.0:8080" + protocol "http" + } +} + +routes { + route "app" { + matches { + host "app.example.com" + } + upstream "app" + } +} + +upstreams { + upstream "app" { + targets { + target { address "127.0.0.1:3000" } + } + } +} +``` + +### Key Differences + +| Aspect | Traefik | Zentinel | +|--------|---------|----------| +| Configuration | Dynamic (labels, API) | Explicit (KDL files) | +| Let's Encrypt | Built-in | Planned | +| Forward Auth | Middleware | Agent-based | +| Extension model | Plugins (Go) | Agents (any language) | +| Isolation | In-process | Process-level | + +## Zentinel vs Caddy + +### Architecture Philosophy + +**Caddy** is known for its simplicity and automatic HTTPS. It pioneered zero-config TLS with built-in Let's Encrypt integration and uses a human-friendly Caddyfile syntax. + +**Zentinel** shares Caddy's focus on simplicity but prioritizes security extensibility over automatic configuration. The agent model provides flexibility that Caddy's module system cannot match for security use cases. + +### When to Choose Caddy + +- Want zero-config automatic HTTPS +- Simple static file serving with automatic TLS +- Prefer minimal configuration +- Need the extensive Caddy module ecosystem + +### When to Choose Zentinel + +- Need isolated security agents (WAF, auth, rate limiting) +- Building custom security controls +- Want process-level isolation for extensions +- Require inference/LLM-specific features (token counting, model routing) +- Need distributed rate limiting across instances + +### Configuration Comparison + +**Caddy** (Caddyfile): +``` +app.example.com { + reverse_proxy localhost:3000 +} + +static.example.com { + root * /var/www/public + file_server +} +``` + +**Zentinel** (KDL): +```kdl +listeners { + listener "https" { + address "0.0.0.0:443" + tls { + cert-path "/etc/zentinel/certs/app.crt" + key-path "/etc/zentinel/certs/app.key" + } + } +} + +routes { + route "app" { + matches { host "app.example.com" } + upstream "backend" + } + + route "static" { + matches { host "static.example.com" } + service-type "static" + static-files { + root "/var/www/public" + fallback "index.html" + } + } +} + +upstreams { + upstream "backend" { + targets { + target { address "localhost:3000" } + } + } +} +``` + +### Key Differences + +| Aspect | Caddy | Zentinel | +|--------|-------|----------| +| Automatic HTTPS | Built-in | Planned | +| Configuration | Caddyfile/JSON | KDL | +| Extension model | Modules (Go) | Agents (any language) | +| Isolation | In-process | Process-level | +| Static files | Built-in | Built-in with SPA fallback | + +## Agent Protocol Comparison + +Beyond proxy-level comparisons, it's important to understand how Zentinel's Agent Protocol V2 compares to extension mechanisms in other proxies. This is critical for security use cases where external processing is required. + +### Agent Protocol V2 vs Envoy ext_proc + +[Envoy's External Processing filter](https://www.envoyproxy.io/docs/envoy/latest/configuration/http/http_filters/ext_proc_filter) (ext_proc) is the closest analog to Zentinel's agent protocol. Both enable external services to inspect and modify requests/responses. + +| Aspect | Envoy ext_proc | Zentinel Agent Protocol V2 | +|--------|---------------|---------------------------| +| **Transport** | gRPC only | gRPC, UDS Binary, Reverse Connections | +| **Connection model** | Per-request stream | Pooled connections (reused) | +| **Default timeout** | 200ms | Configurable (30s default) | +| **Flow control** | [In development](https://github.com/envoyproxy/envoy/issues/33319) | Implemented (pause/resume) | +| **Body streaming** | Full duplex available | Zero-copy with MessagePack (62 GiB/s) | +| **Binary encoding** | Protobuf | MessagePack + JSON | +| **Typical added latency** | 1-6ms | ~230ns hot path | +| **Circuit breaker** | Via Envoy config | Built into protocol | +| **NAT traversal** | Not supported | Reverse connections | + +**Where Zentinel wins:** + +- **3 transport options** — ext_proc is gRPC-only; Zentinel supports UDS for same-host deployment (0.4ms vs 1.2ms latency) and reverse connections for agents behind NAT/firewalls +- **Connection pooling** — ext_proc creates a new gRPC stream per request; Zentinel reuses pooled connections with configurable strategies (RoundRobin, LeastConnections, HealthBased) +- **Flow control** — ext_proc is [still developing this](https://github.com/envoyproxy/envoy/issues/33319); Zentinel has working pause/resume signals with backpressure +- **Performance** — Zentinel's hot path completes in ~230ns; ext_proc typically adds 1-6ms +- **Simpler configuration** — KDL vs complex protobuf/YAML with Envoy's filter chain + +**Where ext_proc wins:** + +- Mature ecosystem, battle-tested at massive scale +- Native Envoy integration (no separate proxy) +- Larger community and more documentation +- Part of the CNCF ecosystem + +### Agent Protocol V2 vs HAProxy SPOE + +[HAProxy's Stream Processing Offload Engine](https://www.haproxy.com/blog/extending-haproxy-with-the-stream-processing-offload-engine) (SPOE) enables external agents to process traffic. It uses a custom binary protocol (SPOP) over TCP. + +| Aspect | HAProxy SPOE | Zentinel Agent Protocol V2 | +|--------|-------------|---------------------------| +| **Protocol** | Custom binary (SPOP) | gRPC + MessagePack + JSON | +| **Matured in** | HAProxy 1.8 (2017) | 2025-2026 | +| **Encoding** | Custom binary | Industry-standard formats | +| **Body access** | Limited (header-focused) | Full streaming (62 GiB/s) | +| **Connection model** | Persistent TCP | Pooled with affinity tracking | +| **Language support** | C, Go, Python, Lua | Any (standard protocols) | +| **Metrics** | Via HAProxy stats | Built-in Prometheus export | +| **Circuit breaker** | Via HAProxy config | Built into protocol | + +**Where Zentinel wins:** + +- **Standard protocols** — gRPC and MessagePack vs custom binary; easier to implement agents in any language with existing libraries +- **Full body streaming** — SPOE is primarily designed for header inspection; Zentinel has zero-copy body chunks with 62 GiB/s throughput +- **Built-in observability** — Protocol-level metrics (counters, histograms, gauges) with native Prometheus export +- **Modern features** — Health-based load balancing, connection affinity for streaming requests, NAT traversal + +**Where SPOE wins:** + +- Tight HAProxy integration with minimal overhead +- Very lightweight for header-only inspection use cases +- Battle-tested in production for 7+ years +- Simpler mental model for basic use cases + +### Agent Protocol V2 vs NGINX njs + +[NGINX njs](https://nginx.org/en/docs/njs/) is a JavaScript runtime embedded in NGINX for request processing. Unlike external agents, njs runs in-process. + +| Aspect | NGINX njs | Zentinel Agent Protocol V2 | +|--------|----------|---------------------------| +| **Execution model** | In-process JavaScript | External process (any language) | +| **Isolation** | None (crash = NGINX crash) | Full process isolation | +| **Language** | JavaScript (ES2023 with QuickJS) | Any (Rust, Go, Python, etc.) | +| **Memory** | Shared with NGINX | Separate process memory | +| **Body streaming** | Callback-based | True streaming with backpressure | +| **Garbage collection** | Yes (QuickJS GC) | Language-dependent (none for Rust) | +| **Hot reload** | Requires NGINX reload | Independent agent updates | +| **Throughput** | Limited by JS overhead | 62 GiB/s body streaming | + +**Where Zentinel wins:** + +- **Process isolation** — A buggy or crashing agent cannot take down the proxy; njs errors can crash NGINX +- **Language flexibility** — Write agents in Rust, Go, Python, Java, or any language; njs is JavaScript-only +- **Independent scaling** — Scale agents separately from the proxy; njs scales with NGINX workers +- **True streaming** — Flow control and backpressure vs JavaScript callbacks +- **No GC pauses** — Rust agents have no garbage collection; [njs/QuickJS has GC overhead](https://blog.nginx.org/blog/quickjs-engine-support-for-njs) +- **Performance** — 62 GiB/s body throughput vs JavaScript processing overhead + +**Where njs wins:** + +- Zero network overhead (in-process execution) +- Simpler deployment (no separate service to manage) +- Good for lightweight transformations and header manipulation +- [Context reuse](https://blog.nginx.org/blog/quickjs-engine-support-for-njs) minimizes per-request overhead +- Familiar JavaScript syntax + +### Protocol Feature Matrix + +| Feature | Zentinel V2 | Envoy ext_proc | HAProxy SPOE | NGINX njs | +|---------|:-----------:|:--------------:|:------------:|:---------:| +| Process isolation | ✓ | ✓ | ✓ | ✗ | +| Multiple transports | ✓ (3) | ✗ (gRPC only) | ✗ (TCP only) | N/A | +| Connection pooling | ✓ | ✗ | ✓ | N/A | +| Flow control | ✓ | 🚧 In progress | ✓ | N/A | +| Body streaming | ✓ Zero-copy | ✓ | Limited | Callbacks | +| Binary encoding | ✓ MessagePack | Protobuf | Custom | N/A | +| Circuit breaker | ✓ Built-in | Via Envoy | Via HAProxy | ✗ | +| NAT traversal | ✓ Reverse conn | ✗ | ✗ | N/A | +| Any language | ✓ | ✓ | ✓ | ✗ JS only | +| Metrics export | ✓ Prometheus | Via Envoy | Via HAProxy | ✗ | +| Connection affinity | ✓ | ✗ | ✗ | N/A | + +### Performance Comparison + +Based on benchmarks run on the same hardware: + +| Metric | Zentinel V2 | Typical ext_proc | SPOE | njs | +|--------|-------------|------------------|------|-----| +| Hot path latency | ~230ns | 1-6ms | ~500μs | ~100μs | +| Body throughput | 62 GiB/s | N/A | Limited | ~1 GiB/s | +| Connection overhead | Pooled | Per-request | Persistent | None | +| Serialization | 150-560ns | Protobuf | Custom | N/A | + +**Note**: These numbers are indicative. Actual performance depends on workload, configuration, and hardware. + +### When to Use Each + +| Use Case | Recommended | +|----------|-------------| +| Security gateway with WAF/auth | **Zentinel V2** | +| Service mesh sidecar | Envoy ext_proc | +| Simple header inspection | HAProxy SPOE | +| Lightweight request transforms | NGINX njs | +| High-throughput body processing | **Zentinel V2** | +| Agents behind NAT/firewall | **Zentinel V2** | +| Maximum ecosystem maturity | Envoy ext_proc | +| Minimal operational complexity | NGINX njs | + +--- + +## Zentinel Unique Features + +Beyond standard proxy capabilities, Zentinel offers features designed for modern workloads: + +### Inference/LLM Gateway + +Zentinel has first-class support for LLM and inference workloads: + +| Feature | Description | +|---------|-------------| +| **Token-aware rate limiting** | Rate limit by tokens (not just requests) using tiktoken | +| **Token budgets** | Daily/monthly cumulative token limits per client | +| **Cost tracking** | Per-request cost attribution ($) | +| **Model-based routing** | Route `gpt-4*` to OpenAI, `claude-*` to Anthropic | +| **Streaming token counting** | Count tokens in SSE responses | +| **Least-tokens load balancing** | Route to backend with lowest token queue | + +No other reverse proxy offers these capabilities natively. + +### External Agent Architecture + +Zentinel's agent model provides unique isolation guarantees: + +| Capability | Benefit | +|------------|---------| +| **Process isolation** | Agent crash never takes down proxy | +| **Language flexibility** | Write agents in Python, Go, Rust, TypeScript, Elixir | +| **Independent deployment** | Update agents without proxy restart | +| **Resource limits** | Per-agent concurrency limits and circuit breakers | +| **WASM sandbox** | In-process agents with Wasmtime isolation | + +### Distributed Rate Limiting + +Native support for distributed rate limiting across instances: + +- Redis backend (feature: `distributed-rate-limit`) +- Memcached backend (feature: `distributed-rate-limit-memcached`) +- Graceful degradation to local limits if backend fails + +### Service Discovery + +Built-in discovery for dynamic environments: + +- Consul integration +- Kubernetes service discovery (feature: `kubernetes`) +- DNS resolution with TTL + +### Security Features + +- **GeoIP filtering** - Block/allow by country (MaxMind, IP2Location) +- **Decompression bomb protection** - Ratio limits (max 100x, 10MB output) +- **Guardrails** - Prompt injection detection for LLM workloads +- **PII detection** - Identify and mask sensitive data + +## Feature Comparison Matrix + +### Core Proxy Features + +| Feature | Zentinel | Envoy | HAProxy | Nginx | Traefik | Caddy | +|---------|:--------:|:-----:|:-------:|:-----:|:-------:|:-----:| +| HTTP/1.1 | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ | +| HTTP/2 | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ | +| HTTP/3 (QUIC) | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ | +| WebSocket | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ | +| gRPC | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ | +| TCP proxy | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ | +| TLS termination | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ | +| mTLS | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ | +| Static files | ✓ | - | - | ✓ | ✓ | ✓ | +| SPA fallback (try_files) | ✓ | - | - | ✓ | - | ✓ | + +### Load Balancing + +| Feature | Zentinel | Envoy | HAProxy | Nginx | Traefik | Caddy | +|---------|:--------:|:-----:|:-------:|:-----:|:-------:|:-----:| +| Round robin | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ | +| Least connections | ✓ | ✓ | ✓ | ✓ | - | ✓ | +| Consistent hashing | ✓ | ✓ | ✓ | ✓ | - | - | +| Weighted | ✓ | ✓ | ✓ | ✓ | ✓ | - | +| Least tokens (LLM) | ✓ | - | - | - | - | - | +| Adaptive (latency) | ✓ | ✓ | - | - | - | - | +| Active health checks | ✓ | ✓ | ✓ | ✓* | ✓ | ✓ | +| Passive health checks | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ | +| Circuit breakers | ✓ | ✓ | - | - | ✓ | - | + +*Nginx Plus only for active health checks + +### Security & Extensions + +| Feature | Zentinel | Envoy | HAProxy | Nginx | Traefik | Caddy | +|---------|:--------:|:-----:|:-------:|:-----:|:-------:|:-----:| +| External agents | ✓ | - | SPOE | - | - | - | +| WASM extensions | ✓ | ✓ | - | - | ✓ | - | +| Rate limiting | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ | +| Distributed rate limit | ✓ | - | - | - | - | - | +| Token-aware rate limit | ✓ | - | - | - | - | - | +| Forward auth | Planned | - | - | - | ✓ | ✓ | +| JWT validation | ✓ | ✓ | Lua | Module | ✓ | ✓ | +| GeoIP filtering | ✓ | - | - | Module | - | - | +| WAF (OWASP CRS) | Agent | - | SPOE | Module | - | - | + +### Observability + +| Feature | Zentinel | Envoy | HAProxy | Nginx | Traefik | Caddy | +|---------|:--------:|:-----:|:-------:|:-----:|:-------:|:-----:| +| Prometheus metrics | ✓ | ✓ | ✓ | Module | ✓ | ✓ | +| Distributed tracing | ✓ | ✓ | ✓ | Module | ✓ | ✓ | +| Access logs | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ | +| Structured logging | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ | + +### Operations + +| Feature | Zentinel | Envoy | HAProxy | Nginx | Traefik | Caddy | +|---------|:--------:|:-----:|:-------:|:-----:|:-------:|:-----:| +| Hot reload config | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ | +| Zero-downtime restart | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ | +| Auto HTTPS (ACME) | Planned | - | - | - | ✓ | ✓ | +| Dynamic config (API) | ✓ | ✓ (xDS) | ✓ | Plus | ✓ | ✓ | +| Graceful shutdown | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ | +| Service discovery | ✓ | ✓ | ✓ | Plus | ✓ | - | + +## Memory Safety + +A key differentiator for Zentinel is memory safety through Rust: + +| Proxy | Language | Memory Safe | CVEs (2020-2024) | +|-------|----------|:-----------:|:----------------:| +| Zentinel | Rust | ✓ | 0 | +| Envoy | C++ | - | 30+ | +| HAProxy | C | - | 15+ | +| Nginx | C | - | 25+ | +| Traefik | Go | ✓ | 5+ | +| Caddy | Go | ✓ | 3+ | + +Memory safety eliminates entire classes of vulnerabilities: +- Buffer overflows +- Use-after-free +- Double-free +- Null pointer dereferences + +## Performance Characteristics + +All six proxies are capable of handling high traffic loads. The primary differences are: + +| Aspect | Zentinel | Envoy | HAProxy | Nginx | Traefik | Caddy | +|--------|----------|-------|---------|-------|---------|-------| +| Latency | Low | Low | Very low | Low | Low | Low | +| Throughput | High | High | Very high | High | High | High | +| Memory usage | Predictable | Higher | Very low | Low | Moderate | Moderate | +| CPU efficiency | High | High | Very high | High | High | High | + +**Note**: Benchmark results vary significantly based on workload, configuration, and hardware. Always benchmark with your specific use case. + +### Agent Overhead + +Zentinel's agent model adds latency for agent calls: +- Unix socket: ~50-200µs per agent +- gRPC: ~200-500µs per agent + +This overhead is acceptable for security use cases where the alternative is in-process complexity or external service calls. + +## Migration Paths + +### From Nginx to Zentinel + +1. Map `server` blocks to `listeners` +2. Convert `location` blocks to `routes` +3. Translate `upstream` blocks +4. Replace modules with agents + +See the [Migration Guide](/operations/migration/) for detailed examples. + +### From HAProxy to Zentinel + +1. Map `frontend` to `listeners` +2. Convert `backend` to `upstreams` +3. Translate ACLs to route matching +4. Replace Lua/SPOE with agents + +### From Envoy to Zentinel + +1. Simplify listener configuration +2. Convert clusters to upstreams +3. Replace filters with agents +4. Remove xDS dependency (if applicable) + +### From Traefik to Zentinel + +1. Convert routers to `routes` blocks +2. Map services to `upstreams` +3. Replace middlewares with agents +4. Move from Docker labels to KDL files +5. Replace automatic HTTPS with manual certs (ACME support planned) + +### From Caddy to Zentinel + +1. Convert Caddyfile blocks to KDL +2. Map `reverse_proxy` to routes + upstreams +3. Move from automatic HTTPS to manual certs (ACME support planned) +4. Replace modules with agents for security policies + +## Summary + +Choose **Zentinel** when: +- Security is a primary concern +- You want isolated, updateable security components +- Memory safety matters for your threat model +- You prefer explicit, readable configuration +- Building custom security policies +- Need LLM/inference gateway features (token limiting, model routing) + +Choose **Envoy** when: +- Building a service mesh +- Need extensive protocol support +- Using xDS-based control planes +- Require Wasm extensibility + +Choose **HAProxy** when: +- Maximum performance is critical +- Pure load balancing use case +- Deep TCP-level control needed +- Established HAProxy expertise + +Choose **Nginx** when: +- Serving static files alongside proxying +- Need the extensive module ecosystem +- Using OpenResty/Lua extensively +- Established Nginx expertise + +Choose **Traefik** when: +- Heavy Docker/Kubernetes environment +- Want automatic service discovery +- Need built-in Let's Encrypt support +- Prefer dynamic, label-based configuration + +Choose **Caddy** when: +- Want zero-config automatic HTTPS +- Simple use case with minimal configuration +- Need the Caddy module ecosystem +- Prefer Caddyfile simplicity + +## Next Steps + +- [Architecture](../architecture/) - Understand Zentinel's design +- [Agents](/agents/) - Explore the agent ecosystem +- [Migration Guide](/operations/migration/) - Migrate from other proxies diff --git a/content/v/26.04/concepts/components.md b/content/v/26.04/concepts/components.md new file mode 100644 index 0000000..cb974dc --- /dev/null +++ b/content/v/26.04/concepts/components.md @@ -0,0 +1,445 @@ ++++ +title = "Component Design" +weight = 2 +updated = 2026-02-19 ++++ + +Zentinel is organized as a Cargo workspace with four core crates. This page explains each component's responsibilities and how they interact. + +## Crate Structure + +``` +zentinel/ +├── crates/ +│ ├── proxy/ # Main proxy binary and library +│ ├── config/ # Configuration parsing and validation +│ ├── agent-protocol/ # Agent communication protocol +│ └── common/ # Shared types and utilities +└── agents/ + └── echo/ # Reference agent implementation +``` + +## Proxy Crate + +**Package**: `zentinel-proxy` +**Binary**: `zentinel` + +The main proxy implementation that ties everything together. + +### Key Modules + +| Module | Purpose | +|--------|---------| +| `main.rs` | CLI entry point, signal handling | +| `proxy/` | Core proxy logic, Pingora integration | +| `routing.rs` | Route matching and compilation | +| `upstream/` | Upstream pool management, load balancing | +| `agents/` | Agent manager and coordination | +| `static_files/` | Static file serving | +| `health.rs` | Active and passive health checking | +| `reload/` | Configuration hot reload | +| `logging.rs` | Access, error, and audit logging | + +### Proxy Module + +The `ZentinelProxy` struct implements Pingora's `ProxyHttp` trait: + +```rust +impl ProxyHttp for ZentinelProxy { + // Select upstream target for request + async fn upstream_peer(&self, session: &mut Session, ctx: &mut Context) + -> Result>; + + // Process request before forwarding + async fn request_filter(&self, session: &mut Session, ctx: &mut Context) + -> Result; + + // Process response before returning to client + async fn response_filter(&self, session: &mut Session, ctx: &mut Context) + -> Result<()>; + + // Log after request completes + async fn logging(&self, session: &mut Session, ctx: &mut Context); +} +``` + +### Routing Module + +Routes are compiled at startup for efficient matching: + +```rust +pub struct RouteMatcher { + routes: Vec, // Sorted by priority + cache: LruCache, // Path cache +} + +pub struct CompiledRoute { + id: RouteId, + priority: Priority, + matchers: Vec, + specificity: u32, // For tie-breaking +} + +pub enum CompiledMatcher { + Path(String), + PathPrefix(String), + PathRegex(Regex), + Host(String), + Method(Vec), + Header { name: String, value: Option }, + QueryParam { key: String, value: Option }, +} +``` + +### Upstream Module + +Manages backend server pools with multiple load balancing strategies: + +```rust +pub struct UpstreamPool { + id: UpstreamId, + targets: Vec, + load_balancer: Box, + health_checker: HealthChecker, + circuit_breaker: CircuitBreaker, + connection_pool: ConnectionPool, +} + +pub trait LoadBalancer: Send + Sync { + fn select(&self, targets: &[Target], ctx: &RequestContext) -> Option<&Target>; +} +``` + +**Load Balancing Algorithms**: + +| Algorithm | Description | Use Case | +|-----------|-------------|----------| +| `round_robin` | Sequential rotation | General purpose | +| `least_connections` | Fewest active connections | Variable latency backends | +| `ip_hash` | Hash client IP | Session affinity | +| `consistent_hash` | Consistent hashing | Cache distribution | +| `p2c` | Power of Two Choices | Low latency selection | +| `adaptive` | Adjusts based on response times | Mixed workloads | + +### Agent Module + +Coordinates external agent communication: + +```rust +pub struct AgentManager { + agents: HashMap, + pools: HashMap, + circuit_breakers: HashMap, + metrics: AgentMetrics, +} + +pub struct Agent { + id: AgentId, + transport: AgentTransport, + timeout: Duration, + failure_mode: FailureMode, + events: Vec, +} +``` + +## Config Crate + +**Package**: `zentinel-config` + +Handles configuration parsing, validation, and hot reload. + +### Supported Formats + +- **KDL** (primary) - Human-friendly document language +- **TOML** - Standard configuration format +- **YAML** - For Kubernetes integration +- **JSON** - For programmatic generation + +### Configuration Structure + +```rust +pub struct Config { + pub server: ServerConfig, + pub listeners: Vec, + pub routes: Vec, + pub upstreams: HashMap, + pub filters: HashMap, + pub agents: Vec, + pub waf: Option, + pub limits: Limits, + pub observability: ObservabilityConfig, +} +``` + +### Key Types + +```rust +pub struct RouteConfig { + pub id: String, + pub priority: Priority, + pub matches: Vec, + pub upstream: Option, + pub service_type: ServiceType, + pub filters: Vec, + pub policies: RoutePolicies, +} + +pub struct UpstreamConfig { + pub targets: Vec, + pub load_balancing: LoadBalancingAlgorithm, + pub health_check: Option, + pub timeouts: TimeoutConfig, + pub circuit_breaker: Option, +} + +pub struct AgentConfig { + pub id: String, + pub agent_type: AgentType, + pub transport: AgentTransport, + pub events: Vec, + pub timeout_ms: u64, + pub failure_mode: FailureMode, +} +``` + +### Validation + +Configuration is validated at multiple levels: + +1. **Schema validation** - Structure and types +2. **Semantic validation** - Cross-references (route → upstream) +3. **Custom validators** - Business rules + +```rust +pub trait ConfigValidator { + fn validate(&self, config: &Config) -> Result<(), Vec>; +} +``` + +### Hot Reload + +Configuration changes are applied atomically: + +```rust +pub struct ConfigManager { + config: ArcSwap, + watcher: FileWatcher, + validators: Vec>, + subscribers: Vec>, +} + +pub enum ReloadEvent { + Applied(Arc), + Failed(ValidationError), +} +``` + +## Agent Protocol Crate + +**Package**: `zentinel-agent-protocol` + +Defines the contract between Zentinel and external agents. + +### Transport Options + +| Transport | Format | Use Case | +|-----------|--------|----------| +| Unix Socket | JSON | Default, same-host agents | +| gRPC | Protobuf | Cross-host, high-performance | + +### Protocol Types + +```rust +pub enum EventType { + RequestHeaders, + RequestBodyChunk, + ResponseHeaders, + ResponseBodyChunk, + RequestComplete, +} + +pub struct AgentRequest { + pub event_type: EventType, + pub correlation_id: String, + pub request_id: String, + pub metadata: RequestMetadata, + pub headers: Vec
, + pub body_chunk: Option>, +} + +pub struct AgentResponse { + pub decision: Decision, + pub header_mutations: HeaderMutations, + pub metadata: HashMap, + pub audit: AuditInfo, +} + +pub enum Decision { + Allow, + Block { status: u16, body: Option }, + Redirect { url: String, status: u16 }, + Challenge { challenge_type: String, data: String }, +} +``` + +### Header Mutations + +Agents can modify request and response headers: + +```rust +pub struct HeaderMutations { + pub request: HeaderOps, + pub response: HeaderOps, +} + +pub struct HeaderOps { + pub set: HashMap, // Replace or create + pub add: HashMap, // Append + pub remove: Vec, // Delete +} +``` + +### Client and Server + +```rust +// Proxy side - manages agent connections +pub struct AgentPool { + agents: HashMap, + config: AgentPoolConfig, +} + +// Agent side - receives calls +pub struct UdsAgentServerV2 { + name: String, + socket_path: PathBuf, + handler: Box, +} + +pub trait AgentHandlerV2: Send + Sync { + fn capabilities(&self) -> AgentCapabilities; + async fn on_request_headers(&self, event: RequestHeadersEvent) -> AgentResponse; + // ... other event handlers +} +``` + +## Common Crate + +**Package**: `zentinel-common` + +Shared types and utilities used across all crates. + +### Type-Safe IDs + +```rust +// Strongly typed identifiers prevent mix-ups +pub struct CorrelationId(String); +pub struct RequestId(String); +pub struct RouteId(String); +pub struct UpstreamId(String); +pub struct AgentId(String); +``` + +### Error Types + +```rust +pub enum ZentinelError { + Config(ConfigError), + Routing(RoutingError), + Upstream(UpstreamError), + Agent(AgentError), + Validation(ValidationError), + Io(std::io::Error), +} + +pub type ZentinelResult = Result; +``` + +### Circuit Breaker + +```rust +pub struct CircuitBreaker { + state: AtomicState, + failure_threshold: u32, + success_threshold: u32, + timeout: Duration, + failure_count: AtomicU32, + success_count: AtomicU32, + last_failure: AtomicInstant, +} + +pub enum CircuitState { + Closed, // Normal operation + Open, // Failing, fast-reject + HalfOpen, // Testing recovery +} +``` + +### Limits + +```rust +pub struct Limits { + pub max_header_count: usize, + pub max_header_size_bytes: usize, + pub max_body_size_bytes: usize, + pub max_connections_per_client: usize, + pub max_total_connections: usize, + pub max_in_flight_requests: usize, +} +``` + +### Observability + +```rust +pub struct RequestMetrics { + pub route_id: RouteId, + pub method: Method, + pub status_code: u16, + pub latency: Duration, + pub upstream_latency: Option, + pub bytes_in: u64, + pub bytes_out: u64, +} +``` + +## Component Interactions + +``` +┌─────────────────────────────────────────────────────────────────┐ +│ zentinel-proxy │ +│ ┌─────────┐ ┌─────────┐ ┌─────────┐ ┌─────────────────────┐│ +│ │ Routing │ │Upstream │ │ Agents │ │ Static Files ││ +│ └────┬────┘ └────┬────┘ └────┬────┘ └─────────────────────┘│ +│ │ │ │ │ +└───────┼────────────┼────────────┼───────────────────────────────┘ + │ │ │ + │ │ │ +┌───────▼────────────▼────────────▼───────────────────────────────┐ +│ zentinel-config │ +│ ┌─────────────┐ ┌──────────────┐ ┌────────────────────────┐ │ +│ │ Parsing │ │ Validation │ │ Hot Reload │ │ +│ └─────────────┘ └──────────────┘ └────────────────────────┘ │ +└─────────────────────────────────────────────────────────────────┘ + │ + │ +┌───────▼─────────────────────────────────────────────────────────┐ +│ zentinel-agent-protocol │ +│ ┌─────────────┐ ┌──────────────┐ ┌────────────────────────┐ │ +│ │ Client │ │ Types │ │ Server │ │ +│ └─────────────┘ └──────────────┘ └────────────────────────┘ │ +└─────────────────────────────────────────────────────────────────┘ + │ + │ +┌───────▼─────────────────────────────────────────────────────────┐ +│ zentinel-common │ +│ ┌─────────┐ ┌──────────┐ ┌─────────┐ ┌──────────────────┐ │ +│ │ Types │ │ Errors │ │ Circuit │ │ Observability │ │ +│ │ (IDs) │ │ │ │ Breaker │ │ │ │ +│ └─────────┘ └──────────┘ └─────────┘ └──────────────────┘ │ +└─────────────────────────────────────────────────────────────────┘ +``` + +## Next Steps + +- [Architecture Overview](../architecture/) - High-level design +- [Request Flow](../request-flow/) - Detailed request lifecycle +- [Routing System](../routing/) - How routes are matched diff --git a/content/v/26.04/concepts/pingora.md b/content/v/26.04/concepts/pingora.md new file mode 100644 index 0000000..a43b407 --- /dev/null +++ b/content/v/26.04/concepts/pingora.md @@ -0,0 +1,418 @@ ++++ +title = "Pingora Foundation" +weight = 4 +updated = 2026-02-19 ++++ + +Zentinel is built on [Cloudflare's Pingora](https://github.com/cloudflare/pingora), a battle-tested HTTP proxy framework written in Rust. This page explains what Pingora provides and how Zentinel extends it. + +## What is Pingora? + +Pingora is an open-source proxy framework that Cloudflare uses to handle **over 1 trillion requests per day**. It provides: + +- **High-performance async HTTP handling** using Tokio +- **Connection pooling** to upstream servers +- **TLS termination** with modern cipher suites +- **HTTP/1.1 and HTTP/2** support +- **Zero-copy buffer management** for efficiency +- **Graceful shutdown and upgrades** + +Zentinel uses Pingora as its foundation, adding routing, load balancing, agent coordination, and configuration management on top. + +## Why Pingora? + +| Requirement | Pingora Solution | +|-------------|------------------| +| **Performance** | Handles millions of requests/sec with low latency | +| **Safety** | Written in Rust with memory safety guarantees | +| **Production-proven** | Powers Cloudflare's global edge network | +| **Extensibility** | Clean trait-based architecture for customization | +| **Operational** | Built-in graceful restart and upgrade support | + +### Compared to Alternatives + +| Framework | Language | Trade-offs | +|-----------|----------|------------| +| **Pingora** | Rust | Best performance + safety, smaller ecosystem | +| **Envoy** | C++ | Feature-rich but complex, memory safety concerns | +| **HAProxy** | C | Mature but harder to extend, no memory safety | +| **Nginx** | C | Ubiquitous but module development is challenging | + +## Core Pingora Concepts + +### Server and Services + +Pingora applications start with a `Server` that manages one or more services: + +```rust +// Create Pingora server with options +let mut server = Server::new(Some(pingora_opt))?; +server.bootstrap(); + +// Create HTTP proxy service +let proxy_service = http_proxy_service(&server.configuration, proxy); + +// Add listeners +proxy_service.add_tcp("0.0.0.0:8080"); + +// Register service and run +server.add_service(proxy_service); +server.run_forever(); +``` + +The server handles: +- Worker process management +- Signal handling (SIGHUP, SIGTERM) +- Graceful restarts and upgrades +- Daemonization + +### Session + +A `Session` represents a single HTTP request/response cycle. It provides access to: + +```rust +// Request information +session.req_header() // HTTP request headers +session.req_header_mut() // Mutable access for modifications +session.client_addr() // Client IP address + +// Response information +session.response_written() // Response after sending + +// Body handling +session.read_request_body() // Read request body chunks +session.write_response_body() // Write response body +``` + +### HttpPeer + +An `HttpPeer` represents an upstream server connection target: + +```rust +let peer = HttpPeer::new( + ("backend.example.com", 8080), // Address + false, // TLS enabled + "backend.example.com".into() // SNI hostname +); + +// Connection options +peer.options.connection_timeout = Some(Duration::from_secs(5)); +peer.options.read_timeout = Some(Duration::from_secs(30)); +``` + +Pingora maintains connection pools to peers for efficiency. + +## The ProxyHttp Trait + +The `ProxyHttp` trait is the heart of Pingora's extensibility. Zentinel implements this trait to inject custom logic at each stage of request processing: + +```rust +#[async_trait] +impl ProxyHttp for ZentinelProxy { + type CTX = RequestContext; + + // Create per-request context + fn new_ctx(&self) -> Self::CTX { + RequestContext::new() + } + + // Select upstream server + async fn upstream_peer( + &self, + session: &mut Session, + ctx: &mut Self::CTX, + ) -> Result, Box>; + + // Process request before forwarding + async fn request_filter( + &self, + session: &mut Session, + ctx: &mut Self::CTX, + ) -> Result>; + + // Process response before returning + async fn response_filter( + &self, + session: &mut Session, + upstream_response: &mut ResponseHeader, + ctx: &mut Self::CTX, + ) -> Result<(), Box>; + + // Final logging after request completes + async fn logging( + &self, + session: &mut Session, + error: Option<&Error>, + ctx: &mut Self::CTX, + ); +} +``` + +### Request Context + +Each request gets its own context that persists throughout the lifecycle: + +```rust +pub struct RequestContext { + pub trace_id: String, + pub start_time: Instant, + pub route_id: Option, + pub upstream: Option, + pub client_ip: String, + pub method: String, + pub path: String, + pub upstream_attempts: u32, + // ... more fields +} +``` + +## How Zentinel Uses Pingora + +### 1. Route Matching (`upstream_peer`) + +When a request arrives, Zentinel matches it to a route and selects an upstream: + +``` +Request arrives + │ + ▼ +┌────────────────────┐ +│ Parse request info │ +│ (method, path, │ +│ host, headers) │ +└────────┬───────────┘ + │ + ▼ +┌────────────────────┐ +│ Match against │ +│ compiled routes │ +└────────┬───────────┘ + │ + ▼ +┌────────────────────┐ +│ Select peer from │ +│ upstream pool │ +└────────┬───────────┘ + │ + ▼ + Return HttpPeer +``` + +### 2. Request Processing (`request_filter`) + +Before forwarding, Zentinel applies filters and calls agents: + +```rust +async fn request_filter(&self, session: &mut Session, ctx: &mut Self::CTX) + -> Result> +{ + // Handle static files and builtins + if route.service_type == ServiceType::Static { + return self.handle_static_route(session, ctx).await; + } + + // Enforce limits + if headers.len() > config.limits.max_header_count { + return Err(Error::explain("Too many headers")); + } + + // Add tracing headers + req_header.insert_header("X-Correlation-Id", &ctx.trace_id)?; + req_header.insert_header("X-Forwarded-By", "Zentinel")?; + + // Call external agents + self.process_agents(session, ctx).await?; + + Ok(false) // Continue to upstream +} +``` + +Returning `Ok(true)` short-circuits processing (response already sent). +Returning `Ok(false)` continues to the upstream. + +### 3. Response Processing (`response_filter`) + +After receiving the upstream response: + +```rust +async fn response_filter( + &self, + session: &mut Session, + upstream_response: &mut ResponseHeader, + ctx: &mut Self::CTX, +) -> Result<(), Box> { + // Add security headers + upstream_response.insert_header("X-Content-Type-Options", "nosniff")?; + upstream_response.insert_header("X-Frame-Options", "DENY")?; + + // Add correlation ID + upstream_response.insert_header("X-Correlation-Id", &ctx.trace_id)?; + + // Record metrics + self.metrics.record_request( + ctx.route_id.as_deref().unwrap_or("unknown"), + &ctx.method, + upstream_response.status.as_u16(), + ctx.elapsed(), + ); + + // Update health status + self.passive_health.record_outcome(&upstream, success).await; + + Ok(()) +} +``` + +### 4. Logging (`logging`) + +After the response is sent to the client: + +```rust +async fn logging(&self, session: &mut Session, error: Option<&Error>, ctx: &mut Self::CTX) { + // Decrement active request counter + self.reload_coordinator.dec_requests(); + + // Write structured access log + let entry = AccessLogEntry { + timestamp: Utc::now().to_rfc3339(), + trace_id: ctx.trace_id.clone(), + method: ctx.method.clone(), + path: ctx.path.clone(), + status: session.response_written().map(|r| r.status.as_u16()), + duration_ms: ctx.elapsed().as_millis(), + // ... + }; + + self.log_manager.log_access(&entry); +} +``` + +## Connection Pooling + +Pingora automatically pools connections to upstream servers: + +``` +┌─────────────────────────────────────────────────────────┐ +│ Zentinel Proxy │ +│ ┌───────────────────────────────────────────────────┐ │ +│ │ Connection Pool Manager │ │ +│ │ ┌─────────────┐ ┌─────────────┐ ┌───────────┐ │ │ +│ │ │ backend-1 │ │ backend-2 │ │ backend-3 │ │ │ +│ │ │ ┌─┐┌─┐┌─┐ │ │ ┌─┐┌─┐┌─┐ │ │ ┌─┐┌─┐ │ │ │ +│ │ │ │C││C││C│ │ │ │C││C││C│ │ │ │C││C│ │ │ │ +│ │ │ └─┘└─┘└─┘ │ │ └─┘└─┘└─┘ │ │ └─┘└─┘ │ │ │ +│ │ └─────────────┘ └─────────────┘ └───────────┘ │ │ +│ └───────────────────────────────────────────────────┘ │ +└─────────────────────────────────────────────────────────┘ + │ │ │ + ▼ ▼ ▼ + ┌─────────┐ ┌─────────┐ ┌─────────┐ + │ Backend │ │ Backend │ │ Backend │ + │ 1 │ │ 2 │ │ 3 │ + └─────────┘ └─────────┘ └─────────┘ +``` + +Benefits: +- **Reduced latency** - Reuses existing TCP connections +- **Lower resource usage** - Fewer connections to manage +- **Connection limits** - Prevents overwhelming backends + +## Graceful Operations + +### Hot Restart + +Pingora supports zero-downtime restarts: + +``` +┌──────────────┐ SIGUSR2 ┌──────────────┐ +│ Old Worker │ ───────────────▶ │ New Worker │ +│ (draining) │ │ (starting) │ +└──────┬───────┘ └──────┬───────┘ + │ │ + │ Existing connections │ New connections + │ finish gracefully │ accepted + ▼ ▼ + [exit when done] [fully operational] +``` + +### Graceful Shutdown + +On SIGTERM/SIGINT: + +1. Stop accepting new connections +2. Wait for in-flight requests (with timeout) +3. Close connection pools +4. Exit cleanly + +Zentinel extends this with reload coordination: + +```rust +pub struct GracefulReloadCoordinator { + active_requests: AtomicUsize, + max_drain_time: Duration, +} + +impl GracefulReloadCoordinator { + pub fn inc_requests(&self) { /* ... */ } + pub fn dec_requests(&self) { /* ... */ } + pub async fn wait_for_drain(&self) { /* ... */ } +} +``` + +## Error Handling + +Pingora uses a typed error system: + +```rust +pub enum ErrorType { + InvalidHTTPHeader, + ConnectTimedout, + ConnectRefused, + ConnectNoRoute, + ReadError, + WriteError, + // ... many more +} +``` + +Zentinel maps these to appropriate HTTP responses: + +| Error Type | HTTP Status | Response | +|------------|-------------|----------| +| `ConnectTimedout` | 504 | Gateway Timeout | +| `ConnectRefused` | 502 | Bad Gateway | +| `ReadError` | 502 | Bad Gateway | +| `InvalidHTTPHeader` | 400 | Bad Request | + +## Performance Characteristics + +Pingora's architecture enables: + +| Metric | Typical Value | +|--------|---------------| +| Requests/sec (per core) | 100,000+ | +| P99 latency overhead | < 1ms | +| Memory per connection | ~10KB | +| Connection reuse rate | > 95% | + +## Dependencies + +Zentinel uses these Pingora crates: + +```toml +[dependencies] +pingora = { version = "0.7", features = ["proxy", "lb"] } +pingora-core = "0.7" +pingora-http = "0.7" +pingora-proxy = "0.7" +pingora-load-balancing = "0.7" +pingora-timeout = "0.7" +``` + +> **Note:** Zentinel uses a fork (`raskell-io/pingora`) that disables the prometheus protobuf default feature to remove the RUSTSEC-2024-0437 vulnerability. The fork tracks upstream Pingora 0.7 with this single change. + +## Next Steps + +- [Architecture Overview](../architecture/) - High-level design +- [Component Design](../components/) - Zentinel's crate structure +- [Request Flow](../request-flow/) - Detailed request lifecycle diff --git a/content/v/26.04/concepts/request-flow.md b/content/v/26.04/concepts/request-flow.md new file mode 100644 index 0000000..b031d72 --- /dev/null +++ b/content/v/26.04/concepts/request-flow.md @@ -0,0 +1,727 @@ ++++ +title = "Request Lifecycle" +weight = 5 +updated = 2026-02-19 ++++ + +This page details the complete lifecycle of an HTTP request through Zentinel, from client connection to response delivery. + +## Overview + +``` +┌────────┐ ┌──────────┐ +│ Client │ │ Upstream │ +└───┬────┘ └────┬─────┘ + │ │ + │ 1. TCP Connect │ + │────────────────────▶┌─────────────────────────────────┐ │ + │ │ │ │ + │ 2. TLS Handshake │ Zentinel Proxy │ │ + │────────────────────▶│ │ │ + │ │ ┌───────────────────────────┐ │ │ + │ 3. HTTP Request │ │ Request Pipeline │ │ │ + │────────────────────▶│ │ │ │ │ + │ │ │ Parse → Route → Filter │ │ │ + │ │ │ → Agents → Forward │ │ 4. Forward │ + │ │ └───────────────────────────┘ │────────────▶│ + │ │ │ │ + │ │ ┌───────────────────────────┐ │ 5. Response│ + │ │ │ Response Pipeline │ │◀────────────│ + │ 6. HTTP Response │ │ │ │ │ + │◀────────────────────│ │ Filter → Headers → Send │ │ │ + │ │ └───────────────────────────┘ │ │ + │ │ │ │ + │ └─────────────────────────────────┘ │ + │ │ +``` + +## Phase 1: Connection Establishment + +### TCP Accept + +When a client connects, Pingora's listener accepts the TCP connection: + +``` +Client Zentinel + │ │ + │──── TCP SYN ─────────────────▶│ + │◀─── TCP SYN-ACK ──────────────│ + │──── TCP ACK ─────────────────▶│ + │ │ + │ Connection established │ +``` + +**What happens:** +1. Pingora accepts connection from the listener socket +2. Connection is assigned to a worker thread +3. Client address is captured for logging and rate limiting + +### TLS Handshake (HTTPS only) + +For HTTPS listeners, TLS negotiation occurs: + +``` +Client Zentinel + │ │ + │──── ClientHello ─────────────▶│ Supported ciphers, SNI + │◀─── ServerHello ──────────────│ Selected cipher, certificate + │──── Key Exchange ────────────▶│ + │◀─── Finished ─────────────────│ + │ │ + │ TLS session established │ +``` + +**Configuration impact:** +- TLS versions allowed (1.2, 1.3) +- Cipher suite selection +- Certificate chain validation +- SNI-based certificate selection + +## Phase 2: Request Reception + +### HTTP Parsing + +Zentinel parses the incoming HTTP request: + +``` +┌─────────────────────────────────────────────────────────────┐ +│ HTTP Request │ +├─────────────────────────────────────────────────────────────┤ +│ POST /api/users HTTP/1.1 ◀── Request Line│ +│ Host: api.example.com ◀── Headers │ +│ Content-Type: application/json │ +│ Authorization: Bearer eyJ... │ +│ X-Request-Id: abc-123 │ +│ │ +│ {"name": "Alice", "email": "alice@..."} ◀── Body │ +└─────────────────────────────────────────────────────────────┘ +``` + +**Extracted information:** +- Method (GET, POST, etc.) +- Path and query string +- Host header +- All request headers +- Content-Length or Transfer-Encoding + +### Limit Enforcement + +Before processing, hard limits are checked: + +```rust +// Header count limit +if headers.len() > config.limits.max_header_count { + return Error::TooManyHeaders; // 400 Bad Request +} + +// Header size limit +let total_size: usize = headers.iter() + .map(|(k, v)| k.len() + v.len()) + .sum(); + +if total_size > config.limits.max_header_size_bytes { + return Error::HeadersTooLarge; // 431 Request Header Fields Too Large +} +``` + +| Limit | Default | Purpose | +|-------|---------|---------| +| `max_header_count` | 100 | Prevent header flooding | +| `max_header_size_bytes` | 8KB | Prevent memory exhaustion | +| `max_body_size_bytes` | 10MB | Prevent large payload attacks | + +### Trace ID Assignment + +Every request gets a correlation ID for distributed tracing: + +``` +┌──────────────────────────────────────────────────────────────┐ +│ Trace ID Sources │ +├──────────────────────────────────────────────────────────────┤ +│ 1. Incoming header (X-Request-Id, X-Correlation-Id) │ +│ └─▶ Reuse existing ID from upstream services │ +│ │ +│ 2. Generate new ID if not present │ +│ ├─▶ UUID v4: 550e8400-e29b-41d4-a716-446655440000 │ +│ └─▶ UUID v7: 018f6b1c-8a1d-7000-8000-000000000000 │ +└──────────────────────────────────────────────────────────────┘ +``` + +The trace ID propagates through: +- Request headers to upstream +- Response headers to client +- All log entries +- Metrics labels +- Agent requests + +## Phase 3: Route Matching + +### Route Selection + +Zentinel matches the request against compiled routes: + +``` +Request: POST /api/users/123/profile + Host: api.example.com + + │ + ▼ + ┌──────────────────┐ + │ Compiled Routes │ + │ (sorted by │ + │ priority) │ + └────────┬─────────┘ + │ + ┌─────────────┼─────────────┐ + │ │ │ + ▼ ▼ ▼ +┌────────┐ ┌────────┐ ┌────────┐ +│Route A │ │Route B │ │Route C │ +│pri: 100│ │pri: 50 │ │pri: 10 │ +│ │ │ │ │ │ +│ path: │ │ path: │ │ path: │ +│ /api/* │ │ /api/ │ │ /* │ +│ │ │ users/*│ │ │ +└────────┘ └────────┘ └────────┘ + │ │ + │ ✓ MATCH (more specific) + │ + ✓ MATCH (lower priority) + +Winner: Route B (highest priority match) +``` + +### Match Criteria + +Routes can match on multiple criteria: + +| Criteria | Example | Evaluation | +|----------|---------|------------| +| **Path exact** | `/api/health` | String equality | +| **Path prefix** | `/api/` | Starts with | +| **Path regex** | `/users/\d+` | Regex match | +| **Host** | `api.example.com` | Host header match | +| **Method** | `GET`, `POST` | Method in list | +| **Header** | `X-Api-Version: 2` | Header exists/equals | +| **Query param** | `?version=2` | Param exists/equals | + +### No Route Found + +If no route matches: + +``` +┌─────────────────────────────────────────┐ +│ No Matching Route │ +├─────────────────────────────────────────┤ +│ Status: 404 Not Found │ +│ │ +│ Response: │ +│ { │ +│ "error": "no_route", │ +│ "message": "No route matched", │ +│ "path": "/unknown/path", │ +│ "trace_id": "abc-123" │ +│ } │ +└─────────────────────────────────────────┘ +``` + +## Phase 4: Service Type Handling + +Based on the matched route's service type, different handlers take over: + +``` + Route Matched + │ + ┌──────────────┼──────────────┐ + │ │ │ + ▼ ▼ ▼ + ┌──────────┐ ┌──────────┐ ┌──────────┐ + │ Static │ │ Builtin │ │ Proxy │ + │ Files │ │ Handlers │ │ (Web/API)│ + └────┬─────┘ └────┬─────┘ └────┬─────┘ + │ │ │ + ▼ ▼ ▼ + Serve from Handle Continue to + filesystem internally agent processing +``` + +### Static File Serving + +For `service_type = "static"`: + +``` +Request: GET /assets/logo.png + + │ + ▼ +┌────────────────────────┐ +│ Resolve file path │ +│ root + request_path │ +└───────────┬────────────┘ + │ + ▼ +┌────────────────────────┐ +│ Security checks: │ +│ • Path traversal │ +│ • Symlink validation │ +│ • Extension allowlist │ +└───────────┬────────────┘ + │ + ┌───────┴───────┐ + │ │ + ▼ ▼ + Found Not Found + │ │ + ▼ ▼ + Stream file Try index.html + with correct or return 404 + Content-Type +``` + +### Builtin Handlers + +For `service_type = "builtin"`: + +| Handler | Path | Response | +|---------|------|----------| +| `health` | `/-/health` | `{"status": "healthy"}` | +| `ready` | `/-/ready` | `{"status": "ready"}` | +| `metrics` | `/-/metrics` | Prometheus metrics | +| `version` | `/-/version` | Build info | + +## Phase 5: Agent Processing + +For routes with configured agents, external processing occurs: + +``` + Request + │ + ▼ + ┌────────────────┐ + │ Agent Manager │ + └───────┬────────┘ + │ + ┌──────────────┼──────────────┐ + │ │ │ + ▼ ▼ ▼ + ┌─────────┐ ┌─────────┐ ┌─────────┐ + │ Auth │ │ WAF │ │ Rate │ + │ Agent │ │ Agent │ │ Limit │ + └────┬────┘ └────┬────┘ └────┬────┘ + │ │ │ + ▼ ▼ ▼ + Decision Decision Decision + │ │ │ + └──────────────┼──────────────┘ + │ + ▼ + ┌────────────────┐ + │ Aggregate │ + │ Decisions │ + └───────┬────────┘ + │ + ┌────────────┼────────────┐ + │ │ │ + ▼ ▼ ▼ + ALLOW BLOCK REDIRECT + │ │ │ + ▼ ▼ ▼ + Continue Return Return + to upstream error redirect +``` + +### Agent Request + +```json +{ + "event_type": "request_headers", + "correlation_id": "abc-123", + "request_id": "req-456", + "metadata": { + "client_ip": "192.168.1.100", + "client_port": 54321, + "method": "POST", + "path": "/api/users", + "host": "api.example.com" + }, + "headers": [ + {"name": "content-type", "value": "application/json"}, + {"name": "authorization", "value": "Bearer eyJ..."} + ] +} +``` + +### Agent Response + +```json +{ + "decision": "allow", + "header_mutations": { + "request": { + "set": {"X-User-Id": "user-789"}, + "remove": ["Authorization"] + }, + "response": { + "set": {"X-RateLimit-Remaining": "99"} + } + }, + "metadata": { + "auth_method": "jwt", + "user_role": "admin" + }, + "audit": { + "rules_matched": ["auth-jwt-valid"], + "processing_time_us": 1234 + } +} +``` + +### Timeout and Failure Handling + +``` +Agent call started + │ + ├─── timeout_ms exceeded ───▶ Timeout! + │ │ + │ ┌───────┴───────┐ + │ │ │ + ▼ ▼ ▼ + Response fail-closed fail-open + received │ │ + │ ▼ ▼ + │ Block request Allow request + │ (503 error) (continue) + │ + ▼ + Process decision +``` + +## Phase 6: Upstream Selection + +### Load Balancing + +Zentinel selects a backend server from the upstream pool: + +``` +Upstream Pool: "backend" +┌─────────────────────────────────────────────────────────────┐ +│ │ +│ ┌─────────────┐ ┌─────────────┐ ┌─────────────┐ │ +│ │ Server A │ │ Server B │ │ Server C │ │ +│ │ 10.0.0.1:80 │ │ 10.0.0.2:80 │ │ 10.0.0.3:80 │ │ +│ │ │ │ │ │ │ │ +│ │ weight: 5 │ │ weight: 3 │ │ weight: 2 │ │ +│ │ healthy: ✓ │ │ healthy: ✓ │ │ healthy: ✗ │ │ +│ └─────────────┘ └─────────────┘ └─────────────┘ │ +│ │ │ │ +│ └────────────────┘ │ +│ │ │ +│ ▼ │ +│ Load Balancer (round_robin) │ +│ │ │ +│ ▼ │ +│ Selected: Server A │ +│ │ +└─────────────────────────────────────────────────────────────┘ +``` + +### Health Filtering + +Unhealthy servers are excluded: + +| Check Type | Mechanism | Action | +|------------|-----------|--------| +| **Active** | Periodic HTTP probes | Mark unhealthy after N failures | +| **Passive** | Real traffic errors | Mark unhealthy on connection failures | +| **Circuit Breaker** | Error rate threshold | Temporarily exclude | + +### Connection Pooling + +Zentinel reuses connections to upstreams: + +``` +┌────────────────────────────────────────┐ +│ Connection Pool │ +│ ┌──────────────────────────────────┐ │ +│ │ Idle Connections │ │ +│ │ ┌────┐ ┌────┐ ┌────┐ │ │ +│ │ │Conn│ │Conn│ │Conn│ │ │ +│ │ │ #1 │ │ #2 │ │ #3 │ │ │ +│ │ └────┘ └────┘ └────┘ │ │ +│ └──────────────────────────────────┘ │ +│ │ +│ Request arrives: │ +│ 1. Check for idle connection │ +│ 2. If available, reuse │ +│ 3. If not, create new (up to limit) │ +│ 4. If at limit, queue or reject │ +└────────────────────────────────────────┘ +``` + +## Phase 7: Upstream Communication + +### Request Forwarding + +The request is sent to the selected upstream: + +``` +Original Request Modified Request (to upstream) +┌──────────────────┐ ┌──────────────────────────────┐ +│ POST /api/users │ │ POST /api/users HTTP/1.1 │ +│ Host: api.ex.com │ ──▶ │ Host: api.example.com │ +│ Auth: Bearer ... │ │ X-Correlation-Id: abc-123 │ +└──────────────────┘ │ X-Forwarded-For: 192.168.1.1 │ + │ X-Forwarded-Proto: https │ + │ X-Forwarded-By: Zentinel │ + │ X-User-Id: user-789 │ ◀── From agent + │ Content-Type: application/json│ + │ │ + │ {"name": "Alice", ...} │ + └──────────────────────────────┘ +``` + +### Added Headers + +| Header | Value | Purpose | +|--------|-------|---------| +| `X-Correlation-Id` | Trace ID | Distributed tracing | +| `X-Forwarded-For` | Client IP | Original client address | +| `X-Forwarded-Proto` | `http`/`https` | Original protocol | +| `X-Forwarded-Host` | Original host | Original Host header | +| `X-Forwarded-By` | `Zentinel` | Proxy identification | + +### Retry Logic + +On upstream failure, retries may occur: + +``` +Attempt 1: Server A + │ + ├─── Success ───▶ Continue to response + │ + ├─── Failure (connection refused) + │ │ + │ ▼ + │ Wait (exponential backoff) + │ 100ms × 2^(attempt-1) + │ │ + │ ▼ +Attempt 2: Server B (different server) + │ + ├─── Success ───▶ Continue to response + │ + ├─── Failure + │ │ + │ ▼ +Attempt 3: Server A (back to healthy server) + │ + └─── Final failure ───▶ Return 502/504 +``` + +**Retry configuration:** + +```kdl +routes { + route "api" { + retry-policy { + max-attempts 3 + retry-on "connection_error" "5xx" + backoff-ms 100 + } + } +} +``` + +## Phase 8: Response Processing + +### Upstream Response Received + +``` +┌─────────────────────────────────────────────────────────────┐ +│ Upstream Response │ +├─────────────────────────────────────────────────────────────┤ +│ HTTP/1.1 200 OK ◀── Status Line │ +│ Content-Type: application/json ◀── Headers │ +│ X-Request-Id: upstream-456 │ +│ Cache-Control: no-cache │ +│ │ +│ {"id": 123, "name": "Alice", ...} ◀── Body │ +└─────────────────────────────────────────────────────────────┘ +``` + +### Response Filter + +Zentinel processes the response before sending to client: + +```rust +async fn response_filter(&self, upstream_response: &mut ResponseHeader) { + // 1. Add security headers + upstream_response.insert_header("X-Content-Type-Options", "nosniff"); + upstream_response.insert_header("X-Frame-Options", "DENY"); + upstream_response.insert_header("X-XSS-Protection", "1; mode=block"); + upstream_response.insert_header("Referrer-Policy", "strict-origin-when-cross-origin"); + + // 2. Remove server identification + upstream_response.remove_header("Server"); + upstream_response.remove_header("X-Powered-By"); + + // 3. Add correlation ID + upstream_response.insert_header("X-Correlation-Id", &ctx.trace_id); + + // 4. Apply agent response mutations + for (name, value) in agent_response.header_mutations.response.set { + upstream_response.insert_header(&name, &value); + } +} +``` + +### Security Headers Added + +| Header | Value | Protection | +|--------|-------|------------| +| `X-Content-Type-Options` | `nosniff` | MIME type sniffing | +| `X-Frame-Options` | `DENY` | Clickjacking | +| `X-XSS-Protection` | `1; mode=block` | XSS attacks | +| `Referrer-Policy` | `strict-origin-when-cross-origin` | Referrer leakage | + +### Headers Removed + +| Header | Reason | +|--------|--------| +| `Server` | Hide upstream technology | +| `X-Powered-By` | Hide framework information | + +## Phase 9: Response Delivery + +### Streaming to Client + +Responses are streamed as they arrive: + +``` +Upstream Zentinel Client + │ │ │ + │── Headers ───────────────▶│ │ + │ │── Headers ───────────────▶│ + │ │ │ + │── Body chunk 1 ──────────▶│ │ + │ │── Body chunk 1 ──────────▶│ + │ │ │ + │── Body chunk 2 ──────────▶│ │ + │ │── Body chunk 2 ──────────▶│ + │ │ │ + │── Body chunk N (final) ──▶│ │ + │ │── Body chunk N (final) ──▶│ + │ │ │ +``` + +This streaming approach: +- Minimizes memory usage (no full buffering) +- Reduces time-to-first-byte (TTFB) +- Handles large responses efficiently + +### Error Responses + +When errors occur, Zentinel generates appropriate responses: + +| Condition | Status | Response | +|-----------|--------|----------| +| No route matched | 404 | Not Found | +| Agent blocked | 403 | Forbidden | +| Agent redirect | 302/307 | Redirect | +| Upstream timeout | 504 | Gateway Timeout | +| Upstream refused | 502 | Bad Gateway | +| All upstreams down | 503 | Service Unavailable | +| Rate limited | 429 | Too Many Requests | + +## Phase 10: Logging and Metrics + +### Access Log Entry + +After the response is sent: + +```json +{ + "timestamp": "2024-01-15T10:30:45.123Z", + "trace_id": "abc-123", + "instance_id": "zentinel-pod-xyz", + "client_ip": "192.168.1.100", + "method": "POST", + "path": "/api/users", + "query": "version=2", + "host": "api.example.com", + "status": 200, + "body_bytes": 1234, + "duration_ms": 45, + "route_id": "api-users", + "upstream": "backend", + "upstream_attempts": 1, + "user_agent": "Mozilla/5.0...", + "referer": "https://example.com/" +} +``` + +### Metrics Updated + +``` +# Request counter +zentinel_requests_total{route="api-users",method="POST",status="200"} 1 + +# Latency histogram +zentinel_request_duration_seconds_bucket{route="api-users",le="0.05"} 1 +zentinel_request_duration_seconds_bucket{route="api-users",le="0.1"} 1 + +# Upstream metrics +zentinel_upstream_requests_total{upstream="backend",status="200"} 1 +zentinel_upstream_latency_seconds_bucket{upstream="backend",le="0.05"} 1 + +# Agent metrics +zentinel_agent_requests_total{agent="auth-agent",decision="allow"} 1 +zentinel_agent_latency_seconds_bucket{agent="auth-agent",le="0.01"} 1 +``` + +### Request Complete + +Finally, the reload coordinator is notified: + +```rust +// In logging() callback +self.reload_coordinator.dec_requests(); +``` + +This enables graceful shutdown - Zentinel waits for all in-flight requests to complete before stopping. + +## Complete Timeline + +``` +Time Event +───── ───────────────────────────────────────────────── +0ms TCP connection accepted +2ms TLS handshake complete (HTTPS) +3ms HTTP request headers received +3ms Trace ID assigned: abc-123 +4ms Limits checked (headers count, size) +4ms Route matched: api-users +5ms Agent: auth-agent called +15ms Agent: auth-agent responded (ALLOW) +16ms Agent: waf-agent called +25ms Agent: waf-agent responded (ALLOW) +26ms Upstream selected: backend-1 (10.0.0.1:80) +27ms Connection acquired from pool +28ms Request forwarded to upstream +65ms Upstream response headers received +66ms Security headers added +66ms Response headers sent to client +70ms Response body streamed +75ms Response complete +75ms Access log written +75ms Metrics updated +75ms Request counter decremented +───── ───────────────────────────────────────────────── +Total: 75ms (client perspective) +``` + +## Next Steps + +- [Routing System](../routing/) - Deep dive into route matching +- [Pingora Foundation](../pingora/) - Underlying framework +- [Agents](/agents/) - External processing details diff --git a/content/v/26.04/concepts/routing.md b/content/v/26.04/concepts/routing.md new file mode 100644 index 0000000..0f50f6f --- /dev/null +++ b/content/v/26.04/concepts/routing.md @@ -0,0 +1,681 @@ ++++ +title = "Routing System" +weight = 6 +updated = 2026-02-19 ++++ + +Zentinel's routing system determines how incoming requests are matched to configured routes. This page covers match conditions, priority rules, and performance optimizations. + +## Overview + +``` + Incoming Request + │ + ▼ + ┌─────────────────┐ + │ Route Matcher │ + │ │ + │ ┌───────────┐ │ + │ │ Cache │◀─┼── Cache hit? Return immediately + │ └───────────┘ │ + │ │ │ + │ ▼ │ + │ ┌───────────┐ │ + │ │ Compiled │ │ + │ │ Routes │ │ + │ │ (sorted) │ │ + │ └───────────┘ │ + └────────┬────────┘ + │ + ┌────────────┼────────────┐ + │ │ │ + ▼ ▼ ▼ + Route A Route B Route C + (pri:100) (pri:50) (pri:10) + │ │ │ + │ ✓ First Match │ + │ │ │ + └────────────┴────────────┘ + │ + ▼ + Return RouteMatch +``` + +## Route Configuration + +Routes are defined in your configuration file: + +```kdl +routes { + route "api-users" { + priority 100 + + matches { + path-prefix "/api/users" + method "GET" "POST" "PUT" "DELETE" + } + + upstream "user-service" + service-type "api" + } + + route "static-assets" { + priority 50 + + matches { + path-prefix "/static/" + } + + service-type "static" + static-files { + root "/var/www/static" + } + } + + route "catch-all" { + priority 1 + + matches { + path-prefix "/" + } + + upstream "default-backend" + } +} +``` + +## Match Conditions + +Routes can match on multiple criteria. All conditions must match (AND logic). + +### Path Matching + +#### Exact Path + +Matches the exact path string: + +```kdl +matches { + path "/api/health" +} +``` + +| Request Path | Match? | +|--------------|--------| +| `/api/health` | Yes | +| `/api/health/` | No | +| `/api/healthcheck` | No | + +#### Path Prefix + +Matches paths starting with the prefix: + +```kdl +matches { + path-prefix "/api/" +} +``` + +| Request Path | Match? | +|--------------|--------| +| `/api/` | Yes | +| `/api/users` | Yes | +| `/api/users/123` | Yes | +| `/apiv2/users` | No | + +#### Path Regex + +Matches paths against a regular expression: + +```kdl +matches { + path-regex "/users/[0-9]+/profile" +} +``` + +| Request Path | Match? | +|--------------|--------| +| `/users/123/profile` | Yes | +| `/users/456/profile` | Yes | +| `/users/abc/profile` | No | + +**Common regex patterns:** + +| Pattern | Description | +|---------|-------------| +| `/api/v[0-9]+/.*` | Versioned API paths | +| `/users/[0-9]+` | Numeric user IDs | +| `/.*/health` | Health endpoints at any level | +| `/[a-z]{2}/.*` | Two-letter locale prefix | + +### Host Matching + +Match based on the `Host` header: + +#### Exact Host + +```kdl +matches { + host "api.example.com" +} +``` + +#### Wildcard Host + +Matches subdomains: + +```kdl +matches { + host "*.example.com" +} +``` + +| Host Header | Match? | +|-------------|--------| +| `api.example.com` | Yes | +| `www.example.com` | Yes | +| `example.com` | No | +| `deep.sub.example.com` | No (single level only) | + +#### Host Regex + +For complex host patterns: + +```kdl +matches { + host-regex "^(api|www)\\.example\\.(com|io)$" +} +``` + +### Method Matching + +Match specific HTTP methods: + +```kdl +matches { + method "GET" "POST" +} +``` + +| Request Method | Match? | +|----------------|--------| +| `GET` | Yes | +| `POST` | Yes | +| `PUT` | No | +| `DELETE` | No | + +### Header Matching + +#### Header Presence + +Match if header exists (any value): + +```kdl +matches { + header "Authorization" +} +``` + +#### Header Value + +Match if header has specific value: + +```kdl +matches { + header "X-Api-Version" value="2" +} +``` + +| Headers | Match? | +|---------|--------| +| `X-Api-Version: 2` | Yes | +| `X-Api-Version: 1` | No | +| (no header) | No | + +### Query Parameter Matching + +#### Parameter Presence + +```kdl +matches { + query-param "debug" +} +``` + +| URL | Match? | +|-----|--------| +| `/api?debug=true` | Yes | +| `/api?debug=` | Yes | +| `/api?debug` | Yes | +| `/api?other=value` | No | + +#### Parameter Value + +```kdl +matches { + query-param "version" value="2" +} +``` + +| URL | Match? | +|-----|--------| +| `/api?version=2` | Yes | +| `/api?version=1` | No | + +## Combining Conditions + +Multiple conditions are combined with AND logic: + +```kdl +route "admin-api" { + matches { + path-prefix "/admin/" + method "GET" "POST" + header "X-Admin-Token" + host "admin.example.com" + } + upstream "admin-service" +} +``` + +This route only matches if: +- Path starts with `/admin/` **AND** +- Method is GET or POST **AND** +- `X-Admin-Token` header is present **AND** +- Host is `admin.example.com` + +## Priority System + +When multiple routes could match, priority determines the winner. + +### Priority Levels + +```kdl +route "high-priority" { + priority 100 // Evaluated first +} + +route "normal-priority" { + priority 50 // Evaluated second +} + +route "low-priority" { + priority 10 // Evaluated last +} +``` + +Higher numbers = higher priority = evaluated first. + +### Named Priority Levels + +You can also use named levels: + +| Name | Numeric Value | +|------|---------------| +| `critical` | 1000 | +| `high` | 100 | +| `normal` | 50 (default) | +| `low` | 10 | +| `background` | 1 | + +```kdl +route "critical-health" { + priority critical + matches { path "/-/health" } +} +``` + +### Priority Example + +``` +Request: GET /api/users/123 + Host: api.example.com + +Routes evaluated in order: +┌────────────────────────────────────────────────────────────┐ +│ 1. route "api-user-detail" pri=100 │ +│ matches: path-regex "/api/users/[0-9]+" │ +│ Result: ✓ MATCH → Selected! │ +├────────────────────────────────────────────────────────────┤ +│ 2. route "api-users" pri=80 │ +│ matches: path-prefix "/api/users" │ +│ Result: (not evaluated - already matched) │ +├────────────────────────────────────────────────────────────┤ +│ 3. route "api-catchall" pri=50 │ +│ matches: path-prefix "/api/" │ +│ Result: (not evaluated - already matched) │ +└────────────────────────────────────────────────────────────┘ +``` + +## Specificity Tie-Breaking + +When routes have the same priority, specificity breaks ties: + +``` +Specificity Scores: +┌─────────────────────────────────────┐ +│ Match Type │ Score │ +├─────────────────────┼───────────────┤ +│ Exact path │ 1000 │ +│ Path regex │ 500 │ +│ Path prefix │ 100 │ +│ Host │ 50 │ +│ Header (with value) │ 30 │ +│ Header (presence) │ 20 │ +│ Query param (value) │ 25 │ +│ Query param (pres.) │ 15 │ +│ Method │ 10 │ +└─────────────────────────────────────┘ +``` + +**Example:** + +```kdl +// Both have priority 50 + +route "specific" { + priority 50 + matches { + path "/api/users" // +1000 + method "GET" // +10 + } + // Total specificity: 1010 +} + +route "general" { + priority 50 + matches { + path-prefix "/api/" // +100 + } + // Total specificity: 100 +} +``` + +For request `GET /api/users`, the "specific" route wins due to higher specificity. + +## Route Compilation + +Routes are compiled at startup for efficient matching: + +``` +Configuration Compiled +┌──────────────────┐ ┌──────────────────────────┐ +│ route "api" { │ │ CompiledRoute { │ +│ matches { │ ────▶ │ id: "api", │ +│ path-prefix │ │ priority: 50, │ +│ "/api/" │ │ matchers: [ │ +│ method │ │ PathPrefix("/api/"), │ +│ "GET" │ │ Method(["GET"]), │ +│ } │ │ ], │ +│ } │ │ specificity: 110, │ +└──────────────────┘ │ } │ + └──────────────────────────┘ +``` + +**What's compiled:** +- Regex patterns are pre-compiled +- Host wildcards are parsed +- Routes are sorted by priority +- Specificity scores are calculated + +## Route Cache + +Zentinel caches route matches for performance: + +``` +┌───────────────────────────────────────────────────────────┐ +│ Route Cache │ +│ ┌─────────────────────────────────────────────────────┐ │ +│ │ Cache Key: "{method}:{host}:{path}" │ │ +│ │ │ │ +│ │ "GET:api.example.com:/users/123" → route-id: "api" │ │ +│ │ "POST:api.example.com:/login" → route-id: "auth" │ │ +│ │ "GET:www.example.com:/about" → route-id: "web" │ │ +│ └─────────────────────────────────────────────────────┘ │ +│ │ +│ Max Size: 1000 entries │ +│ Eviction: LRU (Least Recently Used) │ +│ │ +└───────────────────────────────────────────────────────────┘ +``` + +**Cache behavior:** + +1. **Cache hit**: Return route immediately (no evaluation) +2. **Cache miss**: Evaluate all routes, cache the result +3. **Eviction**: When full, remove least recently used entries +4. **Invalidation**: Cache clears on configuration reload + +### When Caching Doesn't Help + +Cache is bypassed when requests vary significantly: +- Random query parameters in cache key +- Unique paths (e.g., UUIDs in path) +- Many different hosts + +## Default Route + +Configure a catch-all route for unmatched requests: + +```kdl +routes { + // Specific routes first + route "api" { + priority 100 + matches { path-prefix "/api/" } + upstream "api-service" + } + + // Default route (lowest priority) + route "default" { + priority 1 + matches { path-prefix "/" } + upstream "default-backend" + } +} +``` + +Or specify a default route explicitly: + +```kdl +routing { + default-route "fallback" +} + +routes { + route "fallback" { + upstream "default-backend" + } +} +``` + +## No Match Behavior + +When no route matches and no default is configured: + +```json +{ + "status": 404, + "error": "no_route", + "message": "No route matched request", + "path": "/unknown/path", + "trace_id": "abc-123" +} +``` + +## Best Practices + +### 1. Order by Specificity + +Put more specific routes before general ones: + +```kdl +// Good: Specific first +route "user-profile" { priority 100; matches { path-regex "/users/[0-9]+/profile" } } +route "user-detail" { priority 90; matches { path-regex "/users/[0-9]+" } } +route "users-list" { priority 80; matches { path-prefix "/users" } } +route "api-catchall" { priority 50; matches { path-prefix "/api/" } } +``` + +### 2. Use Exact Paths When Possible + +Exact paths are faster and more predictable: + +```kdl +// Prefer this for known endpoints +route "health" { + matches { path "/-/health" } +} + +// Over regex for simple cases +route "health" { + matches { path-regex "^/-/health$" } // Slower +} +``` + +### 3. Limit Regex Complexity + +Simple regexes match faster: + +```kdl +// Fast +matches { path-regex "/users/[0-9]+" } + +// Slower (backtracking) +matches { path-regex "/users/.*?/profile/.*" } +``` + +### 4. Use Priority Gaps + +Leave gaps for future routes: + +```kdl +system { + worker-threads 0 +} + +listeners { + listener "http" { + address "0.0.0.0:8080" + protocol "http" + } +} + +// Example priority values with gaps for future insertion +routes { + route "critical" { + priority 1000 + matches { path-prefix "/critical" } + upstream "backend" + } + route "high" { + priority 100 // Gap allows 101-999 + matches { path-prefix "/high" } + upstream "backend" + } + route "normal" { + priority 50 // Gap allows 51-99 + matches { path-prefix "/normal" } + upstream "backend" + } + route "low" { + priority 10 // Gap allows 11-49 + matches { path-prefix "/low" } + upstream "backend" + } + route "default" { + priority 1 + matches { path-prefix "/" } + upstream "backend" + } +} + +upstreams { + upstream "backend" { + targets { + target { address "127.0.0.1:3000" } + } + } +} +``` + +### 5. Avoid Overlapping Routes + +Overlapping routes with same priority cause confusion: + +```kdl +// Avoid: Both could match /api/users, same priority +route "api-a" { priority 50; matches { path-prefix "/api/" } } +route "api-b" { priority 50; matches { path-prefix "/api/users" } } + +// Better: Different priorities +route "api-users" { priority 60; matches { path-prefix "/api/users" } } +route "api-other" { priority 50; matches { path-prefix "/api/" } } +``` + +## Debugging Routes + +### Test Route Matching + +Use the CLI to test which route matches: + +```bash +zentinel route-test --path "/api/users/123" --method GET --host api.example.com +``` + +Output: +``` +Matched route: api-users + Priority: 100 + Specificity: 610 + Upstream: user-service + +Evaluated routes: + 1. api-users (pri=100, spec=610) ✓ MATCHED + 2. api-catchall (pri=50, spec=100) - (not evaluated) + 3. default (pri=1, spec=100) - (not evaluated) +``` + +### View Compiled Routes + +```bash +zentinel routes --compiled +``` + +### Monitor Cache Performance + +```bash +zentinel stats routes +``` + +``` +Route Cache Statistics: + Entries: 847/1000 + Hit Rate: 94.2% + Evictions: 12,453 + +Top Cached Routes: + 1. api-users: 45,231 hits + 2. static-assets: 23,456 hits + 3. health: 12,345 hits +``` + +## Performance Characteristics + +| Operation | Complexity | Typical Time | +|-----------|------------|--------------| +| Cache lookup | O(1) | < 1μs | +| Route evaluation (no cache) | O(n) | 10-100μs | +| Regex match | O(m) | 1-10μs per regex | +| Cache insert | O(1) | < 1μs | +| LRU eviction | O(1) | < 1μs | + +Where: +- n = number of routes +- m = path length + +## Next Steps + +- [Request Lifecycle](../request-flow/) - See routing in context +- [Basic Configuration](/getting-started/basic-configuration/) - Configuration syntax +- [First Route](/getting-started/first-route/) - Getting started with routes diff --git a/content/v/26.04/configuration/_index.md b/content/v/26.04/configuration/_index.md new file mode 100644 index 0000000..e5243ed --- /dev/null +++ b/content/v/26.04/configuration/_index.md @@ -0,0 +1,133 @@ ++++ +title = "Configuration" +weight = 3 +sort_by = "weight" +template = "section.html" ++++ + +Zentinel uses KDL (a human-friendly document language) for configuration. This section covers all configuration options organized by component. + +## Configuration Blocks + +| Block | Purpose | +|-------|---------| +| [File Format](file-format/) | KDL syntax and file structure | +| [Server](server/) | Worker threads, process management, shutdown | +| [Listeners](listeners/) | Network binding, TLS, SNI, HTTP/2 | +| [Routes](routes/) | Request matching and routing rules | +| [Upstreams](upstreams/) | Backend pools, load balancing, health checks | +| [Limits](limits/) | Request limits, rate limiting, memory protection | +| [Filters](filters/) | Rate limiting, CORS, compression, geo-blocking | +| [Caching](cache/) | HTTP response caching configuration | +| [Observability](observability/) | Logging, metrics, and distributed tracing | +| [Agents](agents/) | External processing agent configuration | +| [WAF](waf/) | Web Application Firewall configuration | +| [Namespaces & Services](namespaces/) | Hierarchical organization and runtime isolation | + +## Quick Example + +```kdl +system { + worker-threads 0 + max-connections 10000 + trace-id-format "tinyflake" +} + +listeners { + listener "https" { + address "0.0.0.0:443" + protocol "https" + tls { + cert-file "/etc/zentinel/certs/server.crt" + key-file "/etc/zentinel/certs/server.key" + min-version "1.2" + } + } +} + +routes { + route "api" { + priority 100 + matches { + path-prefix "/api/" + } + upstream "backend" + filters "rate-limit" "cors" + + cache { + enabled #true + default-ttl-secs 60 + } + } +} + +upstreams { + upstream "backend" { + targets { + target { address "10.0.1.1:8080" } + target { address "10.0.1.2:8080" } + } + load-balancing "round_robin" + health-check { + type "http" { + path "/health" + expected-status 200 + } + } + } +} + +filters { + filter "rate-limit" { + type "rate-limit" + max-rps 100 + burst 20 + key "client-ip" + } + + filter "cors" { + type "cors" + allowed-origins "https://example.com" + allowed-methods "GET" "POST" "PUT" "DELETE" + } +} + +cache { + enabled #true + backend "memory" + max-size 104857600 +} + +observability { + logging { + level "info" + format "json" + } + metrics { + enabled #true + address "0.0.0.0:9090" + } +} + +limits { + max-body-size-bytes 10485760 +} +``` + +## Validation + +Always validate configuration before applying: + +```bash +zentinel --config zentinel.kdl --validate +``` + +## Hot Reload + +Reload configuration without restart: + +```bash +kill -HUP $(cat /var/run/zentinel.pid) +# or +curl -X POST http://localhost:9090/admin/reload +``` diff --git a/content/v/26.04/configuration/agents.md b/content/v/26.04/configuration/agents.md new file mode 100644 index 0000000..85aed9e --- /dev/null +++ b/content/v/26.04/configuration/agents.md @@ -0,0 +1,886 @@ ++++ +title = "Agents" +weight = 9 +updated = 2026-02-27 ++++ + +Agents are external processes that extend Zentinel's functionality. They handle security policies, authentication, rate limiting, and custom business logic. The `agents` block configures how Zentinel connects to and communicates with these agents. + +## Basic Configuration + +```kdl +agents { + agent "waf-agent" type="waf" { + unix-socket "/var/run/zentinel/waf.sock" + events "request_headers" "request_body" + timeout-ms 200 + failure-mode "closed" + } + + agent "auth-agent" type="auth" { + grpc "http://localhost:50051" + events "request_headers" + timeout-ms 100 + failure-mode "closed" + circuit-breaker { + failure-threshold 5 + timeout-seconds 30 + } + } +} +``` + +## Agent Types + +| Type | Description | +|------|-------------| +| `waf` | Web Application Firewall | +| `auth` | Authentication and authorization | +| `rate_limit` | Rate limiting decisions | +| `custom` | Custom agent type (specify name) | + +## Transports + +### Unix Socket + +Low-latency local communication: + +```kdl +agent "local-agent" type="auth" { + unix-socket "/var/run/zentinel/agent.sock" + events "request_headers" + timeout-ms 100 +} +``` + +### gRPC + +Remote agent over HTTP/2: + +```kdl +agent "remote-agent" type="waf" { + grpc "http://waf-service:50051" + events "request_headers" "request_body" + timeout-ms 200 +} +``` + +With TLS: + +```kdl +agent "secure-agent" type="auth" { + grpc "https://auth-service:50051" { + ca-cert "/etc/zentinel/ca.crt" + client-cert "/etc/zentinel/client.crt" + client-key "/etc/zentinel/client.key" + } + events "request_headers" +} +``` + +### HTTP + +REST API agent using JSON over HTTP. This is the simplest transport option, making it easy to build agents in any language with HTTP support. + +#### Basic HTTP + +```kdl +agent "http-agent" type="custom" { + http "http://policy-service:8080/agent" + events "request_headers" + timeout-ms 150 +} +``` + +#### HTTPS with TLS + +```kdl +agent "secure-http-agent" type="auth" { + http "https://auth-service:8443/agent" { + ca-cert "/etc/zentinel/certs/ca.crt" + } + events "request_headers" + timeout-ms 100 +} +``` + +#### HTTPS with mTLS + +```kdl +agent "mtls-agent" type="waf" { + http "https://waf-service:8443/agent" { + ca-cert "/etc/zentinel/certs/ca.crt" + client-cert "/etc/zentinel/certs/client.crt" + client-key "/etc/zentinel/certs/client.key" + } + events "request_headers" "request_body" + timeout-ms 200 +} +``` + +#### HTTP Protocol + +Zentinel sends events as JSON POST requests: + +```http +POST /agent HTTP/1.1 +Host: policy-service:8080 +Content-Type: application/json +X-Zentinel-Protocol-Version: 1 + +{ + "version": 1, + "event_type": "request_headers", + "payload": { + "metadata": { + "correlation_id": "abc123", + "request_id": "req-456", + "client_ip": "192.168.1.100", + "route_id": "api-route" + }, + "method": "POST", + "uri": "/api/users", + "headers": { + "content-type": ["application/json"], + "authorization": ["Bearer token..."] + } + } +} +``` + +Agents respond with JSON: + +```json +{ + "version": 1, + "decision": "allow", + "request_headers": [ + {"set": {"name": "X-User-ID", "value": "user-123"}} + ], + "audit": { + "tags": ["authenticated"], + "rule_ids": ["auth-001"] + } +} +``` + +#### When to Use HTTP + +| Use Case | Recommended Transport | +|----------|----------------------| +| Simple agents in any language | HTTP | +| High-throughput, low-latency | Unix Socket | +| Binary protocol, streaming | gRPC | +| Cross-network, load-balanced | HTTP or gRPC | +| Development/prototyping | HTTP | + +HTTP advantages: +- Works with any language/framework that handles HTTP +- Easy to debug with curl or browser tools +- Simple JSON payloads +- Standard load balancers and proxies work out of the box + +HTTP trade-offs: +- Higher overhead than Unix sockets +- No streaming (full request/response per call) +- JSON parsing overhead vs binary protocols + +## Events + +Specify which lifecycle events the agent handles: + +```kdl +agent "waf-agent" type="waf" { + unix-socket "/var/run/zentinel/waf.sock" + events "request_headers" "request_body" "response_headers" +} +``` + +| Event | Description | +|-------|-------------| +| `request_headers` | HTTP request headers received | +| `request_body` | Request body chunks | +| `response_headers` | Upstream response headers received | +| `response_body` | Response body chunks | +| `log` | Request complete (for logging) | +| `websocket_frame` | WebSocket frames (after upgrade) | + +### Request vs Response Phase + +Events are processed in two phases: + +- **Request phase** (`request_headers`, `request_body`): Before the request is forwarded to the upstream. The agent can allow, block, or modify the request. +- **Response phase** (`response_headers`, `response_body`): After the upstream responds. The agent can inspect and modify response headers and body. + +An agent that needs both phases (e.g. image optimization) must subscribe to events from both: + +```kdl +agent "image-optimizer" type="custom" { + unix-socket "/var/run/zentinel/image-opt.sock" + events "request_headers" "response_headers" "response_body" + timeout-ms 5000 + failure-mode "open" +} +``` + +In the request phase, the agent receives the client's `Accept` header to negotiate the output format. In the response phase, it receives the upstream's response headers (to check `Content-Type`) and the response body (to perform the conversion). + +### Response Header Modifications + +When an agent subscribes to `response_headers`, its Decision can include `response_headers` operations that modify headers before they are sent to the client: + +```json +{ + "decision": "allow", + "response_headers": [ + {"set": {"name": "content-type", "value": "image/avif"}}, + {"set": {"name": "vary", "value": "Accept"}}, + {"remove": {"name": "content-length"}} + ], + "needs_more": true +} +``` + +Setting `needs_more: true` tells the proxy that the agent expects to receive subsequent events (e.g. response body chunks). + +### Response Body Mutations + +When an agent subscribes to `response_body`, it receives body chunks (base64-encoded) and can return a `BodyMutation` to replace the body: + +```json +{ + "decision": "allow", + "response_body_mutation": { + "data": "" + } +} +``` + +| `data` value | Behavior | +|--------------|----------| +| `null` / absent | Pass through original body | +| `""` (empty) | Drop the chunk | +| `""` | Replace body with decoded content | + +When response body processing is active, the proxy automatically sets `Connection: close` and removes the original `Content-Length` header since the body size may change. + +## Failure Handling + +### Failure Mode + +Configure behavior when the agent is unavailable: + +```kdl +agent "auth-agent" type="auth" { + failure-mode "closed" // Block requests if agent fails +} + +agent "analytics-agent" type="custom" { + failure-mode "open" // Allow requests if agent fails +} +``` + +| Mode | Behavior | +|------|----------| +| `closed` | Block requests when agent unavailable (security-first) | +| `open` | Allow requests through when agent unavailable (availability-first) | + +### Circuit Breaker + +Prevent cascading failures with circuit breakers: + +```kdl +agent "waf-agent" type="waf" { + circuit-breaker { + failure-threshold 5 // Open after 5 failures + success-threshold 2 // Close after 2 successes + timeout-seconds 30 // Wait before half-open + half-open-max-requests 1 // Requests in half-open state + } +} +``` + +| Option | Default | Description | +|--------|---------|-------------| +| `failure-threshold` | `5` | Failures to open circuit | +| `success-threshold` | `2` | Successes to close circuit | +| `timeout-seconds` | `30` | Seconds before half-open | +| `half-open-max-requests` | `1` | Test requests in half-open | + +Circuit breaker states: +- **Closed**: Normal operation +- **Open**: Agent bypassed (fails immediately) +- **Half-Open**: Testing if agent recovered + +## Timeouts + +```kdl +agent "waf-agent" type="waf" { + timeout-ms 200 // Total call timeout + chunk-timeout-ms 5000 // Per-chunk timeout (streaming) +} +``` + +## Body Processing + +### Body Size Limits + +```kdl +agent "waf-agent" type="waf" { + max-request-body-bytes 10485760 // 10MB + max-response-body-bytes 5242880 // 5MB +} +``` + +### Body Streaming Modes + +Control how bodies are sent to agents: + +```kdl +// Buffer entire body (default) +agent "waf-agent" type="waf" { + request-body-mode "buffer" + response-body-mode "buffer" +} + +// Stream chunks as they arrive +agent "streaming-agent" type="custom" { + request-body-mode "stream" + chunk-timeout-ms 5000 +} + +// Hybrid: buffer small, stream large +agent "hybrid-agent" type="custom" { + request-body-mode "hybrid:65536" // Buffer up to 64KB +} +``` + +| Mode | Description | +|------|-------------| +| `buffer` | Collect entire body before sending (default) | +| `stream` | Send chunks as they arrive | +| `hybrid:` | Buffer up to threshold, then stream | + +## WAF Body Inspection + +For WAF agents that need to inspect request bodies, Zentinel provides a dedicated body inspection pipeline with security controls. + +### WAF Configuration Block + +Configure body inspection globally via the `waf` block: + +```kdl +waf { + body-inspection { + inspect-request-body #true + inspect-response-body #false + max-body-inspection-bytes 1048576 // 1MB + content-types "application/json" "application/x-www-form-urlencoded" "text/xml" + decompress #true + max-decompression-ratio 100.0 + } +} +``` + +### Body Inspection Options + +| Option | Default | Description | +|--------|---------|-------------| +| `inspect-request-body` | `false` | Enable request body inspection | +| `inspect-response-body` | `false` | Enable response body inspection | +| `max-body-inspection-bytes` | `1048576` | Max bytes to buffer for inspection | +| `content-types` | See below | Content types eligible for inspection | +| `decompress` | `false` | Decompress bodies before inspection | +| `max-decompression-ratio` | `100.0` | Max compression ratio (zip bomb protection) | + +Default content types for inspection: +- `application/json` +- `application/x-www-form-urlencoded` +- `text/xml` +- `application/xml` +- `text/plain` + +### Body Decompression + +When `decompress` is enabled, Zentinel automatically decompresses request bodies before sending them to WAF agents. This allows WAF rules to inspect the actual content of compressed payloads. + +**Supported encodings:** +- `gzip` - Most common compression +- `deflate` - Raw deflate compression +- `br` - Brotli compression + +**Security protections:** + +| Protection | Default | Description | +|------------|---------|-------------| +| Max ratio | 100x | Prevents zip bombs (rejects if decompressed/compressed > ratio) | +| Max output size | 10MB | Hard limit on decompressed size | +| Fail mode | Route setting | Uses route's `failure-mode` for decompression errors | + +```kdl +waf { + body-inspection { + decompress #true + max-decompression-ratio 50.0 // Stricter limit for sensitive routes + } +} +``` + +### Decompression Behavior + +| Scenario | fail-open | fail-closed | +|----------|-----------|-------------| +| Decompression succeeds | Inspect decompressed body | Inspect decompressed body | +| Ratio exceeded | Inspect compressed body | Block with 400 | +| Size exceeded | Inspect compressed body | Block with 400 | +| Invalid data | Inspect compressed body | Block with 400 | + +### Metrics + +Decompression operations are tracked via Prometheus metrics: + +| Metric | Labels | Description | +|--------|--------|-------------| +| `zentinel_decompression_total` | `encoding`, `result` | Total decompression operations | +| `zentinel_decompression_ratio` | `encoding` | Histogram of compression ratios | + +Result labels: `success`, `ratio_exceeded`, `size_exceeded`, `invalid_data`, `io_error` + +### Complete WAF Example + +```kdl +waf { + body-inspection { + inspect-request-body #true + max-body-inspection-bytes 1048576 + content-types "application/json" "application/xml" "text/plain" + decompress #true + max-decompression-ratio 100.0 + } +} + +agents { + agent "modsecurity" type="waf" { + unix-socket "/var/run/zentinel/modsec.sock" + events "request_headers" "request_body" + timeout-ms 200 + failure-mode "closed" + request-body-mode "buffer" + circuit-breaker { + failure-threshold 10 + timeout-seconds 60 + } + config { + rules-path "/etc/modsecurity/crs" + paranoia-level 2 + } + } +} + +routes { + route "api" { + matches { path-prefix "/api/" } + upstream "backend" + policies { + failure-mode "closed" // Used for decompression errors + } + filters "waf-filter" + } +} + +filters { + filter "waf-filter" { + type "agent" + agent "modsecurity" + phase "request" + failure-mode "closed" + } +} +``` + +## Agent-Specific Configuration + +Pass configuration to agents via the `config` block: + +```kdl +agent "waf-agent" type="waf" { + unix-socket "/var/run/zentinel/waf.sock" + config { + rules-path "/etc/zentinel/waf-rules" + paranoia-level 2 + block-suspicious #true + } +} +``` + +The configuration is passed to the agent when it connects. + +## Attaching Agents to Routes + +Reference agents in route configuration: + +```kdl +routes { + // Via filters + route "api" { + filters "waf-filter" // Filter referencing agent + } +} + +filters { + filter "waf-filter" { + type "agent" + agent "waf-agent" + phase "request" + timeout-ms 200 + failure-mode "closed" + } +} +``` + +## Complete Examples + +### WAF Agent + +```kdl +agents { + agent "modsecurity" type="waf" { + unix-socket "/var/run/zentinel/modsec.sock" + events "request_headers" "request_body" + timeout-ms 200 + failure-mode "closed" + max-request-body-bytes 1048576 // 1MB for inspection + request-body-mode "buffer" + circuit-breaker { + failure-threshold 10 + timeout-seconds 60 + } + config { + rules-path "/etc/modsecurity/crs" + paranoia-level 1 + } + } +} +``` + +### Authentication Agent + +```kdl +agents { + agent "jwt-auth" type="auth" { + grpc "http://auth-service:50051" + events "request_headers" + timeout-ms 50 + failure-mode "closed" + circuit-breaker { + failure-threshold 5 + timeout-seconds 30 + } + config { + issuer "https://auth.example.com" + audience "api.example.com" + } + } +} +``` + +### Rate Limiting Agent + +```kdl +agents { + agent "rate-limiter" type="rate_limit" { + grpc "http://ratelimit-service:50051" + events "request_headers" + timeout-ms 20 + failure-mode "open" // Allow through if service down + circuit-breaker { + failure-threshold 3 + timeout-seconds 15 + } + } +} +``` + +### Response Processing Agent (Image Optimization) + +```kdl +agents { + agent "image-optimizer" type="custom" { + unix-socket "/var/run/zentinel/image-opt.sock" + events "request_headers" "response_headers" "response_body" + timeout-ms 5000 + failure-mode "open" + response-body-mode "buffer" + max-response-body-bytes 10485760 // 10MB + } +} + +filters { + filter "image-optimization" { + type "agent" + agent "image-optimizer" + timeout-ms 5000 + failure-mode "open" + } +} + +routes { + route "images" { + matches { path-prefix "/images/" } + upstream "backend" + filters "image-optimization" + } +} +``` + +The agent receives `request_headers` to extract the client's `Accept` header, `response_headers` to check the upstream's `Content-Type` and set output headers, and `response_body` to perform the image conversion. + +### Logging/Analytics Agent + +```kdl +agents { + agent "analytics" type="custom" { + http "http://analytics:8080/log" + events "log" // Only receive completion events + timeout-ms 1000 + failure-mode "open" // Don't block requests for logging + } +} +``` + +## TLS Configuration + +Secure gRPC connections to agents with TLS and mutual TLS (mTLS). + +### TLS Options + +| Option | Description | +|--------|-------------| +| `ca-cert` | Path to CA certificate for verifying the agent's server certificate | +| `client-cert` | Path to client certificate for mTLS authentication | +| `client-key` | Path to client private key for mTLS authentication | +| `insecure-skip-verify` | Skip certificate verification (development only) | + +### Server TLS (One-Way) + +Verify the agent's identity using TLS: + +```kdl +agent "secure-agent" type="auth" { + grpc "https://auth-service.internal:50051" { + ca-cert "/etc/zentinel/certs/ca.crt" + } + events "request_headers" + timeout-ms 100 +} +``` + +This configuration: +- Encrypts traffic between Zentinel and the agent +- Verifies the agent's certificate against the provided CA +- Automatically extracts domain name for SNI from the address + +### Mutual TLS (mTLS) + +For bidirectional authentication where both Zentinel and the agent verify each other: + +```kdl +agent "secure-waf" type="waf" { + grpc "https://waf-service.internal:50051" { + ca-cert "/etc/zentinel/certs/ca.crt" + client-cert "/etc/zentinel/certs/zentinel-client.crt" + client-key "/etc/zentinel/certs/zentinel-client.key" + } + events "request_headers" "request_body" + timeout-ms 200 + failure-mode "closed" +} +``` + +This configuration: +- Encrypts traffic with TLS +- Verifies the agent's certificate against the CA +- Presents Zentinel's client certificate to the agent for verification +- Provides strong mutual authentication for security-sensitive agents + +### Using System CA Store + +When no `ca-cert` is specified, Zentinel uses the system's native certificate store for server verification: + +```kdl +agent "public-agent" type="custom" { + grpc "https://agent.example.com:50051" + events "request_headers" +} +``` + +This works well for agents using certificates from public CAs (Let's Encrypt, DigiCert, etc.). + +### Skip Verification (Development Only) + +**Warning:** Only use this for local development. Never use in production. + +```kdl +agent "dev-agent" type="custom" { + grpc "https://localhost:50051" { + insecure-skip-verify + } + events "request_headers" +} +``` + +When enabled, Zentinel logs a security warning: +``` +WARN: TLS certificate verification disabled for agent connection +``` + +### Certificate Setup + +#### Generate CA and Certificates + +```bash +# Create CA +openssl genrsa -out ca.key 4096 +openssl req -new -x509 -days 3650 -key ca.key -out ca.crt \ + -subj "/CN=Zentinel Agent CA" + +# Create agent server certificate +openssl genrsa -out agent.key 2048 +openssl req -new -key agent.key -out agent.csr \ + -subj "/CN=waf-service.internal" +openssl x509 -req -days 365 -in agent.csr -CA ca.crt -CAkey ca.key \ + -CAcreateserial -out agent.crt + +# Create Zentinel client certificate (for mTLS) +openssl genrsa -out zentinel-client.key 2048 +openssl req -new -key zentinel-client.key -out zentinel-client.csr \ + -subj "/CN=zentinel-proxy" +openssl x509 -req -days 365 -in zentinel-client.csr -CA ca.crt -CAkey ca.key \ + -CAcreateserial -out zentinel-client.crt +``` + +#### File Permissions + +```bash +# Secure the private keys +chmod 600 /etc/zentinel/certs/*.key +chown zentinel:zentinel /etc/zentinel/certs/* +``` + +### Complete Secure Agent Example + +```kdl +agents { + // WAF with mTLS - highest security + agent "modsecurity" type="waf" { + grpc "https://waf.internal:50051" { + ca-cert "/etc/zentinel/certs/ca.crt" + client-cert "/etc/zentinel/certs/zentinel-client.crt" + client-key "/etc/zentinel/certs/zentinel-client.key" + } + events "request_headers" "request_body" + timeout-ms 200 + failure-mode "closed" + circuit-breaker { + failure-threshold 10 + timeout-seconds 60 + } + } + + // Auth with server-only TLS + agent "jwt-auth" type="auth" { + grpc "https://auth.internal:50051" { + ca-cert "/etc/zentinel/certs/ca.crt" + } + events "request_headers" + timeout-ms 50 + failure-mode "closed" + } + + // Rate limiter - internal network, no TLS + agent "rate-limiter" type="rate_limit" { + grpc "http://ratelimit.internal:50051" + events "request_headers" + timeout-ms 20 + failure-mode "open" + } +} +``` + +### Troubleshooting TLS + +| Error | Cause | Solution | +|-------|-------|----------| +| `certificate verify failed` | CA doesn't match agent cert | Verify CA certificate is correct | +| `certificate has expired` | Agent cert expired | Renew agent certificate | +| `handshake failure` | TLS version mismatch | Check both ends support TLS 1.2+ | +| `unknown ca` | Missing CA cert | Add `ca-cert` option | +| `bad certificate` | Client cert rejected | Verify client cert signed by agent's CA | + +## Default Values + +| Setting | Default | +|---------|---------| +| `timeout-ms` | `1000` (1 second) | +| `failure-mode` | `open` | +| `chunk-timeout-ms` | `5000` (5 seconds) | +| `request-body-mode` | `buffer` | +| `response-body-mode` | `buffer` | +| `circuit-breaker.failure-threshold` | `5` | +| `circuit-breaker.success-threshold` | `2` | +| `circuit-breaker.timeout-seconds` | `30` | +| `circuit-breaker.half-open-max-requests` | `1` | + +## Metrics + +Agent-related metrics: + +| Metric | Description | +|--------|-------------| +| `zentinel_agent_requests_total` | Agent calls by agent and status | +| `zentinel_agent_duration_seconds` | Agent call latency | +| `zentinel_agent_errors_total` | Agent errors | +| `zentinel_agent_timeouts_total` | Agent timeouts | +| `zentinel_agent_circuit_breaker_state` | Circuit breaker state | + +## Configuration Validation + +Zentinel validates agent configuration at startup: + +### Transport Validation + +| Transport | Validation | +|-----------|------------| +| Unix Socket | Path exists and is a socket file | +| gRPC | Valid URL format (http/https with host) | +| HTTP | Valid URL format | + +Example validation errors: + +``` +Error: Agent 'auth' socket path '/var/run/zentinel/auth.sock' does not exist +Error: Agent 'waf' path '/tmp/not-a-socket' exists but is not a socket +Error: Agent 'remote' gRPC address 'invalid-url' is not a valid URL +``` + +### Pre-flight Checks + +Run validation before deployment: + +```bash +zentinel --config zentinel.kdl --validate +``` + +This checks: +- All referenced agents exist +- Transport paths/URLs are valid +- Timeout values are within bounds +- Circuit breaker thresholds are valid + +> **Note:** Agent TLS (configured in this section) secures the connection between Zentinel and the agent process. This is separate from **upstream TLS**, which secures the connection between Zentinel and your backend servers. If your backend serves HTTPS, see [Upstream TLS](/configuration/upstreams/#upstream-tls). + +## Next Steps + +- [Agent Protocol](../../agents/protocol/) - Wire protocol specification +- [Building Agents](../../agents/building/) - Creating custom agents +- [Filters](../filters/) - Using agents in filter chains diff --git a/content/v/26.04/configuration/cache.md b/content/v/26.04/configuration/cache.md new file mode 100644 index 0000000..6cd7831 --- /dev/null +++ b/content/v/26.04/configuration/cache.md @@ -0,0 +1,453 @@ ++++ +title = "Caching" +weight = 7 +updated = 2026-02-19 ++++ + +Zentinel provides HTTP response caching to reduce upstream load and improve response times. Caching is configured at two levels: global storage settings and per-route caching policies. + +## Global Cache Storage + +Configure the cache storage backend at the server level: + +```kdl +cache { + enabled #true + backend "memory" // Storage backend + max-size 104857600 // 100MB total cache size + eviction-limit 104857600 // When to start evicting + lock-timeout 10 // Seconds (thundering herd protection) +} +``` + +### Storage Backends + +| Backend | Description | Use Case | +|---------|-------------|----------| +| `memory` | In-memory LRU cache (default) | Single instance, low latency | +| `disk` | Disk-based cache | Large cache, persistence across restarts | +| `hybrid` | Memory + disk tiered cache | Hot data in memory, cold on disk | + +#### Memory Backend + +Fast, in-memory caching using LRU eviction: + +```kdl +cache { + enabled #true + backend "memory" + max-size 209715200 // 200MB + eviction-limit 104857600 // Start evicting at 100MB + lock-timeout 10 +} +``` + +#### Disk Backend + +Persistent disk-based caching that survives proxy restarts. Cache entries are stored as files on disk using a sharded directory layout for performance: + +```kdl +cache { + enabled #true + backend "disk" + max-size 1073741824 // 1GB + disk-path "/var/cache/zentinel" + disk-shards 16 // Parallelism + lock-timeout 10 +} +``` + +**Directory layout:** + +``` +/var/cache/zentinel/ +├── shard-00/ +│ ├── 00/ +│ │ ├── .meta // Response metadata (headers, freshness) +│ │ └── .body // Response body +│ ├── 01/ +│ │ └── ... +│ ├── ... +│ ├── ff/ +│ └── tmp/ // Temporary files during writes +├── shard-01/ +│ └── ... +├── ... +├── shard-0f/ +└── eviction/ // LRU eviction state (persisted) +``` + +Each shard contains 256 hex-prefix subdirectories to prevent too many files in a single directory. The number of shards is configurable via `disk-shards` (default: 16). + +**Crash safety:** All writes use atomic temp-file-then-rename, so a crash during a write never leaves a corrupted cache entry. Orphaned temporary files from previous crashes are cleaned up automatically on startup. + +**Eviction state:** The LRU eviction manager's state is saved to `/eviction/` on shutdown and restored on startup, so the proxy knows which entries to evict first without scanning access patterns from scratch. + +> **Note:** `disk-path` is required when using the `disk` backend. The proxy will fail to start if it is not set. + +#### Hybrid Backend + +Two-tier caching with memory for hot data: + +```kdl +cache { + enabled #true + backend "hybrid" + max-size 1073741824 // 1GB total + disk-path "/var/cache/zentinel" + disk-shards 16 + lock-timeout 15 +} +``` + +> **Not yet implemented.** Configuring `backend "hybrid"` currently falls back to the memory backend with a warning logged at startup. `disk-path` is still required in the config so that switching to hybrid in a future release requires no config changes. Track progress in [#90](https://github.com/zentinelproxy/zentinel/issues/90). + +### Global Cache Options + +| Option | Default | Description | +|--------|---------|-------------| +| `enabled` | `true` | Enable caching globally | +| `backend` | `memory` | Storage backend | +| `max-size` | `104857600` (100MB) | Maximum cache size in bytes | +| `eviction-limit` | None | Size at which to start evicting entries | +| `lock-timeout` | `10` | Cache lock timeout in seconds | +| `disk-path` | None | Path for disk cache (required for disk/hybrid) | +| `disk-shards` | `16` | Number of disk cache shards | + +## Per-Route Caching + +Enable and configure caching for specific routes: + +```kdl +route "api" { + matches { + path-prefix "/api/v1/" + } + upstream "backend" + + cache { + enabled #true + default-ttl-secs 3600 + max-size-bytes 10485760 + stale-while-revalidate-secs 60 + stale-if-error-secs 300 + cacheable-methods "GET" "HEAD" + } +} +``` + +### Route Cache Options + +| Option | Default | Description | +|--------|---------|-------------| +| `enabled` | `false` | Enable caching for this route | +| `default-ttl-secs` | `3600` | Default TTL if no Cache-Control header | +| `max-size-bytes` | `10485760` (10MB) | Maximum cacheable response size | +| `cache-private` | `false` | Cache responses with `Cache-Control: private` | +| `stale-while-revalidate-secs` | `60` | Serve stale while revalidating in background | +| `stale-if-error-secs` | `300` | Serve stale on upstream error | +| `exclude-extensions` | `[]` | File extensions to skip caching (without dot) | +| `exclude-paths` | `[]` | Path glob patterns to skip caching | + +### Cacheable Methods + +Specify which HTTP methods are cacheable: + +```kdl +cache { + enabled #true + cacheable-methods "GET" "HEAD" +} +``` + +Default: `GET`, `HEAD` + +### Cacheable Status Codes + +Specify which response status codes are cacheable: + +```kdl +cache { + enabled #true + cacheable-status-codes 200 203 204 206 300 301 308 404 405 410 414 501 +} +``` + +Default: `200`, `203`, `204`, `206`, `300`, `301`, `308`, `404`, `405`, `410`, `414`, `501` + +### Vary Headers + +Headers that affect cache key: + +```kdl +cache { + enabled #true + vary-headers "Accept-Encoding" "Accept-Language" +} +``` + +When specified, these headers become part of the cache key, allowing different cached responses for different header values. + +### Ignoring Query Parameters + +Parameters to exclude from cache key: + +```kdl +cache { + enabled #true + ignore-query-params "utm_source" "utm_medium" "utm_campaign" "_" +} +``` + +Useful for ignoring tracking parameters or cache-busting tokens. + +### Excluding Extensions + +Exclude specific file extensions from caching. This is useful for broad routes (e.g., `path-prefix "/"`) where most content should be cached but dynamic file types should not: + +```kdl +cache { + enabled #true + default-ttl-secs 3600 + exclude-extensions "php" "html" "asp" +} +``` + +Extensions are specified without the leading dot. The check is case-insensitive. + +### Excluding Paths + +Exclude specific paths from caching using glob patterns (`*`, `**`, `?`): + +```kdl +cache { + enabled #true + default-ttl-secs 3600 + exclude-paths "/wp-admin/**" "/login" "/api/auth/**" +} +``` + +| Pattern | Matches | +|---------|---------| +| `/login` | Exact path `/login` | +| `/admin/*` | One level under `/admin/` (e.g., `/admin/users`) | +| `/api/auth/**` | Any depth under `/api/auth/` (e.g., `/api/auth/oauth/callback`) | +| `/files/*.tmp` | `.tmp` files directly under `/files/` | + +Requests matching an excluded extension or path are bypassed from caching and forwarded directly to the upstream. The cache status for these requests is reported as `BYPASS`. + +## Stale Content Handling + +### Stale-While-Revalidate + +Serve stale content while fetching fresh content in the background: + +```kdl +cache { + enabled #true + default-ttl-secs 3600 + stale-while-revalidate-secs 60 +} +``` + +When a cached response expires: +1. Immediately serve the stale response +2. Trigger background revalidation +3. Update cache when fresh response arrives + +### Stale-If-Error + +Serve stale content when upstream returns an error: + +```kdl +cache { + enabled #true + stale-if-error-secs 300 +} +``` + +When upstream returns 5xx or times out: +1. Check if stale content exists within grace period +2. Serve stale if available +3. Return error if no stale content + +## Cache Lock (Thundering Herd Protection) + +The `lock-timeout` prevents multiple requests from simultaneously fetching the same uncached resource: + +```kdl +cache { + lock-timeout 10 // Seconds to wait for cache lock +} +``` + +When a cache miss occurs: +1. First request acquires the lock and fetches from upstream +2. Subsequent requests wait for the lock (up to timeout) +3. All waiting requests receive the cached response +4. If lock times out, request proceeds to upstream + +## Complete Examples + +### API Caching + +```kdl +// Global cache storage +cache { + enabled #true + backend "memory" + max-size 536870912 // 512MB + lock-timeout 5 +} + +routes { + // Cache API responses + route "api" { + matches { + path-prefix "/api/v1/" + method "GET" + } + upstream "api-backend" + + cache { + enabled #true + default-ttl-secs 60 + max-size-bytes 1048576 // 1MB per response + stale-while-revalidate-secs 30 + stale-if-error-secs 120 + vary-headers "Authorization" + ignore-query-params "callback" "_" + } + } + + // Don't cache mutations + route "api-mutations" { + matches { + path-prefix "/api/v1/" + method "POST" "PUT" "DELETE" "PATCH" + } + upstream "api-backend" + // No cache block = caching disabled + } +} +``` + +### Static Asset Caching + +```kdl +cache { + enabled #true + backend "disk" + max-size 10737418240 // 10GB + disk-path "/var/cache/zentinel/static" + disk-shards 32 +} + +routes { + route "static-assets" { + matches { + path-prefix "/assets/" + } + upstream "cdn-origin" + + cache { + enabled #true + default-ttl-secs 86400 // 24 hours + max-size-bytes 52428800 // 50MB per file + stale-if-error-secs 86400 // Serve stale for 24h on error + cacheable-methods "GET" "HEAD" + } + } +} +``` + +### Multi-Tenant Caching + +```kdl +cache { + enabled #true + backend "memory" + max-size 1073741824 // 1GB +} + +routes { + route "tenant-api" { + matches { + path-prefix "/api/" + } + upstream "backend" + + cache { + enabled #true + default-ttl-secs 300 + vary-headers "X-Tenant-ID" "Accept-Language" + } + } +} +``` + +## Cache Behavior + +### Cache Key Generation + +The cache key is generated from: +1. Request method (if cacheable) +2. Request path +3. Query parameters (excluding ignored ones) +4. Vary headers (if configured) + +### Cache-Control Directives + +Zentinel respects Cache-Control headers from upstream: + +| Directive | Behavior | +|-----------|----------| +| `no-store` | Never cache response | +| `no-cache` | Cache but revalidate before use | +| `private` | Don't cache (unless `cache-private` enabled) | +| `max-age=N` | Cache for N seconds | +| `s-maxage=N` | Override max-age for shared caches | +| `must-revalidate` | Don't serve stale content | + +### Cache Invalidation + +Caches are automatically invalidated: +- When TTL expires +- On configuration reload (optional) +- Via admin endpoint (if enabled) + +```bash +# Purge all cached content +curl -X POST http://localhost:9090/admin/cache/purge + +# Purge specific path +curl -X POST http://localhost:9090/admin/cache/purge?path=/api/v1/users +``` + +## Metrics + +Cache-related metrics: + +| Metric | Description | +|--------|-------------| +| `zentinel_cache_hits_total` | Total cache hits | +| `zentinel_cache_misses_total` | Total cache misses | +| `zentinel_cache_size_bytes` | Current cache size | +| `zentinel_cache_entries` | Number of cached entries | +| `zentinel_cache_evictions_total` | Total evictions | +| `zentinel_cache_stale_served_total` | Stale responses served | + +## Best Practices + +1. **Set appropriate TTLs**: Match TTL to data freshness requirements +2. **Use stale-while-revalidate**: Improve perceived performance +3. **Configure vary headers carefully**: Too many = cache fragmentation +4. **Monitor cache hit rate**: Low hit rate indicates misconfiguration +5. **Size cache appropriately**: Undersized cache = frequent evictions +6. **Use disk cache for large content**: Keep memory for small, hot data + +## Next Steps + +- [Routes](../routes/) - Configuring route-level caching +- [Upstreams](../upstreams/) - Upstream configuration +- [Observability](../observability/) - Monitoring cache performance diff --git a/content/v/26.04/configuration/file-format.md b/content/v/26.04/configuration/file-format.md new file mode 100644 index 0000000..f27bcc9 --- /dev/null +++ b/content/v/26.04/configuration/file-format.md @@ -0,0 +1,461 @@ ++++ +title = "File Format" +weight = 1 +updated = 2026-02-27 ++++ + +Zentinel uses [KDL](https://kdl.dev/) as its primary configuration format. KDL is a human-friendly document language that's easy to read, write, and diff. + +## Why KDL? + +| Feature | Benefit | +|---------|---------| +| **Human-readable** | Clean syntax without excessive punctuation | +| **Git-friendly** | Diffs are clear and meaningful | +| **Typed values** | Numbers, strings, booleans, #null | +| **Comments** | Both line (`//`) and block (`/* */`) | +| **Hierarchical** | Natural nesting for configuration blocks | + +## Basic Syntax + +### Nodes + +KDL documents are made of nodes. Each node has a name and optional values/properties: + +```kdl +// Simple node +system + +// Node with a value +worker-threads 4 + +// Node with a property +listener "http" address="0.0.0.0:8080" + +// Node with children (block) +system { + worker-threads 4 + max-connections 10000 +} +``` + +### Values and Properties + +**Values** are positional arguments: +```kdl +route "api" // "api" is a value +targets "10.0.0.1" "10.0.0.2" // Multiple values +``` + +**Properties** are named with `=`: +```kdl +target address="10.0.0.1" weight=5 +health-check type="http" interval-secs=10 +``` + +### Data Types + +```kdl +// Example showing KDL data types in Zentinel config + +system { + // Numbers (integer) + worker-threads 4 + max-connections 1000 + graceful-shutdown-timeout-secs 30 +} + +listeners { + listener "http" { + // Strings (quoted) + address "0.0.0.0:8080" + protocol "http" + } +} + +routes { + route "default" { + // Booleans (#true, #false) + enabled #true + + matches { path-prefix "/" } + upstream "backend" + } +} + +upstreams { + upstream "backend" { + targets { + target { + address "127.0.0.1:3000" + // Numbers (float) + weight 1.5 + } + } + + // Null values (#null) + health-check #null + } +} +``` + +### Comments + +```kdl +// Line comment + +/* Block comment + spanning multiple + lines */ + +system { + worker-threads 4 // Inline comment +} +``` + +## File Structure + +A typical Zentinel configuration has these top-level blocks: + +```kdl +// System settings (use "system", not "server") +system { + // ... +} + +// Network listeners +listeners { + // ... +} + +// Request routing +routes { + // ... +} + +// Backend servers +upstreams { + // ... +} + +// External agents (optional) +agents { + // ... +} + +// Request/response limits +limits { + // ... +} + +// Logging and metrics (optional) +observability { + // ... +} + +// Hierarchical organization (optional) +namespace "api" { + limits { /* namespace-scoped limits */ } + listeners { /* namespace-scoped listeners */ } + upstreams { /* namespace-scoped upstreams */ } + routes { /* namespace-scoped routes */ } + agents { /* namespace-scoped agents */ } + + // Fine-grained isolation within namespace + service "payments" { + limits { /* service-scoped limits */ } + listener { /* dedicated listener */ } + upstreams { /* service-scoped upstreams */ } + routes { /* service-scoped routes */ } + } +} +``` + +> **Note:** The `server` block name is deprecated but still supported for backward compatibility. Use `system` for new configurations. + +## Schema Versioning + +Zentinel configurations include a schema version for compatibility checking. This helps catch configuration issues when upgrading Zentinel. + +```kdl +// Declare schema version at the top of your config +schema-version "1.0" + +system { + // ... +} +``` + +### Version Format + +Schema versions use `major.minor` format: + +| Version | Meaning | +|---------|---------| +| `1.0` | Initial stable schema | +| `1.1` | Minor additions (backward compatible) | +| `2.0` | Major changes (may require migration) | + +### Compatibility Behavior + +| Config Version vs Zentinel | Result | +|---------------------------|--------| +| Exact match | ✓ Loads normally | +| Config older but supported | ✓ Loads normally | +| Config newer than Zentinel | ⚠ Loads with warning (some features may not work) | +| Config older than minimum | ✗ Rejected with error | +| Invalid format | ✗ Rejected with error | + +### Omitting Version + +If `schema-version` is not specified, Zentinel assumes the current version. For production deployments, explicitly specifying the version is recommended: + +```kdl +// Explicit version (recommended for production) +schema-version "1.0" + +system { /* ... */ } +``` + +This ensures configuration files remain compatible when upgrading Zentinel, and provides clear error messages if migration is needed. + +## Complete Example + +```kdl +// Zentinel Configuration +// Production API Gateway + +schema-version "1.0" + +system { + worker-threads 0 // 0 = auto-detect CPU cores + max-connections 10000 + graceful-shutdown-timeout-secs 30 +} + +listeners { + listener "https" { + address "0.0.0.0:443" + protocol "https" + tls { + cert-file "/etc/zentinel/certs/server.crt" + key-file "/etc/zentinel/certs/server.key" + min-version "1.2" + } + } + + listener "admin" { + address "127.0.0.1:9090" + protocol "http" + } +} + +routes { + route "api" { + priority 100 + matches { + path-prefix "/api/" + method "GET" "POST" "PUT" "DELETE" + } + upstream "backend" + agents "auth" "ratelimit" + } + + route "health" { + priority 1000 + matches { + path "/health" + } + service-type "builtin" + builtin-handler "health" + } +} + +upstreams { + upstream "backend" { + targets { + target { address "10.0.1.1:8080" weight=3 } + target { address "10.0.1.2:8080" weight=2 } + target { address "10.0.1.3:8080" weight=1 } + } + load-balancing "weighted_round_robin" + health-check { + type "http" + path "/health" + interval-secs 10 + timeout-secs 5 + } + } +} + +agents { + agent "auth" { + type "auth" + transport "unix_socket" { + path "/var/run/zentinel/auth.sock" + } + timeout-ms 100 + failure-mode "closed" + } + + agent "ratelimit" { + type "rate_limit" + transport "unix_socket" { + path "/var/run/zentinel/ratelimit.sock" + } + timeout-ms 50 + failure-mode "open" + } +} + +limits { + max-header-size-bytes 8192 + max-header-count 100 + max-body-size-bytes 10485760 // 10MB +} +``` + +## Alternative Formats + +Zentinel also supports JSON and TOML for programmatic generation: + +### JSON + +```json +{ + "system": { + "worker_threads": 4, + "max_connections": 10000 + }, + "listeners": [ + { + "id": "http", + "address": "0.0.0.0:8080", + "protocol": "http" + } + ] +} +``` + +### TOML + +```toml +[system] +worker_threads = 4 +max_connections = 10000 + +[[listeners]] +id = "http" +address = "0.0.0.0:8080" +protocol = "http" +``` + +File format is auto-detected by extension: +- `.kdl` → KDL +- `.json` → JSON +- `.toml` → TOML + +## Multi-File Configuration + +For complex deployments, split configuration across multiple files: + +``` +/etc/zentinel/ +├── zentinel.kdl # Main config (includes others) +├── routes/ +│ ├── api.kdl +│ ├── static.kdl +│ └── admin.kdl +├── upstreams/ +│ ├── backend.kdl +│ └── cache.kdl +└── agents/ + └── security.kdl +``` + +Use directory loading: + +```bash +zentinel --config-dir /etc/zentinel/ +``` + +Or use `include` directives in your main config: + +```kdl +// zentinel.kdl +include "routes/*.kdl" +include "upstreams/*.kdl" +include "agents/*.kdl" +``` + +### Include Behavior + +The `include` directive is processed as a pre-processing step when loading configuration with `Config::from_file()` or `zentinel --config`. Include directives are resolved before the configuration is parsed, so included files can contain any top-level blocks (`routes`, `upstreams`, `agents`, etc.). + +**Glob patterns** — Include paths support glob patterns (`*`, `**`, `?`). Matched files are sorted alphabetically for deterministic ordering. + +**Relative paths** — Patterns are resolved relative to the directory of the file containing the `include` directive. This means included files can themselves include other files using paths relative to their own location. + +**Recursive includes** — Included files can contain their own `include` directives, which are expanded recursively. + +**Circular include detection** — Zentinel tracks which files have already been included (using canonical paths) and returns an error if a circular include is detected. + +**No-match behavior** — If a glob pattern matches no files, Zentinel logs a warning but continues loading. This allows optional includes like `include "overrides/*.kdl"` where the directory may be empty. + +```kdl +// routes/api.kdl — included file contains a top-level block +routes { + route "api" { + match { + path-prefix "/api" + } + upstream "backend" + } +} +``` + +## Validation + +Validate configuration before applying: + +```bash +# Check syntax and semantics +zentinel --config zentinel.kdl --validate + +# Dry-run mode +zentinel --config zentinel.kdl --dry-run +``` + +Common validation errors: + +| Error | Cause | +|-------|-------| +| Route references unknown upstream | Typo in upstream name | +| No listeners defined | Missing `listeners` block | +| Invalid socket address | Wrong format (need `host:port`) | +| Duplicate route ID | Two routes with same name | + +## Hot Reload + +Zentinel supports configuration reload without restart: + +```bash +# Send SIGHUP to reload +kill -HUP $(cat /var/run/zentinel.pid) + +# Or use the admin endpoint +curl -X POST http://localhost:9090/admin/reload +``` + +Reload behavior: +1. Parse new configuration +2. Validate syntax and semantics +3. If valid, atomically swap configuration +4. If invalid, keep old configuration and log error + +## Next Steps + +- [Server Configuration](../server/) - Server block settings +- [Listeners](../listeners/) - Network binding and TLS +- [Routes](../routes/) - Request routing +- [Namespaces & Services](../namespaces/) - Hierarchical organization diff --git a/content/v/26.04/configuration/filters.md b/content/v/26.04/configuration/filters.md new file mode 100644 index 0000000..b444b6c --- /dev/null +++ b/content/v/26.04/configuration/filters.md @@ -0,0 +1,440 @@ ++++ +title = "Filters" +weight = 6 +updated = 2026-02-27 ++++ + +Filters provide a flexible pipeline for request and response processing. They can be built-in (rate-limit, headers, CORS, compression) or external agents. Filters are defined centrally in the `filters` block with unique IDs, then referenced by name in route configurations. + +## Basic Configuration + +```kdl +filters { + filter "api-rate-limit" { + type "rate-limit" + max-rps 100 + burst 20 + key "client-ip" + } + + filter "add-security-headers" { + type "headers" + phase "response" + set { + "X-Content-Type-Options" "nosniff" + "X-Frame-Options" "DENY" + } + } +} + +routes { + route "api" { + matches { + path-prefix "/api/" + } + upstream "backend" + filters "api-rate-limit" "add-security-headers" + } +} +``` + +## Filter Types + +### Rate Limiting + +Rate limiting using a token bucket algorithm. Supports local (single-instance) or distributed (Redis) backends. + +```kdl +filter "rate-limiter" { + type "rate-limit" + max-rps 100 // Maximum requests per second + burst 20 // Token bucket size + key "client-ip" // Rate limit key + on-limit "reject" // Action when exceeded + status-code 429 // Response status + limit-message "Too many requests" +} +``` + +#### Rate Limit Keys + +| Key | Description | +|-----|-------------| +| `client-ip` | Rate limit per client IP address (default) | +| `path` | Rate limit per request path | +| `route` | Global rate limit for the route | +| `client-ip-and-path` | Combined client IP and path | +| `header:X-API-Key` | Rate limit by specific header value | + +#### Rate Limit Actions + +| Action | Description | +|--------|-------------| +| `reject` | Reject with 429 status (default) | +| `delay` | Queue request until tokens available (up to max-delay-ms) | +| `log-only` | Log but allow request through | + +#### Distributed Rate Limiting with Redis + +```kdl +filter "distributed-limiter" { + type "rate-limit" + max-rps 1000 + burst 100 + backend "redis" { + url "redis://127.0.0.1:6379" + key-prefix "zentinel:ratelimit:" + pool-size 10 + timeout-ms 50 + fallback-local #true + } +} +``` + +### Global Rate Limits + +Apply rate limits at the server level before route-specific limits: + +```kdl +rate-limits { + default-rps 100 // Default for routes without explicit limits + default-burst 20 + key "client-ip" + + global { + max-rps 10000 // Server-wide limit + burst 1000 + key "client-ip" + } +} +``` + +### Headers Filter + +Manipulate request or response headers: + +```kdl +filter "security-headers" { + type "headers" + phase "response" // "request", "response", or "both" + + // Set headers (overwrites existing) + set { + "X-Content-Type-Options" "nosniff" + "X-Frame-Options" "DENY" + "X-XSS-Protection" "1; mode=block" + "Strict-Transport-Security" "max-age=31536000; includeSubDomains" + } + + // Add headers (preserves existing) + add { + "X-Request-ID" "zentinel-generated" + } + + // Remove headers + remove "Server" "X-Powered-By" +} +``` + +### CORS Filter + +Handle Cross-Origin Resource Sharing: + +```kdl +filter "cors" { + type "cors" + allowed-origins "https://example.com" "https://app.example.com" + allowed-methods "GET" "POST" "PUT" "DELETE" "OPTIONS" + allowed-headers "Content-Type" "Authorization" "X-Request-ID" + exposed-headers "X-Request-ID" "X-RateLimit-Remaining" + allow-credentials #true + max-age-secs 86400 // Preflight cache duration +} +``` + +| Option | Default | Description | +|--------|---------|-------------| +| `allowed-origins` | `*` | Origins allowed to access (use `*` for any) | +| `allowed-methods` | Common methods | HTTP methods allowed | +| `allowed-headers` | None | Request headers client can send | +| `exposed-headers` | None | Response headers client can read | +| `allow-credentials` | `false` | Allow cookies/auth headers | +| `max-age-secs` | `86400` | Preflight cache duration | + +### Compression Filter + +Compress response bodies: + +```kdl +filter "compress" { + type "compress" + algorithms "brotli" "gzip" "zstd" + min-size 1024 // Minimum size to compress (bytes) + level 6 // Compression level (1-9) + content-types "text/html" "text/css" "application/json" "application/javascript" +} +``` + +| Algorithm | Description | +|-----------|-------------| +| `gzip` | Widely supported, good compression | +| `brotli` | Better compression, modern browsers | +| `deflate` | Legacy, wide support | +| `zstd` | Fast compression/decompression | + +### GeoIP Filter + +Filter requests based on geographic location: + +```kdl +// Block mode - block specific countries +filter "block-countries" { + type "geo" + database-path "/etc/zentinel/GeoLite2-Country.mmdb" + action "block" + countries "RU" "CN" "KP" "IR" + on-failure "closed" // Block if lookup fails + status-code 403 + block-message "Access denied from your region" +} + +// Allow mode - allow only specific countries +filter "us-only" { + type "geo" + database-path "/etc/zentinel/GeoLite2-Country.mmdb" + action "allow" + countries "US" "CA" + status-code 451 // Unavailable for legal reasons +} + +// Log-only mode - tag requests with country +filter "geo-tagging" { + type "geo" + database-path "/etc/zentinel/GeoLite2-Country.mmdb" + action "log-only" + add-country-header #true +} +``` + +#### Geo Filter Options + +| Option | Default | Description | +|--------|---------|-------------| +| `database-path` | Required | Path to GeoIP database (.mmdb or .bin) | +| `database-type` | Auto-detect | `maxmind` or `ip2location` | +| `action` | `block` | `block`, `allow`, or `log-only` | +| `countries` | Empty | ISO 3166-1 alpha-2 codes (e.g., `US`, `CN`) | +| `on-failure` | `open` | `open` (allow) or `closed` (block) on lookup failure | +| `status-code` | `403` | HTTP status for blocked requests | +| `block-message` | None | Custom message for blocked requests | +| `add-country-header` | `true` | Add `X-Country-Code` header | +| `cache-ttl-secs` | `3600` | Cache TTL for lookups | + +### Timeout Filter + +Override timeouts for specific routes: + +```kdl +filter "long-timeout" { + type "timeout" + request-timeout-secs 300 // Total request timeout + upstream-timeout-secs 120 // Backend timeout + connect-timeout-secs 30 // Connection timeout +} +``` + +### Log Filter + +Add detailed logging for specific routes: + +```kdl +filter "debug-logging" { + type "log" + log-request #true + log-response #true + log-body #true + max-body-log-size 4096 + level "debug" + fields "user-agent" "content-type" "x-request-id" +} +``` + +### Agent Filter + +Delegate processing to an external agent: + +```kdl +filter "waf" { + type "agent" + agent "waf-agent" // Reference to agent defined in agents block + timeout-ms 200 + failure-mode "open" // "open" or "closed" +} +``` + +Agent filters automatically process both request-phase and response-phase events based on the agent's `events` subscription. An agent subscribed to `response_headers` and `response_body` will be called during the response phase to inspect and modify the upstream response: + +```kdl +// Agent that transforms response content (e.g. image optimization) +filter "image-optimization" { + type "agent" + agent "image-optimizer" + timeout-ms 5000 + failure-mode "open" +} +``` + +## Filter Phases + +Filters execute at different phases of the request lifecycle: + +| Phase | Description | +|-------|-------------| +| `request` | Before forwarding to upstream | +| `response` | After receiving from upstream | +| `both` | Both request and response phases | + +Filter phase mapping: + +| Filter Type | Default Phase | +|-------------|---------------| +| `rate-limit` | Request | +| `headers` | Configurable | +| `cors` | Both | +| `compress` | Response | +| `geo` | Request | +| `timeout` | Request | +| `log` | Configurable | +| `agent` | Configurable | + +## Filter Ordering + +Filters execute in the order specified in the route: + +```kdl +route "api" { + filters "rate-limit" "auth" "cors" "logging" + // ^^^^^^^^^ ^^^^ ^^^^ ^^^^^^^ + // 1st 2nd 3rd 4th +} +``` + +For request phase: +1. Rate limit check +2. Auth validation +3. CORS headers (preflight) +4. Logging + +For response phase (reverse order of declaration): +1. Logging +2. CORS headers +3. (Auth typically doesn't run on response) +4. (Rate limit doesn't run on response) + +## Complete Example + +```kdl +filters { + // Rate limiting with Redis backend + filter "api-limiter" { + type "rate-limit" + max-rps 100 + burst 20 + key "header:X-API-Key" + on-limit "reject" + backend "redis" { + url "redis://redis:6379" + fallback-local #true + } + } + + // Geo-blocking + filter "geo-block" { + type "geo" + database-path "/etc/zentinel/GeoLite2-Country.mmdb" + action "block" + countries "RU" "CN" + on-failure "open" + } + + // CORS for API + filter "api-cors" { + type "cors" + allowed-origins "https://app.example.com" + allowed-methods "GET" "POST" "PUT" "DELETE" + allowed-headers "Content-Type" "Authorization" + allow-credentials #true + } + + // Security headers + filter "security" { + type "headers" + phase "response" + set { + "X-Content-Type-Options" "nosniff" + "X-Frame-Options" "DENY" + } + remove "Server" + } + + // Response compression + filter "compress" { + type "compress" + algorithms "brotli" "gzip" + min-size 1024 + } + + // WAF agent + filter "waf" { + type "agent" + agent "waf-agent" + timeout-ms 100 + failure-mode "closed" + } +} + +routes { + route "public-api" { + matches { + path-prefix "/api/v1/" + } + upstream "api-backend" + filters "geo-block" "api-limiter" "api-cors" "security" "compress" + } + + route "secure-api" { + matches { + path-prefix "/api/admin/" + } + upstream "admin-backend" + filters "geo-block" "waf" "api-limiter" "security" + } +} +``` + +## Default Values + +| Filter | Setting | Default | +|--------|---------|---------| +| Rate Limit | `burst` | `10` | +| Rate Limit | `key` | `client-ip` | +| Rate Limit | `on-limit` | `reject` | +| Rate Limit | `status-code` | `429` | +| Rate Limit | `max-delay-ms` | `5000` | +| Headers | `phase` | `request` | +| Compress | `algorithms` | `gzip`, `brotli` | +| Compress | `min-size` | `1024` | +| Compress | `level` | `6` | +| CORS | `allowed-origins` | `*` | +| CORS | `max-age-secs` | `86400` | +| Geo | `action` | `block` | +| Geo | `on-failure` | `open` | +| Geo | `status-code` | `403` | +| Log | `level` | `info` | +| Log | `max-body-log-size` | `4096` | + +## Next Steps + +- [Agents](../../agents/) - External processing agents +- [Routes](../routes/) - Applying filters to routes +- [Observability](../observability/) - Logging and metrics diff --git a/content/v/26.04/configuration/limits.md b/content/v/26.04/configuration/limits.md new file mode 100644 index 0000000..88a3ecc --- /dev/null +++ b/content/v/26.04/configuration/limits.md @@ -0,0 +1,528 @@ ++++ +title = "Limits" +weight = 6 +updated = 2026-02-19 ++++ + +The `limits` block configures request/response limits, connection limits, and rate limiting. These settings are critical for predictable behavior, resource protection, and "sleepable operations." + +## Basic Configuration + +```kdl +limits { + max-header-size-bytes 8192 + max-header-count 100 + max-body-size-bytes 10485760 // 10MB +} +``` + +## Header Limits + +```kdl +limits { + max-header-size-bytes 8192 // Total headers size + max-header-count 100 // Maximum header count + max-header-name-bytes 256 // Per-header name size + max-header-value-bytes 4096 // Per-header value size +} +``` + +| Setting | Default | Description | +|---------|---------|-------------| +| `max-header-size-bytes` | `8192` (8KB) | Total size of all headers | +| `max-header-count` | `100` | Maximum number of headers | +| `max-header-name-bytes` | `256` | Maximum header name length | +| `max-header-value-bytes` | `4096` (4KB) | Maximum header value length | + +**Security note:** Large header limits can be exploited for denial-of-service. Keep defaults unless you have specific requirements. + +### Recommended Header Limits + +| Environment | header-size | header-count | header-value | +|-------------|-------------|--------------|--------------| +| Production | 4096-8192 | 50-100 | 2048-4096 | +| Development | 16384 | 200 | 8192 | +| API Gateway | 8192 | 100 | 4096 | + +## Body Limits + +```kdl +limits { + max-body-size-bytes 10485760 // 10MB - maximum request body + max-body-buffer-bytes 1048576 // 1MB - buffer for inspection + max-body-inspection-bytes 1048576 // 1MB - bytes sent to agents +} +``` + +| Setting | Default | Description | +|---------|---------|-------------| +| `max-body-size-bytes` | `10485760` (10MB) | Maximum request body size | +| `max-body-buffer-bytes` | `1048576` (1MB) | Maximum buffered body size | +| `max-body-inspection-bytes` | `1048576` (1MB) | Maximum body sent to agents | + +### Body Size Guidelines + +| Use Case | Recommended Size | +|----------|------------------| +| API endpoints | 1-10 MB | +| File uploads | 100 MB - 1 GB | +| JSON APIs | 1-5 MB | +| Form submissions | 1-10 MB | + +**Memory impact:** `max-body-buffer-bytes × max-in-flight-requests` = potential memory usage for body buffering. + +### Per-Route Body Limits + +Override body limits on specific routes: + +```kdl +routes { + route "upload" { + matches { + path-prefix "/upload/" + } + upstream "storage" + policies { + max-body-size "500MB" + } + } + + route "api" { + matches { + path-prefix "/api/" + } + upstream "backend" + policies { + max-body-size "1MB" + } + } +} +``` + +## Decompression Limits + +Protect against decompression bombs: + +```kdl +limits { + max-decompression-ratio 100.0 // Max 100:1 compression ratio + max-decompressed-size-bytes 104857600 // 100MB decompressed +} +``` + +| Setting | Default | Description | +|---------|---------|-------------| +| `max-decompression-ratio` | `100.0` | Maximum compression ratio allowed | +| `max-decompressed-size-bytes` | `104857600` (100MB) | Maximum decompressed size | + +**Security note:** Compression bombs (zip bombs) can use 1KB of compressed data to expand to gigabytes. These limits prevent resource exhaustion. + +## Connection Limits + +```kdl +limits { + max-connections-per-client 100 + max-connections-per-route 1000 + max-total-connections 10000 + max-idle-connections-per-upstream 100 +} +``` + +| Setting | Default | Description | +|---------|---------|-------------| +| `max-connections-per-client` | `100` | Per-client connection limit | +| `max-connections-per-route` | `1000` | Per-route connection limit | +| `max-total-connections` | `10000` | Global connection limit | +| `max-idle-connections-per-upstream` | `100` | Idle connections per upstream | + +### Connection Limit Sizing + +| Deployment | per-client | per-route | total | +|------------|------------|-----------|-------| +| Small | 50 | 500 | 5000 | +| Medium | 100 | 1000 | 10000 | +| Large | 200 | 2000 | 50000 | +| API Gateway | 100 | 5000 | 100000 | + +**Formula:** `max-total-connections` should be less than or equal to OS file descriptor limit minus a safety margin. + +Check your limits: +```bash +ulimit -n # Current soft limit +cat /proc/sys/fs/file-max # System limit +``` + +## Request Limits + +```kdl +limits { + max-in-flight-requests 10000 + max-in-flight-requests-per-worker 1000 + max-queued-requests 1000 +} +``` + +| Setting | Default | Description | +|---------|---------|-------------| +| `max-in-flight-requests` | `10000` | Total concurrent requests | +| `max-in-flight-requests-per-worker` | `1000` | Per-worker concurrent requests | +| `max-queued-requests` | `1000` | Pending requests in queue | + +When limits are reached: +- New requests receive `503 Service Unavailable` +- `Retry-After` header indicates when to retry + +## Agent Limits + +```kdl +limits { + max-agent-queue-depth 100 + max-agent-body-bytes 1048576 // 1MB to agents + max-agent-response-bytes 10240 // 10KB from agents +} +``` + +| Setting | Default | Description | +|---------|---------|-------------| +| `max-agent-queue-depth` | `100` | Pending requests per agent | +| `max-agent-body-bytes` | `1048576` (1MB) | Request body sent to agents | +| `max-agent-response-bytes` | `10240` (10KB) | Response size from agents | + +These limits protect the proxy from misbehaving agents and ensure bounded memory usage. + +## Rate Limiting + +### Global Rate Limits + +```kdl +limits { + max-requests-per-second-global 10000 +} +``` + +Global limit applies across all clients and routes. + +### Per-Client Rate Limits + +```kdl +limits { + max-requests-per-second-per-client 100 +} +``` + +Limits requests per unique client IP address. + +### Per-Route Rate Limits + +```kdl +limits { + max-requests-per-second-per-route 1000 +} +``` + +Limits total requests to each route. + +### Rate Limit Behavior + +Rate limiting uses token bucket algorithm: +- **Burst capacity:** 10× the per-second limit +- **Refill rate:** Continuous refill at configured rate +- **Response:** `429 Too Many Requests` with `Retry-After` header + +Example: +```kdl +limits { + max-requests-per-second-per-client 100 +} +``` +- Burst: 1000 requests +- Sustained: 100 requests/second +- Over limit: 429 response, retry in ~1 second + +### Route-Level Rate Limits + +For fine-grained control, configure rate limits per route: + +```kdl +routes { + route "api" { + matches { + path-prefix "/api/" + } + upstream "backend" + policies { + rate-limit { + requests-per-second 100 + burst 500 + key "client_ip" // or "header:X-API-Key" + } + } + } +} +``` + +Rate limit keys: +- `client_ip` - Client IP address +- `header:X-API-Key` - Header value +- `path` - Request path +- `route` - Route ID + +## Memory Limits + +```kdl +limits { + max-memory-bytes 2147483648 // 2GB hard limit + max-memory-percent 80.0 // 80% of system memory +} +``` + +| Setting | Default | Description | +|---------|---------|-------------| +| `max-memory-bytes` | None | Absolute memory limit | +| `max-memory-percent` | None | Percentage of system memory | + +When memory limits are reached: +1. New connections are rejected +2. Request queues are flushed +3. Alert is raised via metrics/logs + +## Profile Presets + +### Development + +```kdl +system { + worker-threads 0 +} + +listeners { + listener "http" { + address "0.0.0.0:8080" + protocol "http" + } +} + +limits { + // Permissive for testing + max-header-size-bytes 16384 + max-header-count 200 + max-body-size-bytes 104857600 // 100MB + max-in-flight-requests 100000 + + // No rate limits + max-requests-per-second-global #null + max-requests-per-second-per-client #null +} + +routes { + route "default" { + matches { path-prefix "/" } + upstream "backend" + } +} + +upstreams { + upstream "backend" { + targets { + target { address "127.0.0.1:3000" } + } + } +} +``` + +### Production (Conservative) + +```kdl +limits { + // Restrictive headers + max-header-size-bytes 4096 + max-header-count 50 + max-header-name-bytes 128 + max-header-value-bytes 2048 + + // Limited body + max-body-size-bytes 1048576 // 1MB + max-body-buffer-bytes 524288 // 512KB + + // Connection protection + max-connections-per-client 50 + max-total-connections 5000 + max-in-flight-requests 5000 + + // Rate limiting + max-requests-per-second-global 10000 + max-requests-per-second-per-client 100 + + // Memory protection + max-memory-percent 80.0 +} +``` + +### High-Traffic API + +```kdl +limits { + // Standard headers + max-header-size-bytes 8192 + max-header-count 100 + + // JSON payloads + max-body-size-bytes 10485760 // 10MB + + // High throughput + max-connections-per-client 200 + max-connections-per-route 5000 + max-total-connections 50000 + max-in-flight-requests 50000 + max-in-flight-requests-per-worker 5000 + + // Rate limiting + max-requests-per-second-global 100000 + max-requests-per-second-per-client 1000 +} +``` + +## Complete Example + +```kdl +limits { + // Headers + max-header-size-bytes 8192 + max-header-count 100 + max-header-name-bytes 256 + max-header-value-bytes 4096 + + // Bodies + max-body-size-bytes 10485760 + max-body-buffer-bytes 1048576 + max-body-inspection-bytes 1048576 + + // Decompression protection + max-decompression-ratio 100.0 + max-decompressed-size-bytes 104857600 + + // Connections + max-connections-per-client 100 + max-connections-per-route 1000 + max-total-connections 10000 + max-idle-connections-per-upstream 100 + + // Requests + max-in-flight-requests 10000 + max-in-flight-requests-per-worker 1000 + max-queued-requests 1000 + + // Agents + max-agent-queue-depth 100 + max-agent-body-bytes 1048576 + max-agent-response-bytes 10240 + + // Rate limiting + max-requests-per-second-global 10000 + max-requests-per-second-per-client 100 + max-requests-per-second-per-route 1000 + + // Memory + max-memory-percent 80.0 +} +``` + +## Default Values Summary + +| Category | Setting | Default | +|----------|---------|---------| +| **Headers** | `max-header-size-bytes` | 8192 | +| | `max-header-count` | 100 | +| | `max-header-name-bytes` | 256 | +| | `max-header-value-bytes` | 4096 | +| **Bodies** | `max-body-size-bytes` | 10485760 (10MB) | +| | `max-body-buffer-bytes` | 1048576 (1MB) | +| | `max-body-inspection-bytes` | 1048576 (1MB) | +| **Decompression** | `max-decompression-ratio` | 100.0 | +| | `max-decompressed-size-bytes` | 104857600 (100MB) | +| **Connections** | `max-connections-per-client` | 100 | +| | `max-connections-per-route` | 1000 | +| | `max-total-connections` | 10000 | +| | `max-idle-connections-per-upstream` | 100 | +| **Requests** | `max-in-flight-requests` | 10000 | +| | `max-in-flight-requests-per-worker` | 1000 | +| | `max-queued-requests` | 1000 | +| **Agents** | `max-agent-queue-depth` | 100 | +| | `max-agent-body-bytes` | 1048576 (1MB) | +| | `max-agent-response-bytes` | 10240 (10KB) | +| **Rate Limits** | All rate limits | None (disabled) | +| **Memory** | Memory limits | None (disabled) | + +## Monitoring Limits + +### Metrics + +Zentinel exposes limit-related metrics: + +``` +zentinel_header_size_exceeded_total +zentinel_body_size_exceeded_total +zentinel_connection_limit_reached_total +zentinel_rate_limit_exceeded_total +zentinel_memory_usage_bytes +zentinel_connections_active +zentinel_requests_in_flight +``` + +### Logging + +Limit violations are logged at WARN level: + +```json +{ + "level": "WARN", + "message": "Request body size exceeded limit", + "limit": 10485760, + "actual": 15728640, + "client_ip": "10.0.0.5", + "trace_id": "2kF8xQw4BnM" +} +``` + +## Troubleshooting + +### 413 Payload Too Large + +Request body exceeds `max-body-size-bytes`: + +```bash +# Check current limit +grep max-body-size zentinel.kdl + +# Increase for specific route +route "upload" { + policies { + max-body-size "100MB" + } +} +``` + +### 429 Too Many Requests + +Rate limit exceeded: + +```bash +# Check Retry-After header +curl -I https://api.example.com/endpoint +# Retry-After: 1 + +# Increase limits or implement client-side throttling +``` + +### 503 Service Unavailable (Connection Limit) + +Connection or request limits reached: + +1. Check current connections: `curl localhost:9090/admin/stats` +2. Review `max-total-connections` and `max-in-flight-requests` +3. Check for connection leaks in clients + +## Next Steps + +- [Server Configuration](../server/) - Worker threads and global settings +- [Upstreams](../upstreams/) - Connection pool settings diff --git a/content/v/26.04/configuration/listeners.md b/content/v/26.04/configuration/listeners.md new file mode 100644 index 0000000..b4a0238 --- /dev/null +++ b/content/v/26.04/configuration/listeners.md @@ -0,0 +1,1092 @@ ++++ +title = "Listeners" +weight = 3 +updated = 2026-02-19 ++++ + +The `listeners` block defines network endpoints where Zentinel accepts incoming connections. Each listener binds to an address, specifies a protocol, and optionally configures TLS. + +## Basic Configuration + +```kdl +listeners { + listener "http" { + address "0.0.0.0:8080" + protocol "http" + } + + listener "https" { + address "0.0.0.0:443" + protocol "https" + tls { + cert-file "/etc/zentinel/certs/server.crt" + key-file "/etc/zentinel/certs/server.key" + } + } +} +``` + +## Listener Options + +### Address + +```kdl +listener "api" { + address "0.0.0.0:8080" +} +``` + +Socket address in `host:port` format: + +| Format | Example | Use Case | +|--------|---------|----------| +| All interfaces | `0.0.0.0:8080` | Production, accept from anywhere | +| Localhost only | `127.0.0.1:8080` | Admin endpoints, local testing | +| IPv6 all | `[::]:8080` | IPv6 networks | +| IPv6 localhost | `[::1]:8080` | IPv6 local only | +| Specific interface | `10.0.1.5:8080` | Multi-homed servers | + +### Protocol + +```kdl +listener "secure" { + protocol "https" +} +``` + +| Protocol | Description | TLS Required | +|----------|-------------|--------------| +| `http` | Plain HTTP/1.1 | No | +| `https` | HTTP/1.1 over TLS | Yes | +| `h2` | HTTP/2 (with TLS via ALPN) | Yes | +| `h3` | HTTP/3 (QUIC) | Yes | + +### Timeouts + +```kdl +listener "api" { + address "0.0.0.0:8080" + protocol "http" + request-timeout-secs 60 + keepalive-timeout-secs 75 +} +``` + +| Setting | Default | Description | +|---------|---------|-------------| +| `request-timeout-secs` | `60` | Maximum time to receive complete request | +| `keepalive-timeout-secs` | `75` | Idle connection timeout | + +**Timeout recommendations:** + +| Scenario | Request Timeout | Keep-Alive | +|----------|-----------------|------------| +| API traffic | 30-60s | 60-120s | +| File uploads | 300s+ | 75s | +| WebSocket upgrade | 60s | 3600s+ | +| Internal services | 10-30s | 30s | + +### HTTP/2 Settings + +```kdl +listener "h2" { + address "0.0.0.0:443" + protocol "h2" + max-concurrent-streams 100 + tls { + cert-file "/etc/zentinel/certs/server.crt" + key-file "/etc/zentinel/certs/server.key" + } +} +``` + +| Setting | Default | Description | +|---------|---------|-------------| +| `max-concurrent-streams` | `100` | Maximum concurrent HTTP/2 streams per connection | + +### Default Route + +```kdl +listener "api" { + address "0.0.0.0:8080" + protocol "http" + default-route "fallback" +} +``` + +Route to use when no other route matches. If not set and no route matches, Zentinel returns 404. + +## TLS Configuration + +This section covers **listener TLS** (encrypting connections between clients and Zentinel). If you need to connect to a backend that serves HTTPS, see [Upstream TLS](/configuration/upstreams/#upstream-tls) instead. + +### Basic TLS + +```kdl +listener "https" { + address "0.0.0.0:443" + protocol "https" + tls { + cert-file "/etc/zentinel/certs/server.crt" + key-file "/etc/zentinel/certs/server.key" + } +} +``` + +### TLS Options Reference + +```kdl +system { + worker-threads 0 +} + +listeners { + listener "https" { + address "0.0.0.0:443" + protocol "https" + tls { + // Required + cert-file "/path/to/cert.pem" + key-file "/path/to/key.pem" + + // Version control + min-version "1.2" // Minimum: 1.0, 1.1, 1.2, 1.3 + max-version "1.3" // Maximum TLS version + + // Client authentication (mTLS) + ca-file "/path/to/ca.pem" + client-auth #true + + // Performance + session-resumption #true // TLS session tickets + ocsp-stapling #true // OCSP stapling + + // Cipher control (optional) + cipher-suites "TLS_AES_256_GCM_SHA384" "TLS_CHACHA20_POLY1305_SHA256" + } + } +} + +routes { + route "default" { + matches { path-prefix "/" } + upstream "backend" + } +} + +upstreams { + upstream "backend" { + targets { + target { address "127.0.0.1:3000" } + } + } +} +``` + +### TLS Version + +| Version | Status | Notes | +|---------|--------|-------| +| `1.0` | Deprecated | Avoid unless required for legacy clients | +| `1.1` | Deprecated | Avoid unless required for legacy clients | +| `1.2` | **Default minimum** | Good balance of compatibility and security | +| `1.3` | Recommended | Best performance and security | + +**Production recommendation:** + +```kdl +tls { + min-version "1.2" + max-version "1.3" +} +``` + +### Client Authentication (mTLS) + +For mutual TLS, require clients to present certificates: + +```kdl +listener "internal-api" { + address "0.0.0.0:8443" + protocol "https" + tls { + cert-file "/etc/zentinel/certs/server.crt" + key-file "/etc/zentinel/certs/server.key" + ca-file "/etc/zentinel/certs/client-ca.crt" + client-auth #true + } +} +``` + +Client certificates are validated against the CA certificate. Failed validation results in TLS handshake failure. + +### Session Resumption + +```kdl +system { + worker-threads 0 +} + +listeners { + listener "https" { + address "0.0.0.0:443" + protocol "https" + tls { + cert-file "/etc/zentinel/tls/cert.pem" + key-file "/etc/zentinel/tls/key.pem" + session-resumption #true // Default: true + } + } +} + +routes { + route "default" { + matches { path-prefix "/" } + upstream "backend" + } +} + +upstreams { + upstream "backend" { + targets { + target { address "127.0.0.1:3000" } + } + } +} +``` + +Enables TLS session tickets for faster reconnections. Reduces handshake overhead for returning clients. + +**Security note:** Session tickets are encrypted with server-side keys that rotate automatically. + +### OCSP Stapling + +```kdl +system { + worker-threads 0 +} + +listeners { + listener "https" { + address "0.0.0.0:443" + protocol "https" + tls { + cert-file "/etc/zentinel/tls/cert.pem" + key-file "/etc/zentinel/tls/key.pem" + ocsp-stapling #true // Default: true + } + } +} + +routes { + route "default" { + matches { path-prefix "/" } + upstream "backend" + } +} + +upstreams { + upstream "backend" { + targets { + target { address "127.0.0.1:3000" } + } + } +} +``` + +Server fetches and staples OCSP responses, proving certificate validity without clients contacting the CA. + +Benefits: +- Faster TLS handshakes +- Better client privacy +- Reduced CA load + +### Custom Cipher Suites + +```kdl +tls { + cipher-suites "TLS_AES_256_GCM_SHA384" "TLS_CHACHA20_POLY1305_SHA256" +} +``` + +Override default cipher suite selection. Leave empty to use secure defaults. + +**TLS 1.3 cipher suites:** +- `TLS_AES_256_GCM_SHA384` +- `TLS_AES_128_GCM_SHA256` +- `TLS_CHACHA20_POLY1305_SHA256` + +**TLS 1.2 cipher suites (recommended):** +- `TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384` +- `TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256` +- `TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305_SHA256` + +## SNI (Server Name Indication) + +Serve different certificates based on the hostname the client requests. This enables hosting multiple domains on a single IP address. + +### Basic SNI Configuration + +```kdl +listener "https" { + address "0.0.0.0:443" + protocol "https" + tls { + // Default certificate (when no SNI match) + cert-file "/etc/zentinel/certs/default.crt" + key-file "/etc/zentinel/certs/default.key" + + // Additional certificates for SNI + additional-certs { + sni-cert { + hostnames "example.com" "www.example.com" + cert-file "/etc/zentinel/certs/example.crt" + key-file "/etc/zentinel/certs/example.key" + } + + sni-cert { + hostnames "api.example.com" + cert-file "/etc/zentinel/certs/api.crt" + key-file "/etc/zentinel/certs/api.key" + } + + sni-cert { + hostnames "*.staging.example.com" + cert-file "/etc/zentinel/certs/staging-wildcard.crt" + key-file "/etc/zentinel/certs/staging-wildcard.key" + } + } + } +} +``` + +### SNI Hostname Patterns + +| Pattern | Matches | +|---------|---------| +| `example.com` | Exact match only | +| `www.example.com` | Exact match only | +| `*.example.com` | Any single subdomain (e.g., `api.example.com`, `www.example.com`) | +| `*.*.example.com` | Two subdomain levels | + +### SNI Resolution Order + +1. Exact hostname match +2. Wildcard pattern match (most specific wins) +3. Default certificate + +### Multi-Domain Example + +```kdl +listener "https" { + address "0.0.0.0:443" + protocol "https" + tls { + // Default for unmatched hostnames + cert-file "/etc/zentinel/certs/default.crt" + key-file "/etc/zentinel/certs/default.key" + + additional-certs { + // Production domains + sni-cert { + hostnames "myapp.com" "www.myapp.com" + cert-file "/etc/zentinel/certs/myapp.crt" + key-file "/etc/zentinel/certs/myapp.key" + } + + // API subdomain with separate cert + sni-cert { + hostnames "api.myapp.com" + cert-file "/etc/zentinel/certs/api.myapp.crt" + key-file "/etc/zentinel/certs/api.myapp.key" + } + + // Customer domains + sni-cert { + hostnames "customer1.myapp.com" "customer1-custom.com" + cert-file "/etc/zentinel/certs/customer1.crt" + key-file "/etc/zentinel/certs/customer1.key" + } + + // Wildcard for all other subdomains + sni-cert { + hostnames "*.myapp.com" + cert-file "/etc/zentinel/certs/wildcard.myapp.crt" + key-file "/etc/zentinel/certs/wildcard.myapp.key" + } + } + } +} +``` + +### Certificate Hot Reload + +All SNI certificates are reloaded during configuration reload: + +```bash +# Update certificates +cp new-cert.crt /etc/zentinel/certs/example.crt +cp new-key.key /etc/zentinel/certs/example.key + +# Reload configuration (graceful) +kill -HUP $(cat /var/run/zentinel.pid) +``` + +Connections in progress continue with old certificates. New connections use updated certificates. + +## ACME (Automatic Certificate Management) + +Zentinel supports automatic TLS certificate management using the ACME protocol (RFC 8555). This eliminates manual certificate management by automatically requesting, validating, and renewing certificates from ACME-compatible CAs (Let's Encrypt, ZeroSSL, Step-ca, etc.). + +### Basic ACME Configuration + +```kdl +listener "https" { + address "0.0.0.0:443" + protocol "https" + tls { + acme { + email "admin@example.com" + domains "example.com" "www.example.com" + } + } +} +``` + +With ACME enabled, Zentinel will: +1. Create or restore an ACME account (Let's Encrypt by default, or a custom CA) +2. Request certificates for configured domains +3. Complete HTTP-01 or DNS-01 domain validation automatically +4. Store certificates securely on disk +5. Renew certificates before expiration +6. Hot-reload certificates without proxy restart + +### ACME Options Reference + +```kdl +listener "https" { + address "0.0.0.0:443" + protocol "https" + tls { + acme { + // Required + email "admin@example.com" + domains "example.com" "www.example.com" + + // Optional + staging #false // Use staging environment for testing + storage "/var/lib/zentinel/acme" // Certificate storage directory + renew-before-days 30 // Days before expiry to renew + key-type "ecdsa-p256" // Key type: ecdsa-p256, ecdsa-p384 + + // Custom ACME server (e.g., ZeroSSL, Step-ca) + // server-url "https://acme.zerossl.com/v2/DV90" + + // External Account Binding (required by some CAs) + // eab { + // kid "your-eab-kid" + // hmac-key "your-base64url-encoded-hmac-key" + // } + } + } +} +``` + +| Option | Type | Default | Description | +|--------|------|---------|-------------| +| `email` | string | **required** | Contact email for ACME account | +| `domains` | string[] | **required** | Domains to include in certificate | +| `server-url` | string | - | Custom ACME directory URL (e.g., ZeroSSL, Step-ca) | +| `staging` | bool | `false` | Use Let's Encrypt staging environment (ignored if `server-url` is set) | +| `eab` | block | - | External Account Binding credentials (required by some CAs) | +| `storage` | path | `/var/lib/zentinel/acme` | Directory for certificates and credentials | +| `renew-before-days` | u32 | `30` | Days before expiry to trigger renewal | +| `challenge-type` | string | `"http-01"` | Challenge type: `http-01` or `dns-01` | +| `key-type` | string | `"ecdsa-p256"` | Certificate key type: `ecdsa-p256`, `ecdsa-p384` | +| `dns-provider` | block | - | DNS provider config (required for `dns-01`) | + +### HTTP-01 Challenge (Default) + +ACME uses HTTP-01 challenges to validate domain ownership. Zentinel automatically handles these challenges by serving responses at `/.well-known/acme-challenge/`. + +**Requirements:** +- Port 80 must be accessible from the internet +- DNS must point to the server running Zentinel +- Firewall must allow incoming HTTP traffic + +For HTTP-01 challenges to work, you typically need an HTTP listener on port 80: + +```kdl +listeners { + // HTTP listener for ACME challenges (and optional redirect) + listener "http" { + address "0.0.0.0:80" + protocol "http" + } + + // HTTPS listener with ACME + listener "https" { + address "0.0.0.0:443" + protocol "https" + tls { + acme { + email "admin@example.com" + domains "example.com" + } + } + } +} +``` + +### DNS-01 Challenge (For Wildcard Certificates) + +DNS-01 challenges validate domain ownership by creating TXT records in DNS. This is **required for wildcard certificates** and works even when port 80 is not accessible. + +```kdl +listener "https" { + address "0.0.0.0:443" + protocol "https" + tls { + acme { + email "admin@example.com" + domains "example.com" "*.example.com" + challenge-type "dns-01" + + dns-provider { + type "hetzner" + credentials-file "/etc/zentinel/secrets/hetzner-dns.json" + api-timeout-secs 30 + + propagation { + initial-delay-secs 10 + check-interval-secs 5 + timeout-secs 120 + nameservers "8.8.8.8" "1.1.1.1" + } + } + } + } +} +``` + +**DNS-01 Flow:** +1. Zentinel creates a TXT record at `_acme-challenge.example.com` +2. Waits for DNS propagation (checks against configured nameservers) +3. Notifies Let's Encrypt to validate +4. Cleans up TXT records after validation + +**Supported DNS Providers:** + +| Provider | Type | Description | +|----------|------|-------------| +| Cloudflare | `cloudflare` | Cloudflare DNS API v4 | +| Hetzner | `hetzner` | Hetzner DNS API | +| Webhook | `webhook` | Generic webhook for custom integrations | + +#### Cloudflare DNS Provider + +```kdl +dns-provider { + type "cloudflare" + credentials-file "/etc/zentinel/secrets/cloudflare-token.txt" + api-timeout-secs 30 + + propagation { + initial-delay-secs 20 + check-interval-secs 10 + timeout-secs 300 + nameservers "1.1.1.1" "8.8.8.8" + } +} +``` + +The token needs **Zone.DNS:Edit** and **Zone.Zone:Read** permissions. Credential file is plain text (the token itself) or JSON `{"token": "..."}`. + +Zone IDs are resolved and cached automatically from the domain name. + +#### Hetzner DNS Provider + +```kdl +dns-provider { + type "hetzner" + credentials-file "/etc/zentinel/secrets/hetzner.json" + // or + credentials-env "HETZNER_DNS_TOKEN" +} +``` + +Credential file format: +```json +{"token": "your-hetzner-dns-api-token"} +``` + +#### Webhook Provider (Custom DNS) + +For custom DNS integrations, use the webhook provider: + +```kdl +dns-provider { + type "webhook" + url "https://dns-api.internal/v1" + auth-header "X-API-Key" + credentials-file "/etc/zentinel/secrets/webhook.json" +} +``` + +The webhook provider makes HTTP calls: +- `POST /records` - Create TXT record (returns `{"record_id": "..."}`) +- `DELETE /records/{record_id}` - Delete record +- `GET /domains/{domain}/supported` - Check domain support + +#### DNS Provider Options + +| Option | Type | Default | Description | +|--------|------|---------|-------------| +| `type` | string | **required** | Provider type: `cloudflare`, `hetzner`, `webhook` | +| `credentials-file` | path | - | Path to credentials JSON file | +| `credentials-env` | string | - | Environment variable with credentials | +| `api-timeout-secs` | u64 | `30` | API request timeout | +| `url` | string | - | Webhook URL (webhook provider only) | +| `auth-header` | string | - | Auth header name (webhook provider only) | + +#### Propagation Check Options + +| Option | Type | Default | Description | +|--------|------|---------|-------------| +| `initial-delay-secs` | u64 | `10` | Wait before first DNS check | +| `check-interval-secs` | u64 | `5` | Interval between checks | +| `timeout-secs` | u64 | `120` | Max time to wait for propagation | +| `nameservers` | string[] | public DNS | DNS servers to query | + +#### Credential File Formats + +**Token format (Hetzner, simple webhooks):** +```json +{"token": "your-api-token"} +``` + +**Key/Secret format:** +```json +{"api_key": "your-key", "api_secret": "your-secret"} +``` + +**Plain text (entire file is the token):** +``` +your-api-token +``` + +Security: Credential files should have mode `0600` or `0400`. + +### Custom ACME Server and EAB + +By default, Zentinel uses Let's Encrypt. To use a different ACME-compatible CA (ZeroSSL, BuyPass, Step-ca), set `server-url` to the CA's directory URL. Some CAs also require External Account Binding (EAB) credentials. + +```kdl +tls { + acme { + email "admin@example.com" + domains "example.com" "www.example.com" + + // Custom ACME directory URL + server-url "https://acme.zerossl.com/v2/DV90" + + // EAB credentials (obtain from your CA's dashboard) + eab { + kid "your-eab-kid" + hmac-key "your-base64url-encoded-hmac-key" + } + } +} +``` + +| EAB Option | Type | Description | +|------------|------|-------------| +| `kid` | string | Key ID provided by the ACME CA | +| `hmac-key` | string | HMAC key (base64url-encoded) provided by the ACME CA | + +When `server-url` is set, the `staging` option is ignored. + +### Certificate Key Type + +Zentinel allows configuring the key algorithm for ACME certificates: + +```kdl +tls { + acme { + email "admin@example.com" + domains "example.com" + key-type "ecdsa-p384" + } +} +``` + +| Value | Description | +|-------|-------------| +| `ecdsa-p256` | ECDSA with NIST P-256 curve (default, fast and widely supported) | +| `ecdsa-p384` | ECDSA with NIST P-384 curve (higher security strength) | + +Invalid values produce a config parse error. + +### Staging Environment + +Use Let's Encrypt's staging environment for testing to avoid rate limits: + +```kdl +tls { + acme { + email "admin@example.com" + domains "example.com" + staging #true // Uses staging, certificates won't be trusted by browsers + } +} +``` + +**Rate limits (production):** +- 50 certificates per registered domain per week +- 5 duplicate certificates per week +- 300 new orders per account per 3 hours + +Staging has much higher limits for testing. + +### Certificate Storage + +ACME stores certificates and account credentials on disk: + +``` +/var/lib/zentinel/acme/ +├── credentials.json # ACME account credentials (keep secure) +├── account.json # Account metadata +└── domains/ + └── example.com/ + ├── cert.pem # Certificate chain + ├── key.pem # Private key (mode 0600) + └── meta.json # Expiry, issued date, domains +``` + +**Security:** +- Storage directory created with mode `0700` +- Private keys stored with mode `0600` +- Keep `credentials.json` secure—it contains your account private key + +### Certificate Renewal + +Certificates are automatically renewed when they're within `renew-before-days` of expiration: + +- Default renewal window: 30 days before expiry +- Renewal checks run every 12 hours +- Let's Encrypt certificates are valid for 90 days +- After renewal, certificates are hot-reloaded without restart + +### Combining ACME with Manual Certificates + +You can use ACME alongside manually managed certificates: + +```kdl +listener "https" { + address "0.0.0.0:443" + protocol "https" + tls { + // Manual certificates (takes precedence if both exist) + cert-file "/etc/zentinel/certs/manual.crt" + key-file "/etc/zentinel/certs/manual.key" + + // ACME for automatic management + acme { + email "admin@example.com" + domains "auto.example.com" + } + + // SNI for domain-specific certificates + additional-certs { + sni-cert { + hostnames "api.example.com" + cert-file "/etc/zentinel/certs/api.crt" + key-file "/etc/zentinel/certs/api.key" + } + } + } +} +``` + +When both ACME and manual certificates are configured, manual certificates are used if present. ACME certificates are stored and used as fallback or for specified domains. + +### Multi-Domain Certificates + +Request a single certificate covering multiple domains: + +```kdl +tls { + acme { + email "admin@example.com" + domains "example.com" "www.example.com" "api.example.com" "cdn.example.com" + } +} +``` + +All domains must pass HTTP-01 validation and point to the server. + +### ACME Troubleshooting + +#### Challenge Failed + +``` +Error: ACME challenge validation failed for domain 'example.com' +``` + +- Verify DNS points to this server: `dig +short example.com` +- Ensure port 80 is accessible from the internet +- Check firewall allows incoming HTTP traffic +- Verify no other service is handling `/.well-known/acme-challenge/` + +#### Rate Limit Exceeded + +``` +Error: Rate limit exceeded +``` + +- Wait for the rate limit window to reset (typically 1 week) +- Use `staging true` for testing +- Consolidate multiple domains into one certificate request + +#### Storage Permission Denied + +``` +Error: Permission denied writing to storage directory +``` + +- Ensure the Zentinel process has write access to the storage directory +- Check directory ownership: `chown zentinel:zentinel /var/lib/zentinel/acme` +- Verify parent directories exist and are accessible + +#### Certificate Not Renewing + +Check the Zentinel logs for renewal status. Renewals are attempted: +- Every 12 hours (check interval) +- When certificate is within `renew-before-days` of expiry + +Manually trigger a reload to force renewal check: +```bash +kill -HUP $(cat /var/run/zentinel.pid) +``` + +## Multiple Listeners + +Run multiple listeners for different purposes: + +```kdl +listeners { + // Public HTTPS + listener "public" { + address "0.0.0.0:443" + protocol "https" + request-timeout-secs 30 + tls { + cert-file "/etc/zentinel/certs/public.crt" + key-file "/etc/zentinel/certs/public.key" + min-version "1.2" + } + } + + // HTTP redirect to HTTPS + listener "http-redirect" { + address "0.0.0.0:80" + protocol "http" + default-route "https-redirect" + } + + // Admin interface (localhost only) + listener "admin" { + address "127.0.0.1:9090" + protocol "http" + } + + // Internal mTLS API + listener "internal" { + address "10.0.0.5:8443" + protocol "https" + tls { + cert-file "/etc/zentinel/certs/internal.crt" + key-file "/etc/zentinel/certs/internal.key" + ca-file "/etc/zentinel/certs/internal-ca.crt" + client-auth #true + } + } +} +``` + +## Certificate Management + +### Certificate Formats + +Zentinel accepts PEM-encoded certificates and keys: + +``` +/etc/zentinel/certs/ +├── server.crt # Certificate (PEM) +├── server.key # Private key (PEM) +├── chain.crt # Intermediate certificates (optional) +└── ca.crt # CA certificate for client auth +``` + +### Full Chain Certificates + +For proper certificate chain validation, include intermediates in the cert file: + +```bash +cat server.crt intermediate.crt > fullchain.crt +``` + +Then reference the full chain: + +```kdl +tls { + cert-file "/etc/zentinel/certs/fullchain.crt" + key-file "/etc/zentinel/certs/server.key" +} +``` + +### Certificate Reload + +Certificates are reloaded on configuration reload (SIGHUP): + +```bash +# Update certificates, then reload +cp new-cert.crt /etc/zentinel/certs/server.crt +cp new-key.key /etc/zentinel/certs/server.key +kill -HUP $(cat /var/run/zentinel.pid) +``` + +## Complete Example + +```kdl +system { + worker-threads 0 +} + +listeners { + // Production HTTPS with modern TLS + listener "https" { + + tls { + cert-file "/etc/zentinel/certs/fullchain.crt" + key-file "/etc/zentinel/certs/server.key" + min-version "1.2" + max-version "1.3" + ocsp-stapling #true + session-resumption #true + } + address "0.0.0.0:443" + protocol "https" + request-timeout-secs 60 + keepalive-timeout-secs 120 + max-concurrent-streams 200 + + } + + // HTTP to HTTPS redirect + listener "http" { + address "0.0.0.0:80" + protocol "http" + request-timeout-secs 5 + default-route "redirect-https" + } + + // Admin and metrics (internal only) + listener "admin" { + address "127.0.0.1:9090" + protocol "http" + request-timeout-secs 10 + } +} + +routes { + route "default" { + matches { path-prefix "/" } + upstream "backend" + } +} + +upstreams { + upstream "backend" { + targets { + target { address "127.0.0.1:3000" } + } + } +} + +``` + +## Default Values + +| Setting | Default | +|---------|---------| +| `request-timeout-secs` | `60` | +| `keepalive-timeout-secs` | `75` | +| `max-concurrent-streams` | `100` | +| `tls.min-version` | `1.2` | +| `tls.ocsp-stapling` | `true` | +| `tls.session-resumption` | `true` | +| `tls.client-auth` | `false` | +| `tls.acme.staging` | `false` | +| `tls.acme.storage` | `/var/lib/zentinel/acme` | +| `tls.acme.renew-before-days` | `30` | +| `tls.acme.challenge-type` | `"http-01"` | +| `tls.acme.dns-provider.api-timeout-secs` | `30` | +| `tls.acme.dns-provider.propagation.initial-delay-secs` | `10` | +| `tls.acme.dns-provider.propagation.check-interval-secs` | `5` | +| `tls.acme.dns-provider.propagation.timeout-secs` | `120` | + +## Troubleshooting + +### Port Already in Use + +``` +Error: Address already in use (os error 98) +``` + +Another process is using the port: + +```bash +# Find what's using the port +lsof -i :8080 +# or +ss -tlnp | grep 8080 +``` + +### Permission Denied (Privileged Ports) + +``` +Error: Permission denied (os error 13) +``` + +Ports below 1024 require root or capabilities: + +```bash +# Option 1: Run as root (not recommended) +sudo zentinel + +# Option 2: Grant capability (recommended) +sudo setcap cap_net_bind_service=+ep /usr/local/bin/zentinel + +# Option 3: Use user/group in config +system { + user "zentinel" + group "zentinel" +} +``` + +### Certificate Issues + +``` +Error: Invalid certificate chain +``` + +- Verify certificate format is PEM +- Include intermediate certificates in cert file +- Check certificate dates: `openssl x509 -in cert.crt -noout -dates` +- Verify key matches certificate: `openssl x509 -noout -modulus -in cert.crt | md5sum` vs `openssl rsa -noout -modulus -in key.key | md5sum` + +## Next Steps + +- [Routes](../routes/) - Request routing rules +- [Upstreams](../upstreams/) - Backend server configuration diff --git a/content/v/26.04/configuration/namespaces.md b/content/v/26.04/configuration/namespaces.md new file mode 100644 index 0000000..0e64409 --- /dev/null +++ b/content/v/26.04/configuration/namespaces.md @@ -0,0 +1,619 @@ ++++ +title = "Namespaces & Services" +weight = 11 +updated = 2026-02-19 ++++ + +Namespaces and services provide hierarchical organization for Zentinel configuration, enabling multi-tenant deployments with runtime isolation. + +## Overview + +Zentinel supports three scope levels: + +| Scope | Description | Use Case | +|-------|-------------|----------| +| **Global** | Root-level resources visible everywhere | Shared infrastructure | +| **Namespace** | Grouped resources with isolation | Teams, environments, domains | +| **Service** | Fine-grained resources within a namespace | Individual microservices | + +## Basic Namespace + +```kdl +namespace "api" { + upstreams { + upstream "backend" { + targets { + target { address "10.0.1.1:8080" } + } + } + } + + routes { + route "main" { + matches { + path-prefix "/api/" + } + upstream "backend" + } + } +} +``` + +## Namespace with Limits + +Each namespace can have its own limits for isolation: + +```kdl +namespace "public-api" { + limits { + max-body-size-bytes 1048576 // 1MB + max-requests-per-second-global 1000 + max-requests-per-second-per-client 50 + } + + upstreams { + upstream "api-backend" { ... } + } + + routes { + route "public" { + matches { + path-prefix "/v1/" + } + upstream "api-backend" + } + } +} + +namespace "internal-api" { + limits { + max-body-size-bytes 104857600 // 100MB + max-requests-per-second-global 10000 + // No per-client limit for internal services + } + + upstreams { + upstream "internal-backend" { ... } + } + + routes { + route "internal" { + matches { + path-prefix "/internal/" + } + upstream "internal-backend" + } + } +} +``` + +## Services Within Namespaces + +Services provide finer-grained isolation within a namespace: + +```kdl +namespace "payments" { + limits { + max-body-size-bytes 10485760 // 10MB default + } + + // Shared upstream for the namespace + upstreams { + upstream "shared-db" { ... } + } + + service "checkout" { + limits { + max-requests-per-second-global 500 + } + + listener { + address "0.0.0.0:8443" + protocol "https" + tls { + cert-file "/etc/zentinel/certs/checkout.crt" + key-file "/etc/zentinel/certs/checkout.key" + } + } + + upstreams { + upstream "checkout-backend" { + targets { + target { address "checkout-1:8080" } + target { address "checkout-2:8080" } + } + } + } + + routes { + route "process" { + matches { + path-prefix "/checkout/" + } + upstream "checkout-backend" + } + } + } + + service "refunds" { + limits { + max-requests-per-second-global 100 // Lower limit for refunds + } + + upstreams { + upstream "refunds-backend" { ... } + } + + routes { + route "process" { + matches { + path-prefix "/refunds/" + } + upstream "refunds-backend" + } + } + } +} +``` + +## Scope Resolution + +When a route references an upstream, Zentinel resolves it in order: + +1. **Service scope** - Resources in the same service +2. **Namespace scope** - Resources in the parent namespace +3. **Exported resources** - Resources exported from other namespaces +4. **Global scope** - Root-level resources + +### Resolution Example + +```kdl +// Global upstream (available everywhere) +upstreams { + upstream "shared-auth" { + targets { target { address "auth:8080" } } + } +} + +namespace "api" { + // Namespace-level upstream + upstreams { + upstream "backend" { + targets { target { address "api-backend:8080" } } + } + } + + routes { + route "main" { + upstream "backend" // Resolves to api:backend + } + route "auth" { + upstream "shared-auth" // Resolves to global shared-auth + } + } + + service "users" { + upstreams { + upstream "backend" { // Shadows namespace backend + targets { target { address "users-backend:8080" } } + } + } + + routes { + route "list" { + upstream "backend" // Resolves to api:users:backend + } + } + } +} +``` + +## Qualified References + +Use qualified names to explicitly reference resources from other scopes: + +```kdl +namespace "frontend" { + routes { + route "api-proxy" { + matches { + path-prefix "/api/" + } + // Explicit reference to 'api' namespace + upstream "api:backend" + } + } +} +``` + +### Reference Formats + +| Format | Scope | Example | +|--------|-------|---------| +| `name` | Current scope chain | `upstream "backend"` | +| `namespace:name` | Specific namespace | `upstream "api:backend"` | +| `namespace:service:name` | Specific service | `upstream "api:users:backend"` | + +## Exporting Resources + +Make namespace resources available globally: + +```kdl +namespace "infrastructure" { + upstreams { + upstream "redis" { + targets { target { address "redis:6379" } } + } + upstream "postgres" { + targets { target { address "postgres:5432" } } + } + } + + // Export these upstreams for use by other namespaces + exports { + upstreams "redis" "postgres" + } +} + +namespace "api" { + routes { + route "cached" { + upstream "redis" // Resolves via export + } + } +} +``` + +### Export Configuration + +```kdl +exports { + upstreams "upstream-1" "upstream-2" + agents "auth-agent" + filters "rate-limit" "cors" +} +``` + +## Runtime Isolation + +Each namespace/service has isolated: + +### Rate Limiting + +```kdl +namespace "api" { + limits { + max-requests-per-second-global 5000 + } + + service "public" { + limits { + max-requests-per-second-global 1000 + max-requests-per-second-per-client 50 + } + } + + service "partner" { + limits { + max-requests-per-second-global 2000 + max-requests-per-second-per-client 200 + } + } +} +``` + +Rate limits are enforced independently per scope. A rate limit hit in `api:public` does not affect `api:partner`. + +### Circuit Breakers + +Circuit breakers are isolated per scope. An upstream failure in one namespace does not trip circuit breakers in other namespaces. + +```kdl +namespace "critical" { + upstreams { + upstream "backend" { + health-check { + type "http" { path "/health" } + } + circuit-breaker { + failure-threshold 3 + success-threshold 2 + timeout-secs 30 + } + } + } +} + +namespace "best-effort" { + upstreams { + upstream "backend" { + circuit-breaker { + failure-threshold 10 // More tolerant + timeout-secs 10 + } + } + } +} +``` + +### Metrics + +Scoped metrics include `namespace` and `service` labels: + +``` +zentinel_scoped_requests_total{namespace="api", service="users", route="list", status="200"} +zentinel_scoped_request_duration_seconds{namespace="api", service="users", route="list"} +zentinel_scoped_rate_limit_hits_total{namespace="api", service="public", route="main"} +zentinel_scoped_circuit_breaker_state{namespace="payments", service="checkout", upstream="backend"} +``` + +### Access Logs + +Access logs include scope information in JSON format: + +```json +{ + "timestamp": "2024-01-15T10:30:00Z", + "trace_id": "2kF8xQw4BnM", + "method": "POST", + "path": "/checkout/process", + "status": 200, + "namespace": "payments", + "service": "checkout", + "route_id": "process", + "upstream": "checkout-backend" +} +``` + +## Migration from Flat Configuration + +Existing flat configurations continue to work unchanged. All resources are treated as global scope. + +### Before (Flat) + +```kdl +upstreams { + upstream "api-backend" { ... } + upstream "web-backend" { ... } +} + +routes { + route "api" { + upstream "api-backend" + } + route "web" { + upstream "web-backend" + } +} +``` + +### After (Namespaced) + +```kdl +// Shared infrastructure remains global +upstreams { + upstream "shared-auth" { ... } +} + +namespace "api" { + upstreams { + upstream "backend" { ... } // Renamed from api-backend + } + + routes { + route "main" { + upstream "backend" + // Can still access global: upstream "shared-auth" + } + } +} + +namespace "web" { + upstreams { + upstream "backend" { ... } // Same local name, different scope + } + + routes { + route "main" { + upstream "backend" + } + } +} +``` + +## Complete Example + +```kdl +// Global configuration +system { + worker-threads 0 + trace-id-format "tinyflake" +} + +// Global shared resources +upstreams { + upstream "auth-service" { + targets { + target { address "auth-1:8080" } + target { address "auth-2:8080" } + } + load-balancing "round_robin" + } +} + +// API namespace +namespace "api" { + limits { + max-body-size-bytes 10485760 + max-requests-per-second-global 10000 + } + + upstreams { + upstream "backend" { + targets { + target { address "api-1:8080" } + target { address "api-2:8080" } + } + } + } + + routes { + route "main" { + matches { + path-prefix "/api/v1/" + } + upstream "backend" + } + + route "auth" { + matches { + path-prefix "/api/auth/" + } + upstream "auth-service" // Global + } + } + + service "users" { + limits { + max-requests-per-second-per-client 100 + } + + upstreams { + upstream "users-backend" { + targets { + target { address "users-1:8080" } + } + } + } + + routes { + route "crud" { + matches { + path-prefix "/api/v1/users/" + } + upstream "users-backend" + } + } + } + + exports { + upstreams "backend" + } +} + +// Web namespace +namespace "web" { + listeners { + listener "https" { + address "0.0.0.0:443" + protocol "https" + tls { + cert-file "/etc/zentinel/certs/web.crt" + key-file "/etc/zentinel/certs/web.key" + } + } + } + + upstreams { + upstream "frontend" { + targets { + target { address "web-1:3000" } + } + } + } + + routes { + route "static" { + matches { + path-prefix "/" + } + upstream "frontend" + } + + route "api-proxy" { + matches { + path-prefix "/api/" + } + upstream "api:backend" // Cross-namespace reference + } + } +} + +// Observability +observability { + metrics { + enabled #true + address "0.0.0.0:9090" + } +} +``` + +## Validation + +Zentinel validates namespace configuration: + +- Unique IDs within each scope +- Valid cross-namespace references +- No circular dependencies in exports +- Reserved character (`:`) not used in resource IDs + +```bash +zentinel --config zentinel.kdl --validate +``` + +## Best Practices + +### Naming Conventions + +```kdl +// Use descriptive, hierarchical names +namespace "payments" { ... } +namespace "users" { ... } + +// Within namespaces, use simple names +service "api" { ... } +service "worker" { ... } + +// Upstreams can use generic names within their scope +upstream "backend" { ... } // Clear within api:users context +``` + +### Scope Organization + +| Level | Use For | +|-------|---------| +| Global | Shared infrastructure (auth, logging, metrics) | +| Namespace | Team boundaries, environments, domains | +| Service | Individual microservices, isolated workloads | + +### When to Use Services + +Use services when you need: +- Dedicated listeners with separate TLS certificates +- Independent rate limits for subcomponents +- Fine-grained circuit breaker isolation +- Separate metrics dashboards + +### Export Sparingly + +Only export resources that genuinely need cross-namespace access: + +```kdl +namespace "infrastructure" { + upstreams { + upstream "redis" { ... } + upstream "postgres" { ... } + upstream "internal-tool" { ... } // Don't export + } + + exports { + upstreams "redis" "postgres" // Only shared infra + } +} +``` + +## Next Steps + +- [Limits](../limits/) - Configure per-scope limits +- [Upstreams](../upstreams/) - Backend pool configuration +- [Routes](../routes/) - Request matching and routing diff --git a/content/v/26.04/configuration/observability.md b/content/v/26.04/configuration/observability.md new file mode 100644 index 0000000..0d65110 --- /dev/null +++ b/content/v/26.04/configuration/observability.md @@ -0,0 +1,644 @@ ++++ +title = "Observability" +weight = 8 +updated = 2026-02-25 ++++ + +Zentinel provides comprehensive observability through metrics, logging, and distributed tracing. All observability features are configured in the `observability` block. + +## Basic Configuration + +```kdl +observability { + logging { + level "info" + format "json" + + access-log { + enabled #true + file "/var/log/zentinel/access.log" + format "json" + } + + error-log { + enabled #true + file "/var/log/zentinel/error.log" + level "warn" + } + + audit-log { + enabled #true + file "/var/log/zentinel/audit.log" + log-blocked #true + log-agent-decisions #true + log-waf-events #true + } + } + + metrics { + enabled #true + address "0.0.0.0:9090" + path "/metrics" + } + + tracing { + backend "otlp" { + endpoint "http://jaeger:4317" + } + sampling-rate 0.01 + service-name "zentinel" + } +} +``` + +## Logging + +### Application Logging + +Configure the main application log output: + +```kdl +observability { + logging { + level "info" // Log level + format "json" // Log format + timestamps #true // Include timestamps + file "/var/log/zentinel/app.log" // Optional file path + } +} +``` + +#### Log Levels + +| Level | Description | +|-------|-------------| +| `trace` | Very detailed debugging | +| `debug` | Debugging information | +| `info` | Informational messages (default) | +| `warn` | Warnings | +| `error` | Errors only | + +#### Log Formats + +| Format | Description | +|--------|-------------| +| `json` | Structured JSON (default, recommended for production) | +| `pretty` | Human-readable format | + +### Access Log + +HTTP request/response logging: + +```kdl +observability { + logging { + access-log { + enabled #true + file "/var/log/zentinel/access.log" + format "json" + buffer-size 8192 + include-trace-id #true + } + } +} +``` + +#### Access Log Options + +| Option | Default | Description | +|--------|---------|-------------| +| `enabled` | `true` | Enable access logging | +| `file` | `/var/log/zentinel/access.log` | Log file path | +| `format` | `json` | Log format (`json`, `combined`, `custom`) | +| `buffer-size` | `8192` | Write buffer size | +| `include-trace-id` | `true` | Include trace ID in logs | + +#### Access Log Fields (JSON format) + +```json +{ + "timestamp": "2024-01-15T10:30:45.123Z", + "trace_id": "2Kj8mNpQ3xR", + "method": "GET", + "path": "/api/v1/users", + "status": 200, + "duration_ms": 45, + "bytes_sent": 1234, + "client_ip": "192.168.1.100", + "user_agent": "Mozilla/5.0...", + "upstream": "api-backend", + "upstream_addr": "10.0.1.5:8080", + "upstream_duration_ms": 42, + "cache_status": "HIT", + "route_id": "api-users" +} +``` + +### Error Log + +Error and warning logging. **Error logging is enabled by default** — even without explicit configuration, Zentinel writes errors and warnings to `/var/log/zentinel/error.log`. The directory is created automatically if it doesn't exist. + +To customize: + +```kdl +observability { + logging { + error-log { + enabled #true + file "/var/log/zentinel/error.log" + level "warn" + buffer-size 8192 + } + } +} +``` + +To disable error file logging: + +```kdl +observability { + logging { + error-log { + enabled #false + } + } +} +``` + +#### Error Log Options + +| Option | Default | Description | +|--------|---------|-------------| +| `enabled` | `true` | Enable error logging | +| `file` | `/var/log/zentinel/error.log` | Log file path | +| `level` | `warn` | Minimum level to log (`warn` or `error`) | +| `buffer-size` | `8192` | Write buffer size | + +### Audit Log + +Security-focused logging for compliance and forensics: + +```kdl +observability { + logging { + audit-log { + enabled #true + file "/var/log/zentinel/audit.log" + buffer-size 8192 + log-blocked #true + log-agent-decisions #true + log-waf-events #true + } + } +} +``` + +#### Audit Log Options + +| Option | Default | Description | +|--------|---------|-------------| +| `enabled` | `true` | Enable audit logging | +| `file` | `/var/log/zentinel/audit.log` | Log file path | +| `buffer-size` | `8192` | Write buffer size | +| `log-blocked` | `true` | Log blocked requests | +| `log-agent-decisions` | `true` | Log agent allow/deny decisions | +| `log-waf-events` | `true` | Log WAF rule matches | + +#### Audit Log Events + +```json +{ + "timestamp": "2024-01-15T10:30:45.123Z", + "event_type": "request_blocked", + "trace_id": "2Kj8mNpQ3xR", + "client_ip": "192.168.1.100", + "method": "POST", + "path": "/api/v1/admin", + "reason": "rate_limit_exceeded", + "rule_id": "rate-limit-api", + "action": "block", + "metadata": { + "limit": 100, + "current": 101 + } +} +``` + +## Metrics + +Prometheus-compatible metrics endpoint: + +```kdl +observability { + metrics { + enabled #true + address "0.0.0.0:9090" + path "/metrics" + high-cardinality #false + } +} +``` + +### Metrics Options + +| Option | Default | Description | +|--------|---------|-------------| +| `enabled` | `true` | Enable metrics endpoint | +| `address` | `0.0.0.0:9090` | Metrics server address | +| `path` | `/metrics` | Metrics endpoint path | +| `high-cardinality` | `false` | Include high-cardinality labels | + +### Available Metrics + +#### Request Metrics + +| Metric | Type | Description | +|--------|------|-------------| +| `zentinel_requests_total` | Counter | Total requests by route, method, status | +| `zentinel_request_duration_seconds` | Histogram | Request latency distribution | +| `zentinel_request_size_bytes` | Histogram | Request body size | +| `zentinel_response_size_bytes` | Histogram | Response body size | +| `zentinel_active_requests` | Gauge | Currently active requests | + +#### Upstream Metrics + +| Metric | Type | Description | +|--------|------|-------------| +| `zentinel_upstream_requests_total` | Counter | Requests to upstreams | +| `zentinel_upstream_duration_seconds` | Histogram | Upstream latency | +| `zentinel_upstream_healthy_backends` | Gauge | Healthy backends per upstream | +| `zentinel_upstream_connections` | Gauge | Active upstream connections | +| `zentinel_upstream_retries_total` | Counter | Retry attempts | + +#### Cache Metrics + +| Metric | Type | Description | +|--------|------|-------------| +| `zentinel_cache_hits_total` | Counter | Cache hits | +| `zentinel_cache_misses_total` | Counter | Cache misses | +| `zentinel_cache_size_bytes` | Gauge | Current cache size | +| `zentinel_cache_entries` | Gauge | Number of cached entries | +| `zentinel_cache_evictions_total` | Counter | Cache evictions | + +#### Rate Limiting Metrics + +| Metric | Type | Description | +|--------|------|-------------| +| `zentinel_rate_limit_hits_total` | Counter | Rate limit triggers | +| `zentinel_rate_limit_allowed_total` | Counter | Allowed requests | +| `zentinel_rate_limit_delayed_total` | Counter | Delayed requests | + +#### Agent Metrics + +| Metric | Type | Description | +|--------|------|-------------| +| `zentinel_agent_requests_total` | Counter | Agent call count | +| `zentinel_agent_duration_seconds` | Histogram | Agent call latency | +| `zentinel_agent_errors_total` | Counter | Agent errors | +| `zentinel_agent_timeouts_total` | Counter | Agent timeouts | +| `zentinel_agent_circuit_breaker_state` | Gauge | Circuit breaker state (0=closed, 1=open, 2=half-open) | + +#### Connection Metrics + +| Metric | Type | Description | +|--------|------|-------------| +| `zentinel_connections_total` | Counter | Total connections | +| `zentinel_active_connections` | Gauge | Current connections | +| `zentinel_connection_duration_seconds` | Histogram | Connection lifetime | +| `zentinel_tls_handshake_duration_seconds` | Histogram | TLS handshake time | + +### Prometheus Scrape Config + +```yaml +scrape_configs: + - job_name: 'zentinel' + static_configs: + - targets: ['zentinel:9090'] + scrape_interval: 15s + metrics_path: /metrics +``` + +## Distributed Tracing + +Zentinel provides OpenTelemetry-compatible distributed tracing for end-to-end visibility across your services. Traces show the complete request journey through the proxy, including agent processing and upstream calls. + +> **Note**: Tracing requires building Zentinel with the `opentelemetry` feature flag: +> ```bash +> cargo build --release --features opentelemetry +> ``` + +### Basic Configuration + +```kdl +observability { + tracing { + backend "otlp" { + endpoint "http://jaeger:4317" + } + sampling-rate 0.1 // 10% of requests + service-name "zentinel" + } +} +``` + +### Tracing Backends + +#### OTLP (OpenTelemetry Protocol) - Recommended + +The OpenTelemetry Protocol (OTLP) is the standard for sending telemetry data. Use this with Jaeger, Tempo, or any OTLP-compatible backend: + +```kdl +tracing { + backend "otlp" { + endpoint "http://otel-collector:4317" // gRPC endpoint + } + sampling-rate 0.1 + service-name "zentinel-prod" +} +``` + +#### Jaeger (Direct) + +```kdl +tracing { + backend "jaeger" { + endpoint "http://jaeger:14268/api/traces" + } +} +``` + +#### Zipkin + +```kdl +tracing { + backend "zipkin" { + endpoint "http://zipkin:9411/api/v2/spans" + } +} +``` + +### Tracing Options + +| Option | Default | Description | +|--------|---------|-------------| +| `sampling-rate` | `0.01` | Fraction of requests to trace (0.0 to 1.0) | +| `service-name` | `zentinel` | Service name shown in trace UI | + +#### Sampling Rate Guidelines + +| Environment | Recommended Rate | Notes | +|-------------|------------------|-------| +| Development | `1.0` | Trace all requests | +| Staging | `0.1` - `0.5` | 10-50% sampling | +| Production (low traffic) | `0.05` - `0.1` | 5-10% sampling | +| Production (high traffic) | `0.01` - `0.05` | 1-5% sampling | + +### Trace Propagation + +#### W3C Trace Context (Standard) + +Zentinel implements [W3C Trace Context](https://www.w3.org/TR/trace-context/) for distributed tracing propagation: + +| Header | Format | Description | +|--------|--------|-------------| +| `traceparent` | `{version}-{trace-id}-{parent-id}-{flags}` | Trace context parent | +| `tracestate` | Vendor-specific key-value pairs | Trace context state | + +Example `traceparent` header: +``` +00-0af7651916cd43dd8448eb211c80319c-b7ad6b7169203331-01 +│ │ │ │ +│ │ │ └─ Flags (01 = sampled) +│ │ └─ Parent Span ID (16 hex chars) +│ └─ Trace ID (32 hex chars) +└─ Version (00) +``` + +#### Upstream Propagation + +When proxying to upstream services, Zentinel: +1. Parses incoming `traceparent` header (if present) +2. Creates a child span for the request +3. Propagates `traceparent` with the new span ID to upstream + +This enables end-to-end tracing across your service mesh. + +#### Agent Propagation + +Agents receive the `traceparent` in the `RequestMetadata`: + +```json +{ + "metadata": { + "correlation_id": "2Kj8mNpQ3xR", + "traceparent": "00-0af7651916cd43dd8448eb211c80319c-b7ad6b7169203331-01", + ... + } +} +``` + +Agents can use this to create child spans for their processing, enabling visibility into agent latency in your traces. + +### Request Span Lifecycle + +Each request creates a span with the following lifecycle: + +1. **Start**: Span created when request headers are received +2. **Upstream**: `traceparent` propagated to upstream service +3. **Response**: Status code recorded on span +4. **End**: Span completed when response is sent to client + +### Span Attributes + +Each request span includes semantic convention attributes: + +| Attribute | Description | +|-----------|-------------| +| `http.method` | HTTP method (GET, POST, etc.) | +| `http.target` | Request path | +| `http.status_code` | Response status code | +| `service.name` | Configured service name | + +### Testing with Jaeger + +Quick start with Jaeger all-in-one: + +```bash +# Start Jaeger +docker run -d --name jaeger \ + -p 4317:4317 \ + -p 16686:16686 \ + jaegertracing/all-in-one:latest + +# Run Zentinel with tracing +cargo run --features opentelemetry -- --config zentinel.kdl + +# View traces +open http://localhost:16686 +``` + +### Testing with Grafana Tempo + +For production-grade tracing with Grafana: + +```yaml +# docker-compose.yml +services: + tempo: + image: grafana/tempo:latest + command: ["-config.file=/etc/tempo.yaml"] + volumes: + - ./tempo.yaml:/etc/tempo.yaml + ports: + - "4317:4317" # OTLP gRPC + - "3200:3200" # Tempo API + + grafana: + image: grafana/grafana:latest + ports: + - "3000:3000" + environment: + - GF_AUTH_ANONYMOUS_ENABLED=true + - GF_AUTH_ANONYMOUS_ORG_ROLE=Admin +``` + +Configure Zentinel: +```kdl +tracing { + backend "otlp" { + endpoint "http://tempo:4317" + } + sampling-rate 0.1 + service-name "zentinel" +} +``` + +### Connecting Logs and Traces + +Include trace IDs in access logs for log-to-trace correlation: + +```kdl +observability { + logging { + access-log { + enabled #true + format "json" + include-trace-id #true // Adds trace_id to log entries + } + } + tracing { + backend "otlp" { endpoint "http://tempo:4317" } + sampling-rate 0.1 + service-name "zentinel" + } +} +``` + +Log entries will include the trace ID: +```json +{ + "timestamp": "2024-01-15T10:30:45.123Z", + "trace_id": "0af7651916cd43dd8448eb211c80319c", + "method": "GET", + "path": "/api/users", + "status": 200 +} +``` + +In Grafana, you can then jump from logs to traces using the trace ID. + +## Trace ID Format + +Configure the format for request trace IDs: + +```kdl +system { + trace-id-format "tinyflake" // or "uuid" +} +``` + +| Format | Example | Description | +|--------|---------|-------------| +| `tinyflake` | `2Kj8mNpQ3xR` | 11-char Base58, operator-friendly (default) | +| `uuid` | `550e8400-e29b-41d4-a716-446655440000` | 36-char UUID v4 | + +## Complete Example + +```kdl +system { + worker-threads 0 + trace-id-format "tinyflake" +} + +observability { + logging { + level "info" + format "json" + + access-log { + enabled #true + file "/var/log/zentinel/access.log" + format "json" + include-trace-id #true + } + + error-log { + enabled #true + file "/var/log/zentinel/error.log" + level "warn" + } + + audit-log { + enabled #true + file "/var/log/zentinel/audit.log" + log-blocked #true + log-agent-decisions #true + } + } + + metrics { + enabled #true + address "0.0.0.0:9090" + path "/metrics" + } + + tracing { + backend "otlp" { + endpoint "http://otel-collector:4317" + } + sampling-rate 0.05 + service-name "zentinel-prod" + } +} +``` + +## Log Rotation + +Zentinel logs are designed for external rotation. Use logrotate or similar: + +``` +/var/log/zentinel/*.log { + daily + rotate 30 + compress + delaycompress + missingok + notifempty + copytruncate +} +``` + +## Best Practices + +1. **Use JSON logging in production**: Enables log aggregation and analysis +2. **Set appropriate log levels**: `info` for production, `debug` for troubleshooting +3. **Enable audit logging**: Required for security compliance +4. **Configure sampling for tracing**: 1-5% is typical for production +5. **Use separate log files**: Easier rotation and analysis +6. **Monitor metrics endpoints**: Set up alerting on error rates and latencies + +## Next Steps + +- [Operations](../../operations/) - Operational procedures +- [Monitoring](../../deployment/monitoring/) - Monitoring setup +- [Troubleshooting](../../operations/troubleshooting/) - Debug procedures diff --git a/content/v/26.04/configuration/routes.md b/content/v/26.04/configuration/routes.md new file mode 100644 index 0000000..2e810e8 --- /dev/null +++ b/content/v/26.04/configuration/routes.md @@ -0,0 +1,1205 @@ ++++ +title = "Routes" +weight = 4 +updated = 2026-02-19 ++++ + +The `routes` block defines how incoming requests are matched and forwarded to upstreams or handlers. Routes are evaluated by priority, with higher priority routes checked first. + +## Basic Configuration + +```kdl +routes { + route "api" { + priority 100 + matches { + path-prefix "/api/" + } + upstream "backend" + } + + route "static" { + priority 50 + matches { + path-prefix "/static/" + } + service-type "static" + static-files { + root "/var/www/static" + } + } +} +``` + +## Route Options + +### Priority + +```kdl +route "api" { + priority 100 +} +``` + +Higher priority routes are evaluated first. When multiple routes could match, the highest priority wins. + +| Priority | Typical Use | +|----------|-------------| +| 1000+ | Health checks, admin endpoints | +| 100-999 | API routes | +| 50-99 | Static files | +| 1-49 | Catch-all routes | + +### Match Conditions + +Routes support multiple match conditions. All conditions within a route must match (AND logic). + +#### Path Matching + +```kdl +matches { + // Exact path match + path "/api/v1/users" + + // Prefix match + path-prefix "/api/" + + // Regex match + path-regex "^/api/v[0-9]+/.*$" +} +``` + +| Match Type | Example | Matches | +|------------|---------|---------| +| `path` | `/users` | `/users` only | +| `path-prefix` | `/api/` | `/api/`, `/api/users`, `/api/v1/data` | +| `path-regex` | `^/user/[0-9]+$` | `/user/123`, `/user/456` | + +#### Host Matching + +```kdl +matches { + host "api.example.com" +} +``` + +Match by the `Host` header. Useful for virtual hosting. + +#### Method Matching + +```kdl +matches { + method "GET" "POST" "PUT" "DELETE" +} +``` + +Match specific HTTP methods. Multiple methods are OR'd together. + +#### Header Matching + +```kdl +matches { + // Match if header exists + header name="X-Api-Key" + + // Match header with specific value + header name="X-Api-Version" value="2" +} +``` + +#### Query Parameter Matching + +```kdl +matches { + // Match if parameter exists + query-param name="debug" + + // Match parameter with value + query-param name="format" value="json" +} +``` + +### Service Types + +```kdl +route "api" { + service-type "web" // Default +} +``` + +| Type | Description | +|------|-------------| +| `web` | Standard HTTP proxy (default) | +| `api` | API service with JSON error responses | +| `static` | Static file serving | +| `builtin` | Built-in handlers | + +#### Static File Serving + +```kdl +route "assets" { + matches { + path-prefix "/static/" + } + service-type "static" + static-files { + root "/var/www/static" + index "index.html" + directory-listing #false + cache-control "public, max-age=86400" + compress #true + fallback "index.html" // For SPAs + } +} +``` + +| Option | Default | Description | +|--------|---------|-------------| +| `root` | Required | Root directory for files | +| `index` | `index.html` | Default index file | +| `directory-listing` | `false` | Enable directory browsing | +| `cache-control` | `public, max-age=3600` | Cache-Control header | +| `compress` | `true` | Enable gzip/brotli compression | +| `fallback` | None | Fallback file for 404s (SPA routing) | + +#### Built-in Handlers + +```kdl +route "health" { + priority 1000 + matches { + path "/health" + } + service-type "builtin" + builtin-handler "health" +} +``` + +| Handler | Path | Description | +|---------|------|-------------| +| `health` | `/health` | Health check (200 OK) | +| `status` | `/status` | JSON status with version/uptime | +| `metrics` | `/metrics` | Prometheus metrics | +| `not-found` | Any | 404 handler | +| `config` | `/admin/config` | Configuration dump (admin) | +| `upstreams` | `/admin/upstreams` | Upstream health status (admin) | + +#### API Schema Validation + +The `api` service type supports JSON Schema validation for requests and responses. This enables contract validation at the proxy layer using OpenAPI/Swagger specifications or inline JSON schemas. + +##### OpenAPI/Swagger File Reference + +Reference an OpenAPI 3.0 or Swagger 2.0 specification (YAML or JSON): + +```kdl +route "api-v1" { + matches { + path-prefix "/api/v1" + } + upstream "api-backend" + service-type "api" + api-schema { + schema-file "/etc/zentinel/schemas/api-v1-openapi.yaml" + validate-requests #true + validate-responses #false + strict-mode #false + } +} +``` + +The schema file is loaded at startup and used to validate requests against the paths, methods, and schemas defined in the OpenAPI specification. + +##### Inline OpenAPI/Swagger Specification + +Embed an OpenAPI specification directly in the configuration as a string: + +```kdl +system { + worker-threads 0 +} + +listeners { + listener "http" { + address "0.0.0.0:8080" + protocol "http" + } +} + +route "api-v1" { + matches { + path-prefix "/api/v1" + } + upstream "api-backend" +} + +routes { + route "default" { + matches { path-prefix "/" } + upstream "backend" + } +} + +upstreams { + upstream "backend" { + targets { + target { address "127.0.0.1:3000" } + } + } +} + +``` + +**Important**: The `schema-file` and `schema-content` options are **mutually exclusive**. Use one or the other, not both. + +Inline specs are useful for: +- Small APIs that don't warrant a separate file +- Testing and prototyping +- Self-contained configuration that includes all dependencies + +For large or shared schemas, prefer `schema-file` to keep configuration maintainable. + +##### Inline JSON Schema + +Define JSON schemas directly in the configuration using KDL syntax: + +```kdl +route "user-registration" { + matches { + path "/api/register" + } + upstream "api-backend" + service-type "api" + api-schema { + validate-requests #true + request-schema { + type "object" + properties { + email { + type "string" + format "email" + description "User email address" + } + password { + type "string" + minLength 8 + maxLength 128 + description "Password (min 8 characters)" + } + username { + type "string" + minLength 3 + maxLength 32 + pattern "^[a-zA-Z0-9_-]+$" + } + age { + type "integer" + minimum 13 + maximum 120 + } + terms_accepted { + type "boolean" + } + } + required "email" "password" "username" "terms_accepted" + } + } +} +``` + +The inline schema is converted to JSON Schema and compiled at startup. It follows the JSON Schema specification and supports all standard JSON Schema keywords. + +##### Request and Response Validation + +Configure separate validation for requests and responses: + +```kdl +route "user-profile" { + matches { + path-prefix "/api/profile" + } + upstream "api-backend" + service-type "api" + api-schema { + validate-requests #true + validate-responses #true // Enable response validation + strict-mode #true // Reject additional properties + + // Schema for profile updates + request-schema { + type "object" + properties { + display_name { + type "string" + minLength 1 + maxLength 100 + } + bio { + type "string" + maxLength 500 + } + avatar_url { + type "string" + format "uri" + } + } + minProperties 1 // At least one field required + } + + // Schema for profile responses + response-schema { + type "object" + properties { + id { + type "string" + format "uuid" + } + email { + type "string" + format "email" + } + username { type "string" } + display_name { type "string" } + bio { type "string" } + avatar_url { + type "string" + format "uri" + } + created_at { + type "string" + format "date-time" + } + updated_at { + type "string" + format "date-time" + } + } + required "id" "email" "username" "created_at" + } + } +} +``` + +##### Complex Nested Schemas + +Support for complex object hierarchies and arrays: + +```kdl +route "create-order" { + matches { + path "/api/orders" + method "POST" + } + upstream "api-backend" + service-type "api" + api-schema { + validate-requests #true + strict-mode #true + request-schema { + type "object" + properties { + customer { + type "object" + properties { + name { + type "string" + minLength 1 + } + email { + type "string" + format "email" + } + phone { + type "string" + pattern "^\\+?[1-9]\\d{1,14}$" + } + } + required "name" "email" + } + items { + type "array" + minItems 1 + items { + type "object" + properties { + product_id { type "string" } + quantity { + type "integer" + minimum 1 + } + price { + type "number" + minimum 0 + } + } + required "product_id" "quantity" "price" + } + } + shipping_address { + type "object" + properties { + street { type "string" } + city { type "string" } + state { + type "string" + minLength 2 + maxLength 2 + } + zip { + type "string" + pattern "^\\d{5}(-\\d{4})?$" + } + country { + type "string" + enum "US" "CA" "MX" + } + } + required "street" "city" "state" "zip" "country" + } + } + required "customer" "items" "shipping_address" + } + } +} +``` + +##### Validation Options + +| Option | Default | Description | +|--------|---------|-------------| +| `schema-file` | None | Path to OpenAPI/Swagger spec file (YAML or JSON) | +| `request-schema` | None | Inline JSON Schema for request validation | +| `response-schema` | None | Inline JSON Schema for response validation | +| `validate-requests` | `true` | Enable request body validation | +| `validate-responses` | `false` | Enable response body validation | +| `strict-mode` | `false` | Reject additional properties not in schema | + +##### Validation Error Responses + +When validation fails, Zentinel returns a structured JSON error response: + +```json +{ + "error": "Validation failed", + "status": 400, + "request_id": "req-123", + "validation_errors": [ + { + "field": "$.email", + "message": "'not-an-email' is not a valid email", + "value": "not-an-email" + }, + { + "field": "$.password", + "message": "String is too short (expected minimum 8 characters)", + "value": "short" + } + ] +} +``` + +##### JSON Schema Support + +Zentinel supports JSON Schema Draft 7 with the following features: + +- **Types**: `string`, `number`, `integer`, `boolean`, `array`, `object`, `null` +- **String validation**: `minLength`, `maxLength`, `pattern`, `format` (email, uri, uuid, date-time, etc.) +- **Numeric validation**: `minimum`, `maximum`, `multipleOf` +- **Array validation**: `minItems`, `maxItems`, `uniqueItems`, `items` +- **Object validation**: `properties`, `required`, `minProperties`, `maxProperties`, `additionalProperties` +- **Logical operators**: `allOf`, `anyOf`, `oneOf`, `not` +- **References**: `$ref` (for OpenAPI specs) + +##### OpenAPI Integration + +When using `schema-file`, Zentinel: + +1. Loads the OpenAPI/Swagger specification at startup +2. Extracts schemas for each path and HTTP method +3. Validates incoming requests against the operation's `requestBody` schema +4. Validates responses against the operation's `responses` schema (if enabled) +5. Matches requests to operations by path and method + +The schema file is monitored for changes and automatically reloaded (if hot-reload is enabled). + +##### Performance Considerations + +- Schemas are compiled once at startup for maximum performance +- Request validation adds minimal latency (typically <1ms) +- Response validation requires buffering the full response body +- Use `validate-responses` only in development/testing environments +- For high-throughput APIs, consider validating only critical endpoints + +##### Best Practices + +1. **Use OpenAPI specs** for complex APIs with multiple endpoints +2. **Enable strict-mode** to catch unexpected fields early +3. **Validate requests in production**, responses in development +4. **Keep schemas focused** - validate only what's necessary +5. **Use meaningful descriptions** for better error messages +6. **Test validation** with invalid payloads before deploying +7. **Version your schemas** alongside your API versions + +##### Example: Complete API Route + +```kdl +route "user-api" { + priority 200 + matches { + path-prefix "/api/v2/users" + method "GET" "POST" "PUT" "DELETE" + } + upstream "user-service" + service-type "api" + + // Schema validation + api-schema { + schema-file "/etc/zentinel/schemas/user-api-v2.yaml" + validate-requests #true + validate-responses #false + strict-mode #true + } + + // Authentication and rate limiting + filters "jwt-auth" "rate-limit" + + // Error handling + error-pages { + default-format "json" + pages { + "400" { + format "json" + message "Invalid request" + } + "401" { + format "json" + message "Authentication required" + } + } + } + + // Performance tuning + policies { + timeout-secs 30 + max-body-size "10MB" + buffer-requests #true // Required for validation + } + + // Resilience + retry-policy { + max-attempts 3 + retryable-status-codes 502 503 504 + } +} +``` + +### Upstream Reference + +```kdl +route "api" { + upstream "backend" +} +``` + +Reference an upstream defined in the `upstreams` block. Required for `web` and `api` service types. + +> **Note:** If the referenced upstream connects to an HTTPS backend, make sure it has a `tls` block configured. See [Upstream TLS](/configuration/upstreams/#upstream-tls). + +### Filters and Agents + +```kdl +route "api" { + matches { + path-prefix "/api/" + } + upstream "backend" + filters "auth" "rate-limit" "cors" +} +``` + +Apply filters in order. Filters are defined in the top-level `filters` block. + +Enable WAF shorthand: + +```kdl +route "api" { + waf-enabled #true +} +``` + +## Route Policies + +### Header Modifications + +```kdl +route "api" { + upstream "backend" + policies { + request-headers { + // Set or replace header + set { + "X-Forwarded-Proto" "https" + } + // Add header (preserves existing) + add { + "X-Custom-Header" "value" + } + // Remove headers + remove "X-Internal-Header" "X-Debug" + } + response-headers { + set { + "X-Content-Type-Options" "nosniff" + "X-Frame-Options" "DENY" + } + remove "Server" "X-Powered-By" + } + } +} +``` + +### Timeout Override + +```kdl +route "upload" { + matches { + path-prefix "/upload/" + } + upstream "backend" + policies { + timeout-secs 300 // 5 minutes for uploads + } +} +``` + +### Body Size Limit + +```kdl +route "upload" { + policies { + max-body-size "100MB" + } +} +``` + +Supports units: `B`, `KB`, `MB`, `GB` + +### Failure Mode + +```kdl +route "api" { + policies { + failure-mode "closed" // Block on failure (default) + } +} + +route "metrics" { + policies { + failure-mode "open" // Allow through on failure + } +} +``` + +| Mode | Behavior | Use Case | +|------|----------|----------| +| `closed` | Block traffic on agent/upstream failure | Security-sensitive routes | +| `open` | Allow traffic through on failure | Non-critical observability | + +### Request/Response Buffering + +```kdl +system { + worker-threads 0 +} + +listeners { + listener "http" { + address "0.0.0.0:8080" + protocol "http" + } +} + +route "api" { + policies { + buffer-requests #true // Buffer full request before forwarding + buffer-responses #true // Buffer full response before sending + } +} + +routes { + route "default" { + matches { path-prefix "/" } + upstream "backend" + } +} + +upstreams { + upstream "backend" { + targets { + target { address "127.0.0.1:3000" } + } + } +} + +``` + +Buffering is required for body inspection by agents. Be mindful of memory usage with large bodies. + +## Retry Policy + +```kdl +route "api" { + upstream "backend" + retry-policy { + max-attempts 3 + timeout-ms 30000 + backoff-base-ms 100 + backoff-max-ms 10000 + retryable-status-codes 502 503 504 + } +} +``` + +| Option | Default | Description | +|--------|---------|-------------| +| `max-attempts` | `3` | Maximum retry attempts | +| `timeout-ms` | `30000` | Total timeout for all attempts | +| `backoff-base-ms` | `100` | Initial backoff delay | +| `backoff-max-ms` | `10000` | Maximum backoff delay | +| `retryable-status-codes` | `502, 503, 504` | Status codes to retry | + +Backoff uses exponential delay: `min(base * 2^attempt, max)` + +## Circuit Breaker + +```kdl +route "api" { + upstream "backend" + circuit-breaker { + failure-threshold 5 + success-threshold 2 + timeout-seconds 30 + half-open-max-requests 1 + } +} +``` + +| Option | Default | Description | +|--------|---------|-------------| +| `failure-threshold` | `5` | Failures before opening circuit | +| `success-threshold` | `2` | Successes to close circuit | +| `timeout-seconds` | `30` | Time before trying half-open | +| `half-open-max-requests` | `1` | Requests allowed in half-open | + +Circuit breaker states: +- **Closed**: Normal operation, requests flow through +- **Open**: Requests fail immediately (circuit tripped) +- **Half-Open**: Limited requests to test recovery + +## Traffic Mirroring / Shadow Traffic + +Traffic mirroring (also called shadow traffic or dark traffic) duplicates live requests to a secondary upstream for testing purposes, while the client receives the response from the primary upstream. This enables safe canary deployments, performance testing, and debug workflows. + +```kdl +route "api" { + upstream "production" + + shadow { + upstream "canary" + percentage 100.0 + timeout-ms 5000 + buffer-body #false + max-body-bytes 1048576 + } +} +``` + +### Key Characteristics + +- **Fire-and-forget**: Shadow requests are sent asynchronously and non-blocking +- **No client impact**: Shadow failures don't affect the primary response +- **Zero latency**: Shadow execution happens in a separate tokio task +- **Sampling control**: Percentage-based and header-based filtering +- **Independent failure domain**: Separate connection pools for shadow upstreams + +### Shadow Configuration Options + +| Option | Type | Default | Description | +|--------|------|---------|-------------| +| `upstream` | string | *required* | Shadow target upstream ID (must exist in upstreams block) | +| `percentage` | float | `100.0` | Percentage of requests to mirror (0.0-100.0) | +| `sample-header` | tuple | none | Only mirror if request header matches `(name, value)` | +| `timeout-ms` | int | `5000` | Shadow request timeout in milliseconds | +| `buffer-body` | bool | `false` | Whether to buffer request bodies for POST/PUT/PATCH | +| `max-body-bytes` | int | `1048576` | Maximum body size to shadow (1MB default) | + +### Example: Full Shadow (100% Mirrored) + +Mirror all traffic to a canary deployment for comprehensive testing: + +```kdl +route "api-full-shadow" { + matches { + path-prefix "/api/v1" + } + upstream "production" + + shadow { + upstream "canary" + percentage 100.0 + timeout-ms 5000 + } +} +``` + +**Use case**: Initial canary deployment - validate stability with all traffic. + +### Example: Partial Shadow (10% Sampling) + +Mirror a percentage of requests to reduce shadow load: + +```kdl +route "api-partial-shadow" { + matches { + path-prefix "/api/v2" + } + upstream "production" + + shadow { + upstream "canary" + percentage 10.0 // Mirror 10% of requests + timeout-ms 5000 + } +} +``` + +**Use case**: Gradual rollout - representative traffic sampling with lower overhead. + +### Example: Header-Based Shadow + +Mirror only requests with specific headers for targeted testing: + +```kdl +route "api-debug-shadow" { + matches { + path-prefix "/api/v3" + } + upstream "production" + + shadow { + upstream "canary" + percentage 100.0 + sample-header "X-Debug-Shadow" "true" // Only if header present + timeout-ms 5000 + } +} +``` + +**Use case**: Developer testing - mirror only debug-flagged requests. + +### Body Buffering + +By default, shadow requests **do not** include request bodies to avoid buffering overhead. For POST/PUT/PATCH requests that need body inspection in the shadow: + +```kdl +shadow { + upstream "canary" + buffer-body #true // Enable body buffering + max-body-bytes 1048576 // Limit to 1MB +} +``` + +⚠️ **Important**: Buffering request bodies has memory and latency implications. Use `max-body-bytes` to enforce strict limits. + +**When to buffer bodies:** +- ✅ Small payloads (<1MB), testing form submissions, API validation +- ❌ Large uploads, streaming data, file uploads, high-throughput APIs + +### Metrics + +Zentinel exposes Prometheus metrics for shadow traffic monitoring: + +```prometheus +# Total shadow requests sent (labels: route, upstream, result) +shadow_requests_total{route="api-full-shadow",upstream="canary",result="success"} 1234 + +# Shadow request errors (labels: route, upstream, error_type) +shadow_errors_total{route="api-full-shadow",upstream="canary",error_type="timeout"} 5 + +# Shadow request latency histogram (labels: route, upstream) +shadow_latency_seconds_bucket{route="api-full-shadow",upstream="canary",le="0.1"} 980 +``` + +**Key metrics to monitor:** +- `shadow_requests_total{result="success"}` - Successful shadow requests +- `shadow_requests_total{result="error"}` - Failed shadow requests +- `shadow_errors_total{error_type="timeout"}` - Shadow timeouts +- `shadow_latency_seconds` - Shadow request latency distribution + +### Security Considerations + +Shadow traffic contains **real user data**. Ensure shadow upstreams: + +- Have equivalent security controls (TLS, auth, encryption) +- Comply with data residency and privacy requirements +- Use the same data handling policies as production +- Audit shadow traffic access appropriately + +For regulated environments (PCI, HIPAA, GDPR): +- **Do not** mirror sensitive data to less-secure environments +- Use `sample-header` to exclude sensitive requests +- Consider data masking/scrubbing before shadowing +- Document shadow data flows in compliance audits + +### Best Practices + +1. **Start with low sampling**: Begin with 1-5% and gradually increase +2. **Use timeouts**: Configure shadow timeouts shorter than primary timeouts +3. **Monitor shadow health**: Set up alerts for shadow error rates +4. **Body buffering limits**: Enforce strict size limits when buffering +5. **Header-based targeting**: Use headers for targeted testing without impacting all traffic + +### Complete Example + +```kdl +routes { + // Production route with canary shadow + route "api-v2" { + priority 200 + matches { + path-prefix "/api/v2/" + method "GET" "POST" "PUT" "DELETE" + } + upstream "production" + + // Mirror 10% of traffic to canary + shadow { + upstream "canary" + percentage 10.0 + timeout-ms 3000 + buffer-body #false + } + + // Additional configuration + filters "auth" "rate-limit" + retry-policy { + max-attempts 3 + retryable-status-codes 502 503 504 + } + } + + // Debug route with 100% shadow for beta users + route "api-beta" { + priority 250 + matches { + path-prefix "/api/v2/" + header name="X-User-Tier" value="beta" + } + upstream "production" + + shadow { + upstream "canary" + percentage 100.0 + sample-header "X-Enable-Shadow" "true" + timeout-ms 5000 + } + } +} +``` + +For a complete traffic mirroring example with Docker Compose and test scripts, see the [Traffic Mirroring Example](../../examples/traffic-mirroring/). + +## Error Pages + +```kdl +route "api" { + error-pages { + default-format "json" + pages { + "404" { + format "json" + message "Resource not found" + } + "500" { + format "json" + message "Internal server error" + } + "503" { + format "html" + template "/etc/zentinel/errors/503.html" + } + } + } +} +``` + +| Format | Content-Type | +|--------|--------------| +| `json` | `application/json` | +| `html` | `text/html` | +| `text` | `text/plain` | +| `xml` | `application/xml` | + +## Complete Examples + +### API Gateway + +```kdl +routes { + // Health check (highest priority) + route "health" { + priority 1000 + matches { + path "/health" + } + service-type "builtin" + builtin-handler "health" + } + + // Metrics endpoint (admin only) + route "metrics" { + priority 999 + matches { + path "/metrics" + header name="X-Admin-Token" + } + service-type "builtin" + builtin-handler "metrics" + } + + // API v2 (current) + route "api-v2" { + priority 200 + matches { + path-prefix "/api/v2/" + method "GET" "POST" "PUT" "DELETE" "PATCH" + } + upstream "api-v2" + filters "auth" "rate-limit" + retry-policy { + max-attempts 3 + retryable-status-codes 502 503 504 + } + policies { + timeout-secs 30 + failure-mode "closed" + request-headers { + set { + "X-Api-Version" "2" + } + } + } + } + + // API v1 (legacy) + route "api-v1" { + priority 100 + matches { + path-prefix "/api/v1/" + } + upstream "api-v1" + filters "auth" + policies { + timeout-secs 60 + response-headers { + set { + "X-Deprecation-Notice" "API v1 is deprecated. Please migrate to v2." + } + } + } + } + + // Static assets + route "static" { + priority 50 + matches { + path-prefix "/static/" + } + service-type "static" + static-files { + root "/var/www/static" + cache-control "public, max-age=31536000, immutable" + compress #true + } + } + + // SPA fallback + route "spa" { + priority 1 + matches { + path-prefix "/" + method "GET" + } + service-type "static" + static-files { + root "/var/www/app" + fallback "index.html" + } + } +} +``` + +### Multi-tenant Routing + +```kdl +routes { + route "tenant-a" { + priority 100 + matches { + host "tenant-a.example.com" + path-prefix "/api/" + } + upstream "tenant-a-backend" + policies { + request-headers { + set { + "X-Tenant-Id" "tenant-a" + } + } + } + } + + route "tenant-b" { + priority 100 + matches { + host "tenant-b.example.com" + path-prefix "/api/" + } + upstream "tenant-b-backend" + policies { + request-headers { + set { + "X-Tenant-Id" "tenant-b" + } + } + } + } +} +``` + +## Default Values + +| Setting | Default | +|---------|---------| +| `priority` | `0` | +| `service-type` | `web` | +| `policies.failure-mode` | `closed` | +| `policies.buffer-requests` | `false` | +| `policies.buffer-responses` | `false` | +| `static-files.index` | `index.html` | +| `static-files.directory-listing` | `false` | +| `static-files.compress` | `true` | +| `retry-policy.max-attempts` | `3` | +| `circuit-breaker.failure-threshold` | `5` | + +## Route Evaluation Order + +1. Routes sorted by priority (descending) +2. First matching route wins +3. If no route matches and listener has `default-route`, use that +4. Otherwise, return 404 + +## Next Steps + +- [Upstreams](../upstreams/) - Backend server configuration +- [Limits](../limits/) - Request limits and performance diff --git a/content/v/26.04/configuration/security-headers.md b/content/v/26.04/configuration/security-headers.md new file mode 100644 index 0000000..f5bc6e5 --- /dev/null +++ b/content/v/26.04/configuration/security-headers.md @@ -0,0 +1,287 @@ ++++ +title = "Security Headers" +weight = 8 +updated = 2026-02-22 ++++ + +Zentinel does not inject any response headers by default. Security headers are configured explicitly via `response-headers` policies on your routes or via a reusable `headers` filter. This gives you full control over which headers are set and what values they use. + +## Adding Security Headers + +### Per-Route Policy + +The simplest way to add headers to a specific route: + +```kdl +route "app" { + matches { + path-prefix "/" + } + upstream "backend" + + policies { + response-headers { + set { + "X-Content-Type-Options" "nosniff" + "X-Frame-Options" "SAMEORIGIN" + "Referrer-Policy" "strict-origin-when-cross-origin" + "Permissions-Policy" "geolocation=(), microphone=(), camera=()" + } + } + } +} +``` + +### Reusable Headers Filter + +For headers shared across multiple routes, define a `headers` filter once and reference it by name: + +```kdl +filters { + filter "security-headers" { + type "headers" + phase "response" + set { + "X-Content-Type-Options" "nosniff" + "X-Frame-Options" "SAMEORIGIN" + "Referrer-Policy" "strict-origin-when-cross-origin" + "Permissions-Policy" "geolocation=(), microphone=(), camera=()" + } + } +} + +routes { + route "api" { + matches { path-prefix "/api/" } + upstream "api-backend" + filters "security-headers" + } + + route "app" { + matches { path-prefix "/" } + upstream "app-backend" + filters "security-headers" + } +} +``` + +### Removing Headers + +Strip headers that leak server information: + +```kdl +policies { + response-headers { + remove "Server" "X-Powered-By" "X-AspNet-Version" "X-AspNetMvc-Version" + } +} +``` + +## Recommended Headers + +### Content-Type Options + +Prevents browsers from MIME-sniffing a response away from the declared `Content-Type`. Always safe to set. + +```kdl +"X-Content-Type-Options" "nosniff" +``` + +| Value | Behavior | +|-------|----------| +| `nosniff` | Browser trusts the declared Content-Type; blocks style/script loads with wrong MIME type | + +**Recommendation:** Set on all routes. + +### Frame Options + +Controls whether the page can be embedded in an `