feat: ship multi-tenancy, Entra auth, retention, and pre-UI hardening

Backend
- Multi-tenancy: tenant_id column + ctx propagation across all reads/writes;
  X-Tenant-ID HTTP header, x-tenant-id gRPC metadata, tenant.id resource attr;
  per-tenant API key file support that overrides client header.
- Azure Entra ID DB auth (DefaultAzureCredential); strict sslmode gate
  (require|verify-ca|verify-full); 30m conn lifetime cap for token TTL.
- Drain log clustering (pure-Go, persisted templates, O(1) LRU via
  container/list); retains TF-IDF vectordb as fallback similarity ranker.
- RetentionScheduler: hourly batched purge + daily VACUUM/ANALYZE via raw
  sqlDB (escapes GORM implicit tx); atomic.Bool overlap guard; cross-tenant.
- Self-signed TLS bootstrap: package mutex + flock cross-process serialization,
  ECDSA-P256, 10yr validity, SANs for localhost/hostname.
- Panic recovery across HTTP middleware, gRPC interceptors, GraphRAG workers.
- DB health 503 fast-fail middleware with 5s ping loop.
- DLQ: typed envelopes, collision-safe tempfiles via os.CreateTemp.
- Decompression bomb guard: 64 MiB io.LimitReader post-gzip.
- OTel self-instrumentation via otlptracegrpc + otelhttp.
- API auth failure metric hook (missing_header/bad_scheme/bad_key reasons).

Storage fixes
- PurgeTracesBatched: Unscoped().Delete() to avoid GORM soft-delete.
- Span sweep: exclude spans whose trace still exists (clock-skew safe).
- UpdateLogInsight: tenant-scoped WHERE + ErrLogNotFoundOrWrongTenant.
- GetLogsV2 parallel queries now use WithContext(ctx).
- Drain template ID collisions on merge folded into existing bucket.

UI
- Mantine v8 (scoped migration from Radix UI).
- ErrorBoundary + WebSocket exponential backoff and heartbeat.
- react-window virtualization for Traces/Logs pages.
- Full TypeScript type contracts in ui/src/types/api.ts.

Ops
- docs/OPERATIONS.md (First Run, Production Checklist, Backup/Restore,
  Incident Response, Observability, Known Limitations, Upgrade Path).
- .env.example covering every os.Getenv key.
- CLAUDE.md synced with new subsystems.

Verification: go build + go vet + golangci-lint (0 issues) + go test -race
across all packages green.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

Author: aksOps

.env.example

@@ -1,14 +1,131 @@
-# Application Environment
-APP_ENV=development
-HTTP_PORT=8080
-GRPC_PORT=4317
+# ==============================================================================
+# OtelContext — Environment Configuration Reference
+# ==============================================================================
+# Copy to `.env` and uncomment the lines you want to override.
+# Booleans accept (case-insensitive): 1 | true | yes | on  (parseTruthy in config.go)
+# Lines shown as `=<default>` document the default applied when the var is unset.
+# Keys marked [REQUIRED IN PROD] must be set for any production deployment.
+# ------------------------------------------------------------------------------
 
-# Database Configuration
-# Options: mysql, sqlite, sqlserver
-DB_DRIVER=mysql
-DB_DSN=root:admin@tcp(127.0.0.1:3306)/OtelContext?charset=utf8mb4&parseTime=True&loc=Local
+# ---- Application ------------------------------------------------------------
+# APP_ENV=development            # development|production — gates DevMode (WS origin checks relaxed)
+# LOG_LEVEL=INFO                 # DEBUG|INFO|WARN|ERROR
+# HTTP_PORT=8080                 # HTTP API + OTLP HTTP + WebSocket + UI
+# GRPC_PORT=4317                 # OTLP gRPC
 
-# For SQLite:
-# DB_DRIVER=sqlite
-# DB_DSN=OtelContext.db
+# ---- Database ---------------------------------------------------------------
+# DB_DRIVER=sqlite               # sqlite|postgres|mysql|sqlserver
+# DB_DSN=                        # driver-specific — pick ONE of the blocks below
 
+#   SQLite (default, zero-config):
+# DB_DSN=otelcontext.db
+
+#   PostgreSQL:
+# DB_DRIVER=postgres
+# DB_DSN=host=localhost user=otel password=otel dbname=otelcontext port=5432 sslmode=disable TimeZone=UTC
+
+#   MySQL:
+# DB_DRIVER=mysql
+# DB_DSN=root:admin@tcp(127.0.0.1:3306)/OtelContext?charset=utf8mb4&parseTime=True&loc=Local
+
+#   SQL Server:
+# DB_DRIVER=sqlserver
+# DB_DSN=sqlserver://user:password@host:1433?database=OtelContext
+
+# DB_AUTOMIGRATE=true            # GORM AutoMigrate on startup. Set false in Postgres prod (schema out-of-band)
+
+# ---- Database Pool ----------------------------------------------------------
+# DB_MAX_OPEN_CONNS=50           # Max concurrent DB connections
+# DB_MAX_IDLE_CONNS=10           # Idle connections kept in pool
+# DB_CONN_MAX_LIFETIME=1h        # Conn recycle window. Internally capped to 30m when DB_AZURE_AUTH=true
+
+# ---- Azure Entra (passwordless Postgres) ------------------------------------
+# DB_AZURE_AUTH=false            # Enables DefaultAzureCredential for Postgres. Requires strict TLS
+#                                # (sslmode=require|verify-ca|verify-full). DSN must omit password.
+#                                # Credential order: env vars → workload identity → managed identity → az CLI → dev creds.
+#                                # Local dev: `az login` is sufficient. AKS: workload or pod-managed identity.
+# DB_DSN=host=my-server.postgres.database.azure.com user=my-mi@tenant.onmicrosoft.com dbname=otelcontext port=5432 sslmode=require
+
+# ---- TLS (HTTP + gRPC) ------------------------------------------------------
+# Explicit cert files take precedence over self-signed. Both files must be set together.
+# TLS_CERT_FILE=/etc/otelcontext/tls/server.crt
+# TLS_KEY_FILE=/etc/otelcontext/tls/server.key
+#
+# Self-signed bootstrap (dev/internal). Ignored if TLS_CERT_FILE is set.
+# Generates ECDSA-P256 cert at first start, caches under TLS_CACHE_DIR, reuses until expiry.
+# Clients must trust the generated cert (insecure skip or CA pin).
+# TLS_AUTO_SELFSIGNED=false
+# TLS_CACHE_DIR=./data/tls
+
+# ---- Auth -------------------------------------------------------------------
+# API_KEY=                       # [REQUIRED IN PROD] Bearer token for /api/*, /v1/*, /mcp. Empty = auth disabled (dev only).
+
+# ---- OTLP Ingest Filtering --------------------------------------------------
+# INGEST_MIN_SEVERITY=INFO       # Drop logs below this severity before storage
+# INGEST_ALLOWED_SERVICES=       # CSV allowlist of service.name (empty = accept all)
+# INGEST_EXCLUDED_SERVICES=      # CSV denylist (applied after allowlist)
+
+# ---- Adaptive Sampling ------------------------------------------------------
+# SAMPLING_RATE=1.0              # 0.0..1.0 probability for non-error, non-slow spans
+# SAMPLING_ALWAYS_ON_ERRORS=true # Keep every error span regardless of rate
+# SAMPLING_LATENCY_THRESHOLD_MS=500  # Keep every span slower than this
+
+# ---- TSDB -------------------------------------------------------------------
+# TSDB_RING_BUFFER_DURATION=1h   # In-memory metric ring buffer window (e.g. 30m, 2h)
+
+# ---- GraphRAG / Cardinality / Vector ----------------------------------------
+# METRIC_ATTRIBUTE_KEYS=         # CSV allowlist of attribute keys included in metric series key
+# METRIC_MAX_CARDINALITY=10000   # Max unique series per metric; new series dropped above this
+# VECTOR_INDEX_MAX_ENTRIES=100000  # TF-IDF index capacity (FIFO eviction)
+
+# ---- DLQ (Dead Letter Queue) ------------------------------------------------
+# DLQ_PATH=./data/dlq            # Directory for typed-envelope files
+# DLQ_REPLAY_INTERVAL=5m         # Retry cadence with exponential backoff
+# DLQ_MAX_FILES=1000             # Cap on enqueued envelope count
+# DLQ_MAX_DISK_MB=500            # Disk budget — new writes fail when exceeded
+# DLQ_MAX_RETRIES=10             # Give up after this many failed replays
+
+# ---- Rate Limiting ----------------------------------------------------------
+# API_RATE_LIMIT_RPS=100         # Per-IP token bucket rate for /api/*. 0 disables.
+
+# ---- MCP Server -------------------------------------------------------------
+# MCP_ENABLED=true               # Expose MCP JSON-RPC 2.0 (POST) + SSE (GET) for AI agents
+# MCP_PATH=/mcp                  # Mount path
+
+# ---- Compression ------------------------------------------------------------
+# COMPRESSION_LEVEL=default      # default|fast|best — zstd level for compressed columns
+
+# ---- Retention --------------------------------------------------------------
+# HOT_RETENTION_DAYS=7           # RetentionScheduler purge cutoff. Range 1..36500. Set explicitly in prod.
+
+# ---- OTel Self-Instrumentation ----------------------------------------------
+# OTEL_EXPORTER_OTLP_ENDPOINT=   # When set, OtelContext exports its own spans to this OTLP gRPC endpoint.
+#                                # Use `localhost:4317` for dogfooding (self-ingest).
+
+# ---- Multi-tenancy ----------------------------------------------------------
+# DEFAULT_TENANT=default         # Tenant ID for rows ingested without X-Tenant-ID (HTTP) /
+#                                # x-tenant-id (gRPC metadata).
+#
+# OTLP_TRUST_RESOURCE_TENANT=false
+#                                # When true, OTLP ingest falls back to the `tenant.id` resource
+#                                # attribute if no header/metadata tenant was supplied. Disabled
+#                                # by default because resource attributes are client-controlled —
+#                                # a compromised SDK could forge another tenant's data. Only turn
+#                                # on in closed environments where every OTLP producer is trusted.
+#
+# API_TENANT_KEYS_FILE=          # Path to a file of `key=tenant` pairs (one per line; `#` comments).
+#                                # When set, each API bearer token is bound to a specific tenant
+#                                # and the matched tenant OVERRIDES any X-Tenant-ID header —
+#                                # callers cannot read other tenants by swapping headers.
+#                                # When empty, behaviour falls back to the single shared API_KEY
+#                                # + self-asserted X-Tenant-ID header (legacy dev mode).
+
+# ---- AI Service (optional — Azure OpenAI log insights) ----------------------
+# AI_ENABLED=false               # Master switch. When false, AI workers are not started.
+# AZURE_OPENAI_ENDPOINT=         # e.g. https://my-aoai.openai.azure.com/
+# AZURE_OPENAI_KEY=              # API key
+# AZURE_OPENAI_MODEL=            # Base model name
+# AZURE_OPENAI_DEPLOYMENT=       # Deployment name (overrides MODEL if set)
+# AZURE_OPENAI_API_VERSION=      # e.g. 2024-02-15-preview
+# AI_QUEUE_SIZE=100              # Backlog capacity for AI log analysis
+# AI_WORKER_POOL=3               # Concurrent AI workers

.golangci.yml

@@ -0,0 +1,71 @@
+# golangci-lint v2 configuration for OtelContext.
+# Curated set of linters — no "enable-all" sprawl.
+version: "2"
+
+run:
+  timeout: 5m
+
+linters:
+  default: none
+  enable:
+    # Core correctness
+    - errcheck       # unchecked errors
+    - govet          # standard vet
+    - ineffassign    # unused assignments
+    - staticcheck    # core correctness + Go 1.25 hints (rangeint, stringsseq, etc.)
+    - unused         # dead code
+
+    # Security & idioms
+    - gosec          # security issues
+    - gocritic       # idiomatic Go
+    - misspell       # comment/string typos
+    - unconvert      # redundant type conversions
+    - prealloc       # slice preallocation opportunities
+
+    # Resource hygiene
+    - bodyclose      # HTTP response body close
+    - rowserrcheck   # sql.Rows.Err() check
+    - sqlclosecheck  # sql resource leak detection
+
+    # Language pitfalls
+    - copyloopvar    # pre-Go-1.22 loop variable semantics
+    - nolintlint     # well-formed //nolint directives
+
+  exclusions:
+    rules:
+      # Test files commonly ignore errors intentionally (t.Cleanup, defer close, etc.).
+      - path: _test\.go
+        linters:
+          - errcheck
+          - gosec
+      # Drain tests exercise edge cases that trip gosec heuristics.
+      - path: internal/graphrag/drain_test\.go
+        linters:
+          - gosec
+      - path: internal/graphrag/drain_aggressive_test\.go
+        linters:
+          - gosec
+      # test/ contains microservice simulators (chaos testing). They intentionally
+      # use math/rand for chaos probabilities (G404), ListenAndServe without timeouts
+      # (G114), and log.Fatal (exitAfterDefer). Not production code.
+      - path: ^test/
+        linters:
+          - gosec
+          - gocritic
+
+formatters:
+  enable:
+    - gofmt
+    - goimports
+
+issues:
+  max-issues-per-linter: 0
+  max-same-issues: 0
+
+output:
+  formats:
+    text:
+      path: stdout
+      colors: true
+      print-linter-name: true
+      print-issued-lines: true