Add optional OpenTelemetry tracing spans for evaluation

[se-jo-ma]

Context

Fathom's observability is Prometheus-only today. src/fathom/metrics.py:17 imports prometheus_client (Counter/Gauge/Histogram) behind a graceful optional-import no-op, and Engine.evaluate() records metrics in a finally block (src/fathom/engine.py:1182). A grep for opentelemetry|otel|tracing|trace_id|span across src/fathom/ returns nothing (only unrelated HTML <span> in Studio JSX).

Fathom is positioned as a policy/guardrail step inside agent pipelines (LangChain/CrewAI/OpenAI-Agents/Google-ADK adapters live in src/fathom/integrations/). In that role, operators debug a request that crossed several services with distributed tracing, not just aggregate counters. Prometheus tells you the denial rate; a trace tells you why this request was denied and how long the eval took within the surrounding agent request. Adding an optional OTel span for evaluate() that nests under the caller's trace closes that gap.

Task

Add optional OpenTelemetry instrumentation behind a new optional-dependency group, mirroring the existing optional-import no-op pattern used for prometheus_client:

• Wrap Engine.evaluate() in a span carrying attributes such as decision, rules_fired_count, module_count, duration_us, and session_id.
• Extract and propagate W3C trace context from incoming REST (traceparent header) and gRPC (invocation metadata) requests so the eval span nests under the caller's trace.
• No-op cleanly with zero import error when opentelemetry is not installed.
• Document setup in a new how-to (observability/tracing) and link it from the how-to index.

Where

• src/fathom/metrics.py (or a new src/fathom/tracing.py) — span helpers with graceful optional import, mirroring metrics.py:17.
• src/fathom/engine.py:1148 (def evaluate) — wrap the eval; note metrics already use a finally block at line 1182 as a model.
• src/fathom/integrations/rest.py — REST call site at evaluate() (~line 249); Header/Request already imported for header extraction.
• src/fathom/integrations/grpc_server.py — Evaluate RPC (line 173); context.invocation_metadata() already used for auth, reuse for trace-context extraction.
• pyproject.toml:49 — add an otel group under [project.optional-dependencies], alongside the existing metrics group.
• docs/how-to/index.md — add link to a new observability/tracing how-to guide.

Acceptance criteria

• With OTel installed and a span exporter configured, an evaluate() produces a span carrying decision/duration attributes.
• Without OTel installed, the full test suite passes with zero import error and zero overhead in the no-op path.
• The eval span is a child of the incoming REST/gRPC request span when W3C trace headers/metadata are present.
• pyproject.toml exposes an otel optional-dependency group.
• A how-to doc covers enabling tracing, linked from docs/how-to/index.md.
• uv run mypy src/ (strict) and uv run ruff check src/ tests/ pass.

Size: M