feat(observability): OpenTelemetry (OTel) & Prometheus Structured Metrics Exporter

[sdageltc]

Summary & Motivation

letitloop includes built-in execution tracking (orchestrator/metrics.py and telemetry.py), recording contract runtimes, retry counts, token usage, and verifier exit codes to local JSON ledgers (run_dir/metrics.json).

As teams deploy multi-hour macro-task loops in CI/CD pipelines and production agent swarms, having standardized OpenTelemetry (OTel) spans and Prometheus metrics export is essential for real-time observability in Grafana, Datadog, or Jaeger.

---

Architectural Proposal

1. Standardized Telemetry Spans:
   • Trace the lifecycle of a Goal -> Contracts -> Attempts -> Verifications -> QC Audits.
   • Span tags: task_id, worker_type, risk_tier, verifier_exit_code, retry_attempt, status.
2. Prometheus Metrics:
   • letitloop_contract_duration_seconds (Histogram)
   • letitloop_contracts_total{status="COMPLETE|FAILED|ESCALATED"} (Counter)
   • letitloop_budget_tokens_total{type="prompt|completion"} (Counter)
   • letitloop_three_strike_escalations_total (Counter)
3. Pluggable Exporter Interface:
   • Keep core dependencies zero-dependency by default; provide OTel/Prometheus support via pip install "letitloop[telemetry]" or optional imports.

---

Key Files & Areas

• orchestrator/metrics.py: Add metric aggregator & export hooks.
• orchestrator/telemetry.py: Add OpenTelemetry trace context propagation.
• orchestrator/supervisor.py: Emit span start/end events during contract execution loop.
• pyproject.toml: Add optional dependency group [project.optional-dependencies] telemetry = ["opentelemetry-api", "opentelemetry-sdk", "prometheus-client"].

---

Acceptance Criteria

• Emits structured spans for goal execution and individual contract transitions.
• Exposes a Prometheus /metrics endpoint or file-based Prometheus text exporter.
• Zero runtime crashes if optional telemetry packages are not installed.
• Comprehensive unit tests verifying metric increments and trace attributes.