Skip to content

feat: OpenTelemetry traces/metrics, JSON logs, job locks - #15

Merged
FYWinds merged 2 commits into
masterfrom
feat/observability
Oct 7, 2026
Merged

FYWinds merged 2 commits into
masterfrom
feat/observability

Conversation

@FYWinds

@FYWinds FYWinds commented Oct 7, 2026

Copy link
Copy Markdown
Collaborator

Two commits, keep them separate (rebase merge):

  1. chore: normalize line endings to LF: the tree mixed CRLF and LF commits; .gitattributes pins LF everywhere. Line endings only.
  2. feat: OTel traces/metrics, JSON logs, job locks

What changes

  • Tracing (OpenTelemetry): FastAPI, SQLAlchemy, Redis and httpx spans, plus a root span per scheduler job and consensus <kind> business spans. The sampler is parentbased_always_on, so requests continue Cloudflare's/Traefik's W3C traceparent. Enabled by OTEL_EXPORTER_OTLP_ENDPOINT; disabled otherwise.
  • Metrics (OTLP push): wcs.submissions, wcs.submission.items, wcs.consensus.*, wcs.job.*, wcs.cache.requests, wcs.rate_limited, process/GC and DB pool metrics. http.server.* is dropped because HTTP RED comes from Tempo span metrics and Traefik. Every pod sets a unique service.instance.id.
  • Sentry: errors and logs only, tagged with the OTel trace id through OTLPIntegration. Its propagator is disabled, since it only speaks sentry-trace and would break W3C continuation. Profiling switches to continuous mode (WCS_SENTRY_PROFILE_SESSION_SAMPLE_RATE).
  • Scheduler jobs run once across replicas: a Redis SET NX PX lock held for just under the job's period. Jobs that only touch process-local state (the mapping refresh, registering boost jobs) still run on every replica.
  • Structured logs: LOG_JSON=true (the chart default) writes one JSON object per line with trace_id/span_id. uvicorn access logs are split into fields.
  • Security:
    • Query values are redacted in spans and access logs, because the manage endpoints take user tokens in the query.
    • init_db no longer logs the DSN password.
  • Container: the image now starts uvicorn in exec form. fastapi-cli re-attached rich log handlers that bypassed LOG_JSON, and the shell form blocked SIGTERM, which skipped the telemetry flush.
  • Dev:
    • Windows instructions (wslc, PYTHONUTF8).
    • An optional local OTel collector.
    • pytest pythonpath, so plain pytest works.
  • Version bumped to 0.1.7. pyproject.toml still said 0.1.5 while v0.1.6 was tagged.

Verified locally (Windows, wslc PG/Redis/collector)

  • Two replicas sharing one Redis: compute_gambit_consensus ran ok on one replica and was skipped on the other.
  • A sampled traceparent is continued.
  • ?token= shows up as token=REDACTED in spans and access logs.
  • JSON logs only.
  • The image built with wslc build shuts down cleanly on SIGTERM and flushes its spans and metrics.
  • ruff check, basedpyright (0 errors) and pytest pass.

The tree mixed CRLF and LF commits, so a ruff format (line-ending = lf) on Windows rewrote whole files. .gitattributes pins LF on every OS.
Requests now join the Cloudflare/Traefik trace (W3C propagation; Sentry
kept to errors so its sentry-trace propagator can't break that) and
metrics are pushed per pod so Grafana/Tempo cover the app itself.

Each replica ran every scheduler job; a Redis lock held for just under
the period now gives one run per period. Jobs that only touch
process-local state stay unlocked.

Query values are redacted in spans and access logs: manage endpoints
take user tokens in the query. init_db logged the DSN with password.

uvicorn starts directly in exec form: fastapi-cli re-attached its rich
log handlers (bypassing LOG_JSON) and the shell form blocked SIGTERM.
@FYWinds
FYWinds merged commit a520f3f into master Oct 7, 2026
2 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant