Repository navigation
feat: OpenTelemetry traces/metrics, JSON logs, job locks - #15
Merged
Merged
Conversation
The tree mixed CRLF and LF commits, so a ruff format (line-ending = lf) on Windows rewrote whole files. .gitattributes pins LF on every OS.
Requests now join the Cloudflare/Traefik trace (W3C propagation; Sentry kept to errors so its sentry-trace propagator can't break that) and metrics are pushed per pod so Grafana/Tempo cover the app itself. Each replica ran every scheduler job; a Redis lock held for just under the period now gives one run per period. Jobs that only touch process-local state stay unlocked. Query values are redacted in spans and access logs: manage endpoints take user tokens in the query. init_db logged the DSN with password. uvicorn starts directly in exec form: fastapi-cli re-attached its rich log handlers (bypassing LOG_JSON) and the shell form blocked SIGTERM.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Two commits, keep them separate (rebase merge):
chore: normalize line endings to LF: the tree mixed CRLF and LF commits;.gitattributespins LF everywhere. Line endings only.feat: OTel traces/metrics, JSON logs, job locksWhat changes
consensus <kind>business spans. The sampler isparentbased_always_on, so requests continue Cloudflare's/Traefik's W3Ctraceparent. Enabled byOTEL_EXPORTER_OTLP_ENDPOINT; disabled otherwise.wcs.submissions,wcs.submission.items,wcs.consensus.*,wcs.job.*,wcs.cache.requests,wcs.rate_limited, process/GC and DB pool metrics.http.server.*is dropped because HTTP RED comes from Tempo span metrics and Traefik. Every pod sets a uniqueservice.instance.id.OTLPIntegration. Its propagator is disabled, since it only speakssentry-traceand would break W3C continuation. Profiling switches to continuous mode (WCS_SENTRY_PROFILE_SESSION_SAMPLE_RATE).SET NX PXlock held for just under the job's period. Jobs that only touch process-local state (the mapping refresh, registering boost jobs) still run on every replica.LOG_JSON=true(the chart default) writes one JSON object per line withtrace_id/span_id. uvicorn access logs are split into fields.init_dbno longer logs the DSN password.uvicornin exec form. fastapi-cli re-attached rich log handlers that bypassedLOG_JSON, and the shell form blocked SIGTERM, which skipped the telemetry flush.wslc,PYTHONUTF8).pythonpath, so plainpytestworks.pyproject.tomlstill said 0.1.5 whilev0.1.6was tagged.Verified locally (Windows, wslc PG/Redis/collector)
compute_gambit_consensusranokon one replica and wasskippedon the other.traceparentis continued.?token=shows up astoken=REDACTEDin spans and access logs.wslc buildshuts down cleanly on SIGTERM and flushes its spans and metrics.