Skip to content

feat: add bounded per-worker organization traffic logs - #965

Merged
WorksOnMyVM merged 3 commits into
mainfrom
feat/org-traffic-observability
Oct 13, 2026
Merged

WorksOnMyVM merged 3 commits into
mainfrom
feat/org-traffic-observability

Conversation

@WorksOnMyVM

@WorksOnMyVM WorksOnMyVM commented Oct 13, 2026 •

Copy link
Copy Markdown
Contributor

What this does

Adds trusted Org attribution to root call logs so operators can identify traffic/pool spikes within one or two reporting intervals. Covers /call/, /catalog/call/, /table/, and MCP call/media adapters; unresolved auth logs org_id: null. Adds the effective limit to guard 429 logs.

Main only had Uvicorn transport access logs, which MCP's internal HTTP calls bypass, and no dedicated guard busy log. These are new application-owned lines; Render edge logs are unchanged. No IP/UA, names, emails, keys, payloads, external counters, new settings/endpoints or policy changes.

One process-local timer emits Top 10 every 60s. Arrivals count after caller resolution; statuses/latency count completion after streaming and awaited cleanup. Actual live in-flight survives rollover; internal children and MCP envelopes are not extra roots. Storage is capped at 4,096 orgs with 33 latency buckets each, and overflow is explicit. p95 is a nearest-rank bucket upper bound (up to 2x overestimate).

Synthetic samples:

request {"worker_pid":123,"surface":"call","adapter":"call","org_id":42,"status":503,"outcome":"completed","handler_ms":800.2}
org_concurrency rejected org_id=42 limit=8 status=429
org_traffic_top {"worker_pid":123,"window_s":60,"overflow_requests":0,"overflow_in_flight":0,"overflow_completed":0,"top":[{"org_id":42,"requests":120,"completed":118,"in_flight_max":8,"status_429":3,"status_503":2,"p95_ms":1024}],"window_end_unix_s":1800000060.0}

Combine worker counts over timestamp-aligned windows, then filter request/rejection logs by org. Top-10 truncation makes combined counts lower bounds; summed peaks are upper bounds. Do not add/average p95s; a global p95 requires request durations. Full boundaries/operator notes are documented; composition and org-concurrency fragments/indexes are updated.

How it was tested

  • .venv/bin/python -m pytest -q tests/test_org_traffic.py tests/test_org_concurrency.py: 81 passed.
  • Final relevant regression selection: 1,028 passed, 4 skipped (callmatrix, guard, call architecture/lifecycle/cancellation/pool/streaming, relay, table/OAuth, MCP/directory, marketplace and routing). Synthetic loopback servers only; host proxy variables unset for CLI subprocess tests.
  • .venv/bin/lint-imports: 15 contracts kept. git diff --check and bash .agents/skills/tools-registry-context/scripts/drift.sh pass.
  • After the single-line formatter fix: 218 passed (test_org_traffic, guard, error capture, MCP and directory). After updating the intentional middleware composition snapshot: 37 passed (surface snapshots, roles and traffic); route/OpenAPI/lifespan snapshots are unchanged.
  • All CI checks passed on final head 73a26826821aaceb89dde36537d59036ff911b24: full Python suite, PostgreSQL/Redis regressions, frontend, gitleaks and catalog-date checks. CI run.

Branched from main 70058fbaa108eaf6d5452a2dcad6ea58bca9f0eb; #964 (8b0af9bd) verified as an ancestor before edits. No production SQL/traffic/jobs, config changes, deployment, restart or merge.

Checklist

  • Full uv run --with pytest-xdist pytest -n auto -q locally (focused/relevant regressions above; full suite and PostgreSQL/Redis run in CI)
  • Added/updated tests, including bounded storage, Top 10, cross-window occupancy, exceptions/cancellation, log failure isolation, and MCP/root deduplication
  • Updated relevant docs/context/ fragments
  • No secrets in the diff

@github-actions github-actions Bot added the area:docs Documentation & design fragments label Oct 13, 2026
@WorksOnMyVM
WorksOnMyVM marked this pull request as ready for review October 13, 2026 14:33
@WorksOnMyVM
WorksOnMyVM merged commit e337f1a into main Oct 13, 2026
10 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

area:docs Documentation & design fragments

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant