Feat/health checks

[madsysharma]

Description

Adds operational endpoints requested in #219: a Prometheus metrics scrape target and Kubernetes-style liveness/readiness probes.

New endpoints

Endpoint	Purpose	Behaviour
GET /healthz/live	Liveness probe	Returns 200 {"status":"ok"} while the process can answer HTTP. Does not check external dependencies - Kubernetes restarts the container on failure, so this must never depend on recoverable backends.
GET /healthz/ready	Readiness probe	Runs SELECT 1 against the SQLAlchemy engine. Returns 200 when all checks pass, 503 with a per-check breakdown when any fail. Kubernetes pulls the pod out of the Service load balancer on failure but does not restart it.
GET /metrics	Prometheus exposition	Counters/histograms/gauges for request totals, latency, in-flight count, and unaddressed exceptions, plus an app_info gauge.

Metric families exposed

Metric	Type	Labels	Description
qyverixai_http_requests_total	Counter	method, endpoint, status_code	Total number of requests processed.
qyverixai_http_request_duration_seconds	Histogram	method, endpoint	Request latency. Buckets: 5 ms -> 30 s.
qyverixai_http_requests_in_progress	Gauge	method, endpoint	Concurrent in-flight requests.
qyverixai_http_request_exceptions_total	Counter	method, endpoint, exception_type	Unaddressed exceptions raised during request handling.
qyverixai_app_info	Gauge	version, ai_provider	Static identity, which is always 1.

Design choices worth highlighting in review

• Route template, not raw path, as the endpoint label.: the middleware reads request.scope["route"].path after routing, so /share/abc and /share/def collapse into a single series labelled endpoint="/share/{token}". This avoids the classic Prometheus footgun where dynamic path segments balloon label cardinality. There's a dedicated test (test_metrics_uses_route_template_not_raw_path) guarding this invariant.

• /metrics is excluded from observation.: scrapes would otherwise feed back into the request_total counter on every interval. Tested via test_metrics_endpoint_excludes_itself.

• METRICS_ENABLED and METRICS_AUTH_TOKEN are read at request time, not import time.: operators can flip them without restarting; tests don't have to reload modules to exercise them (which would re-register metrics and trip Duplicated timeseries in CollectorRegistry).

• PROMETHEUS_MULTIPROC_DIR is honoured.: when running uvicorn --workers N > 1, set this env var to a writable directory and /metrics will aggregate across workers via prometheus_client.multiprocess.MultiProcessCollector.

• Optional bearer auth on /metrics.: set METRICS_AUTH_TOKEN to require Authorization: Bearer <token> on scrapes - this is useful when the endpoint is reachable from outside the cluster.

• Existing /health and /ping are untouched.: anything currently pointing at them keeps functioning.
  Files added

• backend/app/observability.py - metric definitions + HTTP middleware.

• backend/app/routers/health.py - /healthz/live, /healthz/ready.

• backend/app/routers/metrics.py - /metrics endpoint with auth and disabled handling.

• backend/tests/test_health_metrics.py - 8 tests covering happy paths, the degraded readiness path, label cardinality, scrape self-exclusion, bearer-auth gating, and the disabled state.

• deploy/k8s/deployment.example.yaml - example Deployment + Service with livenessProbe, readinessProbe, startupProbe, and Prometheus scrape annotations.

• deploy/prometheus/scrape-config.example.yaml - drop-in scrape job.
  Files modified

• backend/app/main.py - now registers the metrics middleware (installed unconditionally; self-disables per request when METRICS_ENABLED=false), includes the two new routers, and initialises the app_info gauge in the lifespan handler.

• backend/app/schemas.py - now adds LivenessResponse and ReadinessResponse.

• backend/requirements.txt - now adds prometheus-client>=0.20.0.

• Dockerfile and backend/Dockerfile - now adds a HEALTHCHECK directive that hits /healthz/live.

• .env.example - now documents METRICS_ENABLED, METRICS_AUTH_TOKEN, PROMETHEUS_MULTIPROC_DIR.

• README.md - now documents the new Observability section with metric tables, degraded-response example, and links to the deploy/ examples.
  Example: degraded readiness response

{
  "status": "degraded",
  "checks": {
    "database": {
      "ok": false,
      "elapsed_ms": 2003.41,
      "error": "OperationalError: connection refused"
    }
  }
}

Related Issue

Fixes #219

Type of change

• Bug fix
• New feature / enhancement
• Documentation update
• Test addition
• Refactor

Checklist

• I have read CONTRIBUTING.md
• My branch is up to date with main
• I have run pytest -v and all tests pass
• I have not introduced duplicate issues or features
• My PR title follows the format: feat/fix/docs/test: short description
• I have added tests for new features (Level 2 and 3 issues)
• No hardcoded secrets or API keys in my code
• This PR is linked to a GSSoC 2026 issue

Screenshots (if frontend change)

Not applicable as this is a backend-only change.

Test evidence

Full suite: 76 passed (68 pre-existing + 8 new). Ruff clean against the same selectors the CI workflow uses (ruff check backend/app --select E,F,W --ignore E501).

$ cd backend && pytest -v
...
tests/test_health_metrics.py::test_liveness_returns_ok PASSED            [ 77%]
tests/test_health_metrics.py::test_readiness_returns_ok_when_db_reachable PASSED [ 78%]
tests/test_health_metrics.py::test_readiness_returns_503_when_db_check_fails PASSED [ 80%]
tests/test_health_metrics.py::test_metrics_endpoint_returns_prometheus_format PASSED [ 81%]
tests/test_health_metrics.py::test_metrics_uses_route_template_not_raw_path PASSED [ 82%]
tests/test_health_metrics.py::test_metrics_endpoint_excludes_itself PASSED [ 84%]
tests/test_health_metrics.py::test_metrics_requires_token_when_configured PASSED [ 85%]
tests/test_health_metrics.py::test_metrics_endpoint_404_when_disabled PASSED [ 86%]
...
============================== 76 passed in 2.07s ==============================

Manual smoke test against a running instance

# Liveness - always cheap, no dependencies
$ curl -s http://localhost:8000/healthz/live
{"status":"ok"}

# Readiness - runs SELECT 1; flips to 503 + per-check breakdown when the DB is down
$ curl -s -o /dev/null -w "%{http_code}\n" http://localhost:8000/healthz/ready
200

# Prometheus scrape
$ curl -s http://localhost:8000/metrics | grep '^qyverixai_http_requests_total' | head
qyverixai_http_requests_total{endpoint="/healthz/live",method="GET",status_code="200"} 1.0
qyverixai_http_requests_total{endpoint="/healthz/ready",method="GET",status_code="200"} 2.0

Note the endpoint label values are route templates, not raw URLs.

[madsysharma]

Hi @imDarshanGK , please review this PR. Also, could you please add the "gssoc:approved" label? Thank you.

[madsysharma]

Hi @imDarshanGK , please review this PR. Also, could you please add the "gssoc:approved" label? Thank you.

[madsysharma]

Hi @imDarshanGK , please review this updated PR when you find the time to. Also, please add the 'gssoc:approved' label for this contribution to be tracked in GSSoC. Thanks.

[imDarshanGK]

@madsysharma update the branch with the latest main changes and CI failures

[madsysharma]

Hi @imDarshanGK , please check the updated PR now (have addressed the merge issues and CI errors). And please add the 'gssoc:approved' label once you've checked that it's good to go. Thank you.

[imDarshanGK]

@madsysharma update the branch with the latest main changes.

[madsysharma]

Hi @imDarshanGK , have resolved the merge conflicts. Please review this updated PR, and add the 'gssoc:approved' label once it's good to go. Thank you.

[imDarshanGK]

@madsysharma update the branch with the latest main changes, CI failing

[madsysharma]

Hi @imDarshanGK , could you please check now? Thanks.

[madsysharma]

Hi @imDarshanGK , could you please review this updated PR? Thanks.

[imDarshanGK]

@madsysharma Sync the branch with the latest main

[madsysharma]

Hi @imDarshanGK , please check now. Thank you.

[imDarshanGK]

@madsysharma Resolve conflicts

[madsysharma]

Hi @imDarshanGK , have resolved the merge conflicts. Please check. Thank you.

[madsysharma]

Hi @imDarshanGK , please review this updated PR. Thank you.

[madsysharma]

Updated the PR

[madsysharma]

Updated files in PR