Skip to content

Grafana + Loki: every service's logs, the metrics and the audit trail at one URL - #75

Merged
CodexManvik merged 1 commit into
masterfrom
feat/grafana-logs
Sep 22, 2026
Merged

CodexManvik merged 1 commit into
masterfrom
feat/grafana-logs

Conversation

@CodexManvik

Copy link
Copy Markdown
Owner

Every service's logs, the Prometheus metrics and the Postgres audit trail are now viewable at their own URL.

Deploy URL
docker-compose http://localhost:3000
Helm on kind http://localhost:30300 (or kubectl port-forward svc/grafana 3000:3000)

Grafana opens on a provisioned IntelliOps — Logs, metrics and audit dashboard, and viewing it needs no login:

  • Service logs from every container, newest first, with a service filter and a regex search box. The Explore view accepts raw LogQL, e.g. {service="action"} |= "remediation".
  • Log volume and errors/warnings per minute, by service.
  • Metrics: CPU, p99 latency, error rate and service_up from the existing Prometheus.
  • Audit trail: the latest 500 audit_records rows from Postgres.

How it works

  • Loki (single binary, 72h retention) stores the logs.
  • Grafana Alloy ships them: through the Docker socket in compose, and through the Kubernetes API in the chart. The chart uses loki.source.kubernetes with a namespace-scoped read-only Role, so there's no hostPath and no DaemonSet. Alloy drops /health, /ready and /metrics access lines, which were most of the volume, and lifts level out of JSON lines into a label.
  • One copy of the config, under deploy/k8s/platform/files/observability/. Compose bind-mounts it and Helm reads it with .Files.Get. The dashboard is generated by scripts/make-grafana-dashboard.py.
  • Chart settings: observability.enabled (on by default) turns the stack on or off. When it's on, the services switch to JSON logs, so levels are searchable. kind-config-full.yaml maps NodePort 30300; an existing cluster has to be recreated to pick that up.
  • Pinned images: grafana/grafana:11.3.0, grafana/loki:3.2.1, grafana/alloy:v1.4.3 (tags verified on Docker Hub).
  • Access: anonymous visitors are read-only Viewers. That's switchable off with GRAFANA_ANONYMOUS=false or observability.grafana.anonymous=false. The admin password is admin unless overridden.

Verification

Ran locally against the real compose stack:

  • Loki and Grafana healthy, and all three datasources (Prometheus, Loki, Postgres) report OK.
  • All 18 compose services appear in Loki, and JSON lines carry a level label.
  • The probe-noise filter works: zero /health, /ready and /metrics lines ingested after it was enabled, while other lines keep flowing.
  • In the browser, the log-volume chart, the log stream, the CPU panel (5 services) and the audit-trail table (878 rows) all render.
  • The Kubernetes Alloy config was checked with the real v1.4.3 binary. It parses and evaluates, including the drop stage and sys.env; the only errors were the expected "no in-cluster config".
  • helm lint passes, and helm template renders all resources. The embedded ConfigMap files parse as JSON/YAML, and --set observability.enabled=false renders none of them.
  • tests/test_observability_config.py checks the dashboard only references provisioned datasource UIDs, and that every file the chart reads exists.
  • The CI compose smoke now also checks Loki and Grafana readiness, datasource health, and that service logs actually reach Loki.
  • Not run: a live kind install.

Independent of #74; the two branches merge cleanly (checked with git merge-tree).

🤖 Generated with Claude Code

…cs and the audit trail at one URL

Grafana at http://localhost:3000 (compose) / http://localhost:30300 (kind),
opening on a provisioned "IntelliOps - Logs, metrics and audit" dashboard:

- Service logs from every container (filter by service, regex search), log
  volume and errors/warnings per minute by service - via Loki.
- CPU, p99 latency, error rate and service_up - via the existing Prometheus.
- The latest 500 audit_records rows - via the Postgres store.

Grafana Alloy ships the logs: through the Docker socket in compose, through
the Kubernetes API (loki.source.kubernetes, namespace-scoped read-only Role,
no hostPath/DaemonSet) in the chart. It drops the /health, /ready and /metrics
access lines, which were most of the volume, and lifts `level` out of JSON
lines. Enabling the stack in the chart switches the services to JSON logs.

One copy of the config under deploy/k8s/platform/files/observability/ - compose
bind-mounts it, Helm reads it with .Files.Get. The dashboard is generated by
scripts/make-grafana-dashboard.py. Anonymous visitors are read-only Viewers
(switchable off in both targets). Images pinned: grafana 11.3.0, loki 3.2.1,
alloy v1.4.3. CI's compose smoke now checks Loki/Grafana readiness, all three
datasources healthy, and service logs actually reaching Loki.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Copilot AI lite review requested due to automatic review settings September 22, 2026 20:07
@coderabbitai

coderabbitai Bot commented Sep 22, 2026

Copy link
Copy Markdown

Important

  • 🔍 Trigger review

This repository does not receive automatic reviews because it has fewer than 10 stars.

⚙️ Run configuration

Configuration used: defaults

Review profile: CHILL

Plan: Advanced

Run ID: b945461f-899f-4073-b8c2-7d08077e0beb


Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

Comment @coderabbitai help to get the list of available commands.

Copilot AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copilot was unable to review this pull request because the user who requested the review has reached their quota limit.

@CodexManvik
CodexManvik merged commit c24c81d into master Sep 22, 2026
6 checks passed
@CodexManvik
CodexManvik deleted the feat/grafana-logs branch September 22, 2026 20:13
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants