Repository navigation
Grafana + Loki: every service's logs, the metrics and the audit trail at one URL - #75
Merged
Merged
Conversation
…cs and the audit trail at one URL Grafana at http://localhost:3000 (compose) / http://localhost:30300 (kind), opening on a provisioned "IntelliOps - Logs, metrics and audit" dashboard: - Service logs from every container (filter by service, regex search), log volume and errors/warnings per minute by service - via Loki. - CPU, p99 latency, error rate and service_up - via the existing Prometheus. - The latest 500 audit_records rows - via the Postgres store. Grafana Alloy ships the logs: through the Docker socket in compose, through the Kubernetes API (loki.source.kubernetes, namespace-scoped read-only Role, no hostPath/DaemonSet) in the chart. It drops the /health, /ready and /metrics access lines, which were most of the volume, and lifts `level` out of JSON lines. Enabling the stack in the chart switches the services to JSON logs. One copy of the config under deploy/k8s/platform/files/observability/ - compose bind-mounts it, Helm reads it with .Files.Get. The dashboard is generated by scripts/make-grafana-dashboard.py. Anonymous visitors are read-only Viewers (switchable off in both targets). Images pinned: grafana 11.3.0, loki 3.2.1, alloy v1.4.3. CI's compose smoke now checks Loki/Grafana readiness, all three datasources healthy, and service logs actually reaching Loki. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
|
Important
This repository does not receive automatic reviews because it has fewer than 10 stars. ⚙️ Run configurationConfiguration used: defaults Review profile: CHILL Plan: Advanced Run ID: Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Every service's logs, the Prometheus metrics and the Postgres audit trail are now viewable at their own URL.
kubectl port-forward svc/grafana 3000:3000)Grafana opens on a provisioned IntelliOps — Logs, metrics and audit dashboard, and viewing it needs no login:
{service="action"} |= "remediation".service_upfrom the existing Prometheus.audit_recordsrows from Postgres.How it works
loki.source.kuberneteswith a namespace-scoped read-only Role, so there's no hostPath and no DaemonSet. Alloy drops/health,/readyand/metricsaccess lines, which were most of the volume, and liftslevelout of JSON lines into a label.deploy/k8s/platform/files/observability/. Compose bind-mounts it and Helm reads it with.Files.Get. The dashboard is generated byscripts/make-grafana-dashboard.py.observability.enabled(on by default) turns the stack on or off. When it's on, the services switch to JSON logs, so levels are searchable.kind-config-full.yamlmaps NodePort 30300; an existing cluster has to be recreated to pick that up.grafana/grafana:11.3.0,grafana/loki:3.2.1,grafana/alloy:v1.4.3(tags verified on Docker Hub).GRAFANA_ANONYMOUS=falseorobservability.grafana.anonymous=false. The admin password isadminunless overridden.Verification
Ran locally against the real compose stack:
levellabel./health,/readyand/metricslines ingested after it was enabled, while other lines keep flowing.sys.env; the only errors were the expected "no in-cluster config".helm lintpasses, andhelm templaterenders all resources. The embedded ConfigMap files parse as JSON/YAML, and--set observability.enabled=falserenders none of them.tests/test_observability_config.pychecks the dashboard only references provisioned datasource UIDs, and that every file the chart reads exists.Independent of #74; the two branches merge cleanly (checked with
git merge-tree).🤖 Generated with Claude Code