feat: self-hosted logging on platform (otel-collector -> Loki -> Grafana) - #44
Draft
jonathandieu wants to merge 4 commits into
Draft
feat: self-hosted logging on platform (otel-collector -> Loki -> Grafana)#44jonathandieu wants to merge 4 commits into
jonathandieu wants to merge 4 commits into
Conversation
…self-hosted Grafana - Add a loki-spaces-credentials ExternalSecret to the Loki chart (was referenced by extraEnvFrom but nothing created it). - Enable self-hosted Grafana on the platform cluster only (stage stays on Grafana Cloud), with Prometheus and Loki wired as datasources. - Add a dedicated logging ApplicationSet (loki, otel-collector), scoped to purpose=platform for now; widen the selector to include stage once this proves out end-to-end. Still needs, before logs actually persist: a DO Spaces bucket + scoped access key (terraform) and the corresponding 1Password item for loki-spaces-credentials.
Helm template diffOnly in /tmp/head: loki.yaml
Only in /tmp/head: otel-collector.yaml
diff -u --recursive --label base --label head base head
--- base
+++ head
@@ -219,7 +219,7 @@
app.kubernetes.io/managed-by: Helm
type: Opaque
data:
- SECRET_KEY_BASE: b1NCT2l3UjFGMW5kYVg2VUJLRDRCRWdBZlVWYU9qWnNydDVyMmdjQXhqYkhmdGNhZFc0djBrQUZvUlEyblNjZ2swbHBFcUliT3kzSThLTHc2dTNCNzRieWtC
+ SECRET_KEY_BASE: bXdOWHRXNjVpZndGOEp5cHF6Q0puYnBTSkd4QTNXNE1Od3BEYkFFdlNVa2ZweUVGSmFJQ0NrODR4MEFKSm1ON2lpSWlCd1FibTFqRnhUSFZ0dVZTaVQ0Q3Bw
TOTP_VAULT_KEY: ZHN4dmJuM2p4RGQxNmF6MlFwc1g1QjhPK2xseGpRMlNKRTJpNUJ6eDM4ST0=
DATABASE_URL: cG9zdGdyZXM6Ly9wb3N0Z3Jlczpwb3N0Z3Jlc0BwbGF1c2libGUtcG9zdGdyZXNxbDo1NDMyL3BsYXVzaWJsZV9kYg==
CLICKHOUSE_DATABASE_URL: aHR0cDovL2NsaWNraG91c2U6cGFzc3dvcmRAcGxhdXNpYmxlLWNsaWNraG91c2U6ODEyMy9wbGF1c2libGVfZXZlbnRzX2Ri |
The chart defaults singleBinary and gateway to `resources: {}` (unbounded).
The platform node is already tight on memory even before this deploys; give
both explicit, modest limits rather than let Loki grow unbounded on it.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Get container logs flowing on the platform cluster into a self-hosted Grafana, closing
the gap that let a stuck plausible pod sit unnoticed for 15 days. Tracking: #40.
Changes
charts/infrastructure/otel-collector/- filelog DaemonSet -> Loki OTLP.charts/infrastructure/loki/- self-hosted single-binary Loki backed by DO Spaces,plus a
loki-spaces-credentialsExternalSecret (was referenced byextraEnvFrombutnothing created it until now).
clusters/platform-do-atl1/overrides/kube-prometheus-stack.yaml- enables the bundledGrafana on platform only (stage keeps using Grafana Cloud, untouched), with
Prometheus and Loki wired as datasources.
argocd/apps/infrastructure/logging-appset.yaml- a dedicated ApplicationSet forloki+otel-collector, scoped topurpose: platformfor now. Widen the selectorto include stage once this proves out end-to-end.
Verified
helm lint+helm templateclean on both charts; kubeconform valid on all renderedresources (CRDs skipped); the new appset passes a server dry-run against the live
ApplicationSet CRD; the platform Grafana override merges correctly and injects both
datasource URLs.
Prerequisite before logs actually persist (not in this PR)
A DO Spaces bucket + scoped access key (terraform) and the corresponding 1Password item
for
loki-spaces-credentials(fields:access_key_id,secret_access_key). Until thatexists, Loki starts but sits waiting on the ExternalSecret; nothing else is affected.
Draft until that prerequisite lands and the pipeline is confirmed end-to-end.