Repository navigation
fix(observability): give grafana room to render dashboards - #254
Merged
Merged
Conversation
…ashboard On eks-dev Grafana was OOMKilled (exit 137) three times at its 256Mi limit while an operator opened the golden-signals dashboard. The contract requires at least 512Mi and fails: FAIL: Grafana's memory limit must be at least 512Mi (found 256Mi in infrastructure/grafana)
Juanmadiaz45
approved these changes
Sep 29, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What changes
infrastructure/grafana/deployment.yamlrequests 256Mi and is limited to 512Mi, instead of 128Mi and 256Mi. The full profile builds on this base and does not override it.tests/contract/observability.shrequires a limit of at least 512Mi.Why
On
eks-dev, Grafana restarted each time an operator opened the golden-signals dashboard to take the observability evidence:The port-forward then failed with
connect: connection refusedand the browser could not load the dashboard.Tasks
gitops specs/006-observability-platform-foundation T059 (added and delivered here).
How it is verified
microservice-app-ai-agents/scripts/check-mcp.sh gitops .passes. No MCP server documents Grafana's memory needs; the evidence is the OOMKilled status above.The
observabilitynamespace has no ResourceQuota or LimitRange (kubectl get resourcequota,limitrangereturns none), so the higher request schedules on the existing nodes.Risk and rollback
Grafana restarts once, and each node's reservable memory drops by 128Mi. Rollback is a revert of the merge commit.
What this PR does not do
Merge method: merge commit. This PR carries a Spec-Driven Development test/fix pair (docs #40).
Approval required: one maintainer approval (EstebanGZam or Juanmadiaz45).
🤖 Generated with Claude Code