Skip to content

ROSAENG-59880: Add Grafana ingress with GitHub OAuth#616

Open
iamkirkbater wants to merge 2 commits into
openshift-online:mainfrom
iamkirkbater:ROSAENG-59880/grafana-ingress-github-auth
Open

ROSAENG-59880: Add Grafana ingress with GitHub OAuth#616
iamkirkbater wants to merge 2 commits into
openshift-online:mainfrom
iamkirkbater:ROSAENG-59880/grafana-ingress-github-auth

Conversation

@iamkirkbater

@iamkirkbater iamkirkbater commented Jun 15, 2026

Copy link
Copy Markdown
Contributor

Summary

  • Adds internet-facing Grafana access behind GitHub OAuth via oauth2-proxy, replacing the port-forward workflow
  • New Terraform module (grafana-ingress) creates an ALB with ACM TLS termination, Route53 DNS record (grafana.<deployment>.<domain>), and security groups
  • oauth2-proxy deployed as a Helm subchart in the Grafana chart, with a TargetGroupBinding connecting pod IPs to the ALB target group
  • Gated behind enable_grafana_ingress feature flag — enabled for integration, disabled by default

How it works

Browser → ALB (HTTPS/443, ACM cert) → Target Group → oauth2-proxy (HTTP/4180) → Grafana (HTTP/80)
                                                        ↕
                                                   GitHub OAuth

Manual prerequisites (per environment, one-time)

  1. Create a GitHub OAuth App with callback URL https://grafana.<deployment>.<domain>/oauth2/callback
  2. Store credentials in Secrets Manager as <regional_id>-grafana-oauth-proxy with keys: client-id, client-secret, cookie-secret

Test plan

  • make helm-lint passes (verified locally)
  • uv run scripts/render.py --check passes (verified locally)
  • terraform validate on terraform/config/regional-cluster/
  • Deploy to ephemeral — verify oauth2-proxy is not deployed (flag is false)
  • Deploy to integration — verify full flow: DNS → ALB → GitHub OAuth → Grafana
  • Verify HTTP→HTTPS redirect on port 80

JIRA: ROSAENG-59880

🤖 Generated with Claude Code

Summary by CodeRabbit

  • New Features

    • Added optional OAuth2 proxy in front of Grafana with GitHub OAuth authentication.
    • Introduced an internet-facing Grafana ingress (ALB) with automatic DNS and TLS certificate provisioning.
    • When Grafana ingress is enabled, routing is configured to the OAuth proxy target group, with required credentials loaded from a secrets manager.
  • Configuration

    • Grafana ingress enablement is now controlled by observability.grafana_ingress.enabled (default: disabled).

Expose Grafana behind an internet-facing ALB with GitHub OAuth via
oauth2-proxy, replacing the finicky port-forward workflow. Gated behind
the `enable_grafana_ingress` feature flag (enabled for integration).

Terraform module (grafana-ingress): ALB with ACM TLS termination,
Route53 DNS record, security groups. oauth2-proxy runs plain HTTP
behind the ALB; a TargetGroupBinding registers pod IPs with the
target group.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
@openshift-ci-robot openshift-ci-robot added the jira/valid-reference Indicates that this PR references a valid Jira ticket of any type. label Jun 15, 2026
@openshift-ci-robot

openshift-ci-robot commented Jun 15, 2026

Copy link
Copy Markdown
Collaborator

@iamkirkbater: This pull request references ROSAENG-59880 which is a valid jira issue.

Warning: The referenced jira issue has an invalid target version for the target branch this PR targets: expected the story to target the "5.0.0" version, but no target version was set.

Details

In response to this:

Summary

  • Adds internet-facing Grafana access behind GitHub OAuth via oauth2-proxy, replacing the port-forward workflow
  • New Terraform module (grafana-ingress) creates an ALB with ACM TLS termination, Route53 DNS record (grafana.<deployment>.<domain>), and security groups
  • oauth2-proxy deployed as a Helm subchart in the Grafana chart, with a TargetGroupBinding connecting pod IPs to the ALB target group
  • Gated behind enable_grafana_ingress feature flag — enabled for integration, disabled by default

How it works

Browser → ALB (HTTPS/443, ACM cert) → Target Group → oauth2-proxy (HTTP/4180) → Grafana (HTTP/80)
                                                       ↕
                                                  GitHub OAuth

Manual prerequisites (per environment, one-time)

  1. Create a GitHub OAuth App with callback URL https://grafana.<deployment>.<domain>/oauth2/callback
  2. Store credentials in Secrets Manager as <regional_id>-grafana-oauth-proxy with keys: client-id, client-secret, cookie-secret

Test plan

  • make helm-lint passes (verified locally)
  • uv run scripts/render.py --check passes (verified locally)
  • terraform validate on terraform/config/regional-cluster/
  • Deploy to ephemeral — verify oauth2-proxy is not deployed (flag is false)
  • Deploy to integration — verify full flow: DNS → ALB → GitHub OAuth → Grafana
  • Verify HTTP→HTTPS redirect on port 80

JIRA: ROSAENG-59880

🤖 Generated with Claude Code

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the openshift-eng/jira-lifecycle-plugin repository.

@openshift-ci

openshift-ci Bot commented Jun 15, 2026

Copy link
Copy Markdown

[APPROVALNOTIFIER] This PR is NOT APPROVED

This pull-request has been approved by:
Once this PR has been reviewed and has the lgtm label, please assign iamkirkbater for approval. For more information see the Code Review Process.

The full list of commands accepted by this bot can be found here.

Details Needs approval from an approver in each of these files:

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

@coderabbitai

coderabbitai Bot commented Jun 15, 2026

Copy link
Copy Markdown
Contributor

No actionable comments were generated in the recent review. 🎉

ℹ️ Recent review info
⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: CHILL

Plan: Enterprise

Run ID: db6a33af-1e15-478c-8a39-f89860cbdb71

📥 Commits

Reviewing files that changed from the base of the PR and between 50ce0ae and 76b986a.

📒 Files selected for processing (6)
  • argocd/config/regional-cluster/grafana/values.yaml
  • config/defaults.yaml
  • deploy/ephemeral/us-east-1/_merged_config.yaml
  • deploy/integration/us-east-1/_merged_config.yaml
  • terraform/modules/grafana-ingress/main.tf
  • terraform/modules/grafana-ingress/variables.tf
🚧 Files skipped from review as they are similar to previous changes (5)
  • argocd/config/regional-cluster/grafana/values.yaml
  • terraform/modules/grafana-ingress/variables.tf
  • config/defaults.yaml
  • terraform/modules/grafana-ingress/main.tf
  • deploy/ephemeral/us-east-1/_merged_config.yaml

📝 Walkthrough

Walkthrough

Adds an optional internet-facing Grafana OAuth2 proxy ingress. A new grafana-ingress Terraform module provisions an ALB, ACM certificate, Route53 DNS, and security groups. The module is wired into the regional cluster config, and the resulting target group ARN is propagated through ECS bootstrap and ArgoCD ApplicationSet annotations into a Grafana Helm chart that gains oauth2-proxy dependency, ExternalSecret, and TargetGroupBinding templates. Config defaults and generated environment artifacts are updated to reflect the new feature flag.

Changes

Grafana OAuth2 Proxy Ingress Feature

Layer / File(s) Summary
New grafana-ingress Terraform module
terraform/modules/grafana-ingress/variables.tf, terraform/modules/grafana-ingress/versions.tf, terraform/modules/grafana-ingress/main.tf, terraform/modules/grafana-ingress/security-groups.tf, terraform/modules/grafana-ingress/acm.tf, terraform/modules/grafana-ingress/dns.tf, terraform/modules/grafana-ingress/outputs.tf
Full new module: internet-facing ALB with target group on port 4180 and HTTP→HTTPS listeners, ACM DNS-validated certificate, Route53 A-alias record, ALB/node security groups with rules for ports 80/443/4180, and three module outputs. Input variables enforce ≥2 public subnets and domain-name regex.
Regional cluster Terraform wiring
terraform/config/regional-cluster/variables.tf, terraform/config/regional-cluster/main.tf, terraform/config/regional-cluster/outputs.tf
Adds enable_grafana_ingress boolean variable (default false), conditionally instantiates the grafana_ingress module with VPC/subnet/security-group/domain/hosted-zone inputs, and exposes grafana_target_group_arn as a conditional Terraform output.
Bootstrap and ECS ARN propagation
scripts/bootstrap-argocd.sh, terraform/modules/ecs-bootstrap/main.tf
ECS bootstrap container's kubectl apply payload annotates the cluster-identity Secret with grafana_target_group_arn. Bootstrap script reads the ARN from Terraform outputs for regional clusters and passes it as an ECS environment override.
Config feature flag and template wiring
config/defaults.yaml, config/integration/defaults.yaml, config/templates/argocd-bootstrap/applicationset.yaml.j2, config/templates/pipeline-regional-cluster-inputs/terraform.json.j2
Introduces observability.grafana_ingress.enabled (default false, true in integration), wires oauth2-proxy.enabled to that flag, adds grafana.targetGroup.arn to the ApplicationSet template, and adds enable_grafana_ingress to the Terraform pipeline-inputs template.
Grafana Helm chart oauth2-proxy integration
argocd/config/regional-cluster/grafana/Chart.yaml, argocd/config/regional-cluster/grafana/values.yaml, argocd/config/regional-cluster/grafana/templates/oauth-external-secret.yaml, argocd/config/regional-cluster/grafana/templates/targetgroupbinding.yaml
Adds oauth2-proxy Helm dependency. Extends values with global.cluster_name, oauth2-proxy block (GitHub OAuth, upstream config), and grafana.targetGroup.arn. New ExternalSecret template pulls three OAuth secrets from AWS Secrets Manager; new TargetGroupBinding template routes the service to the ALB target group.
Generated deploy artifacts (ephemeral and integration)
deploy/ephemeral/us-east-1/..., deploy/integration/us-east-1/...
Generated files for both environments gain observability.grafana_ingress, oauth2-proxy.enabled, grafana.targetGroup.arn in ApplicationSets, and enable_grafana_ingress in Terraform inputs (false for ephemeral, true for integration).

Sequence Diagram(s)

sequenceDiagram
  participant Terraform as Terraform<br/>(regional-cluster)
  participant BootstrapScript as bootstrap-argocd.sh
  participant ECSTask as ECS Bootstrap Task
  participant ClusterSecret as local-cluster-identity<br/>Secret
  participant ArgoCD as ArgoCD ApplicationSet
  participant GrafanaChart as Grafana Helm Chart
  participant ExternalSecrets as External Secrets<br/>Operator
  participant SecretsManager as AWS Secrets Manager
  participant ALB as Internet-Facing ALB

  rect rgba(99, 179, 237, 0.5)
    note over Terraform: Infrastructure provisioning
    Terraform->>ALB: create ALB, target group (port 4180), ACM cert, Route53 record
    Terraform-->>BootstrapScript: output grafana_target_group_arn
  end

  rect rgba(154, 230, 180, 0.5)
    note over BootstrapScript: Bootstrap ARN propagation
    BootstrapScript->>ECSTask: pass GRAFANA_TARGET_GROUP_ARN env var
    ECSTask->>ClusterSecret: kubectl annotate grafana_target_group_arn
  end

  rect rgba(246, 173, 85, 0.5)
    note over ClusterSecret: Helm chart rendering
    ArgoCD->>ClusterSecret: read grafana_target_group_arn annotation
    ArgoCD->>GrafanaChart: deploy with oauth2-proxy.enabled, grafana.targetGroup.arn
    GrafanaChart->>ExternalSecrets: create ExternalSecret grafana-oauth-proxy
    ExternalSecrets->>SecretsManager: fetch client-id, client-secret, cookie-secret
    SecretsManager-->>ExternalSecrets: secret values
    GrafanaChart->>ALB: create TargetGroupBinding grafana-oauth2-proxy (port 4180)
  end
Loading

Estimated code review effort

🎯 4 (Complex) | ⏱️ ~60 minutes

Suggested labels

ok-to-test

Suggested reviewers

  • syncrou
  • typeid
🚥 Pre-merge checks | ✅ 5
✅ Passed checks (5 passed)
Check name Status Explanation
Description Check ✅ Passed Check skipped - CodeRabbit’s high-level summary is enabled.
Title check ✅ Passed The title accurately reflects the main change: adding Grafana ingress with GitHub OAuth authentication. It directly maps to the primary objective of enabling internet-facing Grafana access through a new oauth2-proxy integration and ALB infrastructure.
Docstring Coverage ✅ Passed No functions found in the changed files to evaluate docstring coverage. Skipping docstring coverage check.
Linked Issues check ✅ Passed Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check ✅ Passed Check skipped because no linked issues were found for this pull request.

✏️ Tip: You can configure your own custom pre-merge checks in the settings.

✨ Finishing Touches
🧪 Generate unit tests (beta)
  • Create PR with unit tests

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

Comment @coderabbitai help to get the list of available commands and usage tips.

@coderabbitai coderabbitai Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actionable comments posted: 6

🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Inline comments:
In `@argocd/config/regional-cluster/grafana/values.yaml`:
- Around line 210-212: The YAML file has a duplicate root-level `grafana` key
that overrides the entire earlier Grafana configuration block. Instead of
defining `grafana` again with only `targetGroup.arn`, nest the `targetGroup` and
`arn` properties under the existing `grafana` key that appears earlier in the
file. Remove the duplicate `grafana` key definition and add `targetGroup.arn` as
a nested property within the original `grafana` configuration block to ensure
all Grafana settings are preserved and merged correctly.

In `@config/defaults.yaml`:
- Line 98: The docstring comment on line 98 of config/defaults.yaml incorrectly
refers to an "internet-facing NLB" when the configuration actually provisions an
"ALB" (Application Load Balancer). Update the text in the `@doc` annotation to
replace "NLB" with "ALB" to provide accurate documentation to operators who will
be setting up and debugging this Grafana ingress configuration.

In `@deploy/ephemeral/us-east-1/_merged_config.yaml`:
- Around line 100-103: The documentation comment for the grafana_ingress.enabled
configuration incorrectly specifies "NLB" (Network Load Balancer) when the
actual implementation provisions an "ALB" (Application Load Balancer). Locate
the source template file containing this doc comment (the _merged_config.yaml
file is a generated artifact), update the comment to replace "NLB" with "ALB"
for accuracy, and then regenerate the merged configuration artifact. Apply the
same correction to the other affected location mentioned in the review (around
lines 149-152) in the source template before regenerating.

In `@terraform/modules/grafana-ingress/main.tf`:
- Around line 17-27: The aws_lb resource named "grafana" is missing the security
configuration to drop invalid HTTP headers. Add the
enable_drop_invalid_header_fields attribute set to true within the aws_lb
"grafana" resource block to explicitly enable dropping of malformed and invalid
HTTP headers as a security best practice.

In `@terraform/modules/grafana-ingress/variables.tf`:
- Around line 35-37: The current regex pattern in the validation condition for
var.domain_name accepts single-label domain names (like `grafana`) which fail
later during ACM certificate provisioning. Update the regex pattern to require
at least one dot, ensuring the domain name is in FQDN (Fully Qualified Domain
Name) format with at least a domain and top-level domain (e.g.,
`grafana.example.com`). Modify the condition in the validation block to enforce
this requirement while keeping the character restrictions.
- Around line 1-4: The regional_id variable lacks length validation, which can
cause ALB and target-group resource creation failures when the variable is too
long. Add a validation block to the regional_id variable in the variables
definition that enforces a reasonable maximum length constraint. This will
ensure that when regional_id is used in constructing resource names like
${var.regional_id}-grafana, the resulting names stay within AWS ALB and
target-group name length limits. Determine the appropriate max length by
accounting for the "-grafana" suffix and AWS naming constraints.
🪄 Autofix (Beta)

Fix all unresolved CodeRabbit comments on this PR:

  • Push a commit to this branch (recommended)
  • Create a new PR with the fixes

ℹ️ Review info
⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: CHILL

Plan: Enterprise

Run ID: 41fbecd0-3af8-4f0b-9b08-44b60a732ceb

📥 Commits

Reviewing files that changed from the base of the PR and between 8decb36 and 50ce0ae.

📒 Files selected for processing (30)
  • argocd/config/regional-cluster/grafana/Chart.yaml
  • argocd/config/regional-cluster/grafana/templates/oauth-external-secret.yaml
  • argocd/config/regional-cluster/grafana/templates/targetgroupbinding.yaml
  • argocd/config/regional-cluster/grafana/values.yaml
  • config/defaults.yaml
  • config/integration/defaults.yaml
  • config/templates/argocd-bootstrap/applicationset.yaml.j2
  • config/templates/pipeline-regional-cluster-inputs/terraform.json.j2
  • deploy/ephemeral/us-east-1/_merged_config.yaml
  • deploy/ephemeral/us-east-1/argocd-bootstrap-management-cluster/applicationset.yaml
  • deploy/ephemeral/us-east-1/argocd-bootstrap-regional-cluster/applicationset.yaml
  • deploy/ephemeral/us-east-1/argocd-values-regional-cluster.yaml
  • deploy/ephemeral/us-east-1/pipeline-regional-cluster-inputs/terraform.json
  • deploy/integration/us-east-1/_merged_config.yaml
  • deploy/integration/us-east-1/argocd-bootstrap-management-cluster/applicationset.yaml
  • deploy/integration/us-east-1/argocd-bootstrap-regional-cluster/applicationset.yaml
  • deploy/integration/us-east-1/argocd-values-regional-cluster.yaml
  • deploy/integration/us-east-1/pipeline-regional-cluster-inputs/terraform.json
  • scripts/bootstrap-argocd.sh
  • terraform/config/regional-cluster/main.tf
  • terraform/config/regional-cluster/outputs.tf
  • terraform/config/regional-cluster/variables.tf
  • terraform/modules/ecs-bootstrap/main.tf
  • terraform/modules/grafana-ingress/acm.tf
  • terraform/modules/grafana-ingress/dns.tf
  • terraform/modules/grafana-ingress/main.tf
  • terraform/modules/grafana-ingress/outputs.tf
  • terraform/modules/grafana-ingress/security-groups.tf
  • terraform/modules/grafana-ingress/variables.tf
  • terraform/modules/grafana-ingress/versions.tf

Comment thread argocd/config/regional-cluster/grafana/values.yaml Outdated
Comment thread config/defaults.yaml Outdated
Comment on lines +100 to +103
grafana_ingress:
# @doc observability.grafana_ingress.enabled Enable Grafana OAuth proxy (internet-facing NLB + GitHub OAuth). Requires Secrets Manager secrets for GitHub OAuth credentials and TLS cert.
# @used-by observability.grafana_ingress.enabled _context
enabled: false

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

⚠️ Potential issue | 🟡 Minor | ⚡ Quick win

Correct load balancer type in the doc comment.

Line 101 says “NLB”, but this rollout provisions an ALB. Please update the source template comment and regenerate this artifact to avoid operational confusion.

Also applies to: 149-152

🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@deploy/ephemeral/us-east-1/_merged_config.yaml` around lines 100 - 103, The
documentation comment for the grafana_ingress.enabled configuration incorrectly
specifies "NLB" (Network Load Balancer) when the actual implementation
provisions an "ALB" (Application Load Balancer). Locate the source template file
containing this doc comment (the _merged_config.yaml file is a generated
artifact), update the comment to replace "NLB" with "ALB" for accuracy, and then
regenerate the merged configuration artifact. Apply the same correction to the
other affected location mentioned in the review (around lines 149-152) in the
source template before regenerating.

Comment thread terraform/modules/grafana-ingress/main.tf
Comment on lines +1 to +4
variable "regional_id" {
description = "Regional cluster identifier for resource naming"
type = string
}

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

⚠️ Potential issue | 🟠 Major | ⚡ Quick win

Add a max-length validation for regional_id to prevent ALB/TG name apply failures.

Line 18 and Line 41 in terraform/modules/grafana-ingress/main.tf build names as ${var.regional_id}-grafana. ALB and target-group names are length-limited, so an unbounded regional_id can fail at apply time.

Suggested fix
 variable "regional_id" {
   description = "Regional cluster identifier for resource naming"
   type        = string
+
+  validation {
+    condition     = length(var.regional_id) <= 24
+    error_message = "regional_id must be <= 24 characters to satisfy ALB/target group naming limits."
+  }
 }
📝 Committable suggestion

‼️ IMPORTANT
Carefully review the code before committing. Ensure that it accurately replaces the highlighted code, contains no missing lines, and has no issues with indentation. Thoroughly test & benchmark the code to ensure it meets the requirements.

Suggested change
variable "regional_id" {
description = "Regional cluster identifier for resource naming"
type = string
}
variable "regional_id" {
description = "Regional cluster identifier for resource naming"
type = string
validation {
condition = length(var.regional_id) <= 24
error_message = "regional_id must be <= 24 characters to satisfy ALB/target group naming limits."
}
}
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@terraform/modules/grafana-ingress/variables.tf` around lines 1 - 4, The
regional_id variable lacks length validation, which can cause ALB and
target-group resource creation failures when the variable is too long. Add a
validation block to the regional_id variable in the variables definition that
enforces a reasonable maximum length constraint. This will ensure that when
regional_id is used in constructing resource names like
${var.regional_id}-grafana, the resulting names stay within AWS ALB and
target-group name length limits. Determine the appropriate max length by
accounting for the "-grafana" suffix and AWS naming constraints.

Comment thread terraform/modules/grafana-ingress/variables.tf Outdated
- Move targetGroup.arn under existing grafana subchart key to avoid
  duplicate root-level YAML key that would override subchart config
- Fix NLB→ALB in @doc annotation and remove stale TLS cert mention
- Add drop_invalid_header_fields on internet-facing ALB
- Require FQDN (at least one dot) in domain_name validation

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
@openshift-ci

openshift-ci Bot commented Jun 18, 2026

Copy link
Copy Markdown

PR needs rebase.

Details

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.

@openshift-ci openshift-ci Bot added the needs-rebase Indicates a PR cannot be merged because it has merge conflicts with HEAD. label Jun 18, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

jira/valid-reference Indicates that this PR references a valid Jira ticket of any type. needs-rebase Indicates a PR cannot be merged because it has merge conflicts with HEAD.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants