diff --git a/README.md b/README.md index ddc2599..3d00301 100644 --- a/README.md +++ b/README.md @@ -28,7 +28,6 @@ See [docs/DEPLOYMENT.md](docs/DEPLOYMENT.md) for detailed setup instructions. . ├── authentik/ # Authentik SSO identity provider (prd-apps) ├── harbor/ # Harbor registry deployment -├── github-runner/ # GitHub Actions runners ├── gitlab-runner/ # GitLab CI/CD runners ├── grafana/ # Grafana Stack (Loki + Promtail + Grafana) ├── scripts/ # Setup and utility scripts diff --git a/docs/DEPLOYMENT.md b/docs/DEPLOYMENT.md index 91dfc19..14139be 100644 --- a/docs/DEPLOYMENT.md +++ b/docs/DEPLOYMENT.md @@ -1,6 +1,6 @@ # Deployment Guide -This guide walks you through deploying Harbor, GitHub, and GitLab runners to your cluster. +This guide walks you through deploying Harbor and the GitLab runner to your cluster. ## Prerequisites @@ -10,22 +10,13 @@ This guide walks you through deploying Harbor, GitHub, and GitLab runners to you - RBAC enabled - Sufficient resources for runners -2. **GitHub Access** (for GitHub Runner): - - GitHub Personal Access Token (PAT) with `repo` scope, OR - - GitHub App credentials - -3. **GitLab Access** (for GitLab Runner): +2. **GitLab Access** (for GitLab Runner): - GitLab instance URL - Runner authentication token (`glrt-*`, created via GitLab UI) ## Quick Start -### Step 1: Get Tokens - -**GitHub Token:** -1. Go to: https://github.com/settings/tokens -2. Create token with `repo` and `admin:org` scopes -3. Copy the token +### Step 1: Get Token **GitLab Token:** 1. Go to your GitLab RaaS group @@ -41,15 +32,12 @@ Run the setup script: ./scripts/runner-setup.sh # OR non-interactive mode -GITHUB_TOKEN= GITLAB_TOKEN= GITLAB_URL= ./scripts/runner-setup.sh all +GITLAB_TOKEN= GITLAB_URL= ./scripts/runner-setup.sh gitlab ``` ### Step 3: Update Configuration -1. **GitHub Runner**: Edit `github-runner/base/runnerdeployment.yaml` - - Replace `` with your organization name - -2. **GitLab Runner**: Edit `gitlab-runner/base/gitlab-runner-helmchart.yaml` +1. **GitLab Runner**: Edit `gitlab-runner/base/gitlab-runner-helmchart.yaml` - Set `gitlabUrl` to your GitLab instance URL - Use `./scripts/runner-config.sh` to update token via HelmChartConfig @@ -57,7 +45,7 @@ GITHUB_TOKEN= GITLAB_TOKEN= GITLAB_URL= ./scripts/runner-setu ```bash git add . -git commit -m "feat: configure runners" +git commit -m "feat: configure gitlab runner" git push ``` @@ -70,31 +58,9 @@ Fleet will automatically deploy! ```bash # Create managed-cicd namespace (if it doesn't exist) kubectl create namespace managed-cicd --dry-run=client -o yaml | kubectl apply -f - - -# The actions-runner-system namespace will be created by Helm ``` -### Step 2: Create GitHub Authentication Secret - -**Option A: Using the script (Recommended)** - -```bash -./scripts/runner-setup.sh github -``` - -**Option B: Manual creation** - -```bash -# Create namespace -kubectl create namespace actions-runner-system - -# Create secret with PAT -kubectl create secret generic actions-runner-controller \ - --from-literal=github_token='' \ - -n actions-runner-system -``` - -### Step 3: Create GitLab Runner Token Secret +### Step 2: Create GitLab Runner Token Secret **Option A: Using the script (Recommended)** @@ -111,21 +77,7 @@ kubectl create secret generic gitlab-runner-secret \ -n managed-cicd ``` -### Step 4: Update Configuration Files - -**GitHub Runner:** - -1. Edit `github-runner/base/runnerdeployment.yaml`: - - Update `repository: /` - - Or change to `organization: ` for org-level runners - -2. (Optional) Adjust autoscaling in `github-runner/base/horizontalrunnerautoscaler.yaml`: - - `minReplicas`: Minimum number of runners (default: 1) - - `maxReplicas`: Maximum number of runners (default: 10) - - `scaleUpThreshold`: When to scale up (default: 0.75 = 75% busy) - - `scaleDownThreshold`: When to scale down (default: 0.25 = 25% busy) - -**GitLab Runner:** +### Step 3: Update Configuration Files 1. Edit `gitlab-runner/base/gitlab-runner-helmchart.yaml`: - Update `gitlabUrl: https://gitlab.com` (or your GitLab instance URL) @@ -141,7 +93,7 @@ kubectl create secret generic gitlab-runner-secret \ - Current: `concurrent: 4` (4 parallel jobs) - Increase for more capacity (e.g., `concurrent: 10`) -### Step 5: Configure Fleet GitRepo +### Step 4: Configure Fleet GitRepo Ensure your Fleet GitRepo is monitoring the appropriate paths: @@ -155,12 +107,11 @@ spec: repo: branch: main paths: - - github-runner/overlays/nprd-apps - gitlab-runner/overlays/nprd-apps - harbor/overlays/nprd-apps ``` -### Step 6: Update Fleet Cluster Targeting +### Step 5: Update Fleet Cluster Targeting Edit the `fleet.yaml` files in the overlay directories to match your cluster labels: @@ -169,7 +120,6 @@ Edit the `fleet.yaml` files in the overlay directories to match your cluster lab kubectl get clusters.management.cattle.io -o yaml | grep -A 10 labels # Update fleet.yaml files: -# - github-runner/overlays/nprd-apps/fleet.yaml # - gitlab-runner/overlays/nprd-apps/fleet.yaml # - harbor/overlays/nprd-apps/fleet.yaml ``` @@ -183,16 +133,16 @@ targetCustomizations: managed.cattle.io/cluster-name: nprd-apps ``` -### Step 7: Commit and Push Changes +### Step 6: Commit and Push Changes ```bash # Commit your configuration changes git add . -git commit -m "feat: configure runners and Harbor for deployment" +git commit -m "feat: configure gitlab runner and Harbor for deployment" git push ``` -### Step 8: Monitor Deployment +### Step 7: Monitor Deployment **Check Fleet Status:** @@ -206,21 +156,6 @@ kubectl get bundle -n fleet-default kubectl describe bundle -n fleet-default ``` -**Check GitHub Runner Controller:** - -```bash -# Check controller pod -kubectl get pods -n actions-runner-system -kubectl logs -n actions-runner-system -l app=actions-runner-controller - -# Check RunnerDeployment -kubectl get runnerdeployment -n managed-cicd -kubectl describe runnerdeployment github-runner-deployment -n managed-cicd - -# Check runner pods -kubectl get pods -n managed-cicd -l runner-deployment-name=github-runner-deployment -``` - **Check GitLab Runner:** ```bash @@ -233,16 +168,7 @@ kubectl get helmchart -n managed-cicd kubectl describe helmchart gitlab-runner -n managed-cicd ``` -### Step 9: Verify Runners are Active - -**GitHub Runner:** - -1. Go to your GitHub repository -2. Navigate to **Settings** → **Actions** → **Runners** -3. Verify runners appear with status "Online" -4. Check that autoscaling is working by triggering a workflow - -**GitLab Runner:** +### Step 8: Verify Runners are Active 1. Go to your GitLab project/group/instance 2. Navigate to **Settings** → **CI/CD** → **Runners** @@ -251,35 +177,6 @@ kubectl describe helmchart gitlab-runner -n managed-cicd ## Token Setup Details -### GitHub Organization Runner Token - -For organization-level runners, you need a GitHub Personal Access Token (PAT) or GitHub App. - -**Option 1: Personal Access Token (Recommended for quick setup)** - -1. Go to GitHub → Settings → Developer settings → Personal access tokens → Tokens (classic) -2. Click "Generate new token (classic)" -3. Give it a name (e.g., "Kubernetes Runner Controller") -4. Select scopes: - - ✅ `repo` (Full control of private repositories) - - ✅ `admin:org` (if managing organization runners) -5. Click "Generate token" -6. **Copy the token immediately** (you won't see it again) - -**Option 2: GitHub App (Recommended for organizations)** - -1. Go to your organization → Settings → Developer settings → GitHub Apps -2. Click "New GitHub App" -3. Configure: - - Name: "Kubernetes Runner Controller" - - Homepage URL: Your organization URL - - Permissions: - - Actions: Read and write - - Metadata: Read-only -4. Generate a private key -5. Install the app on your organization -6. Note the App ID, Installation ID, and save the private key - ### GitLab Group Runner Token (RaaS Group) 1. Go to your GitLab instance @@ -292,60 +189,8 @@ For organization-level runners, you need a GitHub Personal Access Token (PAT) or - Ensure you have Maintainer/Owner permissions on the group - Or create an instance-level runner from Admin Area → Runners -## GitHub Organization Setup - -### Option 1: Create New DataKnife Organization (Recommended) - -**Pros:** -- Keeps personal account separate -- Better aligns with your domain (dataknife.net) -- More professional setup -- Can transfer repos as needed - -**Steps:** -1. Go to https://github.com/organizations/new -2. Choose organization name: `DataKnife` or `dataknife` -3. Choose plan (Free tier works for most cases) -4. Create organization -5. Transfer repositories from personal account to `DataKnife` (optional) -6. Update runner configuration to use `DataKnife` organization - -### Option 2: Convert Personal Account to Organization - -**Pros:** -- Keeps existing repositories in place -- No need to transfer repos - -**Cons:** -- **Irreversible** - cannot convert back to personal account -- Requires creating a new personal account first -- Some personal data won't transfer (SSH keys, OAuth tokens, etc.) - ## Troubleshooting -### GitHub Runner Issues - -**Controller not starting:** -```bash -# Check secret exists -kubectl get secret actions-runner-controller -n actions-runner-system - -# Check controller logs -kubectl logs -n actions-runner-system -l app=actions-runner-controller -``` - -**Runners not appearing:** -```bash -# Check RunnerDeployment status -kubectl describe runnerdeployment github-runner-deployment -n managed-cicd - -# Check autoscaler status -kubectl describe horizontalrunnerautoscaler github-runner-autoscaler -n managed-cicd - -# Check for runner pods -kubectl get pods -n managed-cicd -l runner-deployment-name=github-runner-deployment -``` - ### GitLab Runner Issues **Runner not registering:** @@ -373,7 +218,7 @@ kubectl auth can-i create pods --namespace=managed-cicd ## Next Steps After Deployment -1. **Configure runner labels** (GitHub) or **tags** (GitLab) for workflow targeting +1. **Configure runner tags** for job targeting 2. **Adjust resource limits** based on your workload requirements 3. **Monitor autoscaling behavior** and tune thresholds if needed 4. **Set up monitoring/alerting** for runner health @@ -381,6 +226,5 @@ kubectl auth can-i create pods --namespace=managed-cicd ## Additional Resources -- [GitHub Actions Runner Controller Docs](https://github.com/actions/actions-runner-controller) - [GitLab Runner Kubernetes Executor Docs](https://docs.gitlab.com/runner/executors/kubernetes/) - [Fleet Documentation](https://fleet.rancher.io/) diff --git a/docs/FLEET_STRUCTURE.md b/docs/FLEET_STRUCTURE.md index 4abf3e3..18800ee 100644 --- a/docs/FLEET_STRUCTURE.md +++ b/docs/FLEET_STRUCTURE.md @@ -64,18 +64,6 @@ Each overlay directory contains **all necessary files** (copied from base): ``` . -├── github-runner/ -│ ├── base/ # Base configuration (reference only) -│ │ ├── kustomization.yaml -│ │ ├── github-runner-controller-helmchart.yaml -│ │ └── runnerdeployment.yaml -│ └── overlays/ -│ └── nprd-apps/ # Cluster-specific overlay (deployed) -│ ├── fleet.yaml # Cluster targeting -│ ├── kustomization.yaml -│ ├── github-runner-controller-helmchart.yaml # Copied from base -│ └── runnerdeployment.yaml # Copied from base -│ ├── gitlab-runner/ │ ├── base/ # Base configuration (reference only) │ └── overlays/ @@ -172,7 +160,6 @@ Base directories serve as: Fleet creates bundles with names based on the monitored path: -- `github-runner/overlays/nprd-apps` → `gitops-tools-nprd-apps-github-runner-overlays-n-` - `harbor/overlays/nprd-apps` → `gitops-tools-nprd-apps-harbor-overlays-nprd-apps` - `grafana/overlays/nprd-apps` → `gitops-tools-nprd-apps-grafana-overlays-nprd-apps` diff --git a/docs/GITHUB_RUNNER.md b/docs/GITHUB_RUNNER.md deleted file mode 100644 index 81e8d08..0000000 --- a/docs/GITHUB_RUNNER.md +++ /dev/null @@ -1,26 +0,0 @@ -# GitHub Runner Documentation - -This directory contains comprehensive documentation for GitHub Actions Runner setup and configuration. - -## Documentation Index - -- **[OFFICIAL_ARC.md](github-runner/OFFICIAL_ARC.md)** - Official vs Community ARC Controller comparison -- **[MIGRATION_TO_OFFICIAL_ARC.md](github-runner/MIGRATION_TO_OFFICIAL_ARC.md)** - Guide for migrating to official ARC -- **[RUNNER_GROUPS.md](github-runner/RUNNER_GROUPS.md)** - Runner groups configuration guide -- **[RUNNER_GROUP_ISSUE.md](github-runner/RUNNER_GROUP_ISSUE.md)** - Troubleshooting runner group assignment -- **[SETUP_ORG_RUNNERS.md](github-runner/SETUP_ORG_RUNNERS.md)** - Setting up organization-level runners - -## Quick Links - -- [GitHub Actions Runner Controller Official Docs](https://docs.github.com/en/actions/tutorials/use-actions-runner-controller) -- [Runner Groups Documentation](https://docs.github.com/en/actions/hosting-your-own-runners/managing-access-to-self-hosted-runners-using-groups) - -## Current Setup - -This repository uses GitHub Actions Runner Controller to manage self-hosted runners in Kubernetes. - -- **Controller**: Official GitHub-supported ARC (`gha-runner-scale-set-controller`) -- **Runner Group**: NRPD Auto Scale -- **Organization**: DataKnifeAI -- **Namespace**: managed-cicd - diff --git a/docs/README.md b/docs/README.md index fb4db4f..3fb9912 100644 --- a/docs/README.md +++ b/docs/README.md @@ -4,14 +4,6 @@ This directory contains comprehensive documentation for all tools and services m ## Documentation Index -### GitHub Runner -- **[GitHub Runner Overview](GITHUB_RUNNER.md)** - Complete guide for GitHub Actions Runner Controller - - [Official ARC Controller](github-runner/OFFICIAL_ARC.md) - Official vs Community ARC comparison - - [Migration Guide](github-runner/MIGRATION_TO_OFFICIAL_ARC.md) - Migrating to official ARC - - [Runner Groups](github-runner/RUNNER_GROUPS.md) - Runner groups configuration - - [Runner Group Troubleshooting](github-runner/RUNNER_GROUP_ISSUE.md) - Troubleshooting runner group issues - - [Setup Organization Runners](github-runner/SETUP_ORG_RUNNERS.md) - Setting up org-level runners - ### GitLab Runner - **[GitLab Runner Base Configuration](../gitlab-runner/base/README.md)** - GitLab CI/CD runner setup, configuration, and Harbor integration @@ -42,7 +34,6 @@ This directory contains comprehensive documentation for all tools and services m ``` . ├── docs/ # Documentation (this directory) -├── github-runner/ # GitHub Actions Runner Controller ├── gitlab-runner/ # GitLab Runner ├── harbor/ # Harbor container registry ├── grafana/ # Grafana Stack (Loki + Promtail + Grafana) diff --git a/docs/github-runner/MIGRATION_TO_OFFICIAL_ARC.md b/docs/github-runner/MIGRATION_TO_OFFICIAL_ARC.md deleted file mode 100644 index fb9112d..0000000 --- a/docs/github-runner/MIGRATION_TO_OFFICIAL_ARC.md +++ /dev/null @@ -1,222 +0,0 @@ -# Migration to Official ARC Controller - -## Current Status - -**✅ Official Controller Deployed:** -- Controller: `gha-runner-scale-set-controller-gha-rs-controller` -- Version: `0.13.1` -- Image: `ghcr.io/actions/gha-runner-scale-set-controller` -- Status: Running - -**⚠️ Old Controller Still Running:** -- Controller: `actions-runner-controller` (community version) -- Version: `v0.27.6` -- Image: `summerwind/actions-runner-controller` -- Status: Still managing old RunnerDeployment - -**📋 Old Configuration Still Active:** -- RunnerDeployment: `github-runner-deployment` -- Runners: 2 active -- Group: "NRPD Auto Scale" - -## Next Steps - -### Step 1: Create AutoscalingRunnerSet - -Two options: - -**Option A: Using HelmChart (Recommended for Fleet)** -- Use `gha-runner-scale-set-helmchart.yaml` -- Fleet will manage via HelmChart resource -- Easier to maintain with Fleet - -**Option B: Direct AutoscalingRunnerSet Resource** -- Use `autoscalingrunnerset.yaml` -- Direct CRD deployment -- More explicit control - -### Step 2: Authentication Setup - -Official ARC requires authentication. Two options: - -**Option A: GitHub App (Recommended)** -```bash -# Create GitHub App in organization settings -# Install on organization -# Get App ID, Installation ID, and private key - -kubectl create secret generic github-app-secret \ - --from-literal=github_app_id='' \ - --from-literal=github_app_installation_id='' \ - --from-literal=github_app_private_key='' \ - -n managed-cicd -``` - -**Option B: Personal Access Token (PAT)** -```bash -# Less secure but simpler -# Create PAT with admin:org scope - -kubectl create secret generic github-pat-secret \ - --from-literal=github_token='' \ - -n managed-cicd -``` - -### Step 3: Deploy AutoscalingRunnerSet - -**If using HelmChart:** -```bash -# Update kustomization.yaml to include helmchart -# Commit and push - Fleet will deploy -``` - -**If using direct resource:** -```bash -kubectl apply -f autoscalingrunnerset.yaml -``` - -### Step 4: Verify New Runners - -1. Check AutoscalingRunnerSet: - ```bash - kubectl --context=nprd-apps get autoscalingrunnerset -n managed-cicd - ``` - -2. Check Listeners: - ```bash - kubectl --context=nprd-apps get autoscalinglistener -n managed-cicd - ``` - -3. Check Ephemeral Runners: - ```bash - kubectl --context=nprd-apps get ephemeralrunner -n managed-cicd - ``` - -4. Check in GitHub: - - Go to: https://github.com/organizations/DataKnifeAI/settings/actions/runners - - Click on "NRPD Auto Scale" group - - Verify new runners appear (different names) - -### Step 5: Test Workflows - -1. Trigger a workflow -2. Verify it uses new runners -3. Check runner logs -4. Verify ephemeral runners clean up after job - -### Step 6: Remove Old Configuration - -Once new runners are working: - -1. **Remove RunnerDeployment:** - ```bash - kubectl --context=nprd-apps delete runnerdeployment github-runner-deployment -n managed-cicd - ``` - -2. **Remove HorizontalRunnerAutoscaler:** - ```bash - kubectl --context=nprd-apps delete horizontalrunnerautoscaler github-runner-autoscaler -n managed-cicd - ``` - -3. **Wait for old runners to clean up:** - ```bash - kubectl --context=nprd-apps get runner -n managed-cicd - # Wait until all runners are deleted - ``` - -4. **(Optional) Remove old controller:** - ```bash - # Only if you're sure you don't need it - kubectl --context=nprd-apps delete helmchart actions-runner-controller -n managed-cicd - ``` - -## Key Differences - -| Feature | Community (Old) | Official (New) | -|---------|----------------|----------------| -| **CRD** | RunnerDeployment | AutoscalingRunnerSet | -| **Architecture** | Direct pods | Listener + Ephemeral | -| **Labels** | Single label only | Multiple labels supported | -| **Scaling** | HorizontalRunnerAutoscaler | Built-in scaling | -| **Ephemeral** | Optional field | Default behavior | -| **Authentication** | PAT or GitHub App | GitHub App recommended | -| **Runner Groups** | Limited support | Full support | - -## Configuration Mapping - -**Old (RunnerDeployment):** -```yaml -spec: - replicas: 1 - template: - spec: - organization: DataKnifeAI - group: NRPD Auto Scale - labels: [self-hosted] - ephemeral: true -``` - -**New (AutoscalingRunnerSet):** -```yaml -spec: - githubConfigUrl: https://github.com/DataKnifeAI - runnerGroup: NRPD Auto Scale - runnerScaleSetName: nprd-autoscale-runners - minRunners: 2 - maxRunners: 10 - labels: [self-hosted, kubernetes, linux] - # Ephemeral is default - no need to specify -``` - -## Benefits of Official ARC - -✅ **Better Runner Group Support**: Full support for runner groups -✅ **Multiple Labels**: Can use multiple labels (unlike community version) -✅ **Efficient Scaling**: Ephemeral runners are default and more efficient -✅ **Official Support**: GitHub maintains and supports -✅ **Better Metrics**: Built-in Prometheus metrics -✅ **Future-Proof**: Active development by GitHub - -## Troubleshooting - -### Runners Not Appearing - -1. **Check AutoscalingRunnerSet status:** - ```bash - kubectl --context=nprd-apps describe autoscalingrunnerset -n managed-cicd - ``` - -2. **Check listener logs:** - ```bash - kubectl --context=nprd-apps logs -n managed-cicd -l app.kubernetes.io/name=gha-rs-listener - ``` - -3. **Check controller logs:** - ```bash - kubectl --context=nprd-apps logs -n actions-runner-system -l app.kubernetes.io/name=gha-rs-controller - ``` - -4. **Verify runner group exists:** - - https://github.com/organizations/DataKnifeAI/settings/actions/runners - - Ensure "NRPD Auto Scale" group exists - -### Authentication Issues - -1. **Check secret exists:** - ```bash - kubectl --context=nprd-apps get secret github-app-secret -n managed-cicd - ``` - -2. **Verify secret keys:** - ```bash - kubectl --context=nprd-apps get secret github-app-secret -n managed-cicd -o jsonpath='{.data}' | jq 'keys' - ``` - -3. **Check controller logs for auth errors** - -## References - -- [Official ARC Documentation](https://docs.github.com/en/actions/tutorials/use-actions-runner-controller) -- [Deploy Runner Scale Sets](https://docs.github.com/en/actions/tutorials/use-actions-runner-controller/deploy-runner-scale-sets) -- [Quickstart Guide](https://docs.github.com/en/actions/tutorials/quickstart-for-actions-runner-controller) -- [Authentication](https://docs.github.com/en/actions/tutorials/use-actions-runner-controller/authenticate-to-the-api) diff --git a/docs/github-runner/OFFICIAL_ARC.md b/docs/github-runner/OFFICIAL_ARC.md deleted file mode 100644 index 564ee00..0000000 --- a/docs/github-runner/OFFICIAL_ARC.md +++ /dev/null @@ -1,179 +0,0 @@ -# Official Actions Runner Controller (ARC) - -## Overview - -There are **two versions** of Actions Runner Controller: - -1. **Community Version** (what you're currently using) -2. **Official GitHub-Supported Version** (newer, recommended) - -## Community Version (Current Setup) - -**Repository:** `summerwind/actions-runner-controller` - -**Status:** Community-maintained, still active but not officially supported by GitHub - -**CRDs:** -- `RunnerDeployment` (actions.summerwind.dev/v1alpha1) -- `RunnerReplicaSet` -- `Runner` -- `HorizontalRunnerAutoscaler` - -**Architecture:** -- Direct runner pod management -- Runner pods register directly with GitHub -- Simpler but less scalable - -**Helm Chart:** -```yaml -chart: actions-runner-controller -repo: https://actions-runner-controller.github.io/actions-runner-controller -image: summerwind/actions-runner-controller -``` - -**Current Version:** v0.27.6 - -**Pros:** -- ✅ Mature and stable -- ✅ Well-documented -- ✅ Large community -- ✅ Simple architecture - -**Cons:** -- ❌ Not officially supported by GitHub -- ❌ Limited runner group support -- ❌ Less efficient scaling -- ❌ No official SLA - -## Official GitHub-Supported Version - -**Repository:** `actions/actions-runner-controller` (GitHub official) - -**Status:** Officially developed and maintained by GitHub - -**CRDs:** -- `AutoscalingRunnerSet` (actions.github.com/v1beta1) -- `AutoscalingListener` -- `EphemeralRunnerSet` -- `EphemeralRunner` - -**Architecture:** -- Listener-based architecture -- Ephemeral runners (created on-demand) -- More efficient scaling -- Better resource utilization - -**Helm Charts:** -- `gha-runner-scale-set-controller` (controller) -- `gha-runner-scale-set` (runner scale sets) - -**Repository:** -``` -https://github.com/actions/actions-runner-controller -``` - -**Documentation:** -- https://docs.github.com/en/actions/tutorials/use-actions-runner-controller -- https://docs.github.com/en/actions/hosting-your-own-runners/managing-self-hosted-runners-with-actions-runner-controller - -**Pros:** -- ✅ Officially supported by GitHub -- ✅ Better runner group support -- ✅ More efficient scaling (ephemeral runners) -- ✅ Better resource utilization -- ✅ Active development by GitHub -- ✅ Official documentation and support - -**Cons:** -- ⚠️ Different architecture (requires migration) -- ⚠️ Newer (less community examples) -- ⚠️ Different CRDs (not compatible with old version) - -## Key Differences - -| Feature | Community Version | Official Version | -|---------|------------------|-----------------| -| **Maintainer** | Community | GitHub | -| **CRD** | RunnerDeployment | AutoscalingRunnerSet | -| **API Group** | actions.summerwind.dev | actions.github.com | -| **Architecture** | Direct pods | Listener + Ephemeral | -| **Scaling** | Pod-based | Ephemeral runners | -| **Runner Groups** | Limited support | Full support | -| **Documentation** | Community | Official GitHub docs | -| **Support** | Community | GitHub official | - -## Your Current Setup - -**You're using:** Community version (summerwind) - -**Evidence:** -- Controller: `summerwind/actions-runner-controller:v0.27.6` -- CRDs: `runnerdeployments.actions.summerwind.dev` -- Configuration: `RunnerDeployment` CRD - -**Note:** You also have the new CRDs installed (`autoscalingrunnersets.actions.github.com`), but you're not using them yet. - -## Should You Migrate? - -### Keep Community Version If: -- ✅ Current setup works well -- ✅ You don't need advanced features -- ✅ You want to avoid migration effort -- ✅ Community support is sufficient - -### Migrate to Official Version If: -- ✅ You want official GitHub support -- ✅ You need better runner group support -- ✅ You want more efficient scaling -- ✅ You want future-proof solution -- ✅ You need official SLA/support - -## Migration Path - -If you decide to migrate: - -1. **Install Official ARC:** - ```bash - helm repo add actions-runner-controller \ - https://actions-runner-controller.github.io/actions-runner-controller - helm install arc \ - actions-runner-controller/gha-runner-scale-set-controller \ - -n actions-runner-system - ``` - -2. **Create AutoscalingRunnerSet:** - - Different CRD structure - - Uses values.yaml configuration - - See: https://docs.github.com/en/actions/tutorials/use-actions-runner-controller/deploy-runner-scale-sets - -3. **Migrate Configuration:** - - Convert RunnerDeployment → AutoscalingRunnerSet - - Update authentication (GitHub App recommended) - - Test in staging first - -4. **Remove Old Version:** - - Uninstall community ARC - - Remove old CRDs (if desired) - -## References - -### Official GitHub Documentation -- [About Actions Runner Controller](https://docs.github.com/en/actions/hosting-your-own-runners/managing-self-hosted-runners-with-actions-runner-controller/about-actions-runner-controller) -- [Deploy Runner Scale Sets](https://docs.github.com/en/actions/tutorials/use-actions-runner-controller/deploy-runner-scale-sets) -- [Quickstart](https://docs.github.com/en/actions/tutorials/use-actions-runner-controller/quickstart) - -### Community Version -- [GitHub Repository](https://github.com/actions/actions-runner-controller) -- [Community Documentation](https://github.com/actions/actions-runner-controller/blob/master/docs/) - -## Recommendation - -**For now:** Keep using the community version since it's working well. - -**Future consideration:** Plan migration to official version when: -- You need better runner group support -- You want official GitHub support -- You're doing a major infrastructure update -- You need more efficient scaling - -The official version is the future direction, but the community version is still viable for current needs. diff --git a/docs/github-runner/RUNNER_GROUPS.md b/docs/github-runner/RUNNER_GROUPS.md deleted file mode 100644 index 902ce31..0000000 --- a/docs/github-runner/RUNNER_GROUPS.md +++ /dev/null @@ -1,173 +0,0 @@ -# GitHub Actions Runner Groups Configuration - -## Current Setup - -**Repository-Level Runners** (Currently Active) -- ✅ Working and jobs are executing -- ✅ Simple - no additional configuration needed -- ✅ Secure - only `DataKnifeAI/gitops-tools` can use these runners -- ❌ One deployment per repository -- ❌ Less flexible for multiple repositories - -## When to Use Runner Groups - -Use **Organization-Level Runners with Groups** if you: -- Have multiple repositories that need runners -- Want centralized runner management -- Need to control which repos can use which runners -- Want to share runners across repositories - -## Setting Up Organization-Level Runners with Groups - -### Step 1: Create a Runner Group in GitHub - -1. Go to: https://github.com/organizations/DataKnifeAI/settings/actions/runners -2. Click **"New runner group"** -3. Name it (e.g., "kubernetes-runners" or "default") -4. Choose repository access: - - **All repositories** - All repos in org can use - - **Selected repositories** - Choose specific repos -5. Click **"Create group"** - -### Step 2: Configure Repository Access - -1. In the runner group settings, go to **"Repository access"** -2. Add repositories that should have access: - - Click **"Add repository"** - - Select repositories (e.g., `gitops-tools`) - - Click **"Add"** - -### Step 3: Enable Public Repository Access (REQUIRED for Public Repos) - -⚠️ **Security Warning**: Allowing public repos to use self-hosted runners is risky! - -**IMPORTANT**: If your repository is **public**, you MUST enable this flag for workflows to run. - -To enable public repository access: - -1. Go to the runner group settings: https://github.com/organizations/DataKnifeAI/settings/actions/runner-groups -2. Click on your runner group name (e.g., "NRPD Auto Scale") -3. Find the **"Allow public repositories"** checkbox -4. **Enable the checkbox** (this is REQUIRED for public repos) -5. Click **"Save changes"** - -**Understand the risks**: - - Forks of public repos can run code on your runners - - Malicious code could access your infrastructure - - Only enable if you trust all public repos in the group - -**Note**: This setting is REQUIRED if your repository is public. Without it, workflows in public repositories will remain queued and will not be assigned to runners. - -**Note on Forking**: -- **Public repositories** cannot have forking disabled (GitHub requirement) -- **Private repositories** can disable forking to reduce security risk -- If you need to disable forking, make the repository private first - -### Step 4: Update RunnerDeployment - -Update `runnerdeployment.yaml` to use organization-level with runner group: - -```yaml -spec: - template: - spec: - # Organization-level runner - organization: DataKnifeAI - - # Assign to runner group (created in Step 1) - runnerGroup: kubernetes-runners # or "default" - - # Labels (single label only - ARC limitation) - labels: - - self-hosted -``` - -### Step 5: Apply Changes - -```bash -git add github-runner/overlays/nprd-apps/runnerdeployment.yaml -git commit -m "feat: switch to organization-level runners with group" -git push -``` - -Fleet will update the deployment, and new organization-level runners will be created. - -## Comparison - -| Feature | Repository-Level | Organization-Level with Groups | -|---------|-----------------|-------------------------------| -| Setup Complexity | Simple | Requires group configuration | -| Multi-Repo Support | No (one per repo) | Yes (shared across repos) | -| Access Control | Automatic | Manual (via groups) | -| Security | High (isolated) | Medium (shared) | -| Management | Per-repo | Centralized | -| Public Repo Support | Automatic | Requires flag | - -## Current Recommendation - -**Keep Repository-Level** if: -- You only have a few repositories -- Each repo needs isolated runners -- You want the simplest setup - -**Switch to Organization-Level** if: -- You have many repositories -- You want to share runners -- You need centralized management -- You want to control access via groups - -## Security Best Practices - -1. **Runner Groups**: Use groups to restrict access -2. **Public Repos**: Disable public repo access unless necessary -3. **Selected Repos**: Use "Selected repositories" instead of "All repositories" -4. **Monitoring**: Monitor runner activity regularly -5. **Isolation**: Consider separate groups for different security levels -6. **Forking**: Disable forking for private repos to reduce risk - -## Disabling Repository Forking - -### For Private Repositories - -**Repository-Level:** -1. Go to repository **Settings** → **General** -2. Scroll to **Features** section -3. Uncheck **"Allow forking"** -4. Click **Save** - -**Organization-Level:** -1. Go to: https://github.com/organizations/DataKnifeAI/settings/member_privileges -2. Under **"Repository forking"**, choose: - - **Allow forking of private repositories** (enabled) - - **Disallow forking of private repositories** (disabled) -3. Click **Save** - -### For Public Repositories - -⚠️ **Cannot be disabled**: Public repositories cannot have forking disabled. This is a fundamental requirement of making a repository public on GitHub. - -**Options:** -- Make the repository private (then forking can be disabled) -- Use repository-level runners (only your repo can use them, not forks) -- Keep public repo access disabled in runner groups - -### Security Implications - -**With Forking Enabled:** -- Anyone can fork your repository -- Forks can trigger workflows if public repo access is enabled -- Higher security risk for self-hosted runners - -**With Forking Disabled (Private Repos Only):** -- Only repository collaborators can access -- No forks can trigger workflows -- Lower security risk for self-hosted runners - -**Recommendation:** -- For private repos: Disable forking if using self-hosted runners -- For public repos: Use repository-level runners or keep public repo access disabled - -## References - -- [GitHub Docs: Managing access to self-hosted runners using groups](https://docs.github.com/en/actions/hosting-your-own-runners/managing-access-to-self-hosted-runners-using-groups) -- [GitHub Docs: Security hardening for GitHub Actions](https://docs.github.com/en/actions/security/security-hardening-for-github-actions) diff --git a/docs/github-runner/RUNNER_GROUP_ISSUE.md b/docs/github-runner/RUNNER_GROUP_ISSUE.md deleted file mode 100644 index 6f34ae1..0000000 --- a/docs/github-runner/RUNNER_GROUP_ISSUE.md +++ /dev/null @@ -1,92 +0,0 @@ -# Runner Group Assignment Issue - -## Problem - -Runners are going to the "default" group instead of "NRPD Auto Scale" group, even though the `group` field is correctly configured in `RunnerDeployment`. - -## Root Cause - -The old community version of Actions Runner Controller (summerwind/actions-runner-controller) has limitations with runner group assignment: - -1. **Runner group must exist BEFORE runners are created** - If the group doesn't exist when runners register, they default to "default" group -2. **Group field may not be used during registration** - The old ARC version may not properly pass the group during the GitHub registration process -3. **Runners created before group exists** - If runners were created before the "NRPD Auto Scale" group existed, they won't automatically move - -## Current Configuration - -The configuration is correct: -```yaml -spec: - template: - spec: - organization: DataKnifeAI - group: NRPD Auto Scale # ✅ Field is set correctly -``` - -## Solutions - -### Option 1: Ensure Group Exists First (Recommended) - -1. **Create the runner group in GitHub UI first:** - - Go to: https://github.com/organizations/DataKnifeAI/settings/actions/runners - - Click "New runner group" - - Name: "NRPD Auto Scale" - - Repository access: "All repositories" - - Create the group - -2. **Delete and recreate runners:** - ```bash - kubectl --context=nprd-apps delete runner -n managed-cicd --all - ``` - Runners will automatically recreate and should register with the correct group. - -### Option 2: Manually Move Runners in GitHub UI - -1. Go to: https://github.com/organizations/DataKnifeAI/settings/actions/runners -2. Click on "default" group -3. Select runners you want to move -4. Click "Move to group" → Select "NRPD Auto Scale" -5. Confirm the move - -Runners will stay in the correct group after manual assignment. - -### Option 3: Upgrade to New GitHub-Supported ARC - -The new GitHub-supported version (gha-runner-scale-set) has better support for runner groups. However, this requires: - -- Different CRDs (AutoscalingRunnerSet instead of RunnerDeployment) -- Different architecture (Listener-based) -- Migration from old to new version -- See: https://docs.github.com/en/actions/tutorials/use-actions-runner-controller/deploy-runner-scale-sets - -## Verification - -After applying a solution, verify: - -1. **Check runners in Kubernetes:** - ```bash - kubectl --context=nprd-apps get runner -n managed-cicd -o json | jq '.items[].spec.group' - ``` - Should show: `"NRPD Auto Scale"` - -2. **Check in GitHub UI:** - - Go to: https://github.com/organizations/DataKnifeAI/settings/actions/runners - - Click on "NRPD Auto Scale" group - - Verify runners appear there (not in "default") - -3. **Test workflow:** - - Trigger a workflow in any repository - - Verify it uses runners from "NRPD Auto Scale" group - -## Notes - -- The old community ARC version (summerwind) has known limitations with runner groups -- The `group` field exists and is applied, but may not be used during registration -- Manual assignment in GitHub UI is the most reliable method for the old version -- The new GitHub-supported ARC has better runner group support - -## References - -- [GitHub Docs: Runner Scale Sets](https://docs.github.com/en/actions/tutorials/use-actions-runner-controller/deploy-runner-scale-sets) -- [Community ARC: RunnerDeployment](https://github.com/actions/actions-runner-controller) -- [GitHub Docs: Managing Runner Groups](https://docs.github.com/en/actions/hosting-your-own-runners/managing-access-to-self-hosted-runners-using-groups) diff --git a/docs/github-runner/SETUP_ORG_RUNNERS.md b/docs/github-runner/SETUP_ORG_RUNNERS.md deleted file mode 100644 index 6e89270..0000000 --- a/docs/github-runner/SETUP_ORG_RUNNERS.md +++ /dev/null @@ -1,126 +0,0 @@ -# Setting Up Organization-Level Runners - -## Quick Setup Guide - -This guide walks you through setting up organization-level runners with a runner group that has access to all repositories. - -## Step 1: Create Runner Group in GitHub - -1. Navigate to organization settings: - ``` - https://github.com/organizations/DataKnifeAI/settings/actions/runners - ``` - -2. Click **"New runner group"** button - -3. Configure the group: - - **Name**: `NRPD Auto Scale` (must match `runnerGroup` in `runnerdeployment.yaml`) - - **Repository access**: Select **"All repositories"** - - Click **"Create group"** - -4. **REQUIRED for Public Repos**: Enable public repository access: - - In the group settings, find **"Allow public repositories"** - - **Enable the checkbox** (REQUIRED if your repository is public) - - ⚠️ **Security Warning**: Understand the risks before enabling - - Forks of public repos can run code on your runners if enabled - - **Without this flag enabled, workflows in public repos will not run** - -## Step 2: Verify Configuration - -The runner group should: -- ✅ Be named `NRPD Auto Scale` -- ✅ Have access to "All repositories" -- ✅ Be visible in the runner groups list - -## Step 3: Apply Configuration - -Once the runner group is created in GitHub: - -```bash -# The configuration is already updated in runnerdeployment.yaml -# Just commit and push: -git add github-runner/overlays/nprd-apps/runnerdeployment.yaml -git commit -m "feat: switch to organization-level runners with group" -git push -``` - -Fleet will automatically: -1. Remove old repository-level runners -2. Create new organization-level runners -3. Register them with the `kubernetes-runners` group -4. Make them available to all repositories in the organization - -## Step 4: Verify Runners - -After Fleet updates (1-2 minutes): - -1. Check runners in Kubernetes: - ```bash - kubectl --context=nprd-apps get runner -n managed-cicd - ``` - -2. Check runners in GitHub: - - Go to: https://github.com/organizations/DataKnifeAI/settings/actions/runners - - Click on the `NRPD Auto Scale` group - - Verify runners appear and are "Online" - -3. Test with a workflow: - - Trigger a workflow in any repository - - Verify it uses the organization-level runners - -## Troubleshooting - -### Runners Not Appearing - -- **Check runner group exists**: Verify `NRPD Auto Scale` group exists in GitHub -- **Check group name matches**: Must exactly match `runnerGroup` in YAML -- **Check Fleet status**: `kubectl --context=nprd-apps get gitrepo -A` - -### Jobs Not Starting - -- **Check repository access**: Verify group has "All repositories" access -- **Check runner labels**: Ensure workflow uses `runs-on: self-hosted` -- **Check runner status**: Runners should be "Online" in GitHub - -### Public Repo Access Issues - -- **If public repos can't use runners**: Enable "Allow public repositories" in group settings (REQUIRED) -- **Jobs stuck in queued**: This is often because "Allow public repositories" is not enabled -- **Security risk**: Understand that forks can trigger workflows if enabled -- **How to enable**: - 1. Go to: https://github.com/organizations/DataKnifeAI/settings/actions/runner-groups - 2. Click on your runner group (e.g., "NRPD Auto Scale") - 3. Check **"Allow public repositories"** checkbox - 4. Click **"Save changes"** - -## Rollback - -If you need to rollback to repository-level runners: - -1. Update `runnerdeployment.yaml`: - ```yaml - repository: DataKnifeAI/gitops-tools - # Comment out organization and runnerGroup - ``` - -2. Commit and push: - ```bash - git add github-runner/overlays/nprd-apps/runnerdeployment.yaml - git commit -m "revert: switch back to repository-level runners" - git push - ``` - -## Benefits of Organization-Level Runners - -- ✅ **Centralized Management**: One deployment serves all repos -- ✅ **Resource Efficiency**: Shared runners across repositories -- ✅ **Flexibility**: Easy to add/remove repositories from group -- ✅ **Scalability**: Better for organizations with many repos - -## Security Considerations - -1. **Runner Groups**: Use groups to control access -2. **Repository Access**: Start with "Selected repositories" if unsure -3. **Public Repos**: Disable public repo access unless necessary -4. **Monitoring**: Monitor runner activity regularly -5. **Isolation**: Consider separate groups for different security levels diff --git a/docs/github-runner/TERRAFORM_INTEGRATION.md b/docs/github-runner/TERRAFORM_INTEGRATION.md deleted file mode 100644 index d72f486..0000000 --- a/docs/github-runner/TERRAFORM_INTEGRATION.md +++ /dev/null @@ -1,182 +0,0 @@ -# Terraform Integration for GitHub Runner Controller - -## Overview - -This repository manages **only runner definitions** (AutoscalingRunnerSet), not the controller itself. The GitHub Actions Runner Controller (`gha-runner-scale-set-controller`) is managed by a separate Terraform project. - -## Responsibilities - -### This Repository (GitOps) -- ✅ Manages runner definitions (`AutoscalingRunnerSet`) -- ✅ Configures runner scale sets (min/max runners, labels, resources) -- ✅ Defines runner groups and organization/repository associations -- ✅ Manages runner pod templates and resources - -### Terraform Project -- ✅ Manages controller deployment (`gha-runner-scale-set-controller`) -- ✅ Creates and manages authentication secrets -- ✅ Manages controller RBAC and service accounts -- ✅ Handles controller configuration and updates - -## Secret Configuration - -The `AutoscalingRunnerSet` requires a `githubConfigSecret` field that references a Kubernetes secret containing GitHub authentication credentials. - -### Secret Name - -The secret name must match what the Terraform project creates. Common names: -- `github-app-secret` (for GitHub App authentication) -- `github-token-secret` (for PAT authentication) -- Custom name configured in Terraform - -**Action Required**: Verify the secret name in your Terraform project and update `autoscalingrunnerset.yaml` if different: - -```yaml -spec: - githubConfigSecret: -``` - -### Secret Location - -The secret must exist in the same namespace as the AutoscalingRunnerSet: -- Namespace: `managed-cicd` -- Secret: Must exist before AutoscalingRunnerSet is deployed - -### Secret Contents - -**For GitHub App (Recommended):** -```yaml -apiVersion: v1 -kind: Secret -metadata: - name: github-app-secret - namespace: managed-cicd -type: Opaque -data: - github_app_id: - github_app_installation_id: - github_app_private_key: -``` - -**For Personal Access Token:** -```yaml -apiVersion: v1 -kind: Secret -metadata: - name: github-token-secret - namespace: managed-cicd -type: Opaque -data: - github_token: -``` - -## Current Configuration - -**AutoscalingRunnerSet** (`github-runner/overlays/nprd-apps/autoscalingrunnerset.yaml`): -- `githubConfigSecret: github-app-secret` (TODO: Verify with Terraform) -- `githubConfigUrl: https://github.com/DataKnifeAI` -- `runnerGroup: NRPD Auto Scale` -- `minRunners: 2` -- `maxRunners: 10` - -## Verification - -### Check if Secret Exists - -```bash -# List secrets in managed-cicd namespace -kubectl --context=nprd-apps get secret -n managed-cicd | grep -i github - -# Check specific secret (if you know the name) -kubectl --context=nprd-apps get secret github-app-secret -n managed-cicd -``` - -### Verify Secret Name in AutoscalingRunnerSet - -```bash -# Check current configuration -kubectl --context=nprd-apps get autoscalingrunnerset github-runner-scale-set -n managed-cicd -o yaml | grep githubConfigSecret -``` - -### Verify Secret is Used Correctly - -```bash -# Check AutoscalingRunnerSet status -kubectl --context=nprd-apps describe autoscalingrunnerset github-runner-scale-set -n managed-cicd - -# Check for authentication errors in listener logs -kubectl --context=nprd-apps logs -n managed-cicd -l app.kubernetes.io/name=gha-rs-listener | grep -i "auth\|secret\|error" -``` - -## Troubleshooting - -### Secret Not Found - -If AutoscalingRunnerSet fails with "secret not found" error: - -1. **Verify secret exists:** - ```bash - kubectl --context=nprd-apps get secret -n managed-cicd - ``` - -2. **Check Terraform output** for secret name: - ```bash - # In Terraform project - terraform output -json | jq '.github_secret_name' - ``` - -3. **Update AutoscalingRunnerSet** to use correct secret name: - ```yaml - spec: - githubConfigSecret: - ``` - -### Authentication Errors - -If runners fail to authenticate: - -1. **Verify secret contents** (if you have access): - ```bash - kubectl --context=nprd-apps get secret github-app-secret -n managed-cicd -o yaml - ``` - -2. **Check listener logs** for authentication errors: - ```bash - kubectl --context=nprd-apps logs -n managed-cicd -l app.kubernetes.io/name=gha-rs-listener --tail=100 - ``` - -3. **Contact Terraform project maintainers** to verify: - - Secret name is correct - - Secret contains required fields - - Secret is in the correct namespace - -## Integration Points - -### Terraform Outputs - -The Terraform project should provide outputs that this repository can reference: -- Secret name -- Namespace -- Controller version/deployment info - -### GitOps Workflow - -1. **Terraform deploys controller and secrets** -2. **This repository deploys AutoscalingRunnerSet** (via Fleet) -3. **Controller watches AutoscalingRunnerSet** and creates listeners/runners -4. **Runners register with GitHub** using credentials from secret - -## Best Practices - -1. **Secret Management**: Secrets should be created by Terraform, not this repository -2. **Namespace Consistency**: Ensure secrets are in the same namespace as AutoscalingRunnerSet -3. **Secret Naming**: Use consistent naming between Terraform and GitOps -4. **Documentation**: Document secret names and locations in both projects -5. **Validation**: Verify secret exists before deploying AutoscalingRunnerSet - -## References - -- [Official ARC Documentation](https://docs.github.com/en/actions/tutorials/use-actions-runner-controller) -- [Authentication Guide](https://docs.github.com/en/actions/tutorials/use-actions-runner-controller/authenticate-to-the-api) -- [AutoscalingRunnerSet Spec](https://docs.github.com/en/actions/tutorials/use-actions-runner-controller/deploy-runner-scale-sets) - diff --git a/fleet-gitrepo.yaml b/fleet-gitrepo.yaml index 0b56e6a..8272844 100644 --- a/fleet-gitrepo.yaml +++ b/fleet-gitrepo.yaml @@ -7,7 +7,6 @@ spec: repo: https://github.com/DataKnifeAI/gitops-tools branch: main paths: - # github-runner/overlays/nprd-apps # Removed - ARC AutoscalingRunnerSet blocking; add back when ARC installed - gitlab-runner/overlays/nprd-apps - harbor/overlays/nprd-apps - grafana/overlays/nprd-apps diff --git a/github-runner/base/README.md b/github-runner/base/README.md deleted file mode 100644 index b4d6c88..0000000 --- a/github-runner/base/README.md +++ /dev/null @@ -1,130 +0,0 @@ -# GitHub Actions Runner - Runner Definitions - -This directory contains runner definitions for GitHub Actions Runner Controller using the **official GitHub-supported ARC**. - -## Architecture - -**IMPORTANT**: The GitHub Actions Runner Controller (`gha-runner-scale-set-controller`) is managed by a separate Terraform project. This repository only manages runner definitions. - -- **Controller**: `gha-runner-scale-set-controller` (managed by Terraform - not in this repo) -- **Runner Definitions**: `AutoscalingRunnerSet` (actions.github.com/v1beta1) - managed here -- **Architecture**: Listener-based with ephemeral runners -- **Repository**: https://github.com/actions/actions-runner-controller - -## Documentation - -For complete documentation, see: -- [GitHub Runner Overview](../../docs/GITHUB_RUNNER.md) -- [Official ARC Documentation](../../docs/github-runner/OFFICIAL_ARC.md) -- [Migration Guide](../../docs/github-runner/MIGRATION_TO_OFFICIAL_ARC.md) -- [GitHub Official Docs](https://docs.github.com/en/actions/tutorials/use-actions-runner-controller) - -## Quick Start - -### Prerequisites - -**The controller (`gha-runner-scale-set-controller`) must already be deployed.** The controller is managed by a separate Terraform project, not this repository. This repository only manages runner definitions. - -### 1. Set Up Authentication - -**NOTE**: Authentication secrets should already exist if the controller was deployed via Terraform. If not, you may need to create them manually. - -Official ARC requires authentication. Two options: - -**Option A: GitHub App (Recommended)** - -```bash -# Create GitHub App in organization settings -# Install on organization -# Get App ID, Installation ID, and private key - -kubectl create secret generic github-app-secret \ - --from-literal=github_app_id='' \ - --from-literal=github_app_installation_id='' \ - --from-literal=github_app_private_key='' \ - -n managed-cicd -``` - -**Option B: Personal Access Token (PAT)** - -```bash -# Create PAT with admin:org scope -kubectl create secret generic github-pat-secret \ - --from-literal=github_token='' \ - -n managed-cicd -``` - -### 2. Deploy Runner Definitions - -Use the overlay configuration in `overlays/nprd-apps/` for cluster-specific settings. - -The overlay contains: -- `autoscalingrunnerset.yaml` - AutoscalingRunnerSet CRD resource (runner definition) - -Fleet will deploy this to the target cluster when the GitRepo syncs. - -## Configuration - -### Runner Definitions - -This repository manages runner definitions using `AutoscalingRunnerSet` CRD resources. The runner definitions specify: -- Organization/Repository configuration -- Runner group assignment -- Scaling parameters (min/max runners) -- Labels -- Resource requirements -- Runner pod templates - -The controller (managed by Terraform) watches for these definitions and creates the necessary listeners and ephemeral runners. - -## Key Features - -- ✅ **Official GitHub Support**: Maintained by GitHub -- ✅ **Ephemeral Runners**: Automatically clean up after jobs -- ✅ **Built-in Scaling**: No separate autoscaler needed -- ✅ **Multiple Labels**: Supports multiple runner labels -- ✅ **Runner Groups**: Full support for runner groups -- ✅ **Efficient Resource Usage**: Better than community version - -## Troubleshooting - -### Check Controller Status - -**NOTE**: Controller is managed by Terraform. Check with the Terraform project maintainers if issues occur. - -```bash -kubectl get deployment -n actions-runner-system gha-runner-scale-set-controller-gha-rs-controller -``` - -### Check AutoscalingRunnerSet - -```bash -kubectl get autoscalingrunnerset -n managed-cicd -kubectl describe autoscalingrunnerset -n managed-cicd -``` - -### Check Listeners - -```bash -kubectl get autoscalinglistener -n managed-cicd -kubectl logs -n managed-cicd -l app.kubernetes.io/name=gha-rs-listener -``` - -### Check Ephemeral Runners - -```bash -kubectl get ephemeralrunner -n managed-cicd -``` - -### Controller Logs - -```bash -kubectl logs -n actions-runner-system -l app.kubernetes.io/name=gha-rs-controller -``` - -## References - -- [Official ARC Documentation](https://docs.github.com/en/actions/tutorials/use-actions-runner-controller) -- [Deploy Runner Scale Sets](https://docs.github.com/en/actions/tutorials/use-actions-runner-controller/deploy-runner-scale-sets) -- [Quickstart Guide](https://docs.github.com/en/actions/tutorials/quickstart-for-actions-runner-controller) -- [Authentication Guide](https://docs.github.com/en/actions/tutorials/use-actions-runner-controller/authenticate-to-the-api) diff --git a/github-runner/base/fleet.yaml b/github-runner/base/fleet.yaml deleted file mode 100644 index 42189a5..0000000 --- a/github-runner/base/fleet.yaml +++ /dev/null @@ -1,22 +0,0 @@ -# Fleet configuration for GitHub Runner base -# This file configures the base GitHub Runner definitions bundle -# -# NOTE: The controller is managed by Terraform, not this repository. -# This bundle contains base configuration for runner definitions only. -# -# NOTE: This base bundle should typically NOT be deployed directly. -# It's meant to be used by overlays (e.g., github-runner/overlays/nprd-apps). -# -# If your GitRepo monitors github-runner/base directly: -# - This bundle will be deployed to ALL clusters (unless you add cluster targeting) -# - Consider monitoring github-runner/overlays/nprd-apps instead -# -# If your GitRepo monitors the root: -# - Fleet will create separate bundles for base and overlays -# - Overlays will use their own fleet.yaml for cluster targeting - -defaultNamespace: managed-cicd - -# Uncomment below to prevent this base bundle from being deployed -# (only allow overlays to use it) -# targetCustomizations: [] diff --git a/github-runner/base/kustomization.yaml b/github-runner/base/kustomization.yaml deleted file mode 100644 index 2566c87..0000000 --- a/github-runner/base/kustomization.yaml +++ /dev/null @@ -1,18 +0,0 @@ -apiVersion: kustomize.config.k8s.io/v1beta1 -kind: Kustomization - -namespace: managed-cicd - -# Base configuration for GitHub Actions Runner Controller -# This is a base template - use overlays for cluster-specific configurations -# -# Official GitHub-supported ARC uses AutoscalingRunnerSet -# See overlays/nprd-apps for cluster-specific configurations - -resources: - # Base Fleet configuration - - fleet.yaml - -commonLabels: - app: github-runner - managed-by: gitops diff --git a/github-runner/overlays/nprd-apps/autoscalingrunnerset.yaml b/github-runner/overlays/nprd-apps/autoscalingrunnerset.yaml deleted file mode 100644 index add6f30..0000000 --- a/github-runner/overlays/nprd-apps/autoscalingrunnerset.yaml +++ /dev/null @@ -1,82 +0,0 @@ -apiVersion: actions.github.com/v1alpha1 -kind: AutoscalingRunnerSet -metadata: - name: nprd-autoscale-runners - namespace: managed-cicd - labels: - # Version label (as per official documentation) - # Documentation mentions app.kubernetes.io/version= for pods - # Trying same format for AutoscalingRunnerSet - # Downgraded to 0.12.1 to avoid JIT config double-encoding bug in 0.13.0/0.13.1 - app.kubernetes.io/version: "0.12.1" - annotations: - # NOTE: Official troubleshooting docs don't mention version annotations - # Version may be managed via Helm chart, not manual annotations - # Keeping minimal annotations for now -spec: - # Organization-level runner scale set - # Available to all repositories in the organization - # Runners are assigned to a runner group for access control - githubConfigUrl: https://github.com/DataKnifeAI - - # Authentication secret (REQUIRED) - # Secret created by Terraform with GitHub App credentials - githubConfigSecret: github-app-secret - - # Runner group name (must exist in GitHub org settings) - # Create this group in: https://github.com/organizations/DataKnifeAI/settings/actions/runners - # Configure it to allow "All repositories" or select specific repos - # IMPORTANT: For PUBLIC repositories, you MUST enable "Allow public repositories" flag - # in the runner group settings, otherwise workflows will remain queued! - # Go to: https://github.com/organizations/DataKnifeAI/settings/actions/runner-groups - # Click on your group → Enable "Allow public repositories" → Save - # NOTE: Runner group must exist BEFORE creating this resource - runnerGroup: NRPD Auto Scale - - # Runner scale set name (used in GitHub) - # IMPORTANT: ARC runner scale sets expose ONLY this name as a label - # Per official docs: https://docs.github.com/en/actions/tutorials/use-actions-runner-controller/use-arc-in-a-workflow - # "You cannot use additional labels to target runners created by ARC. - # You can only use the installation name...or runnerScaleSetName field. - # These are used as the 'single label' to use as your runs-on target." - # Workflows must use: runs-on: - # - # CRITICAL: Known bug with "self-hosted" label (GitHub issue #3330) - # ARC runners may not automatically receive "self-hosted" label - # Using "kubernetes" instead to avoid this known issue - # Workflow updated to use: runs-on: kubernetes - runnerScaleSetName: kubernetes - - # Maximum number of runners that can be created - maxRunners: 0 - - # Minimum number of runners to maintain - # Set to 0 for true ephemeral behavior - runners are created on-demand and cleaned up after jobs - minRunners: 0 - - # Container spec for the runner pods - template: - spec: - # CRITICAL FIX: Set restartPolicy to Never for ephemeral runners - # With Always (default), Kubernetes restarts completed pods, causing endless loop - # Ephemeral runners should exit cleanly and NOT restart - # This is the KEY missing configuration causing the endless loop! - restartPolicy: Never - - containers: - - name: runner - image: ghcr.io/actions/actions-runner:latest - resources: - # Increased requests to prevent premature exits - # Low resource requests can cause pods to exit prematurely - requests: - cpu: 500m # Increased from 100m to prevent resource starvation - memory: 512Mi # Increased from 128Mi to prevent OOM kills - limits: - cpu: "2" # Use normalized format to match Kubernetes (2000m = 2) - memory: 4Gi - - # Official ARC uses ephemeral runners by default - # Ephemeral runners automatically clean up after job completion - # This is built-in behavior - there's no ephemeral field to configure - # With minRunners: 0 and maxRunners: 0, runners are created on-demand and scale to zero when idle diff --git a/github-runner/overlays/nprd-apps/fleet.yaml b/github-runner/overlays/nprd-apps/fleet.yaml deleted file mode 100644 index 6b060d7..0000000 --- a/github-runner/overlays/nprd-apps/fleet.yaml +++ /dev/null @@ -1,24 +0,0 @@ -# Fleet configuration for nprd-apps cluster -# This file configures how Fleet deploys GitHub Runner definitions to the nprd-apps cluster -# -# NOTE: The controller (gha-runner-scale-set-controller) is managed by Terraform. -# This repository only manages runner definitions (AutoscalingRunnerSet). -# -# Fleet will use this configuration when the GitRepo monitors: -# - The root directory (creates bundles for each directory) -# - Or specifically the github-runner/overlays/nprd-apps path -# -# If your GitRepo monitors github-runner/base, it won't see this overlay. -# In that case, either: -# 1. Change GitRepo to monitor github-runner/overlays/nprd-apps -# 2. Or change GitRepo to monitor root and use fleet.yaml for targeting - -defaultNamespace: managed-cicd - -# Target this bundle to the nprd-apps cluster -targetCustomizations: - - name: nprd-apps - clusterSelector: - matchLabels: - # Target nprd-apps cluster by display name - management.cattle.io/cluster-display-name: nprd-apps diff --git a/github-runner/overlays/nprd-apps/kustomization.yaml b/github-runner/overlays/nprd-apps/kustomization.yaml deleted file mode 100644 index e792bde..0000000 --- a/github-runner/overlays/nprd-apps/kustomization.yaml +++ /dev/null @@ -1,24 +0,0 @@ -apiVersion: kustomize.config.k8s.io/v1beta1 -kind: Kustomization - -namespace: managed-cicd - -resources: - # Runner Scale Set Definition - # NOTE: The controller (gha-runner-scale-set-controller) is managed by Terraform - # This repository only manages runner definitions (AutoscalingRunnerSet) - - autoscalingrunnerset.yaml - -# Cluster-specific overrides for nprd-apps -# Add any nprd-apps specific configurations here -# Examples: -# - resource limits/requests -# - storage class -# - environment-specific values - -patches: - # Add patches here for nprd-apps specific configurations - # - path: resource-limits.yaml - # target: - # kind: AutoscalingRunnerSet - # name: github-runner-scale-set diff --git a/scripts/runner-setup.sh b/scripts/runner-setup.sh index d1a7d58..0f464f4 100755 --- a/scripts/runner-setup.sh +++ b/scripts/runner-setup.sh @@ -1,10 +1,10 @@ #!/bin/bash # Runner Setup Script -# This script creates secrets for GitHub and GitLab runners +# This script creates the GitLab runner secret # # Usage: -# ./scripts/runner-setup.sh [github|gitlab|all] -# GITHUB_TOKEN= GITLAB_TOKEN= GITLAB_URL= ./scripts/runner-setup.sh all +# ./scripts/runner-setup.sh [gitlab] +# GITLAB_TOKEN= GITLAB_URL= ./scripts/runner-setup.sh gitlab set -e @@ -15,10 +15,10 @@ RED='\033[0;31m' BLUE='\033[0;34m' NC='\033[0m' -ACTION="${1:-all}" +ACTION="${1:-gitlab}" echo -e "${BLUE}========================================${NC}" -echo -e "${BLUE}GitHub & GitLab Runner Setup${NC}" +echo -e "${BLUE}GitLab Runner Setup${NC}" echo -e "${BLUE}========================================${NC}" echo "" @@ -28,54 +28,12 @@ if ! command -v kubectl &> /dev/null; then exit 1 fi -# Create namespaces -echo -e "${YELLOW}Creating namespaces...${NC}" +# Create namespace +echo -e "${YELLOW}Creating namespace...${NC}" kubectl create namespace managed-cicd --dry-run=client -o yaml | kubectl apply -f - > /dev/null -kubectl create namespace actions-runner-system --dry-run=client -o yaml | kubectl apply -f - > /dev/null -echo -e "${GREEN}✓ Namespaces created${NC}" +echo -e "${GREEN}✓ Namespace created${NC}" echo "" -# Function: Create GitHub secret -create_github_secret() { - echo -e "${YELLOW}=== GitHub Runner Setup ===${NC}" - - # Get token from env or prompt - if [ -z "$GITHUB_TOKEN" ]; then - echo "For organization-level runners, you need:" - echo " - GitHub Personal Access Token (PAT) with 'repo' scope, OR" - echo " - GitHub App credentials" - echo "" - read -p "Do you have a GitHub PAT? (y/n): " -n 1 -r - echo - - if [[ $REPLY =~ ^[Yy]$ ]]; then - read -sp "Enter GitHub Personal Access Token: " GITHUB_TOKEN - echo "" - - if [ -z "$GITHUB_TOKEN" ]; then - echo -e "${RED}Error: Token cannot be empty${NC}" - exit 1 - fi - else - echo -e "${YELLOW}Skipping GitHub secret creation${NC}" - return 0 - fi - fi - - # Check if secret exists - if kubectl get secret actions-runner-controller -n actions-runner-system &>/dev/null; then - echo -e "${YELLOW}Secret already exists. Deleting...${NC}" - kubectl delete secret actions-runner-controller -n actions-runner-system - fi - - kubectl create secret generic actions-runner-controller \ - --from-literal=github_token="$GITHUB_TOKEN" \ - -n actions-runner-system > /dev/null - - echo -e "${GREEN}✓ GitHub secret created${NC}" - echo "" -} - # Function: Create GitLab secret create_gitlab_secret() { echo -e "${YELLOW}=== GitLab Runner Setup ===${NC}" @@ -121,37 +79,25 @@ create_gitlab_secret() { # Main execution case "$ACTION" in - github) - create_github_secret - ;; - gitlab) - create_gitlab_secret - ;; - all) - create_github_secret + gitlab|all) create_gitlab_secret echo -e "${GREEN}========================================${NC}" echo -e "${GREEN}Setup Complete!${NC}" echo -e "${GREEN}========================================${NC}" echo "" echo "Next steps:" - echo "1. Update github-runner/base/runnerdeployment.yaml with your GitHub organization" - echo "2. Update gitlab-runner/base/gitlab-runner-helmchart.yaml with GitLab URL: ${GITLAB_URL:-}" - echo "3. Commit and push changes" + echo "1. Update gitlab-runner/base/gitlab-runner-helmchart.yaml with GitLab URL: ${GITLAB_URL:-}" + echo "2. Commit and push changes" echo "" - echo "To verify secrets:" - echo " kubectl get secret actions-runner-controller -n actions-runner-system" + echo "To verify the secret:" echo " kubectl get secret gitlab-runner-secret -n managed-cicd" ;; *) - echo "Usage: $0 [github|gitlab|all]" + echo "Usage: $0 [gitlab]" echo "" - echo " github - Create GitHub runner secret only" - echo " gitlab - Create GitLab runner secret only" - echo " all - Create both secrets (default)" + echo " gitlab - Create GitLab runner secret (default; 'all' is an alias)" echo "" echo "Environment variables:" - echo " GITHUB_TOKEN - GitHub Personal Access Token" echo " GITLAB_TOKEN - GitLab runner authentication token (glrt-*)" echo " GITLAB_URL - GitLab instance URL" exit 1 diff --git a/secrets/github-runner/README.md b/secrets/github-runner/README.md deleted file mode 100644 index c2f1e40..0000000 --- a/secrets/github-runner/README.md +++ /dev/null @@ -1,48 +0,0 @@ -# GitHub Runner Secrets - -This directory contains example secret configurations for GitHub Actions Runner Controller. - -## Authentication Methods - -The GitHub Actions Runner Controller supports two authentication methods: - -### Method 1: Personal Access Token (PAT) - -1. Create a GitHub Personal Access Token: - - Go to GitHub → Settings → Developer settings → Personal access tokens → Tokens (classic) - - Generate a new token with `repo` scope - - Copy the token - -2. Create the secret: - ```bash - kubectl create namespace actions-runner-system - kubectl create secret generic actions-runner-controller \ - --from-literal=github_token='' \ - -n actions-runner-system - ``` - -### Method 2: GitHub App (Recommended for Organizations) - -1. Create a GitHub App: - - Go to your organization → Settings → Developer settings → GitHub Apps - - Create a new app with appropriate permissions - - Generate a private key - - Install the app on your organization/repositories - -2. Create the secret: - ```bash - kubectl create namespace actions-runner-system - kubectl create secret generic actions-runner-controller \ - --from-literal=github_app_id='' \ - --from-literal=github_app_installation_id='' \ - --from-literal=github_app_private_key='' \ - -n actions-runner-system - ``` - -## Security Notes - -- **Never commit actual tokens or keys to git** -- Secrets are encrypted at rest in Kubernetes -- Use RBAC to restrict access to secrets -- Rotate tokens/keys regularly -- Use GitHub Apps for better security and auditability in organizations diff --git a/secrets/github-runner/github-token-secret.yaml.example b/secrets/github-runner/github-token-secret.yaml.example deleted file mode 100644 index 35ca27d..0000000 --- a/secrets/github-runner/github-token-secret.yaml.example +++ /dev/null @@ -1,32 +0,0 @@ -# Example secret for GitHub Actions Runner Controller authentication -# -# This file is for reference only. Create the actual secret using kubectl: -# -# kubectl create secret generic actions-runner-controller \ -# --from-literal=github_token='' \ -# -n actions-runner-system -# -# Or for GitHub App: -# kubectl create secret generic actions-runner-controller \ -# --from-literal=github_app_id='' \ -# --from-literal=github_app_installation_id='' \ -# --from-literal=github_app_private_key='' \ -# -n actions-runner-system - -apiVersion: v1 -kind: Secret -metadata: - name: actions-runner-controller - namespace: actions-runner-system -type: Opaque -stringData: - # GitHub Personal Access Token (PAT) method - github_token: "" - - # OR GitHub App method (use one or the other, not both) - # github_app_id: "" - # github_app_installation_id: "" - # github_app_private_key: | - # -----BEGIN RSA PRIVATE KEY----- - # - # -----END RSA PRIVATE KEY-----