Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
734 changes: 734 additions & 0 deletions bugs.md

Large diffs are not rendered by default.

22 changes: 22 additions & 0 deletions charts/gpu-dev-server/Chart.yaml
Original file line number Diff line number Diff line change
@@ -0,0 +1,22 @@
apiVersion: v2
name: gpu-dev-server
description: GPU Development Server Infrastructure - On-demand GPU development environments
type: application
version: 0.1.0
appVersion: "1.0.0"
keywords:
- gpu
- kubernetes
- development
- pytorch
- cuda
maintainers:
- name: GPU Dev Team
home: https://github.com/your-org/osdc
sources:
- https://github.com/your-org/osdc
dependencies:
- name: nvidia-device-plugin
version: "0.14.0"
repository: https://nvidia.github.io/k8s-device-plugin
condition: nvidia.devicePlugin.enabled
232 changes: 232 additions & 0 deletions charts/gpu-dev-server/README.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,232 @@
# GPU Dev Server Helm Chart

Helm chart for deploying GPU Development Server infrastructure on Kubernetes.

## Overview

This chart deploys the following components:

- **PostgreSQL** with PGMQ extension (primary + replica)
- **API Service** - REST API for job submission with IAM auth
- **Reservation Processor** - Polls PGMQ and manages GPU pod lifecycle
- **Availability Updater** - CronJob that tracks GPU availability
- **Reservation Expiry** - CronJob that handles reservation cleanup
- **Registry Caches** - Pull-through caches for ghcr.io and Docker Hub

## Prerequisites

- Kubernetes 1.24+
- Helm 3.8+
- A storage class (gp3 for AWS, pd-ssd for GCP)
- For AWS: EKS cluster with IAM roles configured
- For GCP: GKE cluster with Workload Identity configured

## Installation

### AWS (EKS)

```bash
# Create a values file with your AWS-specific settings
cat > my-values.yaml <<EOF
cloudProvider:
name: "aws"
region: "us-east-2"
aws:
eksClusterName: "my-gpu-cluster"
primaryAvailabilityZone: "us-east-2a"

postgres:
auth:
password: "your-secure-password" # Or use existingSecret

apiService:
image:
repository: "your-registry/api-service"
serviceAccount:
annotations:
eks.amazonaws.com/role-arn: "arn:aws:iam::ACCOUNT:role/api-service-role"

reservationProcessor:
image:
repository: "your-registry/reservation-processor"
serviceAccount:
annotations:
eks.amazonaws.com/role-arn: "arn:aws:iam::ACCOUNT:role/reservation-processor-role"

availabilityUpdater:
image:
repository: "your-registry/availability-updater"
serviceAccount:
annotations:
eks.amazonaws.com/role-arn: "arn:aws:iam::ACCOUNT:role/availability-updater-role"

reservationExpiry:
image:
repository: "your-registry/reservation-expiry"
serviceAccount:
annotations:
eks.amazonaws.com/role-arn: "arn:aws:iam::ACCOUNT:role/reservation-expiry-role"
EOF

# Install the chart
helm install gpu-dev ./charts/gpu-dev-server \
-f charts/gpu-dev-server/values-aws.yaml \
-f my-values.yaml
```

### GCP (GKE)

```bash
# Create a values file with your GCP-specific settings
cat > my-values.yaml <<EOF
cloudProvider:
name: "gcp"
region: "us-central1"
gcp:
gkeClusterName: "my-gpu-cluster"
zone: "us-central1-a"
projectId: "my-project"

postgres:
auth:
password: "your-secure-password"

apiService:
image:
repository: "gcr.io/my-project/api-service"
serviceAccount:
annotations:
iam.gke.io/gcp-service-account: "api-service@my-project.iam.gserviceaccount.com"

# ... similar for other services
EOF

# Install the chart
helm install gpu-dev ./charts/gpu-dev-server \
-f charts/gpu-dev-server/values-gcp.yaml \
-f my-values.yaml
```

## Configuration

See [values.yaml](values.yaml) for the full list of configurable parameters.

### Key Parameters

| Parameter | Description | Default |
|-----------|-------------|---------|
| `global.prefix` | Prefix for resource naming | `gpu-dev` |
| `namespaces.controlplane` | Controlplane namespace | `gpu-controlplane` |
| `namespaces.workloads` | GPU workloads namespace | `gpu-dev` |
| `storage.class` | Storage class name | `gp3` |
| `postgres.enabled` | Enable PostgreSQL | `true` |
| `postgres.auth.password` | PostgreSQL password | `""` (generate) |
| `apiService.enabled` | Enable API Service | `true` |
| `apiService.replicas` | API Service replicas | `2` |
| `reservationProcessor.enabled` | Enable Reservation Processor | `true` |
| `availabilityUpdater.enabled` | Enable Availability Updater | `true` |
| `reservationExpiry.enabled` | Enable Reservation Expiry | `true` |

### Cloud Provider Configuration

#### AWS

```yaml
cloudProvider:
name: "aws"
region: "us-east-2"
aws:
eksClusterName: "my-cluster"
primaryAvailabilityZone: "us-east-2a"
efsSecurityGroupId: "sg-xxx"
efsSubnetIds: "subnet-xxx,subnet-yyy"
```

#### GCP

```yaml
cloudProvider:
name: "gcp"
region: "us-central1"
gcp:
gkeClusterName: "my-cluster"
zone: "us-central1-a"
projectId: "my-project"
```

## Upgrading

```bash
helm upgrade gpu-dev ./charts/gpu-dev-server -f my-values.yaml
```

## Uninstalling

```bash
helm uninstall gpu-dev
```

**Note:** PersistentVolumeClaims are not deleted by default to prevent data loss.
To delete them manually:

```bash
kubectl delete pvc -n gpu-controlplane -l app.kubernetes.io/instance=gpu-dev
```

## Architecture

```
┌─────────────────────────────────────────────────────┐
│ gpu-controlplane │
│ │
│ ┌──────────────┐ ┌──────────────┐ │
│ │ API Service │ │ PostgreSQL │ │
│ │ (FastAPI) │◀─│ + PGMQ │ │
│ └──────┬───────┘ └──────▲───────┘ │
│ │ │ │
│ │ Push jobs │ Poll jobs │
│ ▼ │ │
│ ┌──────────────────────┴─────────┐ │
│ │ Reservation Processor │ │
│ └────────────────────────────────┘ │
│ │
│ ┌────────────────┐ ┌────────────────┐ │
│ │ Availability │ │ Reservation │ │
│ │ Updater │ │ Expiry │ │
│ │ (CronJob) │ │ (CronJob) │ │
│ └────────────────┘ └────────────────┘ │
└─────────────────────────────────────────────────────┘
```

## Troubleshooting

### Check pod status

```bash
kubectl get pods -n gpu-controlplane
```

### View logs

```bash
# API Service
kubectl logs -f -n gpu-controlplane -l app=api-service

# Reservation Processor
kubectl logs -f -n gpu-controlplane -l app=reservation-processor

# CronJobs
kubectl logs -n gpu-controlplane -l app=availability-updater
kubectl logs -n gpu-controlplane -l app=reservation-expiry
```

### Database access

```bash
kubectl port-forward -n gpu-controlplane svc/postgres-primary 5432:5432
psql -h localhost -U gpudev -d gpudev
```

## License

See repository LICENSE file.
130 changes: 130 additions & 0 deletions charts/gpu-dev-server/templates/_helpers.tpl
Original file line number Diff line number Diff line change
@@ -0,0 +1,130 @@
{{/*
Expand the name of the chart.
*/}}
{{- define "gpu-dev-server.name" -}}
{{- default .Chart.Name .Values.nameOverride | trunc 63 | trimSuffix "-" }}
{{- end }}

{{/*
Create a default fully qualified app name.
*/}}
{{- define "gpu-dev-server.fullname" -}}
{{- if .Values.fullnameOverride }}
{{- .Values.fullnameOverride | trunc 63 | trimSuffix "-" }}
{{- else }}
{{- $name := default .Chart.Name .Values.nameOverride }}
{{- if contains $name .Release.Name }}
{{- .Release.Name | trunc 63 | trimSuffix "-" }}
{{- else }}
{{- printf "%s-%s" .Release.Name $name | trunc 63 | trimSuffix "-" }}
{{- end }}
{{- end }}
{{- end }}

{{/*
Create chart name and version as used by the chart label.
*/}}
{{- define "gpu-dev-server.chart" -}}
{{- printf "%s-%s" .Chart.Name .Chart.Version | replace "+" "_" | trunc 63 | trimSuffix "-" }}
{{- end }}

{{/*
Common labels
*/}}
{{- define "gpu-dev-server.labels" -}}
helm.sh/chart: {{ include "gpu-dev-server.chart" . }}
{{ include "gpu-dev-server.selectorLabels" . }}
{{- if .Chart.AppVersion }}
app.kubernetes.io/version: {{ .Chart.AppVersion | quote }}
{{- end }}
app.kubernetes.io/managed-by: {{ .Release.Service }}
{{- end }}

{{/*
Selector labels
*/}}
{{- define "gpu-dev-server.selectorLabels" -}}
app.kubernetes.io/name: {{ include "gpu-dev-server.name" . }}
app.kubernetes.io/instance: {{ .Release.Name }}
{{- end }}

{{/*
Controlplane namespace
*/}}
{{- define "gpu-dev-server.controlplaneNamespace" -}}
{{- .Values.namespaces.controlplane }}
{{- end }}

{{/*
Workloads namespace
*/}}
{{- define "gpu-dev-server.workloadsNamespace" -}}
{{- .Values.namespaces.workloads }}
{{- end }}

{{/*
PostgreSQL primary service name
*/}}
{{- define "gpu-dev-server.postgresPrimaryService" -}}
postgres-primary
{{- end }}

{{/*
PostgreSQL replica service name
*/}}
{{- define "gpu-dev-server.postgresReplicaService" -}}
postgres-replica
{{- end }}

{{/*
PostgreSQL connection host (primary)
*/}}
{{- define "gpu-dev-server.postgresHost" -}}
{{- printf "%s.%s.svc.cluster.local" (include "gpu-dev-server.postgresPrimaryService" .) (include "gpu-dev-server.controlplaneNamespace" .) }}
{{- end }}

{{/*
PostgreSQL credentials secret name
*/}}
{{- define "gpu-dev-server.postgresSecretName" -}}
{{- if .Values.postgres.auth.existingSecret }}
{{- .Values.postgres.auth.existingSecret }}
{{- else }}
postgres-credentials
{{- end }}
{{- end }}

{{/*
Registry native DNS name
*/}}
{{- define "gpu-dev-server.registryNativeDns" -}}
{{- printf "registry-native.%s.svc.cluster.local:5000" (include "gpu-dev-server.controlplaneNamespace" .) }}
{{- end }}

{{/*
Registry ghcr DNS name
*/}}
{{- define "gpu-dev-server.registryGhcrDns" -}}
{{- printf "registry-ghcr.%s.svc.cluster.local:5000" (include "gpu-dev-server.controlplaneNamespace" .) }}
{{- end }}

{{/*
Is AWS cloud provider
*/}}
{{- define "gpu-dev-server.isAws" -}}
{{- eq .Values.cloudProvider.name "aws" }}
{{- end }}

{{/*
Is GCP cloud provider
*/}}
{{- define "gpu-dev-server.isGcp" -}}
{{- eq .Values.cloudProvider.name "gcp" }}
{{- end }}

{{/*
Storage class name
*/}}
{{- define "gpu-dev-server.storageClass" -}}
{{- .Values.storage.class }}
{{- end }}
Loading