Merge pull request #16 from formula-code/grafana-dashboard-and-filter-improvements

Add Grafana dashboard, monitoring docs, and loosen symbolic filter

Author: atharvas

.gitignore

@@ -200,6 +200,9 @@ scratch/artifacts/
 supabase/.temp/
 supabase/.branches/
 
+# Grafana
+grafana/data/
+
 # MkDocs build output
 site/
 

Makefile

@@ -36,6 +36,36 @@ docker-clean: ## Clean up dangling Docker images and containers
 	@echo "Cleaning up dangling Docker images and containers"
 	@docker system prune -f
 
+.PHONY: supabase-up
+supabase-up: ## Start local Supabase instance
+	@npx supabase start
+
+.PHONY: supabase-down
+supabase-down: ## Stop local Supabase instance
+	@npx supabase stop
+
+.PHONY: supabase-status
+supabase-status: ## Show Supabase service status and URLs
+	@npx supabase status
+
+.PHONY: grafana-migrate
+grafana-migrate: ## Apply the grafana_ro read-only database role
+	@docker exec supabase_db_datasmith_new psql -U postgres -d postgres -f /dev/stdin < supabase/migrations/00009_grafana_readonly.sql
+	@echo "grafana_ro role created"
+
+.PHONY: grafana-up
+grafana-up: ## Start Grafana dashboard (http://localhost:3001)
+	@docker compose -f grafana/docker-compose.yml up -d
+	@echo "Grafana is running at http://localhost:3001"
+
+.PHONY: grafana-down
+grafana-down: ## Stop Grafana dashboard
+	@docker compose -f grafana/docker-compose.yml down
+
+.PHONY: grafana-logs
+grafana-logs: ## Tail Grafana container logs
+	@docker compose -f grafana/docker-compose.yml logs -f
+
 
 .PHONY: help
 help:

README.md

@@ -16,6 +16,9 @@
   <a href="https://formula-code.github.io/datasmith/">
     <img src="https://img.shields.io/badge/%F0%9F%93%9A%20Docs-4B0082?style=for-the-badge" alt="fc-data Documentation">
   </a>
+  <a href="https://nonconstricting-zara-unhinderingly.ngrok-free.dev/">
+    <img src="https://img.shields.io/badge/%F0%9F%93%8A%20Dashboard-FF6600?style=for-the-badge" alt="Live Dashboard">
+  </a>
 </p>
 
 [FormulaCode](https://formula-code.github.io/) is a *continually updating*  benchmark for evaluating  the holistic ability of LLM agents to optimize codebases. FormulaCode consists of two parts: a [pipeline](https://github.com/formula-code/datasmith) to construct performance optimization tasks, and an [execution harness](https://github.com/formula-code/terminal-bench) that connects a language model to our terminal sandbox. _This repository contains the task generation pipeline._

docs/guide/monitoring.md

@@ -0,0 +1,90 @@
+# Monitoring with Grafana
+
+fc-data includes a pre-configured Grafana dashboard for monitoring the pipeline, browsing data, and tracking dataset growth.
+
+## Prerequisites
+
+- Local Supabase instance running (`make supabase-up`)
+- Docker available on the host
+
+## Quick Start
+
+```bash
+# Apply the read-only database role (first time only)
+make grafana-migrate
+
+# Start Grafana
+make grafana-up
+```
+
+Grafana is now available at **<http://localhost:3001>** with no login required (anonymous read-only access).
+
+To expose it publicly via ngrok:
+
+```bash
+ngrok http 3001
+```
+
+## Architecture
+
+Grafana runs as a Docker container that joins the existing Supabase Docker network. It connects to Postgres via a **read-only** `grafana_ro` database role that can only `SELECT` — no writes are possible, even from the Explore SQL editor.
+
+```
+┌──────────────┐     ┌──────────────────────────┐
+│   Browser    │────▶│  Grafana (port 3001)     │
+└──────────────┘     │  Anonymous Viewer access  │
+                     └───────────┬──────────────┘
+                                 │ SELECT only
+                     ┌───────────▼──────────────┐
+                     │  Supabase PostgreSQL      │
+                     │  (grafana_ro role)        │
+                     └──────────────────────────┘
+```
+
+## Dashboard Panels
+
+The provisioned dashboard ("DataSmith Pipeline Overview") includes:
+
+| Section | Panels |
+|---------|--------|
+| **Dataset Growth** | Containers built by month, stat counts (PRs, containers, packages, repos), PR-to-container rate |
+| **Repository Overview** | All repos by stars, containers-by-repo treemap |
+| **Pull Request Insights** | Performance commit ratio, dataset time span, difficulty distribution, PR volume over time, optimization type distribution |
+| **Package Resolution** | Python version distribution, installability by repo, resolution strategy breakdown |
+| **Pipeline Status** | Active runners table, completion gauge, recent failures |
+| **Synthesis** | Success rate over time, failure stage distribution, top error messages |
+| **Synthesis Deeper Analysis** | First-attempt vs retry success, hardest repos, agent head-to-head |
+| **Attempt Details** | Attempts per repo, duration by agent, duration over time |
+| **Resource Metrics** | Build metrics table, image size vs build time, avg resources over time |
+| **Pipeline Funnel** | End-to-end conversion: Repos → PRs → Perf PRs → Resolved → Built → Published |
+| **Data Explorer** | Filtered table of performance PRs with referenced issues |
+
+## Ad-hoc SQL Queries
+
+Use Grafana's **Explore** mode (compass icon in the sidebar) to run arbitrary read-only SQL against the database. The `grafana_ro` role has `SELECT` access to all tables.
+
+## Admin Access
+
+To edit dashboards in the Grafana UI, log in with:
+
+- **Username:** `admin`
+- **Password:** value of `GRAFANA_ADMIN_PASSWORD` in `tokens.env` (defaults to `admin`)
+
+## Configuration Files
+
+| File | Purpose |
+|------|---------|
+| `grafana/docker-compose.yml` | Grafana service definition |
+| `grafana/provisioning/datasources/supabase-postgres.yml` | PostgreSQL datasource (read-only) |
+| `grafana/provisioning/dashboards/dashboard-provider.yml` | Dashboard file provider config |
+| `grafana/provisioning/dashboards-json/datasmith-overview.json` | Dashboard panels and queries |
+| `supabase/migrations/00009_grafana_readonly.sql` | Read-only Postgres role |
+
+## Makefile Targets
+
+```bash
+make grafana-up       # Start Grafana
+make grafana-down     # Stop Grafana
+make grafana-logs     # Tail container logs
+make grafana-migrate  # Apply the grafana_ro database role
+```

docs/index.md

@@ -1,6 +1,6 @@
 ![banner](https://github.com/formula-code/datasmith/raw/main/static/formula-code-datasmith.png)
 
-<p align="center">
+I<p align="center">
   <a href="https://formula-code.github.io/">
     <img src="https://img.shields.io/badge/%F0%9F%8C%90%20Website-0A7A5E?style=for-the-badge" alt="FormulaCode Website">
   </a>

grafana/docker-compose.yml

@@ -0,0 +1,37 @@
+services:
+  grafana:
+    image: grafana/grafana-oss:11.6.0
+    container_name: datasmith_grafana
+    restart: unless-stopped
+    ports:
+      - "3001:3000"
+    env_file:
+      - ../tokens.env
+    environment:
+      # Anonymous read-only access (no login required)
+      - GF_AUTH_ANONYMOUS_ENABLED=true
+      - GF_AUTH_ANONYMOUS_ORG_ROLE=Viewer
+      # Admin account for dashboard editing — password from tokens.env
+      - GF_SECURITY_ADMIN_USER=admin
+      - GF_SECURITY_ADMIN_PASSWORD=${GRAFANA_ADMIN_PASSWORD:-admin}
+      # Home dashboard
+      - GF_DASHBOARDS_DEFAULT_HOME_DASHBOARD_UID=datasmith-overview
+      # Minimum refresh interval
+      - GF_DASHBOARDS_MIN_REFRESH_INTERVAL=1m
+      # Plugins
+      - GF_INSTALL_PLUGINS=netsage-sankey-panel,marcusolsson-treemap-panel
+    volumes:
+      - grafana-data:/var/lib/grafana
+      - ./provisioning/datasources:/etc/grafana/provisioning/datasources:ro
+      - ./provisioning/dashboards:/etc/grafana/provisioning/dashboards:ro
+      - ./provisioning/dashboards-json:/var/lib/grafana/dashboards:ro
+    networks:
+      - supabase_net
+
+volumes:
+  grafana-data:
+
+networks:
+  supabase_net:
+    external: true
+    name: supabase_network_datasmith_new