feat: add rate limiting and 429 retry for API requests

CLAUDE.md

@@ -26,6 +26,7 @@ internal/
     gitlab.go          glab CLI token fallback
   provider/            Repository listing from APIs
     provider.go        Provider interface definition
+    ratelimit.go       Rate-limited HTTP transport (429 retry + token-bucket)
     bitbucket.go       Bitbucket Cloud REST API v2.0
     github.go          GitHub REST API
     gitlab.go          GitLab REST API v4
@@ -70,12 +71,14 @@ internal/
 
 - **Provider abstraction**: `provider.Provider` interface allows adding new git hosting providers. Each provider implements `ListRepos(ctx, ListOpts)`.
 - **Worker pool**: Bounded goroutine pool with semaphore pattern. Configurable concurrency via `--concurrency` flag.
+- **Rate limiting**: `RateLimitTransport` in `provider/ratelimit.go` implements `http.RoundTripper` with token-bucket rate limiting and 429 retry (exponential backoff, `Retry-After` header). Injected via `--rate-limit` flag (default: 0 = unlimited, retry-only). All providers accept `*http.Client` to share the transport.
 - **Partial failure**: Repos that fail to clone or analyze are recorded as errors in the report; the run continues.
 - **Auth**: Credentials stored at `~/.config/codemium/credentials.json` (0600 perms). Resolution order: env vars (`CODEMIUM_<PROVIDER>_TOKEN`) → saved credentials → CLI fallback (`gh auth token` for GitHub, `glab auth token` for GitLab).
 - **Clone strategy**: Shallow clone (depth 1, single branch, no tags) to temp dir, deleted after analysis.
 - **scc initialization**: `processor.ProcessConstants()` called via `sync.Once` since scc requires global initialization.
 - **AI estimation**: When `--ai-estimate` is used, a second pass fetches commit history via provider REST APIs. `provider.CommitLister` interface provides `ListCommits` and `CommitStats`. `aidetect.Detect` classifies commits, `aiestimate.Estimate` orchestrates per-repo. Results attach to existing report model as optional fields.
-- **Health classification**: When `--health` is used, repos are classified as Active (<180d), Maintained (180-365d), or Abandoned (>365d) based on last commit date. `--health-details` adds deep analysis: per-window author counts, code churn, bus factor, and velocity trend. Uses the same `CommitLister` interface.
+- **Health classification**: When `--health` is used, repos are classified as Active (<180d), Maintained (180-365d), or Abandoned (>365d) based on last commit date. Repos where commit history cannot be fetched (API errors, permissions) are classified as Failed with the error message stored in `RepoHealth.Error`. `--health-details` adds deep analysis: per-window author counts, code churn, bus factor, and velocity trend. Uses the same `CommitLister` interface.
+- **Error logging**: API errors from health, health-details, AI estimation, and partial commit stat failures are collected and written to `<report>.error.log` (derived from the report path, e.g. `report.error.log` for `report.json`) when any errors occur. Each line is prefixed with a category for easy filtering. `AnalyzeDetails` and `aiestimate.Estimate` return `(result, []string, error)` where `[]string` contains partial error messages.
 - **Vendor/generated filtering**: Always-on filtering using `go-enry` to skip vendor, generated, and binary files during analysis. `FilteredFiles` count is tracked per repo and in report totals.
 - **License detection**: After analysis, `license.Detect` scans the cloned repo directory for SPDX license identifiers (e.g., "MIT", "Apache-2.0"). Results appear in the per-repo License column.
 - **Code churn / hotspots**: Opt-in via `--churn` flag. Uses provider REST APIs to fetch per-file change data (`--churn-limit N` sets max commits, default 500). `churn.Analyze` collects per-file change frequencies; `churn.ComputeHotspots` ranks files by churn x complexity. Top 20 hotspots shown per repo.

README.md

@@ -111,12 +111,6 @@ export CODEMIUM_GITHUB_TOKEN=your_personal_access_token
    ```
    This prompts for your token, verifies it against the GitLab API, and stores it at `~/.config/codemium/credentials.json`.
 
-For self-hosted GitLab instances, set `CODEMIUM_GITLAB_URL`:
-```bash
-export CODEMIUM_GITLAB_URL=https://gitlab.example.com
-codemium auth login --provider gitlab
-```
-
 **Option 2: glab CLI**
 
 If you have the [GitLab CLI](https://gitlab.com/gitlab-org/cli) installed and authenticated, codemium can use its token automatically:
@@ -183,9 +177,6 @@ codemium analyze --provider gitlab --group myorg/mysubgroup
 
 # Specific repos
 codemium analyze --provider gitlab --group mygroup --repos api,frontend
-
-# Self-hosted GitLab
-CODEMIUM_GITLAB_URL=https://gitlab.example.com codemium analyze --provider gitlab --group mygroup
 ```
 
 ### Analyze trends over time
@@ -267,7 +258,7 @@ This is especially useful when Bitbucket project codes or repo naming convention
 
 ### Repository health classification
 
-Classify repositories as Active, Maintained, or Abandoned based on last commit date:
+Classify repositories as Active, Maintained, Abandoned, or Failed based on commit history:
 
 ```bash
 # Quick health check (1 API call per repo, no cloning)
@@ -284,11 +275,17 @@ Health categories:
 - **Active**: last commit < 180 days ago
 - **Maintained**: 180–365 days ago
 - **Abandoned**: > 365 days ago
+- **Failed**: commit history could not be fetched (API error, permissions, etc.)
+
+API requests that receive a 429 (Too Many Requests) response are automatically retried with exponential backoff (up to 5 retries). Use `--rate-limit` to proactively throttle requests and avoid hitting rate limits (e.g., `--rate-limit 5` for GitLab's 300 req/min raw endpoint limit).
+
+When API errors occur during health classification, AI estimation, or detailed analysis, an error log is automatically written next to the JSON report (e.g., `output/report.error.log` for `output/report.json`). Each line is prefixed with a category (`[health]`, `[health-details]`, `[ai-estimate]`, `[ai-estimate-detail]`) for easy filtering with `grep`.
 
 ### Additional flags
 
 ```bash
 --concurrency 10            # Parallel workers (default: 5)
+--rate-limit 5              # Max API requests per second (default: unlimited)
 --include-archived          # Include archived repos (excluded by default)
 --include-forks             # Include forked repos (excluded by default)
 --ai-estimate               # Estimate AI-generated code via commit history analysis

cmd/codemium/main.go

@@ -12,6 +12,7 @@ import (
 	"path/filepath"
 	"sort"
 	"strings"
+	"sync"
 	"time"
 
 	tea "github.com/charmbracelet/bubbletea"
@@ -20,11 +21,11 @@ import (
 
 	"github.com/dsablic/codemium/internal/aiestimate"
 	"github.com/dsablic/codemium/internal/analyzer"
-	"github.com/dsablic/codemium/internal/churn"
-	"github.com/dsablic/codemium/internal/license"
 	"github.com/dsablic/codemium/internal/auth"
+	"github.com/dsablic/codemium/internal/churn"
 	"github.com/dsablic/codemium/internal/health"
 	"github.com/dsablic/codemium/internal/history"
+	"github.com/dsablic/codemium/internal/license"
 	"github.com/dsablic/codemium/internal/model"
 	"github.com/dsablic/codemium/internal/narrative"
 	"github.com/dsablic/codemium/internal/output"
@@ -39,6 +40,13 @@ var (
 	date    = "unknown"
 )
 
+// errorEntry represents a diagnostic error for the error log.
+type errorEntry struct {
+	Category string
+	Repo     string
+	Message  string
+}
+
 func main() {
 	root := &cobra.Command{
 		Use:     "codemium",
@@ -265,6 +273,7 @@ func newAnalyzeCmd() *cobra.Command {
 	cmd.Flags().Int("health-commit-limit", 500, "Max commits to scan per repo for health details (0 = unlimited)")
 	cmd.Flags().Bool("churn", false, "Analyze code churn and hotspots")
 	cmd.Flags().Int("churn-limit", 500, "Max commits to scan per repo for churn analysis (0 = unlimited)")
+	cmd.Flags().Float64("rate-limit", 0, "Max API requests per second (0 = unlimited)")
 
 	cmd.MarkFlagRequired("provider")
 
@@ -287,6 +296,10 @@ func runAnalyze(cmd *cobra.Command, args []string) error {
 	includeForks, _ := cmd.Flags().GetBool("include-forks")
 	concurrency, _ := cmd.Flags().GetInt("concurrency")
 	outputPath, _ := cmd.Flags().GetString("output")
+	rateLimit, _ := cmd.Flags().GetFloat64("rate-limit")
+
+	// Create rate-limited HTTP client
+	httpClient := &http.Client{Transport: &provider.RateLimitTransport{ReqPerSec: rateLimit}}
 
 	// Load credentials
 	store := auth.NewFileStore(auth.DefaultStorePath())
@@ -314,21 +327,21 @@ func runAnalyze(cmd *cobra.Command, args []string) error {
 		if workspace == "" {
 			return fmt.Errorf("--workspace is required for bitbucket")
 		}
-		prov = provider.NewBitbucket(cred.AccessToken, cred.Username, "")
+		prov = provider.NewBitbucket(cred.AccessToken, cred.Username, "", httpClient)
 	case "github":
 		if org != "" && user != "" {
 			return fmt.Errorf("--org and --user are mutually exclusive for github")
 		}
 		if org == "" && user == "" {
 			return fmt.Errorf("--org or --user is required for github")
 		}
-		prov = provider.NewGitHub(cred.AccessToken, "")
+		prov = provider.NewGitHub(cred.AccessToken, "", httpClient)
 	case "gitlab":
 		if group == "" {
 			return fmt.Errorf("--group is required for gitlab")
 		}
 		baseURL := os.Getenv("CODEMIUM_GITLAB_URL")
-		prov = provider.NewGitLab(cred.AccessToken, baseURL)
+		prov = provider.NewGitLab(cred.AccessToken, baseURL, httpClient)
 	default:
 		return fmt.Errorf("unsupported provider: %s", providerName)
 	}
@@ -441,6 +454,10 @@ func runAnalyze(cmd *cobra.Command, args []string) error {
 		program = nil
 	}
 
+	// Diagnostic error collection (written to error.log if non-empty)
+	var diagErrors []errorEntry
+	var diagMu sync.Mutex
+
 	// AI estimation phase
 	aiEstimateFlag, _ := cmd.Flags().GetBool("ai-estimate")
 	aiCommitLimit, _ := cmd.Flags().GetInt("ai-commit-limit")
@@ -471,7 +488,14 @@ func runAnalyze(cmd *cobra.Command, args []string) error {
 		}
 
 		aiResults := worker.RunWithProgress(ctx, repoList, concurrency, func(ctx context.Context, repo model.Repo) (*model.RepoStats, error) {
-			est, err := aiestimate.Estimate(ctx, commitLister, repo, aiCommitLimit)
+			est, partialErrs, err := aiestimate.Estimate(ctx, commitLister, repo, aiCommitLimit)
+			if len(partialErrs) > 0 {
+				diagMu.Lock()
+				for _, pe := range partialErrs {
+					diagErrors = append(diagErrors, errorEntry{Category: "ai-estimate-detail", Repo: repo.Slug, Message: pe})
+				}
+				diagMu.Unlock()
+			}
 			if err != nil {
 				return nil, err
 			}
@@ -491,7 +515,11 @@ func runAnalyze(cmd *cobra.Command, args []string) error {
 		// Attach AI estimates to analysis results
 		aiByRepo := make(map[string]*model.AIEstimate)
 		for _, r := range aiResults {
-			if r.Err == nil && r.Stats != nil && r.Stats.AIEstimate != nil {
+			if r.Err != nil {
+				diagErrors = append(diagErrors, errorEntry{Category: "ai-estimate", Repo: r.Repo.Slug, Message: r.Err.Error()})
+				continue
+			}
+			if r.Stats != nil && r.Stats.AIEstimate != nil {
 				aiByRepo[r.Repo.Slug] = r.Stats.AIEstimate
 			}
 		}
@@ -548,16 +576,40 @@ func runAnalyze(cmd *cobra.Command, args []string) error {
 		healthResults := worker.RunWithProgress(ctx, repoList, concurrency, func(ctx context.Context, repo model.Repo) (*model.RepoStats, error) {
 			commits, err := commitLister.ListCommits(ctx, repo, commitLimit)
 			if err != nil {
-				return nil, err
+				diagMu.Lock()
+				diagErrors = append(diagErrors, errorEntry{Category: "health", Repo: repo.Slug, Message: err.Error()})
+				diagMu.Unlock()
+				return &model.RepoStats{
+					Repository: repo.Slug,
+					Health: &model.RepoHealth{
+						Category:        model.HealthFailed,
+						DaysSinceCommit: -1,
+						Error:           err.Error(),
+					},
+				}, nil
 			}
 
 			h := health.ClassifyFromCommits(commits, now)
 
 			var details *model.RepoHealthDetails
 			if healthDetailsFlag && len(commits) > 0 {
-				details, err = health.AnalyzeDetails(ctx, commitLister, repo, commits, now)
+				var partialErrs []string
+				details, partialErrs, err = health.AnalyzeDetails(ctx, commitLister, repo, commits, now)
+				if len(partialErrs) > 0 {
+					diagMu.Lock()
+					for _, pe := range partialErrs {
+						diagErrors = append(diagErrors, errorEntry{Category: "health-details", Repo: repo.Slug, Message: pe})
+					}
+					diagMu.Unlock()
+				}
 				if err != nil {
-					return nil, err
+					diagMu.Lock()
+					diagErrors = append(diagErrors, errorEntry{Category: "health-details", Repo: repo.Slug, Message: err.Error()})
+					diagMu.Unlock()
+					return &model.RepoStats{
+						Repository: repo.Slug,
+						Health:     h,
+					}, nil
 				}
 			}
 
@@ -648,6 +700,24 @@ func runAnalyze(cmd *cobra.Command, args []string) error {
 		}
 	}
 
+	// Write error.log if there were any diagnostic errors
+	if len(diagErrors) > 0 {
+		ext := filepath.Ext(outputPath)
+		errorLogPath := strings.TrimSuffix(outputPath, ext) + ".error.log"
+		if err := os.MkdirAll(filepath.Dir(errorLogPath), 0o755); err != nil {
+			return fmt.Errorf("create error log directory: %w", err)
+		}
+		f, err := os.Create(errorLogPath)
+		if err != nil {
+			return fmt.Errorf("create error log: %w", err)
+		}
+		for _, e := range diagErrors {
+			fmt.Fprintf(f, "[%s] %s | %s\n", e.Category, e.Repo, e.Message)
+		}
+		f.Close()
+		fmt.Fprintf(os.Stderr, "Error log written to %s (%d entries)\n", errorLogPath, len(diagErrors))
+	}
+
 	// Build report — use user/group as organization in metadata when set
 	reportOrg := org
 	if user != "" {
@@ -881,6 +951,7 @@ func newTrendsCmd() *cobra.Command {
 	cmd.Flags().Bool("include-forks", false, "Include forked repos")
 	cmd.Flags().Int("concurrency", 5, "Number of parallel workers")
 	cmd.Flags().String("output", "output/report.json", "Write JSON to file")
+	cmd.Flags().Float64("rate-limit", 0, "Max API requests per second (0 = unlimited)")
 
 	cmd.MarkFlagRequired("provider")
 	cmd.MarkFlagRequired("since")
@@ -907,6 +978,7 @@ func runTrends(cmd *cobra.Command, args []string) error {
 	includeForks, _ := cmd.Flags().GetBool("include-forks")
 	concurrency, _ := cmd.Flags().GetInt("concurrency")
 	outputPath, _ := cmd.Flags().GetString("output")
+	rateLimit, _ := cmd.Flags().GetFloat64("rate-limit")
 
 	if interval != "monthly" && interval != "weekly" {
 		return fmt.Errorf("--interval must be 'monthly' or 'weekly'")
@@ -929,27 +1001,29 @@ func runTrends(cmd *cobra.Command, args []string) error {
 		store.Save(providerName, cred)
 	}
 
+	httpClient := &http.Client{Transport: &provider.RateLimitTransport{ReqPerSec: rateLimit}}
+
 	var prov provider.Provider
 	switch providerName {
 	case "bitbucket":
 		if workspace == "" {
 			return fmt.Errorf("--workspace is required for bitbucket")
 		}
-		prov = provider.NewBitbucket(cred.AccessToken, cred.Username, "")
+		prov = provider.NewBitbucket(cred.AccessToken, cred.Username, "", httpClient)
 	case "github":
 		if org != "" && user != "" {
 			return fmt.Errorf("--org and --user are mutually exclusive for github")
 		}
 		if org == "" && user == "" {
 			return fmt.Errorf("--org or --user is required for github")
 		}
-		prov = provider.NewGitHub(cred.AccessToken, "")
+		prov = provider.NewGitHub(cred.AccessToken, "", httpClient)
 	case "gitlab":
 		if group == "" {
 			return fmt.Errorf("--group is required for gitlab")
 		}
 		baseURL := os.Getenv("CODEMIUM_GITLAB_URL")
-		prov = provider.NewGitLab(cred.AccessToken, baseURL)
+		prov = provider.NewGitLab(cred.AccessToken, baseURL, httpClient)
 	default:
 		return fmt.Errorf("unsupported provider: %s", providerName)
 	}

internal/aiestimate/estimate.go

@@ -2,6 +2,7 @@ package aiestimate
 
 import (
 	"context"
+	"fmt"
 	"sync"
 
 	"github.com/dsablic/codemium/internal/aidetect"
@@ -12,10 +13,11 @@ import (
 const statsConcurrency = 10
 
 // Estimate computes AI attribution metrics for a single repo.
-func Estimate(ctx context.Context, cl provider.CommitLister, repo model.Repo, commitLimit int) (*model.AIEstimate, error) {
+// It returns the estimate, a list of partial error messages (per-commit stat failures), and a fatal error.
+func Estimate(ctx context.Context, cl provider.CommitLister, repo model.Repo, commitLimit int) (*model.AIEstimate, []string, error) {
 	commits, err := cl.ListCommits(ctx, repo, commitLimit)
 	if err != nil {
-		return nil, err
+		return nil, nil, err
 	}
 
 	est := &model.AIEstimate{
@@ -68,10 +70,12 @@ func Estimate(ctx context.Context, cl provider.CommitLister, repo model.Repo, co
 	}
 	wg.Wait()
 
+	var partialErrors []string
 	for i, fc := range flagged {
 		d := details[i]
 		if d.err != nil {
-			continue // partial failure — skip this commit's stats
+			partialErrors = append(partialErrors, fmt.Sprintf("CommitStats %s: %v", fc.info.Hash, d.err))
+			continue
 		}
 
 		est.AIAdditions += d.additions
@@ -95,5 +99,5 @@ func Estimate(ctx context.Context, cl provider.CommitLister, repo model.Repo, co
 		})
 	}
 
-	return est, nil
+	return est, partialErrors, nil
 }

internal/aiestimate/estimate_test.go

@@ -40,7 +40,7 @@ func TestEstimateBasic(t *testing.T) {
 	}
 
 	repo := model.Repo{Slug: "test-repo", URL: "https://github.com/org/test-repo"}
-	estimate, err := aiestimate.Estimate(context.Background(), mock, repo, 500)
+	estimate, _, err := aiestimate.Estimate(context.Background(), mock, repo, 500)
 	if err != nil {
 		t.Fatalf("Estimate: %v", err)
 	}
@@ -68,7 +68,7 @@ func TestEstimateNoAICommits(t *testing.T) {
 	}
 
 	repo := model.Repo{Slug: "test-repo", URL: "https://github.com/org/test-repo"}
-	estimate, err := aiestimate.Estimate(context.Background(), mock, repo, 500)
+	estimate, _, err := aiestimate.Estimate(context.Background(), mock, repo, 500)
 	if err != nil {
 		t.Fatalf("Estimate: %v", err)
 	}
@@ -95,7 +95,7 @@ func TestEstimatePercentCalculation(t *testing.T) {
 	}
 
 	repo := model.Repo{Slug: "r", URL: "https://github.com/o/r"}
-	est, err := aiestimate.Estimate(context.Background(), mock, repo, 500)
+	est, _, err := aiestimate.Estimate(context.Background(), mock, repo, 500)
 	if err != nil {
 		t.Fatalf("Estimate: %v", err)
 	}
@@ -116,7 +116,7 @@ func TestEstimateFirstLineMessage(t *testing.T) {
 	}
 
 	repo := model.Repo{Slug: "r", URL: "https://github.com/o/r"}
-	est, err := aiestimate.Estimate(context.Background(), mock, repo, 500)
+	est, _, err := aiestimate.Estimate(context.Background(), mock, repo, 500)
 	if err != nil {
 		t.Fatalf("Estimate: %v", err)
 	}

internal/auth/gitlab.go

@@ -9,7 +9,7 @@ import (
 // GlabCLIToken attempts to get a GitLab token from the glab CLI tool.
 // Returns the token and true if successful, or empty string and false otherwise.
 func GlabCLIToken() (string, bool) {
-	out, err := exec.Command("glab", "auth", "token").Output()
+	out, err := exec.Command("glab", "config", "get", "token", "--host", "gitlab.com").Output()
 	if err != nil {
 		return "", false
 	}