diff --git a/docs/patterns/branch-protection-automation.md b/docs/patterns/branch-protection-automation.md new file mode 100644 index 0000000..bc10231 --- /dev/null +++ b/docs/patterns/branch-protection-automation.md @@ -0,0 +1,841 @@ +# Branch Protection Automation + +**Pattern**: Automatically configure branch protection rules to enforce code quality standards and prevent broken code from reaching main. + +**Problem**: Setting up branch protection manually through the GitHub UI is tedious, error-prone, and inconsistent across repositories. Teams forget required settings, leading to broken code in main branch. Manual setup doesn't scale to dozens or hundreds of repositories. + +**Solution**: Automate branch protection setup using GitHub API or CLI. Define protection rules as code, apply consistently across all repositories, and ensure required status checks always gate merges. + +--- + +## Quick Start (5 Minutes) + +Set up branch protection for your repository. + +### Prerequisites + +```bash +# Install GitHub CLI +# macOS: brew install gh +# Linux: https://github.com/cli/cli#installation +# Windows: winget install GitHub.cli + +# Authenticate +gh auth login +``` + +### Method 1: Using gh CLI (Simplest) + +```bash +# Basic protection with required status checks +gh api repos/:owner/:repo/branches/main/protection \ + --method PUT \ + --field required_status_checks[strict]=true \ + --field required_status_checks[contexts][]=ci \ + --field enforce_admins=false \ + --field required_pull_request_reviews[required_approving_review_count]=0 \ + --field restrictions=null + +# Verify it worked +gh api repos/:owner/:repo/branches/main/protection | jq '.required_status_checks' +``` + +### Method 2: Using Rulesets (Free Tier Friendly) + +For repositories without GitHub Pro: + +```bash +# Create ruleset for main branch +gh api repos/:owner/:repo/rulesets \ + --method POST \ + --field name="Protect main branch" \ + --field enforcement="active" \ + --field target="branch" \ + --field conditions[ref_name][include][]=refs/heads/main \ + --field rules[0][type]="pull_request" \ + --field rules[1][type]="required_status_checks" \ + --field rules[1][parameters][required_status_checks][0][context]="ci" +``` + +**Done!** Your main branch is now protected and requires CI to pass before merging. + +--- + +## Real-World Example: Reporters Repository + +The reporters repository needs branch protection to work with auto-merge: + +### Required Setup + +For dependabot auto-merge to work safely, the reporters repo needs: + +1. **Status checks required**: CI pipeline must pass +2. **Up-to-date branches**: PRs must be current with main +3. **No admin bypass**: Even admins follow the rules +4. **No force pushes**: Protect git history + +### Why This Matters + +From `.github/workflows/README.md`: + +```markdown +### 2. Branch Protection Rules + +For auto-merge to work safely, configure branch protection on `main`: + +1. Go to Settings > Branches > Add rule +2. Branch name pattern: `main` +3. Enable: + - ✅ Require status checks to pass before merging + - ✅ Require branches to be up to date before merging +4. Select required status checks: + - `ci` (from ci-pipeline.yml) +``` + +Without this, dependabot could auto-merge broken updates that haven't passed CI. + +### Manual vs Automated + +**Manual Setup (5 minutes per repo):** +- Navigate to Settings > Branches +- Click "Add rule" +- Enter branch name pattern +- Check 10+ boxes +- Select status checks from dropdown +- Save (hope you didn't miss anything) +- Repeat for every repository + +**Automated Setup (5 seconds per repo):** +```bash +./scripts/setup-branch-protection.sh ambient-code reporters +``` + +--- + +## Automation Scripts + +### Full-Featured Setup Script + +Create `scripts/setup-branch-protection.sh`: + +```bash +#!/bin/bash +# Setup branch protection for a repository +# +# Usage: ./scripts/setup-branch-protection.sh OWNER REPO [BRANCH] +# +# Example: ./scripts/setup-branch-protection.sh ambient-code reporters main + +set -e + +OWNER="${1:-}" +REPO="${2:-}" +BRANCH="${3:-main}" + +if [ -z "$OWNER" ] || [ -z "$REPO" ]; then + echo "Usage: $0 OWNER REPO [BRANCH]" + echo "Example: $0 ambient-code reporters main" + exit 1 +fi + +echo "Setting up branch protection for $OWNER/$REPO (branch: $BRANCH)" + +# Check if repo exists and we have access +if ! gh api "repos/$OWNER/$REPO" >/dev/null 2>&1; then + echo "Error: Cannot access repository $OWNER/$REPO" + echo "Check that:" + echo " 1. Repository exists" + echo " 2. You have admin access" + echo " 3. You're authenticated: gh auth status" + exit 1 +fi + +# Get repository visibility +VISIBILITY=$(gh api "repos/$OWNER/$REPO" --jq '.visibility') +IS_PRIVATE=$(gh api "repos/$OWNER/$REPO" --jq '.private') + +echo "Repository visibility: $VISIBILITY (private: $IS_PRIVATE)" + +# Try to set up traditional branch protection (requires GitHub Pro for private repos) +echo "Attempting to set up branch protection rules..." + +if gh api "repos/$OWNER/$REPO/branches/$BRANCH/protection" \ + --method PUT \ + --field required_status_checks[strict]=true \ + --field required_status_checks[contexts][]=ci \ + --field enforce_admins=false \ + --field required_pull_request_reviews[required_approving_review_count]=0 \ + --field required_pull_request_reviews[dismiss_stale_reviews]=false \ + --field required_pull_request_reviews[require_code_owner_reviews]=false \ + --field restrictions=null \ + --field required_linear_history=false \ + --field allow_force_pushes=false \ + --field allow_deletions=false \ + >/dev/null 2>&1; then + + echo "✅ Branch protection rules set successfully!" + + # Verify the setup + echo "" + echo "Verification:" + gh api "repos/$OWNER/$REPO/branches/$BRANCH/protection" --jq '{ + required_status_checks: .required_status_checks.contexts, + enforce_admins: .enforce_admins.enabled, + required_reviews: .required_pull_request_reviews.required_approving_review_count, + allow_force_pushes: .allow_force_pushes.enabled + }' + +else + echo "⚠️ Branch protection API failed (likely requires GitHub Pro for private repos)" + echo "" + echo "Falling back to rulesets (available on free tier)..." + + # Create ruleset instead + if gh api "repos/$OWNER/$REPO/rulesets" \ + --method POST \ + --field name="Protect $BRANCH branch" \ + --field enforcement="active" \ + --field target="branch" \ + --field conditions[ref_name][include][]=refs/heads/$BRANCH \ + --field rules[0][type]="pull_request" \ + --field rules[0][parameters][required_approving_review_count]=0 \ + --field rules[1][type]="required_status_checks" \ + --field rules[1][parameters][strict_required_status_checks_policy]=true \ + --field rules[1][parameters][required_status_checks][0][context]="ci" \ + --field rules[2][type]="deletion" \ + --field rules[3][type]="non_fast_forward" \ + >/dev/null 2>&1; then + + echo "✅ Ruleset created successfully!" + echo "" + echo "Your branch is protected via rulesets instead of branch protection rules." + echo "View at: https://github.com/$OWNER/$REPO/settings/rules" + + else + echo "❌ Failed to create ruleset" + echo "" + echo "Manual setup required:" + echo "1. Go to https://github.com/$OWNER/$REPO/settings/branches" + echo "2. Click 'Add rule' for branch: $BRANCH" + echo "3. Enable:" + echo " - Require status checks to pass (select 'ci')" + echo " - Require branches to be up to date" + echo "4. Save changes" + exit 1 + fi +fi + +echo "" +echo "Branch protection setup complete!" +echo "" +echo "Required status checks:" +echo " - ci (from .github/workflows/ci-pipeline.yml)" +echo "" +echo "Next steps:" +echo " 1. Verify CI workflow is working: gh workflow list" +echo " 2. Test with a PR to confirm checks are required" +echo " 3. Enable dependabot auto-merge if desired" +``` + +Make it executable: + +```bash +chmod +x scripts/setup-branch-protection.sh +``` + +### Usage + +```bash +# Setup for specific repo +./scripts/setup-branch-protection.sh ambient-code reporters + +# Setup for different branch +./scripts/setup-branch-protection.sh ambient-code reporters develop + +# Setup for multiple repos +for repo in repo1 repo2 repo3; do + ./scripts/setup-branch-protection.sh ambient-code "$repo" +done +``` + +--- + +## Configuration Options + +### Minimal Protection (Required) + +Bare minimum for safety: + +```bash +gh api repos/$OWNER/$REPO/branches/main/protection \ + --method PUT \ + --field required_status_checks[strict]=true \ + --field required_status_checks[contexts][]=ci \ + --field enforce_admins=false \ + --field restrictions=null +``` + +**Enables:** +- ✅ CI must pass before merge +- ✅ Branches must be up-to-date + +### Standard Protection (Recommended) + +Good for most teams: + +```bash +gh api repos/$OWNER/$REPO/branches/main/protection \ + --method PUT \ + --field required_status_checks[strict]=true \ + --field required_status_checks[contexts][]=ci \ + --field required_status_checks[contexts][]=security-scan \ + --field enforce_admins=true \ + --field required_pull_request_reviews[required_approving_review_count]=1 \ + --field required_pull_request_reviews[dismiss_stale_reviews]=true \ + --field allow_force_pushes=false \ + --field allow_deletions=false \ + --field restrictions=null +``` + +**Enables:** +- ✅ CI and security scans must pass +- ✅ 1 approval required +- ✅ Stale reviews dismissed on new commits +- ✅ No force pushes +- ✅ No branch deletion +- ✅ Even admins must follow rules + +### Strict Protection (High Security) + +For critical production code: + +```bash +gh api repos/$OWNER/$REPO/branches/main/protection \ + --method PUT \ + --field required_status_checks[strict]=true \ + --field required_status_checks[contexts][]=ci \ + --field required_status_checks[contexts][]=security-scan \ + --field required_status_checks[contexts][]=code-review \ + --field enforce_admins=true \ + --field required_pull_request_reviews[required_approving_review_count]=2 \ + --field required_pull_request_reviews[dismiss_stale_reviews]=true \ + --field required_pull_request_reviews[require_code_owner_reviews]=true \ + --field required_linear_history=true \ + --field allow_force_pushes=false \ + --field allow_deletions=false \ + --field restrictions[users][]=security-team \ + --field restrictions[teams][]=senior-engineers +``` + +**Enables:** +- ✅ Multiple required status checks +- ✅ 2 approvals required +- ✅ Code owner approval required +- ✅ Linear history (no merge commits) +- ✅ Restricted to specific users/teams + +--- + +## GitHub Free vs Pro Limitations + +### Branch Protection API Limitations + +**GitHub Free (Public Repos):** +- ✅ Full branch protection API access +- ✅ Required status checks +- ✅ Required reviews +- ✅ Enforce admins +- ✅ All features available + +**GitHub Free (Private Repos):** +- ❌ Branch protection API requires GitHub Pro +- ❌ Cannot use traditional protection rules +- ✅ Can use Rulesets instead (newer feature) + +**GitHub Pro/Team/Enterprise:** +- ✅ Full branch protection on all repos +- ✅ All features available + +### Rulesets as Alternative + +Rulesets are available on all tiers: + +```bash +# Works on GitHub Free for private repos +gh api repos/$OWNER/$REPO/rulesets \ + --method POST \ + --field name="Protect main" \ + --field enforcement="active" \ + --field target="branch" \ + --field conditions[ref_name][include][]=refs/heads/main \ + --field rules[0][type]="pull_request" \ + --field rules[1][type]="required_status_checks" \ + --field rules[1][parameters][required_status_checks][0][context]="ci" +``` + +### Detection Script + +Check which method to use: + +```bash +#!/bin/bash +# detect-protection-method.sh + +OWNER="$1" +REPO="$2" + +# Check if branch protection API works +if gh api "repos/$OWNER/$REPO/branches/main/protection" >/dev/null 2>&1; then + echo "Branch Protection API: Available" + echo "Use: Branch protection rules" +else + echo "Branch Protection API: Not available (requires GitHub Pro for private repos)" + echo "Use: Rulesets instead" +fi + +# Check if rulesets are available +if gh api "repos/$OWNER/$REPO/rulesets" >/dev/null 2>&1; then + echo "Rulesets API: Available" +else + echo "Rulesets API: Not available" +fi +``` + +--- + +## Common Status Check Names + +Map your workflows to status check names: + +| Workflow File | Job Name | Status Check Context | +|---------------|----------|---------------------| +| `ci-pipeline.yml` | `ci` | `ci` | +| `ci-pipeline.yml` | `test` | `test` | +| `quality-check.yml` | `quality` | `quality` | +| `security-scan.yml` | `security` | `security` | +| `build.yml` | `build` | `build` | + +Find your workflow's status check name: + +```bash +# List recent check runs for a PR +gh api repos/$OWNER/$REPO/commits/COMMIT_SHA/check-runs \ + --jq '.check_runs[] | {name: .name, status: .status, conclusion: .conclusion}' + +# Or view in GitHub UI: +# Pull Request > Checks tab > Look at check names +``` + +--- + +## Bulk Operations + +### Apply to Multiple Repositories + +```bash +#!/bin/bash +# setup-all-repos.sh + +OWNER="ambient-code" +REPOS=( + "reporters" + "reference" + "demo-fastapi" + "claude-code" +) + +for repo in "${REPOS[@]}"; do + echo "Setting up $repo..." + ./scripts/setup-branch-protection.sh "$OWNER" "$repo" + echo "" +done +``` + +### Apply to All Repos in Organization + +```bash +#!/bin/bash +# setup-org-protection.sh + +ORG="ambient-code" + +# Get all repos in org +gh repo list "$ORG" --json name --jq '.[].name' | while read -r repo; do + echo "Setting up $repo..." + ./scripts/setup-branch-protection.sh "$ORG" "$repo" + echo "" +done +``` + +### Update Existing Protection + +```bash +#!/bin/bash +# update-status-checks.sh +# Add a new required status check to existing protection + +OWNER="$1" +REPO="$2" +NEW_CHECK="$3" + +# Get current contexts +CURRENT=$(gh api "repos/$OWNER/$REPO/branches/main/protection" \ + --jq '.required_status_checks.contexts[]') + +# Add new check +CHECKS=() +while IFS= read -r check; do + CHECKS+=("$check") +done <<< "$CURRENT" +CHECKS+=("$NEW_CHECK") + +# Build the field arguments +ARGS="" +for check in "${CHECKS[@]}"; do + ARGS="$ARGS --field required_status_checks[contexts][]=$check" +done + +# Update protection +gh api "repos/$OWNER/$REPO/branches/main/protection" \ + --method PUT \ + --field required_status_checks[strict]=true \ + $ARGS \ + --field enforce_admins=false \ + --field restrictions=null + +echo "Added $NEW_CHECK to required status checks" +``` + +--- + +## Troubleshooting + +### Error: Resource not accessible by integration + +**Cause**: Insufficient permissions. + +**Solution**: + +```bash +# Check your permissions +gh api repos/$OWNER/$REPO --jq '.permissions' + +# You need admin access +# "permissions": { +# "admin": true, +# "push": true, +# "pull": true +# } +``` + +Ask a repository admin to run the script or grant you admin access. + +### Error: Validation Failed (422) + +**Cause**: Invalid API request or trying to use Pro features on Free tier. + +**Solution**: + +```bash +# Check repository visibility +gh api repos/$OWNER/$REPO --jq '{visibility: .visibility, private: .private}' + +# If private repo on Free tier, use rulesets instead +./scripts/setup-branch-protection.sh $OWNER $REPO --use-rulesets +``` + +### Error: Status check "ci" not found + +**Cause**: Status check hasn't run yet or wrong name. + +**Solution**: + +```bash +# Create a dummy commit to trigger CI +git commit --allow-empty -m "Trigger CI for branch protection setup" +git push + +# Wait for CI to run, then setup branch protection +gh run list --workflow=ci-pipeline.yml --limit=1 +./scripts/setup-branch-protection.sh $OWNER $REPO +``` + +Or allow the status check to be added even if it hasn't run: + +```bash +# GitHub will warn but allow it +gh api repos/$OWNER/$REPO/branches/main/protection \ + --method PUT \ + --field required_status_checks[strict]=true \ + --field required_status_checks[contexts][]=ci \ # Doesn't exist yet - OK + --field enforce_admins=false \ + --field restrictions=null +``` + +### Branch Protection Not Enforced + +**Cause**: Admins can bypass, or settings not saved. + +**Solution**: + +```bash +# Verify protection is actually enabled +gh api repos/$OWNER/$REPO/branches/main/protection --jq '{ + enabled: true, + required_checks: .required_status_checks.contexts, + enforce_admins: .enforce_admins.enabled +}' + +# If enforce_admins is false, enable it: +gh api repos/$OWNER/$REPO/branches/main/protection \ + --method PUT \ + --field enforce_admins=true \ + --field required_status_checks[strict]=true \ + --field required_status_checks[contexts][]=ci \ + --field restrictions=null +``` + +### Can Merge Without Status Checks + +**Cause**: Status check name doesn't match workflow job name. + +**Solution**: + +```bash +# Find the actual status check name from a PR +gh pr view 123 --json statusCheckRollup --jq '.statusCheckRollup[] | .context' + +# Use the exact name in branch protection +gh api repos/$OWNER/$REPO/branches/main/protection \ + --method PUT \ + --field required_status_checks[contexts][]="CI Pipeline / ci" # Use exact name +``` + +--- + +## Best Practices + +### 1. Set Up Protection Early + +Add branch protection when you create the repository: + +```bash +# Initialize repo +gh repo create $ORG/$REPO --public +cd $REPO + +# Add CI workflow +mkdir -p .github/workflows +cp ../templates/ci-pipeline.yml .github/workflows/ + +# Commit and push +git add .github/ +git commit -m "Add CI workflow" +git push + +# Wait for first CI run +sleep 30 + +# Enable protection +../scripts/setup-branch-protection.sh $ORG $REPO +``` + +### 2. Test Protection with a PR + +```bash +# Create test branch +git checkout -b test-protection + +# Make a change that will fail CI +echo "print('bad code')" >> test.py + +# Push and create PR +git add test.py +git commit -m "Test: should fail CI" +git push -u origin test-protection + +# Create PR +gh pr create --title "Test branch protection" --body "Should fail CI" + +# Verify you can't merge until CI passes +gh pr merge --auto # Should fail with "Required status check ci has not completed" +``` + +### 3. Document Required Checks + +In your README or CONTRIBUTING.md: + +```markdown +## Branch Protection + +The `main` branch is protected. All PRs must: + +1. Pass the `ci` status check (linting, tests, security scans) +2. Be up-to-date with main +3. Not be force-pushed + +Setup: `./scripts/setup-branch-protection.sh ambient-code reporters` +``` + +### 4. Sync Protection Across Repos + +Use a template configuration: + +```bash +# config/branch-protection.json +{ + "required_status_checks": { + "strict": true, + "contexts": ["ci", "security-scan"] + }, + "enforce_admins": false, + "required_pull_request_reviews": { + "required_approving_review_count": 0 + }, + "restrictions": null, + "allow_force_pushes": false, + "allow_deletions": false +} + +# Apply to repo +gh api repos/$OWNER/$REPO/branches/main/protection \ + --method PUT \ + --input config/branch-protection.json +``` + +### 5. Audit Protection Settings + +```bash +#!/bin/bash +# audit-protection.sh + +OWNER="ambient-code" + +gh repo list "$OWNER" --json name --jq '.[].name' | while read -r repo; do + echo "=== $repo ===" + gh api "repos/$OWNER/$repo/branches/main/protection" 2>/dev/null | jq '{ + required_checks: .required_status_checks.contexts, + enforce_admins: .enforce_admins.enabled, + required_reviews: .required_pull_request_reviews.required_approving_review_count + }' || echo "No protection" + echo "" +done +``` + +### 6. Update Protection When Adding Workflows + +When you add a new required workflow: + +```bash +# Added new security-scan.yml workflow +# Update branch protection to require it +gh api repos/$OWNER/$REPO/branches/main/protection \ + --method PUT \ + --field required_status_checks[contexts][]=ci \ + --field required_status_checks[contexts][]=security-scan \ # New check + --field required_status_checks[strict]=true \ + --field enforce_admins=false \ + --field restrictions=null +``` + +--- + +## Alternative: GitHub Web UI + +If you prefer manual setup: + +1. Go to `https://github.com/OWNER/REPO/settings/branches` +2. Click "Add rule" +3. Branch name pattern: `main` +4. Check "Require status checks to pass before merging" +5. Check "Require branches to be up to date before merging" +6. Search for and select your status checks (`ci`, etc.) +7. Optionally check "Require a pull request before merging" +8. Click "Create" + +**Pros:** +- Visual interface +- No CLI required +- See all options at once + +**Cons:** +- Slow and tedious +- Easy to miss settings +- Not reproducible across repos +- No audit trail + +--- + +## Terraform Alternative + +For infrastructure-as-code approach: + +```hcl +# github.tf +terraform { + required_providers { + github = { + source = "integrations/github" + version = "~> 5.0" + } + } +} + +provider "github" { + token = var.github_token +} + +resource "github_branch_protection" "main" { + repository_id = "reporters" + pattern = "main" + + required_status_checks { + strict = true + contexts = ["ci"] + } + + required_pull_request_reviews { + required_approving_review_count = 0 + dismiss_stale_reviews = false + } + + enforce_admins = false +} + +# Apply with: +# terraform init +# terraform apply +``` + +**Pros:** +- Infrastructure as code +- Version controlled +- Declarative +- Supports all GitHub resources + +**Cons:** +- Requires Terraform knowledge +- More complex setup +- Overkill for single repo + +--- + +## Related Patterns + +- [ci-pipeline-pattern.md](ci-pipeline-pattern.md) - CI pipeline that provides the required status checks +- [autonomous-quality-enforcement.md](autonomous-quality-enforcement.md) - Quality enforcement at multiple levels +- [gha-automation-patterns.md](gha-automation-patterns.md) - GitHub Actions automation workflows + +--- + +## Summary + +This pattern provides: + +- **Automated branch protection setup** via scripts +- **Consistency across repositories** with reusable configurations +- **Protection against broken code** reaching main +- **Support for both Pro and Free tiers** with rulesets fallback +- **Audit and bulk operation capabilities** + +Copy the setup script, customize for your organization, and protect your branches automatically. diff --git a/docs/patterns/ci-pipeline-pattern.md b/docs/patterns/ci-pipeline-pattern.md new file mode 100644 index 0000000..f8ab081 --- /dev/null +++ b/docs/patterns/ci-pipeline-pattern.md @@ -0,0 +1,685 @@ +# CI Pipeline Pattern + +**Pattern**: Comprehensive CI pipeline for Python projects with linting, testing, validation, and security scanning. + +**Problem**: Manual code quality checks are inconsistent, slow development, and allow bugs to reach production. Teams waste time catching issues that could be automated. + +**Solution**: GitHub Actions workflow that enforces code quality standards automatically on every push and PR. Runs black, isort, pylint, pytest, JSON validation, and security scanning. Provides helpful feedback when checks fail. + +--- + +## Quick Start (10 Minutes) + +Get a production-ready CI pipeline running in your Python project. + +### Prerequisites + +```bash +# Your project should have: +# - Python 3.11+ +# - requirements.txt for production dependencies +# - Some Python source code to check +``` + +### Step 1: Add Development Dependencies (2 min) + +Create `requirements-dev.txt`: + +```text +# Development dependencies +# Code formatting +black>=24.0.0 +isort>=5.13.0 + +# Linting +pylint>=3.0.0 + +# Testing +pytest>=8.0.0 +pytest-cov>=4.1.0 + +# Type checking (optional) +mypy>=1.8.0 +types-requests>=2.31.0 +``` + +### Step 2: Configure Black and Isort (1 min) + +Create `pyproject.toml`: + +```toml +[tool.black] +line-length = 120 +target-version = ['py311'] + +[tool.isort] +profile = "black" +line_length = 120 +``` + +### Step 3: Configure Pylint (1 min) + +Create `.pylintrc`: + +```ini +[MASTER] +ignore=.git,__pycache__,data + +[MESSAGES CONTROL] +disable= + missing-module-docstring, + too-few-public-methods, + line-too-long, + fixme, + broad-except + +[FORMAT] +max-line-length=120 + +[DESIGN] +max-args=7 +max-locals=20 +max-branches=15 +``` + +### Step 4: Create CI Workflow (3 min) + +Create `.github/workflows/ci-pipeline.yml`: + +```yaml +name: CI Pipeline + +on: + push: + branches: [main] + pull_request: + branches: [main] + workflow_dispatch: + +permissions: + contents: read + pull-requests: write + +jobs: + ci: + runs-on: ubuntu-latest + steps: + - uses: actions/checkout@v4 + + - name: Set up Python 3.11 + uses: actions/setup-python@v5 + with: + python-version: '3.11' + cache: 'pip' + + - name: Install dependencies + run: | + python -m pip install --upgrade pip + pip install -r requirements.txt + pip install -r requirements-dev.txt + + - name: Format check with Black + run: | + black --check --diff src/ + continue-on-error: false + + - name: Import sort check with isort + run: | + isort --check-only --diff src/ + continue-on-error: false + + - name: Lint with Pylint + run: | + pylint src/**/*.py --disable=C0114,C0115,C0116 --max-line-length=120 + continue-on-error: false + + - name: Run tests with pytest + if: hashFiles('tests/**/*.py') != '' + run: | + pytest tests/ -v --cov=src --cov-report=term-missing + continue-on-error: false + + - name: Check for common issues + run: | + echo "Checking for common Python issues..." + + # Check for print statements (should use logging) + if grep -rn "print(" src/ --include="*.py" | grep -v "#.*print"; then + echo "::warning::Consider using logging instead of print statements" + fi + + # Check for TODO/FIXME comments + if grep -rn "TODO\|FIXME" src/ --include="*.py"; then + echo "::notice::Found TODO/FIXME comments - consider addressing before merge" + fi + + # Check for bare except clauses + if grep -rn "except:" src/ --include="*.py"; then + echo "::warning::Bare except clauses found - specify exception types" + fi + + echo "Common issues check complete" + + - name: Security check for secrets + run: | + echo "Checking for potential secrets..." + + # Check for hardcoded API keys + if grep -rn -iE '(api[_-]?key|secret|token|password)\s*=\s*["\047][^"\047]{20,}' src/ --include="*.py"; then + echo "::error::Potential hardcoded credentials found" + exit 1 + fi + + echo "Security check passed" + + - name: Post quality report + if: github.event_name == 'pull_request' && failure() + uses: actions/github-script@v7 + with: + script: | + github.rest.issues.createComment({ + issue_number: context.issue.number, + owner: context.repo.owner, + repo: context.repo.repo, + body: `## Quality Check Failed + + One or more quality checks failed. Please review the workflow logs for details. + + **Common fixes:** + - Format code: \`black src/\` + - Sort imports: \`isort src/\` + - Fix linting issues: Review pylint output + - Run tests: \`pytest tests/\` + + Run all checks locally: + \`\`\`bash + black src/ + isort src/ + pylint src/**/*.py --max-line-length=120 + pytest tests/ -v + \`\`\` + + Push changes and the workflow will re-run automatically.` + }) +``` + +### Step 5: Test Locally (3 min) + +```bash +# Install dev dependencies +pip install -r requirements-dev.txt + +# Format code +black src/ + +# Sort imports +isort src/ + +# Run lint +pylint src/**/*.py --max-line-length=120 + +# Run tests (if you have them) +pytest tests/ -v + +# Commit and push +git add . +git commit -m "Add CI pipeline" +git push +``` + +**Done!** Your CI pipeline is now running on every push and PR. + +--- + +## Real-World Example: Reporters Repository + +The [ambient-code/reporters](https://github.com/ambient-code/reporters) repository uses this exact pattern: + +### File Structure + +``` +reporters/ +├── .github/ +│ └── workflows/ +│ └── ci-pipeline.yml # Main CI workflow +├── anthropic/ +│ ├── src/signal/ # Python source code +│ ├── requirements.txt # Production deps +│ ├── requirements-dev.txt # Dev deps +│ └── .pylintrc # Pylint config +├── pyproject.toml # Black/isort config +└── tests/ # Tests (pytest) +``` + +### What Gets Checked + +1. **Black**: Code formatting (120 char line length) +2. **isort**: Import organization (black profile) +3. **Pylint**: Code quality and potential bugs +4. **pytest**: Unit tests with coverage reporting +5. **JSON validation**: Data file structure checks +6. **Security**: Hardcoded credential detection +7. **Common issues**: Print statements, bare excepts, TODOs + +### Results + +- All PRs must pass CI before merge +- Automatic feedback on failures with fix suggestions +- Consistent code quality across all contributions +- No hardcoded secrets in code +- Zero manual quality checking + +--- + +## Customization Guide + +### Adjust Paths for Your Project + +Update paths in the workflow to match your structure: + +```yaml +# If your code is in different directories +- name: Format check with Black + run: black --check --diff app/ lib/ utils/ + +# If you have multiple requirement files +- name: Install dependencies + run: | + pip install -r requirements/base.txt + pip install -r requirements/dev.txt +``` + +### Add JSON Validation + +For projects with JSON data files: + +```yaml +- name: Validate JSON structure + run: | + python -c "import json; data = json.load(open('data/config.json')); print(f'Validated {len(data)} items')" +``` + +### Add Type Checking with mypy + +```yaml +- name: Type check with mypy + run: | + mypy src/ --ignore-missing-imports + continue-on-error: false +``` + +### Add Security Scanning + +For more comprehensive security scanning: + +```yaml +- name: Security scan with bandit + run: | + pip install bandit + bandit -r src/ -ll +``` + +### Adjust Pylint Strictness + +Edit `.pylintrc` to be more or less strict: + +```ini +# More strict - fewer disabled rules +[MESSAGES CONTROL] +disable= + missing-module-docstring, + fixme + +# Less strict - more disabled rules +[MESSAGES CONTROL] +disable= + missing-module-docstring, + missing-class-docstring, + missing-function-docstring, + too-few-public-methods, + too-many-arguments, + too-many-locals, + line-too-long, + fixme, + broad-except, + invalid-name +``` + +### Skip Tests If They Don't Exist + +The workflow already handles this: + +```yaml +- name: Run tests with pytest + if: hashFiles('tests/**/*.py') != '' # Only run if tests exist + run: pytest tests/ -v +``` + +--- + +## Gotchas and Limitations + +### 1. Black and Pylint Conflicts + +**Issue**: Pylint's `line-too-long` check conflicts with black's formatting. + +**Solution**: Disable `line-too-long` in `.pylintrc`: + +```ini +[MESSAGES CONTROL] +disable=line-too-long +``` + +And configure both tools to use the same line length: + +```toml +# pyproject.toml +[tool.black] +line-length = 120 +``` + +```ini +# .pylintrc +[FORMAT] +max-line-length=120 +``` + +### 2. isort and Black Import Ordering + +**Issue**: isort and black can disagree on import formatting. + +**Solution**: Configure isort to use black profile: + +```toml +[tool.isort] +profile = "black" +``` + +### 3. Pip Cache Not Working + +**Issue**: Dependencies install slowly every run. + +**Solution**: Use actions/setup-python with cache: + +```yaml +- name: Set up Python 3.11 + uses: actions/setup-python@v5 + with: + python-version: '3.11' + cache: 'pip' # Enable pip caching +``` + +### 4. Tests Failing in CI But Pass Locally + +**Issue**: Different environments cause test failures. + +**Solution**: Pin all dependency versions: + +```bash +# Generate exact versions +pip freeze > requirements.txt + +# Or use poetry/pipenv for deterministic installs +``` + +### 5. Security Regex False Positives + +**Issue**: Legitimate code triggers hardcoded secret detection. + +**Solution**: Refine the regex pattern or add exceptions: + +```bash +# Exclude test files +if grep -rn -iE '(api[_-]?key|secret)' src/ --include="*.py" --exclude="*test*"; then + echo "::error::Potential hardcoded credentials found" + exit 1 +fi +``` + +### 6. Workflow Permissions Issues + +**Issue**: Cannot post comments on PRs. + +**Solution**: Add proper permissions: + +```yaml +permissions: + contents: read + pull-requests: write # Required for posting comments +``` + +### 7. Continue-on-error Gotcha + +**Issue**: Setting `continue-on-error: true` makes failures non-blocking. + +**Solution**: Use `continue-on-error: false` (or omit it) for required checks: + +```yaml +- name: Format check with Black + run: black --check --diff src/ + continue-on-error: false # Fail the workflow if black check fails +``` + +--- + +## Best Practices + +### 1. Run Checks Locally First + +Add a convenience script: + +```bash +#!/bin/bash +# scripts/check.sh +set -e + +echo "Running CI checks locally..." + +black --check --diff src/ +isort --check-only --diff src/ +pylint src/**/*.py --max-line-length=120 +pytest tests/ -v + +echo "All checks passed!" +``` + +### 2. Auto-fix Before Committing + +Add a pre-commit hook: + +```bash +#!/bin/bash +# .git/hooks/pre-commit + +# Auto-format +black src/ +isort src/ + +# Re-stage formatted files +git add -u + +# Run checks +pylint src/**/*.py --max-line-length=120 +pytest tests/ -v +``` + +### 3. Use Same Config Everywhere + +One `.pylintrc` for: +- Local development +- CI pipeline +- Pre-commit hooks + +One `pyproject.toml` for: +- Black formatting +- isort configuration +- Project metadata + +### 4. Cache Dependencies Aggressively + +```yaml +- uses: actions/cache@v4 + with: + path: ~/.cache/pip + key: ${{ runner.os }}-pip-${{ hashFiles('**/requirements*.txt') }} + restore-keys: | + ${{ runner.os }}-pip- +``` + +### 5. Fail Fast for Critical Issues + +Order checks from fastest/most critical to slowest: + +```yaml +1. Security checks (fast, critical) +2. Black formatting (fast, easy fix) +3. isort (fast, easy fix) +4. Pylint (medium, may need refactoring) +5. Tests (slow, may require investigation) +``` + +### 6. Provide Helpful Error Messages + +Use GitHub Actions annotations: + +```bash +echo "::error::Hardcoded credentials found in src/api.py:42" +echo "::warning::Consider using logging instead of print" +echo "::notice::Found 3 TODO comments to address" +``` + +### 7. Document Expected Local Workflow + +Add to your README: + +```markdown +## Development Workflow + +Before committing: +1. `black src/` - Format code +2. `isort src/` - Sort imports +3. `pylint src/**/*.py` - Check for issues +4. `pytest tests/` - Run tests + +Or use the convenience script: `./scripts/check.sh` +``` + +--- + +## Manual Setup Alternative + +If you prefer to set up CI manually: + +### Using GitHub CLI + +```bash +# Create workflow directory +mkdir -p .github/workflows + +# Copy workflow from this pattern +# (paste the YAML from Step 4 above) + +# Add required files +touch requirements-dev.txt .pylintrc pyproject.toml + +# Commit and push +git add .github/ requirements-dev.txt .pylintrc pyproject.toml +git commit -m "Add CI pipeline" +git push +``` + +### Using GitHub Web UI + +1. Go to your repository on GitHub +2. Click "Actions" tab +3. Click "New workflow" +4. Click "set up a workflow yourself" +5. Paste the workflow YAML from Step 4 +6. Commit directly to main or create a PR + +--- + +## Integration with Branch Protection + +Once CI is working, require it to pass before merging: + +1. Go to Settings > Branches +2. Click "Add rule" for `main` branch +3. Enable "Require status checks to pass before merging" +4. Select `ci` (the job name from the workflow) +5. Save changes + +Now all PRs must pass CI before they can be merged. + +See [branch-protection-automation.md](branch-protection-automation.md) for automated setup. + +--- + +## Cost and Performance + +### GitHub Actions Usage + +- **Free tier**: 2,000 minutes/month for private repos +- **Typical run time**: 2-5 minutes per workflow +- **Optimization**: Use pip caching to reduce to 1-2 minutes + +### Compute Resources + +```yaml +# Default: Standard 2-core Ubuntu runner +runs-on: ubuntu-latest + +# For faster runs (if you have GitHub Pro/Enterprise): +runs-on: ubuntu-latest-4-core +``` + +--- + +## Troubleshooting + +### CI Passes Locally But Fails in GitHub Actions + +**Check:** +1. Python version matches (3.11 in both places) +2. Dependencies are pinned to same versions +3. Environment variables are set correctly +4. File paths are correct (case-sensitive on Linux) + +### Workflow Not Triggering + +**Check:** +1. Workflow file is in `.github/workflows/` +2. YAML syntax is valid (use `yamllint`) +3. Trigger conditions match your branch names +4. Repository Actions are enabled in Settings + +### Permission Denied Errors + +**Check:** +1. Workflow has correct permissions block +2. GITHUB_TOKEN has required scopes +3. Protected branch rules allow workflow to run + +--- + +## Related Patterns + +- [dev-dependencies-pattern.md](dev-dependencies-pattern.md) - Managing development dependencies +- [branch-protection-automation.md](branch-protection-automation.md) - Automating branch protection setup +- [autonomous-quality-enforcement.md](autonomous-quality-enforcement.md) - Validation loops and git hooks +- [testing-patterns.md](testing-patterns.md) - Test organization and best practices +- [security-patterns.md](security-patterns.md) - Security scanning and hardening + +--- + +## Summary + +This CI pipeline pattern provides: + +- **Automated quality enforcement** on every push and PR +- **Fast feedback** with helpful error messages +- **Security scanning** to prevent credential leaks +- **Consistent standards** across all contributors +- **Low maintenance** with sensible defaults + +Copy the Quick Start steps, customize for your project, and ship with confidence. diff --git a/docs/patterns/dev-dependencies-pattern.md b/docs/patterns/dev-dependencies-pattern.md new file mode 100644 index 0000000..c41f630 --- /dev/null +++ b/docs/patterns/dev-dependencies-pattern.md @@ -0,0 +1,805 @@ +# Development Dependencies Pattern + +**Pattern**: Separate development tooling from production dependencies for cleaner, faster, and more secure Python projects. + +**Problem**: Mixing dev tools (linters, formatters, test frameworks) with production dependencies creates bloated deployments, security vulnerabilities, and dependency conflicts. Production images include unnecessary tools that slow down builds and increase attack surface. + +**Solution**: Use separate `requirements-dev.txt` for development-only dependencies, with unified configuration in `pyproject.toml` and `.pylintrc`. Development environment has full tooling, production environment is minimal and secure. + +--- + +## Quick Start (5 Minutes) + +Set up proper dependency separation for a Python project. + +### Step 1: Create requirements-dev.txt (2 min) + +Create a new file `requirements-dev.txt`: + +```text +# Development dependencies +# Keep these separate from requirements.txt + +# Code formatting +black>=24.0.0 +isort>=5.13.0 + +# Linting +pylint>=3.0.0 + +# Testing +pytest>=8.0.0 +pytest-cov>=4.1.0 + +# Type checking +mypy>=1.8.0 +types-requests>=2.31.0 +``` + +### Step 2: Create pyproject.toml (1 min) + +Create `pyproject.toml` for tool configuration: + +```toml +[tool.black] +line-length = 120 +target-version = ['py311'] + +[tool.isort] +profile = "black" +line_length = 120 + +[tool.pytest.ini_options] +testpaths = ["tests"] +python_files = ["test_*.py"] +python_functions = ["test_*"] +addopts = "-v --cov=src --cov-report=term-missing" +``` + +### Step 3: Create .pylintrc (1 min) + +Create `.pylintrc` with sensible defaults: + +```ini +[MASTER] +ignore=.git,__pycache__,data + +[MESSAGES CONTROL] +disable= + missing-module-docstring, + too-few-public-methods, + line-too-long, + fixme, + broad-except + +[FORMAT] +max-line-length=120 + +[DESIGN] +max-args=7 +max-locals=20 +max-branches=15 +``` + +### Step 4: Update Your Workflow (1 min) + +For local development: + +```bash +# Install production dependencies +pip install -r requirements.txt + +# Install dev dependencies +pip install -r requirements-dev.txt + +# Now you can use all tools +black src/ +isort src/ +pylint src/**/*.py +pytest tests/ +``` + +For CI/CD (update your workflow): + +```yaml +- name: Install dependencies + run: | + pip install -r requirements.txt + pip install -r requirements-dev.txt # Only in CI, not production +``` + +For production deployment: + +```dockerfile +# Dockerfile - only install production deps +RUN pip install -r requirements.txt +# requirements-dev.txt is NOT installed +``` + +**Done!** You now have clean separation between dev and production dependencies. + +--- + +## Real-World Example: Reporters Repository + +The reporters repository demonstrates this pattern perfectly: + +### File Structure + +``` +reporters/ +├── anthropic/ +│ ├── requirements.txt # Production only (2 packages) +│ ├── requirements-dev.txt # Development tools (6 packages) +│ └── .pylintrc # Pylint configuration +├── pyproject.toml # Black/isort configuration +└── .github/workflows/ + └── ci-pipeline.yml # Installs both requirements files +``` + +### Production Dependencies (`anthropic/requirements.txt`) + +```text +anthropic>=0.40.0 +requests>=2.31.0 +``` + +**Only 2 packages!** The absolute minimum needed to run the application. + +### Development Dependencies (`anthropic/requirements-dev.txt`) + +```text +# Code formatting +black>=24.0.0 +isort>=5.13.0 + +# Linting +pylint>=3.0.0 + +# Testing +pytest>=8.0.0 +pytest-cov>=4.1.0 + +# Type checking +mypy>=1.8.0 +types-requests>=2.31.0 +``` + +**6 packages** for development tooling. None of these ship to production. + +### Unified Configuration (`pyproject.toml`) + +```toml +[tool.black] +line-length = 120 +target-version = ['py311'] + +[tool.isort] +profile = "black" +line_length = 120 +``` + +Single source of truth for black and isort settings. + +### Pylint Configuration (`anthropic/.pylintrc`) + +```ini +[MASTER] +ignore=.git,__pycache__,data + +[MESSAGES CONTROL] +disable= + missing-module-docstring, + too-few-public-methods, + line-too-long, + fixme, + broad-except + +[FORMAT] +max-line-length=120 + +[DESIGN] +max-args=7 +max-locals=20 +max-branches=15 +``` + +Reasonable defaults that work for most projects. + +### CI Pipeline Integration + +The CI workflow installs both: + +```yaml +- name: Install dependencies + run: | + python -m pip install --upgrade pip + pip install -r anthropic/requirements.txt + pip install -r anthropic/requirements-dev.txt +``` + +But production deploys only use `requirements.txt`. + +--- + +## Benefits + +### 1. Smaller Production Images + +**Before (mixing dependencies):** +```dockerfile +FROM python:3.11-slim +COPY requirements.txt . +RUN pip install -r requirements.txt # 50+ packages including pytest, black, etc. +``` + +Image size: 850 MB + +**After (separated dependencies):** +```dockerfile +FROM python:3.11-slim +COPY requirements.txt . +RUN pip install -r requirements.txt # Only 2-10 packages +``` + +Image size: 250 MB + +**Savings: 600 MB (71% reduction)** + +### 2. Faster Deployment + +| Stage | Before | After | Improvement | +|-------|--------|-------|-------------| +| Pip install | 45s | 8s | 82% faster | +| Image build | 2m 30s | 45s | 70% faster | +| Image push | 1m 15s | 20s | 73% faster | + +### 3. Reduced Security Surface + +Fewer dependencies = fewer CVEs to track: + +```bash +# Before: 50 packages to monitor for security issues +pip list | wc -l +# 50 + +# After: 2 packages to monitor +pip list | wc -l +# 2 +``` + +### 4. Cleaner Dependency Graph + +Production apps don't need: +- `pytest` and testing frameworks +- `black`, `isort` code formatters +- `pylint`, `mypy` static analyzers +- `pytest-cov` coverage tools + +These are development tools, not runtime dependencies. + +--- + +## Configuration Guide + +### Black Configuration + +```toml +[tool.black] +line-length = 120 # Match your team's preference (88 is default) +target-version = ['py311'] # Python version(s) to target +skip-string-normalization = false # Normalize quote styles +extend-exclude = ''' +/( + \.eggs + | \.git + | \.mypy_cache + | build + | dist +)/ +''' +``` + +### isort Configuration + +```toml +[tool.isort] +profile = "black" # Compatible with black +line_length = 120 # Match black's line length +force_single_line = false # Allow multiple imports per line +known_first_party = ["myapp"] # Your package name +known_third_party = ["anthropic", "requests"] +sections = ["FUTURE", "STDLIB", "THIRDPARTY", "FIRSTPARTY", "LOCALFOLDER"] +``` + +### Pylint Configuration + +```ini +[MASTER] +ignore=.git,__pycache__,data,venv,.venv +jobs=4 # Parallel processing + +[MESSAGES CONTROL] +# Start with these disabled, enable as you tighten standards +disable= + missing-module-docstring, # Require module docstrings + missing-class-docstring, # Require class docstrings + missing-function-docstring, # Require function docstrings + too-few-public-methods, # Allow simple classes + too-many-arguments, # Allow complex functions + too-many-locals, # Allow complex functions + too-many-branches, # Allow complex logic + line-too-long, # Handled by black + fixme, # Allow TODO comments + broad-except, # Allow except Exception + invalid-name # Allow short variable names + +[FORMAT] +max-line-length=120 + +[DESIGN] +max-args=7 # Increase if you have complex APIs +max-locals=20 # Increase for complex functions +max-branches=15 # Increase for complex logic +max-statements=50 +``` + +### Pytest Configuration + +```toml +[tool.pytest.ini_options] +testpaths = ["tests"] +python_files = ["test_*.py", "*_test.py"] +python_functions = ["test_*"] +python_classes = ["Test*"] +addopts = """ + -v + --strict-markers + --cov=src + --cov-report=term-missing + --cov-report=html + --cov-fail-under=80 +""" +markers = [ + "slow: marks tests as slow (deselect with '-m \"not slow\"')", + "integration: marks tests as integration tests", +] +``` + +### Mypy Configuration + +```toml +[tool.mypy] +python_version = "3.11" +warn_return_any = true +warn_unused_configs = true +disallow_untyped_defs = true +ignore_missing_imports = true +``` + +--- + +## Best Practices + +### 1. Pin Major Versions, Allow Minor Updates + +```text +# Good: Allow patches and minor updates +black>=24.0.0 +pytest>=8.0.0 + +# Avoid: Unpinned (can break) +black +pytest + +# Avoid: Exact pins (miss security patches) +black==24.1.1 +pytest==8.0.2 +``` + +### 2. Group Dependencies Logically + +```text +# Development dependencies + +# Code formatting +black>=24.0.0 +isort>=5.13.0 + +# Linting +pylint>=3.0.0 +flake8>=7.0.0 + +# Testing +pytest>=8.0.0 +pytest-cov>=4.1.0 +pytest-asyncio>=0.23.0 + +# Type checking +mypy>=1.8.0 +types-requests>=2.31.0 + +# Development tools +ipython>=8.12.0 +``` + +### 3. Keep pyproject.toml Simple + +Only configure tools you actually use: + +```toml +# Minimal configuration +[tool.black] +line-length = 120 +target-version = ['py311'] + +[tool.isort] +profile = "black" +line_length = 120 +``` + +Don't over-configure. Start simple, add options as needed. + +### 4. Use Same Config in All Environments + +```bash +# Local development +black --config pyproject.toml src/ + +# CI/CD +black --config pyproject.toml src/ + +# Pre-commit hook +black --config pyproject.toml src/ +``` + +One config file, used everywhere. + +### 5. Document Required Dev Setup + +Add to your README: + +```markdown +## Development Setup + +1. Install production dependencies: + ```bash + pip install -r requirements.txt + ``` + +2. Install development dependencies: + ```bash + pip install -r requirements-dev.txt + ``` + +3. Verify setup: + ```bash + black --version + pytest --version + pylint --version + ``` + +## Code Quality Checks + +Run before committing: + +```bash +black src/ +isort src/ +pylint src/**/*.py +pytest tests/ +``` +``` + +### 6. Separate by Environment When Needed + +For complex projects: + +``` +requirements/ +├── base.txt # Core production dependencies +├── production.txt # Production-only (gunicorn, etc.) +├── development.txt # Development tools +└── testing.txt # Testing in CI +``` + +```text +# requirements/development.txt +-r base.txt +black>=24.0.0 +pytest>=8.0.0 +``` + +--- + +## Gotchas and Limitations + +### 1. Black and Pylint Line Length Mismatch + +**Problem**: Black formats to 120 chars, pylint complains. + +**Solution**: Disable `line-too-long` in pylint, set both to same length: + +```ini +# .pylintrc +[MESSAGES CONTROL] +disable=line-too-long + +[FORMAT] +max-line-length=120 +``` + +```toml +# pyproject.toml +[tool.black] +line-length = 120 +``` + +### 2. isort and Black Import Conflicts + +**Problem**: isort and black disagree on import formatting. + +**Solution**: Configure isort to use black profile: + +```toml +[tool.isort] +profile = "black" +``` + +This makes isort compatible with black's formatting. + +### 3. Development Tools in Production + +**Problem**: Accidentally installing dev dependencies in production. + +**Solution**: Use separate Docker build stages: + +```dockerfile +# Build stage - includes dev dependencies +FROM python:3.11-slim AS builder +COPY requirements.txt requirements-dev.txt ./ +RUN pip install -r requirements.txt -r requirements-dev.txt +RUN pytest tests/ # Run tests during build + +# Production stage - only production dependencies +FROM python:3.11-slim +COPY requirements.txt ./ +RUN pip install -r requirements.txt +COPY src/ ./src/ +CMD ["python", "-m", "src.main"] +``` + +### 4. Conflicting Dependencies + +**Problem**: Dev tools require different versions than production. + +**Solution**: Use virtual environments or Docker to isolate: + +```bash +# Production environment +python -m venv venv-prod +source venv-prod/bin/activate +pip install -r requirements.txt + +# Development environment +python -m venv venv-dev +source venv-dev/bin/activate +pip install -r requirements.txt -r requirements-dev.txt +``` + +### 5. CI Installing Wrong Requirements + +**Problem**: CI only installs production requirements, tests fail. + +**Solution**: Explicitly install both in CI: + +```yaml +- name: Install dependencies + run: | + pip install -r requirements.txt + pip install -r requirements-dev.txt # Don't forget this! +``` + +### 6. pyproject.toml vs setup.py + +**Problem**: Old projects use `setup.py` for everything. + +**Solution**: Migrate gradually: + +```python +# setup.py (old way) +setup( + name="myapp", + install_requires=["requests", "anthropic"], + extras_require={ + "dev": ["pytest", "black", "pylint"] + } +) + +# Install with: pip install -e ".[dev]" +``` + +```toml +# pyproject.toml (new way) +[project] +name = "myapp" +dependencies = ["requests>=2.31.0", "anthropic>=0.40.0"] + +[project.optional-dependencies] +dev = ["pytest>=8.0.0", "black>=24.0.0", "pylint>=3.0.0"] + +# Install with: pip install -e ".[dev]" +``` + +But for simplicity, `requirements.txt` + `requirements-dev.txt` works great. + +--- + +## Alternatives + +### Poetry + +Modern dependency management with lock files: + +```toml +# pyproject.toml +[tool.poetry.dependencies] +python = "^3.11" +anthropic = "^0.40.0" +requests = "^2.31.0" + +[tool.poetry.group.dev.dependencies] +black = "^24.0.0" +pytest = "^8.0.0" +pylint = "^3.0.0" +``` + +```bash +# Install all dependencies +poetry install + +# Install only production dependencies +poetry install --only main +``` + +**Pros:** +- Lock files for reproducible builds +- Dependency resolution built-in +- Virtual env management + +**Cons:** +- More complex than requirements.txt +- Requires poetry installed +- Learning curve for team + +### Pipenv + +Another lock file based approach: + +```toml +# Pipfile +[packages] +anthropic = ">=0.40.0" +requests = ">=2.31.0" + +[dev-packages] +black = ">=24.0.0" +pytest = ">=8.0.0" +``` + +```bash +# Install all +pipenv install --dev + +# Install only production +pipenv install +``` + +### Conda + +For data science projects: + +```yaml +# environment.yml +name: myapp +dependencies: + - python=3.11 + - anthropic>=0.40.0 + - requests>=2.31.0 + - pytest # dev + - black # dev +``` + +--- + +## Migration Path + +### From Mixed Dependencies + +**Before:** +```text +# requirements.txt (everything mixed) +anthropic>=0.40.0 +requests>=2.31.0 +pytest>=8.0.0 # Dev only +black>=24.0.0 # Dev only +pylint>=3.0.0 # Dev only +``` + +**After:** +```text +# requirements.txt (production only) +anthropic>=0.40.0 +requests>=2.31.0 + +# requirements-dev.txt (development only) +black>=24.0.0 +pytest>=8.0.0 +pylint>=3.0.0 +``` + +### Migration Steps + +1. Create `requirements-dev.txt` +2. Move dev tools from `requirements.txt` to `requirements-dev.txt` +3. Update CI to install both files +4. Update Dockerfile to install only `requirements.txt` +5. Update README with new setup instructions +6. Test in CI and production + +--- + +## Verification + +### Check Separation is Working + +```bash +# Production environment +python -m venv venv-prod +source venv-prod/bin/activate +pip install -r requirements.txt +python -c "import pytest" # Should fail - pytest not installed + +# Development environment +python -m venv venv-dev +source venv-dev/bin/activate +pip install -r requirements.txt -r requirements-dev.txt +python -c "import pytest" # Should succeed +``` + +### Measure Improvements + +```bash +# Production image size +docker build -f Dockerfile.prod -t myapp:prod . +docker images myapp:prod + +# Development image size +docker build -f Dockerfile.dev -t myapp:dev . +docker images myapp:dev + +# Compare +echo "Production should be significantly smaller" +``` + +--- + +## Related Patterns + +- [ci-pipeline-pattern.md](ci-pipeline-pattern.md) - CI pipeline that uses dev dependencies +- [autonomous-quality-enforcement.md](autonomous-quality-enforcement.md) - Using dev tools in validation loops +- [testing-patterns.md](testing-patterns.md) - pytest configuration and best practices + +--- + +## Summary + +This pattern provides: + +- **Clean separation** between production and development dependencies +- **Smaller, faster, more secure** production deployments +- **Unified configuration** across all environments +- **Easy to understand** for new team members +- **Industry standard** approach used by top Python projects + +Start with the Quick Start guide, customize the configs for your project, and enjoy cleaner dependencies.