From 3e99d6a358838e59f93e42ee9472df481c349db8 Mon Sep 17 00:00:00 2001 From: crypticsaiyan <186129212+crypticsaiyan@users.noreply.github.com> Date: Sun, 14 Dec 2025 20:21:54 +0530 Subject: [PATCH] docs: add comprehensive documentation for InFoundry - Add docs/architecture.md with system architecture overview, component diagrams, data flow documentation, and integration points - Add docs/blueprint.md with complete system design, project structure, component specifications, API contracts, and data schemas - Add docs/runlocally.md with step-by-step local development guide, service setup instructions, and troubleshooting tips - Add orchestrator/kestra_pipelines/README.md documenting all 9 pipeline steps and AI agent capabilities - Add ui/.env.example for environment configuration - Add CHANGELOG.md, ROADMAP.md, and SECURITY.md for project governance - Add GitHub issue templates for bug reports and feature requests --- .github/ISSUE_TEMPLATE/bug_report.md | 38 + .github/ISSUE_TEMPLATE/feature_request.md | 20 + CHANGELOG.md | 29 + README.md | 2 + ROADMAP.md | 17 + SECURITY.md | 18 + docs/BLUEPRINT.md | 779 ++++++++++-------- docs/architecture.md | 289 +++++-- docs/kestra-architecture.md | 210 ----- docs/runlocally.md | 543 +++++++++++- infoundry-mcp-server/README.md | 5 +- .../kestra_pipelines/07-create-pr.yaml | 5 +- orchestrator/kestra_pipelines/README.md | 88 ++ ui/.env.example | 12 + ui/README.md | 16 +- 15 files changed, 1455 insertions(+), 616 deletions(-) create mode 100644 .github/ISSUE_TEMPLATE/bug_report.md create mode 100644 .github/ISSUE_TEMPLATE/feature_request.md create mode 100644 CHANGELOG.md create mode 100644 ROADMAP.md create mode 100644 SECURITY.md delete mode 100644 docs/kestra-architecture.md create mode 100644 orchestrator/kestra_pipelines/README.md create mode 100644 ui/.env.example diff --git a/.github/ISSUE_TEMPLATE/bug_report.md b/.github/ISSUE_TEMPLATE/bug_report.md new file mode 100644 index 0000000..37800b3 --- /dev/null +++ b/.github/ISSUE_TEMPLATE/bug_report.md @@ -0,0 +1,38 @@ +--- +name: Bug report +about: Create a report to help us improve +title: '' +labels: bug +assignees: '' + +--- + +**Describe the bug** +A clear and concise description of what the bug is. + +**To Reproduce** +Steps to reproduce the behavior: +1. Go to '...' +2. Click on '...' +3. Scroll down to '...' +4. See error + +**Expected behavior** +A clear and concise description of what you expected to happen. + +**Screenshots** +If applicable, add screenshots to help explain your problem. + +**Desktop (please complete the following information):** + - OS: [e.g. iOS] + - Browser [e.g. chrome, safari] + - Version [e.g. 22] + +**Smartphone (please complete the following information):** + - Device: [e.g. iPhone6] + - OS: [e.g. iOS8.1] + - Browser [e.g. stock browser, safari] + - Version [e.g. 22] + +**Additional context** +Add any other context about the problem here. diff --git a/.github/ISSUE_TEMPLATE/feature_request.md b/.github/ISSUE_TEMPLATE/feature_request.md new file mode 100644 index 0000000..11fc491 --- /dev/null +++ b/.github/ISSUE_TEMPLATE/feature_request.md @@ -0,0 +1,20 @@ +--- +name: Feature request +about: Suggest an idea for this project +title: '' +labels: enhancement +assignees: '' + +--- + +**Is your feature request related to a problem? Please describe.** +A clear and concise description of what the problem is. Ex. I'm always frustrated when [...] + +**Describe the solution you'd like** +A clear and concise description of what you want to happen. + +**Describe alternatives you've considered** +A clear and concise description of any alternative solutions or features you've considered. + +**Additional context** +Add any other context or screenshots about the feature request here. diff --git a/CHANGELOG.md b/CHANGELOG.md new file mode 100644 index 0000000..9e1d904 --- /dev/null +++ b/CHANGELOG.md @@ -0,0 +1,29 @@ +# Changelog + +All notable changes to this project will be documented in this file. + +The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.0.0/), +and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html). + +## [Unreleased] + +### Added +- `SECURITY.md` for security policy. +- Issue templates for bug reports and feature requests. +- `CHANGELOG.md` (this file). +- `ROADMAP.md` for project roadmap. + +### Changed +- None + +### Deprecated +- None + +### Removed +- None + +### Fixed +- None + +### Security +- None diff --git a/README.md b/README.md index 1c27ea8..1daa22d 100644 --- a/README.md +++ b/README.md @@ -37,6 +37,8 @@ cd infoundry-mcp-server && npm install && npm run build && cd .. ### Run the UI ```bash cd ui +cp .env.example .env.local +# Edit .env.local with your Kestra credentials if needed npm run dev # Open http://localhost:3000 ``` diff --git a/ROADMAP.md b/ROADMAP.md new file mode 100644 index 0000000..40ca853 --- /dev/null +++ b/ROADMAP.md @@ -0,0 +1,17 @@ +# Roadmap + +This document outlines the high-level roadmap for InFoundry. + +## 2026 Q1 +- [ ] Implement enhanced authentication. +- [ ] Add support for more cloud providers. +- [ ] Improve visualization capabilities. + +## 2026 Q2 +- [ ] Automated compliance auditing. +- [ ] Integration with major CI/CD platforms. +- [ ] Advanced cost estimation features. + +## Future Ideas +- AI-driven architecture optimization. +- Collaborative real-time editing. diff --git a/SECURITY.md b/SECURITY.md new file mode 100644 index 0000000..995a6b8 --- /dev/null +++ b/SECURITY.md @@ -0,0 +1,18 @@ +# Security Policy + +## Supported Versions + +Use this section to tell people about which versions of your project are currently being supported with security updates. + +| Version | Supported | +| ------- | ------------------ | +| 1.0.x | :white_check_mark: | +| < 1.0 | :x: | + +## Reporting a Vulnerability + +Use this section to tell people how to report a vulnerability. + +Tell them where to go, how often they can expect to get an update on a reported vulnerability, what to expect if the vulnerability is accepted or rejected, etc. + +To report a vulnerability, please email [security@infoundry.com](mailto:security@infoundry.com). diff --git a/docs/BLUEPRINT.md b/docs/BLUEPRINT.md index 50e98b2..fb08aec 100644 --- a/docs/BLUEPRINT.md +++ b/docs/BLUEPRINT.md @@ -1,365 +1,502 @@ -An app that generates cloud architecture visually - -This is no longer an infra-optimizer or PR generator. -It is now an AI-powered visual cloud architecture generator that: - -Reads a repo or user description - -Understands the system components - -Designs a cloud architecture diagram automatically - -Renders it visually in the UI as a graph or canvas - -Allows users to edit the diagram - -Exports IaC templates (Terraform/Helm) generated by Cline - -Uses Kestra for summarizing metrics or repo profile - -Uses Oumi for deciding the best architecture patterns - -Uses CodeRabbit for validating IaC and architecture quality - -Everything below is fully rewritten for this new objective. - -InFoundry Architect Blueprint (copy–paste friendly) - -A complete plan for building an AI-powered visual cloud architecture generator. - -Table of contents - -What InFoundry Architect does (short) - -High-level architecture (text diagram) - -Component responsibilities - -Repo structure (copyable) - -Development stack & prerequisites - -Step-by-step 7-day build plan - -Example configs & code snippets - -CI/CD & quality gates - -Open-source practices & contributor workflow - -Security & safety checklist - -Metrics & evaluation (what judges want) - -Demo script (3–5 minutes) - -Hardest technical challenges + mitigations - -Stretch goals & future roadmap - -Quick start checklist - -1) What InFoundry Architect does (short) - -Reads codebases OR text descriptions. - -Identifies services, databases, queues, API boundaries. - -Uses Oumi to choose the best cloud architecture pattern (serverless, ECS, EKS, Lambda, Pub/Sub, etc.). - -Generates a visual cloud architecture diagram (nodes + edges). - -Users can drag, edit, remove, or add components. - -Cline converts the diagram into IaC templates. - -CodeRabbit reviews the IaC for safety & correctness. - -Vercel UI renders the entire visual architecture and exports. - -2) High-level architecture (text diagram) - [Input: Repo OR Prompt] - | - v - Repo Analyzer - (detect services, ports, DBs) - | - v - Kestra Summarizer - (optional: summarize metrics or usage) - | - v - Oumi Architecture Brain - (select patterns, best practices, topology) - | - v - Architecture Graph Generator - (nodes, edges, relations, metadata) - | - v - Visual Diagram Renderer (Next.js) - (interactive drag-and-drop UI) - | - +---------+---------+ - | | - v v - Export IaC (Cline) Manual Edits by User - | - v - CodeRabbit Review → Final IaC bundle - -3) Component responsibilities -Repo Analyzer - -Detect services (auth, payments, frontend, worker). - -Detect databases (Postgres/MySQL/Mongo). - -Detect queues, caches, API endpoints. - -Output a structured service graph. - -Kestra - -Summaries runtime or resource usage (optional). - -Useful if patterns depend on cost/traffic. - -Oumi (Architecture Model) - -Decides architecture style: - -Serverless - -Microservices on ECS - -Kubernetes - -Event-driven - -Monolith + DB - -Picks best-fit patterns. - -Outputs architecture actions. - -Architecture Graph Generator - -Converts Oumi output into JSON graph: - -nodes: [{id, type, name, props}], -edges: [{source, target, type}] - -Next.js Visual Editor UI - -Renders the graph visually. - -Users can drag/drop/edit components. +# InFoundry Blueprint + +This document provides the complete system design blueprint for InFoundry, including project structure, component specifications, and implementation details. + +## Project Structure + +``` +infoundry/ +├── docs/ # Documentation +│ ├── architecture.md # System architecture overview +│ ├── blueprint.md # This file - system design +│ └── runlocally.md # Local development guide +│ +├── examples/ # Sample data files +│ ├── sample_architecture_plan.json +│ ├── sample_deploy_result.json +│ ├── sample_graph.json +│ ├── sample_pr_result.json +│ ├── sample_service_profile.json +│ ├── sample_telemetry_summary.json +│ └── sft_examples.jsonl # Training examples +│ +├── infoundry-mcp-server/ # Cline MCP integration +│ ├── src/ +│ │ └── index.ts # MCP server implementation +│ ├── package.json +│ └── tsconfig.json +│ +├── orchestrator/ # Kestra pipeline definitions +│ └── kestra_pipelines/ +│ ├── 00-end-to-end.yaml # Master pipeline +│ ├── 01-ingest-repo.yaml +│ ├── 02-ingest-telemetry.yaml +│ ├── 03-propose-architecture.yaml +│ ├── 04-render-graph.yaml +│ ├── 05-generate-iac.yaml +│ ├── 06-validate-iac.yaml +│ ├── 07-create-pr.yaml +│ ├── 08-validate-pr.yaml +│ ├── 09-evaluate.yaml +│ └── namespace.yaml +│ +├── oumi/ # AI model training & serving +│ ├── serve.py # FastAPI model server +│ ├── train_sft.py # SFT training script +│ ├── run_inference.py # Model testing +│ ├── generate_training_data.py # Training data generator +│ ├── generate_with_oumi.py # Model-based data generator +│ ├── InFoundry_Cloud_Training.ipynb # Colab notebook +│ ├── generated_training_data.jsonl +│ └── trained_model/ # LoRA adapter weights +│ +├── scripts/ # Utility scripts +│ └── generate-pr.sh +│ +├── tests/ # Unit tests +│ ├── test_generate_training_data.py +│ ├── test_kestra.js +│ ├── test_mcp_server.mjs +│ └── test_oumi_serve.py +│ +├── ui/ # Next.js React dashboard +│ ├── app/ # App Router pages +│ │ ├── page.jsx # Landing page +│ │ ├── layout.jsx +│ │ ├── globals.css +│ │ ├── api/ # API routes +│ │ ├── configure/ +│ │ ├── dashboard/ +│ │ └── pipeline/ +│ ├── components/ # React components +│ │ ├── PipelineForm.jsx +│ │ ├── StepProgressBar.jsx +│ │ ├── StepOutputCard.jsx +│ │ └── viewers/ +│ └── lib/ +│ └── kestra.js # Kestra API client +│ +├── requirements.txt # Python dependencies +├── README.md +├── LICENSE +├── CHANGELOG.md +├── CODE_OF_CONDUCT.md +├── CONTRIBUTING.md +├── ROADMAP.md +└── SECURITY.md +``` + +## Component Specifications + +### 1. Oumi AI Model + +**Purpose:** Intelligent cloud architecture recommendations + +**Specification:** +- **Base Model:** Qwen/Qwen2.5-1.5B-Instruct +- **Fine-tuning Method:** LoRA (Low-Rank Adaptation) +- **Training Examples:** 500 +- **Output Format:** JSON with architecture pattern, components, and rationale + +**API Contract (`/v1/chat/completions`):** + +```typescript +// Request +interface CompletionRequest { + model: string; // Default: "codellama" + messages?: Message[]; // Chat messages + prompt?: string; // Direct prompt + max_tokens?: number; // Default: 2000 + temperature?: number; // Default: 0.7 +} -Export button → sends final graph to backend. +// Response +interface CompletionResponse { + id: string; + choices: [{ + message: { + role: "assistant"; + content: string; // JSON architecture recommendation + }; + finish_reason: "stop"; + }]; +} +``` -Cline +**Architecture Output Schema:** +```json +{ + "pattern": "serverless|microservices_ecs|kubernetes|event_driven|lift_and_shift", + "components": ["api_gateway", "lambda", "rds", "..."], + "topology": "N services with PATTERN on CLOUD", + "scaling_strategy": "horizontal_autoscaling|serverless_autoscaling|kubernetes_hpa|vertical_scaling", + "estimated_cost_tier": "low|medium|high", + "rationale": "Explanation of architecture choice" +} +``` -Converts finalized diagram → IaC templates. +### 2. Kestra Pipelines -Creates Terraform/Helm modules. +**Purpose:** Orchestrate the complete IaC generation workflow -Creates a PR. +**Pipeline Inputs (`00-end-to-end.yaml`):** -CodeRabbit +| Input | Type | Required | Default | Description | +|-------|------|----------|---------|-------------| +| `repo_url` | STRING | Yes | - | Git repository URL | +| `branch` | STRING | Yes | "main" | Git branch to clone | +| `repository` | STRING | Yes | "crypticsaiyan/infotest" | GitHub repo for PR (owner/repo) | +| `cloud_provider` | STRING | Yes | "aws" | Target cloud (aws, gcp, azure) | +| `project_name` | STRING | Yes | "infoundry" | Project name for resources | +| `target_folder` | STRING | Yes | "infra" | Target folder for Terraform | +| `skip_pr` | BOOLEAN | Yes | false | Skip PR creation | +| `skip_validation` | BOOLEAN | Yes | false | Skip validation step | -Validates IaC quality, safety, patterns. +**Pipeline Outputs:** -Blocks unsafe infrastructure. +| Output | Type | Description | +|--------|------|-------------| +| `service_profile` | FILE | Repository analysis results | +| `architecture_plan` | FILE | AI-proposed architecture | +| `graph` | FILE | React Flow graph JSON | +| `iac_bundle` | FILE | Generated Terraform ZIP | +| `iac_manifest` | FILE | IaC generation manifest | -4) Repo structure (copyable) -infoundry-architect/ -├─ analyzer/ -│ ├─ repo_analyzer.py -│ └─ graph_builder.py -├─ orchestrator/ -│ ├─ oumi_integration.py -│ ├─ kestra_integration/ -│ └─ cline_integration.py -├─ ui/ -│ ├─ pages/ -│ ├─ components/ -│ └─ diagram/ # visual graph renderer -├─ examples/ -│ ├─ sft_examples.jsonl -│ └─ sample_graph.json -├─ scripts/ -│ └─ export_iac.py -├─ docs/ -│ ├─ architecture.md -│ └─ graph_schema.md -└─ .github/ - └─ workflows/ +**Kestra Namespace:** `infoundry` -5) Development stack & prerequisites +### 3. MCP Server -Next.js + Graph library (React Flow, D3, or Cytoscape). +**Purpose:** Expose InFoundry tools to Cline CLI -Oumi (SFT + LoRA). +**Tools Specification:** -Cline CLI for IaC generation. +#### Tool: `ingest_repo` +```typescript +{ + repoPath: string; // Local path OR GitHub URL +} +// Returns: ServiceProfile JSON +``` -CodeRabbit GitHub App. +#### Tool: `ingest_telemetry` +```typescript +{ + services: string; // Comma-separated service names + metricsJson?: string; // Optional real metrics JSON +} +// Returns: TelemetrySummary JSON +``` -Kestra local or mocked pipeline. +#### Tool: `propose_architecture` +```typescript +{ + serviceProfile: string; // JSON from ingest_repo + telemetrySummary?: string; // JSON from ingest_telemetry + cloudProvider?: string; // aws, gcp, azure (default: aws) +} +// Returns: ArchitecturePlan JSON +``` -Terraform or CDK. +#### Tool: `render_graph` +```typescript +{ + architecturePlan: string; // JSON from propose_architecture +} +// Returns: ReactFlow Graph JSON +``` -6) Step-by-step 7-day build plan -Day 1: Repo Analyzer + basic graph +#### Tool: `generate_iac` +```typescript +{ + graph: string; // JSON from render_graph + cloudProvider?: string; // aws, gcp, azure + projectName?: string; // Resource naming prefix +} +// Returns: TerraformBundle with manifest +``` -Implement service detection. +#### Tool: `validate_iac` +```typescript +{ + iacBundle: string; // Path to Terraform files +} +// Returns: ValidationResult JSON +``` -Generate simple graph JSON. +#### Tool: `create_pr` +```typescript +{ + iacBundle: string; // Path to Terraform files + repository: string; // owner/repo + targetFolder?: string; // Target folder in repo + labels?: string; // Comma-separated labels +} +// Returns: PRResult JSON +``` -Display graph in React Flow. +#### Tool: `validate_pr` +```typescript +{ + prResult: string; // JSON from create_pr + repository: string; // owner/repo +} +// Returns: ValidationStatus JSON +``` -Day 2: Kestra + Summary +#### Tool: `evaluate` +```typescript +{ + deployResult: string; // JSON from validate_iac +} +// Returns: EvaluationResult JSON +``` -Simple pipeline to summarize metrics or repo profile. +### 4. Next.js UI -Day 3: Oumi Training (SFT only) +**Purpose:** Visual dashboard for architecture management -Train Oumi to pick architecture patterns. +**Pages:** -Input: repo profile → output: architecture decision. +| Route | Component | Purpose | +|-------|-----------|---------| +| `/` | `page.jsx` | Landing page with features | +| `/dashboard` | `dashboard/page.jsx` | React Flow architecture editor | +| `/pipeline` | `pipeline/page.jsx` | Pipeline execution & monitoring | +| `/configure` | `configure/page.jsx` | Service configuration | -Day 4: Architecture Generator +**API Routes:** -Convert Oumi output → nodes + edges. +| Endpoint | Method | Purpose | +|----------|--------|---------| +| `/api/kestra/execute` | POST | Trigger Kestra pipeline | +| `/api/kestra/status/[id]` | GET | Get execution status | +| `/api/kestra/file` | GET | Fetch Kestra storage file | -Display visually. +**Environment Variables:** -Day 5: Export IaC & Cline +| Variable | Default | Description | +|----------|---------|-------------| +| `KESTRA_API_URL` | `http://localhost:8080` | Kestra server URL | +| `KESTRA_TENANT` | `main` | Kestra tenant ID | +| `KESTRA_API_TOKEN` | - | Optional API token | +| `KESTRA_USERNAME` | - | Optional basic auth username | +| `KESTRA_PASSWORD` | - | Optional basic auth password | -Send graph to backend → Cline → generate Terraform. +## Data Schemas -Create PR. +### ServiceProfile -Day 6: UI Polish +```json +{ + "source": "https://github.com/owner/repo", + "services": { + "backend": { "type": "nodejs", "path": "/backend" }, + "frontend": { "type": "nodejs", "path": "/frontend" } + }, + "databases": ["postgres"], + "queues": ["redis"], + "service_count": 2, + "primary_language": "nodejs", + "has_infrastructure": false, + "analyzed_at": "2024-01-01T00:00:00Z" +} +``` -Add drag/drop, edit nodes, export diagram, refresh. +### TelemetrySummary -Day 7: Docs, video demo, cleanup +```json +{ + "summary": { + "backend": { + "p50": 120, + "p95": 350, + "avg_latency": 180, + "error_rate": 0.02, + "cost": 45, + "cpu_usage": 0.65, + "memory_mb": 384 + } + }, + "collected_at": "2024-01-01T00:00:00Z", + "source": "mock|provided" +} +``` -Write final docs. +### ArchitecturePlan -Deploy UI to Vercel. +```json +{ + "architecture": { + "pattern": "microservices_ecs", + "components": ["api_gateway", "ecs_cluster", "alb", "rds"], + "topology": "2 services with microservices_ecs pattern on aws", + "scaling_strategy": "horizontal_autoscaling", + "estimated_cost_tier": "medium", + "rationale": "Selected microservices_ecs based on 2 services" + }, + "inputs": { + "service_count": 2, + "cloud_provider": "aws" + }, + "source": "oumi|heuristic" +} +``` -7) Example outputs -7.1 Oumi SFT example -input: repo_profile:{services:[auth,payments],db:"postgres"} -output: pattern: microservices_ecs; add: load_balancer, vpc, rds_instance +### ReactFlow Graph -7.2 Graph JSON example +```json { - "nodes":[ - {"id":"auth","type":"service"}, - {"id":"lb","type":"load_balancer"}, - {"id":"db","type":"database"} + "nodes": [ + { + "id": "api_gateway", + "type": "infrastructureNode", + "data": { + "label": "Api Gateway", + "type": "api_gateway", + "icon": "globe", + "category": "network", + "scaling": false + }, + "position": { "x": 0, "y": 180 }, + "style": { + "background": "#FF9800", + "borderRadius": "8px" + } + } ], - "edges":[ - {"source":"lb","target":"auth","type":"routes"}, - {"source":"auth","target":"db","type":"queries"} + "edges": [ + { + "id": "cdn-api_gateway", + "source": "cdn", + "target": "api_gateway", + "type": "smoothstep", + "animated": true + } ] } +``` -8) CI/CD & quality - -CodeRabbit reviews IaC generated from the diagram. - -Lint Next.js UI, test graph validity. - -Validate no cycles, missing nodes, invalid edges. - -9) Open-source best practices - -LICENSE (Apache 2.0) - -CONTRIBUTING.md - -CODE_OF_CONDUCT - -PR template - -Branch strategy (main, dev, feat/*). - -10) Security - -No secrets in frontend. - -All IaC validated. +### IaC Manifest -Human-in-loop required. - -Diagram changes logged. - -11) Metrics (for judging) - -Time saved designing architecture - -Accuracy of architecture suggestions - -Quality of IaC generation - -Usability of visual editor - -12) Demo script - -Show uploading a repo - -Repo analyzer identifies components - -Oumi proposes architecture - -Diagram appears visually - -Drag nodes → modify architecture - -Export IaC → Cline PR - -CodeRabbit reviews - -Done - -13) Hardest challenges + solutions - -Converting architecture → visual graph → IaC -→ Use a consistent schema + mapping engine. - -Ensuring architectures follow best practices -→ Oumi training + CodeRabbit gating. - -UI graph editing stability -→ Use React Flow with controlled state. - -14) Stretch goals - -Multi-cloud support - -Real-time collaboration - -Reverse-engineer IaC → diagram - -Cost estimation overlay - -15) Quick start checklist - -Repo analyzer working - -Graph generator working - -Visual UI working - -Oumi trained - -Cline integration - -CodeRabbit review - -Vercel deployment \ No newline at end of file +```json +{ + "files": ["main.tf", "variables.tf", "outputs.tf"], + "cloud_provider": "aws", + "project_name": "infoundry", + "components": ["api_gateway", "ecs_cluster", "rds"], + "generated_at": "2024-01-01T00:00:00Z" +} +``` + +## Infrastructure Components Mapping + +### AWS Components + +| Component | Terraform Resource | Description | +|-----------|-------------------|-------------| +| `api_gateway` | `aws_apigatewayv2_api` | HTTP API Gateway | +| `ecs_cluster` | `aws_ecs_cluster` | ECS Fargate Cluster | +| `eks_cluster` | `aws_eks_cluster` | Kubernetes Cluster | +| `lambda_functions` | `aws_lambda_function` | Serverless Functions | +| `alb` | `aws_lb` | Application Load Balancer | +| `rds` | `aws_db_instance` | RDS PostgreSQL/MySQL | +| `dynamodb` | `aws_dynamodb_table` | NoSQL Database | +| `elasticache` | `aws_elasticache_cluster` | Redis Cache | +| `sqs` | `aws_sqs_queue` | Message Queue | +| `sns` | `aws_sns_topic` | Pub/Sub Topic | +| `msk` | `aws_msk_cluster` | Managed Kafka | +| `cdn` | `aws_cloudfront_distribution` | CDN | +| `s3` | `aws_s3_bucket` | Object Storage | + +## Decision Matrix + +### Architecture Pattern Selection + +| Criteria | serverless | microservices_ecs | kubernetes | event_driven | +|----------|------------|-------------------|------------|--------------| +| Service Count | 1-2 | 3-4 | 5+ | Any | +| Traffic Pattern | Low/Burst | Moderate | High | Event-based | +| Cost Sensitivity | High | Medium | Low | Medium | +| Latency Needs | Variable | Low | Very Low | Async OK | +| Team Size | Small | Medium | Large | Medium | + +### Scaling Strategy Selection + +| Strategy | Use Case | Components | +|----------|----------|------------| +| `serverless_autoscaling` | Lambda functions | No configuration needed | +| `vertical_scaling` | Simple apps, DBs | Resize instances | +| `horizontal_autoscaling` | Stateless services | ASG, ECS Service scaling | +| `kubernetes_hpa` | K8s workloads | Horizontal Pod Autoscaler | + +## Testing Strategy + +### Unit Tests + +```bash +# Python tests (pytest) +pytest tests/ -v + +# MCP server tests (Node.js) +node tests/test_mcp_server.mjs + +# Oumi server tests +pytest tests/test_oumi_serve.py -v +``` + +### Integration Tests + +```bash +# Kestra pipeline tests +node tests/test_kestra.js +``` + +### Manual Testing + +1. Start all services (see `runlocally.md`) +2. Open UI at http://localhost:3000 +3. Navigate to `/pipeline` +4. Enter a GitHub repository URL +5. Run the pipeline and monitor progress + +## Deployment Options + +### Local Development +- Docker Compose for all services +- See `runlocally.md` for details + +### Cloud Deployment + +**Vercel (UI):** +```bash +cd ui +vercel deploy +``` + +**Docker (Services):** +```dockerfile +# Example Oumi Dockerfile +FROM python:3.11-slim +WORKDIR /app +COPY requirements.txt . +RUN pip install -r requirements.txt +COPY oumi/ . +CMD ["uvicorn", "serve:app", "--host", "0.0.0.0", "--port", "8000"] +``` + +**Kubernetes:** +- Deploy Kestra via Helm chart +- Deploy Oumi server as Deployment + Service +- Deploy UI as Deployment + Ingress + +## Future Enhancements + +See [ROADMAP.md](../ROADMAP.md) for planned features: + +- Multi-cloud support (GCP, Azure) +- Enhanced authentication +- Automated compliance auditing +- CI/CD platform integrations +- Advanced cost estimation +- AI-driven architecture optimization +- Collaborative real-time editing diff --git a/docs/architecture.md b/docs/architecture.md index f2af086..9b7b41d 100644 --- a/docs/architecture.md +++ b/docs/architecture.md @@ -1,62 +1,259 @@ # InFoundry Architecture +This document provides a comprehensive overview of InFoundry's system architecture, including all components, their interactions, and data flows. + ## System Overview -InFoundry is a self-adaptive Cloud Architect + SRE agent with the following components: +InFoundry is a self-adaptive Cloud Architect + SRE agent that: +1. Inspects codebases and telemetry +2. Proposes deployable IaC/CI changes +3. Runs safe test deployments +4. Iteratively optimizes cost, latency, and reliability with human-in-the-loop approvals ``` -┌─────────────────────────────────────────────────────────────────┐ -│ User Interface │ -│ Next.js React Dashboard with React Flow Architecture Editor │ -└────────────────────────────┬────────────────────────────────────┘ - │ -┌────────────────────────────▼────────────────────────────────────┐ -│ Kestra Orchestrator │ -│ 9-step pipeline: Ingest → Propose → Generate → Deploy → Eval │ -└────────────────────────────┬────────────────────────────────────┘ - │ - ┌────────────────────┼────────────────────┐ - ▼ ▼ ▼ -┌───────────────┐ ┌───────────────┐ ┌───────────────┐ -│ Oumi Model │ │ MCP Server │ │ Terraform │ -│ Architecture │ │ Cline CLI │ │ Generator │ -│ Recommender │ │ Integration │ │ & Validator │ -└───────────────┘ └───────────────┘ └───────────────┘ +┌─────────────────────────────────────────────────────────────────────────────┐ +│ InFoundry Architecture │ +├─────────────────────────────────────────────────────────────────────────────┤ +│ │ +│ ┌──────────────┐ ┌──────────────────┐ ┌─────────────────────────┐ │ +│ │ Next.js │────▶│ Kestra Pipeline │────▶│ Oumi AI Model │ │ +│ │ UI (3000) │◀────│ Orchestrator │◀────│ Server (8000) │ │ +│ └──────────────┘ └──────────────────┘ └─────────────────────────┘ │ +│ │ │ │ │ +│ │ │ │ │ +│ ▼ ▼ ▼ │ +│ ┌──────────────┐ ┌──────────────────┐ ┌─────────────────────────┐ │ +│ │ React Flow │ │ MCP Server │ │ Trained LoRA Model │ │ +│ │ Diagrams │ │ (Cline/Claude) │ │ Qwen2.5-1.5B-Instruct │ │ +│ └──────────────┘ └──────────────────┘ └─────────────────────────┘ │ +│ │ +└─────────────────────────────────────────────────────────────────────────────┘ +``` + +## Core Components + +### 1. Next.js UI (`/ui`) + +The user-facing dashboard built with Next.js 15. + +**Technology Stack:** +- Next.js 15 with App Router +- React 18 +- React Flow (@xyflow/react) for architecture visualization +- Lucide React for icons +- CSS Modules for styling + +**Key Pages:** +| Route | Purpose | +|-------|---------| +| `/` | Landing page with feature overview | +| `/dashboard` | Visual architecture editor with React Flow | +| `/pipeline` | Kestra pipeline runner and monitor | +| `/configure` | Service profile configuration | + +**Key Components:** +- `PipelineForm` - Collects pipeline inputs (repo URL, cloud provider, etc.) +- `StepProgressBar` - Real-time pipeline step tracking +- `StepOutputCard` - Displays step outputs and artifacts +- `ServiceConfigGenerator` - Interactive service configuration + +### 2. Kestra Pipeline Orchestrator (`/orchestrator`) + +Kestra orchestrates the complete infrastructure lifecycle with 9 sequential steps. + +**Pipeline Flow:** +``` +┌─────────────┐ ┌─────────────┐ ┌─────────────┐ ┌─────────────┐ +│ 01. Ingest │──▶│ 02. Ingest │──▶│ 03. Propose │──▶│ 04. Render │ +│ Repo │ │ Telemetry │ │Architecture │ │ Graph │ +└─────────────┘ └─────────────┘ └─────────────┘ └─────────────┘ + │ +┌─────────────┐ ┌─────────────┐ ┌─────────────┐ │ +│09. Evaluate │◀──│ 08. Validate│◀──│ 07. Create │◀──────────┘ +│ │ │ PR │ │ PR │ │ +└─────────────┘ └─────────────┘ └─────────────┘ │ + │ + ┌─────────────┐ ┌─────────────┐ │ + │ 06. Validate│◀──│05. Generate │◀──────────┘ + │ IaC │ │ IaC │ + └─────────────┘ └─────────────┘ +``` + +**Pipeline Steps:** + +| Step | Pipeline File | Description | Outputs | +|------|---------------|-------------|---------| +| 1 | `01-ingest-repo.yaml` | Analyzes repository for services, DBs, queues | `service_profile.json` | +| 2 | `02-ingest-telemetry.yaml` | Collects service telemetry metrics | `telemetry_summary.json` | +| 3 | `03-propose-architecture.yaml` | AI-powered architecture proposal | `architecture_plan.json` | +| 4 | `04-render-graph.yaml` | Converts architecture to React Flow graph | `graph.json` | +| 5 | `05-generate-iac.yaml` | Generates Terraform using AI Agent | `iac_bundle.zip` | +| 6 | `06-validate-iac.yaml` | Validates Terraform (fmt, init, validate) | `deploy_result.json` | +| 7 | `07-create-pr.yaml` | Creates GitHub PR with IaC files | `pr_result.json` | +| 8 | `08-validate-pr.yaml` | Monitors PR status and CI checks | `validation_result.json` | +| 9 | `09-evaluate.yaml` | Computes reward score for optimization | `evaluation_result.json` | + +**AI Agent Integration:** +The `05-generate-iac.yaml` pipeline uses Kestra's built-in AI Agent for: +- **Data Summarization**: Analyzes complex graph data and produces technical summaries +- **Autonomous Decision Making**: Makes IaC configuration decisions (encryption, instance tiers, auto-scaling) + +### 3. Oumi AI Model (`/oumi`) + +The "brain" of InFoundry - a fine-tuned LLM for cloud architecture recommendations. + +**Model Details:** +- Base Model: `Qwen/Qwen2.5-1.5B-Instruct` +- Training: LoRA (Low-Rank Adaptation) fine-tuning +- Training Data: 500 architecture examples +- Published: [crypticsayan/infoundry-architect](https://huggingface.co/crypticsayan/infoundry-architect) + +**Key Files:** +| File | Purpose | +|------|---------| +| `serve.py` | FastAPI server with OpenAI-compatible endpoint | +| `train_sft.py` | SFT training script with LoRA | +| `run_inference.py` | Test the trained model | +| `generate_training_data.py` | Rule-based training data generator | + +**Model Output Schema:** +```json +{ + "architecture": { + "pattern": "serverless|microservices_ecs|kubernetes|event_driven", + "components": ["api_gateway", "lambda", "rds", "..."], + "topology": "N services with PATTERN on CLOUD", + "scaling_strategy": "horizontal_autoscaling|serverless_autoscaling", + "estimated_cost_tier": "low|medium|high", + "rationale": "Explanation of architecture choice" + }, + "inputs": { + "service_count": 3, + "cloud_provider": "aws" + }, + "source": "ai_recommendation" +} ``` -## Components +**Inference Hierarchy:** +1. **Oumi Model** (trained model) - Primary +2. **Ollama** (local LLM) - Fallback +3. **Heuristics** - Final fallback -### 1. UI Dashboard (`ui/`) -- Next.js 15 React application -- React Flow architecture diagram editor -- Real-time Kestra pipeline monitoring -- Service configuration generator +### 4. MCP Server (`/infoundry-mcp-server`) -### 2. Kestra Pipelines (`orchestrator/kestra_pipelines/`) -- `00-end-to-end.yaml` - Master orchestration pipeline -- Steps 01-09 for complete workflow +Model Context Protocol server for Cline CLI integration. -### 3. Oumi Model (`oumi/`) -- Fine-tuned Qwen2.5-1.5B for architecture decisions -- FastAPI server with OpenAI-compatible API -- Training data generator +**Technology:** +- TypeScript with ES modules +- `@modelcontextprotocol/sdk` for MCP protocol +- `zod` for input validation -### 4. MCP Server (`infoundry-mcp-server/`) -- Model Context Protocol server for Cline CLI -- Exposes all 9 workflow steps as tools -- TypeScript with Zod validation +**Exposed Tools (9 steps mirroring Kestra):** -### 5. Scripts (`scripts/`) -- `generate-pr.sh` - Auto-generate PR descriptions +| Tool | Description | +|------|-------------| +| `ingest_repo` | Analyze repository (local or GitHub URL) | +| `ingest_telemetry` | Collect service telemetry metrics | +| `propose_architecture` | AI architecture proposal using Oumi | +| `render_graph` | Convert architecture to React Flow graph | +| `generate_iac` | Generate Terraform from graph | +| `validate_iac` | Validate Terraform code | +| `create_pr` | Create GitHub PR with IaC files | +| `validate_pr` | Check PR status and reviews | +| `evaluate` | AI evaluation with recommendations | ## Data Flow -1. **Ingest**: Analyze repository → Extract services, DBs, queues -2. **Telemetry**: Collect metrics → Summarize latency, errors, CPU -3. **Propose**: AI recommends → Pattern, components, scaling -4. **Render**: Generate graph → React Flow compatible JSON -5. **Generate**: Create Terraform → AWS infrastructure code -6. **Validate**: Run checks → fmt, validate, tflint, tfsec -7. **Create PR**: Open GitHub PR → With generated IaC -8. **Validate PR**: Check CI → Reviews, status checks -9. **Evaluate**: AI assessment → Recommendations for next iteration +### End-to-End Pipeline Flow + +``` +User Input Processing Output +───────────────────────────────────────────────────────────────────── + +GitHub Repo URL ──▶ Clone & Analyze ──▶ service_profile.json + │ │ + ▼ ▼ +Service Names ──▶ Collect Metrics ──▶ telemetry_summary.json + │ │ + ▼ ▼ +Service Profile ──▶ Oumi AI Model ──▶ architecture_plan.json ++ Telemetry │ + │ ▼ + │ Convert to React Flow ──▶ graph.json + │ │ + │ ▼ + │ AI Agent + Heuristics ──▶ iac_bundle.zip + │ │ + │ ▼ + │ Terraform Validation ──▶ deploy_result.json + │ │ + │ ▼ + │ Create GitHub PR ──▶ pr_result.json + │ │ + │ ▼ + │ Monitor CI Checks ──▶ validation_result.json + │ │ + ▼ ▼ + Evaluate Results ──▶ evaluation_result.json +``` + +### Architecture Component Categories + +InFoundry classifies infrastructure components into categories: + +| Category | AWS Components | +|----------|---------------| +| **Network** | CDN (CloudFront), API Gateway, ALB, VPC | +| **Compute** | ECS Cluster, EKS Cluster, Lambda Functions | +| **Database** | RDS, DynamoDB, ElastiCache | +| **Messaging** | SQS, SNS, MSK (Kafka), EventBridge | +| **Storage** | S3, EFS | + +### Architecture Patterns + +| Pattern | Use Case | Components | +|---------|----------|------------| +| `serverless` | 1-2 services, low traffic | API Gateway, Lambda, DynamoDB | +| `microservices_ecs` | 3-4 services, moderate traffic | ECS, ALB, RDS | +| `kubernetes` | 5+ services, high traffic | EKS, ALB, RDS, ElastiCache | +| `event_driven` | Event processing, async | MSK/SQS, Lambda, EventBridge | + +## Integration Points + +### External Services + +| Service | Purpose | Configuration | +|---------|---------|---------------| +| **GitHub** | Repository analysis, PR creation | `GITHUB_TOKEN` | +| **Kestra** | Pipeline orchestration | `KESTRA_API_URL`, `KESTRA_API_TOKEN` | +| **Ollama** | Local LLM fallback | `OLLAMA_URL` (default: localhost:11434) | +| **AWS** | Cloud deployment | `AWS_ACCESS_KEY_ID`, `AWS_SECRET_ACCESS_KEY` | + +### API Endpoints + +**Oumi Server (port 8000):** +- `POST /v1/chat/completions` - OpenAI-compatible chat endpoint +- `GET /health` - Health check + +**Kestra (port 8080):** +- `POST /api/v1/executions` - Trigger pipeline execution +- `GET /api/v1/executions/{id}` - Get execution status + +**Next.js UI (port 3000):** +- `/api/kestra/execute` - Proxy to Kestra execution +- `/api/kestra/status/{id}` - Proxy to Kestra status +- `/api/kestra/file` - Fetch Kestra storage files + +## Security Considerations + +1. **Path Validation**: MCP server validates paths to prevent command injection +2. **Environment Secrets**: Sensitive values stored as environment variables +3. **Temporary Files**: Cloned repos are cleaned up after analysis +4. **API Authentication**: Kestra supports token and basic auth + +## Scalability + +- **Horizontal Scaling**: Kestra workers can be scaled independently +- **Model Serving**: Oumi server can run multiple replicas behind a load balancer +- **UI Scaling**: Next.js can be deployed on Vercel or Kubernetes with auto-scaling diff --git a/docs/kestra-architecture.md b/docs/kestra-architecture.md deleted file mode 100644 index 017f284..0000000 --- a/docs/kestra-architecture.md +++ /dev/null @@ -1,210 +0,0 @@ -# Kestra Orchestration Architecture - -## Overview - -Kestra is the **single source of truth** for all InFoundry Architect operations. Each user action (Analyze, Propose, Apply, Deploy, Evaluate) maps to a Kestra pipeline with full artifact storage, retry logic, and observability. - -``` -UI -> Kestra API (start pipeline) - ↓ - Kestra pipeline(s) -----> calls services: - - repo-analyzer service - - telemetry fetchers - - Oumi model - - graph builder - - Cline IaC generator - - GitHub API - - CodeRabbit - - deploy runner - - metrics collector - ↓ -Kestra saves artifacts & status -> UI polls for updates -``` - ---- - -## Pipelines - -| Pipeline | Description | Inputs | Outputs | -|----------|-------------|--------|---------| -| `01-ingest-repo` | Analyzes repo, detects services | `repo_url`, `repo_path` | `service_profile.json` | -| `02-ingest-telemetry` | Fetches metrics | `prometheus_url`, `services` | `telemetry_summary.json` | -| `03-propose-architecture` | Calls Oumi for pattern | `service_profile`, `telemetry_summary` | `architecture_plan.json` | -| `04-render-graph` | Builds React Flow graph | `architecture_plan` | `graph.json` | -| `05-generate-iac` | Generates Terraform via Cline | `graph` | `iac.zip`, `iac_manifest.json` | -| `06-create-pr` | Creates GitHub PR | `iac_bundle`, `repository` | `pr_result.json` | -| `07-validate-pr` | Polls GitHub/CodeRabbit | `pr_result`, `repository` | `validation_result.json` | -| `08-test-deploy` | Deploys to localstack/kind | `iac_bundle` | `deploy_result.json` | -| `09-evaluate` | Computes reward score | `deploy_result` | `evaluation_result.json` | -| `00-end-to-end` | Master pipeline | All above | All artifacts | - ---- - -## Best Practices - -### Retries & Backoff - -```yaml -retry: - type: exponential - maxAttempt: 3 - interval: PT10S - maxInterval: PT2M -``` - -### Secrets - -Secrets are referenced via `{{ secret('NAME') }}`: - -```yaml -env: - GITHUB_TOKEN: "{{ secret('GITHUB_TOKEN') }}" - OUMI_API_KEY: "{{ secret('OUMI_API_KEY') }}" -``` - -**Required secrets:** -- `GITHUB_TOKEN` - GitHub PAT for PR operations -- `OUMI_API_KEY` - Oumi model API key -- `AWS_ACCESS_KEY_ID` / `AWS_SECRET_ACCESS_KEY` - For deployments - -### Artifacts - -All outputs use `outputFiles` for automatic artifact storage: - -```yaml -outputFiles: - - service_profile.json - - graph.json - -outputs: - - id: graph - type: FILE - value: "{{ outputs.task.outputFiles['graph.json'] }}" -``` - -### Idempotency - -- Each execution has unique `{{ execution.id }}` -- Artifacts include execution ID for deduplication -- Branch names include timestamps for PR creation - -### Error Handling - -```yaml -errors: - - id: notify_failure - type: io.kestra.plugin.core.log.Log - level: ERROR - message: "Pipeline failed: {{ error.message }}" -``` - -### Observability - -- Labels for filtering: `labels: {stage: ingest, component: repo-analyzer}` -- Structured logging in scripts -- Metric emissions for monitoring - ---- - -## UI Integration - -### Starting a Pipeline - -```typescript -// POST /api/kestra/run -const startPipeline = async (flowId: string, inputs: object) => { - const res = await fetch('http://kestra:8080/api/v1/executions', { - method: 'POST', - headers: { - 'Content-Type': 'application/json', - 'Authorization': `Bearer ${KESTRA_API_KEY}` - }, - body: JSON.stringify({ - namespace: 'infoundry', - flowId, - inputs - }) - }); - return res.json(); // { id: "execution-id" } -}; -``` - -### Polling Execution Status - -```typescript -// GET /api/kestra/executions/:id -const pollExecution = async (executionId: string) => { - const res = await fetch( - `http://kestra:8080/api/v1/executions/${executionId}` - ); - const data = await res.json(); - return { - state: data.state.current, // CREATED, RUNNING, SUCCESS, FAILED - outputs: data.outputs, - startDate: data.state.startDate, - endDate: data.state.endDate - }; -}; -``` - -### Fetching Artifacts - -```typescript -// GET artifact file -const getArtifact = async (executionId: string, path: string) => { - const res = await fetch( - `http://kestra:8080/api/v1/executions/${executionId}/file?path=${path}` - ); - return res.json(); -}; -``` - ---- - -## Day-by-Day Migration - -| Day | Tasks | -|-----|-------| -| **1** | Set up Kestra (Docker), create namespace, upload 01-02 pipelines | -| **2** | Implement 03-04 pipelines, test Oumi integration | -| **3** | Implement 05-06 pipelines, test Cline + GitHub | -| **4** | Implement 07-08 pipelines, set up localstack | -| **5** | Implement 09 + 00-end-to-end, add error handlers | -| **6** | Build UI API routes, wire up frontend | -| **7** | Observability setup, documentation | - ---- - -## Quick Start - -1. **Start Kestra**: - ```bash - docker run -d --name kestra -p 8080:8080 kestra/kestra:latest standalone - ``` - -2. **Upload pipelines**: - ```bash - cd orchestrator/kestra_pipelines - for f in *.yaml; do - curl -X POST http://localhost:8080/api/v1/flows \ - -H "Content-Type: application/x-yaml" \ - --data-binary @"$f" - done - ``` - -3. **Configure secrets** in Kestra UI → Secrets - -4. **Test end-to-end**: - ```bash - curl -X POST http://localhost:8080/api/v1/executions \ - -H "Content-Type: application/json" \ - -d '{ - "namespace": "infoundry", - "flowId": "end-to-end", - "inputs": { - "repository": "your-org/your-repo", - "skip_pr": true, - "skip_deploy": true - } - }' - ``` diff --git a/docs/runlocally.md b/docs/runlocally.md index f004dcd..7eb19c6 100644 --- a/docs/runlocally.md +++ b/docs/runlocally.md @@ -1,65 +1,546 @@ -# Run Locally +# Running InFoundry Locally + +This guide walks you through setting up and running InFoundry on your local machine for development and testing. ## Prerequisites -- Python 3.11+ -- Node.js 18+ -- Optional: Docker, kind, localstack + +Before you begin, ensure you have the following installed: + +| Requirement | Minimum Version | Purpose | +|-------------|-----------------|---------| +| Python | 3.11+ | Oumi AI server, tests | +| Node.js | 18+ | UI, MCP server | +| npm | 9+ | Package management | +| Git | 2.x | Repository cloning | +| Docker | 20+ | Kestra, optional services | +| Docker Compose | 2.x | Multi-container setup | + +### Optional Dependencies + +| Tool | Purpose | +|------|---------| +| Ollama | Local LLM fallback | +| kind | Local Kubernetes cluster | +| LocalStack | AWS emulation | +| Terraform | IaC validation | ## Quick Start -### 1. UI Dashboard +### 1. Clone the Repository + ```bash -cd ui -npm install -npm run dev -# Open http://localhost:3000 +git clone https://github.com/crypticsaiyan/infoundry.git +cd infoundry ``` -### 2. Oumi Model Server +### 2. Install Dependencies + +```bash +# Python dependencies +pip install -r requirements.txt + +# UI dependencies +cd ui && npm install && cd .. + +# MCP server dependencies +cd infoundry-mcp-server && npm install && npm run build && cd .. +``` + +### 3. Start All Services + +Open three terminal windows/tabs: + +**Terminal 1 - Oumi AI Server:** ```bash cd oumi +python serve.py +# Server starts at http://localhost:8000 +``` + +**Terminal 2 - Kestra (Docker):** +```bash +docker run --pull=always --rm -it -p 8080:8080 \ + --user=root \ + -v /var/run/docker.sock:/var/run/docker.sock \ + -v /tmp:/tmp \ + kestra/kestra:latest server local +# Kestra UI at http://localhost:8080 +``` + +**Terminal 3 - Next.js UI:** +```bash +cd ui +cp .env.example .env.local +# Edit .env.local if needed +npm run dev +# UI starts at http://localhost:3000 +``` + +## Detailed Setup + +### Python Environment (Recommended) -# Setup virtual environment +```bash +# Create virtual environment python -m venv venv -source venv/bin/activate # or: source venv/bin/activate.fish + +# Activate (Linux/macOS) +source venv/bin/activate + +# Activate (Fish shell) +source venv/bin/activate.fish + +# Activate (Windows) +.\venv\Scripts\activate # Install dependencies -pip install fastapi uvicorn +pip install -r requirements.txt +``` + +### Oumi Model Setup + +The Oumi model provides AI-powered architecture recommendations. + +#### Option A: Use Pre-trained Model (Recommended) + +```bash +cd oumi + +# Install Git LFS if not already installed +git lfs install + +# Clone the trained model from Hugging Face +git clone https://huggingface.co/crypticsayan/infoundry-architect trained_model +``` + +#### Option B: Train Your Own Model + +```bash +cd oumi + +# Generate training data +python generate_training_data.py --count 500 + +# Train the model (requires GPU) +python train_sft.py +``` + +#### Option C: Use with Google Colab + +1. Open `oumi/InFoundry_Cloud_Training.ipynb` in Google Colab +2. Select a T4 GPU runtime +3. Run all cells to train the model +4. Download the trained model -# Run server +### Starting the Oumi Server + +```bash +cd oumi python serve.py -# API at http://localhost:8000 ``` -### 3. MCP Server (for Cline integration) +**Environment Variables:** + +| Variable | Default | Description | +|----------|---------|-------------| +| `USE_OUMI` | `true` | Use trained Oumi model | +| `USE_OLLAMA` | `false` | Fallback to Ollama | +| `OLLAMA_URL` | `http://localhost:11434` | Ollama server URL | + +**Test the server:** +```bash +curl -X POST http://localhost:8000/v1/chat/completions \ + -H "Content-Type: application/json" \ + -d '{ + "model": "oumi", + "messages": [{"role": "user", "content": "Recommend architecture for 3 nodejs services"}] + }' +``` + +### Kestra Setup + +#### Option A: Docker (Recommended for Development) + +```bash +# Single container with local storage +docker run --pull=always --rm -it -p 8080:8080 \ + --user=root \ + -v /var/run/docker.sock:/var/run/docker.sock \ + -v /tmp:/tmp \ + kestra/kestra:latest server local +``` + +#### Option B: Docker Compose (Production-like) + +Create `docker-compose.yml`: + +```yaml +version: '3.8' +services: + kestra: + image: kestra/kestra:latest + ports: + - "8080:8080" + volumes: + - /var/run/docker.sock:/var/run/docker.sock + - /tmp:/tmp + - kestra-data:/app/storage + environment: + - KESTRA_CONFIGURATION_STRING=| + datasources: + postgres: + url: jdbc:postgresql://postgres:5432/kestra + driverClassName: org.postgresql.Driver + username: kestra + password: kestra + depends_on: + - postgres + + postgres: + image: postgres:15 + environment: + POSTGRES_USER: kestra + POSTGRES_PASSWORD: kestra + POSTGRES_DB: kestra + volumes: + - postgres-data:/var/lib/postgresql/data + +volumes: + kestra-data: + postgres-data: +``` + +Run: +```bash +docker-compose up -d +``` + +#### Deploy Pipelines to Kestra + +```bash +cd orchestrator/kestra_pipelines + +# Upload all pipeline files +for f in *.yaml; do + curl -X POST http://localhost:8080/api/v1/flows \ + -H "Content-Type: application/x-yaml" \ + --data-binary @"$f" +done +``` + +**Fish shell version:** +```fish +cd orchestrator/kestra_pipelines + +for f in *.yaml + curl -X POST http://localhost:8080/api/v1/flows \ + -H "Content-Type: application/x-yaml" \ + --data-binary @$f +end +``` + +#### Configure Kestra Secrets + +Set secrets in the Kestra UI (http://localhost:8080) or via API: + +```bash +# GitHub Token (for PR creation) +curl -X PUT http://localhost:8080/api/v1/namespaces/infoundry/secrets/GITHUB_TOKEN \ + -H "Content-Type: text/plain" \ + -d "ghp_your_github_token" + +# AWS Credentials (optional, for deployment) +curl -X PUT http://localhost:8080/api/v1/namespaces/infoundry/secrets/AWS_ACCESS_KEY_ID \ + -H "Content-Type: text/plain" \ + -d "your_access_key" + +curl -X PUT http://localhost:8080/api/v1/namespaces/infoundry/secrets/AWS_SECRET_ACCESS_KEY \ + -H "Content-Type: text/plain" \ + -d "your_secret_key" +``` + +### Next.js UI Setup + +```bash +cd ui + +# Copy environment template +cp .env.example .env.local +``` + +Edit `.env.local`: +```env +KESTRA_API_URL=http://localhost:8080 +KESTRA_TENANT=main +# Optional authentication +# KESTRA_API_TOKEN=your_token +# KESTRA_USERNAME=admin +# KESTRA_PASSWORD=admin +``` + +Start development server: +```bash +npm run dev +``` + +Access the UI at http://localhost:3000 + +### MCP Server Setup (For Cline Integration) + ```bash cd infoundry-mcp-server + +# Install dependencies npm install + +# Build TypeScript npm run build -npm start ``` -### 4. Run Tests +Configure Cline to use the MCP server. Add to `~/.config/cline/mcp.json`: + +```json +{ + "mcpServers": { + "infoundry-architect": { + "command": "node", + "args": ["/path/to/infoundry/infoundry-mcp-server/dist/index.js"], + "env": { + "GITHUB_TOKEN": "ghp_your_github_token" + } + } + } +} +``` + +## Optional: Ollama Setup + +Ollama provides a local LLM fallback when the Oumi model is unavailable. + +### Install Ollama + +```bash +# Linux +curl -fsSL https://ollama.ai/install.sh | sh + +# macOS +brew install ollama + +# Windows - Download from https://ollama.ai +``` + +### Start Ollama and Pull Model + +```bash +# Start Ollama server +ollama serve + +# In another terminal, pull a model +ollama pull codellama +# or +ollama pull qwen2.5:7b +``` + +### Configure Oumi to Use Ollama + +Set environment variable before starting serve.py: + ```bash -# Python tests -pip install pytest fastapi httpx +USE_OLLAMA=true python serve.py +``` + +## Running Tests + +### Python Tests + +```bash +# Install test dependencies +pip install pytest httpx + +# Run all tests pytest tests/ -v -# MCP server integration test -node tests/test-all-tools.mjs +# Run specific tests +pytest tests/test_oumi_serve.py -v +pytest tests/test_generate_training_data.py -v +``` + +### MCP Server Tests + +```bash +# Ensure Oumi server is running first +cd oumi && python serve.py & + +# Run tests +node tests/test_mcp_server.mjs +``` + +### Kestra Pipeline Tests + +```bash +# Ensure Kestra is running +node tests/test_kestra.js +``` + +## Full Stack Development Workflow + +### 1. Start All Services + +```bash +# Terminal 1: Start Oumi server +cd oumi && python serve.py + +# Terminal 2: Start Kestra +docker run --pull=always --rm -it -p 8080:8080 \ + --user=root \ + -v /var/run/docker.sock:/var/run/docker.sock \ + -v /tmp:/tmp \ + kestra/kestra:latest server local + +# Terminal 3: Start UI +cd ui && npm run dev +``` + +### 2. Upload Kestra Pipelines + +```bash +cd orchestrator/kestra_pipelines +for f in *.yaml; do + curl -X POST http://localhost:8080/api/v1/flows \ + -H "Content-Type: application/x-yaml" \ + --data-binary @"$f" +done +``` + +### 3. Access the Application + +- **UI Dashboard:** http://localhost:3000 +- **Kestra UI:** http://localhost:8080 +- **Oumi API:** http://localhost:8000 + +### 4. Test the Pipeline + +1. Open http://localhost:3000/pipeline +2. Enter a GitHub repository URL (e.g., `https://github.com/crypticsaiyan/infotest`) +3. Configure options (cloud provider, project name, etc.) +4. Click "Run Pipeline" +5. Monitor progress in real-time + +### 5. Test via API + +```bash +# Trigger end-to-end pipeline +curl -X POST http://localhost:8080/api/v1/executions \ + -H "Content-Type: application/json" \ + -d '{ + "namespace": "infoundry", + "flowId": "end-to-end", + "inputs": { + "repo_url": "https://github.com/crypticsaiyan/infotest", + "branch": "main", + "repository": "crypticsaiyan/infotest", + "cloud_provider": "aws", + "project_name": "infoundry-demo", + "skip_pr": true + } + }' +``` + +## Troubleshooting + +### Oumi Server Won't Start + +**Problem:** `ModuleNotFoundError: No module named 'oumi'` + +**Solution:** +```bash +pip install oumi[gpu] +# or without GPU +pip install oumi ``` -### 5. Utility Scripts +### Kestra Pipeline Fails + +**Problem:** AI tasks fail with connection refused + +**Solution:** The Kestra container can't reach Ollama on localhost. Use Docker's host gateway: ```bash -# Generate PR description -./scripts/generate-pr.sh dev +# In pipeline YAML, use: +baseUrl: "http://172.17.0.1:11434/v1" # Docker gateway to host ``` -## Kestra Pipelines +### UI Can't Connect to Kestra + +**Problem:** CORS errors or connection refused + +**Solution:** Check `.env.local` has correct `KESTRA_API_URL` and Kestra is running. + +### MCP Server Not Found by Cline + +**Problem:** Cline can't find the InFoundry MCP server + +**Solution:** +1. Verify the path in `mcp.json` is absolute +2. Ensure `npm run build` was run +3. Check `dist/index.js` exists + +### Model Inference is Slow + +**Problem:** Architecture proposals take too long + +**Solution:** +1. Use Ollama with a smaller model: `ollama pull qwen2.5:7b` +2. Enable GPU acceleration if available +3. The heuristic fallback is always fast + +### Out of Memory During Training + +**Problem:** Training fails with OOM error + +**Solution:** +1. Use Google Colab with T4 GPU (free tier) +2. Reduce batch size in `train_sft.py` +3. Use gradient checkpointing + +## Port Summary + +| Service | Port | URL | +|---------|------|-----| +| Next.js UI | 3000 | http://localhost:3000 | +| Oumi Server | 8000 | http://localhost:8000 | +| Kestra | 8080 | http://localhost:8080 | +| Ollama | 11434 | http://localhost:11434 | + +## Environment Variables Summary + +### Oumi Server +| Variable | Default | Description | +|----------|---------|-------------| +| `USE_OUMI` | `true` | Use trained Oumi model | +| `USE_OLLAMA` | `false` | Use Ollama as fallback | +| `OLLAMA_URL` | `http://localhost:11434` | Ollama server URL | + +### Next.js UI +| Variable | Default | Description | +|----------|---------|-------------| +| `KESTRA_API_URL` | `http://localhost:8080` | Kestra server URL | +| `KESTRA_TENANT` | `main` | Kestra tenant ID | +| `KESTRA_API_TOKEN` | - | Optional API token | +| `KESTRA_USERNAME` | - | Optional basic auth | +| `KESTRA_PASSWORD` | - | Optional basic auth | + +### MCP Server +| Variable | Default | Description | +|----------|---------|-------------| +| `GITHUB_TOKEN` | - | GitHub PAT for PR creation | +| `KESTRA_URL` | `http://localhost:8080` | Kestra server URL | + +## Next Steps -The Kestra pipelines are in `orchestrator/kestra_pipelines/`. To run them: +After successfully running locally: -1. Install Kestra: https://kestra.io/docs/installation -2. Import the pipelines from `orchestrator/kestra_pipelines/` -3. Configure environment variables (GITHUB_TOKEN, etc.) -4. Trigger the `00-end-to-end.yaml` pipeline +1. **Explore the UI:** Navigate to `/dashboard` to see the architecture editor +2. **Run a Pipeline:** Go to `/pipeline` and analyze a real repository +3. **Test MCP Tools:** Use Cline to interact with InFoundry tools +4. **Customize Pipelines:** Modify Kestra YAML files for your needs +5. **Train Custom Models:** Generate domain-specific training data diff --git a/infoundry-mcp-server/README.md b/infoundry-mcp-server/README.md index c76a0b8..ff70fcc 100644 --- a/infoundry-mcp-server/README.md +++ b/infoundry-mcp-server/README.md @@ -32,7 +32,10 @@ Add to `~/.config/cline/mcp.json`: "mcpServers": { "infoundry-architect": { "command": "node", - "args": ["/path/to/infoundry/infoundry-mcp-server/dist/index.js"] + "args": ["/path/to/infoundry/infoundry-mcp-server/dist/index.js"], + "env": { + "GITHUB_TOKEN": "ghp_YOUR_GITHUB_TOKEN_HERE" + } } } } diff --git a/orchestrator/kestra_pipelines/07-create-pr.yaml b/orchestrator/kestra_pipelines/07-create-pr.yaml index 9d369b0..efc38a8 100644 --- a/orchestrator/kestra_pipelines/07-create-pr.yaml +++ b/orchestrator/kestra_pipelines/07-create-pr.yaml @@ -22,13 +22,13 @@ inputs: - id: target_folder type: STRING - required: false + required: true defaults: "infra" description: Target folder in repo for Terraform files - id: labels type: STRING - required: false + required: true defaults: "infoundry,infrastructure,terraform" description: Comma-separated labels to add to the PR @@ -45,6 +45,7 @@ tasks: - id: create_pr type: io.kestra.plugin.scripts.python.Script + timeout: PT5M env: GITHUB_TOKEN: "{{ kv('GITHUB_TOKEN') }}" REPOSITORY: "{{ inputs.repository }}" diff --git a/orchestrator/kestra_pipelines/README.md b/orchestrator/kestra_pipelines/README.md new file mode 100644 index 0000000..e790bc3 --- /dev/null +++ b/orchestrator/kestra_pipelines/README.md @@ -0,0 +1,88 @@ +# Kestra Orchestration Pipelines + +This directory contains the Kestra pipelines that power the InFoundry Architect agent. Each pipeline corresponds to a step in the architecture generation and deployment workflow. + +## Overview + +Kestra orchestrates the entire lifecycle: +1. **Ingest**: Analyze repository and telemetry. +2. **Propose**: Generate architecture plan using Oumi AI. +3. **Render**: Visual graph generation. +4. **Generate**: Create Terraform IaC. +5. **Validate**: Verify IaC correctness. +6. **PR**: Create GitHub Pull Request. +7. **Evaluate**: Deploy and score the architecture. + +## AI Agent Capabilities + +InFoundry uses **Kestra's built-in AI Agent** to summarize data from other systems and make intelligent decisions: + +1. **Data Summarization**: The `ai_summarize` task in the `05-generate-iac` pipeline analyzes the complex graph data and produces a concise technical summary. +2. **Autonomous Decision Making**: The `ai_decide` task consumes this summary to make critical architecture decisions (e.g., enabling encryption, selecting instance tiers, configuring auto-scaling) **without human intervention**. + +This demonstrates the power of Kestra's AI capabilities to not just process data, but to strictly reasoning about it and driving the workflow logic. + +## Pipelines + +| Pipeline | Description | Inputs | Outputs | +|----------|-------------|--------|---------| +| `00-end-to-end` | Master pipeline chaining all steps | `repo_url`, `branch`, `repository`, `cloud_provider`, `project_name`, `target_folder`, `skip_pr`, `skip_validation` | All artifacts | +| `01-ingest-repo` | Analyzes repo to detect services | `repo_url`, `branch` | `service_profile.json` | +| `02-ingest-telemetry` | Fetches mock telemetry metrics | `services` | `telemetry_summary.json` | +| `03-propose-architecture` | Generates architecture plan via Oumi | `service_profile`, `telemetry_summary`, `use_oumi`, `cloud_provider`, `oumi_server_url` | `architecture_plan.json` | +| `04-render-graph` | Builds React Flow graph JSON | `architecture_plan` | `graph.json` | +| `05-generate-iac` | Generates Terraform bundle | `graph`, `cloud_provider`, `project_name` | `iac_bundle` (zip), `iac_manifest.json`, `ai_summary.json`, `ai_decisions.json` | +| `06-validate-iac` | Validates Terraform code | `iac_bundle` | `deploy_result.json` | +| `07-create-pr` | Creates GitHub PR with changes | `iac_bundle`, `repository`, `base_branch`, `target_folder`, `labels` | `pr_result.json` | +| `08-validate-pr` | Monitors PR status and checks | `pr_result`, `repository` | `validation_result.json` | +| `09-evaluate` | Computes reward score for RL | `deploy_result` | `evaluation_result.json` | + +## Usage + +### Prerequisites + +- Kestra server running (Docker or standalone) +- Oumi server running (for AI proposals) +- GitHub Personal Access Token (PAT) configured +- AWS/Cloud credentials configured + +### Deployment + +Upload all pipelines to Kestra: + +```bash +cd orchestrator/kestra_pipelines +for f in *.yaml; do + curl -X POST http://localhost:8080/api/v1/flows \ + -H "Content-Type: application/x-yaml" \ + --data-binary @"$f" +done +``` + +### Configuration + +Ensure the following secrets are set in Kestra: +- `GITHUB_TOKEN`: For PR creation and cloning private repos. +- `AWS_ACCESS_KEY_ID`: For deployment steps. +- `AWS_SECRET_ACCESS_KEY`: For deployment steps. + +### Running a Pipeline + +You can trigger the end-to-end flow via API: + +```bash +curl -X POST http://localhost:8080/api/v1/executions \ + -H "Content-Type: application/json" \ + -d '{ + "namespace": "infoundry", + "flowId": "end-to-end", + "inputs": { + "repo_url": "https://github.com/crypticsaiyan/infotest", + "branch": "main", + "repository": "crypticsaiyan/infotest", + "cloud_provider": "aws", + "project_name": "infoundry-demo", + "skip_pr": false + } + }' +``` diff --git a/ui/.env.example b/ui/.env.example new file mode 100644 index 0000000..c3d93ca --- /dev/null +++ b/ui/.env.example @@ -0,0 +1,12 @@ +# Kestra API URL (default: http://localhost:8080) +KESTRA_API_URL=http://localhost:8080 + +# Kestra Tenant (default: main) +KESTRA_TENANT=main + +# Authentication (Option 1: API Token) +# KESTRA_API_TOKEN=your-api-token-here + +# Authentication (Option 2: Username/Password) +# KESTRA_USERNAME=your-username +# KESTRA_PASSWORD=your-password diff --git a/ui/README.md b/ui/README.md index 8fae5c9..1ff1516 100644 --- a/ui/README.md +++ b/ui/README.md @@ -29,11 +29,17 @@ Open [http://localhost:3000](http://localhost:3000) to view the dashboard. ## Environment Variables -Create `.env.local`: - -```bash -KESTRA_API_URL=http://localhost:8080 -``` +Copy `.env.example` to `.env.local` and update the values: + + ```bash + cp .env.example .env.local + ``` + + Variables: + - `KESTRA_API_URL`: URL of the Kestra API (default: `http://localhost:8080`) + - `KESTRA_TENANT`: Kestra tenant ID (default: `main`) + - `KESTRA_API_TOKEN`: Optional API token for authentication + - `KESTRA_USERNAME` / `KESTRA_PASSWORD`: Optional credentials for basic auth ## Build