SlopScan: Advanced AI Code Quality & Slop Detector

SlopScan is a modern, production-grade technical auditor and codebase scanner designed to detect AI-generated "slop"—code, commits, and documentation that look superficially correct but are structurally flawed, incomplete, redundant, or claim more than they actually deliver.

By combining robust static analysis (Abstract Syntax Trees) with LLM validation (Gemini 3.1 Flash-Lite & Qwen 72B), SlopScan identifies fake PR claims, generic AI commit patterns, hallucinated/outdated guides, and dead code before it pollutes your production codebase.

---

📖 Table of Contents

1. Project Overview
2. Core Features
3. Tech Stack
4. Architecture Overview
5. Installation & Setup
6. API Overview
7. Project Structure
8. Usage & Workflows
9. Future Improvements

---

🔍 Project Overview

The Problem Statement

As generative AI coding assistants (like Copilots or coding agents) become deeply integrated into software workflows, codebases are experiencing an influx of high-volume, low-intent code contributions. This often leads to a new category of technical debt known as AI Slop:

• Misleading PR Descriptions: PR descriptions claiming to fix major bugs or add complex features when the underlying diff only contains superficial tweaks or string changes.
• Generic Commit Messages: Commit histories filled with vague AI-generated phrases (e.g., "Update index.js", "Refactor code", "Fix bug") without specific context or actual implementation matching the description.
• Hallucinated Documentation: Readmes and guides containing obsolete version references, repetitive boilerplate, excessive marketing fluff, or references to functions/classes that do not exist in the code.
• Stubs & Placeholder Code: Production files left with empty function bodies (pass), fake try-catch blocks returning static dummy successes ({ "status": "ok" }), hardcoded API secrets, or dead imports.

Key Value Proposition

SlopScan empowers technical leads, open-source maintainers, and engineering judges by serving as an automated, zero-trust codebase auditor. It performs multi-layered static and semantic checks to guarantee that what developers (and their AI assistants) claim in descriptions matches the exact truth inside the repository files.

Target Users

• Technical Reviewers & Judges: Speed up assessment of hackathon submissions or open-source PRs by verifying the truthfulness of claimed work.
• Maintainers & Tech Leads: Automate pre-merge audits to prevent low-quality code boilerplate or security vulnerabilities (secrets/localhost endpoints) from slipping in.
• Enterprise Teams: Ensure commit logs remain pristine, informative, and traceable.

---

🛠️ Core Features

1. PR Reviewer

Compares public GitHub pull requests against the actual underlying code changes.

• Diff Extraction: Fetches unified patches and diffs from GitHub's REST API.
• AI Claim Extraction: Gemini parses the PR description to extract testable claims (e.g., "Adds Auth middleware").
• Mismatch Detection: Evaluates each claim against a localized "change fingerprint" (which files were touched, if logic was modified, whether tests were included).
• Verdict Generation: Flags PRs as TRUSTWORTHY, SUSPICIOUS, or MISLEADING with a structured confidence score.

2. Commit Reviewer

Performs a deep review of a repository's recent commit history to catch generic and hallucinated commit messages.

• Diff Matching: Verifies the commit message against the exact code changes in that specific commit.
• Qwen Audit: Uses Qwen 2.5 72B Instruct to classify commits as TRUSTWORTHY, GENERIC (vague or low detail), or HALLUCINATED (claiming features not in the diff).
• Aggregate Quality Score: Computes a dynamic code-to-commit quality index for the repository.

3. Docs Verifier

Scans repository Markdown documents (.md and .mdx) to detect high-frequency AI patterns:

• Placeholder Finder: Flags unresolved template brackets, coming soon, lorem ipsum, or TODO: write docs.
• Marketing Fluff: Measures the density of generic buzzwords ("revolutionary", "seamless integration", "cutting-edge").
• Repetitive Paragraphs: Utilizes Jaccard similarity and NLTK/Levenshtein algorithms to locate semantic duplication across paragraphs.
• Claim Inconsistency: Qwen verifies all architectural and setup claims in the README against the actual source codebase.

4. Code Scanner

Runs a deep file-by-file audit of codebases to locate structural issues, AI stubs, and anti-patterns:

• Regex Pattern Scorer: Identifies hardcoded keys, localhost development links, throw-not-implemented stubs, and empty stubs.
• Heuristic Scoring: Rates the severity of findings (low, medium, high, critical) and filters false positives in test or mock files.
• Deep AI Analysis: Concurrently calls Qwen 72B via OpenRouter to analyze long files or files with multiple regex hits, generating specific fixes.

---

💻 Tech Stack

Frontend

• Core: Next.js App Router (React 19)
• Styling: Vanilla CSS + Tailwind CSS v4 custom theme extensions
• Icons: lucide-react
• Streaming Client: Native Web ReadableStreams consuming Server-Sent Events (SSE)

Backend

• Web Framework: FastAPI (Uvicorn / asyncio loops)
• Configuration: Pydantic Settings v2
• HTTP Clients: httpx (async pools)
• Database & Cache:
  • Redis (metadata caching)

AI Providers & Models

• Gemini API: gemini-3.1-flash-lite (via google-generativeai SDK and direct betav1 HTTP endpoints)
• OpenRouter: qwen/qwen-2.5-72b-instruct (used for advanced codebase claims verification, commit verification, and deep file analysis)

---

📐 Architecture Overview

SlopScan utilizes a modern direct-streaming, event-driven architecture centered around Server-Sent Events (SSE).

graph TD
    A[Next.js Client] -->|HTTP POST Request| B[FastAPI Web Server]
    B -->|Initiate Async Stream| C[Streaming Handler]
    C -->|Fetch Source / Diffs| D[GitHub API]
    C -->|Run Local Heuristics| E[Regex Scorer]
    C -->|Request Structuring & Claims Verification| F[Gemini 3.1 Flash-Lite]
    C -->|Request Deep Code Audit / Commits Review| G[OpenRouter Qwen 72B]
    C -->|Stream SSE Chunk-by-Chunk| A
    C -->|Cache Output| H[Redis]

Loading

Request Flow

1. Frontend Initiation: The user pastes a GitHub URL into the Next.js input box.
2. Stream Negotiation: The client establishes a fetch() connection configured for text/event-stream.
3. Async Processing Pipeline: The backend clones the repository (or fetches metadata/diffs via GitHub REST endpoints) and runs local heuristics.
4. LLM Verification: Specific payloads are compiled and dispatched asynchronously using concurrent task managers.
5. Real-time Progress Delivery: As phases complete, structured JSON progress packets (e.g. {percent: 50, step: "Extracting claims"}) are streamed back over the HTTP pipe.
6. Result Presentation: The connection finishes with a final result package, rendering custom, interactive tabs on the dashboard.

---

🚀 Installation & Setup

Prerequisites

• Python 3.10+
• Node.js 18+
• Redis Server (running locally or in the cloud)
• Git installed on the system path
• API Keys for Gemini and OpenRouter

---

Backend Setup

1. Navigate to the server directory:

   cd server

2. Create and activate a virtual environment:

   python -m venv venv
   # On Windows:
   venv\Scripts\activate
   # On macOS/Linux:
   source venv/bin/activate

3. Install python packages:

   pip install -r requirements.txt

4. Configure environment variables: Create a .env file in the root of the server/ directory:

   # ── Redis ──
   REDIS_URL=redis://localhost:6379/0
   
   # ── GitHub Token ──
   GITHUB_TOKEN=your_github_personal_access_token_here
   
   # ── AI API Keys ──
   GEMINI_API_KEY=your_gemini_api_key_here
   OPENROUTER_API_KEY=your_openrouter_api_key_here
   
   # ── Settings ──
   SANDBOX_ROOT=./tmp/sandboxed
   MAX_REPO_SIZE_MB=200
   SCAN_TIMEOUT_SECONDS=120
   
   # ── CORS ──
   CORS_ORIGINS=http://localhost:3000

5. Start the FastAPI server:

   uvicorn main:app --host 127.0.0.1 --port 8000 --reload

   The backend API will be available at http://127.0.0.1:8000. You can inspect the interactive docs at http://127.0.0.1:8000/docs.

---

Frontend Setup

1. Navigate to the client directory:

   cd client

2. Install Node packages:

   npm install

3. Configure environment variables: Create a .env.local file in the client/ directory:

   NEXT_PUBLIC_API_URL=http://localhost:8000

4. Launch the development server:

   npm run dev

   Open http://localhost:3000 in your browser to view the application.

---

🔌 API Overview

All major backend routes are mounted on the FastAPI application. The main routes are:

GitHub Data Routes

• GET /github/repo?owner={owner}&name={name}: Fetches basic repository metrics (description, language, forks, stars) with Redis caching.
• GET /github/prs?owner={owner}&name={name}&state={state}: Lists the 20 most recent pull requests.
• GET /github/pr/{number}?owner={owner}&name={name}: Returns extensive details on a single PR, including code files changed, author metadata, and diff patches.
• GET /github/file?owner={owner}&name={name}&path={path}: Fetches raw file content from GitHub to enable in-app file viewing.

Analysis & Streaming Routes (Server-Sent Events)

• POST /api/pr-review/analyze: Evaluates a PR's description against its diff patches. Streams progress events and the final PR evaluation results.
• POST /api/commits/analyze: Scans recent commit messages against diffs using Qwen. Calculates aggregate quality and slop indexes.
• POST /api/docs/analyze: Clones the repository, parses all markdown files, checks paragraph Levinshtein similarity, and runs local stubs tests. Streams progress and outputs the executive summary with HTML Highlights.
• POST /api/code-review/analyze: Initiates a full codebase scan. Runs local regex patterns followed by OpenRouter deep audits on files that exceed length thresholds.

Utilities

• POST /api/code-review/summary: Sends a batch of code issues to Gemini to group, classify, and format into an executive report.
• GET /health: Verifies basic service connectivity.

---

📂 Project Structure

Backend Layout

server/
├── core/
│   ├── config.py           # Configuration definitions & env loaders
│   └── redis.py            # Redis client initialization & lifespan hooks
├── routers/
│   ├── github.py           # Github fetching, details, and raw files
│   ├── pr_review.py        # Pull request auditing endpoint
│   ├── commit_verify.py    # Commit logging scanner
│   ├── docs_verify.py      # Markdown verifier endpoint
│   └── code_scan.py        # Codebase scanner endpoint
├── services/
│   ├── github_service.py   # GitHub REST API wrappers
│   ├── pr_review_service.py# SSE pipeline for PR review
│   ├── commit_verifier_service.py # SSE pipeline for commits
│   ├── docs_verifier_service.py   # SSE pipeline for docs
│   ├── code_review_service.py     # Codebase scanner coordinator
│   ├── openrouter_service.py      # Qwen completions integration
│   ├── gemini_service.py   # Gemini API endpoints integration
│   └── diff_service.py     # Diff hunk parser & stats generator
├── utils/
│   └── pattern_scorer.py   # RegEx rule scoring engine for code quality
└── main.py                 # FastAPI Application Entrypoint

Frontend Layout

client/
├── src/
│   ├── app/
│   │   ├── globals.css     # Global stylesheets and Tailwind v4 theme variables
│   │   ├── layout.jsx      # Google Sans font imports and metadata setup
│   │   ├── page.jsx        # Landing page with repo search box
│   │   └── repo/[owner]/[name]/
│   │       ├── page.jsx    # Repo Dashboard card navigation
│   │       ├── prs/        # PR lists and detailing
│   │       ├── commits/    # Commit Verifier interface
│   │       ├── docs/       # Docs Verifier dashboard
│   │       └── scan/       # Code Scanner file explorer tree
│   ├── components/
│   │   ├── repo/           # RepoDashboard and Sticky Navigation
│   │   ├── pr/             # PRLists, Commit details, and Verdict panels
│   │   ├── commits/        # CommitVerifier component UI
│   │   ├── docs/           # DocsVerifier panel UI
│   │   ├── scanner/        # IDE-like CodeViewer, FileExplorer tree, and Issue panels
│   │   └── ui/             # Badges, Cards, Spinners, and Progress Streams
│   ├── hooks/
│   │   └── useActionStream.js # Core hook consuming SSE text streams
│   └── lib/
│       ├── api.js          # Fetch wrappers for API endpoints
│       ├── constants.js    # Global dictionary for status codes and colors
│       └── github.js       # GitHub repository URL parsing utility

---

📄 License

This project is licensed under the MIT License - see the LICENSE file for details.