DataOps Incident Copilot

DataOps Incident Copilot is a local AI-assisted incident investigation application for analyzing KPI anomalies. It combines deterministic data diagnostics with an LLM-based investigator, then applies rule-based classification so the final incident type is grounded in collected evidence.

The project runs locally with DuckDB, Ollama, Qwen3, FastAPI, and Streamlit.

What It Does

Given a question such as:

Why did completed orders decrease yesterday?

the application:

1. Compares completed-order metrics.
2. Checks data freshness and required-field completeness.
3. Passes deterministic evidence to the investigator Agent.
4. Produces a structured root-cause report.
5. Applies deterministic classification precedence.
6. Stores the investigation lifecycle and result history.
7. Displays the result through the API or Streamlit dashboard.

Supported incident classifications include:

• business_decline
• incomplete_data
• data_quality_issue
• stale_data
• pipeline_failure
• unknown

Key Features

• Local LLM execution through Ollama and Qwen3
• OpenAI Agents SDK orchestration
• Deterministic metric and data-quality diagnostics
• Structured Pydantic root-cause reports
• Rule-based incident classification
• Read-only SQL validation with sqlglot
• Single-statement SQL enforcement
• SQL result limiting and truncation metadata
• Investigation history persisted in DuckDB
• FastAPI service with generated API documentation
• Streamlit dashboard with report history and JSON export
• Unit and integration test coverage
• Standard Python logging for investigation and SQL lifecycles

Architecture

flowchart TD
    U[User Question] --> API[FastAPI or Streamlit]
    API --> S[Investigation Service]

    S --> M[Metric Diagnostics]
    S --> Q[Data Quality Diagnostics]

    M --> C[Investigation Context]
    Q --> C

    C --> A[Investigator Agent]
    A --> L[Ollama / Qwen3]
    A --> R[Structured RootCauseReport]

    R --> D[Deterministic Classifier]
    D --> H[Investigation History]
    D --> O[API or Dashboard Response]

    M --> DB[(DuckDB)]
    Q --> DB
    H --> HDB[(History DuckDB)]

Loading

The LLM creates the explanatory report, while deterministic diagnostics retain priority when classifying known data conditions.

Classification precedence:

1. Required-field data-quality issue
2. Stale or missing data
3. Incomplete current-period data
4. Confirmed business decline
5. Unknown

Technology Stack

• Python
• FastAPI
• Streamlit
• DuckDB
• OpenAI Agents SDK
• Ollama
• Qwen3
• Pydantic
• sqlglot
• pytest

Project Structure

app/
├── agents/          # Investigator Agent configuration
├── api/             # API schemas and routes
├── models/          # Pydantic domain models
├── prompts/         # Investigator system prompt
├── repositories/    # Investigation-history persistence
├── services/        # Investigation orchestration and classification
├── tools/           # SQL, metric, schema, and quality diagnostics
├── ui/              # Streamlit application
├── config.py        # Environment-based settings
└── main.py          # FastAPI application

data/                # DuckDB and local analytical datasets
docs/                # Supporting documentation
scripts/             # Data generation and local utility scripts
sql/                 # Investigation SQL examples
tests/               # Unit, API, UI, and integration tests

Prerequisites

• Python 3.12 or newer
• Ollama
• A local Qwen3 model
• PowerShell, Bash, or another terminal

The project has been tested with Python 3.14.

Setup

1. Create and activate a virtual environment

PowerShell:

python -m venv .ai
.ai\Scripts\Activate.ps1

macOS or Linux:

python -m venv .ai
source .ai/bin/activate

2. Install dependencies

Install the packages used by the application:

python -m pip install fastapi uvicorn streamlit duckdb openai-agents sqlglot pydantic pytest pytest-asyncio

3. Configure environment variables

Copy the example configuration:

PowerShell:

Copy-Item .env.example .env

macOS or Linux:

cp .env.example .env

Default configuration:

DATAOPS_DATABASE_PATH=data/dataops.duckdb
DATAOPS_HISTORY_DATABASE_PATH=data/investigation_history.duckdb

OLLAMA_BASE_URL=http://localhost:11434/v1
OLLAMA_MODEL=qwen3

AGENT_TIMEOUT_SECONDS=120
AGENT_MAX_RETRIES=2
AGENT_TEMPERATURE=0

RECENT_INVESTIGATION_LIMIT=20
DATA_FRESHNESS_THRESHOLD_HOURS=24

4. Start Ollama and prepare Qwen3

ollama pull qwen3

Confirm that Ollama is running before starting an investigation.

5. Initialize local data

The repository includes data generation and database initialization scripts:

python scripts/generate_data.py
python scripts/initialize_database.py

Run the API

python -m uvicorn app.main:app --reload

Open:

• Health check: http://127.0.0.1:8000/health
• Swagger UI: http://127.0.0.1:8000/docs
• OpenAPI schema: http://127.0.0.1:8000/openapi.json

Example health response:

{
  "status": "ok",
  "service": "dataops-incident-copilot"
}

Investigation requests use a natural-language question between 5 and 500 characters. The generated Swagger UI shows the available investigation endpoint and its complete request and response schemas.

Run the Streamlit Dashboard

python -m streamlit run app/ui/streamlit_app.py

The dashboard provides:

• Natural-language investigation input
• Incident type, confidence, and human-review status
• Root-cause summary
• Evidence and recommendations
• Raw structured report
• JSON report download
• Latest-report session state

Run Tests

Run the full suite:

python -m pytest -v

Run only tests that do not require the local model:

python -m pytest -m "not integration" -v

Run integration tests after Ollama and Qwen3 are available:

python -m pytest -m integration -v

Focused regression suites:

python -m pytest tests/test_sql_tool.py -v
python -m pytest tests/test_incident_classifier.py -v
python -m pytest tests/test_investigation_service.py -v
python -m pytest tests/test_data_quality_tool.py -v

SQL Safety Model

Agent-generated SQL is handled as read-only analytical SQL.

Current safeguards include:

• DuckDB parsing through sqlglot
• Exactly one SQL statement
• SELECT-style query enforcement
• Explicit blocking of write and command expressions
• Validation before opening a database connection
• A maximum of 1,000 returned rows
• Truncation metadata for larger result sets
• Read-only database connections

This is an AST-validated read-only execution layer, not a complete operating-system-level SQL sandbox.

Root-Cause Report

Investigations return a structured report containing:

{
  "summary": "Concise investigation summary",
  "incident_type": "incomplete_data",
  "root_cause": "Evidence-supported root cause",
  "confidence": 0.85,
  "evidence": [
    {
      "source": "diagnostic source",
      "finding": "Specific factual finding"
    }
  ],
  "recommendations": [
    "Recommended next action"
  ],
  "requires_human_review": false
}

Design Decisions

Deterministic diagnostics before the Agent

Metric and quality diagnostics run before the LLM. Their results are added to InvestigationContext, giving the Agent concrete evidence instead of relying only on the user question.

Deterministic final classification

The Agent provides a structured report, but the incident classifier can override the incident type when deterministic evidence identifies stale data, incomplete data, or a required-field issue.

Local-first execution

The analytical database and language model run locally. This keeps the demonstration self-contained and avoids external API cost.

Stable integration tests

Integration tests patch changing diagnostic inputs where necessary while retaining the real Agent execution. This prevents test outcomes from changing solely because the local dataset has aged.

Limitations

• The analytical scope is currently centered on the included order and pipeline datasets.
• Ollama must be running for Agent-backed investigations.
• Results depend on the selected local model and available evidence.
• SQL validation does not provide a complete file-system or process sandbox.
• The application does not include authentication or multi-user authorization.

License

Add the license that matches how you want others to use this portfolio project.