An end-to-end AI testing framework and web studio that ingests application documentation, generates robust test cases with function calling specs, executes tests via a LangGraph state engine, evaluates test quality using DeepEval, and visualizes live results in a dark-mode web dashboard.
🌐 Live Web App / Hosted Studio: https://code-killer0.github.io/QA-Automation-Tool/
The QA Automation Studio features a handcrafted, glassmorphism dark-mode UI accessible directly online or hosted locally.
Configure your live public API domain URL, enter your OpenAI API Key (saved locally in browser storage — no .env file required), and upload or paste QA Documentation.
Real-time KPI metrics, total test case counts, pass rates, execution outcome donut charts, and category breakdown bars.
Browse 500–1000+ generated test cases with category badges, full descriptions, and associated assistant Tool_Call JSON chips. Search and filter by category in real time.
Track live execution status codes, dynamic auto-healing of HTTP 422 errors, and detailed failure reason breakdowns.
DeepEval benchmark scores evaluating Test Prompt Quality (94.2%), Tool Call Correctness (98.0%), and Execution Result Accuracy (96.5%).
- 🌐 Zero-Setup Web Studio: Run directly in browser via GitHub Pages or serve locally. Accepts public domain URLs and in-browser OpenAI API Key configuration.
- 📄 Automated Documentation Ingestion: Parses
.pdf,.txt,.docx, and.mdtechnical specs & API docs. - 🧠 AI Loophole & Vulnerability Analysis: Uses GPT-4o to discover failure modes across accuracy, bias, hallucination, data quality, and adversarial security.
- 🤖 Multi-Category Test Prompt Generation: Generates targeted test prompts accompanied by assistant-side
Tool_CallJSON payloads across 12 test categories. - ⚡ LangGraph Execution Engine: State-machine driven HTTP executor (
Parse→Normalize→Execute) with LLM fallback normalizers. - 🔄 Dynamic Value Synthesis: Auto-heals
422 Unprocessable EntityHTTP responses by inferring missing payload fields from FastAPI schema hints. - 🔬 DeepEval Pipeline: Evaluates Test Prompt Quality, Tool Call JSON schema correctness, and Execution Accuracy against configurable thresholds.
- 📊 Modern Glassmorphism UI: High-contrast, handcrafted UI stack built with Plus Jakarta Sans & JetBrains Mono fonts.
+-----------------------------+
| Technical Documentation |
| (PDF / Markdown / DOCX) |
+--------------+--------------+
|
v
+-----------------------------+
| AI Loophole Analyzer |
| (Identifies 15+ failure |
| modes & security risks) |
+--------------+--------------+
|
v
+-----------------------------+
| Test Case Generator |
| (Produces output.csv with |
| Prompts & Tool_Calls) |
+--------------+--------------+
|
v
+-----------------------------+
| LangGraph Executor Engine |
| Parse -> Normalize -> Run |
| (Generates execution report|
| & dynamic auto-healing) |
+--------------+--------------+
|
+--------------+--------------+
| |
v v
+-----------------------------+ +-----------------------------+
| DeepEval Metric Evaluator | | Dark-Mode Web UI |
| (eval_results.json) | | (ui/index.html) |
+-----------------------------+ +-----------------------------+
QA-Automation-Tool/
├── index.html # Root redirect for GitHub Pages deployment
├── main.py # Stage 1: Document parser & test generator CLI
├── executor.py # Stage 2: LangGraph HTTP test executor engine
├── output.csv # Output file containing generated test cases
├── execution_report.csv # Detailed HTTP response & pass/fail execution log
├── execution_summary.json # High-level execution metrics summary
├── raw_llm_output/ # Raw LLM responses per category for auditability
├── docs/ # Documentation & UI screenshots
│ └── images/ # Dashboard screenshots for README
├── eval/ # Stage 3: DeepEval offline evaluation suite
│ ├── __init__.py
│ ├── metrics.py # DeepEval custom metric definitions
│ ├── test_cases.py # CSV & Report to DeepEval test case loaders
│ ├── run_eval.py # Evaluation pipeline runner CLI
│ └── requirements.txt # Eval pipeline dependencies
├── eval_results.json # Output evaluation scores & metric statuses
├── ui/ # Stage 4: Interactive Web Dashboard
│ ├── index.html # Dashboard markup & section routing
│ ├── style.css # Handcrafted glassmorphism styling
│ └── script.js # Client-side generator workflow & chart renderer
├── requirements.txt # Root Python dependencies
└── .env # Environment variable configurations (optional for CLI)
For CLI execution, create a .env file in the project root (optional if using the Web UI):
# LLM Provider Selection: "openai" or "huggingface"
LLM_PROVIDER=openai
# Option A: OpenAI Configuration
OPENAI_API_KEY=your_openai_api_key_here
OPENAI_MODEL=gpt-4o
# Option B: Hugging Face Configuration
HF_TOKEN=your_huggingface_api_token_here
HF_REPO_ID=meta-llama/Llama-3.2-3B-Instruct
# Target API Base URLs
API_BASE_URL=http://localhost:8000
ML_BASE_URL=http://localhost:8001
# File paths
CSV_PATH=output.csv
REPORT_PATH=execution_report.csv
SUMMARY_PATH=execution_summary.json
DOCUMENTATION_PATH=path/to/documentation.pdf
# Executor options
REQUEST_TIMEOUT_SECONDS=20
USE_LLM_NORMALIZER=true
USE_LLM_VALUE_SYNTHESIS=true
LOG_LEVEL=INFO# Clone the repository
git clone https://github.com/code-killer0/QA-Automation-Tool.git
cd QA-Automation-Tool
# Create and activate a virtual environment
python -m venv .venv
source .venv/bin/activate # On Windows: .venv\Scripts\activate
# Install main dependencies
pip install -r requirements.txtOpen https://code-killer0.github.io/QA-Automation-Tool/ in your browser, enter your public domain URL, paste your QA documentation, and click 🚀 Generate Test Suite & Run Execution.
Run main.py to ingest documentation, perform failure mode analysis, and output generated test prompts into output.csv.
python main.py docs/api_spec.pdf --categories ALLRun executor.py to process output.csv via the LangGraph state machine, sending HTTP requests to your target endpoint.
python executor.py --input output.csvOutputs produced:
execution_report.csv: Detailed request logs, HTTP status codes, and pass/fail reasons.execution_summary.json: Pass rates and failure breakdown.
Run the offline evaluation pipeline to benchmark test prompt quality, tool-call schema accuracy, and execution pass rates.
python -m eval.run_eval --csv output.csv --report execution_report.csv --output eval_results.jsonEvaluated Metrics:
- 🎯 Test Prompt Quality (Threshold:
0.50): Evaluates instruction clarity, setup details, and pass/fail criteria. - 🔧 Tool Call Correctness (Threshold:
0.70): Checks JSON formatting, presence ofrole,tool_calls, and parameter schemas. - 📊 Execution Result Accuracy (Threshold:
0.85): Assesses valid HTTP execution outcomes and status classification match.
To run the web studio with a live backend connection to main.py and executor.py (which runs your documentation input and target URL directly through the Python engine):
python3 server.py 8080Navigate to http://localhost:8080 in your web browser.
(If you only need to explore the UI statically without triggering the local Python subprocesses, you can run python3 -m http.server 8080 --directory ui instead. The UI will automatically fall back to browser-based LLM generation or smart client-side heuristic parsing).
- End-to-End Testing
- Integration Testing
- Unit Testing
- Model Accuracy & Regression Testing
- Hallucination & Fabrication Testing
- Bias & Fairness Testing
- Adversarial & Security Testing
- Edge Case & Boundary Testing
- Data Quality & Pipeline Testing
- Performance & Scalability Testing
- Drift & Monitoring Testing
- Explainability & Auditability Testing
- Fork the Repository
- Create a Feature Branch (
git checkout -b feature/AmazingFeature) - Commit your Changes (
git commit -m 'Add some AmazingFeature') - Push to the Branch (
git push origin feature/AmazingFeature) - Open a Pull Request
Distributed under the MIT License. See LICENSE for more information.