Multi-Tool Visual Reasoning Agent

A unified AI system that performs Object Detection, OCR, and Image Captioning, and then synthesizes all outputs into one human-friendly response using a lightweight reasoning module.

Built using HuggingFace Transformers, EasyOCR, and PyTorch. Designed for multi-image queries, visual QA, and advanced scene understanding.

---

Core Technology Stack

---

Table of Contents

• Project Overview
• Core Objectives
• System Architecture
• Key Features
• Workflow
• Visual Examples
• Installation
• Contribution
• License

---

Project Overview

The Multi-Tool Visual Reasoning Agent is a modular AI system that processes images using multiple computer vision tools and automatically decides:

• When to run object detection
• When to extract text using OCR
• When to generate a caption
• How to combine all outputs into one meaningful response

This system mimics the behavior of modern multi-modal agents (e.g., GPT-Vision), but implemented using open-source models and a lightweight reasoning module.

---

Core Objectives

• Provide a unified pipeline for object detection, OCR, and captioning
• Automatically select the appropriate tools based on the user query
• Support multi-image queries and batch reasoning
• Generate concise, natural-language answers
• Produce annotated visual outputs

---

System Architecture

• DETR (facebook/detr-resnet-50) → Object detection
• EasyOCR → Text extraction from regions of interest
• BLIP / Vit-GPT2 → Image captioning
• Planner Module → Simple rule-based or LLM-based tool selector
• Reasoning Layer → Synthesizes detection + OCR + captions
• Visualization Layer → Annotated image outputs

---

Key Features

• ✔️ Multi-tool visual reasoning
• ✔️ Automatic tool selection based on query (planner)
• ✔️ Multi-image support
• ✔️ Annotated images for detection and OCR regions
• ✔️ Natural-language response synthesis
• ✔️ Modular — easy to swap in new models

---

Workflow

1. Load image(s)
2. Parse the user query
3. Planner decides which tools to run
4. Run DETR / OCR / Caption models
5. Generate unified final answer
6. Render annotated images

A complete multi-modal reasoning pipeline.

---

Visual Examples

1. Object Detection + OCR (Planner: Detect=True, OCR=True)

2. Image Captioning Output
_{(Planner: Caption=True, Detect=False, OCR=False)}

3. Captioning + Scene Summary
_{(Planner: Caption=True, Detect=False, OCR=False)}

---

Setup & Installation

# Clone repository
git clone https://github.com/hamaylzahid/multi-tool-vision-agent
cd multi-tool-vision-agent

# Install dependencies
pip install -r requirements.txt

For CPU usage:

pip install easyocr torch torchvision transformers pillow opencv-python

Contact & Contribution

Have feedback, want to collaborate, or just say hello?
Let’s connect and improve automated defect detection together.

maylzahid588@gmail.com   |   💼 LinkedIn   |   🌐 GitHub Repo

Found this project helpful? Give it a star on GitHub!
Want to improve it? Submit a PR and join the mission.
_{Your contributions help enhance real-world defect detection systems.}

License

This project is licensed under the MIT License — free to use, modify, and expand.

✅ Project Status: Complete & Portfolio-Ready
🧾 License: MIT — View License »

Crafted with deep learning expertise & real-world defect detection applications. 🧵✨

• •

_{Designed for real-world fabric defect detection and deep learning showcase.}

🤖 Use this project to demonstrate your expertise in computer vision and AI.
🧵 Clone it, modify it, expand it — and build real-world automated defect detection solutions.