Food & Fruit Recognition Pipeline with Calorie Estimation

This project presents an end-to-end computer vision pipeline for analyzing food images. Given an input image, the system automatically:

1. Classifies the image as Food or Fruit
2. Identifies the specific food/fruit type
3. Handles few-shot food categories using a Siamese / ProtoNet model
4. Performs binary & multiclass segmentation
5. Estimates calories based on image weight and food type

The pipeline integrates TensorFlow/Keras and PyTorch models into a single unified workflow.

---

Demo

---

Project Structure Overview

project/
│
├── Notebooks/
│
├── Weights/
│   ├── Food Fruit Classification/
│   ├── Fruit Classification/
│   └── Few Shot/
│
├── Project Data/
│   ├── Food/
│   ├── Fruit/
│   ├── Test/
│
├── Project Description/
│
├── calories.txt
├── main_pipeline.py
├── README.md

---

Pipeline Overview

🔹 Part A — Food vs Fruit Classification

Goal: Binary classification

• Model: ResNet50 (TensorFlow/Keras)

• Output:

  • Food
  • Fruit

This step routes images to the correct downstream task.

---

🔹 Part B — Fruit Classification

Triggered only if Part A predicts Fruit

• Model: MobileNetV2
• Classes: 30+ fruit categories (e.g. Apple_Gala, Mango_Alphonso, Banana, etc.)
• Output: Fruit name per image

---

🔹 Part C — Food Classification (Few-Shot Learning)

Triggered only if Part A predicts Food

• Model: ProtoNet (Siamese / Few-Shot Learning)

• Backbone: ResNet50

• Distance: Cosine similarity

• Uses:

  • Precomputed class prototypes
  • Similarity-based inference

This allows recognition of food classes with very few training examples.

---

🔹 Part D — Segmentation

This project also includes segmentation models for deeper visual understanding:

✔ Binary Segmentation

• Separates food vs background
• Useful for portion estimation

✔ Multiclass Segmentation

• Segments different food components
• Useful for advanced calorie analysis

---

🔥 Calorie Estimation Module

📥 Inputs

• Classified food / fruit label
• Image filename containing weight (e.g. 100g.png)
• Calorie reference file (calories.txt)

Calculation Formula

Total Calories = weight_in_grams × calories_per_gram

---

📤 Output (Per Image)

For example, for 150g.png classified as Mango_Alphonso:

fruit
Mango_Alphonso
97.50 calories

Each image generates its own text file.

---

Example Execution Flow

Input Image
   ↓
Part A: Food vs Fruit
   ↓
 ┌───────────────┐
 │               │
Fruit            Food
 ↓                ↓
Part B           Part C
 ↓                ↓
Fruit Name       Food Name
       ↓
Calorie Estimation
       ↓
Text File Output

---

Technologies Used

Component	Technology
Classification	TensorFlow / Keras
Few-Shot Learning	PyTorch (ProtoNet)
Backbones	ResNet50, MobileNetV2
Segmentation	CNN-based binary & multiclass models
Image Processing	OpenCV, PIL
Hardware	CUDA GPU support

---

Key Features

• ✔ End-to-end automated pipeline
• ✔ Hybrid TensorFlow + PyTorch system
• ✔ Few-shot learning for rare food classes
• ✔ Weight-aware calorie estimation
• ✔ Supports batch processing

---

🐍 Python 3.10 Virtual Environment Setup

1. Create a Virtual Environment

Make sure you have Python 3.10 installed, then run:

py -3.10 -m venv venv

2. Activate the Virtual Environment

On Windows:

venv\Scripts\activate

On Linux / macOS:

source venv/bin/activate

3. Install Project Requirements

Once the environment is active:

pip install -r requirements.txt

✅ After these steps, your environment will be isolated with Python 3.10 and all dependencies installed from requirements.txt.

---

Author

This project was developed as part of a Computer Vision course, focusing on practical AI pipelines, few-shot and one-shot learning, and real-world applications such as dietary analysis.