IoT Based Smart Pet Feeder with Vision-Based Pet Identification

An ESP32-S3 smart cat feeder that recognises which cat has walked up to the bowl, checks that cat's own feeding rules, and then dispenses a measured portion by weight instead of by time. Every feeding session is weighed three times, logged to on-device storage, and shown on a Wi-Fi dashboard.

In a multi-cat household a shared bowl or a timer-based feeder cannot tell who ate what. This feeder closes that gap: a camera plus an on-device neural network answers who, a load cell answers how much, and a CSV database on flash answers when and how often.

---

1. What the system does

          cat approaches
                |
   HC-SR04  <= 30 cm for 10 samples (~2 s)
                |
        OV2640 captures a JPEG
                |
   MobileNetV2 (int8, 160x160) classifies the cat
                |
     confidence >= 0.70 and not "unknown"?
                |
   profile lookup: enabled? daily limit left? interval elapsed?
                |
        W1 = weigh the empty bowl
                |
  auger turns (NEMA 17 + A4988) while the load cell is polled
                |
        stop when (weight - W1) >= target portion
                |
        W2 = weigh after dispensing
                |
   wait for the cat to leave (> 30 cm held for 5 s)
                |
        W3 = weigh the settled bowl
                |
   dispensed = W2 - W1     consumed = W2 - W3     leftover = W3
                |
      append a LOG row to /littlefs/database.csv

The same cycle can be started manually from the dashboard or the serial console, in which case the trigger is recorded as MANUAL instead of SONAR.

---

2. Hardware

Block	Part	Notes
Controller	Freenove ESP32-S3-WROOM CAM (N16R8)	16 MB flash, 8 MB octal PSRAM
Camera	OV2640 on the board's DVP bus	QVGA (320x240) JPEG, quality 12, frame buffer in PSRAM
Presence sensor	HC-SR04 ultrasonic	the 5 V echo must go through a divider / level shifter
Weighing	5 kg load cell + HX711	24-bit ADC, bit-banged
Dispensing	NEMA 17 stepper + A4988 driver turning a spiral auger	200 steps/rev, full-step, 8 RPM default
Power	12 V 2 A adapter with an LM2596 buck for the 5 V / 3.3 V rails

Pin map

Taken from the #defines in src/pipeline.cpp:

Signal	GPIO	Direction
HX711 DOUT	21	input
HX711 SCK	14	output
HC-SR04 TRIG	42	output
HC-SR04 ECHO	40	input (via a 1k/2k divider from 5 V)
A4988 STEP	1	output
A4988 DIR	47	output
A4988 ENABLE	—	tied to GND (STEPPER_EN_PIN = -1)

Reserved by the board and unavailable: the camera DVP bus (4–13, 15–18), octal PSRAM/flash (26–37), USB-JTAG (19, 20), UART0 console (43, 44), on-board microSD (38–40), strapping pins (0, 3, 45, 46) and the RGB LED (48).

The banner comment at the top of pipeline.cpp still describes an older wiring (ECHO 41, STEP 4). The #defines above are what the firmware actually uses.

---

3. Vision pipeline

• Model — MobileNetV2 fine-tuned at alpha = 0.35, input 160 x 160 x 3 RGB888, quantised to int8 TensorFlow Lite Micro and embedded in the firmware as a C array.
• Classes — Cat1, Cat2, Cat3, Cat4, unknown (include/model_settings.h). Each label index maps directly onto the cat_id used by the database, so the classifier's answer can be looked up with no translation table in between.
• Two model builds ship with the project: mobilenet160_v1.h (original) and mobilenet160_v2.h (retrained, the default). Switch with -DMODEL_VERSION=1 in platformio.ini.
• Runtime — a 1 MB tensor arena is allocated in PSRAM and the interpreter is built with a 9-operator MicroMutableOpResolver (Conv2D, DepthwiseConv2D, Add, Mul, Sub, Mean, Quantize, FullyConnected, Softmax).
• Preprocessing — the JPEG is decoded to RGB888 in PSRAM, then a 160 x 160 window is cropped from the horizontal centre and the bottom edge of the frame, which is where a cat standing at the bowl appears. Frames smaller than 160 x 160 are nearest-neighbour resized instead.
• Gate — a prediction of unknown, or a best-class probability below MIN_CLASSIFICATION_CONFIDENCE (0.70), aborts the cycle and returns to IDLE. Nothing is dispensed for a cat the model is unsure about.

Training artefacts — the Keras float model, the exported cats_int8.tflite, and confusion matrices for the float and int8 validation and test splits — are kept in out copy/.

---

4. Closed-loop dispensing

A timed gate cannot hit a target mass: hopper level, kibble size and humidity all change the flow rate. So the auger runs while the scale decides when to stop.

• stepper_task pulses STEP continuously at the configured RPM on its own FreeRTOS task, keeping the pipeline task responsive.
• On every 200 ms tick, handle_dispense() takes a fast 5-sample HX711 average and computes dispensed = current - W1.
• The motor stops the moment dispensed >= target_g.
• Guards: an 8 s dispense timeout, and a stall detector (less than 0.5 g of gain across 5 bursts means a jam or an empty hopper) that latches the machine into FAULT. A latched fault clears only when an operator sends x.

Measured accuracy across the demo runs was roughly ±2 g of the target portion.

---

5. Feeding rules and the database

Each cat is a CAT row and each completed session appends a LOG row. Both live in a single 31-column CSV on LittleFS at /littlefs/database.csv, seeded from data/database.csv.

A cat is eligible only if all of the following hold (eligibility_checker()):

1. the record exists and enabled == 1;
2. eaten_today_g + portion_g <= daily_limit_g;
3. at least feeding_interval_hr has passed since the last feed.

Interval enforcement prefers the monotonic esp_timer uptime over the wall clock, so it cannot be defeated by a clock that was never set or was reset mid-life. eaten_today_g is incremented by what was actually consumed (W2 - W3), not by what was dispensed, and is cleared automatically when the calendar date rolls over.

The board has no RTC. The clock is set over serial with time <YYYY-MM-DD HH:MM:SS>, and the last known epoch is persisted to NVS so time only ever moves forward across a reboot. The HX711 tare offset and scale factor are persisted in NVS as well.

In RAM the feeding log is a 200-entry ring buffer (oldest overwritten); up to 20 cats are supported.

---

6. Web dashboard

The feeder joins an existing Wi-Fi network in station mode. Set your SSID and password in include/wifi_config.h, then read the assigned IP off the serial monitor at boot.

GET / serves a single self-contained page (compiled in as include/dashboard_html.h) carrying the live view, sensor readouts, cat database, manual feed control, a mirror of the serial console, and the feeding log.

Method	Endpoint	Purpose
GET	/	dashboard page
GET	/api/cats	cat database as JSON
GET	/api/logs	last 100 feeding records, newest first
GET	/api/status	state-machine state, active session, last message
GET	/api/sensors	distance, weight, subsystem-ready flags, stepper state
GET	/api/console	last 60 console lines
POST	/api/feed	{"cat_id":"Cat2","grams":35} — queue a manual feed
POST	/api/tare	tare the load cell with an empty tray
POST	/api/reset	rebuild the database from its seed values
GET	:81/api/camera/stream	MJPEG live view (separate server on port 81)

The HTTP server runs on its own FreeRTOS task, so every accessor into the shared cat, log and session state takes a mutex (StateLock) before touching it.

There is no authentication; the dashboard assumes a trusted local network.

---

7. Serial console

115200 baud on UART0, line-based so commands can carry arguments. Send h for the full list.

Status         h  i  b  p  l [n]  s  dist [n]
Load cell      z (tare)   r (raw average)   scale <counts>
Camera / ML    c  photo  getlast  getphotos  classify
Feeding        id <CAT_ID> | id off    feed <CAT_ID>    resetfed <CAT_ID>    x
Configuration  time <YYYY-MM-DD HH:MM:SS>   portion <CAT_ID> <g>
               limit <CAT_ID> <g>           enable <CAT_ID> <0|1>    resetdb

id <CAT_ID> forces the label the classifier returns, which is how the mechanical and weighing stages get bench-tested without a live cat in front of the lens.

---

8. Building and flashing

Requires PlatformIO. The project builds against ESP-IDF 5.3.1 through espressif32@6.13.0.

pio run -e feeder                 # build
pio run -e feeder -t upload       # flash the firmware
pio run -e feeder -t uploadfs     # flash data/ into LittleFS
pio device monitor -b 115200      # console

Set upload_port in platformio.ini to your own COM port; it is currently COM5.

esp32-camera, littlefs and esp-tflite-micro are pulled in through the IDF Component Manager via src/idf_component.yml, not through lib_deps. PlatformIO's library builder never runs a component's Kconfig, so symbols such as CONFIG_SCCB_CLK_FREQ would never reach sdkconfig.h and the camera build would fail.

The partition table (partitions_16MB_littlefs.csv) gives the app an 8 MB slot with no OTA, because the embedded TFLite model alone is about 648 KB, and hands the remaining ~7.9 MB to LittleFS for photos and the database.

A second environment, env:stepper, builds src/stepper.cpp under the Arduino framework as a standalone A4988 bring-up sketch (1 start, 0 stop, s <rpm>, d <0|1>, ?). Its build_src_filter still points at the file's old name, so fix that line before using it.

Calibration before first real use

1. Put the empty tray on the load cell and send z to tare.
2. Place a known mass of 500 g or more.
3. Send r to print the raw average.
4. scale = (raw - offset) / known_grams, then send scale <counts>.

HX711_SCALE_FACTOR defaults to 430.0 counts/g, which matches this build's cell and mounting. Recalibrate for any other cell or tray.

---

9. Repository layout

ESPFeeder/
├── src/
│   ├── main.cpp                 app_main: UART, NVS, init, 100 ms pipeline tick
│   ├── pipeline.cpp             sensors, TFLite, state machine, CSV database, console
│   ├── web_server.cpp           Wi-Fi STA, HTTP server, JSON API, MJPEG stream
│   ├── stepper.cpp              standalone Arduino A4988 bring-up sketch
│   └── main_idf_capture.cpp     camera-only reference, not compiled
├── include/
│   ├── pipeline.h, web_server.h API between the three translation units
│   ├── camera_pins.h            ESP32-S3-EYE DVP mapping used by this board
│   ├── wifi_config.h            SSID / password
│   ├── model_settings.h         input geometry and class labels
│   ├── mobilenet160*.h          embedded int8 model (v1 / v2 selectable)
│   ├── dashboard_html.h         the dashboard, as a string literal
│   └── test_image.h             embedded JPEG for camera-less inference tests
├── data/database.csv            LittleFS seed database
├── out copy/                    training artefacts and confusion matrices
├── tools/
│   ├── fetch_photos.py          pull base64 JPEGs off the board over serial
│   └── jpeg_to_test_header.py   convert a JPEG into test_image.h
├── partitions_16MB_littlefs.csv
├── sdkconfig.defaults
└── platformio.ini

---

10. Known limitations

• The classifier is trained on four specific cats plus an unknown class. It does not generalise to unseen animals, and adding a cat means new data and retraining.
• Portion accuracy depends on load-cell calibration and on the auger not jamming; a stall latches FAULT rather than silently under-feeding.
• The ultrasonic sensor reports the nearest surface, so it cannot tell a cat from any other object placed in front of the bowl.
• Without a set clock the daily-limit rollover is skipped, and only the uptime-based feeding interval applies.
• The dashboard is unauthenticated and assumes a trusted LAN.

---

Academic project

This is an academic project developed for EEE 416 — Microprocessors and Embedded Systems Laboratory, Department of Electrical and Electronic Engineering, Bangladesh University of Engineering and Technology (BUET). January 2026 · Level-4, Term-1 · Section A1 · Group 4.

Contributors

Student ID	Name
2106007	Md. Abu Saleh Akib
2106008	Md. Redowanul Hoque
2106009	Fariha Anjum Oshin
2106010	Sudipta Mondal