Embodied Active Learning under Limited Annotation and Navigation Budget for Object Detection

Hadrien Crassous¹, Mohamed Yassine Kabouri², Minahil Raza³, Joni Pajarinen³, Riad Akrour¹

¹ Inria Scool, France  ·  ² LAAS-CNRS, France  ·  ³ Aalto University, Finland

Project page & results

---

Overview

We study how to adapt a computer-vision object detector to an unknown environment under a joint navigation-time and annotation budget. Framed as an embodied variant of batch active learning, our approach selects informative robot trajectories and image samples that explicitly target the detector's failure cases. We leverage spatial consistency to find these cases without external supervision: images whose labels are inconsistent across nearby views are the ones most likely to improve the model.

Each round: collect images along a budgeted trajectory, score them by prediction disagreement (Prediction Discrepancy), select the most inconsistent for an oracle to label, and retrain the detector. On AI2-THOR scenes and a real Boston Dynamics Spot robot with YOLOv5, this achieves the highest detection accuracy under the same budget, beating Entropy, Count, and Random baselines.

Repository structure

Path	Description
main.py	Main entry point — runs one embodied active-learning experiment.
constants.py, quick_inits.py	Global constants and controller/detector setup helpers.
data_collection/	Core library: agents, samplers, detector wrapper, AI2-THOR / real-world environments, metrics.
data_collection/baselines/agents/	Navigation strategies (ada, fbe, sweep, random, dqn, offline trajectory selector).
data_collection/baselines/samplers/	Sample-selection strategies (greedy, diversity, two-stage, random).
scripts/	Install, download, single-run and batch/cluster experiment launchers.
notebooks/	Analysis and plotting notebooks.

Installation

See INSTALL.md for setup instructions.

Example Usage

Run an active learning experiment

python main.py \
  --scene-name HousePlan81 \
  --model-version yolov5/yolov5s \
  --agent ada \
  --sampler two-stage \
  --ranking-method discrepancy-total-variation \
  --rounds 10 \
  --timesteps 400 \
  --samples 20 \
  --seed 0 \
  --headless --verbose

Logs are written to logs/ and per-run training results to yolo_train_logs/.

Pretrained models

main.py --model-version looks up checkpoint under models/ (e.g. models/yolov5s-procthor.pt, models/maskrcnn-procthor.pth); download them using download.py or pretrain

# YOLOv5
./scripts/yolo_pretraining.sh procthor yolov5s

# Detectron2 (Mask R-CNN); newly added but slower and without tuned hyperparams
./scripts/d2_pretraining.sh procthor

Run a batch of experiments

scripts/run_experiment.sh sweeps scenes and seeds for one configuration:

# scene_type  n_scenes  n_seeds  model           agent  sampler,ranking-method            rounds samples timesteps
./scripts/run_experiment.sh  procthor  10 10  yolov5/yolov5s  ada  greedy,discrepancy-total-variation  10  20  400

scene_type is one of kitchen, living_room, bedroom, bathroom, procthor.

Citation

Please cite:

@inproceedings{crassous2026embodied,
  author = {Crassous, Hadrien and Kabouri, Mohamed Yassine and Raza, Minahil and Pajarinen, Joni and Akrour, Riad},
  title = {{Embodied Active Learning under Limited Annotation and Navigation Budget for Object Detection}},
  booktitle = {IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS)},
  year = {2026},
}

Acknowledgements

Built on AI2-THOR, YOLOv5.