✨Self in Space✨: Benchmarking Self-Awareness and Spatial Cognition in UAV Embodied Intelligence

Zhishan Zou¹, Guoyan Sun¹, Zhiwei Wei², Jiancheng Pan³, Yujie Li¹, Mugen Peng¹, Wenjia Xu^1†
†Corresponding author
¹Beijing University of Posts and Telecommunications   ²Hunan Normal University   ³Tsinghua University
ACM MM 2026

         

Self-in-Space: We study spatial intelligence in embodied UAV scenarios from two complementary perspectives — Space and Self. We introduce SIS-Bench, SIS-Motion-54K, and SIS-Motion: a benchmark, a training dataset, and a motion-aware model for spatial reasoning.

📢 News

• 🎉 [2026/9/05] Our open-source models and datasets have accumulated over 4 million downloads.
• 🎉 [2026/7/16] Our open-source models and datasets have accumulated over 500k downloads.
• 🔥 [2026/7/14] Our paper is available on arXiv.
• 😊 [2026/7/13] We release our benchmarking dataset SIS-Bench on Hugging Face and ModelScope.
• 😊 [2026/7/13] We release our training dataset SIS-Motion-54K on Hugging Face and ModelScope.
• 😊 [2026/7/13] We release our downstream task dataset OpenUAV-QA on Hugging Face and ModelScope.
• 😊 [2026/7/13] We release our model SIS-Motion on Hugging Face and ModelScope.
• 🎉 [2026/7/10] Our paper is accepted by ACM MM 2026!

🌟 Overview

🍏 SIS-Bench

SIS-Bench: We introduce a benchmark of 4,856 QA pairs from 1,646 real-world UAV videos, evaluating embodied spatial intelligence across two dimensions (spatial cognition and self-awareness) and three cognitive levels (perception, memory, and reasoning).

🏗️ Pipeline

Pipeline: SIS-Bench is built through a four-stage task-conditioned protocol: heterogeneous video processing, task-specific annotation, LLM-assisted QA construction, and dual-expert verification, ensuring scalability and reliability across 13 tasks.

✈️ SIS-Motion

SIS-Motion: A motion-aware extension of a video MLLM that fuses optical-flow-based motion cues with visual features, jointly capturing environmental context and agent dynamics to improve both spatial cognition and self-awareness.

⚙️ Getting Started

🎹 Installation

Requires Python 3.10, PyTorch 2.6 / CUDA 12.4, and an NVIDIA GPU. LoRA and SIS-Motion use separate environments; both include FlashAttention (built from source only when no compatible wheel exists, which needs a CUDA toolkit and CUDA_HOME).

git clone https://github.com/IntelliSensing/Self-in-Space.git
cd Self-in-Space

bash scripts/setup_conda.sh lora      # conda env: sis-motion-lora
bash scripts/setup_conda.sh motion    # conda env: sis-motion-motion
# or with uv: bash scripts/setup_uv.sh {lora|motion}  → .venv-{lora|motion}/

To use a different CUDA stack, edit the definitions under environments/.

📊 Data Preparation

python scripts/download_data.py                  # all datasets
python scripts/download_data.py sis_bench        # or: sis_motion_54k, openuav_qa

Data goes to data/ by default; set SIS_DATA_ROOT to use another directory and check it with python scripts/data_registry.py validate. Paths are defined in configs/data_registry.json.

data/
├── SIS-Bench/           SIS-Bench.jsonl (4,856 QA) + video/{AirScape,UrbanVideo,VisDrone}
├── SIS-Motion-54K/      SIS-Motion-54K.jsonl + AirScape_dataset/
└── OpenUAV-QA/          TravelUAV_test.jsonl (3,895 QA) + TravelUAV_dataset/

Videos may be raw files (.mp4, .mov, .avi, .mkv, .webm) or ordered frame directories (.jpg, .jpeg, .png); a sample/ directory next to a referenced sample.mp4 is also accepted. OpenUAV-QA uses frame directories only.

Note: The paper pre-samples videos with the Qwen2.5-VL sampling strategy (see the appendix). Using raw MP4 inputs may give slightly different results; use the same input format when comparing against baselines.

🚀 Training

Visual-only LoRA baseline:

conda activate sis-motion-lora
CUDA_VISIBLE_DEVICES=0,1 \
MODEL_NAME_OR_PATH=Qwen/Qwen2.5-VL-7B-Instruct \
OUTPUT_DIR=$PWD/lora/output/qwen25vl-7b-lora \
bash scripts/train_lora.sh

SIS-Motion: first download MOF_kitti.pth to checkpoints/VideoFlow/MOF_kitti.pth (or set VIDEOFLOW_CKPT).

conda activate sis-motion-motion
CUDA_VISIBLE_DEVICES=0,1,2 \
PRETRAINED_MODEL_NAME_OR_PATH=Qwen/Qwen2.5-VL-7B-Instruct \
OUTPUT_DIR=$PWD/motion/output/sis-motion \
bash scripts/train_motion.sh

Both use DeepSpeed ZeRO-2 on all visible GPUs and train on SIS-Motion-54K (DATASET_USE=sis_motion_54k). Set SWANLAB_ENABLED=1 for SwanLab tracking.

📊 Evaluation

Set BENCHMARK=openuav_qa on any command below to evaluate on OpenUAV-QA instead of SIS-Bench (default).

# LoRA baseline, or a base Qwen2/2.5/3-VL model zero-shot (pass its HF ID)
conda activate sis-motion-lora
MODEL_PATH=$PWD/lora/output/qwen25vl-7b-lora bash scripts/eval_lora.sh
MODEL_PATH=Qwen/Qwen2.5-VL-7B-Instruct      bash scripts/eval_lora.sh

# SIS-Motion
conda activate sis-motion-motion
MODEL_PATH=choucsan/SIS-Motion bash scripts/eval_motion.sh          # from Hugging Face
bash scripts/eval_motion.sh                                          # local model/ directory
MODEL_DIR=$PWD/motion/output/sis-motion bash scripts/eval_motion.sh  # final model, else latest checkpoint

The SIS-Motion package (adapter_config.json, adapter_model.safetensors, connector_weights.pt, MOF_kitti.pth, sis_motion_config.json) does not include the Qwen2.5-VL-7B base weights; they are downloaded automatically from base_model_name_or_path.

📚 Citation

@misc{zou2026sis,
      title={Self in Space: Benchmarking Self-Awareness and Spatial Cognition in UAV Embodied Intelligence},
      author={Zhishan Zou and Guoyan Sun and Zhiwei Wei and Jiancheng Pan and Yujie Li and Mugen Peng and Wenjia Xu},
      year={2026},
      eprint={2607.12477},
      archivePrefix={arXiv},
      primaryClass={cs.CV},
      url={https://arxiv.org/abs/2607.12477},
}

📖 Acknowledgements

• Code: Qwen2.5-VL and Qwen3-VL, Spatial-MLLM, VideoFlow
• Infra: Transformers, PEFT, DeepSpeed
• Data: AirScape, UrbanVideo-Bench, VisDrone, OpenUAV

📮 Contact

For questions, corrections, or collaboration requests: choucisan@gmail.com