🧠 Crowd Behavior Analysis

Aerial Video Crowd-Behavior Classification with a CNN-LSTM Hybrid Network

A PyTorch notebook that processes drone/aerial video clips and classifies crowd behavior into four categories — aggressive, idle, normal, panic — using a CNN encoder for spatial features and an LSTM for temporal reasoning across frames.

🚀 Try the live demo on Hugging Face Spaces →

---

📑 Table of Contents

• Overview
• Live Demo
• How It Works
• Model Architecture
• Results
• Project Structure
• Getting Started
• Dataset Format
• Related Work & Datasets
• Known Issues
• Team

---

🔍 Overview

This project was built as an ANN/deep-learning coursework project (NUST MCS, Semester 6). It takes short aerial video clips of crowds, breaks each clip into 16-frame sequences, and feeds them through a CNN → LSTM → Fully Connected pipeline to predict one of four crowd-behavior classes:

Class	Label
🟠	Aggressive
⚪	Idle
🟢	Normal
🔴	Panic

Everything — data loading, model definition, training, evaluation and ad-hoc inference — lives in a single notebook: Crowd_Behavior_Analysis_CNN_LSTM.ipynb.

---

🎮 Live Demo

A Gradio-based demo of this project is deployed on Hugging Face Spaces:

👉 huggingface.co/spaces/usman-khn/Crowd-Behavior-Detection

Upload a clip or a 16-frame image sequence and get a predicted crowd-behavior class without running the notebook locally. Below is a real run against the live Space using one of this repo's test sequences:

1. Upload screen	2. 16-frame sequence uploaded	3. Prediction: aggressive (97%)

---

⚙️ How It Works

flowchart LR
    A["🎥 Raw Drone / CCTV Video"] --> B["Frame Extraction<br/>(OpenCV, 1 frame/sec)"]
    B --> C["Frame Renaming &<br/>Ordering"]
    C --> D["16-Frame Sequence<br/>Resize 64×64 · ToTensor"]
    D --> E["CNN Encoder<br/>(per-frame features)"]
    E --> F["LSTM<br/>(temporal aggregation)"]
    F --> G["Fully Connected + Softmax"]
    G --> H{{"Predicted Behavior"}}
    H --> H1["Aggressive"]
    H --> H2["Idle"]
    H --> H3["Normal"]
    H --> H4["Panic"]

Loading

1. Frame extraction — extract_frames() pulls frames from a source video at a fixed interval using OpenCV.
2. Dataset assembly — CrowdBehaviorDataset walks class-labeled folders, groups frames into non-overlapping 16-frame clips, and resizes/tensorizes them.
3. Spatial encoding — a small 2-layer CNN extracts features from every frame independently.
4. Temporal reasoning — an LSTM consumes the per-frame feature sequence and summarizes it into a single hidden state.
5. Classification — a linear layer maps the final LSTM hidden state to 4 class logits.

---

🏗 Model Architecture

flowchart TB
    I["Input Clip<br/>(B, T=16, C=3, H=64, W=64)"] --> R["Reshape → (B·T, 3, 64, 64)"]
    R --> C1["Conv2d 3→32, k3, pad1<br/>ReLU → MaxPool2d(2)"]
    C1 --> C2["Conv2d 32→64, k3, pad1<br/>ReLU → MaxPool2d(2)"]
    C2 --> RS["Reshape → (B, T, 64×16×16)"]
    RS --> L["LSTM<br/>hidden_size=128, batch_first"]
    L --> LAST["Take last timestep<br/>(B, 128)"]
    LAST --> FC["Linear 128 → 4"]
    FC --> OUT["Class logits<br/>aggressive · idle · normal · panic"]

Loading

📋 Full torchinfo summary

==========================================================================================
Layer (type:depth-idx)                   Output Shape              Param #
==========================================================================================
CNNLSTM                                  [1, 4]                    --
├─Sequential: 1-1                        [16, 64, 16, 16]          --
│    └─Conv2d: 2-1                       [16, 32, 64, 64]          896
│    └─ReLU: 2-2                         [16, 32, 64, 64]          --
│    └─MaxPool2d: 2-3                    [16, 32, 32, 32]          --
│    └─Conv2d: 2-4                       [16, 64, 32, 32]          18,496
│    └─ReLU: 2-5                         [16, 64, 32, 32]          --
│    └─MaxPool2d: 2-6                    [16, 64, 16, 16]          --
├─LSTM: 1-2                              [1, 16, 128]              8,455,168
├─Linear: 1-3                            [1, 4]                    516
==========================================================================================
Total params: 8,475,076
Trainable params: 8,475,076
Non-trainable params: 0
==========================================================================================

Hyperparameters

Hyperparameter	Value
Batch size	16
Max epochs	10
Learning rate	0.0005 (Adam)
Early stopping patience	5 epochs
Sequence length	16 frames/clip
Frame size	64 × 64
Loss function	CrossEntropyLoss
Classes	4
Total parameters	8,475,076

---

📊 Results

The dataset used in this run contained 72 sequences total, split 70 / 15 / 15 into train / validation / test (50 / 10 / 12 clips).

Split	Accuracy	Samples
🏋️ Train	100.00%	50
🧪 Validation	90.00%	10
✅ Test	100.00%	12

With only 12 test clips, these numbers are a proof of concept, not a statistically robust benchmark — see Known Issues.

Train set	Validation set

_{Test set}

⚠️ Note: the axis labels on the plots above are generated by a buggy target_names order in the notebook's evaluate_model() — the "panic" and "normal" tick labels are swapped. The model's actual predictions are correct; only the plot legend is mislabeled. Details in Known Issues.

---

📁 Project Structure

CrowdBehaviourAnalysis/
├── Crowd_Behavior_Analysis_CNN_LSTM.ipynb   # Main notebook — data, model, training, evaluation, inference
├── Crowd_Behavior_Analysis_CNN_LSTM.pdf     # Static PDF export of the notebook
├── Group Members names.txt                  # Team roster
├── assets/                                   # Images used in this README
│   ├── train_confusion_matrix.png
│   ├── val_confusion_matrix.png
│   └── test_confusion_matrix.png
└── README.md

---

🚀 Getting Started

1. Install dependencies

pip install torch torchvision torchinfo opencv-python pillow numpy matplotlib scikit-learn tqdm

2. Point the notebook at your dataset

Open the notebook and update the config cell:

DATA_DIR = r"path/to/crowd_behavior_data"   # see Dataset Format below
BATCH_SIZE = 16
EPOCHS = 10
LEARNING_RATE = 0.0005
SEQUENCE_LENGTH = 16

3. Run it

Run the notebook top to bottom. Training checkpoints the best model to best_model.pth whenever validation loss improves, and stops early if it doesn't improve for 5 straight epochs. If best_model.pth already exists, the notebook loads it instead of retraining.

4. Run inference on a new clip

The "Testing on External Inputs" section loads a folder of exactly 16 ordered frames and predicts its class:

TEST_SEQ_DIR = r"path/to/16_frame_folder"
MODEL_PATH = "best_model.pth"

---

🗂 Dataset Format

CrowdBehaviorDataset expects one folder per class, each containing ordered frame images. Folders are sorted alphabetically to assign label indices:

crowd_behavior_data/
├── aggressive/   → label 0
│   ├── frame_0001.jpg
│   ├── frame_0002.jpg
│   └── ...
├── idle/         → label 1
├── normal/       → label 2
└── panic/        → label 3

Frames within each class folder are grouped into consecutive, non-overlapping windows of SEQUENCE_LENGTH (16) frames — each window becomes one training sample. Use the notebook's extract_frames() and rename_files_with_extension() helpers to turn a raw video into a properly named frame folder.

---

📚 Related Work & Datasets

This project trains on a custom, self-collected aerial-video dataset (not included in this repo). For readers looking for large-scale public benchmarks on drone-based crowd analysis:

• VisDrone / DroneCrowd — a large-scale drone-crowd benchmark with 112 video sequences (33,600 frames at 1920×1080) across 70 scenarios, ~20,800 trajectories and 4.8M annotated head points. It targets density-map estimation, person localization, and trajectory tracking (rather than clip-level behavior classification) via the proposed STANet/STNNet architectures.

  @inproceedings{dronecrowd_cvpr2021,
    author    = {Longyin Wen and Dawei Du and others},
    title     = {Detection, Tracking, and Counting Meets Drones in Crowds: A Benchmark},
    booktitle = {CVPR},
    year      = {2021}
  }

---

⚠️ Known Issues

• Confusion-matrix label swap — evaluate_model() plots use target_names=["aggressive", "idle", "panic", "normal"], but the true label order (fixed by sorted(os.listdir(DATA_DIR))) is aggressive, idle, normal, panic. This swaps the "panic" and "normal" names in the printed classification report and confusion matrix plots. The inference cells further down use the correct order.
• Hardcoded local paths — DATA_DIR, TEST_SEQ_DIR, and the frame-extraction example paths all point to a specific machine (C:\Users\hp\Desktop\...) and need to be changed before running elsewhere.
• No dataset included — the crowd_behavior_data folder isn't part of this repo, so the notebook can't be re-run end-to-end without supplying your own clips.
• No environment/requirements file — dependencies are inferred from the notebook's imports (see Getting Started).
• Small dataset — 72 total sequences (12 in the test split) means the reported accuracy is indicative, not a robust benchmark.

---

👥 Team

Group project for NUST — MCS, Semester 6.

#	Name	Role
1	Muhammad Usman	Team Lead
2	Muhammad Aitazaz Ahsan	Member
3	Imran Tahir	Member
4	Shaheer Khan	Member
5	Naveed Ahmad	Member
6	Nimra Muqaddas	Member

(Full contact details in Group Members names.txt.)