SceneMotion AI - Video Sentiment Analysis

SceneMotion AI is a local Python application for short-video sentiment analysis. It analyzes the first 15 seconds of an uploaded video and returns a heuristic Negative, Neutral, or Positive result with a per-signal breakdown.

The active app is a six-signal fusion system. It combines pretrained vision and speech models with deterministic audio, color, motion, and temporal rules. It is intended for experimentation and explainable demos, not for clinical, psychological, safety, or identity inference.

Features

• Local Streamlit interface for video upload and analysis
• Six-signal fusion across visual, speech, acoustic, color, motion, and temporal cues
• CLIP zero-shot visual matching for scene-level semantics
• Whisper transcription plus DistilBERT text sentiment for spoken content
• Librosa audio-feature heuristics for acoustic tone
• OpenCV color and optical-flow analysis
• Per-signal vote bars and effective fusion weights
• Unit tests for the active rule-based pipeline

How The Analyzer Works

Video file
  |
  |-- MoviePy extracts up to 15 seconds of media
  |-- 16 frames are sampled for visual analysis
  |-- audio is extracted to a temporary WAV file when available
  |
  |-- Spatial signal: CLIP zero-shot image/text matching
  |-- Speech signal: Whisper transcription + DistilBERT sentiment
  |-- Acoustic signal: Librosa waveform and prosody heuristics
  |-- Color signal: HSV brightness and hue heuristics
  |-- Motion signal: optical-flow activity signal
  |-- Temporal signal: cut/editing-style signal
  |
  |-- fixed-weight fusion with abstentions
  |
  +-- Negative / Neutral / Positive result

Each signal produces a distribution over:

Negative, Neutral, Positive

A uniform 33% / 33% / 33% distribution represents abstention or no useful preference, not a confident neutral prediction.

Signal Weights

Signal	Implementation	Base weight
Spatial	OpenCLIP ViT-B-32 zero-shot image/text matching	40%
Speech	Whisper base transcription + DistilBERT sentiment	10%
Acoustic	Librosa audio-feature rules	25%
Color	OpenCV HSV color rules	5%
Motion	Optical-flow analysis	15%
Temporal	Editing and frame-change cues	5%

If a signal abstains, it receives no effective weight and the remaining usable weights are normalized to 100%.

Project Structure

.
|-- app.py                         # Main Streamlit application
|-- six_pillar_*.py                # Modular pillar implementations
|-- utils/                         # Frame, audio, dataset, and signal helpers
|-- ui/streamlit_app.py            # Streamlit UI module
|-- tests/                         # Pytest suite
|-- scripts/                       # Cache and semantic-analysis utilities
|-- tools/weight_optimizer/        # Weight tuning helper
|-- requirements.txt               # Runtime dependencies
|-- requirements-dev.txt           # Development/test dependencies
|-- PROJECT_CONTEXT.md             # Additional project notes
|-- CHANGELOG.md                   # Change history

Some files such as train_pipeline.py, run_pipeline.py, api/fastapi_app.py, and related training scripts are research/legacy workflows for dataset-backed model experiments. They may require datasets, checkpoints, or package paths that are not needed for the main Streamlit app.

Requirements

• Python 3.8 or newer
• FFmpeg available on your system path
• Internet access for the first run so pretrained model weights can be downloaded
• A CPU is supported; CUDA is used automatically when available through PyTorch

The first run may download model weights for OpenCLIP, Whisper, and Hugging Face Transformers. After those files are cached locally, normal analysis can run offline.

Installation

From the project root:

python -m venv .venv
.\.venv\Scripts\Activate.ps1
python -m pip install --upgrade pip
python -m pip install -r requirements.txt

For development and tests:

python -m pip install -r requirements-dev.txt

Run The App

python -m streamlit run app.py

Then open the Streamlit URL shown in the terminal, usually:

http://localhost:8501

Upload an mp4, mov, avi, or mkv file. Longer files are accepted, but only the first 15 seconds are analyzed.

Run Tests

python -m pytest -q

Focused examples:

python -m pytest tests/test_rule_based_pillars.py -q
python -m pytest tests/test_six_pillar_pipeline.py -q

Important Limitations

• Scores are heuristic fusion scores, not calibrated probabilities.
• The project does not infer a person's true emotion, intent, mental health, or safety risk.
• Visual prompts, text sentiment, acoustic cues, and color rules are context-dependent and can be wrong.
• Performance claims should only be made after evaluating this exact version on a labeled, held-out dataset that matches the intended use case.

Troubleshooting

FFmpeg or video decoding errors

Install FFmpeg and make sure it is available on your PATH.

First run is slow

The first run downloads and caches pretrained model weights. Later runs should start faster.

Out-of-memory errors

Use a smaller video, close other GPU-heavy applications, or run on CPU. The app automatically selects CUDA when PyTorch reports that it is available.

No speech signal

The speech pillar abstains when audio is missing, too short, has no detected speech, has a low-quality transcript, or produces a weak text-sentiment result.

References

• Radford et al. (2021), Learning Transferable Visual Models From Natural Language Supervision
• Radford et al. (2022), Robust Speech Recognition via Large-Scale Weak Supervision
• Russell (1980), A Circumplex Model of Affect
• Ekman (1992), An Argument for Basic Emotions
• Mehrabian and Ferris (1967), Inference of Attitudes from Nonverbal Communication in Two Channels

License

See LICENSE.