Skip to content

About

No description, website, or topics provided.

Resources

Contributing

Stars

0 stars

Watchers

0 watching

Forks

Repository files navigation

TrueTalent

TrueTalent is a prototype web-based proctoring system that captures webcam video and audio, computes candidate integrity metrics (eyes, head, body movement), streams annotated video to a browser, stores time-series metrics to OpenTSDB, records session audio, and performs post-session analysis using a local LLM (Ollama).

This repository contains a Python Flask backend, optional React frontend (for SPA), and several helper modules for audio recording, STT, metrics emission, and LLM orchestration.

Contents

  • web_proctor.py — Flask server and MJPEG streaming endpoint.
  • truetalent_core.py — core detection helpers (EAR, face/pose helpers).
  • analysis_runner.py — queries stored metrics, builds LLM prompt, calls local Ollama, and formats analysis HTML.
  • audio_monitor.py — audio recorder that writes WAV files per session.
  • stt.py — optional VOSK-based transcription helper.
  • metrics_sender.py / opentsdb_client.py — batching metrics sender and OpenTSDB client.
  • config.py — configuration via .env (.env.example included).
  • templates/ and static/ — fallback server templates and static assets.
  • frontend/ — optional React SPA (dev server and production build in frontend/dist).

Quickstart (development)

  1. Create and activate a virtual environment (recommended):
python -m venv .venv
source .venv/bin/activate
  1. Install Python dependencies (project supplies requirements.txt):
pip install -r requirements.txt
  1. Copy .env.example to .env and edit values (especially OLLAMA_URL, OPENTSDB_HTTP_URL, VOSK_MODEL_PATH if used):
cp .env.example .env
# edit .env
  1. If using Ollama for LLM analysis, ensure the Ollama daemon is running and the model is available:
# start ollama daemon (installed separately)
# example: ollama daemon &
# optionally pull model once:
ollama pull llama3.2:latest
  1. Run the backend server:
python web_proctor.py

By default the server listens on 0.0.0.0:8080. Open http://localhost:8080/ and start a monitor session.

Optional: Frontend dev server

  • If you're iterating on the React SPA inside frontend/, run the Vite dev server (open a separate terminal):
cd frontend
npm install
npm run dev

The dev server proxies API requests to the Flask server (see vite.config.js).

Endpoints

  • / — landing page (or SPA index)
  • /monitor/<candidate_id> — monitor page
  • /video_feed/<candidate_id> — MJPEG stream for annotated video
  • /stop_analyze/<candidate_id> (POST) — stop audio recorder and start async analysis job
  • /analysis_result/<job_id> — poll job status/result

Configuration

  • Copy and edit .env values to tune thresholds, capture size, and external endpoints. Important keys:
    • OLLAMA_URL — base URL for Ollama HTTP API (if using HTTP path) or leave default when using installed ollama lib.
    • OLLAMA_MODEL — model name used for analysis.
    • OPENTSDB_HTTP_URL — HTTP API endpoint for OpenTSDB (required for analysis_runner to query metrics).
    • VOSK_MODEL_PATH — path to a VOSK model for offline STT (optional).
    • CAPTURE_WIDTH, CAPTURE_HEIGHT, PROCESS_EVERY_N — capture and processing tuning to reduce CPU.
    • EAR_THRESH, CLOSED_SEC, AWAY_SEC — detection thresholds.

How the analysis flow works

  1. While streaming, web_proctor emits metrics into a MetricsSender which batches writes to OpenTSDB.
  2. Audio is recorded per-session to WAV files by AudioRecorder and registered so /stop_analyze can stop it.
  3. When you click Stop & Analyze, the server stops audio, optionally transcribes it (VOSK), queries OpenTSDB for the candidate's metrics, builds a natural-language prompt summarizing the metrics and transcript, and calls ollama.generate(...) (via the ollama Python package) to get a score and rationale.
  4. The model output is parsed (JSON extracted from fenced code blocks if present) and a modern HTML card is returned in the analysis job result.

Troubleshooting

  • Slow video: adjust CAPTURE_WIDTH/CAPTURE_HEIGHT and PROCESS_EVERY_N in .env to reduce CPU usage. Lower resolution and larger PROCESS_EVERY_N improve FPS.
  • Ollama 404 or errors: ensure Ollama daemon is running and OLLAMA_MODEL is available. You can ollama pull <model> once.
  • GenerateResponse JSON serialization errors: addressed — responses are sanitized to be JSON-serializable before returning.
  • STT not working: ensure VOSK_MODEL_PATH points to a downloaded VOSK model; install vosk package.
  • OpenTSDB queries failing: set OPENTSDB_HTTP_URL and verify your OpenTSDB instance exposes /api/query.

Development notes

  • The project intentionally uses dotenv-backed config.py so runtime values come from .env.
  • UI templates are in templates/ (server-side fallback). An improved React SPA is in frontend/ for a richer UX.

Security & privacy

  • This tool records webcam video and audio. Use it only in environments where you have explicit consent. Take care when storing or transmitting PII or audio.

License

  • This repository is a prototype. Add your license or NOTICE as needed.

Questions or issues

  • Open an issue or paste server log snippets (tracebacks) when reporting bugs — logs are the fastest way to diagnose runtime problems.

TrueTalent Monitor (Detailed README)

TrueTalent Monitor is a local proctoring prototype that captures webcam video and audio, runs lightweight vision-based checks (eyes, head, body movement), and streams annotated video to a web UI. It also optionally sends metrics to OpenTSDB and can perform a session analysis using a local Ollama LLM (via HTTP or LangChain) with optional speech-to-text transcripts.

This README documents the project layout, how the pieces work together, configuration options, the emitted metrics and their meanings, setup and run instructions, troubleshooting tips, and recommended next steps.

Project layout (important files)

  • truetalent_core.py — core detection code and a standalone CLI version. Contains MediaPipe/OpenCV logic used to compute EAR, head orientation, and shoulder movement.
  • web_proctor.py — Flask web app that streams annotated video (MJPEG), manages per-candidate sessions, starts/stops audio recording, queues metrics, and exposes the Stop & Analyze endpoints.
  • audio_monitor.py — AudioRecorder: records microphone input to WAV in config.AUDIO_RECORD_DIR using sounddevice + soundfile.
  • stt.py — VOSK-based STT helper. Transcribes WAV files and writes transcripts to config.TRANSCRIPT_DIR.
  • analysis_runner.py — Gathers metrics (via OpenTSDB HTTP API), loads transcript if available, builds a prompt and calls a local Ollama model (via HTTP or LangChain) to produce a JSON-style analysis (score + rationale).
  • metrics_sender.py — Background metrics queue, batching and retry logic with local SQLite persistence for failed metric writes.
  • opentsdb_client.py — Minimal helper to send put lines to OpenTSDB (telnet-style). analysis_runner uses the OpenTSDB HTTP API when available.
  • config.py — Centralized configuration for thresholds, hosts/ports, audio settings, overlay colors, etc.
  • templates/ — index.html and monitor.html for candidate ID entry and the monitor UI.
  • static/ — Static assets (logo SVG). Replace zensar_logo.svg with your branding.

High-level flow

  1. Start web_proctor.py (Flask). The index page asks for a candidate id.
  2. When the monitor page is opened, the server starts a frame generator that:
    • Captures frames from the webcam (OpenCV).
    • Runs MediaPipe face_mesh + pose to compute landmarks.
    • Calculates metrics: EAR (eyes), head horizontal ratio (looking away), shoulder movement (body movement), movement magnitude, FPS.
    • Draws overlays: eye polylines, face hull, and skeletal guidelines.
    • Encodes frames as JPEG and streams them to the browser (MJPEG).
    • Enqueues metrics to MetricsSender for best-effort send to OpenTSDB.
  3. AudioRecorder writes a WAV file for the session into config.AUDIO_RECORD_DIR.
  4. Pressing "Stop & Analyze" stops the audio recording, optionally transcribes the WAV (VOSK), queries OpenTSDB for the session metrics, and calls the local Ollama model to return a numeric trust score and short rationale.

Metrics emitted (name -> meaning)

  • truetalent.ear — Eye Aspect Ratio (float). Lower values indicate closed eyes.
  • truetalent.face_present / truetalent.face_detected — 1 if a face is detected in the frame, 0 otherwise.
  • truetalent.looking_away — 1 when head horizontal ratio indicates looking away for at least AWAY_SEC seconds.
  • truetalent.eyes_closed_alert — 1 when eyes have been closed longer than CLOSED_SEC.
  • truetalent.body_movement — 1 when shoulder movement magnitude exceeds MOVE_THRESH * diagonal.
  • truetalent.movement_magnitude — raw movement magnitude in pixels (float).
  • truetalent.fps — Frames per second reported by OpenCV.
  • truetalent.session_start and truetalent.session_end — simple markers to indicate session lifecycle (value 1).

Notes:

  • Metrics are tagged with candidate (candidate id) and DEFAULT_METRIC_TAGS from config.py.
  • MetricsSender batches and retries sends; failed batches are persisted to metrics.db (SQLite) and retried later.

Configuration (what to tweak in config.py)

  • Camera and detection thresholds:

    • CAMERA_INDEX — camera device index (default 0)
    • EAR_THRESH — threshold for EAR below which eyes are considered closed
    • CLOSED_SEC — seconds of continuous closure to trigger eyes_closed_alert
    • AWAY_SEC — seconds looking away to trigger looking_away
    • MOVE_THRESH — fraction of frame diagonal used as threshold for body movement
  • OpenTSDB and metrics:

    • OPENTSDB_ENABLED — toggle; MetricsSender can be started regardless. Use OPENTSDB_HTTP_URL for query-based analysis.
    • OPENTSDB_HOST, OPENTSDB_PORT — the telnet-style endpoint used by the opentsdb_client/MetricsSender.
    • OPENTSDB_HTTP_URL — (optional) HTTP endpoint used by analysis_runner.query_metrics (e.g., http://localhost:4242).
  • Audio & transcript:

    • AUDIO_SAMPLE_RATE — recording sample rate (default 16000)
    • AUDIO_CHANNELS — number of channels (1)
    • AUDIO_RECORD_DIR — where WAVs are saved (recordings/)
    • AUDIO_WAV_SUBTYPE — WAV subtype used when writing files (default PCM_16 which is broadly playable)
    • TRANSCRIPT_DIR — where transcripts are saved (transcripts/)
    • VOSK_MODEL_PATH — path to downloaded VOSK model (required to use stt.py)
  • LLM & analysis:

    • OLLAMA_URL — base URL for your local Ollama server (e.g. http://localhost:11434)
    • OLLAMA_MODEL — model name used with Ollama
  • Overlay customization (used by web_proctor.py):

    • OVERLAY_EYE_COLOR, OVERLAY_FACE_HULL_COLOR, OVERLAY_SKELETON_LINE_COLOR, OVERLAY_JOINT_COLOR — RGB tuples
    • OVERLAY_EYE_THICKNESS, OVERLAY_FACE_HULL_THICKNESS, OVERLAY_SKELETON_THICKNESS, OVERLAY_JOINT_RADIUS

Dependencies and platform notes

  • Python 3.10+ recommended. This project used 3.11/3.12 in development.
  • Key packages: opencv-python, mediapipe, numpy, flask, sounddevice, soundfile, vosk (for STT), requests. LangChain is optional if you want the LangChain Ollama wrapper.
  • macOS audio: install PortAudio first (brew install portaudio) before installing sounddevice.
  • VOSK: download an appropriate model (e.g., vosk-model-small-en-us-0.15) and set config.VOSK_MODEL_PATH to its directory.

Example requirements snippet (use your existing requirements.txt or update accordingly):

flask
opencv-python
mediapipe
numpy
sounddevice
soundfile
requests
vosk
langchain  # optional

How to run (quick start)

  1. Create and activate virtualenv in the project root and install requirements:
python -m venv .venv
source .venv/bin/activate
pip install -r requirements.txt
  1. (Optional) Install portaudio on macOS and download a VOSK model:
# macOS
brew install portaudio
# download a vosk model and set config.VOSK_MODEL_PATH in config.py
  1. Start the web UI:
python web_proctor.py
# open http://localhost:8080 and enter a candidate id
  1. Use the monitor page to stream annotated video. Click Stop & Analyze to stop audio recording and run the analysis job.

React frontend (optional)

We included a minimal Vite + React frontend under frontend/. It provides the candidate ID form and the Stop & Analyze UI.

To run the frontend in development mode:

cd frontend
npm install
npm run dev

The Vite dev server proxies requests (see vite.config.js) to the backend on localhost:8080 so the /video_feed/<candidate> and /stop_analyze/<candidate> endpoints work in dev.

To build a production frontend and serve from the Flask app:

cd frontend
npm install
npm run build
# then start the Flask app; it will serve files from frontend/dist
python web_proctor.py

Note: the React SPA now handles the /monitor/<candidate_id> route client-side. When the frontend build is present, visiting /monitor/<candidate_id> serves the SPA which will render the MJPEG stream.

Audio troubleshooting and playback

  • The recorder writes WAV files using AUDIO_WAV_SUBTYPE (default PCM_16). If a saved file won't play:
    • Confirm the file exists under recordings/.
    • Check server logs for audio_monitor messages: it logs file open/close and any write errors.
    • Try playing locally with afplay (macOS) or ffplay (FFmpeg):
afplay recordings/<candidate>_TIMESTAMP.wav
# or
ffplay recordings/<candidate>_TIMESTAMP.wav

If playback fails but the file exists, use the logs to find write exceptions. If needed, I can add an automatic re-encode step (to convert to PCM_16) before returning the path.

Analysis flow (Stop & Analyze)

  • When you POST to /stop_analyze/<candidate_id> (the UI calls this), the server:
    1. Stops the active recorder for that candidate and returns a job id.
    2. In the background, the job:
      • Optionally runs STT on the saved WAV (if VOSK_MODEL_PATH is configured and stt.py is available).
      • Queries OpenTSDB HTTP API (if OPENTSDB_HTTP_URL is set) for the candidate's metrics in the selected time window.
      • Builds a prompt summarizing metrics and transcript and calls the local Ollama model.
      • Stores the result in an in-memory job map which the UI polls via /analysis_result/<job_id>.

Notes on Ollama:

  • The code tries to use LangChain's Ollama wrapper if langchain and its Ollama integration are installed. Otherwise it posts to OLLAMA_URL/v1/generate. Adjust analysis_runner.call_ollama if your Ollama server expects a different payload/endpoint.

Logs and debugging

  • The server (web_proctor.py) prints log lines to the terminal. Important log messages to watch:
    • Recorder logs: Opening audio file for writing: and Finished writing audio file:.
    • STT logs and exceptions from stt.py if transcription fails.
    • Encoding errors: cv2.imencode failed for frame (size=...) — indicates a frame couldn't be encoded as JPEG.
    • Drawing exceptions: stack traces when overlay drawing fails (these are now logged with logger.exception).

Extension ideas / next steps

  • Temporal smoothing / throttling: compute hull & skeleton every N frames and reuse results in between to improve FPS.
  • Keep persistent session state (DB) for analysis jobs and transcripts rather than the in-memory job map.
  • Improve the scoring heuristic or implement a deterministic aggregation of metrics before sending to the LLM.
  • Add authentication on web endpoints and secure the Ollama/OpenTSDB endpoints.

Security & privacy

  • This project records video and audio. Use responsibly and make sure you have candidate consent and follow applicable laws and policies.

Contact / contribution

  • If you want me to implement any of the extension ideas or run a local end-to-end test with your environment (OpenTSDB/Ollama/VOSK), tell me which parts you want me to run next and I will proceed.

About

No description, website, or topics provided.

Resources

Contributing

Stars

0 stars

Watchers

0 watching

Forks

Releases

Packages

Contributors

Languages