TrueTalent is a prototype web-based proctoring system that captures webcam video and audio, computes candidate integrity metrics (eyes, head, body movement), streams annotated video to a browser, stores time-series metrics to OpenTSDB, records session audio, and performs post-session analysis using a local LLM (Ollama).
This repository contains a Python Flask backend, optional React frontend (for SPA), and several helper modules for audio recording, STT, metrics emission, and LLM orchestration.
Contents
web_proctor.py— Flask server and MJPEG streaming endpoint.truetalent_core.py— core detection helpers (EAR, face/pose helpers).analysis_runner.py— queries stored metrics, builds LLM prompt, calls local Ollama, and formats analysis HTML.audio_monitor.py— audio recorder that writes WAV files per session.stt.py— optional VOSK-based transcription helper.metrics_sender.py/opentsdb_client.py— batching metrics sender and OpenTSDB client.config.py— configuration via.env(.env.exampleincluded).templates/andstatic/— fallback server templates and static assets.frontend/— optional React SPA (dev server and production build infrontend/dist).
Quickstart (development)
- Create and activate a virtual environment (recommended):
python -m venv .venv
source .venv/bin/activate- Install Python dependencies (project supplies
requirements.txt):
pip install -r requirements.txt- Copy
.env.exampleto.envand edit values (especiallyOLLAMA_URL,OPENTSDB_HTTP_URL,VOSK_MODEL_PATHif used):
cp .env.example .env
# edit .env- If using Ollama for LLM analysis, ensure the Ollama daemon is running and the model is available:
# start ollama daemon (installed separately)
# example: ollama daemon &
# optionally pull model once:
ollama pull llama3.2:latest- Run the backend server:
python web_proctor.pyBy default the server listens on 0.0.0.0:8080. Open http://localhost:8080/ and start a monitor session.
Optional: Frontend dev server
- If you're iterating on the React SPA inside
frontend/, run the Vite dev server (open a separate terminal):
cd frontend
npm install
npm run devThe dev server proxies API requests to the Flask server (see vite.config.js).
Endpoints
/— landing page (or SPA index)/monitor/<candidate_id>— monitor page/video_feed/<candidate_id>— MJPEG stream for annotated video/stop_analyze/<candidate_id>(POST) — stop audio recorder and start async analysis job/analysis_result/<job_id>— poll job status/result
Configuration
- Copy and edit
.envvalues to tune thresholds, capture size, and external endpoints. Important keys:OLLAMA_URL— base URL for Ollama HTTP API (if using HTTP path) or leave default when using installedollamalib.OLLAMA_MODEL— model name used for analysis.OPENTSDB_HTTP_URL— HTTP API endpoint for OpenTSDB (required for analysis_runner to query metrics).VOSK_MODEL_PATH— path to a VOSK model for offline STT (optional).CAPTURE_WIDTH,CAPTURE_HEIGHT,PROCESS_EVERY_N— capture and processing tuning to reduce CPU.EAR_THRESH,CLOSED_SEC,AWAY_SEC— detection thresholds.
How the analysis flow works
- While streaming,
web_proctoremits metrics into aMetricsSenderwhich batches writes to OpenTSDB. - Audio is recorded per-session to WAV files by
AudioRecorderand registered so/stop_analyzecan stop it. - When you click Stop & Analyze, the server stops audio, optionally transcribes it (VOSK), queries OpenTSDB for the candidate's metrics, builds a natural-language prompt summarizing the metrics and transcript, and calls
ollama.generate(...)(via theollamaPython package) to get a score and rationale. - The model output is parsed (JSON extracted from fenced code blocks if present) and a modern HTML card is returned in the analysis job result.
Troubleshooting
- Slow video: adjust
CAPTURE_WIDTH/CAPTURE_HEIGHTandPROCESS_EVERY_Nin.envto reduce CPU usage. Lower resolution and larger PROCESS_EVERY_N improve FPS. - Ollama 404 or errors: ensure Ollama daemon is running and
OLLAMA_MODELis available. You canollama pull <model>once. - GenerateResponse JSON serialization errors: addressed — responses are sanitized to be JSON-serializable before returning.
- STT not working: ensure
VOSK_MODEL_PATHpoints to a downloaded VOSK model; installvoskpackage. - OpenTSDB queries failing: set
OPENTSDB_HTTP_URLand verify your OpenTSDB instance exposes/api/query.
Development notes
- The project intentionally uses
dotenv-backedconfig.pyso runtime values come from.env. - UI templates are in
templates/(server-side fallback). An improved React SPA is infrontend/for a richer UX.
Security & privacy
- This tool records webcam video and audio. Use it only in environments where you have explicit consent. Take care when storing or transmitting PII or audio.
License
- This repository is a prototype. Add your license or NOTICE as needed.
Questions or issues
- Open an issue or paste server log snippets (tracebacks) when reporting bugs — logs are the fastest way to diagnose runtime problems.
TrueTalent Monitor is a local proctoring prototype that captures webcam video and audio, runs lightweight vision-based checks (eyes, head, body movement), and streams annotated video to a web UI. It also optionally sends metrics to OpenTSDB and can perform a session analysis using a local Ollama LLM (via HTTP or LangChain) with optional speech-to-text transcripts.
This README documents the project layout, how the pieces work together, configuration options, the emitted metrics and their meanings, setup and run instructions, troubleshooting tips, and recommended next steps.
truetalent_core.py— core detection code and a standalone CLI version. Contains MediaPipe/OpenCV logic used to compute EAR, head orientation, and shoulder movement.web_proctor.py— Flask web app that streams annotated video (MJPEG), manages per-candidate sessions, starts/stops audio recording, queues metrics, and exposes the Stop & Analyze endpoints.audio_monitor.py— AudioRecorder: records microphone input to WAV inconfig.AUDIO_RECORD_DIRusingsounddevice+soundfile.stt.py— VOSK-based STT helper. Transcribes WAV files and writes transcripts toconfig.TRANSCRIPT_DIR.analysis_runner.py— Gathers metrics (via OpenTSDB HTTP API), loads transcript if available, builds a prompt and calls a local Ollama model (via HTTP or LangChain) to produce a JSON-style analysis (score + rationale).metrics_sender.py— Background metrics queue, batching and retry logic with local SQLite persistence for failed metric writes.opentsdb_client.py— Minimal helper to sendputlines to OpenTSDB (telnet-style).analysis_runneruses the OpenTSDB HTTP API when available.config.py— Centralized configuration for thresholds, hosts/ports, audio settings, overlay colors, etc.templates/—index.htmlandmonitor.htmlfor candidate ID entry and the monitor UI.static/— Static assets (logo SVG). Replacezensar_logo.svgwith your branding.
- Start
web_proctor.py(Flask). The index page asks for a candidate id. - When the monitor page is opened, the server starts a frame generator that:
- Captures frames from the webcam (OpenCV).
- Runs MediaPipe face_mesh + pose to compute landmarks.
- Calculates metrics: EAR (eyes), head horizontal ratio (looking away), shoulder movement (body movement), movement magnitude, FPS.
- Draws overlays: eye polylines, face hull, and skeletal guidelines.
- Encodes frames as JPEG and streams them to the browser (MJPEG).
- Enqueues metrics to
MetricsSenderfor best-effort send to OpenTSDB.
- AudioRecorder writes a WAV file for the session into
config.AUDIO_RECORD_DIR. - Pressing "Stop & Analyze" stops the audio recording, optionally transcribes the WAV (VOSK), queries OpenTSDB for the session metrics, and calls the local Ollama model to return a numeric trust score and short rationale.
truetalent.ear— Eye Aspect Ratio (float). Lower values indicate closed eyes.truetalent.face_present/truetalent.face_detected— 1 if a face is detected in the frame, 0 otherwise.truetalent.looking_away— 1 when head horizontal ratio indicates looking away for at leastAWAY_SECseconds.truetalent.eyes_closed_alert— 1 when eyes have been closed longer thanCLOSED_SEC.truetalent.body_movement— 1 when shoulder movement magnitude exceedsMOVE_THRESH * diagonal.truetalent.movement_magnitude— raw movement magnitude in pixels (float).truetalent.fps— Frames per second reported by OpenCV.truetalent.session_startandtruetalent.session_end— simple markers to indicate session lifecycle (value 1).
Notes:
- Metrics are tagged with
candidate(candidate id) andDEFAULT_METRIC_TAGSfromconfig.py. MetricsSenderbatches and retries sends; failed batches are persisted tometrics.db(SQLite) and retried later.
-
Camera and detection thresholds:
CAMERA_INDEX— camera device index (default 0)EAR_THRESH— threshold for EAR below which eyes are considered closedCLOSED_SEC— seconds of continuous closure to triggereyes_closed_alertAWAY_SEC— seconds looking away to triggerlooking_awayMOVE_THRESH— fraction of frame diagonal used as threshold for body movement
-
OpenTSDB and metrics:
OPENTSDB_ENABLED— toggle;MetricsSendercan be started regardless. UseOPENTSDB_HTTP_URLfor query-based analysis.OPENTSDB_HOST,OPENTSDB_PORT— the telnet-style endpoint used by theopentsdb_client/MetricsSender.OPENTSDB_HTTP_URL— (optional) HTTP endpoint used byanalysis_runner.query_metrics(e.g.,http://localhost:4242).
-
Audio & transcript:
AUDIO_SAMPLE_RATE— recording sample rate (default 16000)AUDIO_CHANNELS— number of channels (1)AUDIO_RECORD_DIR— where WAVs are saved (recordings/)AUDIO_WAV_SUBTYPE— WAV subtype used when writing files (defaultPCM_16which is broadly playable)TRANSCRIPT_DIR— where transcripts are saved (transcripts/)VOSK_MODEL_PATH— path to downloaded VOSK model (required to usestt.py)
-
LLM & analysis:
OLLAMA_URL— base URL for your local Ollama server (e.g.http://localhost:11434)OLLAMA_MODEL— model name used with Ollama
-
Overlay customization (used by
web_proctor.py):OVERLAY_EYE_COLOR,OVERLAY_FACE_HULL_COLOR,OVERLAY_SKELETON_LINE_COLOR,OVERLAY_JOINT_COLOR— RGB tuplesOVERLAY_EYE_THICKNESS,OVERLAY_FACE_HULL_THICKNESS,OVERLAY_SKELETON_THICKNESS,OVERLAY_JOINT_RADIUS
- Python 3.10+ recommended. This project used 3.11/3.12 in development.
- Key packages: opencv-python, mediapipe, numpy, flask, sounddevice, soundfile, vosk (for STT), requests. LangChain is optional if you want the LangChain Ollama wrapper.
- macOS audio: install PortAudio first (brew install portaudio) before installing
sounddevice. - VOSK: download an appropriate model (e.g.,
vosk-model-small-en-us-0.15) and setconfig.VOSK_MODEL_PATHto its directory.
Example requirements snippet (use your existing requirements.txt or update accordingly):
flask
opencv-python
mediapipe
numpy
sounddevice
soundfile
requests
vosk
langchain # optional
- Create and activate virtualenv in the project root and install requirements:
python -m venv .venv
source .venv/bin/activate
pip install -r requirements.txt- (Optional) Install portaudio on macOS and download a VOSK model:
# macOS
brew install portaudio
# download a vosk model and set config.VOSK_MODEL_PATH in config.py- Start the web UI:
python web_proctor.py
# open http://localhost:8080 and enter a candidate id- Use the monitor page to stream annotated video. Click
Stop & Analyzeto stop audio recording and run the analysis job.
We included a minimal Vite + React frontend under frontend/. It provides the candidate ID form and the Stop & Analyze UI.
To run the frontend in development mode:
cd frontend
npm install
npm run devThe Vite dev server proxies requests (see vite.config.js) to the backend on localhost:8080 so the /video_feed/<candidate> and /stop_analyze/<candidate> endpoints work in dev.
To build a production frontend and serve from the Flask app:
cd frontend
npm install
npm run build
# then start the Flask app; it will serve files from frontend/dist
python web_proctor.pyNote: the React SPA now handles the /monitor/<candidate_id> route client-side. When the frontend build is present, visiting /monitor/<candidate_id> serves the SPA which will render the MJPEG stream.
- The recorder writes WAV files using
AUDIO_WAV_SUBTYPE(defaultPCM_16). If a saved file won't play:- Confirm the file exists under
recordings/. - Check server logs for
audio_monitormessages: it logs file open/close and any write errors. - Try playing locally with
afplay(macOS) orffplay(FFmpeg):
- Confirm the file exists under
afplay recordings/<candidate>_TIMESTAMP.wav
# or
ffplay recordings/<candidate>_TIMESTAMP.wavIf playback fails but the file exists, use the logs to find write exceptions. If needed, I can add an automatic re-encode step (to convert to PCM_16) before returning the path.
- When you POST to
/stop_analyze/<candidate_id>(the UI calls this), the server:- Stops the active recorder for that candidate and returns a job id.
- In the background, the job:
- Optionally runs STT on the saved WAV (if
VOSK_MODEL_PATHis configured andstt.pyis available). - Queries OpenTSDB HTTP API (if
OPENTSDB_HTTP_URLis set) for the candidate's metrics in the selected time window. - Builds a prompt summarizing metrics and transcript and calls the local Ollama model.
- Stores the result in an in-memory job map which the UI polls via
/analysis_result/<job_id>.
- Optionally runs STT on the saved WAV (if
Notes on Ollama:
- The code tries to use LangChain's Ollama wrapper if
langchainand its Ollama integration are installed. Otherwise it posts toOLLAMA_URL/v1/generate. Adjustanalysis_runner.call_ollamaif your Ollama server expects a different payload/endpoint.
- The server (
web_proctor.py) prints log lines to the terminal. Important log messages to watch:- Recorder logs:
Opening audio file for writing:andFinished writing audio file:. - STT logs and exceptions from
stt.pyif transcription fails. - Encoding errors:
cv2.imencode failed for frame (size=...)— indicates a frame couldn't be encoded as JPEG. - Drawing exceptions: stack traces when overlay drawing fails (these are now logged with
logger.exception).
- Recorder logs:
- Temporal smoothing / throttling: compute hull & skeleton every N frames and reuse results in between to improve FPS.
- Keep persistent session state (DB) for analysis jobs and transcripts rather than the in-memory job map.
- Improve the scoring heuristic or implement a deterministic aggregation of metrics before sending to the LLM.
- Add authentication on web endpoints and secure the Ollama/OpenTSDB endpoints.
- This project records video and audio. Use responsibly and make sure you have candidate consent and follow applicable laws and policies.
- If you want me to implement any of the extension ideas or run a local end-to-end test with your environment (OpenTSDB/Ollama/VOSK), tell me which parts you want me to run next and I will proceed.