Skip to content

Social Explorer

Tests License: Apache-2.0

A social media research platform: collect posts from several platforms, load them into one analytics database, and explore them through dashboards, investigations and notebooks.

Status: early, pre-1.0. The platform is under active development. Interfaces and the database schema may change, and some modules (for example the workflow builder) are experimental. Run it on a trusted network until you have reviewed the security notes.

What it does

  • Collect. A Go crawler gathers Telegram, YouTube and Bluesky content on a schedule; ETL pipelines load datasets such as GDELT; a radio service records and transcribes broadcasts.
  • Analyse. Sentiment and emotion, topic modelling, language detection, engagement, GDELT events and Android usage analytics, with cross-module views over the same posts.
  • Explore. Interactive dashboards (ECharts, Leaflet), keyword-highlighted investigations of transcripts, and embedded Jupyter notebooks.
  • Organise. Projects and workspaces keep datasets separate. Accounts have admin, user and viewer roles.
  • Share. Give a collaborator, a reviewer or a journal one content ID for the dataset behind your findings, and anyone with it can fetch the same package: an open Frictionless Data Package (a plain CSV plus a schema) published over the IPFS peer-to-peer network, which opens in Python, R or Excel and always resolves to the same bytes, so a paper can cite the exact version published. Import a colleague's package by ID into a table you can query. Off by default and admin-only: anything published is public, stays available only while a node holds a copy, and cannot be withdrawn, so it is for open, anonymised data only. See Data sharing over IPFS.

Architecture

flowchart LR
    Browser --> UI["Web app<br/>SvelteKit"]
    UI --> PG[("PostgreSQL<br/>+ pg_duckdb")]
    UI -- Dapr --> Redis[("Redis<br/>state / pub-sub")]
    Crawler["Crawler (Go)"] --> Redis
    Crawler --> PG
    ETL["ETL (Python)"] --> PG
    Radio["WavePulse radio service"] --> PG
    Jupyter --> PG
Loading
Directory Role
ex-accell-ui/ SvelteKit 2 / Svelte 5 web app and API (TypeScript, Tailwind CSS 4)
etl/ Python data pipelines
crawler/ Go crawler for Telegram, YouTube and Bluesky
wavepulse/ Radio recording and transcription, derived from NYU WavePulse
jupyter/ Jupyter Lab environment
deploy/, docker-compose.yml Dapr components and the compose stack
docs-site/ Documentation site source (VitePress)

Try it in one command

Needs Node 22 and Docker. This starts a throwaway database, loads synthetic demo data about a fictional city and runs the app on http://localhost:5173, reachable only from your machine:

git clone https://github.com/spiculedata/social-explorer.git
cd social-explorer/ex-accell-ui
npm run demo

The sign-in is printed in the terminal. npm run demo -- --stop stops the database and npm run demo -- --reset deletes it (then run npm run demo for a fresh one).

Quickstart

To run the full stack yourself, you need Docker with Compose v2.

git clone https://github.com/spiculedata/social-explorer.git
cd social-explorer

cp .env.example .env
# Edit .env: set SESSION_SECRET (openssl rand -hex 32), ADMIN_PASSWORD (12+ characters)
# and change POSTGRES_PASSWORD.

docker compose up -d --build postgres redis dapr-placement ui ui-dapr

Open http://localhost:3000 and sign in with ADMIN_EMAIL / ADMIN_PASSWORD from your .env. The first admin account is created automatically when the user table is empty.

The command above starts only the core stack. The remaining services in docker-compose.yml (Ollama, Jupyter, WavePulse, the crawler, ETL) are optional and need extra setup; see the documentation.

The first sign-in creates the database schema. A fresh database has no data, so dashboards show empty states until you load some; npm run seed:demo loads a synthetic demo dataset (see Try it with demo data).

Troubleshooting

  • port is already allocated for 50006. A local dapr init already runs a container on that port. Stop it (docker stop dapr_scheduler) or leave dapr-placement out of the command; the web app works without it, but Dapr features will not.
  • SESSION_SECRET error on start. The secret is required. Generate one with openssl rand -hex 32 and put it in .env.
  • Login does not stick over plain HTTP. Keep COOKIE_SECURE=false for http://localhost, and set it to true only when serving over HTTPS.
  • Data lives in ./data. Set DATA_DIR in .env to move it. Remove the directory to reset.

Development

cd ex-accell-ui
npm ci
npm run dev        # needs a PostgreSQL with pg_duckdb; see CONTRIBUTING.md
npm run test:unit  # unit tests (Vitest)
npm run check      # type check

CONTRIBUTING.md covers the database container for local development, the full set of checks and commit conventions.

Documentation

Security

Please report vulnerabilities privately; see SECURITY.md.

License

Apache License 2.0. See LICENSE and NOTICE. The wavepulse/ directory is derived from NYU's WavePulse and carries its own notices, including a note about a pending patent application; read wavepulse/NOTICE before commercial use.

Releases

Packages

Contributors

Languages