Skip to content
Open
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
248 changes: 80 additions & 168 deletions README.md
Original file line number Diff line number Diff line change
@@ -1,140 +1,75 @@
# TelemetryTaco

TelemetryTaco is a lightweight self-hosted telemetry MVP built around three concrete workflows:

- capture single events or batches
- inspect recent events in a live dashboard
- query minute-level insight aggregates over a recent lookback window

The codebase now targets a strong single-project MVP rather than a broad PostHog clone. The refactor in this repo keeps the current API contract intact while adding real batching, idempotency, generated frontend types, tests, and a cleaner developer workflow.

## Current Architecture

```text
Python SDK / API clients
|
v
Django + Django Ninja
|
v
Celery batch task queue
|
v
PostgreSQL event store
|
v
React dashboard (React Query + generated OpenAPI types)
```

### Runtime responsibilities

- `backend/`: API surface, ingestion service, selectors, Celery tasks, retention commands
- `frontend/`: dashboard UI, React Query polling, OpenAPI-generated TypeScript types
- `sdk/`: queue-backed Python client that batches to `/api/capture/batch`

## What Exists Today

- `POST /api/capture`: additive single-event capture endpoint
- `POST /api/capture/batch`: batch capture endpoint used by the SDK
- `GET /api/events`: bounded recent-event feed with optional `before` cursor
- `GET /api/insights`: bounded minute-level aggregate series
- `GET /api/health/live` and `GET /api/health/ready`
- event idempotency via caller-supplied `event_uuid`
- OpenAPI export and generated frontend types
- backend pytest coverage, frontend Vitest coverage, and SDK tests
<div align="center">

## Quick Start

### Prerequisites

- Python 3.11, 3.12, or 3.13
- Poetry
- Node.js 18+ and `pnpm`
- Docker and Docker Compose
Lightweight, self-hosted telemetry for queued event ingestion, live event inspection, and minute-level insights.

### Local development
[![CI](https://github.com/Agile-Flimflam/TelemetryTaco/actions/workflows/cicd.yml/badge.svg)](https://github.com/Agile-Flimflam/TelemetryTaco/actions/workflows/cicd.yml)
![Python 3.11-3.13](https://img.shields.io/badge/python-3.11--3.13-3776AB?logo=python&logoColor=white)
![Django 5](https://img.shields.io/badge/django-5.0-092E20?logo=django&logoColor=white)
![React 18 + Vite](https://img.shields.io/badge/react-18%20%2B%20vite-149ECA?logo=react&logoColor=white)
![PostgreSQL + Redis](https://img.shields.io/badge/postgresql%20%2B%20redis-runtime-3B82F6)

1. Copy backend environment defaults:
[Quick Start](#quick-start) • [Architecture](#architecture) • [Usage](#usage) • [Repo](#repo)

```bash
cp backend/.env.example backend/.env
```
</div>

2. Start the database and Redis:
## Why It Exists

```bash
docker-compose up -d db redis
```
- Queued batch ingestion with idempotent event persistence.
- Live recent-event streaming plus minute-level insight aggregates.
- OpenAPI-generated frontend types across a tested monorepo.

3. Start the application stack:
## Architecture

```bash
./start.sh
```mermaid
flowchart LR
A[Python SDK / API clients] --> B[Django + Django Ninja API]
B --> C[Celery worker]
B <--> D[(Redis cache)]
C --> E[(PostgreSQL event store)]
F[React dashboard] <--> B
```

That script will install backend dependencies, run migrations, start Django and Celery in the background, and run the frontend in the foreground.

### Useful commands

```bash
pnpm generate:api-types # export backend OpenAPI and regenerate frontend types
pnpm validate:backend # Ruff + format check + Django check + backend pytest
pnpm validate:frontend # OpenAPI type generation + lint + type-check + Vitest
pnpm validate:all # backend + frontend + SDK validation
pnpm test # backend + frontend + SDK tests
pnpm seed # seed realistic sample events
pnpm seed:clean # wipe and reseed events
```mermaid
sequenceDiagram
participant Client as SDK / caller
participant API as Django API
participant Worker as Celery worker
participant DB as PostgreSQL
participant UI as React dashboard

Client->>API: POST /api/capture or /api/capture/batch
API->>Worker: enqueue normalized event batch
Worker->>DB: persist unique events
UI->>API: GET /api/events
UI->>API: GET /api/insights
API->>UI: recent events + minute buckets
```

## Backend Notes

The backend defaults to development settings via `telemetry_taco.settings`.

Available settings modules:

- `telemetry_taco.settings.development`
- `telemetry_taco.settings.test`
- `telemetry_taco.settings.production`

Important environment variables:

- `DATABASE_URL`
- `REDIS_URL`
- `CACHE_URL`
- `MAX_CAPTURE_BATCH_SIZE`
- `MAX_EVENTS_LIMIT`
- `MAX_INSIGHTS_LOOKBACK_MINUTES`
- `EVENT_RETENTION_DAYS`

Retention cleanup is exposed as a management command:

```bash
cd backend
poetry run python manage.py purge_expired_events
```
## Quick Start

OpenAPI export is also explicit:
Prereqs: Python 3.11-3.13, Poetry, pnpm, Docker.

```bash
cd backend
DJANGO_SETTINGS_MODULE=telemetry_taco.settings.test poetry run python manage.py export_openapi_schema ../frontend/openapi.json
pnpm install
cp backend/.env.example backend/.env
pnpm services
pnpm dev
Comment on lines +57 to +58
# optional
pnpm seed
```

## Frontend Notes
Local endpoints:

The dashboard is a Vite React app that uses:
- Frontend: `http://localhost:5173`
- API: `http://localhost:8000`

- React Query for polling, deduping, and error handling
- lazy loading for the chart surface
- generated API types from `frontend/openapi.json`
`pnpm dev` runs the Django API, Celery worker, and Vite frontend. Seed data is available through `pnpm seed` or `pnpm seed:clean`.

If the backend contract changes, regenerate types before committing:

```bash
pnpm generate:api-types
```
## Usage

## SDK Example
Python SDK:

```python
from telemetry_taco import TelemetryTaco
Expand All @@ -147,68 +82,45 @@ with TelemetryTaco(base_url="http://localhost:8000") as client:
)
```

The SDK batches events in a background worker, attaches `event_uuid` and `sent_at`, and flushes automatically when the context manager exits.

## API Summary

### `POST /api/capture`

```json
{
"distinct_id": "user-123",
"event_name": "page_view",
"properties": {
"path": "/"
},
"event_uuid": "optional-uuid",
"sent_at": "YYYY-MM-DDTHH:MM:SSZ"
}
```

Response:

```json
{
"status": "ok"
}
```
Batch capture:

### `POST /api/capture/batch`

```json
{
"events": [
{
"distinct_id": "user-123",
"event_name": "page_view"
}
]
}
```bash
curl -X POST http://localhost:8000/api/capture/batch \
-H 'Content-Type: application/json' \
-d '{"events":[{"distinct_id":"user-123","event_name":"page_view","properties":{"path":"/"}}]}'
```

### `GET /api/events?limit=100&before=YYYY-MM-DDTHH:MM:SSZ,EVENT_ID`
API surface:

Returns recent events ordered by `timestamp desc, id desc`.
For stable pagination, set `before` to the last event's `timestamp,id` pair.
Plain ISO 8601 timestamps are still accepted for backward compatibility.
| Endpoint | Purpose |
| --- | --- |
| `POST /api/capture` | Capture one event |
| `POST /api/capture/batch` | Capture a batch of events |
| `GET /api/events` | Read the recent event feed |
| `GET /api/insights` | Read minute-level aggregates |
| `GET /api/health/live` | Liveness probe |
| `GET /api/health/ready` | Database and cache readiness |

### `GET /api/insights?lookback_minutes=60`
## Repo

Returns minute buckets shaped like:
| Path | Responsibility |
| --- | --- |
| `backend/` | Django API, ingestion, Celery tasks, retention, tests |
| `frontend/` | React dashboard, React Query polling, generated API types |
| `sdk/` | Python client with background batching and flush-on-close |

```json
[
{ "time": "18:04", "count": 4 }
]
```
Useful commands:

## Developer Workflow
| Command | Purpose |
| --- | --- |
| `pnpm generate:api-types` | Export OpenAPI and regenerate frontend types |
| `pnpm validate:all` | Run backend, frontend, and SDK validation |
| `pnpm test` | Run backend, frontend, and SDK tests |
| `pnpm seed` | Seed realistic sample events |
| `pnpm stop` | Stop local app processes |
Comment on lines +114 to +120

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

medium

The "Useful commands" table is a great addition. To make it even more comprehensive for developers, consider re-introducing some useful commands from the previous README version that are still present in package.json, such as validate:backend, validate:frontend, and seed:clean.

Suggested change
| Command | Purpose |
| --- | --- |
| `pnpm generate:api-types` | Export OpenAPI and regenerate frontend types |
| `pnpm validate:all` | Run backend, frontend, and SDK validation |
| `pnpm test` | Run backend, frontend, and SDK tests |
| `pnpm seed` | Seed realistic sample events |
| `pnpm stop` | Stop local app processes |
| Command | Purpose |
| --- | --- |
| `pnpm generate:api-types` | Export OpenAPI and regenerate frontend types |
| `pnpm validate:backend` | Run backend validation checks |
| `pnpm validate:frontend` | Run frontend validation checks |
| `pnpm validate:all` | Run all validation checks |
| `pnpm test` | Run all tests |
| `pnpm seed` | Seed realistic sample events |
| `pnpm seed:clean` | Wipe and reseed events |
| `pnpm stop` | Stop local app processes |


- Use `pnpm` at the repo root for day-to-day commands.
- Treat `backend/poetry.lock` and `pnpm-lock.yaml` as the dependency source of truth.
- Do not reintroduce `npm` lockfiles or a standalone backend `requirements.txt`.
- Keep frontend API types generated from the backend schema, not hand-maintained.
Deeper backend setup lives in [`backend/SETUP.md`](backend/SETUP.md).

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

medium

For clarity and to ensure compatibility with a wider range of Markdown parsers, it's good practice to use explicit relative links for files within the repository.

Suggested change
Deeper backend setup lives in [`backend/SETUP.md`](backend/SETUP.md).
Deeper backend setup lives in [`backend/SETUP.md`](./backend/SETUP.md).


## Status
## Non-Goals

TelemetryTaco is intentionally not solving multi-tenancy, auth, cohorts, funnels, feature flags, or ClickHouse analytics yet. The current code is optimized for a maintainable ingestion-and-dashboard MVP with clean seams for future expansion.
TelemetryTaco is intentionally not solving multi-tenancy, auth, funnels, feature flags, or warehouse-scale analytics yet. The current repo is optimized for a strong ingestion-and-dashboard MVP with clean seams for future expansion.
Loading