ArkiWatch

ArkiWatch is a Finland-focused life-data platform that turns public signals into practical daily guidance. It gathers weather, air quality, transit, traffic, water, snow, city service, event, and energy data; normalizes those feeds into events and snapshots; and serves them through a FastAPI backend, a Next.js data dashboard, and Telegram delivery paths.

The repository is a polyglot monorepo:

Python owns ingestion, persistence, normalization, domain services, rule engines, delivery, and observability.
TypeScript owns the interactive dashboard and data-platform user experience.

At a Glance

Area	Stack / surface
Backend	FastAPI, SQLAlchemy, Pydantic Settings, PostgreSQL
Frontend	Next.js App Router, React, TypeScript
Data jobs	Collectors, normalization pipeline, rules, dispatch runners
Observability	Health endpoint, Prometheus-compatible metrics, structured logs, request IDs, server timing
Product surfaces	Data platform, public dashboard, personal assistant, recommendation flows, Telegram integration
Status	Active pre-1.0 development with CI, migrations, scheduler, API contracts, and test gates

What It Does

Collects live and near-live public data from FMI, HSY, Digitransit/HSL, Digitraffic, Fingrid, SYKE, Helsinki Service Map, Helsinki Linked Events, and electricity price APIs.
Normalizes raw observations into queryable event and snapshot models.
Produces public and personalized recommendations for commute, weather, outdoor, home-air, and energy scenarios.
Exposes operational APIs for source health, platform summaries, dashboards, preferences, locations, commute profiles, channels, notifications, and Telegram integration.
Ships a responsive dashboard with city scoping, date scoping, dark/light mode, multilingual UI support, event timelines, category workspaces, and an assistant-style explainer widget.

Key Concepts

Concept	Explanation
Source	A configured public-data provider, such as FMI, HSY, Digitransit, Fingrid, SYKE, or Helsinki APIs.
Collector	A backend adapter that fetches one source and stores raw records without leaking provider-specific details into the rest of the app.
Raw record	The preserved source payload plus fetch metadata, used for repeatable normalization and source-health reporting.
Event	A normalized observation that has time, place, category, severity, source, and user-facing message fields.
Snapshot	A current or historical rollup that powers dashboards and category workspaces.
Rule engine	A guidance layer that turns normalized facts into recommendations and notification candidates.
Dispatch	The delivery path for pending notifications, currently centered on Telegram-capable channels.

Architecture

flowchart LR
  classDef source fill:#eef6ff,stroke:#4f8cc9,color:#12395b;
  classDef backend fill:#f4f0ff,stroke:#8067c7,color:#2d2359;
  classDef data fill:#effaf4,stroke:#4f9f6f,color:#173d28;
  classDef surface fill:#fff7e8,stroke:#d99932,color:#4f3208;
  classDef ops fill:#f7f7f7,stroke:#8b95a1,color:#20242a;

  subgraph Sources["External public signals"]
    direction TB
    WeatherAir["Weather + air<br/>FMI, HSY, Open-Meteo"]
    Mobility["Transit + traffic<br/>Digitransit, Digitraffic"]
    Energy["Energy<br/>Fingrid, price APIs"]
    City["Water, snow, services, events<br/>SYKE, Service Map, Linked Events"]
  end

  subgraph Backend["apps/api - FastAPI runtime"]
    direction TB
    Ingestion["Ingestion<br/>collector adapters"]
    Storage["Storage<br/>PostgreSQL raw records"]
    DataProducts["Data products<br/>events + snapshots"]
    Serving["Serving layer<br/>services, presenters, REST routes"]
    Guidance["Guidance + delivery<br/>rules, dispatch, Telegram"]
    Controls["Controls<br/>scheduler, backfill, auth, observability"]
  end

  subgraph Products["apps/web - Next.js surfaces"]
    direction TB
    Overview["Data platform overview"]
    Workspaces["Category workspaces"]
    Public["Public dashboard"]
    Assistant["Data assistant"]
  end

  WeatherAir --> Ingestion
  Mobility --> Ingestion
  Energy --> Ingestion
  City --> Ingestion

  Ingestion --> Storage --> DataProducts
  DataProducts --> Serving
  DataProducts --> Guidance
  Serving --> Overview
  Serving --> Workspaces
  Serving --> Public
  Serving --> Assistant
  Controls -. schedules + replays .-> Ingestion
  Controls -. protects + observes .-> Serving

  class WeatherAir,Mobility,Energy,City source;
  class Ingestion,Serving,Guidance backend;
  class Storage,DataProducts data;
  class Overview,Workspaces,Public,Assistant surface;
  class Controls ops;

  style Sources fill:#f8fbff,stroke:#b8c7da,stroke-width:1px;
  style Backend fill:#fbf9ff,stroke:#c4b5fd,stroke-width:1px;
  style Products fill:#fffaf0,stroke:#f2cf86,stroke-width:1px;

The backend is split between a data plane and a serving layer: collectors isolate provider APIs, raw records keep source payloads auditable, pipeline jobs produce stable events and snapshots, services and presenters shape API contracts, and rule/dispatch workers turn facts into guidance. Scheduler, backfill, auth, and observability sit beside the flow so live jobs, historical replay, guarded operations, and production diagnostics stay explicit.

Repository Layout

.
├── apps
│   ├── api
│   │   ├── backend          # FastAPI app, collectors, pipeline, services, rules, dispatch
│   │   ├── tests            # Backend pytest suite
│   │   ├── scripts          # Backend local helpers
│   │   ├── .env.example     # Backend environment template
│   │   ├── package.json     # Backend lifecycle scripts
│   │   └── pyproject.toml   # Python package and tooling config
│   └── web
│       ├── public/brand     # Brand assets used by README and Next.js
│       ├── src              # Next.js App Router source
│       ├── .env.example     # Web environment template
│       ├── package.json     # Web dependencies and scripts
│       └── tsconfig.json    # Web TypeScript config
├── .github                  # CI and dependency update automation
├── deploy                   # VPS deployment, operations scripts, and runbooks
├── docs                     # Tracked design notes
├── package.json             # Root workspace scripts
├── pnpm-workspace.yaml      # pnpm workspace definition
├── tsconfig.base.json       # Shared TypeScript config
└── LICENSE

Prerequisites

Node.js 20+
Corepack with pnpm available
Python 3.11+
PostgreSQL
Optional: uv for faster backend virtualenv creation

The default local database URL is:

postgresql+psycopg://postgres:postgres@localhost:5432/arkiwatch

Quickstart

Create local environment files:

cp apps/api/.env.example apps/api/.env
cp apps/web/.env.example apps/web/.env.local

Install dependencies:

corepack enable
pnpm install
pnpm run setup:api

Create the local database, run migrations, and bootstrap metadata:

createdb arkiwatch
pnpm run bootstrap:api

Databases created before Alembic support are detected by pnpm run bootstrap:api; existing application tables without an alembic_version row are stamped at the current head before metadata seeding continues.

Start the backend and frontend together:

pnpm dev

Local URLs:

Service	URL
Web app	`http://localhost:3000`
API root	`http://127.0.0.1:8000/api`
OpenAPI docs	`http://127.0.0.1:8000/api/docs`
Health	`http://127.0.0.1:8000/api/health`
Metrics	`http://127.0.0.1:8000/api/metrics`

In production, /api/metrics should be protected with METRICS_BEARER_TOKEN.

To run each side separately:

pnpm run dev:api
pnpm run dev:web

Configuration

Backend configuration lives in apps/api/.env.

Setting	Purpose
`DATABASE_URL`	PostgreSQL connection string.
`APP_ENV`	Runtime environment. Production-like values enable strict secret validation.
`API_PREFIX`	API route prefix, defaulting to `/api`.
`API_HOST`, `API_PORT`	Uvicorn host and port for local API serving.
`BACKEND_CORS_ORIGINS`	Comma-separated allowed frontend origins.
`APP_TIMEZONE`	Runtime timezone, defaulting to `Europe/Helsinki`.
`DEFAULT_PLACE`, `DEFAULT_LAT`, `DEFAULT_LON`	Default city/place context for local flows.
`WEATHER_PLACES`	Comma-separated FMI weather place mappings, for example `helsinki=Helsinki,espoo=Espoo`.
`AIR_QUALITY_PLACES`	Comma-separated HSY air-quality place mappings, for example `helsinki=Helsinki,espoo=Espoo,vantaa=Vantaa`.
`OPEN_METEO_AIR_QUALITY_URL`, `OPEN_METEO_AIR_QUALITY_LOCATIONS`	Open-Meteo air-quality endpoint and city coordinate mappings for PM2.5, PM10, and NO2.
`TELEGRAM_BOT_TOKEN`, `TELEGRAM_WEBHOOK_SECRET`	Enable Telegram notifications and webhook validation.
`ADMIN_USER_ID`, `ADMIN_USERNAME`, `ADMIN_PASSWORD`, `ADMIN_PASSWORD_HASH`	Admin login bootstrap. Prefer `ADMIN_PASSWORD_HASH` outside local development.
`AUTH_TOKEN_SECRET`, `AUTH_TOKEN_ISSUER`, `AUTH_TOKEN_AUDIENCE`, `AUTH_TOKEN_TTL_MINUTES`	Signed access-token configuration and validation claims.
`FMI_`, `HSY_`, `DIGITRANSIT_`, `DIGITRAFFIC_`, `FINGRID_`, `SYKE_`	Source-specific API configuration.
`COLLECTOR_TIMEOUT_SECONDS`	External-source request timeout.
`NORMALIZATION_LOOKBACK_HOURS`, `SNAPSHOT_LOOKAHEAD_HOURS`	Pipeline time windows.
`SCHEDULER_*`	Durable scheduler polling, lock TTL, retry, pipeline/rules/dispatch intervals, and due-job batch size.
`BACKFILL_*`	Historical backfill window size, retry, lock TTL, and due-window batch controls.
`INITIAL_BACKFILL_*`	Optional bootstrap seeding of historical backfill jobs. Defaults to off; source-specific days and window sizes live in `backend.app.source_policies`.
`LOG_LEVEL`, `LOG_FORMAT`, `REQUEST_LOGGING_ENABLED`, `METRICS_ENABLED`, `METRICS_BEARER_TOKEN`	Observability controls and optional metrics endpoint protection.

Frontend configuration lives in apps/web/.env.local.

Setting	Purpose
`NEXT_PUBLIC_APP_NAME`	Display name used by the app shell.
`NEXT_PUBLIC_API_BASE_URL`	Public API base URL used by the dashboard.
`NEXT_PUBLIC_APP_ORIGIN`	Canonical web origin used by server-side POST request origin checks.
`NEXT_PUBLIC_DEFAULT_USER_ID`	Optional user context for personal dashboard and subscription flows.
`APP_TIMEZONE`	Frontend timezone context.

Generate an admin password hash with:

cd apps/api
./.venv/bin/python -c 'from backend.app.security import hash_password; print(hash_password("replace-me"))'

Development Commands

Command	Purpose
`pnpm install`	Install root workspace and frontend dependencies.
`pnpm run setup:api`	Recreate `apps/api/.venv` and install the backend in editable mode.
`pnpm run bootstrap:api`	Run Alembic migrations and seed source/rule metadata.
`pnpm run migrate:api`	Run Alembic migrations against `DATABASE_URL`.
`pnpm run revision:api -- -m "add table"`	Create an Alembic autogenerate revision.
`pnpm dev`	Start FastAPI and Next.js together.
`pnpm run dev:api`	Start FastAPI from `apps/api`.
`pnpm run dev:web`	Start Next.js from `apps/web`.
`pnpm run collectors:api`	Run all collector adapters once.
`pnpm run pipeline:api`	Normalize raw observations and refresh snapshots.
`pnpm run rules:api`	Execute recommendation engines.
`pnpm run dispatch:api`	Dispatch pending notifications.
`pnpm run scheduler:api`	Run the durable backend scheduler loop.
`pnpm run build:web`	Build the Next.js app.
`pnpm run contracts:generate`	Export backend OpenAPI and regenerate frontend API contract types.
`pnpm run contracts:check`	Regenerate API contracts and fail if generated files drift.
`pnpm run lint:api`	Run Ruff against backend code and tests.
`pnpm run lint:web`	Run Next.js linting.
`pnpm run typecheck:web`	Run TypeScript type checking.
`pnpm run test:api`	Run the backend pytest suite.
`pnpm run test:web`	Run the frontend Vitest suite.
`pnpm check`	Run Ruff, pytest, frontend linting, frontend tests, and frontend type checking.

Data Pipeline

The runtime pipeline is split into explicit jobs:

pnpm run collectors:api
pnpm run pipeline:api
pnpm run rules:api
pnpm run dispatch:api

The stages are intentionally independent:

Stage	Responsibility
Collectors	Fetch external observations and persist raw source records.
Pipeline	Normalize raw records into events and snapshots.
Rules	Evaluate public and personal guidance from normalized facts.
Dispatch	Send pending notifications through configured channels.

The durable scheduler persists job state in PostgreSQL (scheduled_jobs and scheduled_job_runs) so restarts do not lose the cadence. It seeds one collector job per active source, then recurring normalize, snapshot, rules, and dispatch jobs:

pnpm run bootstrap:api
pnpm run scheduler:api

Useful one-shot commands:

pnpm run scheduler:api -- --sync-only
pnpm run scheduler:api -- --once

Multiple scheduler processes can run against the same database. Due jobs are claimed with a database row lock and a TTL, then rescheduled after success or a retry delay after failure. Current job state is exposed at GET /api/scheduler/jobs; definitions can be reconciled with POST /api/scheduler/sync. Both scheduler operations require a bearer token from POST /api/auth/login.

API Surface

The backend exposes route groups for:

platform overview and source health
weather, air quality, traffic, transit, and energy signals
public dashboard and public channels
personal dashboard, preferences, locations, commute profiles, and recommendations
users, channels, notifications, scheduler status, Telegram integration, health, and metrics

Useful checks:

curl -i http://127.0.0.1:8000/api/health
curl http://127.0.0.1:8000/api/metrics

When METRICS_BEARER_TOKEN is set, include Authorization: Bearer <token> for metrics.

Frontend Surface

The web app currently leads with the unified data platform:

overview dashboard with source counts, active events, field coverage, and signal health
category workspaces for weather, air quality, energy, transit, road and rail, water and snow, city services, events, geospatial data, and regional profiles
city and date scoping for supported categories
recent event timeline and notification summary
dark/light theme toggle and language switcher
assistant widget for interpreting platform state

If the backend is unavailable, the UI shows an explicit platform error state instead of static mock data.

Current Screenshots

The screenshots below are refreshed from the current local dashboard against a seeded development database.

Full desktop overview dashboard with the Today scenario lead, city and date scoping, daily planning, and live signal cards for commute, weather, air quality, and energy.

Full desktop weather detail workspace with live sync status, current conditions, operational weather cards, and forecast summary metrics.

AI data assistant answering a natural umbrella question from the live Helsinki weather view.

AI data assistant answering a natural commute-impact question using the current platform snapshot.

AI data assistant answering a natural household energy-timing question from the current live data snapshot.

Project Status

ArkiWatch is active, pre-1.0 software. The main product paths are implemented, and the core production foundations now include:

Alembic migrations and bootstrap stamping for existing development databases
admin authentication, signed token claims, production secret validation, and guarded operational routes
CI enforcement for contracts, linting, tests, frontend builds, and Docker image builds
generated OpenAPI contracts with frontend compile-time drift checks
durable scheduler runtime for collectors, normalization, rules, dispatch, and backfill windows
backend pytest coverage and focused frontend Vitest coverage for platform state, auth request guards, and workspace view models
dependency update automation through Dependabot

Still planned:

OpenTelemetry trace export
fuller contributor process, issue templates, and release process

Contributing

ArkiWatch is not yet operating with a full contributor process. Keep changes small, run pnpm run contracts:check, pnpm check, and pnpm run build:web, and include enough context in pull requests for reviewers to understand the data source, API contract, or UI behavior being changed.

License

ArkiWatch is licensed under the GNU Affero General Public License v3.0.

Name		Name	Last commit message	Last commit date
Latest commit History 178 Commits
.github		.github
apps		apps
deploy		deploy
docs		docs
.dockerignore		.dockerignore
.editorconfig		.editorconfig
.eslintrc.base.json		.eslintrc.base.json
.gitignore		.gitignore
Brewfile		Brewfile
LICENSE		LICENSE
README.md		README.md
package.json		package.json
pnpm-lock.yaml		pnpm-lock.yaml
pnpm-workspace.yaml		pnpm-workspace.yaml
tsconfig.base.json		tsconfig.base.json

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Repository files navigation

ArkiWatch

At a Glance

What It Does

Key Concepts

Architecture

Repository Layout

Prerequisites

Quickstart

Configuration

Development Commands

Data Pipeline

API Surface

Frontend Surface

Current Screenshots

Project Status

Contributing

License

About

Releases

Packages

Contributors

Languages

Folders and files

Latest commit

History

Repository files navigation

ArkiWatch

At a Glance

What It Does

Key Concepts

Architecture

Repository Layout

Prerequisites

Quickstart

Configuration

Development Commands

Data Pipeline

API Surface

Frontend Surface

Current Screenshots

Project Status

Contributing

License

About

Resources

Stars

Watchers

Forks

Releases

Packages

Contributors

Languages