PULSE: Urban Mobility Observatory & Day-Ahead Demand Forecasting

PULSE is a cinematic, browser-based urban mobility observatory built on 5,972,150 real NYC Yellow Taxi trip records from January & February 2024. It combines WebGL 2.5D spatial visualization (deck.gl) with strict, leak-free day-ahead demand forecasting (HistGradientBoostingRegressor) and empirical conformal uncertainty intervals.

---

Visual Showcase

View 01: City Replay (2.5D Extrusion & 24h Scrubber)	View 02: Flow Explorer (3D OD Arcs & Companion Matrix)

View 03: Rhythm Compare (Fair-Denominator Normalization)	View 04: Forecast Lab (Scoreboard & Conformal Bands)

View 05: Evidence & Method (100% Invariant Audit)	View 08: High-Resolution Cover Showcase

---

Core Capabilities

1. City Replay (2.5D Spatial Observatory)

• Standalone deck.gl GeoJsonLayer rendering all 263 NYC Taxi Zones with 2.5D extrusion heights driven by hourly pickup volumes.
• Continuous 24-hour interactive scrubber with Play / Pause, step-forward/backward, and mini sparkline volume density.
• Selected Zone Inspector Drawer displaying peak hour of the day, current volume, and complete diurnal sparklines.
• Zero remote map tile tokens or subscription fees: renders directly on WebGL canvas over deep #08121C observatory background.

2. Flow Explorer (Origin-Destination Arcs)

• Illuminated 3D ArcLayer displaying primary travel corridors from top origin hubs (e.g. JFK Airport, LaGuardia, Midtown Center, Penn Station).
• Companion Ranking Table detailing destination zone, borough, trip count, and percentage share.
• Identifies radial inter-borough airport dispersals versus dense Manhattan intra-borough micro-hops.

3. Rhythm Compare (Fair-Denominator Discipline)

• The 2.53× Trap: In Jan–Feb 2024, there are 43 weekdays and 17 weekend days. Summing raw counts distorts weekday volumes by 2.53×.
• Fair-denominator division ($\frac{\sum Y}{43}$ vs $\frac{\sum Y}{17}$) reveals authentic mobility archetypes:
  • Commercial Core (Midtown Center): Intense midday commute surge, falling off on weekends.
  • Nightlife District (East Village): Sustained late-night weekend surge into the early morning hours.
  • Gateway Hub (JFK Airport): Steady 7-day volume governed by flight scheduling.

4. Forecast Lab (Zero-Leakage Benchmark Scoreboard)

• Day-ahead hourly pickup demand forecasts issued strictly at midnight ($D$ 00:00) for horizons $h = 0 \dots 23$.
• Evaluated across 3,360 zone-hours on the held-out Test Split (Feb 23–29, 2024, Leap Year):

Model / Baseline	MAE (Pickups/Hr)	WAPE (%)	80% Empirical Coverage	Outcome
HistGradientBoostingRegressor (Challenger)	22.81	15.94%	81.0%	Winner
Baseline: Seasonal Lag (168h / Same Hour Last Week)	26.71	18.66%	N/A	Baseline
Baseline: Prior-Day Persistence (24h Lag)	43.90	30.66%	N/A	Benchmark

• Empirical Conformal Quantiles: Residual quantiles calibrated on a separate, held-out Calibration split (Feb 16–22) achieve 81.0% empirical coverage on test data.
• "Reveal Actuals" Toggle: Allows users to inspect ground-truth pickups against model forecasts and confidence corridors.

5. Evidence & Method (Audit Integrity)

• 100% Invariant Reconciliation: Complete accounting of 5,972,150 raw records with zero unclassified leakage:
  • VALID_MAPPED_TRIP: 5,911,633 (98.99%)
  • UNMAPPED_DROPOFF: 37,386 (0.63%)
  • UNMAPPED_PICKUP: 23,112 (0.39%)
  • OUT_OF_SCOPE_TIME: 19 (0.00%)
• Data Provenance: Official NYC TLC citations with SHA-256 cryptographic hashes for all ingested files.
• One-Click PDF Brief Export: Print-optimized stylesheet for research briefs and executive presentations.

---

Project Structure

d:/Ran/pulse/
├── artifacts/                  # High-resolution screenshots and presentation assets
├── docs/                       # Technical documentation and audit evidence
│   ├── DATA_DICTIONARY.md      # Field descriptions, constraints, and parquet schemas
│   ├── DEMO_SCRIPT.md          # 2-minute executive demonstration walkthrough
│   ├── INTERVIEW_WALKTHROUGH.md# Senior/Staff engineer system design Q&A
│   ├── LINKEDIN_POST.md        # Formatted social showcase with metrics
│   ├── METRIC_CONTRACT.md      # Statistical invariants and mathematical contracts
│   ├── MODEL_CARD.md           # Model specifications, hyperparameters, and scorecard
│   ├── PERFORMANCE.md          # WebGL, bundle, and runtime benchmarks
│   ├── VALIDATION.md           # Audit verification and test execution report
│   └── screenshots/            # Captured application views (desktop, tablet, mobile)
├── pipeline/                   # Python data preparation and modeling
│   ├── prepare_data.py         # DuckDB ingestion, geometry simplification, aggregations
│   └── train_forecast.py       # scikit-learn HGB training, conformal intervals, export
├── public/data/                # Static JSON and GeoJSON assets loaded by client
│   ├── forecast_results.json   # Held-out test timelines, scoreboard, and metrics
│   ├── manifest.json           # Invariant reconciliation, checksums, and splits
│   ├── replay_sample_day.json  # 24-hour hourly replay grid
│   ├── rhythm_profiles.json    # Fair-denominator weekday/weekend hourly curves
│   ├── taxi_zones.geojson      # Simplified WGS84 zone polygons (689 KB)
│   ├── top_flows.json          # OD flow rankings for top 20 zones
│   └── zone_centers.json       # Representative interior point coordinates
├── src/                        # React 18 + TypeScript client application
│   ├── app/App.tsx             # Root application and view router
│   ├── components/             # Reusable UI components (NavigationRail, TimeRail)
│   ├── data/dataLoader.ts      # Fetch utilities with client-side caching
│   ├── features/               # Five domain observatory views
│   ├── map/DeckMap.tsx         # Deck.gl WebGL canvas (GeoJsonLayer + ArcLayer)
│   └── types/index.ts          # TypeScript domain interfaces
├── tests/                      # Automated test suite
│   └── metric_contract.test.js # Invariant reconciliation and metric contract tests
├── package.json
├── tsconfig.json
└── vite.config.ts

---

Quickstart & Local Reproduction

Prerequisites

• Node.js 18+ (tested on Node v22.12.0)
• Python 3.10+ (or uv)

1. Install Node Dependencies

npm install

2. Run Data Pipeline (Optional — Pre-computed artifacts are included in repository)

uv run --with geopandas --with duckdb --with pyarrow --with pandas --with shapely --with pyproj --with pyogrio python pipeline/prepare_data.py
uv run --with scikit-learn --with pandas --with pyarrow --with numpy python pipeline/train_forecast.py

3. Run Automated Tests

node --test tests/metric_contract.test.js

4. Build & Run Application

npm run build
npm run preview

Visit http://localhost:4173/ in your browser.

---

License

MIT License. TLC data is published by the New York City Taxi & Limousine Commission under the NYC Open Data terms of use.