feat(app): consume frequencies.txt to expand frequency-based trips

docs/concepts/terminology.md

@@ -16,6 +16,9 @@ Canonical names. Use these in code and docs.
 | **feed** | agency (in v2 UX) | A feed may carry multiple agencies; see [feeds.md](feeds.md) |
 | **ETA** | arrival time, time-to-arrive | "ETA" is short and unambiguous |
 | **bucket** | status, state | Specific term for station-view arrival classification |
+| **frequency-based trip** | repeating trip, headway trip, rec-trip | GTFS `frequencies.txt` model: a trip whose stop_times are reused as offsets for N generated departures, one per `headway_secs` within the frequency window. See [vehicle.md](vehicle.md#frequency-based-trips). |
+| **anchor trip** | base trip, source trip | The single `trips.txt` row whose `stop_times` carry the offsets for a frequency-based service. The one entry that physically exists in `trips` + `stop_times`; the per-departure rows are derived. |
+| **generated departure** | synthetic departure, expansion row | One per-departure row produced by the `frequencyExpansion` helper from a single anchor trip. Identified by `Vehicle.id = "trip:<tripId>@<effectiveStartMin>"` and `schedule.tripStartMin = effectiveStartMin`. |
 
 ## Technical
 

docs/concepts/vehicle.md

@@ -74,6 +74,48 @@ Orthogonal to `kind` and to [arrival-buckets](arrival-buckets.md). A
 the depot already broadcasting; `last` and `on-route` rows almost
 always do.
 
+## Frequency-based trips
+
+For trips with rows in `frequencies.txt` (the [GTFS spec's
+headway-based service model](https://gtfs.org/schedule/reference/#frequenciestxt)),
+the active set contains one `Vehicle` per **generated departure** —
+not one per anchor trip. A 15-minute headway running 05:05–22:40
+produces 71 `kind: 'scheduled'` rows, each with a distinct
+`schedule.tripStartMin` (the k-th departure's effective origin time).
+`tripPhase` classification still works unchanged because it's keyed
+on `tripStartMin`.
+
+**Identification.** The `Vehicle.id` for a generated row is
+`trip:<tripId>@<effectiveStartMin>` (the anchor's id without the
+suffix, `trip:<tripId>`, is kept for the original anchor entry where
+present). The `@<min>` suffix is what makes the id stable across
+polls and unique per generated departure.
+
+**Reconciler match key.** Live observations match by the composite
+`(tripId, tripStartMin)` key, not by `tripId` alone. The
+`enrichObservations` index uses `${tripId}|${tripStartMin}` so an
+`UNSCHEDULED` observation on a frequency-based trip (which carries
+the *specific* generated departure's effective start time per the
+[GTFS-RT contract](specs/gtfs-rt-contract.md)) resolves to the
+correct generated row. For observations missing `startTime`, a
+fallback to `tripId`-only matching preserves the legacy lenient
+behaviour for non-conforming producers.
+
+**Per-stop promotion.** The station-board merge in
+[`stationBoard.ts`](../../src/lib/domain/stationBoard.ts) uses the
+same composite key. A frequency-based trip's N per-stop rows each
+match against the N active-set rows by `tripStartMin`; a tolerance
+filter rejects per-stop rows whose `tripStartMin` doesn't match the
+reconciled row's, so the same GPS position doesn't get applied to
+every per-stop entry of an anchor trip.
+
+**Schedule-only fallback.** The publisher's `SCHEMA` adds the
+`frequencies` table in
+[n3ary/gtfs-publisher#252](https://github.com/n3ary/gtfs-publisher/pull/252);
+cached blobs that pre-date that change report `false` from
+`hasFrequenciesTable()` and the per-time query modules fall back to
+schedule-only behaviour without throwing.
+
 ## Visual rendering — kind dot
 
 The `VehicleCard` shows a small dot on the far right encoding `kind`

docs/plan/gtfs-frequencies.md

@@ -0,0 +1,222 @@
+# GTFS frequencies.txt support
+
+Issue: #347. Closes when the app consumes
+`frequencies.txt` to expand frequency-based trips into per-departure
+rows in the active set, the station board, the schedule view, the
+map view, and the weekly pattern. The publisher-side prerequisite
+lives in [n3ary/gtfs-publisher#252](https://github.com/n3ary/gtfs-publisher/pull/252)
+(merged first; the DDL addition is what makes the table present in
+the SQLite blob the app downloads).
+
+## What "frequency-based trip" means in this app
+
+A row in `frequencies.txt` says: "for this `trip_id`, run every
+`headway_secs` seconds from `start_time` to `end_time`, applying the
+anchor trip's `stop_times` as offsets relative to each departure
+time". The cluj-napoca adapter emits these rows for `*-range`
+annotations (e.g. M26 `05:05-22:40` / `10-20min` is the live case;
+see
+[gtfs-adapters#…/cluj-napoca/docs/known-limitations.md](https://github.com/n3ary/gtfs-adapters)).
+
+In the app, a frequency-based trip becomes **N `kind: 'scheduled'`
+rows in the active set, one per generated departure**, each with
+its own `schedule.tripStartMin` (the k-th departure's effective
+origin time) and its own `Vehicle.id`
+(`trip:<tripId>@<effectiveStartMin>`). The reconciler already
+matches on `(routeId, directionId, tripStartMin)`; the active set
+just needs more rows.
+
+## Design
+
+### Data model
+
+No new domain types. The discriminated union at
+`src/lib/domain/types.ts:188-217` is unchanged. A frequency-based
+trip's N generated rows are N `kind: 'scheduled'` Vehicles with
+distinct `schedule.tripStartMin` values and distinct `id` strings
+(the `@<min>` suffix). This is the same shape as the existing
+schedule-based rows; the only difference is the id suffix and the
+fact that the row's `tripStartMin` doesn't equal the anchor's
+`stop_times[0].departure_time`.
+
+The `id` encoding choice — `trip:<tripId>@<effectiveStartMin>` —
+is a public contract: the per-stop promotion path in
+`stationBoard.ts` uses Vehicle.id as a stable Svelte key, and the
+reconciler's matched-scheduled index in
+`stationBoard.ts:mergeReconciledIntoStationBoard` keys off the
+reconciled row's `schedule.tripStartMin` (matched against the
+per-stop row's `schedule.tripStartMin`) to gate the kind: 'tracked'
+promotion. Without the per-row `tripStartMin` match, every per-stop
+row for an anchor trip would get the same GPS position.
+
+### Helper module
+
+`src/lib/workers/gtfs/queries/frequencyExpansion.ts` is the
+shared core. Five exports:
+
+| Export | Purpose |
+|---|---|
+| `hasFrequenciesTable(db)` | PRAGMA probe. Returns false on cached blobs that pre-date the publisher's DDL addition; callers fall back to schedule-only behaviour. |
+| `getFrequenciesForServices(db, serviceIds)` | SQL query joining `frequencies` to `trips.service_id`. Filters out `exact_times=1` (rare, per spec). |
+| `expandFrequencyToDepartures(freq, windowStart, windowEnd)` | Pure JS. One `GeneratedDeparture` per k-th departure in the window. The end-time bound is exclusive per spec ("up to but not including end_time"). |
+| `expandFrequenciesToDepartures(freqs, windowStart, windowEnd)` | Convenience wrapper. Returns `Map<tripId, GeneratedDeparture[]>`. |
+| `getAnchorStopTimes(db, tripId)` | Per-trip stop_times rows in `stop_sequence ASC` order. Used by the per-stop expansion path to derive per-stop offset times. |
+
+The expansion is pure JS (no recursive SQL CTE) because (a) the
+window is small (typical M26 case: 71 generated departures in a
+17.5-hour window) and (b) keeping it pure makes the load-bearing
+function unit-testable without a DB fixture.
+
+### Per-time query changes
+
+Every per-time query gains a `hasFrequencies: boolean` parameter
+(passed in by the worker from
+`state.currentFeedHasFrequencies`). When true, after the existing
+SQL scan, the query calls `getFrequenciesForServices` + the
+expansion helpers and merges the generated rows with the
+schedule-based rows. The five queries touched:
+
+| Query | Expansion model |
+|---|---|
+| `getActiveTrips` | One `Vehicle` per generated departure. `schedule.tripStartMin` = effectiveStartMin, `schedule.scheduledArrival` = anchor.trip_end_time + k*headway. |
+| `getStationArrivals` | One `ScheduleRow` per generated departure whose effective time at THIS stop falls in the query window. The effective per-stop arrival is `anchor.stop_times[stop_id].arrival_time + k*headway_min`. The row's `id` is `trip:<tripId>@<effectiveStartMin>`. |
+| `getRouteSchedule` | One `ScheduleTrip` per generated departure, with the same composite key. |
+| `getActiveRouteIdsInWindow` | Boolean: a route is "active right now" iff any frequency row on it overlaps the query window. Per-row expansion is unnecessary — the route set is all we return. |
+| `getRouteMapView` | Same as `getActiveTrips` but for the per-(route, direction) view; shape_id is shared per anchor (shape doesn't change per generated departure). |
+| `getWeeklySchedule` | Expand each frequency row into synthetic minute slots for each matching day pattern. Headway 15 min × 17.5 h on weekdays = 67 synthetic minutes added to the `weekday` set. |
+
+The frequency-based rows are appended to the schedule-based rows
+in the same `Vehicle[]` (or `ScheduleTrip[]`, etc.) and the rest of
+the pipeline is unaware. `scanSchedule` accepts an optional
+`ScheduleRow.id` override so the per-stop rows get the right
+`@<min>`-suffixed id (the default `trip:<tripId>` is used when
+undefined).
+
+### Reconciler
+
+No change. The reconciler at `src/lib/domain/reconcile.ts:37-203`
+matches on `(routeId, directionId, tripStartMin)` — the
+`tripStartMin` field is already on the active set rows, and the
+generated rows have distinct values. The match tolerance
+(`computeTolerance`, lines 212-235) works unchanged; a
+15-minute-headway cohort's median gap is 15 min, so the median/2
+tolerance is 7-8 min, which is wider than any reasonable
+observation drift.
+
+### `enrichObservations` composite key
+
+`src/lib/domain/enrichObservations.ts:10-19` previously indexed
+the active set by `tripId` alone. With multiple generated rows per
+tripId, that map would collapse to the last-written entry and every
+observation would match the wrong generated departure. Fixed by
+keying on `${tripId}|${tripStartMin}` (the primary index) and
+keeping a `tripId`-only fallback for observations missing
+`startTime` (preserves the legacy lenient behaviour for
+non-conforming producers; for frequency-based trips, the fallback
+resolves to the k=0 entry, which is the anchor's first departure).
+
+### `stationBoard` promotion tolerance
+
+`src/lib/domain/stationBoard.ts:276-297` previously promoted every
+per-stop row whose `tripId` matched a reconciled row to `kind:
+'tracked'`. With frequency expansion, that's wrong — the
+reconciled row carries ONE specific generated departure's position;
+the per-stop set has N rows (one per generated departure). Added a
+`tripStartMin` equality check that gates the promotion. Trivial
+change; non-frequency trips pass the check trivially because the
+anchor's `tripStartMin` is identical to the active-set entry.
+
+### Bootstrap PRAGMA probe
+
+`src/lib/workers/gtfs/bootstrap.ts:385-393` runs a `sqlite_master`
+probe for the `frequencies` table after the `stop_times` integrity
+check. The result is stashed in `state.currentFeedHasFrequencies`
+(`src/lib/workers/gtfs/state.ts:32-36`). `closeCurrent()` resets
+the flag to `false`.
+
+### `id` encoding — public contract
+
+`Vehicle.id` for frequency-based rows is `trip:<tripId>@<effectiveStartMin>`.
+Anchor rows (where the trip has no frequencies row, or the row
+exists but the anchor's stop_times naturally fall in the window
+without expansion) keep the legacy `trip:<tripId>` shape. The
+per-stop promotion path's `Vehicle.id` propagation is unchanged —
+the merged row inherits `v.id` from the per-stop row.
+
+## Stack order
+
+1. **n3ary/gtfs-publisher#252** — DDL addition (merged first).
+2. **#347 (this)** — consumer side. Lands after #252.
+
+## Open design questions
+
+1. **Weekly view display.** A frequency-based trip with headway
+   15 min / window 05:05-22:40 currently shows as 1 anchor
+   departure on the weekly schedule. The data layer makes
+   "every 15 min from 05:05 to 22:40" possible; the
+   rendering is a separate UX call (the data already includes 67
+   synthetic minute slots in the `weekday` set; the existing
+   per-minute rendering just needs to handle the dense output).
+2. **Per-route schedule display.** Same question for the
+   per-route schedule view. 67 individual rows is a lot; one
+   summary row with the headway is probably right.
+3. **Multi-frequency rows per trip.** GTFS allows multiple
+   `frequencies.txt` rows per `trip_id` (e.g. "15 min 05:00-09:00"
+   then "30 min 09:00-22:00"). The cluj adapter only emits one
+   row per anchor, but the spec permits more. The
+   `getFrequenciesForServices` helper already returns N rows per
+   trip; the per-time query loops iterate them all and union the
+   expansions. Tested conceptually via
+   `frequencyExpansion.test.ts` (the `expandFrequenciesToDepartures`
+   test covers N rows per trip) but not end-to-end.
+4. **`exact_times=1` rows.** GTFS allows `exact_times=1`
+   (the frequencies row exists but the trip is still
+   schedule-based). The cluj adapter only emits
+   `exact_times=0`. The helper treats
+   `exact_times=0` (or NULL) as the default expansion target and
+   `exact_times=1` as "ignore the frequencies row, use
+   `stop_times` directly" — the SQL `WHERE (f.exact_times IS NULL
+   OR f.exact_times = 0)` filters these out. Theoretical until a
+   feed actually needs it.
+
+## Verification
+
+- `pnpm check` — clean (svelte-check).
+- `pnpm test` — 330/330 pass.
+- `pnpm build` — vite build emits the production bundle.
+- Unit tests cover the load-bearing expansion math
+  (`frequencyExpansion.test.ts`, 8 cases).
+- Live data path verified at the type level: the per-time queries
+  accept `hasFrequencies: boolean`, the worker passes
+  `state.currentFeedHasFrequencies`, and the `id` encoding is
+  stable across polls. E2E verification (loading a real
+  frequencies-bearing blob and inspecting the station board)
+  needs the publisher's blob to be published to R2 first — that's
+  #252's rollout, out of scope here.
+
+## Out of scope
+
+- A `route_desc` or visual treatment of "headway 15 min" in the
+  route badge. That's a separate UX call.
+- Per-feed opt-in / opt-out toggles in `feeds.json`. The app is
+  feed-agnostic per `docs/standards/feed-agnostic.md`; the
+  publisher's `SCHEMA` is the contract, full stop. Old blobs that
+  pre-date the DDL addition are handled by the bootstrap PRAGMA
+  probe, not by a feed flag.
+- Reimplementing the weekly pattern view to summarize
+  frequency-based trips. The data layer supports it; the rendering
+  is a follow-up.
+- GTFS-RT `TripUpdates` and `service_alerts` consumption. Both are
+  still reserved per `docs/specs/feeds-json.md:69`.
+- The `gtfs-publisher-rt-reconcile` package's `parseFrequencies`
+  reader — has no consumer today. Left for a separate cleanup PR.
+
+## Related
+
+- Producer: `gtfs-adapters/adapters/cluj-napoca/src/assemble/derive/frequencies.ts`
+- Publisher PR: [n3ary/gtfs-publisher#252](https://github.com/n3ary/gtfs-publisher/pull/252)
+- GTFS spec: [frequencies.txt](https://gtfs.org/schedule/reference/#frequenciestxt)
+- GTFS-RT contract: [app/docs/specs/gtfs-rt-contract.md](../specs/gtfs-rt-contract.md)
+- Reconciler match key: [app/src/lib/domain/reconcile.ts](../../src/lib/domain/reconcile.ts)
+- Helper: [app/src/lib/workers/gtfs/queries/frequencyExpansion.ts](../../src/lib/workers/gtfs/queries/frequencyExpansion.ts)
+- Helper tests: [app/src/lib/workers/gtfs/queries/frequencyExpansion.test.ts](../../src/lib/workers/gtfs/queries/frequencyExpansion.test.ts)

src/lib/domain/enrichObservations.ts

@@ -1,37 +1,92 @@
 // Resolve live observations against the static-trip index. Pure: no IO. Caller owns the active-trips snapshot.
 
 import type { LiveVehicleObservation } from '$lib/data/live/gtfsRtClient';
-import { minutesToTime } from './pipeline/timeUtils';
+import { minutesToTime, timeToMinutes } from './pipeline/timeUtils';
 import type { Vehicle } from './types';
 
-type ActiveTripIndex = ReadonlyMap<string, { directionId: 0 | 1; tripStartMin: number }>;
+interface ActiveTripEntry {
+  directionId: 0 | 1;
+  tripStartMin: number;
+}
+
+/** Composite-key index: `${tripId}|${effectiveStartMin}` → entry. The
+ *  composite key is required for frequency-based trips — the app's
+ *  frequency-expansion helper emits one Vehicle per generated
+ *  departure (see #347), all sharing the anchor's tripId but with
+ *  distinct `tripStartMin` (the effective time of the k-th departure).
+ *  A bare tripId key would collapse the map on the last-written entry
+ *  and the reconciler would match every observation against the wrong
+ *  generated departure. The key is the same encoding the per-stop
+ *  promotion path uses in `stationBoard.ts:mergeReconciledIntoStationBoard`. */
+type CompositeIndex = ReadonlyMap<string, ActiveTripEntry>;
 
-// tripId → {direction, startMin}, from the active-trips the worker fetches per tick.
-export function indexActiveTripsByTripId(active: readonly Vehicle[]): ActiveTripIndex {
-  const out = new Map<string, { directionId: 0 | 1; tripStartMin: number }>();
+/** tripId → first entry. Fallback for observations with no
+ *  startTime. For non-frequency trips there's exactly one entry per
+ *  tripId; for frequency-based trips, the iteration finds the k=0
+ *  entry (the anchor's first departure), which is the legacy
+ *  "tripId-only match" behaviour for ambiguous cases. */
+type TripIdIndex = ReadonlyMap<string, ActiveTripEntry>;
+
+/** Build both indices from the worker's active-trips snapshot. */
+export function indexActiveTrips(active: readonly Vehicle[]): {
+  byComposite: CompositeIndex;
+  byTripId: TripIdIndex;
+} {
+  const byComposite: Map<string, ActiveTripEntry> = new Map();
+  const byTripId: Map<string, ActiveTripEntry> = new Map();
   for (const v of active) {
     if (!v.tripId) continue;
     const dir = v.schedule?.directionId;
     const start = v.schedule?.tripStartMin;
     if ((dir !== 0 && dir !== 1) || typeof start !== 'number') continue;
-    out.set(v.tripId, { directionId: dir, tripStartMin: start });
+    const entry: ActiveTripEntry = { directionId: dir, tripStartMin: start };
+    byComposite.set(`${v.tripId}|${start}`, entry);
+    if (!byTripId.has(v.tripId)) byTripId.set(v.tripId, entry);
   }
-  return out;
+  return { byComposite, byTripId };
+}
+
+/** Back-compat shim — the old API returned a single Map keyed by
+ *  tripId. Kept for tests and any external callers; new code should
+ *  use `indexActiveTrips` for the composite path. */
+export function indexActiveTripsByTripId(active: readonly Vehicle[]): TripIdIndex {
+  return indexActiveTrips(active).byTripId;
 }
 
 export function enrichObservations(
   observations: readonly LiveVehicleObservation[],
   active: readonly Vehicle[],
 ): LiveVehicleObservation[] {
-  const byTripId = indexActiveTripsByTripId(active);
-  return observations.map((obs) => enrichOne(obs, byTripId));
+  const { byComposite, byTripId } = indexActiveTrips(active);
+  return observations.map((obs) => enrichOne(obs, byComposite, byTripId));
 }
 
 function enrichOne(
   obs: LiveVehicleObservation,
-  byTripId: ActiveTripIndex,
+  byComposite: CompositeIndex,
+  byTripId: TripIdIndex,
 ): LiveVehicleObservation {
-  const sched = obs.tripId ? byTripId.get(obs.tripId) : undefined;
+  // Primary: composite key. For non-frequency trips the obs's
+  // startTime is the anchor's origin departure, identical to the
+  // active set's `tripStartMin`. For `UNSCHEDULED` observations
+  // (frequency-based trips per
+  // docs/specs/gtfs-rt-contract.md:89-90), the startTime is one
+  // specific generated departure's effective time.
+  const startMin = obs.startTime ? timeToMinutes(obs.startTime) : Number.NaN;
+  let sched: ActiveTripEntry | undefined;
+  if (obs.tripId && Number.isFinite(startMin)) {
+    sched = byComposite.get(`${obs.tripId}|${startMin}`);
+  }
+  // Fallback: tripId-only. Preserves the legacy lenient behaviour
+  // for non-conforming producers that don't populate startTime on
+  // non-frequency observations. For frequency-based trips this
+  // resolves to the k=0 entry, which is the anchor's first
+  // departure — not strictly correct (an observation without
+  // startTime is genuinely ambiguous), but the best we can do and
+  // matches pre-#347 behaviour.
+  if (!sched && obs.tripId) {
+    sched = byTripId.get(obs.tripId);
+  }
   if (sched) {
     return {
       ...obs,