Optimize Live dashboard

[ezio-melotti]

This PR optimizes the Live dashboard by introducing two major changes:

• The list of sensors metadata is now fetched only once at the beginning, the list of active sensors every 5 minutes, and the readings from the active sensors every 5 seconds;
• The SQL query used to retrieve the latest timestamp now assume that the timestamps are already sorted and retrieves the last value directly, instead of scanning the whole content of the tables finding the most recent timestamp.

[Copilot]

Pull request overview

This PR targets Live dashboard performance by reducing how often sensor metadata is fetched and by changing the “latest reading” SQL lookups to avoid scanning sensor tables.

Changes:

• Frontend: cache /api/sensors metadata and background-refresh it every 5 minutes instead of refetching on every live poll.
• Backend: change “latest timestamp / latest reading” queries to fetch the last row via descending id order.

Reviewed changes

Copilot reviewed 2 out of 2 changed files in this pull request and generated 2 comments.

File	Description
src/simoc_sam/api.py	Updates “latest” SQL queries to use ORDER BY id DESC LIMIT 1.
frontend/app.js	Adds sensor-metadata TTL caching and background refresh behavior for the Live dashboard.

Comments suppressed due to low confidence (1)

src/simoc_sam/api.py:174

• /api/latest orders by id instead of timestamp, which changes semantics from “most recent reading” to “most recently inserted row”. If any readings are inserted out of chronological order, the API will return a stale reading even though newer timestamps exist in the table.

            try:
                row = conn.execute(
                    f'SELECT sensor_id, timestamp, {", ".join(metrics)} '
                    f'FROM {sensor} ORDER BY id DESC LIMIT 1'
                ).fetchone()

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

[Copilot]

Pull request overview

Copilot reviewed 4 out of 4 changed files in this pull request and generated no new comments.

Comments suppressed due to low confidence (5)

src/simoc_sam/api.py:173

• Per-sensor timing logs at INFO will generate a log line per sensor per poll, which can be very noisy in production and degrade performance. Consider lowering this to DEBUG (or gating behind a config flag).

            app.logger.info('api_latest(%s): %.3fs', sensor, time.perf_counter() - t0)

frontend/app.js:125

• PR description says the sensor list is fetched once at the beginning and then every 5 minutes; the current implementation only fetches /api/sensors once on page load. Either update the PR description or refetch /api/sensors during the 5-minute discovery window (and optionally URL-encode sensor names when building the filter).

  const isDiscovery = Date.now() - discoveryLastRun > DISCOVERY_TTL;
  if (isDiscovery) discoveryLastRun = Date.now();
  const url = (isDiscovery || !activeSensors.size)
    ? '/api/latest'
    : `/api/latest?sensors=${[...activeSensors].join(',')}`;

src/simoc_sam/api.py:170

• /api/latest currently orders by id to pick the last inserted row. Because sqlwriter inserts the timestamp value coming from MQTT payloads (with no ordering/monotonicity enforcement), id is not guaranteed to correspond to the most recent timestamp; this can return a stale reading as “latest”. Prefer ordering by timestamp (and optionally id as a tie-breaker) to preserve the endpoint’s semantics.

                row = conn.execute(
                    f'SELECT sensor_id, timestamp, {", ".join(metrics)} '
                    f'FROM {sensor} ORDER BY id DESC LIMIT 1'
                ).fetchone()

src/simoc_sam/api.py:184

• The total timing log at INFO will be emitted on every /api/latest request (which is polled frequently by the dashboard). This is likely too noisy at INFO; DEBUG is a better default for request-level timing metrics.

        app.logger.info('api_latest total: %.3fs, %d/%d sensors',
                        time.perf_counter() - t_total, len(result), len(sensors_to_query))

tests/test_api.py:67

• This test name no longer matches the behavior being asserted (the endpoint is now static and omits active/has_data fields rather than returning inactive sensors). Renaming the test will make failures easier to interpret.

def test_sensors_inactive_when_empty(client):