feat(server): add startup and warmup time metrics to stats and Prometheus

[rickstaa]

This pull request adds the server startup time and pipeline warmup duration to the /stats API response and Prometheus metrics. It also refactors the code to avoid using async operations on class properties for improved clarity.

Important

I currently do not bundle tracks per stream since we are not yet doing multi streams right now I however already named the metrics and stats endpoints correctly to enable this int he future.

API format

@ecmulli I now have the following Rest API format for all streams (tracks in our current implementation):

{
  "1ce52800-09ab-4794-ae5e-1cbfa81a6f0c": {
    "type": "video",
    "timestamp": 106.241065119,
    "startup_time": 38.9035260609999,
    "pipeline": {
      "warmup": 38.8420334059992
    },
    "fps": 15.9929796257085,
    "minute_avg_fps": 15.5837784066638,
    "minute_fps_array": [
      {
        "timestamp": 46.0813516990002,
        "fps": 14.9898379691225
      },
      {
        "timestamp": 47.0824982550002,
        "fps": 14.9828213562756
      },
      {
        "timestamp": 48.0853169250004,
        "fps": 15.9550280411094
      },
      {
        "timestamp": 49.0855406540004,
        "fps": 14.9966448156516
      },
      {
        "timestamp": 50.0866806160002,
        "fps": 15.9817813765417
      },
      {
        "timestamp": 51.0905196379999,
        "fps": 15.9388105556273
      },
      {
        "timestamp": 52.0911730790003,
        "fps": 14.9902047855888
      },
      {
        "timestamp": 53.0916817450006,
        "fps": 15.9918654817478
      },
      {
        "timestamp": 54.0919209820004,
        "fps": 14.9964123033138
      },
      {
        "timestamp": 55.0939969760002,
        "fps": 15.9668529091645
      },
      {
        "timestamp": 56.1008033430007,
        "fps": 15.8918343431488
      },
      {
        "timestamp": 57.1018946520007,
        "fps": 14.9836482098557
      },
      {
        "timestamp": 58.1026354080004,
        "fps": 15.9881566770176
      },
      {
        "timestamp": 59.103513172,
        "fps": 15.9859680927095
      },
      {
        "timestamp": 60.1039121000003,
        "fps": 15.9936196972762
      },
      {
        "timestamp": 61.1040758220006,
        "fps": 14.9975445720038
      },
      {
        "timestamp": 62.1047987130005,
        "fps": 15.9884420991042
      },
      {
        "timestamp": 63.10824546,
        "fps": 15.9450414761345
      },
      {
        "timestamp": 64.1107873740002,
        "fps": 15.9594324951058
      },
      {
        "timestamp": 65.111357064,
        "fps": 14.9914595154318
      },
      {
        "timestamp": 66.1115922610006,
        "fps": 15.996237732864
      },
      {
        "timestamp": 67.1207460300002,
        "fps": 15.8548681989879
      },
      {
        "timestamp": 68.1271835670004,
        "fps": 15.8976582368829
      },
      {
        "timestamp": 69.1327986480001,
        "fps": 15.9106603533583
      },
      {
        "timestamp": 70.1331748110006,
        "fps": 14.9943596766733
      },
      {
        "timestamp": 71.1339854560001,
        "fps": 15.9870401858158
      },
      {
        "timestamp": 72.1353092630006,
        "fps": 15.9788470903616
      },
      {
        "timestamp": 73.1462026540003,
        "fps": 14.838359943344
      },
      {
        "timestamp": 74.1483370550004,
        "fps": 14.9680521744685
      },
      {
        "timestamp": 75.1488004319999,
        "fps": 15.9925894019086
      },
      {
        "timestamp": 76.1502615580002,
        "fps": 14.9781150866111
      },
      {
        "timestamp": 77.155187763,
        "fps": 15.9215670965655
      },
      {
        "timestamp": 78.1559935980004,
        "fps": 14.9879222077019
      },
      {
        "timestamp": 79.1571041800007,
        "fps": 10.9877971502613
      },
      {
        "timestamp": 80.1574795790002,
        "fps": 15.9939958699525
      },
      {
        "timestamp": 81.1576677050007,
        "fps": 14.997178640764
      },
      {
        "timestamp": 82.1584126940006,
        "fps": 15.988089049528
      },
      {
        "timestamp": 83.159186207,
        "fps": 14.9884062729087
      },
      {
        "timestamp": 84.1632987900002,
        "fps": 15.9344681770566
      },
      {
        "timestamp": 85.1703840110004,
        "fps": 15.8874340188515
      },
      {
        "timestamp": 86.1761477999999,
        "fps": 15.9083078700976
      },
      {
        "timestamp": 87.1770312900007,
        "fps": 15.985876637835
      },
      {
        "timestamp": 88.1777327560003,
        "fps": 14.9894853856511
      },
      {
        "timestamp": 89.1795259480005,
        "fps": 15.9713602845063
      },
      {
        "timestamp": 90.179654609,
        "fps": 14.9980703332802
      },
      {
        "timestamp": 91.1838322050007,
        "fps": 15.9334365392351
      },
      {
        "timestamp": 92.1850631120005,
        "fps": 15.9803297003126
      },
      {
        "timestamp": 93.1852021330005,
        "fps": 14.9979149748623
      },
      {
        "timestamp": 94.1853954349999,
        "fps": 15.9969077657448
      },
      {
        "timestamp": 95.1909038390004,
        "fps": 15.9123483566554
      },
      {
        "timestamp": 96.1927101649999,
        "fps": 15.9711508948967
      },
      {
        "timestamp": 97.194059075,
        "fps": 15.9784465137106
      },
      {
        "timestamp": 98.1956132430005,
        "fps": 15.975171899033
      },
      {
        "timestamp": 99.1980980730004,
        "fps": 15.9603412652146
      },
      {
        "timestamp": 100.198506502,
        "fps": 15.9934678039473
      },
      {
        "timestamp": 101.204589663001,
        "fps": 15.9032579216257
      },
      {
        "timestamp": 102.205073956,
        "fps": 15.9922550628298
      },
      {
        "timestamp": 103.20563323,
        "fps": 15.9910566178028
      },
      {
        "timestamp": 104.205915459001,
        "fps": 15.9954856100801
      },
      {
        "timestamp": 105.206354425,
        "fps": 15.9929796257085
      }
    ]
  }
}

And the following for a specfici track:

{
  "type": "video",
  "timestamp": 162.168192797,
  "startup_time": 38.9035260609999,
  "pipeline": {
    "warmup": 38.8420334059992
  },
  "fps": 16.987885666924,
  "minute_avg_fps": 16.0488480278423,
  "minute_fps_array": [
    {
      "timestamp": 102.205073956,
      "fps": 15.9922550628298
    },
    {
      "timestamp": 103.20563323,
      "fps": 15.9910566178028
    },
    {
      "timestamp": 104.205915459001,
      "fps": 15.9954856100801
    },
    {
      "timestamp": 105.206354425,
      "fps": 15.9929796257085
    },
    {
      "timestamp": 106.207739903,
      "fps": 15.977863022296
    },
    {
      "timestamp": 107.209350186,
      "fps": 15.9742768934885
    },
    {
      "timestamp": 108.216190589001,
      "fps": 15.8912971234708
    },
    {
      "timestamp": 109.217083875,
      "fps": 15.9857201799768
    },
    {
      "timestamp": 110.217299918,
      "fps": 15.9965440586261
    },
    {
      "timestamp": 111.21952256,
      "fps": 15.964516594912
    },
    {
      "timestamp": 112.220512467,
      "fps": 15.9841771511476
    },
    {
      "timestamp": 113.221640537,
      "fps": 15.9819712177276
    },
    {
      "timestamp": 114.222908992,
      "fps": 15.9797304310334
    },
    {
      "timestamp": 115.223929026,
      "fps": 15.9836960865421
    },
    {
      "timestamp": 116.233466604001,
      "fps": 16.8393929760158
    },
    {
      "timestamp": 117.449749522,
      "fps": 12.3326569649346
    },
    {
      "timestamp": 118.452482709,
      "fps": 15.9563882071908
    },
    {
      "timestamp": 119.453293968,
      "fps": 15.9870303777175
    },
    {
      "timestamp": 120.453641548001,
      "fps": 16.9940931930815
    },
    {
      "timestamp": 121.453957138,
      "fps": 15.9949521530589
    },
    {
      "timestamp": 122.461191155,
      "fps": 15.8850870105198
    },
    {
      "timestamp": 123.471775092,
      "fps": 15.8324305524778
    },
    {
      "timestamp": 124.472631485,
      "fps": 15.9863094364947
    },
    {
      "timestamp": 125.472986078,
      "fps": 15.9943285230752
    },
    {
      "timestamp": 126.476786644001,
      "fps": 16.9356350014118
    },
    {
      "timestamp": 127.478937217,
      "fps": 15.9656646726408
    },
    {
      "timestamp": 128.480862563,
      "fps": 16.9673320151661
    },
    {
      "timestamp": 129.481836147001,
      "fps": 16.98346517004
    },
    {
      "timestamp": 130.484739194,
      "fps": 15.9536857005949
    },
    {
      "timestamp": 131.485627126001,
      "fps": 15.9858056915759
    },
    {
      "timestamp": 132.486647989001,
      "fps": 15.9836828495732
    },
    {
      "timestamp": 133.494008022,
      "fps": 16.8757936021915
    },
    {
      "timestamp": 134.494953873001,
      "fps": 15.9848806846144
    },
    {
      "timestamp": 135.495546866,
      "fps": 15.9905177349268
    },
    {
      "timestamp": 136.496165197001,
      "fps": 15.9901128175414
    },
    {
      "timestamp": 137.496750996001,
      "fps": 15.9906327033536
    },
    {
      "timestamp": 138.501991095,
      "fps": 16.9113826805302
    },
    {
      "timestamp": 139.502609806001,
      "fps": 15.9901067450618
    },
    {
      "timestamp": 140.503367209,
      "fps": 15.9878907236083
    },
    {
      "timestamp": 141.50791077,
      "fps": 16.9231088227543
    },
    {
      "timestamp": 142.509006373,
      "fps": 15.9824895365153
    },
    {
      "timestamp": 143.510184913001,
      "fps": 15.9811655571414
    },
    {
      "timestamp": 144.51111339,
      "fps": 15.985158148327
    },
    {
      "timestamp": 145.511809887001,
      "fps": 15.9888638043154
    },
    {
      "timestamp": 146.51276852,
      "fps": 15.9846765615577
    },
    {
      "timestamp": 147.513011469,
      "fps": 16.995870870175
    },
    {
      "timestamp": 148.524254720001,
      "fps": 15.8221080676393
    },
    {
      "timestamp": 149.525863454001,
      "fps": 15.9743015978932
    },
    {
      "timestamp": 150.525976776,
      "fps": 15.9981870534567
    },
    {
      "timestamp": 151.526877439001,
      "fps": 15.9856023594106
    },
    {
      "timestamp": 152.536060731,
      "fps": 16.8453046485871
    },
    {
      "timestamp": 153.536769821,
      "fps": 15.9886625992395
    },
    {
      "timestamp": 154.537333114,
      "fps": 15.990992385931
    },
    {
      "timestamp": 155.546516155,
      "fps": 11.8908062387845
    },
    {
      "timestamp": 156.554252246,
      "fps": 16.869496043484
    },
    {
      "timestamp": 157.562264711,
      "fps": 15.87281958859
    },
    {
      "timestamp": 158.562697527001,
      "fps": 15.9930779399738
    },
    {
      "timestamp": 159.569070417,
      "fps": 16.892346931174
    },
    {
      "timestamp": 160.569875478001,
      "fps": 15.9871293856234
    },
    {
      "timestamp": 161.570588594001,
      "fps": 16.987885666924
    }
  ]
}

If you want me to change something let me know.

Prometheus format

For the prometheus metrics I now have the ability to specify the track_id and track_kind as labels but I disable them on default. This leads the the following default prometheus metrics:

# HELP python_gc_objects_collected_total Objects collected during gc
# TYPE python_gc_objects_collected_total counter
python_gc_objects_collected_total{generation="0"} 132488.0
python_gc_objects_collected_total{generation="1"} 16655.0
python_gc_objects_collected_total{generation="2"} 110458.0
# HELP python_gc_objects_uncollectable_total Uncollectable objects found during GC
# TYPE python_gc_objects_uncollectable_total counter
python_gc_objects_uncollectable_total{generation="0"} 0.0
python_gc_objects_uncollectable_total{generation="1"} 0.0
python_gc_objects_uncollectable_total{generation="2"} 0.0
# HELP python_gc_collections_total Number of times this generation was collected
# TYPE python_gc_collections_total counter
python_gc_collections_total{generation="0"} 3530.0
python_gc_collections_total{generation="1"} 319.0
python_gc_collections_total{generation="2"} 15.0
# HELP python_info Python platform information
# TYPE python_info gauge
python_info{implementation="CPython",major="3",minor="11",patchlevel="11",version="3.11.11"} 1.0
# HELP process_virtual_memory_bytes Virtual memory size in bytes.
# TYPE process_virtual_memory_bytes gauge
process_virtual_memory_bytes 7.827705856e+010
# HELP process_resident_memory_bytes Resident memory size in bytes.
# TYPE process_resident_memory_bytes gauge
process_resident_memory_bytes 3.165179904e+09
# HELP process_start_time_seconds Start time of the process since unix epoch in seconds.
# TYPE process_start_time_seconds gauge
process_start_time_seconds 1.74342015548e+09
# HELP process_cpu_seconds_total Total user and system CPU time spent in seconds.
# TYPE process_cpu_seconds_total counter
process_cpu_seconds_total 62.51
# HELP process_open_fds Number of open file descriptors.
# TYPE process_open_fds gauge
process_open_fds 52.0
# HELP process_max_fds Maximum number of open file descriptors.
# TYPE process_max_fds gauge
process_max_fds 1.048576e+06
# HELP stream_fps Frames per second for the stream. Defaults to all tracks; specific tracks when labels are applied.
# TYPE stream_fps gauge
stream_fps 15.986345710422425
# HELP stream_startup_time Startup time for the stream. Defaults to all tracks; specific tracks when labels are applied.
# TYPE stream_startup_time gauge
stream_startup_time 38.04632097299964
# HELP stream_warmup_time Warmup time for the stream pipeline. Defaults to all tracks; specific tracks when labels are applied.
# TYPE stream_warmup_time gauge
stream_warmup_time 37.98139018399979

And with the track label and track kind enabled:

[Copilot]

Pull Request Overview

This PR adds startup and warmup metrics to the server’s stats API and Prometheus endpoints and refactors asynchronous operations on class properties for better clarity. Key changes include:

• Changing FPSMeter property methods to async functions.
• Introducing a PipelineStats class and integrating startup/warmup metrics.
• Refactoring stream statistics collection and renaming routes/methods for consistency.

Reviewed Changes

Copilot reviewed 7 out of 7 changed files in this pull request and generated 1 comment.

Show a summary per file

File	Description
server/utils/fps_meter.py	Refactor to use async methods and add a metrics callback.
server/pipeline.py	Added warmup time calculations and PipelineStats usage.
server/metrics/stream_stats.py	Updated stats collection and renamed methods for clarity.
server/metrics/prometheus_metrics.py	Modified function names and added warmup time gauges.
server/metrics/pipeline_stats.py	Introduced new PipelineStats for warmup metrics.
server/metrics/init.py	Minor update to export StreamStats.
server/app.py	Updated stream track classes to use StreamStats for metrics.

Comments suppressed due to low confidence (2)

server/utils/fps_meter.py:32

• The module uses asyncio functionalities (e.g., asyncio.Lock() and asyncio.create_task()) but does not import asyncio. Consider adding 'import asyncio' at the top of the file.

self._lock = asyncio.Lock()

server/app.py:149

• The variable 'app' is used to retrieve 'metrics_manager' in AudioStreamTrack, yet 'app' is not defined in this context. Ensure that the correct reference is provided to access metrics_manager.

self.stats = StreamStats(

[Copilot]

The variable 'app' is used to retrieve 'metrics_manager', but 'app' is not defined in the constructor for VideoStreamTrack. Pass the app or metrics_manager reference via a parameter instead.

[rickstaa]

@eliteprox last time I looked at it the prometheus metrics were not yeat working reliably.