Add Runtime Metrics Interface to `lume_pva.Runner`

[bisegni]

Description

Add lightweight runtime metrics collection to the lume_pva.Runner and expose the collected metrics through a Python method on the Runner class.

This is not an HTTP API and the Runner does not need to expose Prometheus, OpenTelemetry, or any other monitoring protocol directly.

The higher-level application embedding the Runner should be able to call something such as:

metrics = runner.get_metrics()

and combine those metrics with its own application-level metrics before exposing them externally.

The intent is to instrument the work that the Runner itself owns:

Input Updates
     |
     v
Runner Queue
     |
     +-- update coalescing
     |
     v
model.set(...)
     |
     v
model.get(...)
     |
     v
Output Publication
     |
     +-- PVA
     +-- CA
     |
     v
Higher-Level Application
     |
     +-- runner.get_metrics()
     +-- application metrics
     |
     v
External Monitoring

Why this belongs in Runner

The current Runner is the component that:

• receives local CA/PVA writes;
• receives remote PV monitor updates;
• handles snapshot requests;
• queues incoming updates;
• coalesces updates for update_rate;
• transitions the model between Idle and Simulating;
• calls model.reset() when requested;
• calls model.set(...);
• calls model.get(...);
• publishes model outputs through PVA and/or CA;
• reports put completion;
• restores cached model state after simulation failures.

These are the best locations for collecting operational metrics because the higher-level application cannot reliably reconstruct them from outside the Runner.

Proposed Python Interface

Add a method on Runner, for example:

metrics = runner.get_metrics()

The exact name and return type can be decided during implementation.

A typed, immutable snapshot is preferred, for example:

@dataclass(frozen=True)
class RunnerMetrics:
    ...

get_metrics() should return the current snapshot and must not reset counters.

A separate reset method can be added later if needed.

Metrics to Collect

1. Runner State / Health

These metrics describe whether the Runner is alive and able to process model updates.

Suggested fields:

• state
  • idle
  • simulating
  • ...
• simulation_active
• simulation_cycles_total
• simulation_cycles_failed_total
• last_simulation_started_at
• last_simulation_completed_at
• last_simulation_error
• time_since_last_successful_simulation_seconds
• 'simulation_latency_[max/min,mean]_time'

The Runner already tracks Idle / Simulating, so the metrics snapshot should reuse that state rather than create a second independent state model.

The last error should be treated as diagnostic state rather than a counter. It may be omitted if exposing error text through metrics is considered undesirable.

---

2. Input Update Metrics

Inputs can currently arrive through several paths:

• PVA PUT
• CA write
• remote PV monitor callback
• snapshot
• reset/control request
• initial update created when the Runner starts

Track the source where practical.

Suggested counters:

• input_updates_received_total
• input_variables_received_total
• pva_puts_total
• ca_puts_total
• remote_monitor_updates_total
• snapshot_requests_total
• reset_requests_total
• invalid_input_updates_total

It is useful to distinguish:

update events

from:

individual variable values contained in those updates

because one queued update may contain several variables.

For example, a snapshot can update many input variables at once.

---

3. Queue / Backpressure Metrics

The Runner uses an internal queue and intentionally coalesces updates during the configured update_rate window before starting a simulation.

This makes queue metrics particularly important.

Suggested metrics:

• queue_depth
• queue_depth_max
• queue_items_enqueued_total
• queue_items_dequeued_total
• queue_items_coalesced_total
• variables_coalesced_total
• update_batches_total
• average_updates_per_batch
• average_variables_per_batch
• queue_wait_time_ms
  • average
  • p50
  • p95
  • p99

A key metric should show how much incoming activity is being collapsed into a single model execution.

For example:

8 queued updates
      |
      | update_rate window
      v
1 model execution

This is important when determining whether the Runner is keeping up with incoming control-system updates.

---

4. End-to-End Update Latency

The Runner already associates timestamps with incoming values and propagates the newest input timestamp to the output PV updates.

Use this to measure end-to-end Runner latency.

Suggested metric:

• input_to_output_latency_ms
  • average
  • p50
  • p95
  • p99
  • max

Conceptually:

input update timestamp
        |
        v
queue/coalescing
        |
        v
model execution
        |
        v
output publication

This is one of the most useful operational metrics because it represents the delay seen by a user of the virtual accelerator.

---

5. Model Execution Metrics

The Runner has clearly separated execution stages around:

model.reset()
model.set(...)
model.get(...)

Collect latency independently for each stage.

Suggested metrics:

model.set()

• model_set_total
• model_set_failures_total
• model_set_latency_ms
  • average
  • p50
  • p95
  • p99
  • max

model.get()

• model_get_total
• model_get_failures_total
• model_get_latency_ms
  • average
  • p50
  • p95
  • p99
  • max

Full simulation cycle

• simulation_cycle_latency_ms
  • average
  • p50
  • p95
  • p99
  • max

The full cycle should measure from the point at which the Runner starts processing a coalesced update batch until output publication is complete.

This is more useful than a generic "inference latency" metric because not every supported LUME backend is an ML inference model. The same Runner may use Bmad/Tao, surrogate models, staged models, or other LUME implementations.

---

6. Reset and Recovery Metrics

The Runner supports explicit model reset and also restores cached model state after a failed simulation.

Suggested metrics:

• model_resets_total
• model_reset_failures_total
• model_reset_latency_ms
• cached_state_restore_total
• cached_state_restore_failures_total

This makes it possible to differentiate between:

normal execution failure

and:

failure followed by successful/failed state recovery

---

7. Output Publication Metrics

After model.get(...), the Runner iterates over returned values and publishes them through PVA and/or CA.

Suggested counters:

• output_variables_returned_total
• output_variables_published_total
• output_variables_skipped_total
• output_none_values_total
• output_publish_failures_total
• pva_posts_total
• pva_post_failures_total
• ca_updates_total
• ca_update_failures_total

Also collect:

• output_publication_latency_ms
  • average
  • p50
  • p95
  • p99
  • max

The Runner currently skips outputs whose value is None. This should have its own counter because frequent None outputs can indicate a model or configuration problem even when the simulation itself does not throw an exception.

---

8. Put Completion Metrics

The Runner supports two put modes:

• immediate
• complete

When complete mode is used, completion callbacks are invoked after the model execution finishes.

Useful metrics include:

• put_completion_callbacks_total
• put_completion_callback_failures_total
• put_completion_latency_ms

This is useful for diagnosing control clients that experience long PUT completion times or timeouts.

---

9. Snapshot Metrics

Snapshot mode performs synchronous reads of configured remote PVs before enqueuing the model update.

Suggested metrics:

• snapshot_requests_total
• snapshot_remote_pvs_read_total
• snapshot_failures_total
• snapshot_collection_latency_ms

If individual remote get() failures can be handled independently in the future, also track:

• snapshot_remote_pv_read_failures_total

---

10. Remote Input Monitor Metrics

For continuous remote input mode, updates arrive through monitor callbacks.

Suggested metrics:

• remote_monitor_updates_total
• remote_monitor_variables_total

Optionally, if subscription state is available:

• remote_monitors_configured
• remote_monitors_connected
• remote_monitor_disconnects_total

Connection-level metrics should only be added if the underlying PV client exposes them reliably.

---

Configuration / Identity Information

The metrics snapshot should also expose enough static information for the higher-level application to identify the Runner.

Suggested fields:

• model class/name
• Runner/lume-pva version
• configured protocols (ca, pva)
• configured put_mode
• configured remote_model_mode
• configured update_rate
• number of supported variables
• number of local input/output PVs
• number of remote variables

Do not duplicate the complete model_info PV payload unless needed.

---

Metrics That Should NOT Be Owned by Runner

Avoid putting general infrastructure metrics into this interface unless there is a strong reason.

For example:

• process CPU
• host memory
• container memory
• GPU utilization
• network throughput
• Kubernetes state

These are better collected by the higher-level deployment/infrastructure layer.

The Runner should focus on metrics that only it can accurately provide:

input -> queue -> model execution -> output

---

Suggested Initial Scope

The first implementation does not need every metric above.

The highest-value initial set is:

Runner health
    state
    uptime
    successful/failed cycles

Input
    updates received
    variables received
    invalid updates

Queue
    current/max depth
    updates coalesced

Execution
    cycle latency
    model.set latency
    model.get latency

Output
    values returned
    values published
    None values
    publication failures

Recovery
    resets
    simulation failures
    cached-state restores

End-to-end
    input-to-output latency

These metrics cover the major failure and performance boundaries without creating excessive instrumentation.

Example API

The following is only an example of the intended API shape:

metrics = runner.get_metrics()

print(metrics.state)
print(metrics.queue.depth)
print(metrics.execution.cycles_total)
print(metrics.execution.cycle_latency_ms.p95)

Possible structure:

RunnerMetrics(
    uptime_seconds=...,
    state=ModelState.Idle,

    input=InputMetrics(
        updates_received_total=...,
        variables_received_total=...,
        invalid_updates_total=...,
    ),

    queue=QueueMetrics(
        depth=...,
        max_depth=...,
        items_enqueued_total=...,
        items_coalesced_total=...,
    ),

    execution=ExecutionMetrics(
        cycles_total=...,
        cycles_failed_total=...,
        set_latency_ms=...,
        get_latency_ms=...,
        cycle_latency_ms=...,
    ),

    output=OutputMetrics(
        variables_returned_total=...,
        variables_published_total=...,
        none_values_total=...,
        publish_failures_total=...,
    ),
)

The implementation does not need to use this exact schema.

Implementation Requirements

Metrics collection should add negligible overhead to the simulation path.

Prefer:

• monotonic/performance clocks for durations;
• cumulative integer counters;
• gauges for current state;
• bounded histograms or streaming statistics for latency;
• no unbounded list of individual timing samples;
• thread-safe updates;
• immutable/copy-on-read metric snapshots.

get_metrics() must be safe to call while the Runner is processing updates.

Retrieving metrics must not block model execution for a significant amount of time.

The Runner should not depend on Prometheus or any external metrics library unless there is a compelling implementation reason.