feat: Measure system errors using a counter

[morgan-wowk]

TL;DR

Added OpenTelemetry metrics instrumentation to track execution system errors in the orchestrator.

Business value

We will be able to track the rate of system errors and respond to high or increasing rates.

Future iterations

In the future we will emit metrics for state transitions in general, and use measurement attributes to create dimensions on status, then we will have the option to deprecate to deprecate this measurement specific to system errors.

What changed?

• Created a new metrics module (cloud_pipelines_backend/instrumentation/metrics.py) that defines an orchestrator meter and an execution_system_errors counter instrument
• Integrated the metrics counter into the record_system_error_exception function in orchestrator_sql.py to increment the counter when system errors occur
• Added comprehensive OpenTelemetry strategy documentation (otel_strategy.md) covering best practices for meters, instruments, temporality, and aggregation

How to test?

• Cherry pick the OTel stack from https://app.graphite.com/github/pr/TangleML/tangle/101/feat-Add-development-observability-stack-for-testing-OpenTelemetry-with-Jaeger%2C-and-Prometheus
• Run the stack (docker-compose up -d)
• Run export TANGLE_OTEL_TRACE_EXPORTER_ENDPOINT="http://localhost:4317" && export TANGLE_OTEL_TRACE_EXPORTER_PROTOCOL="grpc" && export TANGLE_OTEL_METRIC_EXPORTER_ENDPOINT="http://localhost:4317" && export TANGLE_OTEL_METRIC_EXPORTER_PROTOCOL="grpc"
• Trigger execution nodes that result in SYSTEM_ERROR status and verify that the execution.system_errors metric is incremented in your OpenTelemetry metrics backend. Add a line raise RuntimeError("Temporary") to the start of https://github.com/TangleML/tangle/blob/system-error-metric/cloud_pipelines_backend/orchestrator_sql.py#L209

Why make this change?

This enables monitoring and alerting on system errors in pipeline executions, providing better observability into the health and reliability of the orchestrator component. The metrics follow OpenTelemetry semantic conventions and provide a foundation for expanding observability coverage across the application.

[morgan-wowk]

• feat: Measure system errors using a counter #127 👈 (View in Graphite)
• feat: Setup metric provider for manual and auto-instrumented measurements #126
• chore!: Refactor OTel exporters to be signal-specific #125
• feat: Set service version on OTel data #124
• chore: Refactor OTel concerns in preparation for metric provider #123
• master

This stack of pull requests is managed by Graphite. Learn more about stacking.

[Ark-kun]

Should we increment system error counter or would it be better to record a system error event?

[morgan-wowk]

In terms of Metrics (counter, histogram, gauge), a counter is the most applicable here. However, considering it is an error we could report an error to Observe (they mimic "bugnsag") in the future, with extra details on the error.

[morgan-wowk]

Merge activity

• Mar 5, 9:25 PM UTC: A user started a stack merge that includes this pull request via Graphite.
• Mar 5, 9:27 PM UTC: Graphite couldn't merge this pull request because a downstack PR feat: Set service version on OTel data #124 failed to merge.
• Mar 5, 9:43 PM UTC: A user started a stack merge that includes this pull request via Graphite.
• Mar 5, 9:48 PM UTC: Graphite rebased this pull request as part of a merge.
• Mar 5, 9:49 PM UTC: @morgan-wowk merged this pull request with Graphite.