observability: add metrics instrumentation to DataParallelInferenceCoordinator

[DhineshPonnarasan]

Summary

Implements PR 2 — Observability Enhancements for the inference coordinator as part of:

• Issue: Prefix-Cache Coordinator Follow-ups: Protocol Hardening, Metrics Integration, TTL Policy and Churn Validation #4176
• Follows: [Inference] Protocol Robustness Improvements in Coordinator (Follow-up to #4176) #4419 (Protocol Robustness)

This PR builds on protocol robustness and adds production-ready observability for routing quality, reliability, and latency — without modifying routing logic or protocol behavior.

---

Context

Issue #4176 tracks follow-up improvements after prefix-cache routing enhancements.

• PR 1 ([Inference] Protocol Robustness Improvements in Coordinator (Follow-up to #4176) #4419) → Protocol robustness (validation, crash safety, error classification)
• PR 2 (this PR) → Observability (metrics, routing quality, latency)
• Planned PR 3 → Metadata policy extensions (TTL-based expiration)

This PR is intentionally scoped to observability only to keep review focused and low-risk.

---

What’s Included

1. Metrics Abstraction

New file:

• megatron/core/inference/coordinator_metrics.py

Provides:

• CoordinatorMetrics (minimal interface: inc, observe, gauge)
• NoOpMetrics (default zero-overhead implementation)

Design goals:

• Backend-agnostic (Prometheus / StatsD / OpenTelemetry compatible)
• No global state
• Safe default when observability is disabled

---

2. Coordinator Instrumentation

Modified:

• megatron/core/inference/data_parallel_inference_coordinator.py

Routing Quality

• routing_cache_hit_total
• routing_cache_miss_total
• routing_stale_detected_total

Reliability

• coordinator_invalid_message_total
• coordinator_internal_error_total
• coordinator_unknown_sender_total
• coordinator_engine_unreachable_total
• coordinator_all_engines_exhausted_total

Latency

• coordinator_routing_latency_seconds
• coordinator_message_processing_latency_seconds

System State

• coordinator_active_engines (gauge)

---

3. Implementation Details

• Metrics injected via:

metrics: CoordinatorMetrics | None = None

• Defaults to NoOpMetrics (zero overhead when disabled)

• _log_protocol_error(...):

  • Centralizes error classification
  • Aligns with PR [Inference] Protocol Robustness Improvements in Coordinator (Follow-up to #4176) #4419 (client_error vs internal_error)

• Double-count prevention:

  • record_metrics guard ensures routing metrics are emitted once per request

• Latency:

  • Uses time.monotonic()
  • Message latency recorded via try/finally (covers failures)

• No changes to:

  • Routing decisions
  • Prefix cache logic
  • Protocol semantics

---

4. Tests

New file:

• tests/unit_tests/inference/test_coordinator_observability.py

Includes 32 unit tests covering:

• Metric emission correctness
• Routing scenarios (hit / miss / stale / fallback)
• Unknown sender handling
• Engine failure and exhaustion scenarios
• Latency metrics (routing + message processing)
• NoOpMetrics behavior
• Metrics injection via entrypoint
• Double-count prevention

---

Metric Naming

Prefix	Meaning
coordinator_*	System-level metrics
routing_*	Routing quality metrics

---

Testing

pytest tests/unit_tests/inference/test_coordinator_observability.py -v

Result:

• 32 passed

Note:
Full test suite may not run locally due to pre-existing environment constraints (e.g., missing fused_a2a_config, platform-specific signal handling). These are unrelated to this PR.

---

Scope

• Observability only
• No behavior changes
• No modifications to routing or protocol logic

---

Follow-up Work

This PR completes PR 2 — Observability Enhancements as outlined in #4419.

Planned PR 3 — Metadata Policy Extensions

• TTL-based expiration for prefix metadata
• Integration with existing cap-based eviction

Additional validation and performance testing (as described in #4176) will be addressed separately.

---

Result

The coordinator now exposes:

• Actionable routing quality signals
• Reliability diagnostics
• Latency visibility

while maintaining existing behavior and performance characteristics.

---

Closes #4176 (observability section)

---

@shanmugamr1992 @Phlip79 @YangFei1990 could you please take a look when you have time?

Happy to incorporate any feedback including additional tests, refinements or scope adjustments if needed.

[copy-pr-bot]

This pull request requires additional validation before any workflows can run on NVIDIA's runners.

Pull request vetters can view their responsibilities here.

Contributors can view more details about this message here.