Summary
The NeMo Relay plugin currently records the logical Hermes LLM lifecycle and emits switchyard.routing.requested and switchyard.routing.decision marks, but it does not emit a nested lifecycle for the physical provider request or classifier request performed by Switchyard.
This is a product-observability gap in the initial plugin design, not a regression from the later typed-SDK refactor.
Current behavior
A routed buffered call emits four Relay events:
- Outer logical LLM start
switchyard.routing.requested mark
switchyard.routing.decision mark
- Outer logical LLM end
The decision mark identifies the selected target, and the final response may contain a provider-returned model. However, there is no distinct lifecycle for:
- the physical provider and model attempt
- a classifier-model request
- retry or fallback attempts as independently timed calls
- per-attempt outcome and duration
Proposed behavior
Represent physical provider, classifier, retry, and fallback attempts as nested Relay lifecycles under the outer logical call, while retaining routing decision marks for the decision itself.
The exact event shape should be agreed with Relay maintainers so scope ownership and context propagation remain correct when Switchyard work runs on its own asynchronous executor.
Acceptance criteria
- A routed call preserves the outer logical LLM lifecycle.
- Each physical provider attempt has an attributable provider/model target, outcome, and duration.
- Classifier requests are distinguishable from final provider attempts.
- Retries and trusted fallback attempts are observable independently.
- Streaming attempts close exactly once on success, error, or cancellation.
- Existing routing-requested and routing-decision marks remain available.
- Automated tests verify parent-child relationships and terminal lifecycle balance.
Summary
The NeMo Relay plugin currently records the logical Hermes LLM lifecycle and emits
switchyard.routing.requestedandswitchyard.routing.decisionmarks, but it does not emit a nested lifecycle for the physical provider request or classifier request performed by Switchyard.This is a product-observability gap in the initial plugin design, not a regression from the later typed-SDK refactor.
Current behavior
A routed buffered call emits four Relay events:
switchyard.routing.requestedmarkswitchyard.routing.decisionmarkThe decision mark identifies the selected target, and the final response may contain a provider-returned model. However, there is no distinct lifecycle for:
Proposed behavior
Represent physical provider, classifier, retry, and fallback attempts as nested Relay lifecycles under the outer logical call, while retaining routing decision marks for the decision itself.
The exact event shape should be agreed with Relay maintainers so scope ownership and context propagation remain correct when Switchyard work runs on its own asynchronous executor.
Acceptance criteria