Skip to content

Emit physical provider and classifier attempt lifecycles through Relay #299

Description

@afourniernv

Summary

The NeMo Relay plugin currently records the logical Hermes LLM lifecycle and emits switchyard.routing.requested and switchyard.routing.decision marks, but it does not emit a nested lifecycle for the physical provider request or classifier request performed by Switchyard.

This is a product-observability gap in the initial plugin design, not a regression from the later typed-SDK refactor.

Current behavior

A routed buffered call emits four Relay events:

  1. Outer logical LLM start
  2. switchyard.routing.requested mark
  3. switchyard.routing.decision mark
  4. Outer logical LLM end

The decision mark identifies the selected target, and the final response may contain a provider-returned model. However, there is no distinct lifecycle for:

  • the physical provider and model attempt
  • a classifier-model request
  • retry or fallback attempts as independently timed calls
  • per-attempt outcome and duration

Proposed behavior

Represent physical provider, classifier, retry, and fallback attempts as nested Relay lifecycles under the outer logical call, while retaining routing decision marks for the decision itself.

The exact event shape should be agreed with Relay maintainers so scope ownership and context propagation remain correct when Switchyard work runs on its own asynchronous executor.

Acceptance criteria

  • A routed call preserves the outer logical LLM lifecycle.
  • Each physical provider attempt has an attributable provider/model target, outcome, and duration.
  • Classifier requests are distinguishable from final provider attempts.
  • Retries and trusted fallback attempts are observable independently.
  • Streaming attempts close exactly once on success, error, or cancellation.
  • Existing routing-requested and routing-decision marks remain available.
  • Automated tests verify parent-child relationships and terminal lifecycle balance.

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions