Current implementation status — August 2026
The active direction is now a Switchyard-owned external native plugin built
against NeMo Relay's published native API v1 Rust SDK. The plugin drives
libsy::Algorithm::run_stream in-process while Switchyard owns provider HTTP
dispatch through switchyard-llm-client; it does not require
switchyard-server, the Decision API, a sidecar, or a Relay core/ABI change.
| Work |
Status |
Role in the integration |
| Switchyard #270 |
Open, ready for review, CI green |
Establishes the external nvidia.switchyard plugin on Relay 0.7.0-rc.5. It supports seeded weighted-random and LLM-classifier routing for buffered and streaming OpenAI Chat, OpenAI Responses, and Anthropic Messages traffic, with routing marks, retries, exactly-once fallback, and preserved same-protocol stream events. |
| Switchyard #271 |
Open, ready for review, CI green |
Independently hardens the shared provider HTTP/SSE client with redirect rejection, timeouts, bounded bodies and frames, and diagnostic redaction. It is beneficial for production use but is not a functional or merge dependency of Switchyard #270. |
| Switchyard #274 |
Draft, CI green |
Independently enforces buffered response loss policy. It complements Switchyard #270 by turning unsupported strict cross-protocol response conversion into a translation error that the plugin can send through its trusted-fallback path. |
| Switchyard #250 |
Draft reference implementation |
Prototypes StageRouter, including request mutation, tier prompts, handoff notes, and classifier fallback. It is based on the superseded Switchyard #220/native-API-v2 integration and will be refactored onto the plugin architecture introduced by Switchyard #270 rather than landed as-is. |
| Switchyard #251 |
Draft reference implementation |
Prototypes response-judging escalation, including weak/judge/strong calls and retained raw stream replay. It is also based on the superseded Switchyard #220/native-API-v2 integration and will be refactored onto Switchyard #270, with production bounds and remaining escalation tests added before review. |
Packaging status
Switchyard #270
currently adds a non-published Rust cdylib crate and
crates/switchyard-nemo-relay-plugin/scripts/package_bundle.py. This is enough
to land the source integration and produce an installable bundle containing the
shared library, materialized Relay manifest, configuration schema, and integrity
digest; publishing supporting crates and automating platform-specific release
bundles can follow in Switchyard CI.
Planned sequence
- Review and land Switchyard #270
as the initial random and LLM-classifier integration.
- Review Switchyard #271
independently as shared transport hardening; it does not block the plugin PR.
- Complete review of Switchyard #274
as independent buffered-translation hardening.
- Rebase or reimplement the router-specific work from
Switchyard #250 and
Switchyard #251 as
separate follow-ups on the external plugin introduced by the initial plugin
PR. Do not carry forward their obsolete stacked base or Relay native-API-v2
continuation assumptions.
- Add release automation and published artifacts after the source-built bundle
contract is accepted.
The outcome and scope below predate this external-plugin direction and are
retained as historical context. Their built-in Relay component,
DecisionProvider, Python-native wiring, and sidecar-compatibility assumptions
should be reconciled with the status above after the initial plugin lands.
Outcome
Deliver a production-ready Switchyard integration for NeMo Relay that:
- Works through native Relay runtime and plugin-host lifecycles, including Hermes, LangChain Deep Agents, and future native harnesses, without requiring the CLI wrapper as the integration boundary.
- Can be discovered, configured, and executed by the published Python package.
- Is available as a supported, published Rust crate or clearly defined crate split.
- Supports an in-process Switchyard
DecisionProvider without requiring a separately managed sidecar.
- Retains the HTTP Decision API as a supported compatibility and deployment mode.
Motivation
The current experimental nemo-relay-switchyard integration is source-only, CLI feature-gated, and uses the Switchyard Decision API over HTTP. This is sufficient for the existing sidecar workflow, but it does not support native Relay harness integrations or provide a stable consumer-facing package.
The Python wheel also cannot currently configure the plugin: the native extension neither depends on nor registers nemo-relay-switchyard, so ComponentSpec(kind="switchyard", ...) is treated as an unknown component. Publishing the Rust crate alone does not solve this; Python support also requires native-extension wiring, component registration, configuration coverage, and runtime documentation.
As Switchyard evolves toward a library-first architecture, Relay should be able to use its decision and accumulation components in-process while preserving service-based deployment compatibility.
Scope
Included:
- Define the native host/plugin contract for configuration, lifecycle, capabilities, and error reporting.
- Register and activate the plugin through native and worker/plugin-host lifecycles.
- Support Hermes, LangChain Deep Agents, and future native harnesses without using
nemo-relay-cli as the integration boundary.
- Preserve request and stream intercept behavior, target binding, retries, fallbacks, optimization accounting, and observability semantics.
- Wire the Switchyard component into the Python native extension and expose and test its configuration path.
- Define the public API, ownership boundary, feature flags, compatibility policy, dependency strategy, documentation, and release CI for a publishable Rust crate or supported crate split.
- Keep the integration optional so users do not incur Switchyard dependencies unless they enable it.
- Add an in-process Switchyard
DecisionProvider behind a Relay-owned abstraction.
- Use Switchyard-owned ATOF accumulation, snapshots, decision logic, and provider-protocol translation where appropriate.
- Preserve a service/HTTP adapter during migration.
- Define snapshot freshness, lifecycle, concurrency, and failure semantics for embedded accumulators and decision providers.
- Keep the versioned request/decision contract compatible across service and library modes.
- Add supported examples that do not depend on internal services, credentials, or untracked development worktrees.
Current limitations to address:
- The crate has
publish = false and must be built from a Relay source checkout.
- The CLI feature is the supported activation path; native harness integrations are not covered.
- Decision API routing and ATOF-backed profiles require a reachable Switchyard service.
switchyard-translation is library-backed, but decision logic and ATOF accumulation remain service-bound.
- Existing examples are experimental compatibility and end-to-end workflows rather than a production deployment contract.
Non-goals
- Removing the HTTP Decision API or sidecar deployment mode during the migration.
- Making Switchyard dependencies mandatory for Relay users who do not enable the integration.
- Treating Rust crate publication as a substitute for Python native-extension integration.
- Depending on internal-only services, credentials, or development worktrees in supported examples.
Child work
- Document and implement the native host/plugin lifecycle contract.
- Add native-harness registration and integration tests.
- Wire the Switchyard component into the Python native extension, expose the configuration path, and add package-level tests.
- Define and stabilize the publishable crate boundary and feature/dependency policy.
- Add an in-process Switchyard
DecisionProvider behind a Relay-owned abstraction.
- Add service and in-process conformance tests against the same request/decision fixtures.
- Add snapshot freshness, embedded accumulator lifecycle, concurrency, and failure-mode tests.
- Update user-facing documentation and examples once the supported deployment model is established.
Related work:
Implementation should be split into independently reviewable pull requests.
Completion criteria
- Hermes and LangChain Deep Agents can activate and use Switchyard routing through native integration paths without the CLI wrapper.
- The published Python package can discover, configure, and execute the Switchyard component through its native/plugin API.
- A supported published Rust crate or package can be consumed without a Relay source checkout.
- Relay can run against an in-process Switchyard
DecisionProvider with no mandatory Switchyard sidecar.
- The HTTP Decision API remains available as a compatibility and deployment mode during migration.
- Service and library modes produce equivalent routing, translation, retry and fallback, optimization, and observability results for the same fixtures.
- ATOF-backed routing has documented snapshot freshness, lifecycle, concurrency, and failure behavior.
- Supported examples require no internal-only services, credentials, or development worktrees.
- Production readiness is documented with versioning, compatibility, performance, security, and operational guidance.
Current implementation status — August 2026
The active direction is now a Switchyard-owned external native plugin built
against NeMo Relay's published native API v1 Rust SDK. The plugin drives
libsy::Algorithm::run_streamin-process while Switchyard owns provider HTTPdispatch through
switchyard-llm-client; it does not requireswitchyard-server, the Decision API, a sidecar, or a Relay core/ABI change.nvidia.switchyardplugin on Relay0.7.0-rc.5. It supports seeded weighted-random and LLM-classifier routing for buffered and streaming OpenAI Chat, OpenAI Responses, and Anthropic Messages traffic, with routing marks, retries, exactly-once fallback, and preserved same-protocol stream events.StageRouter, including request mutation, tier prompts, handoff notes, and classifier fallback. It is based on the superseded Switchyard #220/native-API-v2 integration and will be refactored onto the plugin architecture introduced by Switchyard #270 rather than landed as-is.Packaging status
Switchyard #270
currently adds a non-published Rust
cdylibcrate andcrates/switchyard-nemo-relay-plugin/scripts/package_bundle.py. This is enoughto land the source integration and produce an installable bundle containing the
shared library, materialized Relay manifest, configuration schema, and integrity
digest; publishing supporting crates and automating platform-specific release
bundles can follow in Switchyard CI.
Planned sequence
as the initial random and LLM-classifier integration.
independently as shared transport hardening; it does not block the plugin PR.
as independent buffered-translation hardening.
Switchyard #250 and
Switchyard #251 as
separate follow-ups on the external plugin introduced by the initial plugin
PR. Do not carry forward their obsolete stacked base or Relay native-API-v2
continuation assumptions.
contract is accepted.
Outcome
Deliver a production-ready Switchyard integration for NeMo Relay that:
DecisionProviderwithout requiring a separately managed sidecar.Motivation
The current experimental
nemo-relay-switchyardintegration is source-only, CLI feature-gated, and uses the Switchyard Decision API over HTTP. This is sufficient for the existing sidecar workflow, but it does not support native Relay harness integrations or provide a stable consumer-facing package.The Python wheel also cannot currently configure the plugin: the native extension neither depends on nor registers
nemo-relay-switchyard, soComponentSpec(kind="switchyard", ...)is treated as an unknown component. Publishing the Rust crate alone does not solve this; Python support also requires native-extension wiring, component registration, configuration coverage, and runtime documentation.As Switchyard evolves toward a library-first architecture, Relay should be able to use its decision and accumulation components in-process while preserving service-based deployment compatibility.
Scope
Included:
nemo-relay-clias the integration boundary.DecisionProviderbehind a Relay-owned abstraction.Current limitations to address:
publish = falseand must be built from a Relay source checkout.switchyard-translationis library-backed, but decision logic and ATOF accumulation remain service-bound.Non-goals
Child work
DecisionProviderbehind a Relay-owned abstraction.Related work:
Implementation should be split into independently reviewable pull requests.
Completion criteria
DecisionProviderwith no mandatory Switchyard sidecar.