[Enhancement]: Initialize discovered [[plugins.dynamic]] without caller-built specs in Python

[bbednarski9]

Affected area

Python binding, dynamic plugins

Observed behavior

Python can activate caller-supplied DynamicPluginActivationSpec objects, but it cannot initialize the standard [[plugins.dynamic]] records from a resolved or conventionally discovered plugins.toml.

This was reproduced against the published nemo-relay==0.6.0 wheel and the current 0.7.x release candidate.

This leaves embedded Python hosts with no file-backed dynamic initialization call:

Call	Result with the configuration below
await plugin.initialize({})	Discovers the file and initializes [[components]], but silently ignores [[plugins.dynamic]].
await plugin.initialize_with_dynamic_plugins({}, [])	Does not derive activation specs from the discovered file; raises ValueError because the caller-supplied list is empty.
await plugin.initialize_with_dynamic_plugins({}, [manual_spec])	Enters the dynamic loader, proving activation works only after the host has independently parsed and resolved the file into specs.

Use this project configuration:

# .nemo-relay/plugins.toml
version = 1

[[components]]
kind = "proof.static_marker"
enabled = true

[[plugins.dynamic]]
manifest = "./missing/relay-plugin.toml"

Then run:

import asyncio
from pathlib import Path

from nemo_relay import plugin


static_marker_calls = []


class StaticMarkerPlugin:
    def validate(self, config):
        return []

    def register(self, config, _context):
        static_marker_calls.append(config)


plugin.register("proof.static_marker", StaticMarkerPlugin())

# Standard discovery finds the file and initializes [[components]], but it
# silently ignores the missing [[plugins.dynamic]] manifest.
asyncio.run(plugin.initialize({}))
print(static_marker_calls)  # [{}]
plugin.clear()

# This is the integration-shaped failure: even though normal discovery can see
# the plugins.toml, the dynamic initializer does not turn its dynamic record
# into an activation spec.
try:
    asyncio.run(plugin.initialize_with_dynamic_plugins({}, []))
except Exception as error:
    print(type(error).__name__, error)

missing_manifest = Path(".nemo-relay/missing/relay-plugin.toml").resolve()
try:
    asyncio.run(
        plugin.initialize_with_dynamic_plugins(
            {},
            [
                {
                    "plugin_id": "proof.missing",
                    "kind": "rust_dynamic",
                    "manifest_ref": str(missing_manifest),
                    "config": {},
                }
            ],
        )
    )
except Exception as error:
    print(type(error).__name__, error)

Observed results:

1. initialize({}) succeeds and invokes proof.static_marker, proving that standard discovery read the file, even though the referenced dynamic-plugin manifest does not exist.

2. initialize_with_dynamic_plugins({}, []) fails with:

   ValueError: invalid config: dynamic plugin activation requires at least one dynamic plugin; use plugin initialization for a static-only configuration
   

   The discovered [[plugins.dynamic]] record is not used to populate the required list.

3. Supplying the missing manifest as an explicit activation spec makes Relay attempt to load it and raises FileNotFoundError.

The missing manifest is intentional: it is a sentinel showing whether a call consumed the standard dynamic record. Normal initialization never reaches it; an explicit spec immediately does.

Therefore, the missing layer is specifically standard plugins.toml dynamic discovery and resolution into the existing activation path. Dynamic activation itself is already public, but initialize_with_dynamic_plugins() cannot replace this layer because its second argument must already contain caller-resolved specs.

Why this blocks embedded integrations

An embedded host such as Hermes wants to select one ordinary Relay plugins.toml, initialize all configured static and dynamic plugins, retain the returned host activation for its process lifetime, and tear it down safely.

Today the host cannot accomplish that using Relay's Python API alone:

• initialize() omits the dynamic records.
• initialize_with_dynamic_plugins() rejects an empty list instead of discovering those records.
• Making the latter succeed requires the host to parse [[plugins.dynamic]], resolve manifest-relative paths and other activation metadata, construct DynamicPluginActivationSpec objects, and then own the resulting activation lifetime.

That caller-side conversion is the integration blocker. It duplicates Relay configuration semantics in Hermes and can drift from Relay's CLI and future configuration behavior.

Desired behavior

Provide one Python initialization flow where the caller supplies a normal plugins.toml path or uses Relay's standard discovery. Relay should internally parse, resolve, and activate both regular [[components]] and enabled [[plugins.dynamic]] entries from that configuration.

The caller should not need to parse TOML or construct DynamicPluginActivationSpec objects for records already present in the file. Relay may share internal resolution logic with other hosts, but CLI-specific resolver and lifecycle implementation types do not need to become public Python APIs.

The resulting activation must retain native libraries and workers for the full callback lifetime and provide deterministic teardown.

Acceptance criteria

• Python can initialize from one explicitly supplied or conventionally discovered standard plugins.toml.
• Regular components and enabled native or worker dynamic plugins from that configuration are activated together.
• Relay resolves manifest-relative paths internally; a missing dynamic manifest fails initialization instead of being silently ignored.
• Callers do not parse [[plugins.dynamic]] or build activation specs for the standard file-backed workflow.
• Initialization is atomic and rolls back partial dynamic loads and component registrations on failure.
• The owned lifetime removes callbacks and registrations before unloading libraries or stopping workers.
• Static-only and no-dynamic-plugin configurations continue to succeed.
• Existing explicit initialize_with_dynamic_plugins(...) behavior remains available.
• The file-backed path returns or otherwise exposes an owned activation whose lifetime can be retained and deterministically closed by an embedded host.
• Tests cover explicit-path and standard-discovery flows, mixed static/dynamic configuration, missing manifests, rollback, and teardown.

Downstream impact

Without this path, embedded hosts such as NousResearch/hermes-agent#77915 must define a second configuration shape and reproduce the file-to-activation conversion themselves.