fix: queue managed sanitizer publication - #698
Conversation
Signed-off-by: Will Killian <wkillian@nvidia.com>
|
No actionable comments were generated in the recent review. 🎉 ℹ️ Recent review info⚙️ Run configurationConfiguration used: Path: .coderabbit.yaml Review profile: ASSERTIVE Plan: Enterprise Run ID: 📒 Files selected for processing (1)
📜 Recent review details⏰ Context from checks skipped due to timeout. (2)
🧰 Additional context used📓 Path-based instructions (5)**/*📄 CodeRabbit inference engine (.agents/skills/validate-change/SKILL.md)
Files:
**/{test,tests}/**/*📄 CodeRabbit inference engine (CONTRIBUTING.md)
Files:
crates/node/**/*.{js,ts,mjs}📄 CodeRabbit inference engine (AGENTS.md)
Files:
crates/{python,ffi,node}/**/*⚙️ CodeRabbit configuration file
Files:
{crates/**/tests/**,python/tests/**,go/nemo_relay/**/*_test.go}⚙️ CodeRabbit configuration file
Files:
🔇 Additional comments (1)
WalkthroughLLM, tool, and stream lifecycle events now use asynchronous transformed dispatch. Payload and event sanitization occur before subscriber publication without blocking managed results or stream termination. Tests now flush subscribers before validating asynchronous behavior. ChangesQueued observability lifecycle
Estimated code review effort: 4 (Complex) | ~60 minutes Sequence Diagram(s)sequenceDiagram
participant ManagedExecution
participant PublicationDispatcher
participant SanitizerChains
participant Subscribers
ManagedExecution->>PublicationDispatcher: enqueue lifecycle event
PublicationDispatcher->>SanitizerChains: transform payload and event fields
SanitizerChains-->>PublicationDispatcher: return transformed event
PublicationDispatcher->>Subscribers: publish event
ManagedExecution-->>ManagedExecution: return result without waiting
Possibly related PRs
🚥 Pre-merge checks | ✅ 5✅ Passed checks (5 passed)
✨ Finishing Touches🧪 Generate unit tests (beta)
Comment |
![coderabbitai[bot]](https://avatars.githubusercontent.com/in/347564?s=60&v=4){kind=small}
There was a problem hiding this comment.
Actionable comments posted: 3
Caution
Some comments are outside the diff and can’t be posted inline due to platform limitations.
⚠️ Outside diff range comments (3)
crates/core/src/api/llm.rs (1)
418-481: 📐 Maintainability & Code Quality | 🟠 Major | ⚡ Quick winRemove the duplicated
#[cfg(test)]start-event pipeline.emit_llm_start_with_subscribersremains incrates/core/src/api/llm.rsand is reached bycrates/core/tests/unit/llm_api_tests.rsthroughemit_llm_start. It duplicates the sanitization, annotation decoding, freshness projection, and event construction inqueue_llm_start_with_subscribers. Delete the seam and test the queued path withflush_subscribers(), or extract a shared transformation helper.🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@crates/core/src/api/llm.rs` around lines 418 - 481, Remove the test-only emit_llm_start_with_subscribers pipeline and its duplicated sanitization, annotation, freshness, and event-building logic. Update emit_llm_start and the llm_api_tests coverage to exercise queue_llm_start_with_subscribers directly, using flush_subscribers() to observe queued events; alternatively extract and reuse a shared transformation helper so both paths cannot diverge.Source: Coding guidelines
go/nemo_relay/scope_local_test.go (1)
93-110: 🩺 Stability & Availability | 🟠 Major | ⚡ Quick winGuard the callback counter; it is now written from the dispatcher thread.
The guardrail callbacks now run on the native dispatcher instead of the calling goroutine.
FlushSubscribersis a cgo barrier. It orders native work, but it does not create a happens-before edge that the Go race detector recognizes for*calls.
assertScopeLocalCallbackDeregistersreads*callsat Line 96 and Line 108 with no synchronization, while the registered callback increments it from the dispatcher thread.go test -racecan report a data race here. The two priority tests already guardorderwithmu; this helper has no equivalent guard.Change the counter to
*atomic.Int64(or protect it with a mutex) in the helper and in every caller.🔒️ Suggested change
func assertScopeLocalCallbackDeregisters( t *testing.T, label string, - calls *int, + calls *atomic.Int64, register func() error, deregister func() error, runBefore func() error, runAfter func() error, ) { @@ if err := FlushSubscribers(); err != nil { t.Fatalf(scopeLocalFlushSubscribersFailed, err) } - if *calls != 1 { - t.Fatalf("expected %s callback once, got %d", label, *calls) + if got := calls.Load(); got != 1 { + t.Fatalf("expected %s callback once, got %d", label, got) } @@ if err := FlushSubscribers(); err != nil { t.Fatalf(scopeLocalFlushSubscribersFailed, err) } - if *calls != 1 { - t.Fatalf("%s callback still fired after deregister: %d", label, *calls) + if got := calls.Load(); got != 1 { + t.Fatalf("%s callback still fired after deregister: %d", label, got) } }Update each callback to use
calls.Add(1).🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@go/nemo_relay/scope_local_test.go` around lines 93 - 110, Update assertScopeLocalCallbackDeregisters and every caller to synchronize the callback counter across dispatcher and test goroutines: use a shared atomic.Int64 (or mutex-protected counter), increment it with Add(1) inside each callback, and load its value atomically for both assertions while preserving the existing expected-count checks.Source: Path instructions
crates/core/tests/integration/middleware_tests.rs (1)
3441-3455: 📐 Maintainability & Code Quality | 🔵 Trivial | ⚡ Quick winRestore lock-freedom assertions in all sanitizer callbacks.
The sanitizer registries are snapshotted before queued sanitizer chains run. Add
assert_middleware_callback_locks_are_free()to the tool and LLM request and response sanitizer callbacks.🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@crates/core/tests/integration/middleware_tests.rs` around lines 3441 - 3455, Restore lock-freedom checks in every tool and LLM request/response sanitizer callback. After each sanitizer registry is snapshotted and before the callback queues or runs its sanitizer chain, call assert_middleware_callback_locks_are_free(). Apply this to the callbacks registered by the tool and LLM sanitizer test helpers, preserving their existing callback behavior.Source: Path instructions
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In `@crates/core/src/stream.rs`:
- Around line 244-245: Remove the dead finalization-task plumbing around
emit_end_event: delete the finalization field, its polling branch in poll_next,
and the await in close(). Remove the _background_thread parameter and
background_thread argument from finish and emit_end_event, change emit_end_event
to return (), and update its documentation to state that it queues
sanitizer/END-event work rather than running it directly.
In `@crates/core/tests/integration/middleware_tests.rs`:
- Around line 4508-4537: Update the managed queued tool test around
managed_queued_tool_observer to capture subscriber events instead of using an
empty callback, then after flush_subscribers assert the START event contains the
sanitized arguments and that the lifecycle events occur in start-then-end order,
matching the existing LLM test pattern.
In `@docs/about-nemo-relay/concepts/middleware.mdx`:
- Around line 266-275: Update the Detailed Execution Flow diagram to match the
queued middleware ordering described above: route the start and response
sanitizer groups through the Dispatcher node, rather than directly between
RequestIntercepts/StartEvent or InterceptResult/Finalizer. Ensure the diagram
shows start-event enqueueing before execution intercepts and dispatcher-based
sanitization afterward, consistent with queue_llm_start_with_subscribers.
---
Outside diff comments:
In `@crates/core/src/api/llm.rs`:
- Around line 418-481: Remove the test-only emit_llm_start_with_subscribers
pipeline and its duplicated sanitization, annotation, freshness, and
event-building logic. Update emit_llm_start and the llm_api_tests coverage to
exercise queue_llm_start_with_subscribers directly, using flush_subscribers() to
observe queued events; alternatively extract and reuse a shared transformation
helper so both paths cannot diverge.
In `@crates/core/tests/integration/middleware_tests.rs`:
- Around line 3441-3455: Restore lock-freedom checks in every tool and LLM
request/response sanitizer callback. After each sanitizer registry is
snapshotted and before the callback queues or runs its sanitizer chain, call
assert_middleware_callback_locks_are_free(). Apply this to the callbacks
registered by the tool and LLM sanitizer test helpers, preserving their existing
callback behavior.
In `@go/nemo_relay/scope_local_test.go`:
- Around line 93-110: Update assertScopeLocalCallbackDeregisters and every
caller to synchronize the callback counter across dispatcher and test
goroutines: use a shared atomic.Int64 (or mutex-protected counter), increment it
with Add(1) inside each callback, and load its value atomically for both
assertions while preserving the existing expected-count checks.
🪄 Autofix
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: Path: .coderabbit.yaml
Review profile: ASSERTIVE
Plan: Enterprise
Run ID: c9e2ea48-bd61-4d65-bf02-fc45854d0be7
📒 Files selected for processing (15)
.agents/skills/add-middleware/SKILL.mdAGENTS.mdcrates/core/src/api/llm.rscrates/core/src/api/tool.rscrates/core/src/stream.rscrates/core/tests/integration/middleware_tests.rscrates/core/tests/unit/llm_api_tests.rscrates/node/tests/llm_tests.mjsdocs/about-nemo-relay/architecture.mdxdocs/about-nemo-relay/concepts/middleware.mdxgo/nemo_relay/llm_test.gogo/nemo_relay/scope_local_test.gogo/nemo_relay/tools_test.gopython/tests/test_llm.pypython/tests/test_scope_local.py
📜 Review details
🧰 Additional context used
📓 Path-based instructions (47)
**/*.mdx
📄 CodeRabbit inference engine (.agents/skills/review-doc-style/SKILL.md)
MDX top-of-file SPDX comments must use {/* ... */} delimiters instead of HTML comment delimiters (Must-Fix)
**/*.mdx: In MDX files, top-of-file comments must use JSX comment delimiters ({/*and*/}); do not use HTML comments for MDX SPDX headers.
New or regenerated MDX files must use{/* ... */}for top-of-file SPDX comments.
Files:
docs/about-nemo-relay/architecture.mdxdocs/about-nemo-relay/concepts/middleware.mdx
{docs,examples}/**/*
📄 CodeRabbit inference engine (.agents/skills/rename-surfaces/SKILL.md)
Update docs and examples.
Files:
docs/about-nemo-relay/architecture.mdxdocs/about-nemo-relay/concepts/middleware.mdx
**/*
📄 CodeRabbit inference engine (.agents/skills/validate-change/SKILL.md)
**/*: Format changed files with the language-native formatter before the final lint/test pass.
If dynamic plugin behavior changed, usemaintain-dynamic-pluginsand include the native SDK, worker protocol, Python SDK, docs, packaging, and Codecov surfaces in the validation plan.
If code changes alter APIs, bindings, commands, paths, packaging behavior, observability/adaptive semantics, or documented best practices, update any dependent maintainer or consumer skills in the same branch.
During iteration, preferuv run pre-commit run --files <changed files...>.
Before review or handoff, runuv run pre-commit run --all-files.
**/*: Use release tags in raw Rust-compatible SemVer without a leadingv; tags such asv0.1.0are prohibited.
Use branch prefixesfeat/,fix/,docs/,test/, orrefactor/according to the change purpose.
Every commit in a pull request must include a DCOSigned-off-by:sign-off.
Before submitting a pull request, ensure pre-commit hooks, relevant tests, target-specific builds, documentation updates, and a rebase on the latestmainare complete.
Use commit messages in the formtype: short description, with a valid type and a first line under 72 characters.
**/*: Before editing, lock the middleware design: target entities, middleware kind, pipeline stage, callback failure behavior, registration scope, event payload observations, and event-sanitizer mutability.
Expose the new middleware surface in every affected language binding and provide parity coverage across those bindings.
Keep changes scoped, surface assumptions, and define focused validation before editing; validate the completed change.
**/*: Prefer repositoryjustrecipes over raw tool commands; use rawcargo,pytest,go test, ornpmonly for focused debugging or targeted reruns without a corresponding recipe.
Run tests for every language affected by a change. Cross-language runtime-contract changes require validation of every affected binding.
Pref...
Files:
docs/about-nemo-relay/architecture.mdxpython/tests/test_llm.pycrates/core/tests/unit/llm_api_tests.rsAGENTS.mdgo/nemo_relay/tools_test.gocrates/node/tests/llm_tests.mjspython/tests/test_scope_local.pydocs/about-nemo-relay/concepts/middleware.mdxcrates/core/src/api/tool.rscrates/core/src/stream.rsgo/nemo_relay/llm_test.gocrates/core/src/api/llm.rscrates/core/tests/integration/middleware_tests.rsgo/nemo_relay/scope_local_test.go
docs/**/*
📄 CodeRabbit inference engine (.agents/skills/validate-change/SKILL.md)
If documentation examples or commands under
docs/change, run the targeted docs checks appropriate to the change.
Files:
docs/about-nemo-relay/architecture.mdxdocs/about-nemo-relay/concepts/middleware.mdx
**/*.{md,mdx}
📄 CodeRabbit inference engine (.agents/skills/validate-change/SKILL.md)
If links in documentation change, run
just docs-linkcheck.
**/*.{md,mdx}: Prefer the documented public API over internal shortcuts in documentation and examples.
Keep package names, repository references, and build commands current.
Contribution workflow documentation must require an issue before external contribution pull requests and note that NVIDIA contributors may use a GitHub or Linear issue.
Update entry-point documentation when examples or reading paths change.
Keep release-process and release-notes guidance in maintainer documentation such asRELEASING.md, rather than user-facing documentation pages orCHANGELOG.md.
Use stable user-facing wrappers at thescripts/root in documentation and examples; reference namespaced helper paths only for internal maintenance documentation.
When detailed dynamic plugin guides exist, keep Rust native plugin examples, Python worker plugin examples, andgrpc-v1protocol details on separate pages.
Dynamic plugin manifests in documentation and examples should usecompat.relay = ">=0.5,<1.0"unless deliberately narrower.
Render images, diagrams, tables, and other visual content at representative page widths, ensuring legibility and complete access without clipping; use responsive scaling, reflow, or overflow as appropriate and scope visual styling narrowly.
Dynamic plugin entry pages should link to native, worker, Rust example, Python example, and protocol pages when those pages exist.
Images, diagrams, tables, and custom visual content must remain legible and fully accessible at representative desktop and narrow page widths.
Release-policy documentation must point to GitHub Releases as the only release-history source of truth.
Runjust docswhen the documentation site changes; retain./scripts/build-docs.sh htmlas the compatibility wrapper.
Files:
docs/about-nemo-relay/architecture.mdxAGENTS.mddocs/about-nemo-relay/concepts/middleware.mdx
**/*.{md,mdx,py,go,js,ts}
📄 CodeRabbit inference engine (.agents/skills/maintain-observability/SKILL.md)
Examples and documentation must use each exporter's documented flush/deregister order before shutdown.
Files:
docs/about-nemo-relay/architecture.mdxpython/tests/test_llm.pyAGENTS.mdgo/nemo_relay/tools_test.gopython/tests/test_scope_local.pydocs/about-nemo-relay/concepts/middleware.mdxgo/nemo_relay/llm_test.gogo/nemo_relay/scope_local_test.go
docs/**/*.{md,mdx}
📄 CodeRabbit inference engine (CONTRIBUTING.md)
Update relevant reference documentation when public behavior or APIs change.
Files:
docs/about-nemo-relay/architecture.mdxdocs/about-nemo-relay/concepts/middleware.mdx
**/*.{rs,py,go,js,jsx,ts,tsx,c,h,html,md,mdx,toml}
📄 CodeRabbit inference engine (CONTRIBUTING.md)
Include the appropriate SPDX copyright and Apache-2.0 license header in every source file.
Files:
docs/about-nemo-relay/architecture.mdxpython/tests/test_llm.pycrates/core/tests/unit/llm_api_tests.rsAGENTS.mdgo/nemo_relay/tools_test.gopython/tests/test_scope_local.pydocs/about-nemo-relay/concepts/middleware.mdxcrates/core/src/api/tool.rscrates/core/src/stream.rsgo/nemo_relay/llm_test.gocrates/core/src/api/llm.rscrates/core/tests/integration/middleware_tests.rsgo/nemo_relay/scope_local_test.go
**/*.{md,mdx,rst}
📄 CodeRabbit inference engine (.agents/skills/review-doc-style/assets/nvidia-style-technical-docs.md)
**/*.{md,mdx,rst}: Use title case consistently for technical documentation headings and table headers; avoid quotation marks, ampersands, and exclamation marks in headings, while preserving official product, event, research, and whitepaper title case.
Format code elements, commands, parameters, package names, expressions, directories, file names, and paths in monospace; represent path placeholders with angle brackets inside monospace.
Format UI buttons, menus, fields, and labels in bold, and separate consecutive UI navigation labels with>.
Use quotation marks for error messages and strings when appropriate, italics for newly introduced terms and publication titles, and plain text for keyboard shortcuts.
Represent GitHub repositories with owner/repository link text, such as[NVIDIA/NeMo](link), rather than generic repository wording.
Introduce every code block with a complete sentence; do not let a code block complete or interrupt the grammar of surrounding prose; use syntax highlighting when supported.
Keep inline method, function, and class references consistent with nearby documentation; omit empty parentheses in prose when no call is shown.
Use descriptive link text matching the destination title when possible; avoid raw URLs, generic anchors, long-sentence links, and unnecessary links that distract from procedures.
Ensure lists have a complete lead-in sentence, more than one item, no more than two levels, parallel construction, one idea or action per item, and appropriate punctuation; use bullets for unordered items and numbers for ordered tasks.
Format definition lists with a bold term followed by a complete, parallel, punctuated definition.
Use tables for reference information, decision support, compatibility matrices, and comparable choices; flag one-row tables, missing captions or lead-ins, sentence-case headers where title case is expected, unexplained empty cells, and code or links that would be clearer as prose.
Write procedure steps as imperative ...
Files:
docs/about-nemo-relay/architecture.mdxAGENTS.mddocs/about-nemo-relay/concepts/middleware.mdx
**/*.{rs,md,mdx}
📄 CodeRabbit inference engine (.agents/skills/add-middleware/SKILL.md)
Preserve the documented middleware pipeline order for tool execution, LLM execution, queued publication, and mark/scope events.
Files:
docs/about-nemo-relay/architecture.mdxcrates/core/tests/unit/llm_api_tests.rsAGENTS.mddocs/about-nemo-relay/concepts/middleware.mdxcrates/core/src/api/tool.rscrates/core/src/stream.rscrates/core/src/api/llm.rscrates/core/tests/integration/middleware_tests.rs
{fern,docs}/**/*
📄 CodeRabbit inference engine (AGENTS.md)
Documentation site changes must run
just docs; runjust docs-linkcheckwhen links change.
Files:
docs/about-nemo-relay/architecture.mdxdocs/about-nemo-relay/concepts/middleware.mdx
{docs/**,README.md,CONTRIBUTING.md,RELEASING.md,SECURITY.md}
⚙️ CodeRabbit configuration file
{docs/**,README.md,CONTRIBUTING.md,RELEASING.md,SECURITY.md}: Review documentation for technical accuracy against the current API, command correctness, and consistency across language bindings.
Flag stale examples, missing SPDX headers where required, and instructions that no longer match CI or pre-commit behavior.
Files:
docs/about-nemo-relay/architecture.mdxdocs/about-nemo-relay/concepts/middleware.mdx
{crates/**/src/**/*.rs,python/**/*.py}
📄 CodeRabbit inference engine (.agents/skills/maintain-dynamic-plugins/SKILL.md)
Do not add tests under
src; Rust tests belong in cratetests/trees, and Python SDK tests belong underpython/tests.
Files:
python/tests/test_llm.pypython/tests/test_scope_local.pycrates/core/src/api/tool.rscrates/core/src/stream.rscrates/core/src/api/llm.rs
python/tests/**/*.py
📄 CodeRabbit inference engine (.agents/skills/test-python-binding/SKILL.md)
python/tests/**/*.py: Pytest is used to run tests.
Do not add@pytest.mark.asyncioto any test; async tests are automatically detected and run by the async runner.
Do not add a-> Nonereturn type annotation to test functions.
When mocking a class, do not define a new class; useunittest.mock.MagicMockorunittest.mock.AsyncMock, with thespecconstructor argument when necessary.
Name mocked classes with themockprefix, notfake.
Prefer pytest fixtures over helper methods.
Do not repeat fixtures; if a fixture is needed in multiple test files, place it in aconftest.pyfile.
When creating a fixture, use@pytest.fixture(name="<fixture_name>"[, scope="<scope>"])and define the fixture function asdef <fixture_name>_fixture() -> <return_type>:; only specifyscopewhen it is notfunction.
Preferpytest.mark.parametrizeover creating individual tests for different input types.
Files:
python/tests/test_llm.pypython/tests/test_scope_local.py
**/*.{rs,py,go,js,ts}
📄 CodeRabbit inference engine (.agents/skills/validate-change/SKILL.md)
If a language surface changed, always run that language's test target even when Rust core did not change.
**/*.{rs,py,go,js,ts}: When observability configuration or lifecycle is exposed, keep FFI and Python, Go, and Node.js binding-native config objects and subscriber/exporter methods aligned in logical knobs and semantics.
Require every OpenTelemetry endpoint to have a type and nonblank destination; resolveheader_envvalues at activation and reject missing, blank, or duplicate headers.
Concatenate layered ATOF sink, ATIF storage, and OpenTelemetry endpoint lists with higher-precedence entries first.
Preserve correct handling of mark events, start/end events, orphan cases, and span or trajectory fields derived from intended event data.
Run affected Rust tests andjust test-rustwhen event fields change; runjust test-python,just test-go, andjust test-nodewhen binding-native configuration or lifecycle changes.
Files:
python/tests/test_llm.pycrates/core/tests/unit/llm_api_tests.rsgo/nemo_relay/tools_test.gopython/tests/test_scope_local.pycrates/core/src/api/tool.rscrates/core/src/stream.rsgo/nemo_relay/llm_test.gocrates/core/src/api/llm.rscrates/core/tests/integration/middleware_tests.rsgo/nemo_relay/scope_local_test.go
**/*.py
📄 CodeRabbit inference engine (CONTRIBUTING.md)
**/*.py: Lint Python with Ruff using rule setsE,F,W, andI.
Format Python with the Ruff formatter, using a 120-character line length and double quotes.
Runtyfor Python type checking.
Use Pythonsnake_casenaming conventions.Use Python
snake_casenaming; Python wrappers live underpython/nemo_relay/.
Files:
python/tests/test_llm.pypython/tests/test_scope_local.py
**/*.{rs,py,go,js,jsx,ts,tsx,c,h}
📄 CodeRabbit inference engine (CONTRIBUTING.md)
**/*.{rs,py,go,js,jsx,ts,tsx,c,h}: Run tests for every language affected by a change; changes to the core Rust crate require tests across all bindings.
UseSONAR_IGNORE_START/SONAR_IGNORE_ENDonly for documented false positives, keep ignored blocks minimal, explain them with a comment, and obtain reviewer sign-off.
Preserve the layered architecture in which Rust provides the core runtime and C FFI, PyO3, and NAPI provide bindings that mirror the full API surface.
Files:
python/tests/test_llm.pycrates/core/tests/unit/llm_api_tests.rsgo/nemo_relay/tools_test.gopython/tests/test_scope_local.pycrates/core/src/api/tool.rscrates/core/src/stream.rsgo/nemo_relay/llm_test.gocrates/core/src/api/llm.rscrates/core/tests/integration/middleware_tests.rsgo/nemo_relay/scope_local_test.go
**/{test,tests}/**/*
📄 CodeRabbit inference engine (CONTRIBUTING.md)
When adding functionality, include tests in the appropriate test files for each affected language binding.
Files:
python/tests/test_llm.pycrates/core/tests/unit/llm_api_tests.rscrates/node/tests/llm_tests.mjspython/tests/test_scope_local.pycrates/core/tests/integration/middleware_tests.rs
**/*.{rs,py,js,ts,java,go}
📄 CodeRabbit inference engine (.agents/skills/add-middleware/SKILL.md)
**/*.{rs,py,js,ts,java,go}: Test registration, duplicate-name handling, deregistration, missing-name no-ops, priority ordering, callback failure behavior, scope-local inheritance and cleanup, and event payload semantics.
When applicable, test mark and scope-event field semantics, including preservation of immutable identity fields.
Files:
python/tests/test_llm.pycrates/core/tests/unit/llm_api_tests.rsgo/nemo_relay/tools_test.gopython/tests/test_scope_local.pycrates/core/src/api/tool.rscrates/core/src/stream.rsgo/nemo_relay/llm_test.gocrates/core/src/api/llm.rscrates/core/tests/integration/middleware_tests.rsgo/nemo_relay/scope_local_test.go
python/**/*.py
📄 CodeRabbit inference engine (AGENTS.md)
Python binding or wrapper changes must run
just test-python.
Files:
python/tests/test_llm.pypython/tests/test_scope_local.py
**/*.{rs,py,js,mjs,ts,tsx,go,c,h,cpp,md,json,toml,yaml,yml,sh}
📄 CodeRabbit inference engine (AGENTS.md)
Keep SPDX headers on source, documentation, scripts, and configuration files.
Files:
python/tests/test_llm.pycrates/core/tests/unit/llm_api_tests.rsAGENTS.mdgo/nemo_relay/tools_test.gocrates/node/tests/llm_tests.mjspython/tests/test_scope_local.pycrates/core/src/api/tool.rscrates/core/src/stream.rsgo/nemo_relay/llm_test.gocrates/core/src/api/llm.rscrates/core/tests/integration/middleware_tests.rsgo/nemo_relay/scope_local_test.go
{crates/**/tests/**,python/tests/**,go/nemo_relay/**/*_test.go}
⚙️ CodeRabbit configuration file
{crates/**/tests/**,python/tests/**,go/nemo_relay/**/*_test.go}: Tests should cover the behavior promised by the changed API surface, including error paths and cross-request isolation where relevant.
Prefer assertions on lifecycle events, scope stacks, middleware ordering, and binding parity over shallow smoke tests.
Files:
python/tests/test_llm.pycrates/core/tests/unit/llm_api_tests.rsgo/nemo_relay/tools_test.gocrates/node/tests/llm_tests.mjspython/tests/test_scope_local.pygo/nemo_relay/llm_test.gocrates/core/tests/integration/middleware_tests.rsgo/nemo_relay/scope_local_test.go
**/*.rs
📄 CodeRabbit inference engine (.agents/skills/prepare-pr/SKILL.md)
**/*.rs: Any Rust change must runjust test-rust
Any Rust change must runcargo fmt --all
Any Rust change must runcargo clippy --workspace --all-targets -- -D warnings
**/*.rs: Runcargo fmt --allfor all FFI work since it is Rust work
Runjust test-rustto validate FFI changes
Runcargo clippy --workspace --all-targets -- -D warningsto enforce strict linting on FFI workWhen Rust files changed as part of Go work, also run
cargo fmt --all,just test-rust, andcargo clippy --workspace --all-targets -- -D warnings
**/*.rs: Runcargo fmt --allwhen Rust files are changed as part of Node work
Runcargo clippy --workspace --all-targets -- -D warningswhen Rust files are changed as part of Node work
Runjust test-rustwhen Rust files are changed as part of Node work
**/*.rs: If any Rust code changed, always runjust test-rust.
If any Rust code changed, also runcargo fmt --all.
If any Rust code changed, also runcargo clippy --workspace --all-targets -- -D warnings.
For Rust changes headed for review, runcargo fmt --allandcargo clippy --workspace --all-targets -- -D warningseven if relying on pre-commit.
**/*.rs: Format Rust code with rustfmt defaults usingcargo fmt.
Runcargo clippy -- -D warnings; all Rust warnings must be treated as errors.
Use Rustsnake_casenaming conventions.
**/*.rs: Use Rustsnake_casenaming,Json = serde_json::Valuewhere existing Rust-facing runtime APIs expect JSON payloads, andResult<T>withFlowErrorin core runtime paths.
Keep async behavior on the existing Tokio-based model and preserve callback and future lifetimes rather than blocking or unexpectedly hiding async work.
Files:
crates/core/tests/unit/llm_api_tests.rscrates/core/src/api/tool.rscrates/core/src/stream.rscrates/core/src/api/llm.rscrates/core/tests/integration/middleware_tests.rs
{crates/core,crates/adaptive}/**/*
📄 CodeRabbit inference engine (.agents/skills/prepare-pr/SKILL.md)
Changes to
crates/coreorcrates/adaptivemust run the full language matrix
Files:
crates/core/tests/unit/llm_api_tests.rscrates/core/src/api/tool.rscrates/core/src/stream.rscrates/core/src/api/llm.rscrates/core/tests/integration/middleware_tests.rs
crates/core/**/*.rs
📄 CodeRabbit inference engine (.agents/skills/test-go-binding/SKILL.md)
If the change touched
crates/coreor shared runtime semantics, also usevalidate-changefor broader validation
Files:
crates/core/tests/unit/llm_api_tests.rscrates/core/src/api/tool.rscrates/core/src/stream.rscrates/core/src/api/llm.rscrates/core/tests/integration/middleware_tests.rs
crates/{core,adaptive}/**/*
📄 CodeRabbit inference engine (.agents/skills/validate-change/SKILL.md)
If
crates/coreorcrates/adaptivechanged, run the full validation matrix across Rust, Python, Go, and Node.js.
Files:
crates/core/tests/unit/llm_api_tests.rscrates/core/src/api/tool.rscrates/core/src/stream.rscrates/core/src/api/llm.rscrates/core/tests/integration/middleware_tests.rs
crates/**/*.rs
📄 CodeRabbit inference engine (AGENTS.md)
crates/**/*.rs: Rust core or adaptive changes must runjust test-rust; add binding tests when public behavior changes.
Scope stacks must remain hierarchical with a root scope, preserving parent-child events, scope-local visibility, cleanup boundaries, and concurrent request isolation.
Scope-local middleware and subscribers are owned by their scope and disappear when it closes; global registrations remain process-wide until removed.
Middleware must be priority-ordered after merging global and visible scope-local entries.
Request intercepts rewrite requests; execution intercepts wrap or replace callbacks; stream execution intercepts handle streaming lifecycle behavior.
Guardrails may block execution or sanitize observability payloads; sanitizing guardrails must not rewrite real callback arguments or return values.
Managed execution must run conditional guardrails and request intercepts, queue sanitize-request start events, execute intercepts and callbacks, then queue sanitize-response end events; payload sanitizers must not delay the application callback or result.
Use ATOF0.1as the canonical event format; scope events use start/end pairs and mark events record runtime checkpoints.
Keep LLM and tool metadata in the category profile, including fields such asmodel_name,tool_call_id, and customsubtypevalues.
Exporters may transform events to ATIF, OpenTelemetry, or OpenInference-compatible output, and root scope identity must isolate concurrent agents.
Files:
crates/core/tests/unit/llm_api_tests.rscrates/core/src/api/tool.rscrates/core/src/stream.rscrates/core/src/api/llm.rscrates/core/tests/integration/middleware_tests.rs
crates/{core,adaptive}/**/*.rs
⚙️ CodeRabbit configuration file
crates/{core,adaptive}/**/*.rs: Review the Rust runtime for async correctness, scope isolation, middleware ordering, and event lifecycle regressions.
Pay close attention to task-local/thread-local scope propagation, callback lifetimes, stream finalization, and root_uuid isolation.
Public API changes should preserve existing behavior unless tests and docs show the intended migration path.
Files:
crates/core/tests/unit/llm_api_tests.rscrates/core/src/api/tool.rscrates/core/src/stream.rscrates/core/src/api/llm.rscrates/core/tests/integration/middleware_tests.rs
AGENTS.md
📄 CodeRabbit inference engine (CLAUDE.md)
Document agent implementations in AGENTS.md with clear descriptions of functionality and usage
Files:
AGENTS.md
**/*.{md,rst,html,txt}
📄 CodeRabbit inference engine (.agents/skills/review-doc-style/assets/nvidia-style-brand-terminology.md)
**/*.{md,rst,html,txt}: Always spellNVIDIAin all caps. Do not useNvidia,nvidia,nVidia,nVIDIA, orNV.
Usean NVIDIAbefore a noun because the name starts with an 'en' sound.
Do not add a registered trademark symbol afterNVIDIAwhen referring to the company.
Use trademark symbols with product names only when the document type or legal guidance requires them.
Verify official capitalization, spacing, and hyphenation for product names.
Precede NVIDIA product names withNVIDIAon first mention when it is natural and accurate.
Do not rewrite product names for grammar or title-case rules.
Preserve third-party product names according to the owner's spelling.
Include the company name and full model qualifier on first use when it helps identify the model.
Preserve the official capitalization and punctuation of model names.
Use shorter family names only after the full name is established.
Spell out a term on first use and put the acronym in parentheses unless the acronym is widely understood by the intended audience.
Use the acronym on later mentions after it has been defined.
For long documents, reintroduce the full term if readers might lose context.
Form plurals of acronyms withs, not an apostrophe, such asGPUs.
In headings, common acronyms can remain abbreviated. Spell out the term in the first or second sentence of the body.
Common terms such asCPU,GPU,PC,API, andUIusually do not need to be spelled out for developer audiences.
Files:
AGENTS.md
**/*.{md,rst,html}
📄 CodeRabbit inference engine (.agents/skills/review-doc-style/assets/nvidia-style-brand-terminology.md)
Link the first mention of a product name when the destination helps the reader.
Files:
AGENTS.md
**/*.{md,rst,txt}
📄 CodeRabbit inference engine (.agents/skills/review-doc-style/assets/nvidia-style-guide.md)
Spell
NVIDIAin all caps. Do not useNvidia,nvidia, orNV.
Files:
AGENTS.md
**/*.{md,rst}
📄 CodeRabbit inference engine (.agents/skills/review-doc-style/assets/nvidia-style-guide.md)
**/*.{md,rst}: Format commands, code elements, expressions, package names, file names, and paths as inline code.
Use descriptive link text. Avoid raw URLs and weak anchors such as "here" or "read more."
Use title case consistently for technical documentation headings.
Introduce code blocks, lists, tables, and images with complete sentences.
Write procedures as imperative steps. Keep steps parallel and split long procedures into smaller tasks.
Prefer active voice, present tense, short sentences, contractions, and plain English.
Usecanfor possibility and reservemayfor permission.
Useafterfor temporal relationships instead ofonce.
Preferrefer tooverseewhen the wording points readers to another resource.
Avoid culture-specific idioms, unnecessary Latinisms, jokes, and marketing exaggeration in technical docs.
Spell out months in body text, avoid ordinal dates, and use clear time zones.
Spell out whole numbers from zero through nine unless they are technical values, parameters, versions, or UI values.
Use numerals for 10 or greater and include commas in thousands.
Do not add trademark symbols to learning-oriented docs unless the source, platform, or legal guidance explicitly requires them.
Files:
AGENTS.md
go/nemo_relay/**/*.go
📄 CodeRabbit inference engine (.agents/skills/test-go-binding/SKILL.md)
go/nemo_relay/**/*.go: Format changed Go packages withcd go/nemo_relay && go fmt ./...
Run Go tests withjust test-goto build and test the NeMo Relay Go binding
Usejust build-gowhen you want an explicit build-only pass or need the artifact for other work
Usejust ci=true test-gowhen you need the CI-style coverage and JUnit path
On macOS, setDYLD_LIBRARY_PATHto the../../target/releasedirectory before running the rawgo testcommand directly
Files:
go/nemo_relay/tools_test.gogo/nemo_relay/llm_test.gogo/nemo_relay/scope_local_test.go
{crates/python/src/py_api/mod.rs,python/nemo_relay/**/*.py,python/nemo_relay/**/*.pyi,go/nemo_relay/**/*.go,crates/node/src/api/**/*.rs}
📄 CodeRabbit inference engine (.agents/skills/add-binding-feature/SKILL.md)
Update the language-native bindings for every exposed surface in Python, Go, and Node.js.
Files:
go/nemo_relay/tools_test.gogo/nemo_relay/llm_test.gogo/nemo_relay/scope_local_test.go
{python/nemo_relay/**/*.py,python/nemo_relay/**/*.pyi,go/nemo_relay/**/*.go}
📄 CodeRabbit inference engine (.agents/skills/add-binding-feature/SKILL.md)
Update language wrapper helpers such as Python wrapper modules, Python type stubs, and Go shorthand packages when the new behavior belongs in those helper layers.
Files:
go/nemo_relay/tools_test.gogo/nemo_relay/llm_test.gogo/nemo_relay/scope_local_test.go
go/nemo_relay/**
📄 CodeRabbit inference engine (.agents/skills/maintain-optimizer/SKILL.md)
Keep shared plugin helpers in
go/nemo_relayaligned with plugin registration, composition, and lifecycle behavior.
Files:
go/nemo_relay/tools_test.gogo/nemo_relay/llm_test.gogo/nemo_relay/scope_local_test.go
**/*.go
📄 CodeRabbit inference engine (CONTRIBUTING.md)
**/*.go: Format Go code withgofmt.
Rungo vet ./...for Go static analysis.
Use GoPascalCasenaming conventions.Use
PascalCasefor public Go APIs.
Files:
go/nemo_relay/tools_test.gogo/nemo_relay/llm_test.gogo/nemo_relay/scope_local_test.go
go/**/*.go
📄 CodeRabbit inference engine (AGENTS.md)
Go binding changes must run
just test-go; raw FFI changes also require relevant Rust/FFI checks.
Files:
go/nemo_relay/tools_test.gogo/nemo_relay/llm_test.gogo/nemo_relay/scope_local_test.go
go/nemo_relay/**/*
⚙️ CodeRabbit configuration file
go/nemo_relay/**/*: Review Go binding changes for cgo memory ownership, race safety, callback cleanup, idiomatic exported APIs, and parity with Rust/FFI behavior.
Any API change should include focused Go tests and consider race-test behavior.
Files:
go/nemo_relay/tools_test.gogo/nemo_relay/llm_test.gogo/nemo_relay/scope_local_test.go
crates/node/**/*.{js,mjs,ts,tsx}
📄 CodeRabbit inference engine (AGENTS.md)
Node.js binding or wrapper changes must run
just test-node.
Files:
crates/node/tests/llm_tests.mjs
**/*.{js,mjs,ts,tsx}
📄 CodeRabbit inference engine (AGENTS.md)
Use
camelCasefor Node.js APIs.
Files:
crates/node/tests/llm_tests.mjs
crates/{python,ffi,node}/**/*
⚙️ CodeRabbit configuration file
crates/{python,ffi,node}/**/*: Treat binding changes as public API changes. Check for parity with the other language bindings, FFI ownership/lifetime safety,
callback error propagation, stable type conversion, and consistent async/stream semantics.
Flag changes that update one binding without corresponding tests or documentation for the same surface elsewhere.
Files:
crates/node/tests/llm_tests.mjs
docs/about-nemo-relay/concepts/middleware.mdx
📄 CodeRabbit inference engine (.agents/skills/add-middleware/SKILL.md)
Use the middleware concept documentation as the authoritative reference for full pipeline diagrams and ordering.
Files:
docs/about-nemo-relay/concepts/middleware.mdx
crates/core/src/{api/**/*.rs,api/runtime/**/*.rs,codec/**/*.rs,json.rs}
📄 CodeRabbit inference engine (.agents/skills/add-binding-feature/SKILL.md)
Implement the new or changed public runtime behavior first in the Rust core, especially under
crates/core/src/api/and related core modules such ascrates/core/src/api/runtime/,crates/core/src/codec/, andcrates/core/src/json.rs.
Files:
crates/core/src/api/tool.rscrates/core/src/api/llm.rs
crates/core/src/api/{registry.rs,**/*.rs}
📄 CodeRabbit inference engine (.agents/skills/add-middleware/SKILL.md)
Add global and scope-local registration and deregistration APIs, following the
global_*_registry_api!andscope_*_registry_api!patterns, unless the design explicitly excludes one scope.
Files:
crates/core/src/api/tool.rscrates/core/src/api/llm.rs
crates/core/src/api/{tool.rs,llm.rs,shared.rs,scope.rs}
📄 CodeRabbit inference engine (.agents/skills/add-middleware/SKILL.md)
Wire the new middleware chain into the lifecycle owner at the correct pipeline stage: tool/LLM execution paths use
tool.rsorllm.rs; shared mark and scope-event sanitization usesshared.rsand is called fromscope.rs.
Files:
crates/core/src/api/tool.rscrates/core/src/api/llm.rs
🧠 Learnings (3)
📚 Learning: 2026-08-03T19:55:03.931Z
Learnt from: afourniernv
Repo: NVIDIA/NeMo-Relay PR: 558
File: crates/pii-redaction/src/rampart/mod.rs:265-274
Timestamp: 2026-08-03T19:55:03.931Z
Learning: In NeMo Relay first-party plugin registration helpers, treat the documented duplicate-registration `PluginError::RegistrationFailed` result from `register_plugin` as success when registration is intended to be idempotent. Do not locally reclassify this as `PluginError::Conflict`; changing the classification requires a core-wide review of the public API and FFI behavior.
Applied to files:
crates/core/tests/unit/llm_api_tests.rscrates/core/src/api/tool.rscrates/core/src/stream.rscrates/core/src/api/llm.rscrates/core/tests/integration/middleware_tests.rs
📚 Learning: 2026-07-28T20:33:25.156Z
Learnt from: willkill07
Repo: NVIDIA/NeMo-Relay PR: 572
File: go/nemo_relay/adaptive_runtime_test.go:214-238
Timestamp: 2026-07-28T20:33:25.156Z
Learning: When adding/adjusting Go unit tests for `BuildCacheRequestFacts` (request-ID validation and related request parsing), set `CacheRequestFactsInput.Provider` to a valid provider in all tests that are intended to isolate request-ID behavior—because `BuildCacheRequestFacts` does not validate `Provider`. Then add separate test coverage for malformed `AnnotatedRequest` JSON so JSON parsing failures are not conflated with `Provider`-related inputs.
Applied to files:
go/nemo_relay/tools_test.gogo/nemo_relay/llm_test.gogo/nemo_relay/scope_local_test.go
📚 Learning: 2026-07-28T20:07:29.880Z
Learnt from: willkill07
Repo: NVIDIA/NeMo-Relay PR: 571
File: crates/core/src/api/runtime/state.rs:996-1020
Timestamp: 2026-07-28T20:07:29.880Z
Learning: In NeMo Relay (RELAY-509), sanitizer callback failures must be treated as intentional fail-open behavior. When an event/tool (request/response) or LLM (request/response) sanitizer callback fails, the sanitizer chain should retain and publish the last valid event/payload snapshot (rather than dropping/invalidating the data) and log the failure including callback context (e.g., which sanitizer/callback failed and relevant identifiers). Apply this consistently across all sanitizer chains mentioned in the RELAY-509 documentation/migration guide.
Applied to files:
crates/core/src/api/tool.rscrates/core/src/stream.rscrates/core/src/api/llm.rs
🔇 Additional comments (20)
crates/core/src/api/llm.rs (2)
483-506: LGTM!Also applies to: 1144-1162, 1273-1284, 1455-1459, 1486-1490, 1585-1591, 1692-1695, 1792-1798
508-550: 🗄️ Data Integrity & IntegrationNo change needed: preserve the handle timestamp.
build_llm_start_eventsetsBaseEvent.timestampfromhandle.started_at, so rebuilding the START event after sanitization does not move its timestamp.> Likely an incorrect or invalid review comment..agents/skills/add-middleware/SKILL.md (1)
43-50: LGTM!AGENTS.md (1)
180-180: LGTM!docs/about-nemo-relay/architecture.mdx (1)
125-125: LGTM!docs/about-nemo-relay/concepts/middleware.mdx (1)
28-31: LGTM!Also applies to: 43-49, 160-164, 206-211, 245-265
go/nemo_relay/tools_test.go (1)
641-643: LGTM!python/tests/test_llm.py (1)
305-305: LGTM!crates/core/src/api/tool.rs (2)
359-359: LGTM!Also applies to: 375-375, 397-417
552-552: LGTM!Also applies to: 577-605, 685-688
crates/core/src/stream.rs (3)
16-27: LGTM!Also applies to: 64-66
326-336: LGTM!
217-219: 🗄️ Data Integrity & IntegrationKeep the eager
close_for_finalization(None)calls. The later call still records"stream_interrupted"when optimization evidence exists.> Likely an incorrect or invalid review comment.crates/core/tests/integration/middleware_tests.rs (2)
19-19: LGTM!Also applies to: 3519-3519, 3757-3757, 3900-3912
4372-4480: LGTM!Also applies to: 4540-4631
crates/core/tests/unit/llm_api_tests.rs (1)
1182-1183: LGTM!go/nemo_relay/llm_test.go (1)
448-450: LGTM!go/nemo_relay/scope_local_test.go (1)
22-22: LGTM!Also applies to: 522-524, 576-578
python/tests/test_scope_local.py (1)
229-229: LGTM!Also applies to: 257-257
crates/node/tests/llm_tests.mjs (1)
504-504: 📐 Maintainability & Code QualityKeep
flushSubscribers()at both call sites. It waits for queued JavaScript subscriber callbacks. The sanitizer state is already set beforellmCallExecute()resolves orstream.next()returnsnull;flushSubscriberCallbacks()is not required.> Likely an incorrect or invalid review comment.
| _background_thread: bool, | ||
| ) -> Option<tokio::task::JoinHandle<()>> { |
There was a problem hiding this comment.
📐 Maintainability & Code Quality | 🟠 Major | ⚡ Quick win
Remove the now-dead finalization-task plumbing.
emit_end_event always returns None at Line 362. So self.finalization is never Some, and the following code is unreachable:
poll_nextLine 432-449, the finalization polling branch.close()Line 497-501, the finalization await.- The
finalizationfield itself.
The _background_thread parameter and the background_thread argument that finish forwards to it are also dead. The emit_end_event doc comment at Line 236-239 still states that the method runs the sanitizers and emits the END event; it now only queues that work.
Delete the unreachable branches, the finalization field, and the unused parameter, then change the return type to (). Update the doc comment to describe the queued behavior.
♻️ Suggested signature change
- fn emit_end_event(
- &mut self,
- metadata: Option<Json>,
- interrupted: bool,
- _background_thread: bool,
- ) -> Option<tokio::task::JoinHandle<()>> {
+ /// Queue the LLM END event with aggregated response data.
+ ///
+ /// Calls the finalizer to produce the aggregated response, then queues the
+ /// sanitize-response guardrails and END-event publication on the shared
+ /// publication executor. Stream termination does not await that work.
+ fn emit_end_event(&mut self, metadata: Option<Json>, interrupted: bool) { let _ = subscriber_dispatcher::spawn_background_publication(finalize);
- None
}Also applies to: 357-362, 430-431
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
In `@crates/core/src/stream.rs` around lines 244 - 245, Remove the dead
finalization-task plumbing around emit_end_event: delete the finalization field,
its polling branch in poll_next, and the await in close(). Remove the
_background_thread parameter and background_thread argument from finish and
emit_end_event, change emit_end_event to return (), and update its documentation
to state that it queues sanitizer/END-event work rather than running it
directly.
| register_subscriber("managed_queued_tool_observer", Arc::new(|_| {})).unwrap(); | ||
|
|
||
| let call = tokio::spawn(async { | ||
| tool_call_execute( | ||
| nemo_relay::api::tool::ToolCallExecuteParams::builder() | ||
| .name("managed-queued-tool") | ||
| .args(json!({"input": true})) | ||
| .func(Arc::new(|args| Box::pin(async move { Ok(args) }))) | ||
| .build(), | ||
| ) | ||
| .await | ||
| }); | ||
| tokio::time::timeout( | ||
| std::time::Duration::from_secs(2), | ||
| sanitizer_started.notified(), | ||
| ) | ||
| .await | ||
| .expect("managed tool request sanitizer did not start"); | ||
| let result = tokio::time::timeout(std::time::Duration::from_secs(1), call) | ||
| .await | ||
| .expect("request sanitizer blocked managed tool execution") | ||
| .expect("managed tool task should join") | ||
| .expect("managed tool call should succeed"); | ||
| assert_eq!(result, json!({"input": true})); | ||
|
|
||
| sanitizer_release.notify_one(); | ||
| flush_subscribers().unwrap(); | ||
|
|
||
| deregister_tool_sanitize_request_guardrail("managed_queued_tool_request").unwrap(); | ||
| deregister_subscriber("managed_queued_tool_observer").unwrap(); |
There was a problem hiding this comment.
📐 Maintainability & Code Quality | 🔵 Trivial | ⚡ Quick win
Assert the queued tool payload, not only the absence of blocking.
This test registers managed_queued_tool_observer with an empty callback and never inspects an event. After the flush it asserts nothing. So it proves the sanitizer does not block execution, but it does not prove the queued sanitization result reaches the START event.
Capture events in the subscriber and assert the tool START event carries the sanitized arguments and that the lifecycle is start-then-end. The LLM test at Line 4465-4475 already does this.
♻️ Suggested assertion
- register_subscriber("managed_queued_tool_observer", Arc::new(|_| {})).unwrap();
+ let events = Arc::new(Mutex::new(Vec::<Event>::new()));
+ let captured = Arc::clone(&events);
+ register_subscriber(
+ "managed_queued_tool_observer",
+ Arc::new(move |event| captured.lock().unwrap().push(event.clone())),
+ )
+ .unwrap(); sanitizer_release.notify_one();
flush_subscribers().unwrap();
+
+ let lifecycle = events
+ .lock()
+ .unwrap()
+ .iter()
+ .filter(|event| event.name() == "managed-queued-tool")
+ .map(|event| event.scope_category())
+ .collect::<Vec<_>>();
+ assert_eq!(
+ lifecycle,
+ [Some(ScopeCategory::Start), Some(ScopeCategory::End)]
+ );🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
In `@crates/core/tests/integration/middleware_tests.rs` around lines 4508 - 4537,
Update the managed queued tool test around managed_queued_tool_observer to
capture subscriber events instead of using an empty callback, then after
flush_subscribers assert the START event contains the sanitized arguments and
that the lifecycle events occur in start-then-end order, matching the existing
LLM test pattern.
Source: Coding guidelines
| On the publication path, the dispatcher preserves start/end ordering. It runs | ||
| the tool or LLM payload sanitizer first, then the matching scope-event | ||
| sanitizer, and finally delivers the event to subscribers and exporters. | ||
|
|
||
| For streaming LLM flows, the runtime queues the LLM start-event copy before the | ||
| stream execution intercept chain runs. Stream execution intercepts are the | ||
| execution family for streaming provider callbacks. The runtime then collects | ||
| chunks and finalizes the stream before `sanitize-response` guardrails rewrite | ||
| the emitted end-event payload and scope-end event sanitizers run at items 7 and | ||
| 8. | ||
| execution family for streaming provider callbacks. The runtime collects chunks | ||
| and invokes the finalizer, then queues response and event sanitization without | ||
| delaying observable stream termination. A subscriber flush waits for that | ||
| queued end-event work. |
There was a problem hiding this comment.
📐 Maintainability & Code Quality | 🟡 Minor | ⚡ Quick win
Update the Detailed Execution Flow diagram to match the queued ordering.
The simplified diagram and the numbered list now place the start-event enqueue before the execution intercepts, with the sanitize-request chain running on the dispatcher afterwards. The Detailed Execution Flow flowchart later in this page still encodes the old synchronous chain:
RequestIntercepts -->|Transformed Request| SanitizeRequestGuardrailsSanitizeRequestGuardrails -->|Sanitized Start Payload| ScopeStartSanitizersScopeStartSanitizers -->|Sanitized Event Fields| StartEventStartEvent -->|Before Execution Intercepts| HasExecutionIntercept
That path states that sanitize-request and scope-start sanitizers complete before the execution intercepts run. After this change, queue_llm_start_with_subscribers in crates/core/src/api/llm.rs enqueues the start event and returns, and both sanitizer chains run on the dispatcher. The same mismatch applies to SanitizeResponseGuardrails, which the flowchart reaches directly from InterceptResult and Finalizer.
Route both sanitizer groups through the Dispatcher node so the two diagrams agree.
As per coding guidelines: "Use the middleware concept documentation as the authoritative reference for full pipeline diagrams and ordering."
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
In `@docs/about-nemo-relay/concepts/middleware.mdx` around lines 266 - 275, Update
the Detailed Execution Flow diagram to match the queued middleware ordering
described above: route the start and response sanitizer groups through the
Dispatcher node, rather than directly between RequestIntercepts/StartEvent or
InterceptResult/Finalizer. Ensure the diagram shows start-event enqueueing
before execution intercepts and dispatcher-based sanitization afterward,
consistent with queue_llm_start_with_subscribers.
Source: Coding guidelines
github-actions
Bot
added
size:L
Signed-off-by: Will Killian <wkillian@nvidia.com>
1 participant
Overview
Queue managed payload sanitization and event publication so observability work does not add latency to application execution.
Details
Where should the reviewer start?
Start with
crates/core/src/api/llm.rs,crates/core/src/api/tool.rs, andcrates/core/src/stream.rs; the blocked-sanitizer regression coverage is incrates/core/tests/integration/middleware_tests.rs.Related Issues: (use one of the action keywords Closes / Fixes / Resolves / Relates to)
Summary by CodeRabbit
New Features
Documentation
Tests