Skip to content

fix(tier3): handle negative controls in insights - #58

Merged
rng1995 merged 11 commits into
NVIDIA:mainfrom
rookepoole:fix/issue-57-negative-control-insights
Aug 21, 2026
Merged

fix(tier3): handle negative controls in insights#58
rng1995 merged 11 commits into
NVIDIA:mainfrom
rookepoole:fix/issue-57-negative-control-insights

Conversation

@rookepoole

@rookepoole rookepoole commented Aug 20, 2026

Copy link
Copy Markdown
Contributor

Fixes #57

Summary

  • Preserve evaluator-owned skill identity across generated and native Harbor staging, and score negative controls only from exact target-skill evidence; ambiguous evidence remains unknown and fails closed.
  • Apply one tri-state routing contract: explicit should_trigger takes precedence, otherwise bool(expected_skill), while truly unlabeled legacy cases remain distinct.
  • Derive and persist trusted invocation evidence from agent trajectories without allowing custom metrics or authored booleans to impersonate standard routing evidence.
  • Keep sampled trials, cases, agents/attempts, and Harbor evidence links correlated, and reject unsupported negative-control claims after Insights parsing.
  • Redact every persisted reward artifact while preserving finite numeric custom metrics through the documented metric namespaces.
  • Document the user-visible fix in the Unreleased changelog.

Verification

  • Repository suite on the final local head, excluding the known occupied-port case below: 4,853 passed, 17 skipped, 5 deselected.
  • Final reward-storage/collector regression suite: 248 passed, 4 skipped.
  • Independent frozen Review GURU semantic review: 325 passed, with no remaining Critical or Important findings.
  • Credential-free integration: 12 configurations, 60 staged tasks, and 44 real verifier/custom-grader subprocesses; all subprocesses exited successfully and collector evidence matched the expected true/false/unknown states.
  • Ruff lint/format, compilation, import ordering, lock consistency, package build, Twine metadata validation, and git diff --check: passed.

Known verification limits

tests/test_nvidia_build_bridge.py::test_dynamic_bridge_ignores_prebound_old_port_and_checker_authenticates_health was excluded because another local process already owned 127.0.0.1:18080. Docker was unavailable because the configured daemon timed out, so no Docker-backed Harbor run was completed. No paid-provider/model request was made.

Signed-off-by: rookepoole <rookpool97@gmail.com>
@rookepoole
rookepoole marked this pull request as ready for review August 20, 2026 06:17

@chrisknvidia chrisknvidia left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Requesting changes for three negative-control correctness gaps that still reproduce at the current head. The inline comments are limited to behavior required for issue #57: enforcing evidence-grounded output, matching runtime routing semantics, and preserving case/trial correlation.

Comment thread src/skillevaluator/evaluation/insights_judge.py
Comment thread src/skillevaluator/evaluation/insights_judge.py Outdated
Comment thread src/skillevaluator/evaluation/insights_judge.py Outdated

@rng1995 rng1995 left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Reviewed the current head. I independently reproduced the six-case negative-006 case/trial metadata loss already captured in the newly opened correlation thread, so I am not duplicating that inline. I found no additional non-duplicate findings beyond the three current request-changes threads.

@rng1995

rng1995 commented Aug 20, 2026

Copy link
Copy Markdown
Collaborator

@rookepoole - Thanks for your contribution! Please resolve merge conflicts to merge your PR.

@rng1995 rng1995 left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Re-reviewed eed4780. The runtime-routing precedence and sampled trial/case identity fixes now hold, and their focused regressions pass. The post-parse evidence invariant remains bypassable through ordinary paraphrases, so I am requesting changes for the remaining P1 inline.

Comment thread src/skillevaluator/evaluation/insights_judge.py Outdated
@chrisknvidia

Copy link
Copy Markdown
Collaborator

Looks good to me, All the comments are addressed now.

@chrisknvidia

Copy link
Copy Markdown
Collaborator

@rookepoole : Please resolve merge conflicts to merge your PR

Signed-off-by: rookepoole <rookpool97@gmail.com>
rookepoole and others added 4 commits August 20, 2026 19:32
Signed-off-by: Christopher Kevin <christopherk@nvidia.com>
Signed-off-by: Christopher Kevin <christopherk@nvidia.com>
Signed-off-by: Christopher Kevin <christopherk@nvidia.com>

@chrisknvidia chrisknvidia left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Re-reviewed the exact final head 3daa3fd. This approval supersedes my earlier changes-requested review. The complete negative-control grounding and persisted-reward redaction paths passed the final local, integration, and hosted gates; no remaining Critical or Important findings.

@chrisknvidia

Copy link
Copy Markdown
Collaborator

Updated this PR on current main with the complete negative-control grounding fix. Evaluator-owned identity now survives generated/native Harbor staging, exact invocation evidence is preserved as true/false/unknown, custom metrics cannot impersonate standard routing evidence, trial/case links remain correlated, unsupported negative-control findings fail closed, and persisted rewards are credential-redacted. All four prior review threads are resolved.

Final verification on head 3daa3fd:

  • Repository suite: 4,853 passed, 17 skipped, 5 deselected (the known local port-18080 case was explicitly excluded).
  • Reward-storage/collector regression suite: 248 passed, 4 skipped.
  • Credential-free integration: 12 configurations, 60 staged tasks, and 44 real verifier/custom-grader subprocesses.
  • All 15 hosted checks passed, including Linux/Python 3.12 and 3.13, macOS, Windows, DCO, Gitleaks, dependency review, packaging, and both CodeQL results. The prior high CodeQL alert is now marked fixed.
  • Ruff, formatting, import ordering, lock, build, Twine metadata, and diff checks passed.

Docker was unavailable and no paid-provider/model request was made. I have submitted a fresh approval on this exact head. @rng1995, please re-review the final head when convenient so your earlier change request can be superseded.

This PR is ready for final maintainer review and has not been merged or configured for auto-merge.

@rng1995
rng1995 merged commit e70f0e3 into NVIDIA:main Aug 21, 2026
15 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[BUG]: Tier 3 LLM insights misclassify a correctly handled negative-control case as unintended skill application

3 participants