Proposal: rogue-openeval-adapter for portable Scenario/EvaluationResult interchange

[adhabnr-ux]

Summary

Rogue already has well-defined, native pydantic models for both test cases and results — Scenario/Scenarios and EvaluationResult/EvaluationResults in sdks/python/rogue_sdk/types.py, plus the red-team equivalents (VulnerabilityResult, RedTeamResults) in rogue/server/red_teaming/models.py. That makes Rogue a good candidate for a converter to EvalPort, an open JSON interchange format for eval datasets (spec: SPEC.md).

EvalPort defines two shapes:

• Suite — portable test cases + grader definitions
• ResultSet — per-test-case results + grader results

The idea would be a small, separately-installable rogue-openeval-adapter package (following the pattern of the ~36 existing adapters in the EvalPort ecosystem, e.g. adapters/<name>-openeval-adapter/) that converts:

• rogue_sdk.types.Scenarios ⇄ EvalPort Suite
• rogue_sdk.types.EvaluationResults ⇄ EvalPort ResultSet

This would let a scenarios.json authored for Rogue be reused by any other EvalPort-compatible eval tool, and let Rogue's evaluation output (EvaluationResults, exported via --output-report-file or the job store) be consumed by dashboards/tools built against the EvalPort ResultSet shape, without either side depending on the other's internals.

Concrete sketch

# rogue_openeval_adapter/convert.py
from rogue_sdk.types import (
    Scenario, Scenarios, ScenarioType,
    EvaluationResult, EvaluationResults, ConversationEvaluation,
)
from openeval import Suite, TestCase, ResultSet, TestCaseResult, GraderResult


def to_openeval(scenarios: Scenarios) -> Suite:
    """Scenario -> EvalPort TestCase."""
    return Suite(
        test_cases=[
            TestCase(
                id=s.scenario,  # Scenario has no separate id field; text is the identity
                input=s.scenario,
                expected_output=s.expected_outcome,
                metadata={
                    "scenario_type": s.scenario_type.value,
                    "multi_turn": s.multi_turn,
                    "max_turns": s.max_turns,
                    "attempts": s.attempts,
                },
            )
            for s in scenarios.scenarios
        ],
    )


def from_openeval(suite: Suite) -> Scenarios:
    """EvalPort TestCase -> Scenario."""
    return Scenarios(
        scenarios=[
            Scenario(
                scenario=tc.input,
                scenario_type=ScenarioType(
                    tc.metadata.get("scenario_type", "policy"),
                ),
                expected_outcome=tc.expected_output,
                multi_turn=tc.metadata.get("multi_turn", True),
            )
            for tc in suite.test_cases
        ],
    )


def results_to_openeval(results: EvaluationResults) -> ResultSet:
    """EvaluationResult -> EvalPort ResultSet, AND-aggregated across conversations
    (mirrors EvaluationResults.add_result's own pass semantics)."""
    return ResultSet(
        results=[
            TestCaseResult(
                test_case_id=r.scenario.scenario,
                passed=r.passed,
                grader_results=[
                    GraderResult(
                        name="rogue-judge-llm",
                        passed=c.passed,
                        reason=c.reason,
                        metadata={"context_id": c.context_id} if c.context_id else {},
                    )
                    for c in r.conversations
                ],
            )
            for r in results.results
        ],
    )

For red-team mode, VulnerabilityResult (id/name/passed/severity/cvss_score/details) maps naturally onto EvalPort's grader-result shape too, though the policy-mode mapping above is the more direct fit and the natural place to start.

Why raise this here

I haven't run this conversion against a live Rogue install and I'm not proposing any specific test results — just flagging that the data model overlap looks clean enough to be worth a dedicated adapter, and checking whether that's something the maintainers would want living in this repo (e.g. under sdks/), as a separate PyPI package, or not at all.

Happy to put together a PR with the adapter package + round-trip tests if that's of interest — let me know which direction (in-repo vs. separate package) you'd prefer before I do.