[BUG] OAEvalPlugin.run_evaluation returns a silent error result in any process where asyncio.run has already completed

[Nitjsefnie]

Describe the Bug

Once any asyncio.run(...) has completed earlier in the same process, every subsequent call to OAEvalPlugin.run_evaluation(...) fails and returns passed=False with summary={'error': "There is no current event loop in thread 'MainThread'.", ...}. No exception propagates and the process exit code is 0, so the caller cannot distinguish this from a genuine gate failure.

asyncio.run calls asyncio.set_event_loop(None) when it finishes, after closing its loop. From that point asyncio.get_event_loop() in the main thread no longer auto-creates a loop and raises RuntimeError instead. run_evaluation calls it at openagent_eval/cicd/plugin.py:179, inside the try: that begins at line 169, so the if loop.is_running(): test at line 180 is never reached — the failure happens before either branch is chosen. The except Exception as e: at line 214 then converts the RuntimeError into the returned EvaluationResult.

In the documented pytest usage (assert result.passed, f"Evaluation failed: {result.summary}") this surfaces as a test failure with an opaque message, which is what makes it hard to attribute — the evaluation never ran at all.

To Reproduce

1. Create any valid config (the contents do not matter, since the failure occurs after dataset loading and before evaluation; an offline config using the in-repo mock provider with a single dataset item is enough).
2. In one process, run a trivial coroutine to completion first, then call the plugin:

import asyncio

async def trivial():
    return 42

asyncio.run(trivial())          # any completed asyncio.run will do

from openagent_eval.cicd.plugin import OAEvalPlugin
result = OAEvalPlugin.run_evaluation("config.yaml")
print("passed:", result.passed)
print("summary:", result.summary)

3. Observe passed: False and an error key in the summary, with no traceback and exit code 0.
4. Remove the asyncio.run(trivial()) line and re-run to see it succeed — the evaluation completes and returns passed: True with a normal gate summary.

Expected Behavior

run_evaluation should run the evaluation regardless of whether an earlier asyncio.run has completed in the process, since it does not require a pre-existing event loop. Failing that, an infrastructure error of this kind should be distinguishable from an evaluation that ran and failed its gates.

Error Output

Failing run:

mode: close-first
asyncio.run(trivial()) -> 42
returned type: EvaluationResult
passed: False
summary: {'error': "There is no current event loop in thread 'MainThread'.", 'duration_seconds': 0.12048816680908203}
EXIT=0

Control run, same script without the preceding asyncio.run:

mode: control
returned type: EvaluationResult
passed: True
summary: {'total_gates': 0, 'passed_gates': 0, 'failed_gates': 0, 'total_thresholds': 0, 'passed_thresholds': 0, 'failed_thresholds': 0, 'duration_seconds': 0.10222291946411133}
EXIT=0

Environment

• OS: Linux
• Python version: 3.11.15
• OpenAgent Eval version: main at b00ab48704f18e6bd4f13101dabe64e107091262
• Installation method: uv

The mechanism lives in CPython's default event-loop policy, which is unchanged between 3.11 and 3.12, so 3.12 is expected to behave the same — but only 3.11 was actually executed here, so treat 3.12 as inference rather than a measured result.

Additional Context

I grepped openagent_eval/ for get_event_loop, run_until_complete and new_event_loop: plugin.py:179 and plugin.py:189 are the only occurrences, both in this one call site, so this appears to be the last place in the package carrying this pattern. Line 189 (loop.run_until_complete(...)) has a second, narrower exposure — if a caller closes a loop that is still set as current, it raises RuntimeError: Event loop is closed and is swallowed into the same shape. asyncio.run never leaves that state, so it is not the path reproduced above.

Worth noting this is the same bug class as #71 and #49, both already fixed, in metrics/llm_judge and executor.py respectively. This call site was not covered by either.

Unverified fix suggestion: replace the get_event_loop() + is_running() pattern with asyncio.get_running_loop() in a try, using asyncio.run(...) when it raises (no loop running) and the existing thread-pool offload when it succeeds. That removes the dependency on the policy's current-loop state entirely and would cover both line 179 and line 189.

Separately, and arguably the more consequential half: because except Exception at line 214 maps any internal failure to passed=False with an error key, an infrastructure fault is reported through the same channel as a legitimate gate failure. That is the documented error-reporting path so I would not call it the defect, but it is why this took a while to attribute.

I found this while adding regression tests in #260 — they pass in isolation and fail when the full suite runs, because an earlier test leaves the process in this state.