Track multi-route AI observability coverage and blocked evaluator recheck

[gewenyu99]

Status

Parked for now. This tracks the multi-route AI observability reproduction and the blocked before/after evaluation. No further runs are planned until work resumes.

Reproduction and findings

The Workbench fixture PR exercises six inference route groups: OpenAI-compatible chat, Responses, Ollama chat and generate, direct assistant tool calls, and WebSocket generation. It includes JSON, SSE, NDJSON, provider failures, and cancellation.

• Before guidance changes, the Wizard run took 236.5 seconds and incorrectly captured a successful WebSocket generation as an error without output or tokens.
• After applying Context Mill #406 and manually supplying the Markdown from docs #20409, the Wizard run took 352.5 seconds. The WebSocket classification improved, but five successful $ai_output_choices entries across four route groups still lack role. The promised inference coverage ledger/report was not written, and the SSE completion parser remains brittle to valid JSON whitespace.
• Both runs added capture hooks for all six route groups and passed the fixture's five response-contract tests. Those checks do not establish that the emitted AI observability payloads are complete.

The earlier gateway evaluator scored both runs 5/5 despite the remaining payload/report misses. The Workbench PR adds specific rubric criteria and scoring caps; pnpm test:evaluator passed 57/57. The live recheck with that rubric did not yield a usable after score.

Evaluation blocker

In the live recheck, the evaluator logged a completed report result (10,784 output tokens), then another success result (28 output tokens). Its loop assigns resultText = message.result on each success, so the later result appears to replace the full report. It then retried and failed with No result received from agent (including retry). A model mismatch is possible but unconfirmed; the observed result replacement should be investigated first. This is in the Workbench evaluator (services/pr-evaluator/evaluator.ts), not established as the same failure as Wizard's long-running gateway connection issue #581.

When resumed

1. Preserve the complete evaluator report when subsequent SDK result messages arrive, and rerun the updated rubric against the saved before/after diffs.
2. Verify each successful generation has the required $ai_output_choices shape, including role, across all six routes; verify SSE variants and the promised coverage ledger/report.
3. Decide whether any remaining misses need Context Mill guidance, docs, or Wizard behavior changes, using the Workbench fixture as the regression case.