Add sub metrics for conversation correctly finished

docs/metrics/conversation_correctly_finished.md

@@ -53,6 +53,38 @@ Uses the following MetricContext fields:
 }
 ```
 
+## Diagnostic Sub-Metrics
+
+When a record fails (score 0.0), a deterministic classifier assigns **exactly one primary cause** for why the agent went silent, emitted as sub-metrics keyed `conversation_correctly_finished.<cause>_rate`.
+
+**General logic:**
+- **One cause per failure.** Causes are checked in a fixed priority order (infra errors first, since they invalidate the run) and the first match wins — causes never double-count. `unknown_reason` is the residual when nothing else matches.
+- **Score convention (`_rate`)**: `1.0` = this cause occurred, `0.0` = it did not (lower is better). Every cause flag is emitted on every record (all `0.0` on a clean finish), so the cross-record mean reads as "fraction of conversations with this cause".
+- **Signals come from the raw record files** (`agent_perf_stats.csv`, `logs.log`, `pipecat_logs.jsonl`, `user_simulator_events.jsonl`, `audit_log.json`). A missing file just means that cause can't fire — it's noted in `details.notes`, never an error.
+
+**Causes** (in priority order):
+
+| Cause | Detected when |
+|-------|---------------|
+| `tts_api_error` | `pipecat_logs.jsonl` error frame naming a `*TTSService` (infra → `invalid_run`) |
+| `stt_api_error` | `pipecat_logs.jsonl` error frame naming a `*STTService` (infra → `invalid_run`) |
+| `llm_api_error` | fatal LLM error line in `logs.log` (allowlist) with no response after it (infra → `invalid_run`) |
+| `reasoning_too_long` | last `agent_perf_stats.csv` row: empty `response`, no tool call, model reasoned, `stop_reason == length` (ran out of tokens mid-reasoning) |
+| `reasoning_only` | same, but `stop_reason != length` — reasoned (visible text **or** `reasoning_tokens > 0`) yet emitted no answer |
+| `stt_empty_transcription` | cascade, last message from assistant, `VAD fired but no transcription` after the last response, plus a `User turn stopped - complete transcript: ''` line (turn closed empty) |
+| `stt_missing_transcription` | same, but **no** `User turn stopped` line (turn never closed) |
+| `ended_with_user_interruption` | an accepted non-empty user turn, then an interruption, with no response after it |
+| `vad_no_turn_detected` | the user's final utterance came after the agent, but no VAD onset fired in its ±1.5s window |
+| `unknown_reason` | none of the above (residual; a discovery queue for new causes) |
+
+**Input-characteristic flags** are also emitted on failures. These are *orthogonal* — they describe the user's final utterance, not a cause, so they overlap each other and the causes and do **not** sum to 1.0.
+
+| Flag (`…_rate`) | Fires when |
+|-----------------|-----------|
+| `final_turn_short` | 1–2 words, after stripping `[annotation]` tags |
+| `final_turn_acknowledgement` | leads with `yes/no/ok/sure/…` **and** ≤ 6 words |
+| `final_turn_spelled_entity` | a spelled ID/code/name (letter/digit runs ≥3 chars, NATO, "X as in Y", caps codes) that dominates the turn |
+
 ## Related Metrics
 
 - [conversation_valid_end.md](conversation_valid_end.md) - Validates the conversation ended via the `end_call` tool (simulator-side quality gate).
@@ -66,3 +98,4 @@ Uses the following MetricContext fields:
 - **Category**: `diagnostic`
 - **`exclude_from_pass_at_k`**: `True`
 - **Configuration**: None (deterministic)
+- **Sub-metric classifier**: `src/eva/utils/conversation_correctly_finished/` — `classifier.py` (extract raw files into signals → fixed-priority cause selection → build sub-metric flags), `causes/*` (per-cause detect + extract), `final_turn.py` (input-characteristic flags).

src/eva/assistant/agentic/system.py

@@ -316,7 +316,7 @@ async def _run_tool_loop(
                     "tool_calls": json.dumps(response_tool_calls_for_stats, ensure_ascii=False)
                     if response_tool_calls_for_stats
                     else "",
-                    "reasoning": f'"{reasoning_content_for_csv}"',
+                    "reasoning": reasoning_content_for_csv or None,
                     "reasoning_tokens": reasoning_tokens,
                 }
                 self.agent_perf_stats.append(perf_stat)

src/eva/metrics/diagnostic/conversation_correctly_finished.py

@@ -4,14 +4,30 @@
 from eva.metrics.processor import is_agent_timeout_on_user_turn, last_audio_speaker
 from eva.metrics.registry import register_metric
 from eva.models.results import MetricScore
+from eva.utils.conversation_correctly_finished import (
+    Classification,
+    build_conv_finish_sub_metrics,
+    build_final_turn_flag_sub_metrics,
+    classify_conv_finish_failure,
+    extract_conv_finish_signals,
+)
 
 
 @register_metric
 class ConversationCorrectlyFinishedMetric(CodeMetric):
-    """0.0 when the agent timed out on the user's final turn; 1.0 otherwise."""
+    """0.0 when the agent timed out on the user's final turn; 1.0 otherwise.
+
+    It also emits per-cause diagnostic sub-metrics
+    (``conversation_correctly_finished.<cause>_rate``) splitting *why* the agent went silent.
+    These fire (``1.0``) only on failures, but are emitted as ``0.0`` on every clean finish too,
+    so their cross-record mean reads as "fraction of all conversations with this cause" —
+    matching how faithfulness dimension rates aggregate (denominator = all records, not just
+    failures). The orthogonal input-characteristic flags (``final_turn_*``) stay failure-only,
+    since they describe the unanswered final turn.
+    """
 
     name = "conversation_correctly_finished"
-    version = "v0.1"
+    version = "v0.3"
     description = "Diagnostic metric: 0.0 when agent failed to respond to the user's final turn"
     category = "diagnostic"
     exclude_from_pass_at_k = True
@@ -30,12 +46,24 @@ async def compute(self, context: MetricContext) -> MetricScore:
             )
             score = 0.0 if missed_turn else 1.0
 
+            # Classify the cause only when the conversation actually failed. Emit the per-cause
+            # rate sub-metrics on every record regardless (all 0.0 on a clean finish) so the
+            # cross-record mean has an all-records denominator, matching faithfulness.
             if missed_turn:
                 human_reason = "conversation ended with inactivity_timeout and user was the last speaker"
-            elif reason == "inactivity_timeout":
-                human_reason = f"inactivity_timeout but last speaker was {speaker!r}"
+                signals = extract_conv_finish_signals(context)
+                classification = classify_conv_finish_failure(signals)
+                sub_metrics = build_conv_finish_sub_metrics(classification, self.name)
+                # Orthogonal input-characteristic flags (short / acknowledgement / spelled final turn).
+                sub_metrics.update(build_final_turn_flag_sub_metrics(signals.user_final_words, self.name))
             else:
-                human_reason = f"conversation ended with reason={reason!r}"
+                # Clean finish: no cause fired, but emit every cause flag at 0.0 so this record
+                # counts toward the rate denominator.
+                sub_metrics = build_conv_finish_sub_metrics(Classification(None, {}), self.name)
+                if reason == "inactivity_timeout":
+                    human_reason = f"inactivity_timeout but last speaker was {speaker!r}"
+                else:
+                    human_reason = f"conversation ended with reason={reason!r}"
 
             return MetricScore(
                 name=self.name,
@@ -46,6 +74,7 @@ async def compute(self, context: MetricContext) -> MetricScore:
                     "last_audio_speaker": speaker,
                     "reason": human_reason,
                 },
+                sub_metrics=sub_metrics,
             )
 
         except Exception as e:

src/eva/metrics/diagnostic/response_speed.py

@@ -10,6 +10,7 @@
 from eva.metrics.base import CodeMetric, MetricContext
 from eva.metrics.registry import register_metric
 from eva.models.results import MetricScore
+from eva.utils.log_processing import load_audit_log
 
 
 def _load_component_latencies(output_dir: str) -> dict[str, dict]:
@@ -36,13 +37,47 @@ def _load_component_latencies(output_dir: str) -> dict[str, dict]:
     return latencies
 
 
+def _load_inference_tool_counts(output_dir: str) -> list[int] | None:
+    """Read audit_log.json once and return per-inference tool-call counts.
+
+    An *inference* is one model round-trip (one ``llm_prompts`` entry): the
+    agent may run several within a single user turn as it loops through the
+    reason-act cycle. Returns a list where element ``i`` is the number of tool
+    calls emitted by the i-th inference; the caller derives all aggregate counts
+    and rates from it.
+
+    Uses the same inference definition as the cross-run scan so numbers are
+    comparable. Returns None if the audit log is missing/unreadable or has no
+    ``llm_prompts``.
+    """
+    audit = load_audit_log(Path(output_dir) / "audit_log.json")
+    if audit is None:
+        return None
+
+    prompts = audit.get("llm_prompts")
+    prompts = prompts if isinstance(prompts, list) else []
+    if not prompts:
+        return None
+
+    per_inference_tools = []
+    for c in prompts:
+        rm = c.get("response_message") if isinstance(c, dict) else None
+        tool_calls = rm.get("tool_calls") if isinstance(rm, dict) else None
+        per_inference_tools.append(len(tool_calls) if isinstance(tool_calls, list) else 0)
+
+    return per_inference_tools
+
+
+def _tool_call_turn_ids(context: MetricContext) -> set:
+    """Return the set of turn_ids that contain at least one tool call."""
+    return {entry["turn_id"] for entry in (context.conversation_trace or []) if entry.get("type") == "tool_call"}
+
+
 def _split_by_tool_calls(
     context: MetricContext,
 ) -> tuple[list[float], list[float]]:
     """Partition per_turn_latency values into (with_tool_calls, no_tool_calls)."""
-    tool_call_turn_ids = {
-        entry["turn_id"] for entry in (context.conversation_trace or []) if entry.get("type") == "tool_call"
-    }
+    tool_call_turn_ids = _tool_call_turn_ids(context)
 
     with_tool = [v for k, v in context.latency_assistant_turns.items() if k in tool_call_turn_ids]
     no_tool = [v for k, v in context.latency_assistant_turns.items() if k not in tool_call_turn_ids]
@@ -87,7 +122,7 @@ class ResponseSpeedMetric(CodeMetric):
     description = "Diagnostic metric: latency between user utterance end and assistant response start"
     exclude_from_pass_at_k = True
     higher_is_better = False  # Score is latency in seconds — lower is better.
-    version = "v0.2"
+    version = "v0.4"
 
     async def compute(self, context: MetricContext) -> MetricScore:
         try:
@@ -129,6 +164,57 @@ async def compute(self, context: MetricContext) -> MetricScore:
                         details=stats,
                     )
 
+            # Inference / tool-call efficiency diagnostics from audit_log.json.
+            # Rates count tool-calling inferences only (no-tool inferences excluded).
+            # Each sub-metric carries only the counts that explain its own score.
+            per_inference_tools = _load_inference_tool_counts(context.output_dir)
+            if per_inference_tools is not None:
+                num_inferences_with_tool_calls = sum(1 for n in per_inference_tools if n >= 1)
+
+                # Tool calls per tool-using turn: when the model uses tools in a turn,
+                # how many tool calls land in that turn (across all its inferences).
+                num_turns_with_tool_calls = len(_tool_call_turn_ids(context))
+                sub_metrics["num_tool_calls_per_tool_turn"] = MetricScore(
+                    name=f"{self.name}.num_tool_calls_per_tool_turn",
+                    score=(
+                        round(sum(per_inference_tools) / num_turns_with_tool_calls, 3)
+                        if num_turns_with_tool_calls
+                        else 0.0
+                    ),
+                    normalized_score=None,
+                    details={
+                        "num_tool_calls": sum(per_inference_tools),
+                        "num_turns_with_tool_calls": num_turns_with_tool_calls,
+                    },
+                )
+                # Mean tool calls per tool-calling inference — the model's tool-call
+                # parallelism (1.0 = always sequential, >1.0 = emits parallel tool calls).
+                sub_metrics["tool_call_parallelism"] = MetricScore(
+                    name=f"{self.name}.tool_call_parallelism",
+                    score=(
+                        round(sum(per_inference_tools) / num_inferences_with_tool_calls, 3)
+                        if num_inferences_with_tool_calls
+                        else 0.0
+                    ),
+                    normalized_score=None,
+                    details={
+                        "num_tool_calls": sum(per_inference_tools),
+                        "num_inferences_with_tool_calls": num_inferences_with_tool_calls,
+                        "num_inferences_with_parallel_tool_calls": sum(1 for n in per_inference_tools if n > 1),
+                        "max_tools_per_inference": max(per_inference_tools),
+                    },
+                )
+                # Fraction of inferences that call >=1 tool.
+                sub_metrics["tool_inference_rate"] = MetricScore(
+                    name=f"{self.name}.tool_inference_rate",
+                    score=round(num_inferences_with_tool_calls / len(per_inference_tools), 3),
+                    normalized_score=None,
+                    details={
+                        "num_inferences_with_tool_calls": num_inferences_with_tool_calls,
+                        "num_inferences": len(per_inference_tools),
+                    },
+                )
+
             # Add per-component latency sub_metrics from result.json.
             # result.json stores these in milliseconds; convert to seconds here
             # to match the top-level response_speed score (already in seconds).

src/eva/models/config.py

@@ -117,7 +117,11 @@ class ModelConfig(BaseModel):
         "stt_model": "stt",
         "tts_model": "tts",
     }
-    _LEGACY_DROP: ClassVar[set[str]] = {"realtime_model", "realtime_model_params"}
+    _LEGACY_DROP: ClassVar[set[str]] = {
+        "realtime_model",
+        "realtime_model_params",
+        "audio_llm_full_audio_context",
+    }
 
     # STT models that perform their own (server-side) semantic endpointing. They drive turn
     # boundaries themselves, so they must run with the 'external' turn strategies and no local VAD.

src/eva/utils/conversation_correctly_finished/__init__.py

@@ -0,0 +1,27 @@
+"""Classifier for `conversation_correctly_finished` failures — one primary cause per record.
+
+Modules: ``signals`` (contract), ``classifier`` (extract → classify → build sub-metrics),
+``causes/*`` (per-cause detect + extract), ``final_turn`` (input-characteristic flags). Public API
+re-exported below.
+"""
+
+from eva.utils.conversation_correctly_finished.classifier import (
+    CATEGORY_PRIORITY,
+    build_conv_finish_sub_metrics,
+    build_final_turn_flag_sub_metrics,
+    classify_conv_finish_failure,
+    extract_conv_finish_signals,
+)
+from eva.utils.conversation_correctly_finished.final_turn import final_turn_input_flags
+from eva.utils.conversation_correctly_finished.signals import Classification, ConvFinishSignals
+
+__all__ = [
+    "CATEGORY_PRIORITY",
+    "Classification",
+    "ConvFinishSignals",
+    "build_conv_finish_sub_metrics",
+    "build_final_turn_flag_sub_metrics",
+    "classify_conv_finish_failure",
+    "extract_conv_finish_signals",
+    "final_turn_input_flags",
+]

src/eva/utils/conversation_correctly_finished/causes/__init__.py

@@ -0,0 +1 @@
+"""Per-cause modules — each owns one cause-family's log/CSV/pipecat parsing and its ``detect_*`` logic."""

src/eva/utils/conversation_correctly_finished/causes/api_errors.py

@@ -0,0 +1,89 @@
+"""Causes: infra API errors — TTS/STT service frames and fatal LLM API errors.
+
+They invalidate the run, so the classifier ranks them ahead of any behavioral cause.
+"""
+
+import re
+from typing import Any
+
+from eva.utils.conversation_correctly_finished.signals import Classification, ConvFinishSignals
+from eva.utils.log_processing import parse_log_message
+
+# Fatal LLM API-error signatures — allowlist, not a broad `litellm.*Error` match.
+_RE_LLM_ERR = re.compile(
+    r"MidStreamFallbackError|APIConnectionError|Retryable streaming error|RateLimitError|InternalServerError"
+)
+_RE_SERVICE = re.compile(r"([A-Za-z0-9]+(?:TTS|STT)Service)")
+
+
+def extract_service_errors(pipecat_events: list[dict], s: ConvFinishSignals) -> None:
+    """Set tts/stt service-error flags from pipecat error frames."""
+    for e in pipecat_events:
+        if e.get("type") != "error":
+            continue
+        frame = str(e.get("data", {}).get("frame", ""))
+        m = _RE_SERVICE.search(frame)
+        if not m:
+            continue
+        s.num_service_error_frames += 1
+        svc = m.group(1)
+        if "TTS" in svc:
+            s.tts_service_error = True
+            s.service_error_name = svc
+        elif "STT" in svc:
+            s.stt_service_error = True
+            s.service_error_name = svc
+        s.service_error_excerpt = frame[:160]
+
+
+def extract_log_signals(lines: list[str], resp_idx: list[int], s: ConvFinishSignals) -> None:
+    """LLM API error: a fatal error with no non-empty response after the last error line."""
+    err_idx = [i for i, ln in enumerate(lines) if _RE_LLM_ERR.search(ln)]
+    if not err_idx:
+        return
+    last_err = err_idx[-1]
+    s.num_llm_api_error_lines = len(err_idx)
+    s.llm_api_error_terminal = not any(i > last_err for i in resp_idx)
+    if s.llm_api_error_terminal:
+        m = _RE_LLM_ERR.search(lines[last_err])
+        s.llm_error_type = m.group(0) if m else ""
+        s.llm_error_excerpt = parse_log_message(lines[last_err])[:160]
+
+
+def _infra_details(component: str, s: ConvFinishSignals) -> dict[str, Any]:
+    """Shared details block for the tts/stt infra-error classifications."""
+    return {
+        "invalid_run": True,
+        "component": component,
+        "service": s.service_error_name,
+        "error_excerpt": s.service_error_excerpt,
+        "num_error_frames": s.num_service_error_frames,
+        "assistant_audio_events": s.assistant_audio_events,
+    }
+
+
+def detect_service_error(s: ConvFinishSignals, base: dict[str, Any]) -> Classification | None:
+    """Infra: a TTS/STT service raised an error frame — the agent was never exercised."""
+    if s.tts_service_error:
+        return Classification("tts_api_error", {**base, **_infra_details("tts", s)})
+    if s.stt_service_error:
+        return Classification("stt_api_error", {**base, **_infra_details("stt", s)})
+    return None
+
+
+def detect_llm_api_error(s: ConvFinishSignals, base: dict[str, Any]) -> Classification | None:
+    """Infra: a fatal LLM API error with no successful response after it (invalid run)."""
+    if not s.llm_api_error_terminal:
+        return None
+    return Classification(
+        "llm_api_error",
+        {
+            **base,
+            "invalid_run": True,
+            "component": "llm",
+            "error_type": s.llm_error_type,
+            "error_excerpt": s.llm_error_excerpt,
+            "num_api_error_lines": s.num_llm_api_error_lines,
+            "responses_after_last_error": 0,
+        },
+    )