fix(diagnosis): require low relevancy before blaming short answers as off-topic

[Nitjsefnie]

Description

_analyze_generation flagged any answer shorter than ANSWER_TOO_SHORT = 10 characters as OFF_TOPIC_ANSWER, so legitimate short answers — "Yes.", "42", "Paris" — were blamed on the generation component with 0.7 confidence, skewing the blame summary and overall_health.

The length heuristic is now one signal rather than the whole test: a short answer is only flagged when answer_relevancy also indicates a problem.

Type of Change

• Bug fix (non-breaking change which fixes an issue)
• New feature (non-breaking change which adds functionality)
• Breaking change (fix or feature that would cause existing functionality to not work as expected)
• Documentation update
• Refactoring (no functional changes)
• Test update
• CI/CD update

Related Issues

Closes #66

How Has This Been Tested?

• Unit tests pass (uv run pytest tests/unit -v --tb=short → 1010 passed, 5 skipped)
• Whole suite with coverage, matching the CI Coverage job (--cov-fail-under=75 → 1064 passed, 80.67%)
• Linter passes (uv run ruff check .) — see note below
• Type checker passes (uv run mypy openagent_eval/) — see note below
• Manual testing performed

Tests cover both directions, because a narrowed heuristic and a disabled one are otherwise indistinguishable:

• your four examples ("Yes.", "42", "Paris" and friends) with healthy scores are no longer flagged — this fails on unfixed code;
• a genuinely off-topic short answer, where relevancy is low, is still flagged — this fails if the heuristic is neutered entirely.

I picked your second suggestion rather than lowering the threshold, because a smaller constant only moves the boundary — "42" is still two characters, and any threshold admitting it also admits genuinely truncated output.

Checklist

• My code follows the project's coding standards
• I have performed a self-review of my own code
• I have commented my code, particularly in hard-to-understand areas
• I have made corresponding changes to the documentation
• My changes generate no new warnings
• I have added tests that prove my fix is effective or that my feature works
• New and existing unit tests pass locally with my changes

Additional Notes

Two things I checked because they looked like they could be regressions, and want to report either way.

First, empty answers. Requiring corroboration could in principle swallow the clearest true positive there is, so I ran it: an empty answer with a hypothetically healthy relevancy score would indeed go unflagged — but that state is unreachable, because AnswerRelevancy.evaluate returns a hard 0.0 with "No answer provided" for an empty answer (relevancy.py:54), and _run_metrics always records it. End to end, an empty answer is still blamed on generation with overall_health 0.0. I therefore did not add a length-0 special case, since it would guard a state the metric layer already makes impossible.

Second, test_empty_answer_blames_generation needed a fixture adjustment. Worth saying plainly what changed: its assertion is untouched, and despite the name it never contained an empty answer — its fixture was an instance of the very false-positive class this issue describes. It now supplies the corroborating signal explicitly.

One disclosed consequence of this approach: if a user configures a metric set without answer_relevancy, the corroborating signal is absent and the short-answer heuristic goes inert rather than firing. That is inherent to gating on another metric, not something a carve-out should hide — flagging it in case you would rather it fall back to length-only when relevancy is unavailable.

On the two unchecked test boxes. ruff check . exits 1 on pre-existing findings at main itself, so I could not honestly tick it. Per-file at main versus this branch: one pre-existing I001 in the test file, zero new findings. mypy is unticked for the same reason — not clean at main, and unchanged by this diff.

Generated by Claude Opus 5 (brief, review), Kimi K3 (implementation)

[github-actions]

🎉 Congratulations @Nitjsefnie!

Your pull request has been successfully merged into main. 🚀

Thank you for contributing to OpenAgentHQ and helping improve the project.

We truly appreciate your contribution and hope to see you back with more amazing PRs!

Happy Open Sourcing! ❤️