Skip to content

Add feedback tone check (#69) - #1013

Open
SaimM2007 wants to merge 8 commits into
ascherj:mainfrom
SaimM2007:feat/69-feedback-tone-check
Open

Add feedback tone check (#69)#1013
SaimM2007 wants to merge 8 commits into
ascherj:mainfrom
SaimM2007:feat/69-feedback-tone-check

Conversation

@SaimM2007

Copy link
Copy Markdown

Summary

Closes #69

Adds a tone check step to PathReview's feedback generation pipeline. Previously, ContentFilter only caught explicitly harmful content (self-harm, slurs, illegal activity) via regex — it had no concept of "constructive vs. discouraging" tone, and it wasn't even called from the review generation flow. This meant harsh-but-not-technically-harmful feedback could reach users unfiltered.

What changed

safety/content_filter.py

  • Added ToneChecker, an LLM-based classifier that judges whether a piece of feedback is CONSTRUCTIVE (actionable, specific, encouraging — even if critical) or NEGATIVE (discouraging, vague, dismissive).
  • Added ToneCheckResult dataclass (is_constructive: bool, raw_response: str) as the return type.
  • Empty/near-empty content short-circuits to constructive without an LLM call, so it can't get stuck looping on blank input.

rag/generator/review_generator.py

  • Extracted the original single-call generation logic into _generate_section_once().
  • generate_section() now calls _generate_section_once(), runs the result through ToneChecker, and retries generation up to MAX_RETRIES (2) times if the tone check fails.
  • If a section still fails after all retries, the last attempt is returned rather than looping forever, with its confidence score capped at 0.3 and a structlog warning logged so the fallback is visible rather than silent.

Tests

  • tests/unit/test_content_filter.py — reproduction test showing the original gap (ContentFilter has no tone concept), plus new tests for ToneChecker: constructive feedback, negative feedback, critical-but-actionable feedback (must not be falsely flagged), and empty content.
  • tests/unit/test_review_generator_tone_check.py (new) — covers generate_section() passing on the first attempt, succeeding after one regeneration, and falling back correctly after exhausting retries.
  • All 8 tests pass.
  • Ran the full unit suite (428 tests) before and after this change: 53 pre-existing failures exist on main in unrelated modules (review_service, resume_parser, security, skill_extractor, etc.). This PR introduces 0 new test failures and 0 new lint errors.

Open questions

Tone classification strictness is a judgment call — the current prompt is tuned to distinguish "critical but specific" from "vague/dismissive," but I'd welcome feedback on whether it's too strict or too lenient in practice.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Add a "feedback tone check" that ensures all generated feedback is written constructively

1 participant