Repository navigation
Conversation
…memories _detect_language used a standalone full-text zh_ratio >= 0.7 scan that dilutes the Chinese denominator with paths, timestamps, commands and English terms. For developer-facing Chinese memories the ratio drops well below 0.7 (e.g. zh_chars=2110 / total=6644 -> 0.318), so the materials were classified "en" and the L6 schemas were emitted in English, making them nearly unretrievable by Chinese queries (see Tencent-Hunyuan#243). Reuse utils.lang_detect.detect_language, which already handles code-heavy Chinese via a leading-instruction window plus a 0.3 CJK threshold and langdetect fallback, and is already covered by tests. Normalize its result back to the zh/en contract the callers expect. Add regression tests for tech-heavy Chinese, plain English, and empty materials.
Collaborator
|
Thanks! #269 targets the same issue (#243) by voting per memory, which handles batches whose first memory is code-heavy. We'll go with #269; closing this one. 🤖 Generated with Claude Code |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem
system2_agent._detect_language()used a standalone full-text heuristic:The denominator counts paths, timestamps, commands, punctuation and English terms, so developer-facing Chinese memories are structurally diluted below the 0.7 threshold. As reported in #243, a real evolve run of 32 Chinese facts measured
zh_chars=2110 / total=6644 → 0.318, so the materials were classified"en"and the L6 schemas were emitted in English — nearly unretrievable by Chinese queries (cross-language vector similarity 0.17–0.34 vs 0.59 same-language).Fix
Reuse the existing
utils.lang_detect.detect_language, which already solves this exact case (code-heavy Chinese instructions) via a leading-instruction window plus a 0.3 CJK threshold and a langdetect fallback, and is already covered bytests/test_lang_detect.py. Normalize its result back to thezh/encontract that_detect_language's callers expect.Tests
Added two regression tests to
tests/test_lang_detect.py:test_system2_detect_language_handles_tech_heavy_chinese— tech-heavy Chinese facts (paths, timestamps, English terms) now classify aszh.test_system2_detect_language_english_and_empty— plain English and empty materials still classify asen.All 21 tests in
test_lang_detect.pypass locally.Closes #243