Skip to content

fix(system2): reuse shared language detection for tech-heavy Chinese memories - #415

Closed
zuizhuai wants to merge 1 commit into
Tencent-Hunyuan:mainfrom
zuizhuai:fix-system2-language-detection
Closed

zuizhuai wants to merge 1 commit into
Tencent-Hunyuan:mainfrom
zuizhuai:fix-system2-language-detection

Conversation

@zuizhuai

Copy link
Copy Markdown
Contributor

Problem

system2_agent._detect_language() used a standalone full-text heuristic:

zh_ratio = zh_chars / total_chars   # total_chars = every non-whitespace char
lang = "zh" if zh_ratio >= 0.7 else "en"

The denominator counts paths, timestamps, commands, punctuation and English terms, so developer-facing Chinese memories are structurally diluted below the 0.7 threshold. As reported in #243, a real evolve run of 32 Chinese facts measured zh_chars=2110 / total=6644 → 0.318, so the materials were classified "en" and the L6 schemas were emitted in English — nearly unretrievable by Chinese queries (cross-language vector similarity 0.17–0.34 vs 0.59 same-language).

Fix

Reuse the existing utils.lang_detect.detect_language, which already solves this exact case (code-heavy Chinese instructions) via a leading-instruction window plus a 0.3 CJK threshold and a langdetect fallback, and is already covered by tests/test_lang_detect.py. Normalize its result back to the zh/en contract that _detect_language's callers expect.

Tests

Added two regression tests to tests/test_lang_detect.py:

  • test_system2_detect_language_handles_tech_heavy_chinese — tech-heavy Chinese facts (paths, timestamps, English terms) now classify as zh.
  • test_system2_detect_language_english_and_empty — plain English and empty materials still classify as en.

All 21 tests in test_lang_detect.py pass locally.

Closes #243

…memories

_detect_language used a standalone full-text zh_ratio >= 0.7 scan that
dilutes the Chinese denominator with paths, timestamps, commands and
English terms. For developer-facing Chinese memories the ratio drops well
below 0.7 (e.g. zh_chars=2110 / total=6644 -> 0.318), so the materials
were classified "en" and the L6 schemas were emitted in English, making
them nearly unretrievable by Chinese queries (see Tencent-Hunyuan#243).

Reuse utils.lang_detect.detect_language, which already handles code-heavy
Chinese via a leading-instruction window plus a 0.3 CJK threshold and
langdetect fallback, and is already covered by tests. Normalize its result
back to the zh/en contract the callers expect.

Add regression tests for tech-heavy Chinese, plain English, and empty
materials.
@txfei

txfei commented Nov 10, 2026

Copy link
Copy Markdown
Collaborator

Thanks! #269 targets the same issue (#243) by voting per memory, which handles batches whose first memory is code-heavy. We'll go with #269; closing this one.

🤖 Generated with Claude Code

@txfei txfei closed this Nov 10, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[Bug] System2 归纳语言检测对技术型中文记忆失效——zh_ratio 阈值 0.7 在含大量 ASCII 的中文内容下必判 en

2 participants