Skip to content

Help search: crop Chinese and Japanese snippets between characters (#1357) - #1374

Merged
Cidan merged 1 commit into
mainfrom
fix/help-snippet-cjk
Nov 19, 2026
Merged

Cidan merged 1 commit into
mainfrom
fix/help-snippet-cjk

Conversation

@Cidan

@Cidan Cidan commented Nov 19, 2026

Copy link
Copy Markdown
Owner

Fixes #1357.

Problem

search:Snippet (help/search.lua) cropped a sentence longer than the snippet width (140 characters on the Help page) by looking only for ASCII spaces. Chinese and Japanese put no spaces between words, so a long sentence came back broken:

  • With no match, or a match near the start: the first character and ….
  • With a match further in: …, one character, ….
  • With a match near the end: … and the sentence's tail, without the matched run's first character.
  • A translation keeping a Latin name with a space ("BetterBags Next") was cut at that space.
  • The window was centered on the start of the whole run of characters between two punctuation marks, so in a long run with no punctuation the matched characters could fall outside it.

No Help text is translated yet, so no player sees this today; it would appear once tools/translate runs on the Help strings. Korean puts spaces between words and was already cut correctly. WoW has no Japanese client, but kana are handled the same way.

Fix

Part Change
isUnspaced Kana, CJK ideographs, the ideographic space, CJK punctuation, and the full-width punctuation sub-ranges (FF01-FF0F, FF1A-FF20, FF3B-FF40, FF5B-FF65). Not Hangul, and not full-width letters and digits.
breaks A cut is allowed at the sentence's ends, beside a space, or beside an unspaced character, unless it leaves a closing mark (,。、)」”…) at the snippet's start or an opening mark ((「“…) at its end.
matchAt Replaces matches and also returns which term matched, so the window centers on the matched character pair inside a run.
Start of the crop Moves forward to a cut but never past the match. Before, it could step one character past the match, in English too when no space came within a third of the width before it.
End of the crop Moves back to a cut but never before the match's end, and is cut at the width when no cut lies in between. Before, it collapsed to the first character.
Edges Spaces and ideographic spaces are trimmed off both edges.

Tests

spec/help/search_spec.lua adds 12 tests:

  • Chinese sentences with and without full-width commas.
  • A match at a run's start, inside a long run, and among the last characters.
  • Latin and full-width Latin words inside Chinese, and a CJK or full-width comma between two Latin words.
  • Closing and opening marks, and ideographic spaces, at the edges.
  • Japanese kana, and Korean cut only at spaces.
  • English crops whose edges fall on a word's edge, with no space before the match, and with one long token.

On the old code, 10 of the 12 fail. The Korean test and the English word-edge test pass on both and guard against regressions. The full suite (8,181 tests) and luacheck . pass.

Review

A bug review compared the old and new crops on every article sentence longer than the width, with each of its terms (10,377 crops), and on 1,810 snippets from real searches. No English snippet changed.

The review found three defects in Chinese crops, each fixed test-first:

  • A closing mark could be left at the snippet's start, or an opening mark at its end.
  • An ideographic space could be left at either edge.
  • Full-width letters could be cut inside a word.

It also found 12 of 26 mutants surviving the first tests. With the added tests, 17 of 19 mutants of the final code are caught. The two that survive produce the same snippet on any real text; .claude/rules/help.md §2 records them.

Docs

.claude/rules/help.md §2 (the cut rules) and §7 (the limitation is no longer listed as left unfixed), and help/README.md (Snippet).

…1357)

Issue #1357: search:Snippet (help/search.lua) cropped a sentence longer than
the snippet width (140 characters on the Help page) by looking only for ASCII
spaces. Chinese and Japanese put no spaces between words, so a long sentence
came back as its first character and an ellipsis, as one character between two
ellipses, or as the sentence's tail without the matched run's first character.
A translation keeping a Latin name with a space ("BetterBags Next") was cut at
that space. The window was also centered on the start of the whole wide run
(the text between two punctuation marks), so in a long run with no punctuation
the matched characters could fall outside it. No Help text is translated yet,
so no player sees this; it would appear once tools/translate runs on the Help
strings. Korean puts spaces between words and was already cut correctly.

Code (help/search.lua)
- isUnspaced: kana, CJK ideographs, the ideographic space, CJK punctuation and
  the full-width punctuation sub-ranges (FF01-FF0F, FF1A-FF20, FF3B-FF40,
  FF5B-FF65). Not Hangul, so Korean is still cut only at spaces; not
  full-width letters and digits, which are cut only at a word's edge.
- breaks: a cut is allowed at the sentence's ends, beside a space, or beside an
  unspaced character, unless it leaves a closing mark (CLOSES: ,。、)」” and
  the rest) at the snippet's start or an opening mark (OPENS: (「“ and the
  rest) at its end.
- matchAt replaces matches and also returns which term matched, so the window
  centers on the matched character pair inside a wide run.
- The start moves forward to a cut but never past the match. Before, it could
  step one character past the match, in English too when no space came within
  a third of the width before it. The end moves back to a cut but never before
  the match's end, and is cut at the width when no cut lies in between; before,
  it collapsed to the first character.
- Spaces and ideographic spaces are trimmed off both edges.
- isWide's comment no longer says Hangul is written without spaces.

Tests (spec/help/search_spec.lua): Chinese sentences with and without
full-width commas; a match at a run's start, inside a long run, and among the
last characters; Latin and full-width Latin words inside Chinese, and a CJK or
full-width comma between two Latin words; closing and opening marks and
ideographic spaces at the edges; Japanese kana; Korean cut only at spaces; and
English crops whose edges fall on a word's edge, with no space before the
match, and with one long token. 10 of the 12 new tests fail on the old code;
the Korean and English word-edge tests guard against regressions.

Review: a bug review compared the old and new crops on every article sentence
over the width with each of its terms (10,377 crops) and on 1,810 snippets of
real searches: no English snippet changed. It found closing and opening marks
and the ideographic space left at the edges, full-width letters cut inside a
word, and 12 of 26 mutants surviving the first tests. The three defects are
fixed test-first above. With the added tests, 17 of 19 mutants of the final
code are caught; the two left produce the same snippet on any real text
(.claude/rules/help.md §2).

Docs: .claude/rules/help.md §2 (the cut rules) and §7 (no longer listed as
left unfixed), help/README.md (Snippet).

Fixes #1357.
@Cidan
Cidan merged commit fcc6087 into main Nov 19, 2026
4 checks passed
@Cidan
Cidan deleted the fix/help-snippet-cjk branch November 19, 2026 05:47
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Help search snippets crop Chinese and Japanese sentences incorrectly

1 participant