Repository navigation
Help search: crop Chinese and Japanese snippets between characters (#1357) - #1374
Merged
Merged
Conversation
…1357) Issue #1357: search:Snippet (help/search.lua) cropped a sentence longer than the snippet width (140 characters on the Help page) by looking only for ASCII spaces. Chinese and Japanese put no spaces between words, so a long sentence came back as its first character and an ellipsis, as one character between two ellipses, or as the sentence's tail without the matched run's first character. A translation keeping a Latin name with a space ("BetterBags Next") was cut at that space. The window was also centered on the start of the whole wide run (the text between two punctuation marks), so in a long run with no punctuation the matched characters could fall outside it. No Help text is translated yet, so no player sees this; it would appear once tools/translate runs on the Help strings. Korean puts spaces between words and was already cut correctly. Code (help/search.lua) - isUnspaced: kana, CJK ideographs, the ideographic space, CJK punctuation and the full-width punctuation sub-ranges (FF01-FF0F, FF1A-FF20, FF3B-FF40, FF5B-FF65). Not Hangul, so Korean is still cut only at spaces; not full-width letters and digits, which are cut only at a word's edge. - breaks: a cut is allowed at the sentence's ends, beside a space, or beside an unspaced character, unless it leaves a closing mark (CLOSES: ,。、)」” and the rest) at the snippet's start or an opening mark (OPENS: (「“ and the rest) at its end. - matchAt replaces matches and also returns which term matched, so the window centers on the matched character pair inside a wide run. - The start moves forward to a cut but never past the match. Before, it could step one character past the match, in English too when no space came within a third of the width before it. The end moves back to a cut but never before the match's end, and is cut at the width when no cut lies in between; before, it collapsed to the first character. - Spaces and ideographic spaces are trimmed off both edges. - isWide's comment no longer says Hangul is written without spaces. Tests (spec/help/search_spec.lua): Chinese sentences with and without full-width commas; a match at a run's start, inside a long run, and among the last characters; Latin and full-width Latin words inside Chinese, and a CJK or full-width comma between two Latin words; closing and opening marks and ideographic spaces at the edges; Japanese kana; Korean cut only at spaces; and English crops whose edges fall on a word's edge, with no space before the match, and with one long token. 10 of the 12 new tests fail on the old code; the Korean and English word-edge tests guard against regressions. Review: a bug review compared the old and new crops on every article sentence over the width with each of its terms (10,377 crops) and on 1,810 snippets of real searches: no English snippet changed. It found closing and opening marks and the ideographic space left at the edges, full-width letters cut inside a word, and 12 of 26 mutants surviving the first tests. The three defects are fixed test-first above. With the added tests, 17 of 19 mutants of the final code are caught; the two left produce the same snippet on any real text (.claude/rules/help.md §2). Docs: .claude/rules/help.md §2 (the cut rules) and §7 (no longer listed as left unfixed), help/README.md (Snippet). Fixes #1357.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Fixes #1357.
Problem
search:Snippet(help/search.lua) cropped a sentence longer than the snippet width (140 characters on the Help page) by looking only for ASCII spaces. Chinese and Japanese put no spaces between words, so a long sentence came back broken:….…, one character,….…and the sentence's tail, without the matched run's first character.No Help text is translated yet, so no player sees this today; it would appear once
tools/translateruns on the Help strings. Korean puts spaces between words and was already cut correctly. WoW has no Japanese client, but kana are handled the same way.Fix
isUnspacedbreaks,。、)」”…) at the snippet's start or an opening mark ((「“…) at its end.matchAtmatchesand also returns which term matched, so the window centers on the matched character pair inside a run.Tests
spec/help/search_spec.luaadds 12 tests:On the old code, 10 of the 12 fail. The Korean test and the English word-edge test pass on both and guard against regressions. The full suite (8,181 tests) and
luacheck .pass.Review
A bug review compared the old and new crops on every article sentence longer than the width, with each of its terms (10,377 crops), and on 1,810 snippets from real searches. No English snippet changed.
The review found three defects in Chinese crops, each fixed test-first:
It also found 12 of 26 mutants surviving the first tests. With the added tests, 17 of 19 mutants of the final code are caught. The two that survive produce the same snippet on any real text;
.claude/rules/help.md§2 records them.Docs
.claude/rules/help.md§2 (the cut rules) and §7 (the limitation is no longer listed as left unfixed), andhelp/README.md(Snippet).