fix(analyzer): locale-aware case fold for Turkish context words

[fevziegeyurtsevenler]

Change Description

LemmaContextAwareEnhancer lowercases context words and surrounding lemmas with
str.lower() before comparing them. str.lower() is locale-independent, and for
Turkish/Azerbaijani it produces the wrong result for the dotted/dotless I:

>>> "TC KİMLİK NO".lower()
'tc ki̇mli̇k no'      # "İ" (U+0130) -> "i" + U+0307 combining dot
>>> "kimlik" in "TC KİMLİK NO".lower()
False
>>> "NÜFUS CÜZDANI".lower()
'nüfus cüzdani'       # "I" (U+0049) -> "i" instead of "ı" (U+0131)

Because of this, a context word written in uppercase (which is how Turkish ID and
vehicle documents are usually printed) no longer contains the recognizer's context
term, so the context confidence boost is never applied for that input.

This is visible with the bundled Turkish recognizers, whose context lists are
lowercase Turkish ("tc kimlik", "kimlik no", "kayıt", …): those words never
match their own uppercase form. (These recognizers are country-specific and ship
enabled: false in default_recognizers.yaml, so this only affects users who opt
into them — the point is that once enabled, their context matching is locale-broken
for uppercase input.)

Fix

Add a small _fold(text, language) helper on the enhancer. When the recognizer's
supported_language is tr/az, the dotted/dotless I pairs are pre-mapped
(İ→i, I→ı) before lowering; for every other language (and when no language is
given) _fold returns exactly text.lower(). The language is taken from the
matched recognizer.

Which sites fold, which stay plain. The old up-front
context = [word.lower() for word in context] at the top of enhance_using_context
is removed — folding the caller-supplied context early would discard the
original casing that the locale-aware fold needs, so those words are now folded at
comparison time instead. The two comparison sites in
_find_supportive_word_in_context (the substring branch and the whole_word
branch) fold both operands through _fold, and the stored surrounding word
appended in _add_n_words is folded as well. One .lower() deliberately stays
plain: the membership test lemmas[i].lower() in lemmatized_filtered_keywords in
_add_n_words, because those keywords are the plain-str.lower() keywords built by
NlpArtifacts and must be compared like-for-like — folding only the collected word
that is later compared against the recognizer context.

Zero blast radius for non-tr/az: for any other language _fold(x, lang) is
byte-for-byte identical to x.lower(), so existing behaviour is unchanged.

Why this lives in the core enhancer (and why casefold() is not enough)

The fold is applied once, in LemmaContextAwareEnhancer, rather than in each
Turkish recognizer. The lowering that breaks the match happens inside the
enhancer — recognizers only declare their lowercase context lists and never see the
surrounding-word lowering, so a recognizer has no seam at which to intervene.
Putting the rule in each recognizer would duplicate it across every current and
future tr/az recognizer and still could not touch the enhancer's own
str.lower() on surrounding lemmas. Keying off the recognizer's already-declared
supported_language keeps the rule in exactly one place and leaves every other
language on the identical code path.

Switching the existing calls to str.casefold() does not fix this. casefold()
is also locale-independent: "TC KİMLİK NO".casefold() still yields
'tc ki̇mli̇k no' (with the combining dot) and "KAYIT".casefold() still yields
'kayit', not 'kayıt'. The dotted/dotless I has to be mapped explicitly for
tr/az, which is what _fold does before lowering.

Tests

Failing-then-fix unit tests were added for both bundled Turkish recognizers,
asserting that uppercase Turkish context raises the score above the pattern
baseline and records the matching context word:

• test_tr_national_id_recognizer.py: "TC KİMLİK NO" supplied as context.
• test_tr_license_plate_recognizer.py: "KAYIT" as a surrounding lemma
  (exercising the lemma-extraction path).

Both assertions fail on main (score stays at the baseline) and pass with the fix.

A unit test for the whole_word branch was also added to
test_lemma_context_aware_enhancer.py
(test_when_whole_word_turkish_context_then_locale_aware_match) so the Turkish
path of the whole_word fold site is asserted directly:
"KAYIT" matches "kayıt" with language="tr"/"az", and does not match
without a locale. The remaining existing
test_lemma_context_aware_enhancer.py and test_context_support.py cases
continue to pass unchanged.

Issue reference

No pre-existing issue on the tracker; the defect and a runnable reproduction are
described in the Change Description above.

Checklist

• I have reviewed the contribution guidelines
• I agree to follow this project's Code of Conduct
• I confirm that I have the right to submit this contribution and that it does not knowingly contain proprietary or confidential code.
• My code includes unit tests
• All unit tests and lint checks pass locally
• My PR contains documentation updates / additions if required

[fevziegeyurtsevenler]

Gentle ping — could a maintainer approve the CI workflows when they get a chance? The change is narrowly scoped to the lemma context enhancer (tests included, failing-then-fixed).

Also possibly useful context: #2216 raises the same locale/agglutinative-language gap in context enhancement from the benchmarking side — this PR addresses the Turkish/Azerbaijani case-folding portion of that discussion.

[juno-junho]

Author of #2216 here, the link is right. Same layer, different failure mode: your case is the fold before the comparison, mine is the segmentation the comparison runs on. Both are the enhancer assuming text behaves like English.

The repro checks out locally: "TC KİMLİK NO".lower() leaves U+0307 combining dots so "kimlik" never matches, and plain str.casefold() has the same problem, which is a good argument for the locale-aware fold being explicit rather than a one-liner.

Nice to have the Turkish/Azerbaijani side documented with failing-then-fixed tests. One more language family in the "context enhancement needs per-language handling" column.