Add Ukrainian stemmer vocabulary data - #36
Open
polaz wants to merge 14 commits into
Open
Conversation
57,868 words from Ukrainian Wikipedia (dump 2025-01-02), filtered at frequency threshold 300, Cyrillic script only. Generated using scripts/wikipedia-dump-to-freq + scripts/freq-to-voc as documented in CONTRIBUTING.rst. Licensed CC BY-SA 3.0.
These verbal nouns were used as inline examples in ukrainian.sbl to document the R2 guard. Moving them to the corpus proper.
The test cases from ukrainian_stemmer_tests.tsv (now removed from the stemmer repo) are moved here. All stems verified against the current algorithm.
Add лошатко (little foal, diminutive of лоша) to exercise the -атко animal-young diminutive entry. Stem: лош, consistent with лоша→лош. Regenerate output.txt to reflect algorithm improvements from the companion snowball PR: dead adjective alt endings removed, reflexive present-tense stripped correctly, dead noun and derivational entries removed.
Add 27 words exercising diminutive patterns not represented in voc.txt: Intensive diminutives: - одинюсінька/-іньки/-інькі/-інько (-юсіньк- all four forms) - повнісінька/-ісіньки/-ісінькі (-ісіньк- three forms) - прямесенька/-есеньки/-есенькі (-есеньк- three forms) - матусоньки/-онько/-онькі (-оньк- in diminutive; noun→dimstем path) - бабусінька/-іньки/-інько/-інькі (-іньк- in diminutive; noun→dimstем path) - маленьки (-еньки) Standard diminutives: - вікночко/-очкі (-очко/-очкі) - зернечкі (-ечкі) - братічок/-ічко/-ічкі (-ічок/-ічко/-ічкі) Double diminutives: - будиночечка/-ечки/-ечкі (-очечка/-очечки/-очечкі)
Add 4 words that exercise diminutive_stem entries via noun/adj stripping, which the diminutive block does not cover directly: - лошатка: noun strips -а → лошатк → dimstем -атк → лош (лошатко exercises diminutive -атко; gen sg needed for -атк path) - собаченітка: noun strips -а → собаченітк → dimstем -ітк → собачен (собаченітко exercises diminutive -ітко; gen sg needed for -ітк path) - бабусінькій: adj strips -ій → бабусіньк → dimstем -іньк → бабус (all -іньк+vowel forms go through diminutive; -ій form needed for -іньк path) - матусонькій: adj strips -ій → матусоньк → dimstем -оньк → матус (all -оньк+vowel forms go through diminutive; -ій form needed for -оньк path)
братунь (informal diminutive of брат) exercises the -усь/-унь/-інь/-ень diminutive group; the -унь entry was the only form without a hitting word since all -унь corpus words (окунь, грунь) are non-diminutive and their delete is blocked by the pV constraint
Regenerated expected output reflecting corrections to -онн- stemming: - електронних → електрон (was: електр — over-stripped) - телефонних → телефон (was: телеф — over-stripped) - оборонних → оборон (was: обор — over-stripped) - охоронних → охорон (was: охор — over-stripped) - прикордонних → прикордон (was: прикорд — over-stripped) - незаконних → незакон (was: незак — over-stripped) - залізобетонних → залізобетон (was: залізобет — over-stripped) - законних → закон (was: unstemmed) - бетонних → бетон (was: unstemmed) - нейронних → нейрон (was: unstemmed) - синхронних → синхрон (was: unstemmed) - районних → район (was: unstemmed or wrong) - двосторонніх → двосторон (was: двостор — over-stripped)
живуче exercises the adj -уче ending (neuter of -учий present participles). зростаюче exercises the adj -юче ending (neuter of -ючий present participles).
… fix Two classes of changes: 1. Compound adj forms (-ічн-/-уальн-/-ійн-/-аційн-) that were previously unstemmed (R2-blocking bug) now correctly reduce to their adj stems. E.g.: хімічних→хімічн, візуальної→візуальн, логічного→логічн. 2. Two-character words (аа, ав, ай, etc.) no longer incorrectly strip to single letters; they return unchanged per the pV fix (min word length 3).
Reflects 281 improved stemmings: adjective-derived abstract nouns now correctly reduce oblique forms to adjective stem via the new -ості/-остей entries with minimum 4-char stem guard.
Applying the R2 restriction via setlimit in `derivational` merges the -ість nominative form into its adjective stem group for 4 words: впевненість, відданість, людяність, солоність.
Add 3106 product-domain word forms (from a community-collected list of 444 common Ukrainian goods, expanded to full declensions), validated against the LibreOffice dict-uk hunspell dictionary and pymorphy3. Invalid generator artifacts were dropped (declined indeclinable авто, г/ґ spelling, fleeting-vowel errors like оцетом). Update expected output for the -нти over-stemming fix: 53 -нти noun plurals now keep their final т (документи→документ, агенти→агент).
ojwb
reviewed
Jun 10, 2026
| @@ -0,0 +1,14 @@ | |||
| voc.txt and output.txt are licensed as CC BY-SA 3.0. | |||
Member
There was a problem hiding this comment.
Are they?
Taking a large dataset and extracting the most frequent N words is clearly highly transformative and it's not clear to me that the licence of the original text applies to the wordlist. (It's also not clear to me that it automatically doesn't, but apparently an LLM can ignore licensing of code with impunity despite being capable of regenerating significant verbatim sections of the original, whereas you couldn't recreate even one sentence of a wikipedia article working solely from this word list.)
So that's the reason why for the other languages I've just stated how the source data was licensed. Perhaps I should seek a qualified legal opinion on this.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Add test vocabulary for the Ukrainian stemmer (companion to
snowballstem/snowball#277).
(2025-01-02), frequency threshold ≥ 300, Cyrillic script filter
Generated with scripts/wikipedia-dump-to-freq + scripts/freq-to-voc
as described in CONTRIBUTING.rst.