Hi Patrick,
I've noticed an issue when trying to add hyphenated compound words (like the French "pousse-café" or "prêchi-prêcha") to the personal user dictionary. It is currently very difficult, if not impossible, to have these words saved and predicted as a single unit.
Suspected Cause: While looking at the code in AutoCorrectEditScreen.kt, the initial validation helper addReplacementToUserDictionaryIfNeeded only checks if the string is not empty and contains at least one letter or digit. A hyphenated word successfully passes this filter.
However, because Pastiera's virtual keyboard layout and prediction logic are based on AOSP LatinIME, it is highly likely that:
The hyphen (-) is treated by the tokenizer/word-tracker as a word separator rather than a valid word character.
When UserDictionaryStore tries to commit or read the word, the underlying engine splits it into two separate words ("pousse" and "café") or simply ignores it as it contains "punctuation".
Interestingly, hyphenated words work perfectly fine as static auto-correction replacements (for instance, "luimeme": "lui-même" is present in auto_corrections_fr.json). The limitation seems strictly tied to how the dictionary database and the prediction engine index word tokens.
Could the dictionary engine or the word tokenizer be adjusted to treat the soft hyphen/dash as a valid word character instead of a separator?
Thank you for your help!
Best regards,
Hi Patrick,
I've noticed an issue when trying to add hyphenated compound words (like the French "pousse-café" or "prêchi-prêcha") to the personal user dictionary. It is currently very difficult, if not impossible, to have these words saved and predicted as a single unit.
Suspected Cause: While looking at the code in AutoCorrectEditScreen.kt, the initial validation helper addReplacementToUserDictionaryIfNeeded only checks if the string is not empty and contains at least one letter or digit. A hyphenated word successfully passes this filter.
However, because Pastiera's virtual keyboard layout and prediction logic are based on AOSP LatinIME, it is highly likely that:
The hyphen (-) is treated by the tokenizer/word-tracker as a word separator rather than a valid word character.
When UserDictionaryStore tries to commit or read the word, the underlying engine splits it into two separate words ("pousse" and "café") or simply ignores it as it contains "punctuation".
Interestingly, hyphenated words work perfectly fine as static auto-correction replacements (for instance, "luimeme": "lui-même" is present in auto_corrections_fr.json). The limitation seems strictly tied to how the dictionary database and the prediction engine index word tokens.
Could the dictionary engine or the word tokenizer be adjusted to treat the soft hyphen/dash as a valid word character instead of a separator?
Thank you for your help!
Best regards,