Missing languages

[Jibec]

Hello Michal,

here is a list of languages I discovered in Linux files and missing in language-data.

If I can't find the language either in language_codes.LANGUAGES, aliases.ALIASES or countries.DEFAULT, then I check if both filename and "language" po properties are the same.

Here is the result of this.

Language code	Teams observed in po files
aa_dj	(afar (djibouti) (http://www.transifex.com/morevnaproject/synfig/language/aa_dj/))
adh	(language ll@li.org)
agr	(language ll@li.org)
aln	(albanian gheg (http://www.transifex.com/projects/p/fedora/language/aln/))
aln	(albanian gheg (http://www.transifex.com/projects/p/python-meh/language/aln/))
aln	(albanian gheg (http://www.transifex.com/projects/p/initial-setup/language/aln/))
aln	(albanian gheg (http://www.transifex.com/projects/p/certmonger/language/aln/))
ang	(old english modean52@comcast.net)
ang@latin	(old english http://wiki.wesnoth.org/oldenglishtranslation)
ang@latin	(language ll@li.org)
ang@latin	(none)
ast_es	(asturian (spain) (https://www.transifex.com/arctic-games/teams/95/ast_es/))
ayc	(language ll@li.org)
az_ir	(iranian azerbaijani az-ir@lists.sharif.edu)
az_ir	(iranian azerbaiijani az-ir@lists.sharif.edu)
az_ir	(general)
bal	(baluchi)
bal	(balochi (http://www.transifex.com/projects/p/anaconda/language/bal/))
bal	(balochi trans-bal@lists.fedoraproject.org)
bal	(balochi (http://www.transifex.com/projects/p/fedora/language/bal/))
bal	(balochi fedora-trans-bal@redhat.com)
bal	(balochi (http://www.transifex.com/projects/p/certmonger/language/bal/))
bal	(balochi (http://www.transifex.com/projects/p/libvirt-glib/language/bal/))
be@tarask	(belarusian (tarask) (https://www.transifex.com/xfce/teams/16840/be@tarask/))
btk	(none)
cpp	(language ll@li.org)
de@hebrew	(german translation-team-de@lists.sourceforge.net)
en_br	(english (brazil) (http://www.transifex.com/bgk/vba-m/language/en_br/))
en_cz	(english (czech republic) (http://www.transifex.com/morevnaproject/synfig/language/en_cz/))
en@arabic	(none)
en@boldquot	(none)
en@cyrillic	(none)
en@greek	(none)
en@hebrew	(none)
en@piglatin	(none)
en@quot	(none)
en@shaw	(shavian ubuntu-l10n-en-shaw@launchpad.net)
en@shaw	(language ll@li.org)
en@shaw	(none)
en@shaw	(shavian ubuntu-l10n-en-shaw@lists.launchpad.net)
es_co	(spanish (colombia) (https://www.transifex.com/mate/teams/13566/es_co/))
es_co	()
es_co	(spanish (colombia) (http://www.transifex.com/projects/p/powertop/language/es_co/))
es_co	(spanish (colombia) (http://www.transifex.com/mate/mate/language/es_co/))
es_co	(spanish gnome-es-list@gnome.org)
es_co	(spanish)
es_co	(spanish (colombia) (http://www.transifex.com/bgk/vba-m/language/es_co/))
es_co	(spanish (colombia) (https://www.transifex.com/glpi/teams/1637/es_co/))
es_cr	(spanish (costa rica) (https://www.transifex.com/mate/teams/13566/es_cr/))
es_cr	(spanish (costa rica) (http://www.transifex.com/mate/mate/language/es_cr/))
es_cr	(spanish gnome-es-list@gnome.org)
es_cr	(spanish (costa rica) (http://www.transifex.com/bgk/vba-m/language/es_cr/))
es_gt	(spanish gnome-es-list@gnome.org)
es_hn	(spanish gnome-es-list@gnome.org)
es_ni	(spanish (nicaragua) (https://www.transifex.com/mate/teams/13566/es_ni/))
es_ni	(spanish (nicaragua) (http://www.transifex.com/mate/mate/language/es_ni/))
es_ni	(spanish gnome-es-list@gnome.org)
es_pa	(spanish (panama) (https://www.transifex.com/mate/teams/13566/es_pa/))
es_pa	(spanish (panama) (http://www.transifex.com/mate/mate/language/es_pa/))
es_pa	(spanish gnome-es-list@gnome.org)
es_py	(spanish (paraguay) (http://www.transifex.com/morevnaproject/synfig/language/es_py/))
es_sv	(spanish (el salvador) (https://www.transifex.com/mate/teams/13566/es_sv/))
es_sv	(spanish (el salvador) (http://www.transifex.com/needle-and-thread/vocal/language/es_sv/))
es_sv	(spanish (el salvador) (http://www.transifex.com/mate/mate/language/es_sv/))
es_sv	(spanish gnome-es-list@gnome.org)
es_uy	(spanish (uruguay) (https://www.transifex.com/mate/teams/13566/es_uy/))
es_uy	(spanish (uruguay) (http://www.transifex.com/mate/mate/language/es_uy/))
es_uy	(spanish gnome-es-list@gnome.org)
german	(german)
gom	(language ll@li.org)
gom@latin	(language ll@li.org)
grc	(language ll@li.org)
grc	(none)
grc	()
guc	(wayuunaiki)
gug	(language ll@li.org)
gug_py	(guarani (paraguay) (http://www.transifex.com/mate/mate/language/gug_py/))
hch	(language ll@li.org)
hus	(language ll@li.org)
ilo	(iloko)
ilo	(iloko (http://www.transifex.com/projects/p/anaconda/language/ilo/))
ilo	(iloko (http://www.transifex.com/projects/p/blivet/language/ilo/))
ilo	(iloko (http://www.transifex.com/projects/p/fedora/language/ilo/))
ilo	(iloko (http://www.transifex.com/projects/p/certmonger/language/ilo/))
ilo	(iloko (http://www.transifex.com/projects/p/libvirt-glib/language/ilo/))
ja_hira	(language ll@li.org)
kok@latin	(konkani)
kos	(language ll@li.org)
krl	(karelian (http://www.transifex.com/supertuxkart/supertuxkart/language/krl/))
krl	(karelian (http://www.transifex.com/widelands/widelands/language/krl/))
krl	(karelian (https://www.transifex.com/widelands/teams/35159/krl/))
ks@aran	(kashmiri)
ks@deva	(kashmiri)
ks@devanagari	(kashmiri)
ku_iq	(kurdish (iraq) (https://www.transifex.com/mate/teams/13566/ku_iq/))
ku_iq	(kurdish (iraq) (http://www.transifex.com/pidgin/pidgin/language/ku_iq/))
ku_iq	(kurdish (iraq) (http://www.transifex.com/mate/mate/language/ku_iq/))
ku_iq	(kurdish (iraq) (http://www.transifex.com/linuxdeepin/deepin-desktop-environment/language/ku_iq/))
kw@kkcor	(cornish (common orthography))
kw@uccor	(cornish (unified orthography))
ltg	(latgalian (https://www.transifex.com/mageialinux/teams/9361/ltg/))
ltg	(latgalian (http://www.transifex.com/p/spacefm/language/ltg/))
ltg	(latgalian (http://www.transifex.com/calibre/calibre/language/ltg/))
miq	(language ll@li.org)
mjw	(karbi mjw@li.org)
mjw	(karbi karbi.translation@gmail.com)
mni@beng	(gist)
mvo	(language ll@li.org)
my_mm	(burmese (myanmar) (http://www.transifex.com/pidgin/pidgin/language/my_mm/))
nhn	(central nahuatl gnome-nhn-list@gnome.org)
niu	(language ll@li.org)
pam	(kapampangan (http://www.transifex.com/linuxdeepin/deepin-desktop-environment/language/pam/))
pbs	(language ll@li.org)
pis	(language ll@li.org)
quy	(language ll@li.org)
quz	(language ll@li.org)
quz	(voluntarios quechua sugar)
quz	(voluntarios quechuas sugar camp lima)
quz	(voluntarios quechua sugar camp)
quz	(quechua (cuzco-collao))
racv	(language ll@li.org)
racv	(valencian (racv) team)
racv	(none)
ru_ru.cp1251	()
ru_ru.koi8r	()
ru_ru.utf8	()
sat@deva	(language ll@li.org)
sd@deva	(hindi )
sr@ije	(serbian (sr) serbiangnome-lista@nongnu.org)
sr@ije	(serbian (sr) serbiagnome-lista@nongnu.org)
sr@ije	(serbian (sr) gnom@prevod.org)
sr@ijekavian	(serbian kde-i18n-sr@kde.org)
sr@ijekavian	(language ll@li.org)
sr@ijekavian	(none)
sr@ijekavianlatin	(serbian kde-i18n-sr@kde.org)
sr@ijekavianlatin	(language ll@li.org)
sr@ijekavianlatin	(none)
sr@latn	(serbian (sr) serbiangnome-lista@nongnu.org)
sw_tz	(none)
tlh_aa	(klingon)
tvll	(language ll@li.org)
tyv	(language ll@li.org)
tzo	(language ll@li.org)
uz@cyrillic	(uzbek)
uz@cyrillic	(uzbek@cyrillic)
uz@cyrillic	(uzbek (cyrillic) <>)
uz@cyrillic	(http: littlesvr.ca ostd )
uz@cyrillic	(uzbek (cyrillic) <uz@cyrillic@li.org>)
uz@cyrillic	(uzbek https://hosted.weblate.org/projects/remmina/remmina/uz@cyrillic/)
uz@cyrillic	(uzbek bahodir@mansurov.org)
uz@cyrl	(uzbek (cyrillic) (http://www.transifex.com/ethereal/modem-manager-gui/language/uz@cyrl/))
uz@latn	(uzbek (latin) (http://www.transifex.com/xfce/garcon/language/uz@latn/))
uz@latn	(uzbek (latin) (http://www.transifex.com/heldercorreia/speedcrunch/language/uz@latn/))
uz@latn	(uzbek (latin) (http://www.transifex.com/ethereal/modem-manager-gui/language/uz@latn/))
uz@latn	(uzbek (latin) (http://www.transifex.com/xfce/thunar-plugins/language/uz@latn/))
uz@latn	(uzbek (latin) (http://www.transifex.com/calibre/calibre/language/uz@latn/))
uz@latn	(uzbek (latin) (http://www.transifex.com/morevnaproject/synfig/language/uz%40latn/))
uz@latn	(uzbek (latin) (http://www.transifex.com/xfce/xfce-panel-plugins/language/uz@latn/))
uz@latn	(uzbek (latin) (http://www.transifex.com/uget/uget/language/uz@latn/))
wba	(warao)
wba	(wba)
zam	()
zam	(espanol ll@li.org)
zh_cn.gb2312	(chinese (china) (gb2312) (http://www.transifex.com/projects/p/fedora/language/zh_cn.gb2312/))
zh_cn.gb2312	(chinese (china) (gb2312) (http://www.transifex.com/projects/p/python-meh/language/zh_cn.gb2312/))
zh_cn.gb2312	(chinese (china) (gb2312) (http://www.transifex.com/projects/p/initial-setup/language/zh_cn.gb2312/))
zh_cn.gb2312	(chinese (china) (gb2312) (http://www.transifex.com/projects/p/certmonger/language/zh_cn.gb2312/))
zh_cn.gb2312	(chinese (china) (gb2312) (http://www.transifex.com/bgk/vba-m/language/zh_cn.gb2312/))
zh_tw.big5	(chinese (taiwan) (big5) (http://www.transifex.com/projects/p/synfig/language/zh_tw.big5/))
zh_tw.big5	(zh_tw.big5 ll@li.org)
zh_tw.big5	(chinese (taiwan) (big5) (http://www.transifex.com/projects/p/fedora/language/zh_tw.big5/))
zh_tw.big5	(chinese (taiwan) (big5) (http://www.transifex.com/projects/p/python-meh/language/zh_tw.big5/))
zh_tw.big5	(chinese (taiwan) (big5) (http://www.transifex.com/projects/p/initial-setup/language/zh_tw.big5/))
zh_tw.big5	(chinese (taiwan) (big5) (http://www.transifex.com/projects/p/certmonger/language/zh_tw.big5/))
zh_tw.big5	(chinese (taiwan) (big5) (http://www.transifex.com/bgk/vba-m/language/zh_tw.big5/))

[nijel]

The charset specific codes are out of scope here (and they are most likely outdated these days anyway). The others might be added, though for many of them it's hard to find reliable source of plural equations...

[Jibec]

would you mind making categories? I don't know anything about charset and what may be obsolete. I'll help for what I can, missing plurals (by reaching our community) and upstream changes (to remove/replace obsolete content). I'm unsure all of them requires support, I'll give you the number of occurences of each language tomorrow, so that we can focus on most important ones. But as language enthusiasts, even minor language deserves existance. Challenge never is for languages with huge community. There is no rush in this, we can move to something else than a github issue if you think it may be easier. -- Jean-Baptiste

[fitojb]

Hyperspecific things like “ast_ES” and “gug_PY” are ridiculous. Asturian is only spoken in Spain, and gug already refers to Paraguayan Guarani without needing the country code.

[nijel]

I don't know anything about charset and what may be obsolete.

I thought that everything using UTF-8 these days, but I might be wrong. Anyway the syntax is code.CHARSET so you should strip the character set before looking up the code.

I see no problem in adding all language codes defined by ISO (we do this for the two-letter codes for now only). The only problematic thing is to get plural definition for these.

Besides these three-letter codes, I think I've covered nearly all missing variants in the list.

[Jibec]

I was summarizing this discussion and @nijel kindly added new commits, so here are some statistics, if it may help.
Next time you make a release, I'll rerun this.

Language code	number of files (in Fedora 32 packages)
aa	95
aa_dj	2
ab	164
ace	3
ach	41
adh	2
ae	95
af	693
agr	7
ak	121
aln	6
am	477
an	298
ang	16
ang@latin	29
anp	18
ar	1088
ar_dz	2
ar_eg	26
ar_ly	1
ar_ma	1
ar_sa	7
arn	2
as	515
ast	816
ast_es	1
av	95
ay	120
ayc	46
az	470
az_ir	12
ba	94
bal	22
bar	1
be	721
be@tarask	3
be_latn	127
bem	3
ber	3
bg	1248
bh	95
bi	116
bm	96
bn	570
bn_bd	29
bn_in	462
bo	273
br	466
brx	97
bs	738
btk	2
byn	4
ca	1829
ca@valencia	528
ce	104
ceb	5
cgg	3
ch	94
chr	1
ckb	145
ckb_ir	1
co	98
cpp	5
cr	93
crh	127
cs	2217
csb	42
cu	93
cv	114
cy	466
da	1679
de	3042
de@hebrew	1
de_at	7
de_ch	39
doi	3
dsb	78
dv	97
dz	448
ee	94
el	1561
en	199
en@arabic	1
en@boldquot	24
en@cyrillic	1
en@greek	1
en@hebrew	1
en@piglatin	1
en@quot	26
en@shaw	93
en_au	296
en_br	1
en_ca	426
en_cz	2
en_gb	1176
en_ie	1
en_nz	2
en_us	175
en_za	71
eo	997
error	4125
es	2439
es_419	13
es_ar	126
es_cl	58
es_co	68
es_cr	31
es_do	30
es_ec	74
es_gt	1
es_hn	1
es_mx	120
es_ni	30
es_pa	30
es_pe	32
es_pr	45
es_py	2
es_sv	31
es_us	4
es_uy	30
es_ve	37
et	944
eu	1135
fa	737
fa_af	34
ff	122
fi	1575
fil	78
fj	95
fo	151
fr	2785
fr_ca	162
fr_ch	1
frp	62
fur	322
fy	327
ga	749
gd	464
german	1
gez	4
gl	1310
gn	147
gom	2
gom@latin	2
grc	29
gu	602
gu_in	18
guc	2
gug	32
gug_py	2
gun	2
gv	106
ha	175
haw	3
hch	2
he	1100
he_il	40
hi	752
hil	1
hne	1
ho	93
hr	1096
hrx	2
hsb	85
ht	137
hu	1702
hus	32
hy	382
hz	93
ia	365
id	1455
ie	210
ig	217
ii	93
ik	93
ilo	20
io	114
is	720
it	2297
iu	98
ja	2012
ja_hira	1
jam	1
jbo	17
jv	255
ka	508
kab	174
kde	17
kg	109
ki	94
kj	93
kk	616
kl	166
km	457
kn	576
ko	1379
kok	74
kok@latin	1
kos	10
kr	94
krl	27
ks	200
ks@aran	1
ks@deva	1
ks@devanagari	1
ku	427
ku_iq	56
kv	94
kw	136
kw@kkcor	15
kw@uccor	15
ky	408
la	228
lb	204
lg	131
li	172
ln	104
lo	272
lt	1226
ltg	4
lu	93
lv	946
mai	328
mfe	2
mg	277
mh	95
mhr	10
mi	218
miq	3
mjw	49
mk	586
ml	744
mn	456
mni	72
mni@beng	1
mnk	3
mr	707
ms	774
mt	176
mvo	6
my	278
my_mm	1
na	113
nah	37
nan	2
nap	6
nb_no	1375
nd	93
nds	315
ne	583
ng	93
nhn	1
niu	2
nl	1837
nl_be	11
nn	848
nqo	9
nr	163
nso	190
nv	94
ny	93
oc	737
oj	97
om	167
or	511
os	96
pa	778
pam	1
pap	43
pbs	36
pi	94
pis	1
pl	2558
pms	35
pr	1
ps	284
pt	1467
pt_br	2135
pt_pt	226
qu	95
quy	6
quz	36
racv	6
rm	103
rn	93
ro	1253
ro_md	102
ru	2523
ru_ru.cp1251	2
ru_ru.koi8r	1
ru_ru.utf8	1
ru_ua	2
rue	95
rw	499
sa	104
sah	71
sat	71
sat@deva	1
sc	151
sco	9
sd	196
sd@deva	1
se	113
sg	93
shn	11
si	576
sk	1333
sl	1301
sm	115
sma	102
sms	1
sn	94
so	101
son	27
sq	791
sr	1728
sr@ije	19
sr@ijekavian	5
sr@ijekavianlatin	5
sr@latn	1
sr_cyrl	5
sr_latn	531
ss	149
st	179
su	98
sv	2231
sw	171
sw_tz	1
szl	174
ta	734
te	705
tg	414
th	756
ti	174
tig	4
tk	179
tl	183
tlh	5
tlh_aa	1
tn	162
to	113
tr	1695
ts	158
tt	237
tt@iqtelif	4
tvll	1
tw	112
ty	93
tyv	5
tzm	24
tzo	4
ug	608
uk	1846
ur	449
ur_pk	68
uz	322
uz@cyrillic	37
uz@cyrl	2
uz@latn	14
ve	172
vec	81
vi	1201
vo	93
wa	347
wae	6
wal	2
wba	15
wo	112
xh	362
yi	159
yo	207
za	94
zam	2
zh_cn.gb2312	5
zh_hans	2277
zh_hans_sg	1
zh_hant	1565
zh_hant_hk	662
zh_tw.big5	8
zu	374

[Jibec]

Please ask me the list of plurals you need. I'll share with you what I can find in Fedora packages.

[nijel]

Probably for all missing three letter locales. These are typically not present in CLDR and do not have defaults Gettext.

[Jibec]

that was the question I was going to ask, how do we know if what we are doing here isn't a duplicate of what other groups does, like cldr. Do you sync with their releases? I also feel like may join efforts with this project: https://github.com/mike-fabian/langtable would it make sense? -- Jean-Baptiste

[Jibec]

Here is a new list of suggestion actions, with details as an attachment (I had to rename it in txt for github):
error-analyze.json.txt

This attachment is a json file.
First level is "Language" metadata.

As sublevel:

• Count is the number of occurences of this language.
• Plural-Forms are the list of plurals found in po files (unique values only)
• Language Team are the list of teams found in po files (unique values only)

Here is an example for Filipino from Philippine (my assumption here is that fil_PH is an alias for fil).

 "fil_PH": {
    "Count": 2,
    "Files": [
      "./results/f32/packages/bibletime/i18n/howto/howto-/howto-fil_PH.po",
      "./results/f32/packages/bibletime/i18n/handbook/handbook-/handbook-fil_PH.po"
    ],
    "Plural-Forms": [
      "nplurals=2; plural=(n == 1 || n==2 || n==3) || (n % 10 != 4 || n % 10 != 6 || n % 10 != 9);"
    ],
    "Language-Team": [
      "Filipino (Philippines) (http://www.transifex.com/bibletime/bibletime/language/fil_PH/)"
    ]
  },

[Jibec]

In order to help you with plurals, here is a folder with one json file per language: https://jibecfed.fedorapeople.org/partage/languages-data/

If you see multiple first level keys, it means the po-files aggregated have different values.

Example with Asturian: https://jibecfed.fedorapeople.org/partage/languages-data/ast.json

• we can find aliases Asturian, ast_ES, templates, zzz_empty (no value Language metadata), zzz_null (no Language metadata). My algorithm probably took the filename to confirm the correct language code is "ast".

Example with Guarani: https://jibecfed.fedorapeople.org/partage/languages-data/gug.json

• you can see gug_PY is considered as an alias