Extend PoePy and SinoPy to offer basic access to important operations

[LinguList]

Often, this access is already available, but needs to be checked, etc.

• convert short characters to long characters (sinopy)
• identify what IS a Chinese character (sinopy)
• make network from CLDF data (poepy)
• parse rhyant Basic (raw) Formats for Unannotated Rhyme Collections #19 format (poepy)
• provide Middle Chinese readings (sinopy or another dataset that we link to (!))

@nh36 and @ashhenson, I am inclined now to not use sinopy to check for baxter sagart data, etc. but rather make individual CLDF datasets and link to them. We could still discuss to have something like a "generic" access to Guangyun with MC readings, fanqie, etc., but OC reconstructions should anyway be done text-wise.

[nh36]

@LinguList, if there is something specific that Ash can help with here can you please spell it out. Also, (perhaps as a separate issue), for our project I think we should add Schuessler's Han reconstructions in to SinoPy (or CLDF). We have the data for this.

[LinguList]

Yes, I will do. What is crucial is that we start with @ashhenson to make acquaintance with test-based coding (which I often ignored but want to take seriously now), and see how contributions can be done (e.g., writing a CLDF parser for rhyme words to create networks and making a pull request to poepy, etc.). But our start is to make a couple more rhyme datasets now in CLDF (Weingarten, Wang Li 1980, etc.). Then, we will see what repeats in terms of code and can start to integrate more.

[LinguList]

@nh36, @ashhenson, I invited you to a new organization that stores reconstructions etc., and will complement hanproj, where we collect reconstructions and other data related to yinyunxue, I made an initial repo for Schuessler's ABC dictionary: https://github.com/yinyunxue/schuesslerhanchinese

[LinguList]

Schuessler's Han Chinese is now available as a wordlist, but we need to edit the orthography profile.

[ashhenson]

Hey Mattis, I have reconstruction data for 李方桂、王力、高本漢、鄭張尚芳、白一平1992、潘悟雲、董同龢、周法高, plus Baxter & Sagart 2014. If you need any of these, just let me know. A few questions/comments: "convert short characters to long characters" is this referring to Unicode code points? like U+738B vs. U+0738B or maybe long = utf8 and short = utf16? or regular = short, extension characters = long? "identify what IS a Chinese character" I have my own code that does this, but it's in Python 2. I'd have to convert it to Python 3. Ash

…

On Fri, 1 Apr 2022 at 18:15, Johann-Mattis List ***@***.***> wrote: For reconstructions, we may also think of a new collection, namely a dictionary collection (also in CLDF) that is much more suitable for our purpose, I think / hope. I created and invited you all to this new organization "yinyunxue". — Reply to this email directly, view it on GitHub <#21 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AYII6PTCOKCHWCC5AO6BDL3VC4VKVANCNFSM5SH25PKA> . You are receiving this because you were mentioned.Message ID: ***@***.***>

[LinguList]

I have reconstruction data for 李方桂、王力、高本漢、鄭張尚芳、白一平1992、潘悟雲、董同龢、周法高, plus
Baxter & Sagart 2014. If you need any of these, just let me know.

Nice. Then, you can add repositories and put data into the raw/ folder. But ideally the original (Baxter-Sagart would be their file), or some documentation where it comes from.

"convert short characters to long characters" is this referring to Unicode
code points? like U+738B vs. U+0738B or maybe long = utf8 and short =
utf16? or regular = short, extension characters = long?

I thought about long/short characters in Chinese script (gbk,. etc.), and this should be done when converting to CLDF, using long characters (big5) in CLDF.

"identify what IS a Chinese character" I have my own code that does this,
but it's in Python 2. I'd have to convert it to Python 3.

No need, as it is in sinopy, and it also works, so we can just use it to convert to CLDF then. Sinopy is on Pypi, so it qualifies as a normal dependency.

[ashhenson]

Hey Mattis, Ah! I forgot about long/short. You mean traditional vs. simplified, right? We aren't really going to use gbk and big5, are we? UTF8 is automatically cross-platform (i.e., is immune to being eaten by changes in endianness). Shouldn't we stick with that? re: Then, you can add repositories and put data into the raw/ folder. What format do you want the data in? All in one file? Each with their own file? {char}\t{OC reconstruction}\t{fanqie pair} Something like that? Ash

…

On Sat, 2 Apr 2022 at 16:13, Johann-Mattis List ***@***.***> wrote: I have reconstruction data for 李方桂、王力、高本漢、鄭張尚芳、白一平1992、潘悟雲、董同龢、周法高, plus Baxter & Sagart 2014. If you need any of these, just let me know. Nice. Then, you can add repositories and put data into the raw/ folder. But ideally the original (Baxter-Sagart would be their file), or some documentation where it comes from. "convert short characters to long characters" is this referring to Unicode code points? like U+738B vs. U+0738B or maybe long = utf8 and short = utf16? or regular = short, extension characters = long? I thought about long/short characters in Chinese script (gbk,. etc.), and this should be done when converting to CLDF, using long characters (big5) in CLDF. "identify what IS a Chinese character" I have my own code that does this, but it's in Python 2. I'd have to convert it to Python 3. No need, as it is in sinopy, and it also works, so we can just use it to convert to CLDF then. Sinopy is on Pypi, so it qualifies as a normal dependency. — Reply to this email directly, view it on GitHub <#21 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AYII6PQWGZCTPIF6OUWQYULVDBPYZANCNFSM5SH25PKA> . You are receiving this because you were mentioned.Message ID: ***@***.***>

[nh36]

We use UTF8. Mattis, I think means simplified and traditional. We use traditional as strong default.

Typically into the raw folder you would put in the data in whatever format you currently have it.

[LinguList]

Yes, and typically, although all is in Unicode, people call long "BIG5" and short "GBK", so there is a function in SinoPy that does this conversion of a short to along character and vice versa. My strong suggestion is to use this function when data is in short characters. And of course we use utf8.

[LinguList]

What format do you want the data in? All in one file? Each with their own
file?
{char}\t{OC reconstruction}\t{fanqie pair}

Hmm. I want the data in which you received the file, ideally, the direct type-off of the resource, the book. If there is no resource and no book, I do not know if I want it. We have some pseudo-baxter readings from the project of colleagues in Shanghai, but they are not Baxter 1992 readings, so I'd say: if we talk about Baxter 1992, I want to have the version that is tight to the Shijing, which I could not type of when I digitized the rhymes in the past.

The trick is, @ashhenson, that in CLDF we use Python code to tweak the data into the format in which we need it. But we do this in a regular almost standardized way. So we deal with tables, but also parse texts, but what we definitely always want to, idealistically, is original sources that we can tie to a reference.

So more important than the format of the data is that I get from you a concrete story (you can add it to a file NOTES.md in the respective repository) where you inform me and future users, how the data were digitized.

[ashhenson]

Well, that data is mostly from scraping the now the defuct EastLing website (by 鄭張尚芳 or 潘悟雲) and 小學堂. 小學堂 is done by Academia Sinica.

[LinguList]

Then we did the same thing in the past, as I also have it and used it. So we should make an EASTLING directory, where we share that data, ideally just using data from our paper on vowel purity with Nathan, where we used it.

[ashhenson]

I've read through all the comments here and I'm summarizing the tasks, have a few questions, and a recommendation. Let me know what you guys think.

Task list:

• Make CLDF datasets for BnS 2014
• add Schuessler's Han reconstructions in to SinoPy (or CLDF)
• we start with @ashhenson to make acquaintance with test-based coding
• make a couple more rhyme datasets now in CLDF (Weingarten, Wang Li 1980, etc.).
• add OC data to EASTLING diretory
• add OC data to XIAOXUETANG directory
• edit the orthography profile for Schuessler's Han Chinese
• writing a CLDF parser for rhyme words to create networks and making a pull request to poepy

Question list:

• What does “but OC reconstructions should anyway be done text-wise.” mean? The part I'm not sure about is "text-wise"
• Re: " @ashhenson to make acquaintance with test-based coding". How do I do that exactly? Is there something I can read or tests that I can mimic?

Recommendation:
I recommend we don't use the following terms:
“short character” or “gbk” for Simplified Chinese. This is not normal English usage, and since we're doing open source stuff, we should use normal English terms.
Of course, “gbk” is appropriate when talking about the actual gbk encoding, but calling it gbk when actually using utf8 encoding is unnecessarily confusing.

Same for “long character” or “big5”

To make sure it's not just me, I googled “convert long characters to short characters”, and nothing Chinese related even came up. Compare this to googling “convert simplified to traditional” where all of the results are related to Chinese. When I googled “convert long Chinese characters to short characters,” then Chinese related results came up, but the terminology used was all “simplified vs. traditional,” i.e., none of the pages used short/long.

[LinguList]

What does “but OC reconstructions should anyway be done text-wise.” mean? The part I'm not sure about is "text-wise"

A Chinese character is ambiguous, it has multiple readings that manifest a certain FORM in a certain TEXT. Ambiguity can only be resolved in a text, where you have a 1-1 relation, in a dictionary, you have one character but n reconstructions (n often is of course 1, but anyway). This is crucial for modeling.

[LinguList]

Test-based coding means you accompany all code you write by tests (unittests, we use pytest for this task). This guarantees the code base remains stable if you change things, as the tests typically stay the same and are used to make sure the code yields identical results.