Change format of input control file to a standard format?

[ekluzek]

The control file for mizuRoute is in a custom format rather than an industry standard format. When industry standard formats are used (such as XML, JSON, YAML, INI-config, etc.) readers and validators can also be used to handle the file, to read, write, change, and verify syntax. This can be very useful.

In bringing mizuRoute into CESM, we may get pushback unless a standard format is used. I just want to start discussing this to hear how important the file format for the control file is. If there is flexibility in the mizuRoute community maybe we'll just pick on and convert. If there is a desire to keep the same format we can start preparing our arguments. We could also possibly use different formats for standalone vs cesm-coupled.

We don't need to decide on this anytime soon, but should be thinking about it longer term.

[ekluzek]

@nmizukami and I talked about this some yesterday. In terms of a format to pick we thought the INI config file format is preferred (this is the format used for the Externals.cfg file at the top level).

@martynpclark @ShervanGharari feel free to add your thoughts here. Thanks.

[nmizukami]

Personally, I am ok with INI-config or Yaml. These are also more modern? (than namelist - this is ok to me). The format like JSON requires opening and closing brackets and also quote for string, so looks cumbersome and more mistake-prone.

So like Erik said, this is not urgent, and for now current control file works just fine with cesm coupling.

[andywood]

Hi Erik,
Good topic - it's worth making good choices since the control file is often the most handled in workflows that use a model.
I think some requirements are to be: commentable (rules out JSON), easily human readable/editable and concise/non-cluttered (rules out XML), machine-readable/writable w/ common open source libraries & code (eg python), keyword rather than order-controlled, and also ideally handles array and structured/hierarchical entries. I'm not familiar with the INI format but it looks like it might be suitable and consistency with other CESM models also gets a heavy weight here. For our new PyGMET program we adopted a TOML format, but if INI looks better I'd be open to switching. -Andy

[ekluzek]

From wikipedia, it looks like INI config files are almost identical to TOML. But, TOML is standardized, so if we convert it would be the leading candidate.

Here's the quote from wikipedia...

"TOML's syntax primarily consists of key = value pairs, [section names], and # (for comments). TOML's syntax somewhat resembles that of .INI files, but it includes a formal specification, whereas the INI file format suffers from many competing variants."

[martynpclark]

Andy's framing of the problem is excellent. I do not have experience with TOML -- on first glace it looks really good.

[ShervanGharari]

TOML looks very good! One advantage that comes to mind for TOML in comparison with the existing control file is the modularity in reading variables or groups of variables. Flags can be sorted, and if a flag such as lake is True then it is easier to check if needed variables for lakes are given and populated as a subset of network topology or forcing. With a structured config file, it can be easier to have these checks in a centralized location. Just my opinion working with lake parameters (that are sometimes many).

[ekluzek]

I looked more into this for bringing it into CESM/cime, and one caveat is that in python the TOML library is a non-standard library you have to include in your environment. In cime we've tried to make python be limited to the python standard library. We could set it up so that TOML was only required if mizuRoute is used, but you would need to have a python environment that included it. Using conda that would be simple, but it is an extra step. On cheyenne you could use the npl environment, so it would be easy to setup. But, this is an additional concern.

I'm going to propose that we add TOML as an option to cime, so this is a bit of pushback that I do expect to get from that.

[andywood]

Hi Erik, thanks for looking deeper into this. I understand the desire to use a standard library stack (I have that desire myself in creating software). Perhaps this opens up the question of whether CESM/cime could have a defined 'extended library', a kind of supplemental stack that could evolve a bit ahead of the standard library. I don't know enough about the pros/cons to know if that would be radical or incremental to their plans. If TOML is a bridge too far, we can just go with INI. I like the extension, actually.

[andywood]

Also the fact that TOML is in NPL does make it easy to use for those on Cheyenne (nothing overly custom).

[ekluzek]

#646 proposes to use the FORTRAN library

https://github.com/toml-f/toml-f.git

to use in the Fortran code to handle reading in TOML format. This library has reasonable support, seems to be put together well, looks to have reasonable testing, has been around since 2020, and uses the Apache 2.0 license so could be adopted

Python has the toml and tomllib python modules as options which #646 also uses...

[ekluzek]

Talking with @johnpaulalex and @nmizukami we noted that TOML and TOML readers don't preserve comments. But, a common workaround is to do something like this:

_comment_host = "The IP address or domain name of the database server"
host = "192.168.1.50"

And then in our handling we ignore the "comment" variables, but still have them written out in the files and preserve them for the reader to use.