Problems in Data Conversion with Excel of Otoole Examples

[HanHuiHH]

$ otoole convert excel datafile simplicity.xlsx simplicity.txt config.yaml
OtooleIndexError: TechnologyFromStorage -> Indices inconsistent between config and data. Config indices are ['REGION', 'TECHNOLOGY', 'STORAGE', 'MODE_OF_OPERATION']. Data indices are ['REGION', 'TECHNOLOGY', 'STORAGE', 'YEAR'].

After I change 'MODE_OF_OPERATION' to 'YEAR' in config.yaml, the following steps of this section seems work, but there comes another problem in Model Visualization:

$otoole viz res excel simplicity.xlsx res.png config.yaml
TypeError: argument of type 'float' is not iterable

So the res.png was not created in this folder.
I'm new to OSeMOSYS, so I don't know how to solve this problem.

[trevorb1]

Hi @Hansen-hanhui, thanks so much for flagging this issue!

Quick question, are you able to check your otoole version, please? You can do that through running the command otoole -V from the command line.

If you are running otoole version 1.0.2, you may be experiencing the bug described in this issue ticket. If this is the case, can you please update your otoole version to 1.0.3 and try again? To update you otoole version, you should be able to just run the command pip install otoole==1.0.3.

After doing this, can you also report back on this issue ticket please. That way we can either close the issue of figure out what is going on if you are still getting errors. Thanks again! :)

[HanHuiHH]

Hi @trevorb1 , thanks for your quick response. the former problem is fixed after I upgrade to 1.0.3 from 1.0.2. However the latter problem still exist, maybe caused by another small problem in config.yaml.

Thanks again for your detailed help! To save your valuable time, it's ok to reply briefly, and I will keep my attention on this project.

[trevorb1]

Great, happy to hear the first issue is solved @Hansen-hanhui!

Unfortunately, I wasn't able to recreate your second issue. Below are the commands I ran and everything worked for me; I used otoole v1.0.3. Are you using the config file you updated, or the reference one found in the repository? If the issue persists, could you please run otoole in verbose mode (ie. otoole -v viz res ...) and copy over the traceback error?

git clone https://github.com/OSeMOSYS/simplicity.git
cd simplicity
otoole convert csv excel data data.xlsx config.yaml
otoole convert csv datafile data data.txt config.yaml
otoole viz res csv data res-csv.png config.yaml
otoole viz res excel data.xlsx res-xlsx.png config.yaml
otoole viz res datafile data.txt res-txt.png config.yaml

Thanks!

[HanHuiHH]

C:\Users\Administrator>git clone https://github.com/OSeMOSYS/simplicity.git
Cloning into 'simplicity'...
remote: Enumerating objects: 189, done.
remote: Counting objects: 100% (23/23), done.
remote: Compressing objects: 100% (16/16), done.
remote: Total 189 (delta 5), reused 18 (delta 3), pack-reused 166
Receiving objects: 100% (189/189), 59.11 KiB | 517.00 KiB/s, done.
Resolving deltas: 100% (73/73), done.

C:\Users\Administrator>cd simplicity

C:\Users\Administrator\simplicity>otoole convert csv excel data data.xlsx config.yaml

C:\Users\Administrator\simplicity>otoole convert csv datafile data data.txt config.yaml

C:\Users\Administrator\simplicity>otoole viz res csv data res-csv.png config.yaml
TypeError: argument of type 'float' is not iterable

C:\Users\Administrator\simplicity>otoole viz res excel data.xlsx res-xlsx.png config.yaml
TypeError: argument of type 'float' is not iterable

C:\Users\Administrator\simplicity>otoole viz res datafile data.txt res-txt.png config.yaml
TypeError: argument of type 'float' is not iterable

C:\Users\Administrator\simplicity>otoole -V
1.0.3

I tried it in cmd of windows11, maybe the problem was caused by not using linux system?

I am just famillar with python language, and I will further explore the python code of otoole so that I can use pycharm to debug.

Thanks!

[trevorb1]

Hi @Hansen-hanhui, apologies for the delay! Good point on the running of Windows vs. Linux. I am running on Linux via WSL on a Windows machine. If you need an immediate solution, I would suggest that! Else, hopefully I will have time soon or so to look into this issue or shift it to a new issue on the otoole repository.

[HanHuiHH]

Thank you @trevorb1 , I have tried WSL on Windows machine, and ALL CODE WORKS! That's so great!

Maybe I should try to learn more about LINUX as it provide more friendly environment than windows cmd.

Thanks again for your help! :)

[HanHuiHH]

There are another problem in Model Validation 6. otoole validate. I just copied the code in this section to validate.yaml and there came an error:

root@A3K0GQFTQ2V5QX7:/mnt/c/Users/Administrator/simplicity# otoole validate datafile data.txt config.yaml --validate_config validate.yaml
ScannerError: mapping values are not allowed here
in "validate.yaml", line 18, column 12

Then I checked the difference between otoole/src/otoole/validate.yaml, and found that there are several indent error in Schema. After I corrected the indent errors in this file, the code worked with the result:

Beginning validation

Validating FUEL with fuel_name

17 invalid names:
BAGASSE, DSL, ETH, FEL1, FEL2, FER, GAS, LAND, LANDFOREST, LNDSUGCAN, MOLASSES, RAWSUG, RIVERWATER, SEC_EL, SUGCAN, WATAGR, WATIN

Validating TECHNOLOGY with technology_name

26 invalid names:
BACKSTOP1, BACKSTOP2, ETHPLANT, GAS_EXTRACTION, GAS_IMPORT, GRID_EXP, HYD1, HYD2, IMPDSL, IMPFERT, IMPRAWSUG, LNDFORCOV, LNDRES, LNDSUGPL, LNDSUGPLIR, LNDSUGPLRF, NGCC, RIVER, RIVER_2, RIVWATAGR, SOLPV1, SOLPV2, SUGFACTORY, TD, TD2, WINDPOWER

Checking graph structure

1 'technology' nodes are isolated:
TD2

I believe that I have fixed this problem, and there are no other problems in Examples. :)