Using new dataset

[lovelyyunamars]

Dear Dr. Nguyen

Thank you for your awesome work!

Right now, I can get results based on the data you provided. However, when I use my own dataset, there are some problems.

First of all, I listed all the setups I made:
Lat_min=50.6, Lat_max=54.6, Lon_min=-180.0, Lon_max=-174.0
Lat_bins=400, Lon_bins=600
the data I used is from 2017 Jan 1st to Jan 31th, 10 days for training, 10 days for valid, 11 days for tset.

Then I run cvs2pkl, dataset_preprocessing, calculate_AIS_mean, I got the data file similar with the data you provided, but I did not use the ''Loading coastline polygon'' function in the dataset_preprocessing.py

then I start to train the model, it works, but I got two files under the chkpt path with different data_dim, one is 602, one is 1102, I believe 1102 should me mine dataset.

then I run save_logprob, it did not creat errors too

However when I run local_logprob, I can not get any result, even that data/ct_2017010203_10_20/local_logprob-ct_2017010203_10_20_train.pkl-ct_2017010203_10_20_valid.pkl-100-missing_data-False-step-80002 and other results.

So I want to know, why this error happened? is that because I used not enough data?

I will be so grateful if you can reply!

Regards

Yu

[mariamartinmarin]

Incredible work, Dr. Nguyen,

I am also having some problems. I don't know if it could be because of the format of my dataset.

Lovelyyunamars, could you tell me the original format of your dataset? I have downloaded several datasets from MarineCadastre, but when I pass it csv2pkl, it does not work well, because some fields are missing that are not in the csv file provided by MarineCadastre.

Thank you very much for everything!

[lovelyyunamars]

Incredible work, Dr. Nguyen,

I am also having some problems. I don't know if it could be because of the format of my dataset.

Lovelyyunamars, could you tell me the original format of your dataset? I have downloaded several datasets from MarineCadastre, but when I pass it csv2pkl, it does not work well, because some fields are missing that are not in the csv file provided by MarineCadastre.

Thank you very much for everything!
Hi, Maria

I think you should adjust the code line 170-176 in csv2pkl.py according to your original csv file.

[duonginspace]

Hello,

Sorry for the late reply, I'm currently not working on this project.

@lovelyyunamars there is a bug in this version, the sizes of the 4-hot vectors are hard-coded in datasets. You should manually modify those values. I'm sorry for this bug. I'll update a newer version when I have time.

Can you successfully run the code on the provided dataset?

@mariamartinmarin You dont need all the attributes in the *.csv files (we use HEADING, ROT, NAV, etc. for other applications). GeoTrackNet requires only LAT, LON, COG, SOG, MMSI, and TIMESTAMP.
If you want to apply GeoTrackNet on your own datasets, you can re-write the preprocessing part (highly recommend, the preprocessing code of this version is very bad).

[lovelyyunamars]

Hey, Duong,  Thank you for reply, I sloved the problems already, but I am wondering all the parameters in datasets_prepcocessing.py, how to choose all these values like 20 length, 4hours, and 5 mins according to new datasets?

…

------------------ 原始邮件 ------------------ 发件人: "CIA-Oceanix/GeoTrackNet" ***@***.***>; 发送时间: 2021年4月1日(星期四) 晚上6:41 ***@***.***>; ***@***.******@***.***>; 主题: Re: [CIA-Oceanix/GeoTrackNet] Using new dataset (#16) Hello, Sorry for the late reply, I'm currently not working on this project. @lovelyyunamars there is a bug in this version, the sizes of the 4-hot vectors are hard-coded in datasets. You should manually modify those values. I'm sorry for this bug. I'll update a newer version when I have time. Can you successfully run the code on the provided dataset? @mariamartinmarin You dont need all the attributes in the *.csv files (we use HEADING, ROT, NAV, etc. for other applications). GeoTrackNet requires only LAT, LON, COG, SOG, MMSI, and TIMESTAMP. If you want to apply GeoTrackNet on your own datasets, you can re-write the preprocessing part (highly recommend, the preprocessing code of this version is very bad). — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub, or unsubscribe.

[mariamartinmarin]

@lovelyyunamars Thank you for your answer.

How could you do the training? When I try to do the training, I get the following error:

An error occurred while training the embedding layer：ValueError：Dimensions must be equal,but are 702 and 602 for 'sub' with input shapes:[?,??,702],[1,1,602].

It seems that I have to make some changes in the dataset.py file, but I don't really know what to do. Can you help me, please?

Thank you very much!

[lovelyyunamars]

this error means you did not set the correct bins value, you should calculate your own bins and change them in every.py which has bins. 发自我的iPhone

…

------------------ Original ------------------ From: María Martín ***@***.***> Date: Thu,Apr 29,2021 0:55 AM To: CIA-Oceanix/GeoTrackNet ***@***.***> Cc: lovelyyunamars ***@***.***>, Mention ***@***.***> Subject: Re: [CIA-Oceanix/GeoTrackNet] Using new dataset (#16) @lovelyyunamars Thank you for your answer. How could you do the training? When I try to do the training, I get the following error: An error occurred while training the embedding layer：ValueError：Dimensions must be equal,but are 702 and 602 for 'sub' with input shapes:[?,??,702],[1,1,602]. It seems that I have to make some changes in the dataset.py file, but I don't really know what to do. Can you help me, please? Thank you very much! — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub, or unsubscribe.

[mariamartinmarin]

Thank you so much @lovelyyunamars !!!

I was able to fix it by modifying some files!
One question I have now (maybe I haven't done enough research) is how to apply the generated model to perform detections. could you help me?

Thank you very much for your help.

[hariiiseldon]

@mariamartinmarin Hello，I have the same error when I run the GeoTrackNet.py:
An error occurred while training the embedding layer：ValueError：Dimensions must be equal,but are 702 and 602 for 'sub' with input shapes:[?,??,702],[1,1,602].
I would appreciate it if you could tell me how to modify the files.
Thank you so much for your help!

[yche-sflscientific]

@hariiiseldon hope it's still relevant. I had the same issue and succeeded getting pass the mismatched dimension error.
My understanding is that the 602 is from the where you set using the calculate_AIS_mean.py, and the 702 is from where you input when calling geotracknet.py, specifically calculated from lat_min/lat_max and lon_min/lon_max, divide the range by 0.01 deg, should be your lat_bins and lon_bins set in the dataset.py and calculate_AIS_mean.py. Good luck!