[Bug] owntracks recorder integration limited to 10000 points/month for historical data

[wjansenw]

Describe the bug
When loading historical data from owntracks recorder, it is hardcoded to only fetch 10000 data points. I have several months where this is not sufficient, so I assume it is not loading all data - as I do not see there is a retry for after the first 10000 points of each month.

To Reproduce
Load historical owntracks recorder data:

Logs say:
Fetching location data from: http://192.168.0.xxx:8083/api/0/locations?user=wim&device=gsm-wim&from=2025-04-01T00:00:00&to=2025-04-30T23:59:59&limit=10000

Where the month has more data points:

count | 16248
data | (16248)[ {…}, {…}, {…}, {…}, {…}, {…}, {…}, {…}, {…}, {…}, … ]
status | 200
version | "1.0.1"

Expected behavior
Not sure if owntracks sorts the data somehow, but you could fetch the next 10000 points starting with the last datapoint received. Or remove the limit - but that might trigger other issues?

[wjansenw]

When reading owntracks information, it seems limit sorts the data from most recent, going backwards; so you would have to set the end-date of the fetch url to the oldest already received information from that month.
Or change the logic, by removing the current month-approach, and start from the last received 10000 records, and work your way back to the first recording.

[dgraf-gh]

Hi @wjansenw,

thank you for the report and the analysis. Yes, I thought reitti does it exactly how you described it. At least that was in my memory but somehow I also seem to missed it implementing it like that.

Will be fixed in the next version.

[wjansenw]

Tnx @dgraf-gh. Would a reload mean duplication of points? Or does reitti know which points have already been processed, and skip them? Just to know if I need to stop the current processing, or if I can keep the data. Just started to use reitti, so not much information around places etc. has been processed.

[dgraf-gh]

You do not have to stop the current process. Reitti will ignore duplicate points on the second load

[wjansenw]

Note: my use case might not be how reitti is intended to be used...

I can't run reitti on my odroid as it seems to be too demanding (it runs several other services as well), so I (for now) run it on my laptop, which is not always on.
When I launched reitti this morning, it only captured data from yesterday from owntracks, and not from Saturday.

While this is not the same as fetching all historic data, I noticed there is a similar limitation to only capture the 500 last points from owntracks. Ideally, this is also changed to capture all points since the last successful fetch.

[dgraf-gh]

I see, I will also adjust the normal fetching of the data.