Reverse Chronological Pointers

[jshlbrd]

I'm trying to understand the behavior of pointers in reverse chronological connectors and struggling a bit. What I'm observing is that if the connector is a long-running fetch (assume the API retrieves 100 records for each call and the run results in 1000s of parts, as an example) then the window_end pointer is always updated with the latest pointer, but the pointer, next_pointer, and window_start pointers never change during the run.

Is this intentional? If the task fails then it starts over at the pointer again, which in this example would be 100,000 events in the past. Is there a way to force the pointers to change so that if the run fails it picks up in the middle of where it was?

[jshlbrd]

An example might help -- assume the API uses cursor-based pagination and returns the most recent events first, something like this:

{"id": "ccc", "time": "2025-06-19T01:00:00"}
{"id": "bbb", "time": "2025-06-19T00:00:00"}
{"id": "aaa", "time": "2025-06-18T23:00:00"}

The cursor is the id value and it's paginated by searching for values that occur after that value. The results are reverse chronological, but should the connector use the reverse chronological log order? Based on the Grove source code it looks like the chronological order works in this case because the first id value in the response can become the pointer (i.e. on the next run, fetch all events that occur after ccc, such as ddd).

edit: I might have answered my own question -- the _save_chronological function specifically pulls the pointer from the last entry in the list. Is there a way to override the pointer value in a run, or is the simplest solution here to reverse the results before saving them?

[hcpadkins]

Hey @jshlbrd!

Thanks for the report. You're correct around the difference in operation between reverse chronological and chronological. We prefer and recommend passing parameters to a given API to ensure results are in chronological order as it greatly reduces the amount of work Grove needs to do - if the API supports it. Unfortunately, some APIs do not support ordering, which is why the mode exists.

Reverse chronological is expensive as we need to try and ensure we don't drop records if a collection fails. To do this, we "window" over pages and keep track of what records we've seen and flushed / saved as a range so that if the collection fails we don't just take the save the pointer from the record returned by the API (the latest value) and then miss the pages which weren't collected due to this failure.

In this mode the next_pointer will be the value we want to use once we have collected all possible records across all pages, and we shouldn't see the window_start change as you observe. Only the window_end should keep incrementing after each page is flushed / saved.

In the case of a failed collection, it's important to note that although we use the pointer value to start collection we don't actually flush / save records until we see the window_end value - to try and avoid duplication of records already collected.

This said, to mitigate the sort of issues experienced in #95 I'm looking to move this logic to instead use the window_end value as the pointer when picking back up from an incomplete collection. However, I need to do some additional testing and get some test cases in to ensure we don't cause an issue during collection. It's been a minute since I've touched these code paths, so I want to make sure it doesn't go 💥 .

After these changes it should result in reduced collection times and less API calls after a failure when using reverse chronological mode.

I'll leave this one open, and keep it updated as I work through this.

[jshlbrd]

#95 has an issue that I experienced in a connector that I built, hence this issue. :)

For new connectors, can you think of a reason why reversing the list of events prior to calling save() wouldn't work? It appears to be fine for the connector I'm referring to above.

[hcpadkins]

Hey @jshlbrd,

If I'm understanding the flow correctly, that will work in cases where an API only ever returns a single page of results. If not, you would need to perform a full collection into memory before reversing and then saving to ensure the pointer recorded is correct and that no data is lost.

The windowing was implemented to handle this case as if looping until all data is retrieved from the API and then calling save, Grove may never complete or run out of memory.

That said, this is can currently still happen with high volume data sources due to the use of pointer over window_end when picking back up from where a connector left off. Moving this to window_end, as planned, should hopefully resolve this, while still guarding against lost data and high memory footprint.

That said, if you see any cases where this may not be the case or there is an alternate approach, please let me know! :)

[jshlbrd]

We might be describing two different behaviors, but let me know what you think--

There's the behavior of log order and the behavior of pagination. The log order can be reversed chronologically and the pagination chronological. I can't think of any examples of an API that has the reverse behavior, so let's ignore that for now.

If pagination is chronological (use a cursor to move forward in time or through an event stream), then reversing the list of entries should be valid because the most recent event will have its pointer selected from the list. The most recent pointer could be retrieved from each page of results.

Most API that I've seen have one of these two behaviors:

• Pagination and log order is chronological.
• Pagination is chronological and log order is reverse chronological.

I think what you're describing is reverse chronological pagination, but I'm not sure. 🤔

edit: btw I'm aware of the problem with the Google Workspace APIs that write events out of order, but I think that behavior is an edge case unless you've seen otherwise.

[hcpadkins]

Hey @jshlbrd,

Thanks for the quick reply! Let me think on this a little and get back to you :)