Bug: GSuite Activities Pagination Infinite Loop

[brianbeklaviyo]

GSuite Activities connector enters infinite pagination loop when API continuously returns nextPageToken values, causing the connector to never complete collection and preventing logs from being saved to output destinations.

Looking at the logic for the connector, this might be fixed by way of:

• Maximum pagination limits
• Duplicate token detection
• Circuit breakers for identical time ranges
• Progress tracking mechanisms

We have only noticed this with the token operation.

---

• We are running the latest main branch of the Grove repo
• Shared cache (DynamoDB)
• Deployed Grove with 1-2 workers on AWS ECS with CPU/Memory Utilization <50%
• logs output to AWS S3

[hcpadkins]

Hey @brianbeklaviyo!

Thanks for the report!

I think the cause of this is related to the windowing used for reverse chronological collection of logs when collecting data for busy tenants. Effectively, when a collection fails and Grove tries again to catch-up, it needs to find the spot it got to last before flushing new logs to the output - to avoid duplication and missed data.

If the collection failed due to something like the AWS STS token expiring - see #114 - then this will never actually be able to catch-up. I'm trying to understand if there's a better way of resolving this, and whether there is a shortcut we can take in these cases so we don't need to effectively recollect the same data to find our place, but this change should assist to some degree in the interim.

Sorry for this one, it's an annoying issue caused due to Grove trying to be cautious. I'll keep you updated, but I'm actively working on this currently :)

[hcpadkins]

Hey @brianbeklaviyo,

I think this should be resolved now, after the merging of #100. We've not seen any similar issue occur in our deployment since this was merged and deployed.

Are you still seeing this?

Thank you for reporting this one, and your assistance.