Bug: Concurrency/Cache issues when utilizing multiple GSuite connectors

[brianbeklaviyo]

Issue: Frequent Cache set failed as constraint failed and The cached lock does not match ours errors related to pipelines which use the gsuite_activities connectors. Most notably those where operation is in drive,calendar,token. This results in no new logs being acquired - likely due to the cache never having been updated.

Context:

• Deployed containerized via AWS ECS
• Configuration files stores in AWS SSM Parameter Store
• Shared cache/pointer using AWS DynamoDB
• Configuration files exist with gsuite_activities as the connector and each one of the possible operation values from the list of available options
• This issue appears to have occurred when we implemented autoscaling and added several config objects

EDIT: Rather than assuming this is a cache issue I would like to note that based on logs, we can see that:

1. We have several logs indicating pagination is occurring (Using pageToken as a cursor...)
2. This is followed by a log indicating pagination has ended (Got activities from the GSuite API)
3. There may be some issue in the save method, because the connector exits after this line and the logs do not make it to S3.
4. This has been noted with GSuite operations: drive, gcp, token, calendar, chrome, context_aware_access.
5. This began occurring upon introducing several new configuration objects for GSuite activities of various operation types
6. I can find instances of this chain of events occurring without any error logs referring to The cached lock does not match ours.

[brianbeklaviyo]

Adding to this ticket, I can confirm that the cache/pointers for these connectors appear to be progressing with each run despite the logs not being saved.

[brianbeklaviyo]

Temporary fixes that work for us:

• disable the collection via config ("disabled":true)
• clear the dynamodb cache entries for that given operation
• enable collection again

[hcpadkins]

Hey @brianbeklaviyo!

Thank you for the report. If I'm understanding what you're observing correctly, this is expected behaviour.

Originally Grove was never designed to run the same connectors concurrently in multiple locations, as this would require some form robust locking implementation to ensure that logs are not collected twice by different instances of Grove.

However, as it was found to be common for Grove to be accidentally scheduled or executed twice - especially in cloud environments - we added locking to try and avoid duplicates and potential caching issues when this occurred.

This works by both attempting to get a lock, but also failing collection if the values in the cache have unexpectedly changed during a run - which would indicate something else is updating the cache. The latter of which appears to be what you're observing.

This said, given the changes in Grove recently related to locking, as well as new cache backed last run tracking, we may be able to extend this implementation to allow for Grove to properly support concurrent execution, but this would definitely require a bit of design work to ensure we're not getting ourselves into trouble.

[brianbeklaviyo]

@hcpadkins Thank you for the prompt response!

Is there a more practical approach for deploying Grove in this case, perhaps more in the spirit of how it was designed? Between the cache locking mechanism and templates for deploying with ECS, I was under the impression that Grove was intended to scale with multiple workers fetching logs concurrently.

If it is designed to be run optimally as a single instance, this would be something for us to consider.

[hcpadkins]

Hey @brianbeklaviyo!

I've been looking into this recently, and the lift to allow concurrency across workloads (horizonal scaling) may not be as large as initially thought after the last few iterations of changes. I'll loop back with our findings, and information about level of effort to make this a supported configuration soon :)