Skip to content

[FEATURE] Warning system when detector query becomes too large #1655

@AWSHurneyt

Description

@AWSHurneyt

Is your feature request related to a problem?
A detector can potentially be configured with thousands of rules. Rules are used to construct the query that's executed by the detector to generating finds/alerts. If a query becomes too large, the execution of the query could fail with ParseException. E.g.,

ParseException[Query string length exceeds max allowed length 32000 (search.query.max_query_string_length); actual length: 32920];

Increasing the indices.query.bool.max_clause_count and search.query.max_query_string_length cluster settings can resolve those errors, but this is not a good long-term solution as large queries can negatively impact cluster performance (e.g., causing out-of-memory errors by consuming large amounts of heap memory, increasing search latency by taking a long time to parse and complete large queries, etc.). This would likely require scaling up/out the cluster to compensate for the performance issues. This is a limitation of opensearch itself, not so much the plugin.

What solution would you like?
It would be helpful for the plugin to intelligently warn when a detector query is becoming too large so that users can take appropriate action (e.g., adjust the 2 cluster settings mentioned above, auditing rules to make them more succinct/efficient, create more detectors with fewer rules, etc.).

This could be accomplished through the plugin UI as part of a dry-run process when creating/updating a detector.

It would also be helpful for the detectors to generate error alerts that call out the parse exception.

Do you have any additional context?
Ref# V2112755895

Metadata

Metadata

Assignees

No one assigned

    Labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions