-
Notifications
You must be signed in to change notification settings - Fork 19
DOCS-3471 Added troubleshooting recommendations #1808
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Open
egoverdovskaya-wallarm
wants to merge
2
commits into
master
Choose a base branch
from
feature/docs-3471-add-troubleshooting-section
base: master
Could not load branches
Branch not found: {{ refName }}
Loading
Could not load tags
Nothing to show
Loading
Are you sure you want to change the base?
Some commits from the old base branch may be removed from the timeline,
and old review comments may become outdated.
Open
Changes from all commits
Commits
Show all changes
2 commits
Select commit
Hold shift + click to select a range
File filter
Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
There are no files selected for viewing
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
|
|
@@ -127,6 +127,10 @@ wallarm_wcli_job_error{component="blkexp",code="1"} 0 | |
| wallarm_wcli_job_error{component="botexp",code="1"} 0 | ||
| ``` | ||
|
|
||
| **Troubleshooting recommendations:** | ||
|
|
||
| Non-zero values indicate issues. Check the component logs and investigate network connectivity, configuration errors, or Wallarm Cloud failures based on the error code. | ||
|
|
||
| --- | ||
| ### `wallarm_wcli_job_export_period` | ||
|
|
||
|
|
@@ -154,6 +158,10 @@ wallarm_wcli_job_export_period{component="jwtexp"} 15619.140830117 | |
| wallarm_wcli_job_export_period{component="reqexp"} 15608.084553393 | ||
| ``` | ||
|
|
||
| **Troubleshooting recommendations:** | ||
|
|
||
| High values indicate an export delay. Verify network throughput to Wallarm Cloud and check **wcli** logs for queue issues or Wallarm Cloud bottlenecks. | ||
|
|
||
| --- | ||
| ### `wallarm_wcli_cloud_connectivity` | ||
|
|
||
|
|
@@ -174,6 +182,10 @@ Shows whether the Wallarm cloud is responsive. | |
| wallarm_wcli_cloud_connectivity 1 | ||
| ``` | ||
|
|
||
| **Troubleshooting recommendations:** | ||
|
|
||
| If `0`, the controller cannot reach Wallarm Cloud. Check Internet connectivity, firewall rules, and Cloud endpoint configuration. | ||
|
|
||
| --- | ||
| ### `wallarm_wcli_subscription_active` | ||
|
|
||
|
|
@@ -194,6 +206,10 @@ Shows whether the Wallarm subscription is currently active. | |
| wallarm_wcli_subscription_active 1 | ||
| ``` | ||
|
|
||
| **Troubleshooting recommendations:** | ||
|
|
||
| If `0`, verify the Wallarm subscription status. | ||
|
|
||
| ## Bot feature extraction (botexp) metrics | ||
|
|
||
| Metrics from the bot feature extraction `botexp` job used by the [API Abuse Prevention module][api-abuse-prevention] to analyze and detect automated traffic (bots). | ||
|
|
@@ -215,6 +231,10 @@ Average time the `botexp` job spent processing requests. | |
| go_feature_extractor_processing_duration_seconds_bucket{le="0.1"} 15 | ||
| ``` | ||
|
|
||
| **Troubleshooting recommendations:** | ||
|
|
||
| If values rise, the `botexp` job may be overloaded. Check CPU usage and optimize fetch or export frequency. | ||
|
|
||
| --- | ||
| ### `go_feature_extractor_fetching_duration_seconds` | ||
|
|
||
|
|
@@ -239,6 +259,10 @@ It is also labeled by the result of the send operation (`success` or `error`) an | |
| go_feature_extractor_fetching_duration_seconds_bucket{partner_client_uuid="b938ac84-1ac3-11ec-9f1c-4201ac1ff113",result="success",vmrange="1.000e-04...1.136e-04"} 369 | ||
| ``` | ||
|
|
||
| **Troubleshooting recommendations:** | ||
|
|
||
| Increased error values or high durations suggest slow storage or network latency. Verify **wcli** availability. | ||
|
|
||
| --- | ||
| ### go_feature_extractor_fetching_request_total` | ||
|
|
||
|
|
@@ -256,6 +280,10 @@ The total number of requests the `botexp` job fetched from the request storage. | |
| go_feature_extractor_fetching_request_total 104 | ||
| ``` | ||
|
|
||
| **Troubleshooting recommendations:** | ||
|
|
||
| If stagnant or zero during active traffic, check **wcli** connectivity and `botexp` configuration. | ||
|
|
||
| --- | ||
| ### `go_feature_extractor_sending_duration_seconds_bucket` | ||
|
|
||
|
|
@@ -278,6 +306,10 @@ go_feature_extractor_sending_duration_seconds_bucket{result="error",vmrange="7.7 | |
| go_feature_extractor_sending_duration_seconds_bucket{result="success",vmrange="2.783e-01...3.162e-01"} 29 | ||
| ``` | ||
|
|
||
| **Troubleshooting recommendations:** | ||
|
|
||
| Rising `error` values indicate export issues. Check network latency and Wallarm Cloud account credentials. | ||
|
|
||
| --- | ||
| ### `go_feature_extractor_sending_request_total` | ||
|
|
||
|
|
@@ -296,6 +328,10 @@ go_feature_extractor_sending_request_total{result="error"} 1 | |
| go_feature_extractor_sending_request_total{result="success"} 505 | ||
| ``` | ||
|
|
||
| **Troubleshooting recommendations:** | ||
|
|
||
| Non-zero error values indicate export failures. Inspect **wcli** logs and verify the Wallarm Cloud availability. | ||
|
|
||
| --- | ||
| ### `go_feature_extractor_tarantool_queue_total` | ||
|
|
||
|
|
@@ -321,6 +357,10 @@ go_feature_extractor_tarantool_queue_total{partner_client_uuid="b938ac84-1ac3-11 | |
|
|
||
| ``` | ||
|
|
||
| **Troubleshooting recommendations:** | ||
|
|
||
| If throttled values increase, **wcli** is overloaded. Allocate more resources or reduce the fetch frequency. | ||
|
|
||
| ## wcli-layer metrics for bot feature extraction (botexp) | ||
|
|
||
| Metrics for the `botexp` job (used by the [API Abuse Prevention module][api-abuse-prevention]) at the **wcli** layer. Tracks interactions with the request storage (fetched, skipped, acknowledged, or failed requests) and exports to the Wallarm Cloud (successful and failed). Useful for monitoring `botexp` job activity, reliability, and data flow. | ||
|
|
@@ -427,6 +467,10 @@ The total number of exported requests to the Wallarm Cloud by the `botexp` job. | |
| wallarm_wcli_botexp_api_sent 505 | ||
| ``` | ||
|
|
||
| **Troubleshooting recommendations:** | ||
|
|
||
| If stagnant despite incoming traffic, check Wallarm Cloud endpoint availability and export queue status. | ||
|
|
||
| --- | ||
| ### `wallarm_wcli_botexp_api_failed` | ||
|
|
||
|
|
@@ -444,6 +488,10 @@ The total number of failed export attempts to the Wallarm Cloud by the `botexp` | |
| wallarm_wcli_botexp_api_failed 0 | ||
| ``` | ||
|
|
||
| **Troubleshooting recommendations:** | ||
|
|
||
| Any non-zero count signals Wallarm Cloud or network issues. Inspect retry logic and connectivity. | ||
|
|
||
| ## Blocked request exporter metrics (blkexp) | ||
|
|
||
| Metrics for the `blkexp job`, which tracks the export and processing of blocked requests. Includes counts, processing rates, and export status to the Wallarm Cloud. | ||
|
|
@@ -516,6 +564,10 @@ The total number of exported requests the `blkexp` job sent to the Wallarm Cloud | |
| wallarm_wcli_blkexp_api_send 0 | ||
| ``` | ||
|
|
||
| **Troubleshooting recommendations:** | ||
|
|
||
| If stalled, verify Wallarm Cloud account credentials and network bandwidth. | ||
|
|
||
| --- | ||
| ### `wallarm_wcli_blkexp_api_sent_failed` | ||
|
|
||
|
|
@@ -533,6 +585,10 @@ The total number of failed export attempts the `blkexp` job sent to the Wallarm | |
| wallarm_wcli_blkexp_api_sent_failed 0 | ||
| ``` | ||
|
|
||
| **Troubleshooting recommendations:** | ||
|
|
||
| A non-zero value indicates Wallarm Cloud connectivity or rate-limit issues. | ||
|
|
||
| ## Credential Stuffing Detection metrics (credstuff) | ||
|
|
||
| Metrics for the [Credential Stuffing Detection module][cred-stuffing]. Monitors event detection rate, matched credentials, and request processing statistics. | ||
|
|
@@ -605,6 +661,10 @@ The total number of requests the `credstuff` job successfully processed. | |
| wallarm_wcli_credstuff_requests_processed 0 | ||
| ``` | ||
|
|
||
| **Troubleshooting recommendations:** | ||
|
|
||
| If the value remains `0`, check the input feeds and job initialization. | ||
|
|
||
| --- | ||
| ### `wallarm_wcli_credstuff_requests_failed` | ||
|
|
||
|
|
@@ -622,6 +682,10 @@ The total number of requests the `credstuff` job failed to process. | |
| wallarm_wcli_credstuff_requests_failed 0 | ||
| ``` | ||
|
|
||
| **Troubleshooting recommendations:** | ||
|
|
||
| A rising count indicates credential-matching or logic errors. Check logs for details. | ||
|
|
||
| ## JWT token exporter metrics (jwtexp) | ||
|
|
||
| Metrics from the JWT token exporter, which extracts and analyzes [JSON Web Tokens][jwt-tokens] for authentication and abuse detection. | ||
|
|
@@ -694,6 +758,10 @@ The total number of exported requests to the Wallarm Cloud by the `jwtexp` job. | |
| wallarm_wcli_jwtexp_api_requests_sent 0 | ||
| ``` | ||
|
|
||
| **Troubleshooting recommendations:** | ||
|
|
||
| If the value is zero or stagnant, the export is stalled. Check Wallarm Cloud availability. | ||
|
|
||
| --- | ||
| ### `wallarm_wcli_jwtexp_api_requests_failed` | ||
|
|
||
|
|
@@ -711,6 +779,10 @@ The total number of failed request export attempts to the Wallarm Cloud by the ` | |
| wallarm_wcli_jwtexp_api_requests_failed 0 | ||
| ``` | ||
|
|
||
| **Troubleshooting recommendations:** | ||
|
|
||
| A non-zero value indicates network or Wallarm Cloud failures. | ||
|
|
||
| ## Request exporter metrics (reqexp) | ||
|
|
||
| Metrics for the request exporter, responsible for sending analyzed HTTP request data from the Postanalytics module to the Wallarm Cloud. | ||
|
|
@@ -783,6 +855,10 @@ The total number of exported requests to the Wallarm Cloud by the `reqexp` job. | |
| wallarm_wcli_reqexp_api_requests_sent 156 | ||
| ``` | ||
|
|
||
| **Troubleshooting recommendations:** | ||
|
|
||
| If the value stops increasing, exports are stuck. Verify Wallarm Cloud connectivity. | ||
|
|
||
| --- | ||
| ### `wallarm_wcli_reqexp_api_requests_failed` | ||
|
|
||
|
|
@@ -800,6 +876,10 @@ The total number of failed request export attempts to the Wallarm Cloud by the ` | |
| wallarm_wcli_reqexp_api_requests_failed 0 | ||
| ``` | ||
|
|
||
| **Troubleshooting recommendations:** | ||
|
|
||
| A non-zero value indicates Wallarm Cloud errors or network failures. | ||
|
|
||
| ## Counter exporter metrics (cntexp) | ||
|
|
||
| Metrics from the counter exporter, tracking aggregated counters and summary statistics used for analytics and reporting. | ||
|
|
@@ -838,6 +918,10 @@ The total number of exported counters to the Wallarm Cloud by the `cntexp` job. | |
| wallarm_wcli_cntexp_api_counters_sent 869 | ||
| ``` | ||
|
|
||
| **Troubleshooting recommendations:** | ||
|
|
||
| If it lags behind read counts, check the Wallarm Cloud export pipeline and retry queue. | ||
|
Collaborator
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. what is an export pipeline? |
||
|
|
||
| --- | ||
| ### `wallarm_wcli_cntexp_api_counters_failed` | ||
|
|
||
|
|
@@ -855,6 +939,10 @@ The total number of failed counter export attempts to the Wallarm Cloud by the ` | |
| wallarm_wcli_cntexp_api_counters_failed 0 | ||
| ``` | ||
|
|
||
| **Troubleshooting recommendations:** | ||
|
|
||
| A non-zero value indicates Wallarm Cloud connectivity or authorization issues. | ||
|
|
||
| ## API Discovery metrics (api_discovery) | ||
|
|
||
| Metrics for the [API Discovery module][api-discovery], which analyzes incoming API requests to identify and catalog API endpoints. Tracks batch processing, memory usage, request filtering, and data flush operations to monitor system activity, performance, and reliability. | ||
|
|
||
Oops, something went wrong.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Uh oh!
There was an error while loading. Please reload this page.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
general for all troubleshooting recommendations: provide links to related articles (e.g. subscription plan article where needed, amemory allocation article, log article, wallarm cloud avalability, etc)
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
and if recommendations ask a user to review some other metrics, provide a link to the corresponding ones