diff --git a/docs/latest/admin-en/wcli-metrics.md b/docs/latest/admin-en/wcli-metrics.md index 0c69e773be..ae313fe3b1 100644 --- a/docs/latest/admin-en/wcli-metrics.md +++ b/docs/latest/admin-en/wcli-metrics.md @@ -127,6 +127,10 @@ wallarm_wcli_job_error{component="blkexp",code="1"} 0 wallarm_wcli_job_error{component="botexp",code="1"} 0 ``` +**Troubleshooting recommendations:** + +Non-zero values indicate issues. Check the component logs and investigate network connectivity, configuration errors, or Wallarm Cloud failures based on the error code. + --- ### `wallarm_wcli_job_export_period` @@ -154,6 +158,10 @@ wallarm_wcli_job_export_period{component="jwtexp"} 15619.140830117 wallarm_wcli_job_export_period{component="reqexp"} 15608.084553393 ``` +**Troubleshooting recommendations:** + +High values indicate an export delay. Verify network throughput to Wallarm Cloud and check **wcli** logs for queue issues or Wallarm Cloud bottlenecks. + --- ### `wallarm_wcli_cloud_connectivity` @@ -174,6 +182,10 @@ Shows whether the Wallarm cloud is responsive. wallarm_wcli_cloud_connectivity 1 ``` +**Troubleshooting recommendations:** + +If `0`, the controller cannot reach Wallarm Cloud. Check Internet connectivity, firewall rules, and Cloud endpoint configuration. + --- ### `wallarm_wcli_subscription_active` @@ -194,6 +206,10 @@ Shows whether the Wallarm subscription is currently active. wallarm_wcli_subscription_active 1 ``` +**Troubleshooting recommendations:** + +If `0`, verify the Wallarm subscription status. + ## Bot feature extraction (botexp) metrics Metrics from the bot feature extraction `botexp` job used by the [API Abuse Prevention module][api-abuse-prevention] to analyze and detect automated traffic (bots). @@ -215,6 +231,10 @@ Average time the `botexp` job spent processing requests. go_feature_extractor_processing_duration_seconds_bucket{le="0.1"} 15 ``` +**Troubleshooting recommendations:** + +If values rise, the `botexp` job may be overloaded. Check CPU usage and optimize fetch or export frequency. + --- ### `go_feature_extractor_fetching_duration_seconds` @@ -239,6 +259,10 @@ It is also labeled by the result of the send operation (`success` or `error`) an go_feature_extractor_fetching_duration_seconds_bucket{partner_client_uuid="b938ac84-1ac3-11ec-9f1c-4201ac1ff113",result="success",vmrange="1.000e-04...1.136e-04"} 369 ``` +**Troubleshooting recommendations:** + +Increased error values or high durations suggest slow storage or network latency. Verify **wcli** availability. + --- ### go_feature_extractor_fetching_request_total` @@ -256,6 +280,10 @@ The total number of requests the `botexp` job fetched from the request storage. go_feature_extractor_fetching_request_total 104 ``` +**Troubleshooting recommendations:** + +If stagnant or zero during active traffic, check **wcli** connectivity and `botexp` configuration. + --- ### `go_feature_extractor_sending_duration_seconds_bucket` @@ -278,6 +306,10 @@ go_feature_extractor_sending_duration_seconds_bucket{result="error",vmrange="7.7 go_feature_extractor_sending_duration_seconds_bucket{result="success",vmrange="2.783e-01...3.162e-01"} 29 ``` +**Troubleshooting recommendations:** + +Rising `error` values indicate export issues. Check network latency and Wallarm Cloud account credentials. + --- ### `go_feature_extractor_sending_request_total` @@ -296,6 +328,10 @@ go_feature_extractor_sending_request_total{result="error"} 1 go_feature_extractor_sending_request_total{result="success"} 505 ``` +**Troubleshooting recommendations:** + +Non-zero error values indicate export failures. Inspect **wcli** logs and verify the Wallarm Cloud availability. + --- ### `go_feature_extractor_tarantool_queue_total` @@ -321,6 +357,10 @@ go_feature_extractor_tarantool_queue_total{partner_client_uuid="b938ac84-1ac3-11 ``` +**Troubleshooting recommendations:** + +If throttled values increase, **wcli** is overloaded. Allocate more resources or reduce the fetch frequency. + ## wcli-layer metrics for bot feature extraction (botexp) Metrics for the `botexp` job (used by the [API Abuse Prevention module][api-abuse-prevention]) at the **wcli** layer. Tracks interactions with the request storage (fetched, skipped, acknowledged, or failed requests) and exports to the Wallarm Cloud (successful and failed). Useful for monitoring `botexp` job activity, reliability, and data flow. @@ -427,6 +467,10 @@ The total number of exported requests to the Wallarm Cloud by the `botexp` job. wallarm_wcli_botexp_api_sent 505 ``` +**Troubleshooting recommendations:** + +If stagnant despite incoming traffic, check Wallarm Cloud endpoint availability and export queue status. + --- ### `wallarm_wcli_botexp_api_failed` @@ -444,6 +488,10 @@ The total number of failed export attempts to the Wallarm Cloud by the `botexp` wallarm_wcli_botexp_api_failed 0 ``` +**Troubleshooting recommendations:** + +Any non-zero count signals Wallarm Cloud or network issues. Inspect retry logic and connectivity. + ## Blocked request exporter metrics (blkexp) Metrics for the `blkexp job`, which tracks the export and processing of blocked requests. Includes counts, processing rates, and export status to the Wallarm Cloud. @@ -516,6 +564,10 @@ The total number of exported requests the `blkexp` job sent to the Wallarm Cloud wallarm_wcli_blkexp_api_send 0 ``` +**Troubleshooting recommendations:** + +If stalled, verify Wallarm Cloud account credentials and network bandwidth. + --- ### `wallarm_wcli_blkexp_api_sent_failed` @@ -533,6 +585,10 @@ The total number of failed export attempts the `blkexp` job sent to the Wallarm wallarm_wcli_blkexp_api_sent_failed 0 ``` +**Troubleshooting recommendations:** + +A non-zero value indicates Wallarm Cloud connectivity or rate-limit issues. + ## Credential Stuffing Detection metrics (credstuff) Metrics for the [Credential Stuffing Detection module][cred-stuffing]. Monitors event detection rate, matched credentials, and request processing statistics. @@ -605,6 +661,10 @@ The total number of requests the `credstuff` job successfully processed. wallarm_wcli_credstuff_requests_processed 0 ``` +**Troubleshooting recommendations:** + +If the value remains `0`, check the input feeds and job initialization. + --- ### `wallarm_wcli_credstuff_requests_failed` @@ -622,6 +682,10 @@ The total number of requests the `credstuff` job failed to process. wallarm_wcli_credstuff_requests_failed 0 ``` +**Troubleshooting recommendations:** + +A rising count indicates credential-matching or logic errors. Check logs for details. + ## JWT token exporter metrics (jwtexp) Metrics from the JWT token exporter, which extracts and analyzes [JSON Web Tokens][jwt-tokens] for authentication and abuse detection. @@ -694,6 +758,10 @@ The total number of exported requests to the Wallarm Cloud by the `jwtexp` job. wallarm_wcli_jwtexp_api_requests_sent 0 ``` +**Troubleshooting recommendations:** + +If the value is zero or stagnant, the export is stalled. Check Wallarm Cloud availability. + --- ### `wallarm_wcli_jwtexp_api_requests_failed` @@ -711,6 +779,10 @@ The total number of failed request export attempts to the Wallarm Cloud by the ` wallarm_wcli_jwtexp_api_requests_failed 0 ``` +**Troubleshooting recommendations:** + +A non-zero value indicates network or Wallarm Cloud failures. + ## Request exporter metrics (reqexp) Metrics for the request exporter, responsible for sending analyzed HTTP request data from the Postanalytics module to the Wallarm Cloud. @@ -783,6 +855,10 @@ The total number of exported requests to the Wallarm Cloud by the `reqexp` job. wallarm_wcli_reqexp_api_requests_sent 156 ``` +**Troubleshooting recommendations:** + +If the value stops increasing, exports are stuck. Verify Wallarm Cloud connectivity. + --- ### `wallarm_wcli_reqexp_api_requests_failed` @@ -800,6 +876,10 @@ The total number of failed request export attempts to the Wallarm Cloud by the ` wallarm_wcli_reqexp_api_requests_failed 0 ``` +**Troubleshooting recommendations:** + +A non-zero value indicates Wallarm Cloud errors or network failures. + ## Counter exporter metrics (cntexp) Metrics from the counter exporter, tracking aggregated counters and summary statistics used for analytics and reporting. @@ -838,6 +918,10 @@ The total number of exported counters to the Wallarm Cloud by the `cntexp` job. wallarm_wcli_cntexp_api_counters_sent 869 ``` +**Troubleshooting recommendations:** + +If it lags behind read counts, check the Wallarm Cloud export pipeline and retry queue. + --- ### `wallarm_wcli_cntexp_api_counters_failed` @@ -855,6 +939,10 @@ The total number of failed counter export attempts to the Wallarm Cloud by the ` wallarm_wcli_cntexp_api_counters_failed 0 ``` +**Troubleshooting recommendations:** + +A non-zero value indicates Wallarm Cloud connectivity or authorization issues. + ## API Discovery metrics (api_discovery) Metrics for the [API Discovery module][api-discovery], which analyzes incoming API requests to identify and catalog API endpoints. Tracks batch processing, memory usage, request filtering, and data flush operations to monitor system activity, performance, and reliability. diff --git a/docs/latest/admin-en/wstore-metrics.md b/docs/latest/admin-en/wstore-metrics.md index ac7996adae..6b40453d9d 100644 --- a/docs/latest/admin-en/wstore-metrics.md +++ b/docs/latest/admin-en/wstore-metrics.md @@ -48,7 +48,7 @@ You can change the default metrics host and port (`http://localhost:9001/metrics --- ### `wallarm_wstore_connections_total` -The total number of network connections handled by wstore, broken down by connection type (i.e., protocol schema like TCP or TLS). +The total number of network connections handled by **wstore**, broken down by connection type (i.e., protocol schema like TCP or TLS). **Type**: Counter @@ -69,7 +69,7 @@ wallarm_wstore_connections_total{schema="TCP"} 219 --- ### `wallarm_wstore_current_connections` -The number of active connections currently established with wstore. +The number of active connections currently established with **wstore**. **Type**: Gauge @@ -83,6 +83,10 @@ The number of active connections currently established with wstore. wallarm_wstore_current_connections 9 ``` +**Troubleshooting recommendations:** + +Watch for unusually high values (e.g., 1000+), as they may indicate hung or unclosed connections. + --- ### `wallarm_wstore_requests_total` @@ -125,11 +129,16 @@ The total number of iproto CALL/CALL_16 requests, broken down by the function na wallarm_wstore_iproto_calls_total{func="wallarm.blocked_stat.read",result="success"} 621473 ``` +**Troubleshooting recommendations:** + +Rising failed counts may indicate **wstore** errors. Check **wstore** logs and ensure sufficient memory allocation. + + ## Request throttling and load shedding --- ### `wallarm_wstore_throttle_mode` -Shows if wstore is currently throttling requests due to severely insufficient resources. When this metric is `1.0`, wstore is dropping some incoming requests because system resources are critically low. +Shows if **wstore** is currently throttling requests due to severely insufficient resources. When this metric is `1.0`, **wstore** is dropping some incoming requests because system resources are critically low. **Type**: Gauge @@ -171,7 +180,7 @@ The number of requests rejected due to queue throttling, broken down by queue. **Type**: Counter -**Labels**: `queue` - name of the wstore queue +**Labels**: `queue` - name of the **wstore** queue **Unit**: Count @@ -185,7 +194,7 @@ wallarm_wstore_queue_throttled{queue="appstructure"} 0 --- ### `wallarm_wstore_queue_size` -The current number of requests in each wstore queue. +The current number of requests in each **wstore** queue. **Type**: Gauge @@ -193,7 +202,7 @@ The current number of requests in each wstore queue. * `engine` - e.g., `ring` -* `name` - name of the wstore queue +* `name` - name of the **wstore** queue **Unit**: Count @@ -206,11 +215,11 @@ wallarm_wstore_queue_size{engine="ring",name="api_discovery"} 0 --- ### `wallarm_wstore_queue_drops` -The number of requests dropped when a wstore queue reaches its maximum size and begins overwriting entries in the ring buffer, broken down by queue. +The number of requests dropped when a **wstore** queue reaches its maximum size and begins overwriting entries in the ring buffer, broken down by queue. **Type**: Counter -**Labels**: `queue` - name of the wstore queue +**Labels**: `queue` - name of the **wstore** queue **Unit**: Count @@ -220,6 +229,10 @@ The number of requests dropped when a wstore queue reaches its maximum size and wallarm_wstore_queue_drops{queue="appstructure"} 0 ``` +**Troubleshooting recommendations:** + +Watch for unusually high values, which vary by available RAM (e.g., ~50 for 1 GB of RAM and ~500,000 for 50 GB of RAM). + --- ### `wallarm_wstore_queue_take_requests` @@ -229,7 +242,7 @@ The number of requests returned from the queue by the `wallarm.requests_processi **Labels**: -* `queue` - name of the wstore queue +* `queue` - name of the **wstore** queue * `result` - result of the operation (`success` or `failed`) @@ -241,14 +254,18 @@ The number of requests returned from the queue by the `wallarm.requests_processi wallarm_wstore_queue_take_requests{queue="appstructure", result="success"} 313187 ``` +**Troubleshooting recommendations:** + +If failed values increase, check for internal processing or acknowledgment issues in **wstore** logs. + --- ### `wallarm_wstore_queue_ack_drops` -The number of acknowledgement attempts for requests that have already been removed from the wstore queue. +The number of acknowledgement attempts for requests that have already been removed from the **wstore** queue. **Type**: Counter -**Labels**: `queue` - name of the wstore queue +**Labels**: `queue` - name of the **wstore** queue **Unit**: Count @@ -267,7 +284,7 @@ The number of requests that were captured but not acknowledged, and were therefo **Labels**: -* `queue` - name of the wstore queue +* `queue` - name of the **wstore** queue * `result` - result of the operation (`success` or `failed`) @@ -288,7 +305,7 @@ The total number of `put`, `ack`, and `take` actions per queue, maintained for b **Labels**: none -* `queue` - name of the wstore queue +* `queue` - name of the **wstore** queue * `action` - type of the queue operation (`put`, `take`, or `ack`) @@ -321,7 +338,7 @@ wallarm_wstore_request_storage_total_size 2285568 --- ### `wallarm_wstore_request_storage_timeframe_size` -Current time span in seconds between the oldest and newest requests stored in wstore. +Current time span in seconds between the oldest and newest requests stored in **wstore**. **Type**: Gauge @@ -335,6 +352,10 @@ Current time span in seconds between the oldest and newest requests stored in ws wallarm_wstore_request_storage_timeframe_size 308775 ``` +**Troubleshooting recommendations:** + +If value drops significantly, request retention is too short. Allocate more memory or disk for **wstore** to maintain longer history. + --- ### `wallarm_wstore_request_storage_drops` @@ -369,6 +390,10 @@ The number of incoming requests rejected because they are too large to be stored wallarm_wstore_request_storage_rejects 0 ``` +**Troubleshooting recommendations:** + +Non-zero values indicate large requests are being rejected. Review request size limits and adjust configuration. + --- ### `wallarm_wstore_request_storage_misses` @@ -386,11 +411,15 @@ The number of attempts to retrieve full request information for dropped or stale wallarm_wstore_request_storage_misses 0 ``` +**Troubleshooting recommendations:** + +If increasing, data for older requests is unavailable. Allocate more memory or disk for **wstore** to maintain longer history. + ## Internal wstore engine metrics --- ### `wallarm_wstore_kvstore_records_total` -The total number of records currently stored in the wstore key-value store. +The total number of records currently stored in the **wstore** key-value store. **Type**: Counter @@ -407,7 +436,7 @@ wallarm_wstore_kvstore_records_total 770 --- ### `wallarm_wstore_kvstore_cleanups` -The number of old requests cleaned up from the wstore internal key-value store. +The number of old requests cleaned up from the **wstore** internal key-value store. **Type**: Counter @@ -424,7 +453,7 @@ wallarm_wstore_kvstore_cleanups 0 --- ### `wallarm_wstore_kvstore_errors` -The number of errors in the wstore internal key-value store operations, labeled by action type (e.g., cleanup, insert, or drop). +The number of errors in the **wstore** internal key-value store operations, labeled by action type (e.g., cleanup, insert, or drop). **Type**: Counter @@ -449,7 +478,7 @@ wallarm_wstore_kvstore_errors{action="cleanup"} 0 --- ### `wallarm_wstore_kvstore_oom_errors_total` -The number of Out Of Memory (OOM) errors occurred during insertion into the wstore key-value store. +The number of Out Of Memory (OOM) errors occurred during insertion into the **wstore** key-value store. **Type**: Counter @@ -466,7 +495,7 @@ wallarm_wstore_kvstore_oom_errors_total 0 --- ### `wallarm_wstore_kvstore_insertions_total` -The number of requests successfully stored by the wstore into its key-value store. +The number of requests successfully stored by the **wstore** into its key-value store. **Type**: Counter @@ -483,7 +512,7 @@ wallarm_wstore_kvstore_insertions_total 770 --- ### `wallarm_wstore_kvstore_lost_insertions_total` -The number of requests failed to be stored in the wstore key-value store after all retry attempts. +The number of requests failed to be stored in the **wstore** key-value store after all retry attempts. **Type**: Counter @@ -500,7 +529,7 @@ wallarm_wstore_kvstore_lost_insertions_total 0 --- ### `wallarm_wstore_kvstore_drops_total` -The number of requests lost due to failed cleanups from the wstore internal key-value store. +The number of requests lost due to failed cleanups from the **wstore** internal key-value store. **Type**: Counter