feat(helm): update chart victoria-metrics-k8s-stack (0.70.0 → 0.72.5) by bot-akira[bot] · Pull Request #3045 · axeII/home-ops

--- kubernetes/apps/observability/victoria-metrics/app Kustomization: observability/victoria-metrics HelmRelease: observability/victoria-metrics

+++ kubernetes/apps/observability/victoria-metrics/app Kustomization: observability/victoria-metrics HelmRelease: observability/victoria-metrics

@@ -13,13 +13,13 @@

     spec:
       chart: victoria-metrics-k8s-stack
       sourceRef:
         kind: HelmRepository
         name: victoria-metrics
         namespace: flux-system
-      version: 0.70.0
+      version: 0.72.5
   install:
     createNamespace: true
     remediation:
       retries: 3
   interval: 30m
   timeout: 15m

bot-akira · 2026-02-16T17:32:39Z

--- HelmRelease: observability/victoria-metrics ConfigMap: observability/vm-node-exporter-full

+++ HelmRelease: observability/victoria-metrics ConfigMap: observability/vm-node-exporter-full

@@ -65,64 +65,65 @@

     },\"overrides\":[]},\"gridPos\":{\"h\":4,\"w\":3,\"x\":3,\"y\":1},\"id\":20,\"\
     options\":{\"minVizHeight\":75,\"minVizWidth\":75,\"orientation\":\"auto\",\"\
     reduceOptions\":{\"calcs\":[\"lastNotNull\"],\"fields\":\"\",\"values\":false},\"\
     showThresholdLabels\":false,\"showThresholdMarkers\":true,\"sizing\":\"auto\"\
     },\"pluginVersion\":\"11.6.1\",\"targets\":[{\"editorMode\":\"code\",\"exemplar\"\
     :false,\"expr\":\"100 * (1 - avg(rate(node_cpu_seconds_total{mode=\\\"idle\\\"\
-    ,instance=\\\"$node\\\",cluster=~\\\"$cluster\\\"}[$__rate_interval])) by(cluster))\"\
-    ,\"instant\":true,\"legendFormat\":\"\",\"range\":false,\"refId\":\"A\",\"step\"\
-    :240}],\"title\":\"CPU Busy\",\"type\":\"gauge\"},{\"datasource\":{\"type\":\"\
-    prometheus\",\"uid\":\"${ds_prometheus}\"},\"description\":\"System load  over\
-    \ all CPU cores together\",\"fieldConfig\":{\"defaults\":{\"color\":{\"mode\"\
-    :\"thresholds\"},\"decimals\":1,\"mappings\":[{\"options\":{\"match\":\"null\"\
-    ,\"result\":{\"text\":\"N/A\"}},\"type\":\"special\"}],\"max\":100,\"min\":0,\"\
-    thresholds\":{\"mode\":\"absolute\",\"steps\":[{\"color\":\"rgba(50, 172, 45,\
-    \ 0.97)\"},{\"color\":\"rgba(237, 129, 40, 0.89)\",\"value\":85},{\"color\":\"\
-    rgba(245, 54, 54, 0.9)\",\"value\":95}]},\"unit\":\"percent\"},\"overrides\":[]},\"\
-    gridPos\":{\"h\":4,\"w\":3,\"x\":6,\"y\":1},\"id\":155,\"options\":{\"minVizHeight\"\
-    :75,\"minVizWidth\":75,\"orientation\":\"auto\",\"reduceOptions\":{\"calcs\":[\"\
-    lastNotNull\"],\"fields\":\"\",\"values\":false},\"showThresholdLabels\":false,\"\
-    showThresholdMarkers\":true,\"sizing\":\"auto\"},\"pluginVersion\":\"11.6.1\"\
-    ,\"targets\":[{\"editorMode\":\"code\",\"exemplar\":false,\"expr\":\"(scalar(node_load1{instance=\\\
-    \"$node\\\",job=\\\"$job\\\",cluster=~\\\"$cluster\\\"}) * 100) / count(count(node_cpu_seconds_total{instance=\\\
-    \"$node\\\",job=\\\"$job\\\",cluster=~\\\"$cluster\\\"}) by(cpu,cluster)) by(cluster)\"\
-    ,\"format\":\"time_series\",\"instant\":true,\"range\":false,\"refId\":\"A\",\"\
-    step\":240}],\"title\":\"Sys Load\",\"type\":\"gauge\"},{\"datasource\":{\"type\"\
-    :\"prometheus\",\"uid\":\"${ds_prometheus}\"},\"description\":\"Real RAM usage\
-    \ excluding cache and reclaimable memory\",\"fieldConfig\":{\"defaults\":{\"color\"\
-    :{\"mode\":\"thresholds\"},\"decimals\":1,\"mappings\":[],\"max\":100,\"min\"\
-    :0,\"thresholds\":{\"mode\":\"absolute\",\"steps\":[{\"color\":\"rgba(50, 172,\
-    \ 45, 0.97)\"},{\"color\":\"rgba(237, 129, 40, 0.89)\",\"value\":80},{\"color\"\
-    :\"rgba(245, 54, 54, 0.9)\",\"value\":90}]},\"unit\":\"percent\"},\"overrides\"\
-    :[]},\"gridPos\":{\"h\":4,\"w\":3,\"x\":9,\"y\":1},\"id\":16,\"options\":{\"minVizHeight\"\
-    :75,\"minVizWidth\":75,\"orientation\":\"auto\",\"reduceOptions\":{\"calcs\":[\"\
-    lastNotNull\"],\"fields\":\"\",\"values\":false},\"showThresholdLabels\":false,\"\
-    showThresholdMarkers\":true,\"sizing\":\"auto\"},\"pluginVersion\":\"11.6.1\"\
-    ,\"targets\":[{\"editorMode\":\"code\",\"exemplar\":false,\"expr\":\"clamp_min((1\
-    \ - (node_memory_MemAvailable_bytes{instance=\\\"$node\\\",job=\\\"$job\\\",cluster=~\\\
-    \"$cluster\\\"} / node_memory_MemTotal_bytes{instance=\\\"$node\\\",job=\\\"$job\\\
-    \",cluster=~\\\"$cluster\\\"})) * 100, 0)\",\"format\":\"time_series\",\"instant\"\
-    :true,\"range\":false,\"refId\":\"B\",\"step\":240}],\"title\":\"RAM Used\",\"\
+    ,instance=\\\"$node\\\",job=\\\"$job\\\",cluster=~\\\"$cluster\\\"}[$__rate_interval]))\
+    \ by(cluster))\",\"instant\":true,\"legendFormat\":\"\",\"range\":false,\"refId\"\
+    :\"A\",\"step\":240}],\"title\":\"CPU Busy\",\"type\":\"gauge\"},{\"datasource\"\
+    :{\"type\":\"prometheus\",\"uid\":\"${ds_prometheus}\"},\"description\":\"System\
+    \ load over all CPU cores together\",\"fieldConfig\":{\"defaults\":{\"color\"\
+    :{\"mode\":\"thresholds\"},\"decimals\":1,\"mappings\":[{\"options\":{\"match\"\
+    :\"null\",\"result\":{\"text\":\"N/A\"}},\"type\":\"special\"}],\"max\":100,\"\
+    min\":0,\"thresholds\":{\"mode\":\"absolute\",\"steps\":[{\"color\":\"rgba(50,\
+    \ 172, 45, 0.97)\"},{\"color\":\"rgba(237, 129, 40, 0.89)\",\"value\":85},{\"\
+    color\":\"rgba(245, 54, 54, 0.9)\",\"value\":95}]},\"unit\":\"percent\"},\"overrides\"\
+    :[]},\"gridPos\":{\"h\":4,\"w\":3,\"x\":6,\"y\":1},\"id\":155,\"options\":{\"\
+    minVizHeight\":75,\"minVizWidth\":75,\"orientation\":\"auto\",\"reduceOptions\"\
+    :{\"calcs\":[\"lastNotNull\"],\"fields\":\"\",\"values\":false},\"showThresholdLabels\"\
+    :false,\"showThresholdMarkers\":true,\"sizing\":\"auto\"},\"pluginVersion\":\"\
+    11.6.1\",\"targets\":[{\"editorMode\":\"code\",\"exemplar\":false,\"expr\":\"\
+    (scalar(node_load1{instance=\\\"$node\\\",job=\\\"$job\\\",cluster=~\\\"$cluster\\\
+    \"}) * 100) / count(count(node_cpu_seconds_total{instance=\\\"$node\\\",job=\\\
+    \"$job\\\",cluster=~\\\"$cluster\\\"}) by(cpu,cluster)) by(cluster)\",\"format\"\
+    :\"time_series\",\"instant\":true,\"range\":false,\"refId\":\"A\",\"step\":240}],\"\
+    title\":\"Sys Load\",\"type\":\"gauge\"},{\"datasource\":{\"type\":\"prometheus\"\
+    ,\"uid\":\"${ds_prometheus}\"},\"description\":\"Real RAM usage excluding cache\
+    \ and reclaimable memory\",\"fieldConfig\":{\"defaults\":{\"color\":{\"mode\"\
+    :\"thresholds\"},\"decimals\":1,\"mappings\":[],\"max\":100,\"min\":0,\"thresholds\"\
+    :{\"mode\":\"absolute\",\"steps\":[{\"color\":\"rgba(50, 172, 45, 0.97)\"},{\"\
+    color\":\"rgba(237, 129, 40, 0.89)\",\"value\":80},{\"color\":\"rgba(245, 54,\
+    \ 54, 0.9)\",\"value\":90}]},\"unit\":\"percent\"},\"overrides\":[]},\"gridPos\"\
+    :{\"h\":4,\"w\":3,\"x\":9,\"y\":1},\"id\":16,\"options\":{\"minVizHeight\":75,\"\
+    minVizWidth\":75,\"orientation\":\"auto\",\"reduceOptions\":{\"calcs\":[\"lastNotNull\"\
+    ],\"fields\":\"\",\"values\":false},\"showThresholdLabels\":false,\"showThresholdMarkers\"\
+    :true,\"sizing\":\"auto\"},\"pluginVersion\":\"11.6.1\",\"targets\":[{\"editorMode\"\
+    :\"code\",\"exemplar\":false,\"expr\":\"clamp_min((1 - (node_memory_MemAvailable_bytes{instance=\\\
+    \"$node\\\",job=\\\"$job\\\",cluster=~\\\"$cluster\\\"} / node_memory_MemTotal_bytes{instance=\\\
+    \"$node\\\",job=\\\"$job\\\",cluster=~\\\"$cluster\\\"})) * 100, 0)\",\"format\"\
+    :\"time_series\",\"instant\":true,\"range\":false,\"refId\":\"B\",\"step\":240}],\"\
+    title\":\"RAM Used\",\"type\":\"gauge\"},{\"datasource\":{\"type\":\"prometheus\"\
+    ,\"uid\":\"${ds_prometheus}\"},\"description\":\"Percentage of swap space currently\
+    \ used by the system\",\"fieldConfig\":{\"defaults\":{\"color\":{\"mode\":\"thresholds\"\
+    },\"decimals\":1,\"mappings\":[{\"options\":{\"match\":\"null\",\"result\":{\"\
+    text\":\"N/A\"}},\"type\":\"special\"}],\"max\":100,\"min\":0,\"thresholds\":{\"\
+    mode\":\"absolute\",\"steps\":[{\"color\":\"rgba(50, 172, 45, 0.97)\"},{\"color\"\
+    :\"rgba(237, 129, 40, 0.89)\",\"value\":10},{\"color\":\"rgba(245, 54, 54, 0.9)\"\
+    ,\"value\":25}]},\"unit\":\"percent\"},\"overrides\":[]},\"gridPos\":{\"h\":4,\"\
+    w\":3,\"x\":12,\"y\":1},\"id\":21,\"options\":{\"minVizHeight\":75,\"minVizWidth\"\
+    :75,\"orientation\":\"auto\",\"reduceOptions\":{\"calcs\":[\"lastNotNull\"],\"\
+    fields\":\"\",\"values\":false},\"showThresholdLabels\":false,\"showThresholdMarkers\"\
+    :true,\"sizing\":\"auto\"},\"pluginVersion\":\"11.6.1\",\"targets\":[{\"editorMode\"\
+    :\"code\",\"exemplar\":false,\"expr\":\"((node_memory_SwapTotal_bytes{instance=\\\
+    \"$node\\\",job=\\\"$job\\\",cluster=~\\\"$cluster\\\"} >bool 0) * ((node_memory_SwapTotal_bytes{instance=\\\
+    \"$node\\\",job=\\\"$job\\\",cluster=~\\\"$cluster\\\"} - node_memory_SwapFree_bytes{instance=\\\
+    \"$node\\\",job=\\\"$job\\\",cluster=~\\\"$cluster\\\"}) / node_memory_SwapTotal_bytes{instance=\\\
+    \"$node\\\",job=\\\"$job\\\",cluster=~\\\"$cluster\\\"})) * 100\",\"instant\"\
+    :true,\"range\":false,\"refId\":\"A\",\"step\":240}],\"title\":\"SWAP Used\",\"\
     type\":\"gauge\"},{\"datasource\":{\"type\":\"prometheus\",\"uid\":\"${ds_prometheus}\"\
-    },\"description\":\"Percentage of swap space currently used by the system\",\"\
-    fieldConfig\":{\"defaults\":{\"color\":{\"mode\":\"thresholds\"},\"decimals\"\
-    :1,\"mappings\":[{\"options\":{\"match\":\"null\",\"result\":{\"text\":\"N/A\"\
-    }},\"type\":\"special\"}],\"max\":100,\"min\":0,\"thresholds\":{\"mode\":\"absolute\"\
-    ,\"steps\":[{\"color\":\"rgba(50, 172, 45, 0.97)\"},{\"color\":\"rgba(237, 129,\
-    \ 40, 0.89)\",\"value\":10},{\"color\":\"rgba(245, 54, 54, 0.9)\",\"value\":25}]},\"\
-    unit\":\"percent\"},\"overrides\":[]},\"gridPos\":{\"h\":4,\"w\":3,\"x\":12,\"\
-    y\":1},\"id\":21,\"options\":{\"minVizHeight\":75,\"minVizWidth\":75,\"orientation\"\
-    :\"auto\",\"reduceOptions\":{\"calcs\":[\"lastNotNull\"],\"fields\":\"\",\"values\"\
-    :false},\"showThresholdLabels\":false,\"showThresholdMarkers\":true,\"sizing\"\
-    :\"auto\"},\"pluginVersion\":\"11.6.1\",\"targets\":[{\"editorMode\":\"code\"\
-    ,\"exemplar\":false,\"expr\":\"((node_memory_SwapTotal_bytes{instance=\\\"$node\\\
-    \",job=\\\"$job\\\",cluster=~\\\"$cluster\\\"} - node_memory_SwapFree_bytes{instance=\\\
-    \"$node\\\",job=\\\"$job\\\",cluster=~\\\"$cluster\\\"}) / node_memory_SwapTotal_bytes{instance=\\\
-    \"$node\\\",job=\\\"$job\\\",cluster=~\\\"$cluster\\\"}) * 100\",\"instant\":true,\"\
-    range\":false,\"refId\":\"A\",\"step\":240}],\"title\":\"SWAP Used\",\"type\"\
-    :\"gauge\"},{\"datasource\":{\"type\":\"prometheus\",\"uid\":\"${ds_prometheus}\"\
     },\"description\":\"Used Root FS\",\"fieldConfig\":{\"defaults\":{\"color\":{\"\
[Diff truncated by flux-local]
--- HelmRelease: observability/victoria-metrics ConfigMap: observability/vm-victoriametrics-operator

+++ HelmRelease: observability/victoria-metrics ConfigMap: observability/vm-victoriametrics-operator

@@ -9,8 +9,8 @@

     app: victoria-metrics-k8s-stack-grafana
     app.kubernetes.io/instance: victoria-metrics
     app.kubernetes.io/managed-by: Helm
     app.kubernetes.io/name: victoria-metrics-k8s-stack
 data:
   victoriametrics-operator.json: |
[Diff truncated by flux-local]
--- HelmRelease: observability/victoria-metrics ConfigMap: observability/vm-victoriametrics-single-node

+++ HelmRelease: observability/victoria-metrics ConfigMap: observability/vm-victoriametrics-single-node

@@ -22,14 +22,18 @@

     iconColor\":\"red\",\"name\":\"alerts\",\"titleFormat\":\"{{alertname}}\"},{\"\
     datasource\":{\"type\":\"prometheus\",\"uid\":\"$ds\"},\"enable\":true,\"expr\"\
     :\"sum(vm_app_version{job=~\\\"$job\\\",instance=~\\\"$instance\\\",cluster=~\\\
     \"$cluster\\\"}) by(version,cluster) unless sum(vm_app_version{job=~\\\"$job\\\
     \",instance=~\\\"$instance\\\",cluster=~\\\"$cluster\\\"} offset $__interval)\
     \ by(version,cluster)\",\"hide\":true,\"iconColor\":\"dark-blue\",\"name\":\"\
-    version\",\"textFormat\":\"{{version}}\",\"titleFormat\":\"Version change\"}]},\"\
-    description\":\"Overview for single-node VictoriaMetrics v1.117.0 or higher\"\
+    version\",\"textFormat\":\"{{version}}\",\"titleFormat\":\"Version change\"},{\"\
+    datasource\":{\"type\":\"prometheus\",\"uid\":\"$ds\"},\"enable\":true,\"expr\"\
+    :\"sum(changes(vm_app_start_timestamp{job=~\\\"$job\\\",instance=~\\\"$instance\\\
+    \",cluster=~\\\"$cluster\\\"}[$__interval])) by(job,cluster)\",\"hide\":false,\"\
+    iconColor\":\"dark-yellow\",\"name\":\"restarts\",\"textFormat\":\"{{job}} restarted\"\
+    }]},\"description\":\"Overview for single-node VictoriaMetrics v1.117.0 or higher\"\
     ,\"editable\":false,\"fiscalYearStartMonth\":0,\"graphTooltip\":1,\"id\":13,\"\
     links\":[{\"icon\":\"doc\",\"tags\":[],\"targetBlank\":true,\"title\":\"Docs\"\
     ,\"type\":\"link\",\"url\":\"https://docs.victoriametrics.com/victoriametrics/single-server-victoriametrics/\"\
     },{\"asDropdown\":false,\"icon\":\"question\",\"includeVars\":false,\"keepTime\"\
     :false,\"tags\":[],\"targetBlank\":true,\"title\":\"Troubleshooting\",\"tooltip\"\
     :\"\",\"type\":\"link\",\"url\":\"https://docs.victoriametrics.com/victoriametrics/single-server-victoriametrics/#troubleshooting\"\
@@ -326,14 +330,15 @@

     :\"text\",\"axisLabel\":\"\",\"axisPlacement\":\"auto\",\"barAlignment\":0,\"\
     barWidthFactor\":0.6,\"drawStyle\":\"bars\",\"fillOpacity\":100,\"gradientMode\"\
     :\"none\",\"hideFrom\":{\"legend\":false,\"tooltip\":false,\"viz\":false},\"insertNulls\"\
     :false,\"lineInterpolation\":\"linear\",\"lineWidth\":1,\"pointSize\":5,\"scaleDistribution\"\
     :{\"type\":\"linear\"},\"showPoints\":\"auto\",\"showValues\":false,\"spanNulls\"\
     :false,\"stacking\":{\"group\":\"A\",\"mode\":\"none\"},\"thresholdsStyle\":{\"\
-    mode\":\"off\"}},\"links\":[{\"targetBlank\":true,\"title\":\"asd\",\"url\":\"\
-    asd\"}],\"mappings\":[],\"thresholds\":{\"mode\":\"absolute\",\"steps\":[{\"color\"\
+    mode\":\"off\"}},\"links\":[{\"targetBlank\":true,\"title\":\"Drilldown\",\"url\"\
+    :\"/d/wNf0q_kZk?viewPanel=154&var-job=${__field.labels.job}&var-ds=$ds&${__url_time_range}\"\
+    }],\"mappings\":[],\"thresholds\":{\"mode\":\"absolute\",\"steps\":[{\"color\"\
     :\"green\",\"value\":0},{\"color\":\"red\",\"value\":80}]}},\"overrides\":[]},\"\
     gridPos\":{\"h\":8,\"w\":12,\"x\":12,\"y\":29},\"id\":110,\"options\":{\"legend\"\
     :{\"calcs\":[\"lastNotNull\",\"mean\",\"max\"],\"displayMode\":\"table\",\"placement\"\
     :\"bottom\",\"showLegend\":true,\"sortBy\":\"Last *\",\"sortDesc\":true},\"tooltip\"\
     :{\"hideZeros\":false,\"mode\":\"single\",\"sort\":\"none\"}},\"pluginVersion\"\
     :\"12.3.0\",\"targets\":[{\"datasource\":{\"type\":\"prometheus\",\"uid\":\"$ds\"\
@@ -353,15 +358,15 @@

     :\"line\",\"fillOpacity\":0,\"gradientMode\":\"none\",\"hideFrom\":{\"legend\"\
     :false,\"tooltip\":false,\"viz\":false},\"insertNulls\":false,\"lineInterpolation\"\
     :\"linear\",\"lineWidth\":1,\"pointSize\":5,\"scaleDistribution\":{\"type\":\"\
     linear\"},\"showPoints\":\"never\",\"showValues\":false,\"spanNulls\":false,\"\
     stacking\":{\"group\":\"A\",\"mode\":\"none\"},\"thresholdsStyle\":{\"mode\":\"\
     off\"}},\"links\":[{\"targetBlank\":true,\"title\":\"Drilldown - RSS memory usage\"\
-    ,\"url\":\"/d/wNf0q_kZk?viewPanel=148&var-job=${__field.labels.job}&var-ds=$ds&var-instance=$instance&${__url_time_range}\"\
+    ,\"url\":\"/d/wNf0q_kZk?viewPanel=148&var-job=${__field.labels.job}&var-ds=$ds&${__url_time_range}\"\
     },{\"targetBlank\":true,\"title\":\"Drilldown - Memory usage breakdown\",\"url\"\
-    :\"/d/wNf0q_kZk?viewPanel=141&var-job=${__field.labels.job}&var-ds=$ds&var-instance=$instance&${__url_time_range}\"\
+    :\"/d/wNf0q_kZk?viewPanel=141&var-job=${__field.labels.job}&var-ds=$ds&${__url_time_range}\"\
     }],\"mappings\":[],\"min\":0,\"thresholds\":{\"mode\":\"absolute\",\"steps\":[{\"\
     color\":\"green\",\"value\":0},{\"color\":\"red\",\"value\":80}]},\"unit\":\"\
     percentunit\"},\"overrides\":[]},\"gridPos\":{\"h\":8,\"w\":12,\"x\":0,\"y\":31},\"\
     id\":112,\"options\":{\"legend\":{\"calcs\":[\"mean\",\"lastNotNull\",\"max\"\
     ],\"displayMode\":\"table\",\"placement\":\"bottom\",\"showLegend\":true,\"sortBy\"\
     :\"Last *\",\"sortDesc\":true},\"tooltip\":{\"hideZeros\":false,\"mode\":\"multi\"\
@@ -379,13 +384,13 @@

     barWidthFactor\":0.6,\"drawStyle\":\"line\",\"fillOpacity\":0,\"gradientMode\"\
     :\"none\",\"hideFrom\":{\"legend\":false,\"tooltip\":false,\"viz\":false},\"insertNulls\"\
     :false,\"lineInterpolation\":\"linear\",\"lineWidth\":1,\"pointSize\":5,\"scaleDistribution\"\
     :{\"type\":\"linear\"},\"showPoints\":\"never\",\"showValues\":false,\"spanNulls\"\
     :false,\"stacking\":{\"group\":\"A\",\"mode\":\"none\"},\"thresholdsStyle\":{\"\
     mode\":\"off\"}},\"links\":[{\"targetBlank\":true,\"title\":\"Drilldown\",\"url\"\
-    :\"/d/wNf0q_kZk?viewPanel=151&var-job=${__field.labels.job}&var-ds=$ds&var-instance=$instance&${__url_time_range}\"\
+    :\"/d/wNf0q_kZk?viewPanel=151&var-job=${__field.labels.job}&var-ds=$ds&${__url_time_range}\"\
     }],\"mappings\":[],\"min\":0,\"thresholds\":{\"mode\":\"absolute\",\"steps\":[{\"\
     color\":\"green\",\"value\":0},{\"color\":\"red\",\"value\":80}]},\"unit\":\"\
     percentunit\"},\"overrides\":[]},\"gridPos\":{\"h\":8,\"w\":12,\"x\":12,\"y\"\
     :31},\"id\":114,\"options\":{\"legend\":{\"calcs\":[\"mean\",\"lastNotNull\",\"\
     max\"],\"displayMode\":\"table\",\"placement\":\"bottom\",\"showLegend\":true,\"\
     sortBy\":\"Last *\",\"sortDesc\":true},\"tooltip\":{\"hideZeros\":false,\"mode\"\
@@ -405,13 +410,13 @@

     ,\"barAlignment\":0,\"barWidthFactor\":0.6,\"drawStyle\":\"line\",\"fillOpacity\"\
     :0,\"gradientMode\":\"none\",\"hideFrom\":{\"legend\":false,\"tooltip\":false,\"\
     viz\":false},\"insertNulls\":false,\"lineInterpolation\":\"linear\",\"lineWidth\"\
     :1,\"pointSize\":5,\"scaleDistribution\":{\"type\":\"linear\"},\"showPoints\"\
     :\"never\",\"showValues\":false,\"spanNulls\":false,\"stacking\":{\"group\":\"\
     A\",\"mode\":\"none\"},\"thresholdsStyle\":{\"mode\":\"off\"}},\"links\":[{\"\
-    targetBlank\":true,\"title\":\"Drilldown\",\"url\":\"/d/wNf0q_kZk?viewPanel=149&var-job=${__field.labels.job}&var-ds=$ds&var-instance=$instance&${__url_time_range}\"\
+    targetBlank\":true,\"title\":\"Drilldown\",\"url\":\"/d/wNf0q_kZk?viewPanel=149&var-job=${__field.labels.job}&var-ds=$ds&${__url_time_range}\"\
     }],\"mappings\":[],\"min\":0,\"thresholds\":{\"mode\":\"absolute\",\"steps\":[{\"\
     color\":\"green\",\"value\":0},{\"color\":\"red\",\"value\":80}]},\"unit\":\"\
     percentunit\"},\"overrides\":[]},\"gridPos\":{\"h\":8,\"w\":12,\"x\":0,\"y\":39},\"\
     id\":123,\"options\":{\"legend\":{\"calcs\":[\"mean\",\"lastNotNull\",\"max\"\
     ],\"displayMode\":\"table\",\"placement\":\"bottom\",\"showLegend\":true,\"sortBy\"\
     :\"Last *\",\"sortDesc\":true},\"tooltip\":{\"hideZeros\":false,\"mode\":\"multi\"\
@@ -1128,22 +1133,22 @@

     :\"auto\",\"barAlignment\":0,\"barWidthFactor\":0.6,\"drawStyle\":\"line\",\"\
     fillOpacity\":0,\"gradientMode\":\"none\",\"hideFrom\":{\"legend\":false,\"tooltip\"\
     :false,\"viz\":false},\"insertNulls\":false,\"lineInterpolation\":\"linear\",\"\
     lineWidth\":1,\"pointSize\":5,\"scaleDistribution\":{\"type\":\"linear\"},\"showPoints\"\
     :\"auto\",\"showValues\":false,\"spanNulls\":false,\"stacking\":{\"group\":\"\
     A\",\"mode\":\"none\"},\"thresholdsStyle\":{\"mode\":\"off\"}},\"links\":[{\"\
-    targetBlank\":true,\"title\":\"Drilldown\",\"url\":\"/d/wNf0q_kZk?viewPanel=140&var-job=${__field.labels.job}&var-ds=$ds&var-instance=$instance&${__url_time_range}\"\
+    targetBlank\":true,\"title\":\"Drilldown\",\"url\":\"/d/wNf0q_kZk?viewPanel=140&var-job=${__field.labels.job}&var-ds=$ds&${__url_time_range}\"\
     }],\"mappings\":[],\"thresholds\":{\"mode\":\"absolute\",\"steps\":[{\"color\"\
     :\"green\",\"value\":0},{\"color\":\"red\",\"value\":80}]}},\"overrides\":[]},\"\
     gridPos\":{\"h\":8,\"w\":12,\"x\":0,\"y\":73},\"id\":137,\"options\":{\"legend\"\
     :{\"calcs\":[\"mean\",\"lastNotNull\",\"max\"],\"displayMode\":\"table\",\"placement\"\
     :\"bottom\",\"showLegend\":true},\"tooltip\":{\"hideZeros\":false,\"mode\":\"\
     multi\",\"sort\":\"desc\"}},\"pluginVersion\":\"12.2.0\",\"targets\":[{\"datasource\"\
     :{\"type\":\"prometheus\",\"uid\":\"${ds}\"},\"editorMode\":\"code\",\"expr\"\
     :\"sum(rate(process_major_pagefaults_total{job=~\\\"$job\\\",instance=~\\\"$instance\\\
-    \",cluster=~\\\"$cluster\\\"}[$__rate_interval])) by(job,cluster)\",\"legendFormat\"\
+    \",cluster=~\\\"$cluster\\\"}[$__rate_interval])) by(job,cluster) > 0\",\"legendFormat\"\
     :\"__auto\",\"range\":true,\"refId\":\"A\"}],\"title\":\"Major page faults rate\"\
     ,\"type\":\"timeseries\"}],\"title\":\"Troubleshooting\",\"type\":\"row\"},{\"\
     collapsed\":true,\"gridPos\":{\"h\":1,\"w\":24,\"x\":0,\"y\":39},\"id\":14,\"\
     panels\":[{\"datasource\":{\"type\":\"prometheus\",\"uid\":\"$ds\"},\"description\"\
     :\"How many datapoints are inserted into storage per second\",\"fieldConfig\"\
[Diff truncated by flux-local]
--- HelmRelease: observability/victoria-metrics ConfigMap: observability/vm-victoriametrics-vmagent

+++ HelmRelease: observability/victoria-metrics ConfigMap: observability/vm-victoriametrics-vmagent

@@ -24,13 +24,13 @@

     },{\"datasource\":{\"type\":\"prometheus\",\"uid\":\"$ds\"},\"enable\":true,\"\
     expr\":\"sum(changes(vm_app_start_timestamp{job=~\\\"$job\\\",instance=~\\\"$instance\\\
     \",cluster=~\\\"$cluster\\\"}[$__interval])) by(job,instance,cluster)\",\"hide\"\
     :false,\"iconColor\":\"dark-yellow\",\"name\":\"restarts\",\"textFormat\":\"{{job}}:{{instance}}\
     \ restarted\"}]},\"description\":\"Overview for VictoriaMetrics vmagent v1.117.0\
     \ or higher\",\"editable\":false,\"fiscalYearStartMonth\":0,\"graphTooltip\":1,\"\
-    id\":2,\"links\":[{\"icon\":\"doc\",\"tags\":[],\"targetBlank\":true,\"title\"\
+    id\":3,\"links\":[{\"icon\":\"doc\",\"tags\":[],\"targetBlank\":true,\"title\"\
     :\"Docs\",\"type\":\"link\",\"url\":\"https://docs.victoriametrics.com/victoriametrics/vmagent/\"\
     },{\"asDropdown\":false,\"icon\":\"question\",\"includeVars\":false,\"keepTime\"\
     :false,\"tags\":[],\"targetBlank\":true,\"title\":\"Troubleshooting\",\"tooltip\"\
     :\"\",\"type\":\"link\",\"url\":\"https://docs.victoriametrics.com/victoriametrics/vmagent/#troubleshooting\"\
     },{\"icon\":\"info\",\"tags\":[],\"targetBlank\":true,\"title\":\"Community support\"\
     ,\"type\":\"link\",\"url\":\"https://victoriametrics.com/support/community-support/\"\
@@ -144,13 +144,13 @@

     :\"auto\",\"orientation\":\"auto\",\"percentChangeColorMode\":\"standard\",\"\
     reduceOptions\":{\"calcs\":[\"last\"],\"fields\":\"\",\"values\":false},\"showPercentChange\"\
     :false,\"text\":{},\"textMode\":\"auto\",\"wideLayout\":true},\"pluginVersion\"\
     :\"12.2.0\",\"targets\":[{\"datasource\":{\"type\":\"prometheus\",\"uid\":\"$ds\"\
     },\"expr\":\"sum(increase(vm_log_messages_total{job=~\\\"$job\\\",instance=~\\\
     \"$instance\\\",level!=\\\"info\\\",cluster=~\\\"$cluster\\\"}[30m])) by(cluster)\"\
-    ,\"interval\":\"\",\"legendFormat\":\"\",\"refId\":\"A\"}],\"title\":\"Log errors\
+    ,\"interval\":\"\",\"legendFormat\":\"\",\"refId\":\"A\"}],\"title\":\"Log messages\
     \ (30m)\",\"type\":\"stat\"},{\"datasource\":{\"type\":\"prometheus\",\"uid\"\
     :\"$ds\"},\"description\":\"Total size of available memory for selected vmagents.\"\
     ,\"fieldConfig\":{\"defaults\":{\"color\":{\"mode\":\"thresholds\"},\"mappings\"\
     :[],\"thresholds\":{\"mode\":\"absolute\",\"steps\":[{\"color\":\"green\",\"value\"\
     :0}]},\"unit\":\"bytes\"},\"overrides\":[]},\"gridPos\":{\"h\":3,\"w\":6,\"x\"\
     :18,\"y\":4},\"id\":153,\"maxDataPoints\":100,\"options\":{\"colorMode\":\"value\"\
@@ -208,13 +208,13 @@

     axisColorMode\":\"text\",\"axisLabel\":\"\",\"axisPlacement\":\"auto\",\"barAlignment\"\
     :0,\"barWidthFactor\":0.6,\"drawStyle\":\"line\",\"fillOpacity\":0,\"gradientMode\"\
     :\"none\",\"hideFrom\":{\"legend\":false,\"tooltip\":false,\"viz\":false},\"insertNulls\"\
     :false,\"lineInterpolation\":\"linear\",\"lineWidth\":1,\"pointSize\":5,\"scaleDistribution\"\
     :{\"type\":\"linear\"},\"showPoints\":\"never\",\"showValues\":false,\"spanNulls\"\
     :false,\"stacking\":{\"group\":\"A\",\"mode\":\"none\"},\"thresholdsStyle\":{\"\
-    mode\":\"off\"}},\"links\":[{\"title\":\"Drilldown\",\"url\":\"/d/G7Z9GzMGz?viewPanel=123&var-job=${__field.labels.job}&var-ds=$ds&var-instance=$instance&${__url_time_range}\"\
+    mode\":\"off\"}},\"links\":[{\"title\":\"Drilldown\",\"url\":\"/d/G7Z9GzMGz?viewPanel=123&var-job=${__field.labels.job}&var-ds=$ds&${__url_time_range}\"\
     }],\"mappings\":[],\"thresholds\":{\"mode\":\"absolute\",\"steps\":[{\"color\"\
     :\"green\",\"value\":0},{\"color\":\"red\",\"value\":80}]},\"unit\":\"short\"\
     },\"overrides\":[{\"matcher\":{\"id\":\"byRegexp\",\"options\":\"/out .*/\"},\"\
     properties\":[{\"id\":\"custom.transform\",\"value\":\"negative-Y\"}]}]},\"gridPos\"\
     :{\"h\":8,\"w\":12,\"x\":0,\"y\":13},\"id\":5,\"options\":{\"legend\":{\"calcs\"\
     :[\"mean\",\"lastNotNull\",\"max\"],\"displayMode\":\"table\",\"placement\":\"\
@@ -270,27 +270,29 @@

     ,\"axisPlacement\":\"auto\",\"barAlignment\":0,\"barWidthFactor\":0.6,\"drawStyle\"\
     :\"bars\",\"fillOpacity\":100,\"gradientMode\":\"none\",\"hideFrom\":{\"legend\"\
     :false,\"tooltip\":false,\"viz\":false},\"insertNulls\":false,\"lineInterpolation\"\
     :\"linear\",\"lineWidth\":1,\"pointSize\":5,\"scaleDistribution\":{\"type\":\"\
     linear\"},\"showPoints\":\"never\",\"showValues\":false,\"spanNulls\":false,\"\
     stacking\":{\"group\":\"A\",\"mode\":\"normal\"},\"thresholdsStyle\":{\"mode\"\
-    :\"off\"}},\"links\":[],\"mappings\":[],\"min\":0,\"thresholds\":{\"mode\":\"\
-    absolute\",\"steps\":[{\"color\":\"green\",\"value\":0},{\"color\":\"red\",\"\
-    value\":80}]},\"unit\":\"short\"},\"overrides\":[]},\"gridPos\":{\"h\":8,\"w\"\
-    :12,\"x\":0,\"y\":21},\"id\":107,\"options\":{\"legend\":{\"calcs\":[\"mean\"\
-    ,\"lastNotNull\",\"max\"],\"displayMode\":\"table\",\"placement\":\"bottom\",\"\
-    showLegend\":true,\"sortBy\":\"Last *\",\"sortDesc\":true},\"tooltip\":{\"hideZeros\"\
-    :false,\"mode\":\"multi\",\"sort\":\"desc\"}},\"pluginVersion\":\"12.2.0\",\"\
-    targets\":[{\"datasource\":{\"type\":\"prometheus\",\"uid\":\"$ds\"},\"editorMode\"\
-    :\"code\",\"exemplar\":true,\"expr\":\"sum(rate(vm_log_messages_total{job=~\\\"\
-    $job\\\",instance=~\\\"$instance\\\",level!=\\\"info\\\",cluster=~\\\"$cluster\\\
-    \"}[$__rate_interval])) by(job,level,cluster) > 0\",\"format\":\"time_series\"\
-    ,\"hide\":false,\"interval\":\"5m\",\"intervalFactor\":1,\"legendFormat\":\"{{job}}\
-    \ - {{level}}\",\"range\":true,\"refId\":\"A\"}],\"title\":\"Logging rate\",\"\
-    type\":\"timeseries\"},{\"datasource\":{\"type\":\"prometheus\",\"uid\":\"$ds\"\
-    },\"description\":\"Shows rate of dropped [samples](https://docs.victoriametrics.com/victoriametrics/keyconcepts/#raw-samples)\
+    :\"off\"}},\"links\":[{\"targetBlank\":true,\"title\":\"Drilldown\",\"url\":\"\
+    /d/G7Z9GzMGz?viewPanel=162&var-job=${__field.labels.job}&var-ds=$ds&${__url_time_range}\"\
+    }],\"mappings\":[],\"min\":0,\"thresholds\":{\"mode\":\"absolute\",\"steps\":[{\"\
+    color\":\"green\",\"value\":0},{\"color\":\"red\",\"value\":80}]},\"unit\":\"\
+    short\"},\"overrides\":[]},\"gridPos\":{\"h\":8,\"w\":12,\"x\":0,\"y\":21},\"\
+    id\":107,\"options\":{\"legend\":{\"calcs\":[\"mean\",\"lastNotNull\",\"max\"\
+    ],\"displayMode\":\"table\",\"placement\":\"bottom\",\"showLegend\":true,\"sortBy\"\
+    :\"Last *\",\"sortDesc\":true},\"tooltip\":{\"hideZeros\":false,\"mode\":\"multi\"\
+    ,\"sort\":\"desc\"}},\"pluginVersion\":\"12.2.0\",\"targets\":[{\"datasource\"\
+    :{\"type\":\"prometheus\",\"uid\":\"$ds\"},\"editorMode\":\"code\",\"exemplar\"\
+    :true,\"expr\":\"sum(rate(vm_log_messages_total{job=~\\\"$job\\\",instance=~\\\
+    \"$instance\\\",level!=\\\"info\\\",cluster=~\\\"$cluster\\\"}[$__rate_interval]))\
+    \ by(job,level,cluster) > 0\",\"format\":\"time_series\",\"hide\":false,\"interval\"\
+    :\"5m\",\"intervalFactor\":1,\"legendFormat\":\"{{job}} - {{level}}\",\"range\"\
+    :true,\"refId\":\"A\"}],\"title\":\"Logging rate\",\"type\":\"timeseries\"},{\"\
+    datasource\":{\"type\":\"prometheus\",\"uid\":\"$ds\"},\"description\":\"Shows\
+    \ rate of dropped [samples](https://docs.victoriametrics.com/victoriametrics/keyconcepts/#raw-samples)\
     \ from persistent queue. vmagent drops samples from queue if in-memory and on-disk\
     \ queues are full and it is unable to flush them to remote storage.\\nThe max\
     \ size of on-disk queue is configured by `-remoteWrite.maxDiskUsagePerURL` flag.\"\
     ,\"fieldConfig\":{\"defaults\":{\"color\":{\"mode\":\"palette-classic\"},\"custom\"\
     :{\"axisBorderShow\":false,\"axisCenteredZero\":false,\"axisColorMode\":\"text\"\
     ,\"axisLabel\":\"\",\"axisPlacement\":\"auto\",\"barAlignment\":0,\"barWidthFactor\"\
@@ -384,16 +386,16 @@

     ,\"axisPlacement\":\"auto\",\"barAlignment\":0,\"barWidthFactor\":0.6,\"drawStyle\"\
     :\"line\",\"fillOpacity\":0,\"gradientMode\":\"none\",\"hideFrom\":{\"legend\"\
     :false,\"tooltip\":false,\"viz\":false},\"insertNulls\":false,\"lineInterpolation\"\
     :\"linear\",\"lineWidth\":1,\"pointSize\":5,\"scaleDistribution\":{\"type\":\"\
     linear\"},\"showPoints\":\"never\",\"showValues\":false,\"spanNulls\":false,\"\
     stacking\":{\"group\":\"A\",\"mode\":\"none\"},\"thresholdsStyle\":{\"mode\":\"\
-    off\"}},\"links\":[{\"targetBlank\":true,\"title\":\"Drilldown\",\"url\":\"/d/G7Z9GzMGz?viewPanel=117&var-job=${__field.labels.job}&var-ds=$ds&var-instance=$instance&${__url_time_range}\"\
+    off\"}},\"links\":[{\"targetBlank\":true,\"title\":\"Drilldown\",\"url\":\"/d/G7Z9GzMGz?viewPanel=117&var-job=${__field.labels.job}&var-ds=$ds&${__url_time_range}\"\
     }],\"mappings\":[],\"min\":0,\"thresholds\":{\"mode\":\"absolute\",\"steps\":[{\"\
     color\":\"green\",\"value\":0},{\"color\":\"red\",\"value\":80}]},\"unit\":\"\
-    percentunit\"},\"overrides\":[]},\"gridPos\":{\"h\":8,\"w\":12,\"x\":0,\"y\":698},\"\
+    percentunit\"},\"overrides\":[]},\"gridPos\":{\"h\":8,\"w\":12,\"x\":0,\"y\":141},\"\
     id\":111,\"options\":{\"legend\":{\"calcs\":[\"mean\",\"lastNotNull\",\"max\"\
     ],\"displayMode\":\"table\",\"placement\":\"bottom\",\"showLegend\":true,\"sortBy\"\
     :\"Last *\",\"sortDesc\":true},\"tooltip\":{\"hideZeros\":false,\"mode\":\"multi\"\
     ,\"sort\":\"desc\"}},\"pluginVersion\":\"12.2.0\",\"targets\":[{\"datasource\"\
     :{\"type\":\"prometheus\",\"uid\":\"$ds\"},\"editorMode\":\"code\",\"exemplar\"\
     :true,\"expr\":\"max(max_over_time(process_resident_memory_bytes{job=~\\\"$job\\\
@@ -408,17 +410,17 @@

     barWidthFactor\":0.6,\"drawStyle\":\"line\",\"fillOpacity\":0,\"gradientMode\"\
[Diff truncated by flux-local]
--- HelmRelease: observability/victoria-metrics ClusterRole: observability/victoria-metrics-victoria-metrics-operator

+++ HelmRelease: observability/victoria-metrics ClusterRole: observability/victoria-metrics-victoria-metrics-operator

@@ -36,13 +36,12 @@

   - '*'
 - apiGroups:
   - ''
   resources:
   - configmaps/status
   - nodes
-  - nodes/proxy
   - nodes/metrics
   - namespaces
   verbs:
   - get
   - list
   - watch
@@ -167,12 +166,15 @@

   - vmauths
   - vmauths/finalizers
   - vmauths/status
   - vmclusters
   - vmclusters/finalizers
   - vmclusters/status
+  - vmdistributed
+  - vmdistributed/finalizers
+  - vmdistributed/status
   - vmnodescrapes
   - vmnodescrapes/finalizers
   - vmnodescrapes/status
   - vmpodscrapes
   - vmpodscrapes/finalizers
   - vmpodscrapes/status
--- HelmRelease: observability/victoria-metrics ClusterRole: observability/victoria-metrics-victoria-metrics-operator-victoriametrics-admin

+++ HelmRelease: observability/victoria-metrics ClusterRole: observability/victoria-metrics-victoria-metrics-operator-victoriametrics-admin

@@ -42,12 +42,15 @@

   - vmauths
   - vmauths/finalizers
   - vmauths/status
   - vmclusters
   - vmclusters/finalizers
   - vmclusters/status
+  - vmdistributed
+  - vmdistributed/finalizers
+  - vmdistributed/status
   - vmnodescrapes
   - vmnodescrapes/finalizers
   - vmnodescrapes/status
   - vmpodscrapes
   - vmpodscrapes/finalizers
   - vmpodscrapes/status
--- HelmRelease: observability/victoria-metrics ClusterRole: observability/victoria-metrics-victoria-metrics-operator-victoriametrics-view

+++ HelmRelease: observability/victoria-metrics ClusterRole: observability/victoria-metrics-victoria-metrics-operator-victoriametrics-view

@@ -42,12 +42,15 @@

   - vmauths
   - vmauths/finalizers
   - vmauths/status
   - vmclusters
   - vmclusters/finalizers
   - vmclusters/status
+  - vmdistributed
+  - vmdistributed/finalizers
+  - vmdistributed/status
   - vmnodescrapes
   - vmnodescrapes/finalizers
   - vmnodescrapes/status
   - vmpodscrapes
   - vmpodscrapes/finalizers
   - vmpodscrapes/status
--- HelmRelease: observability/victoria-metrics Deployment: observability/victoria-metrics-victoria-metrics-operator

+++ HelmRelease: observability/victoria-metrics Deployment: observability/victoria-metrics-victoria-metrics-operator

@@ -26,13 +26,13 @@

         fsGroup: 2000
         runAsNonRoot: true
         runAsUser: 1000
       serviceAccountName: victoria-metrics-victoria-metrics-operator
       containers:
       - name: operator
-        image: victoriametrics/operator:v0.67.0
+        image: victoriametrics/operator:v0.68.3
         imagePullPolicy: IfNotPresent
         env:
         - name: POD_NAME
           valueFrom:
             fieldRef:
               fieldPath: metadata.name
--- HelmRelease: observability/victoria-metrics VMAgent: observability/vm

+++ HelmRelease: observability/victoria-metrics VMAgent: observability/vm

@@ -13,14 +13,13 @@

   externalLabels:
     cluster: main
   extraArgs:
     promscrape.dropOriginalLabels: 'true'
     promscrape.streamParse: 'true'
   image:
-    tag: v1.135.0
-  license: {}
+    tag: v1.138.0
   port: '8429'
   remoteWrite:
   - url: http://vmsingle-vm.observability.svc.cluster.local:8428/api/v1/write
   replicas: 1
   resources:
     limits:
--- HelmRelease: observability/victoria-metrics VMRule: observability/vm-general.rules

+++ HelmRelease: observability/victoria-metrics VMRule: observability/vm-general.rules

@@ -47,11 +47,12 @@

           other alerts.
           This alert fires whenever there's a severity="info" alert, and stops firing when another alert with a
           severity of 'warning' or 'critical' starts firing on the same namespace.
           This alert should be routed to a null receiver and configured to inhibit alerts with severity="info".
         runbook_url: https://runbooks.prometheus-operator.dev/runbooks/general/infoinhibitor
         summary: Info-level alert inhibition.
-      expr: (ALERTS{severity="info"} == 1) unless on(namespace,cluster) (ALERTS{alertname!="InfoInhibitor",severity=~"warning|critical",alertstate="firing"}
-        == 1)
+      expr: group(ALERTS{severity="info"} == 1) by(namespace,cluster) unless on(namespace,cluster)
+        group(ALERTS{alertname!="InfoInhibitor",alertstate="firing",severity=~"warning|critical"}
+        == 1) by(namespace,cluster)
       labels:
         severity: none
 
--- HelmRelease: observability/victoria-metrics VMRule: observability/vm-kubernetes-apps

+++ HelmRelease: observability/victoria-metrics VMRule: observability/vm-kubernetes-apps

@@ -218,14 +218,16 @@

       annotations:
         description: HPA {{ $labels.namespace }}/{{ $labels.horizontalpodautoscaler  }}
           has been running at max replicas for longer than 15 minutes on cluster {{
           $labels.cluster }}.
         runbook_url: https://runbooks.prometheus-operator.dev/runbooks/kubernetes/kubehpamaxedout
         summary: HPA is running at max replicas
-      expr: kube_horizontalpodautoscaler_status_current_replicas{namespace=~".*",job="kube-state-metrics"}
-        == kube_horizontalpodautoscaler_spec_max_replicas{namespace=~".*",job="kube-state-metrics"}
+      expr: (kube_horizontalpodautoscaler_status_current_replicas{namespace=~".*",job="kube-state-metrics"}
+        == kube_horizontalpodautoscaler_spec_max_replicas{namespace=~".*",job="kube-state-metrics"})
+        and on(namespace,horizontalpodautoscaler,cluster) (kube_horizontalpodautoscaler_spec_max_replicas{namespace=~".*",job="kube-state-metrics"}
+        != kube_horizontalpodautoscaler_spec_min_replicas{namespace=~".*",job="kube-state-metrics"})
       for: 15m
       labels:
         severity: warning
     - alert: KubePdbNotEnoughHealthyPods
       annotations:
         description: PDB {{ $labels.cluster }}/{{ $labels.namespace }}/{{ $labels.poddisruptionbudget
--- HelmRelease: observability/victoria-metrics VMRule: observability/vm-kubernetes-system-kubelet

+++ HelmRelease: observability/victoria-metrics VMRule: observability/vm-kubernetes-system-kubelet

@@ -174,14 +174,16 @@

       expr: increase(kubelet_server_expiration_renew_errors[5m]) > 0
       for: 15m
       labels:
         severity: warning
     - alert: KubeletDown
       annotations:
-        description: Kubelet has disappeared from Prometheus target discovery.
+        description: Kubelet has disappeared from Prometheus target discovery on cluster
+          {{ $labels.cluster }}.
         runbook_url: https://runbooks.prometheus-operator.dev/runbooks/kubernetes/kubeletdown
         summary: Target disappeared from Prometheus target discovery.
-      expr: absent(up{job="kubelet",metrics_path="/metrics"})
+      expr: count(kube_node_info{job="kube-state-metrics"}) by(cluster) unless on(cluster)
+        count(up{job="kubelet",metrics_path="/metrics"} == 1) by(cluster)
       for: 15m
       labels:
         severity: critical
 
--- HelmRelease: observability/victoria-metrics VMRule: observability/vm-node-exporter.rules

+++ HelmRelease: observability/victoria-metrics VMRule: observability/vm-node-exporter.rules

@@ -62,7 +62,27 @@

       record: instance:node_network_receive_drop_excluding_lo:rate5m
     - annotations: {}
       expr: sum(rate(node_network_transmit_drop_total{job="node-exporter",device!="lo"}[5m]))
         without(device)
       labels: {}
       record: instance:node_network_transmit_drop_excluding_lo:rate5m
+    - annotations: {}
+      expr: sum(rate(node_network_receive_bytes_total{job="node-exporter",device!~"lo|veth.+"}[5m]))
+        without(device)
+      labels: {}
+      record: instance:node_network_receive_bytes_physical:rate5m
+    - annotations: {}
+      expr: sum(rate(node_network_transmit_bytes_total{job="node-exporter",device!~"lo|veth.+"}[5m]))
+        without(device)
+      labels: {}
+      record: instance:node_network_transmit_bytes_physical:rate5m
+    - annotations: {}
+      expr: sum(rate(node_network_receive_drop_total{job="node-exporter",device!~"lo|veth.+"}[5m]))
+        without(device)
+      labels: {}
+      record: instance:node_network_receive_drop_physical:rate5m
+    - annotations: {}
+      expr: sum(rate(node_network_transmit_drop_total{job="node-exporter",device!~"lo|veth.+"}[5m]))
+        without(device)
+      labels: {}
+      record: instance:node_network_transmit_drop_physical:rate5m
 
--- HelmRelease: observability/victoria-metrics VMRule: observability/vm-vm-health

+++ HelmRelease: observability/victoria-metrics VMRule: observability/vm-vm-health

@@ -79,16 +79,16 @@

       for: 15m
       labels:
         severity: critical
     - alert: TooManyLogs
       annotations:
         description: |
-          Logging rate for job \"{{ $labels.job }}\" ({{ $labels.instance }}) is {{ $value }} for last 15m. Worth to check logs for specific error messages.
-        summary: Too many logs printed for job "{{ $labels.job }}" ({{ $labels.instance
+          The job \"{{ $labels.job }}\" ({{ $labels.instance }}) generated {{ $value }} log messages with the level higher than info for the last 5 minutes. Check the logs for the given target. Check also the \"location\" label at the vm_log_messages_total metric if -loggerLevel command-line flag is set to value other than INFO. This label contains code locations responsible for generating log messages suppressed by -loggerLevel.
+        summary: Too many logs are generated for job "{{ $labels.job }}" ({{ $labels.instance
           }})
-      expr: sum(increase(vm_log_messages_total{level="error"}[5m])) without(app_version,location)
+      expr: sum(increase(vm_log_messages_total{level!="info"}[5m])) without(app_version,location,is_printed)
         > 0
       for: 15m
       labels:
         severity: warning
     - alert: TooManyTSIDMisses
       annotations:
--- HelmRelease: observability/victoria-metrics VMRule: observability/vm-vmagent

+++ HelmRelease: observability/victoria-metrics VMRule: observability/vm-vmagent

@@ -15,40 +15,40 @@

     interval: 30s
     name: vmagent
     params: {}
     rules:
     - alert: PersistentQueueIsDroppingData
       annotations:
-        dashboard: http:///d/G7Z9GzMGz?viewPanel=49&var-instance={{ $labels.instance
-          }}&var-cluster={{ $labels.cluster }}
+        dashboard: '{{ $externalURL }}/d/G7Z9GzMGz?viewPanel=49&var-instance={{ $labels.instance
+          }}'
         description: Vmagent dropped {{ $value | humanize1024 }} from persistent queue
           on instance {{ $labels.instance }} for the last 10m.
         summary: Instance {{ $labels.instance }} is dropping data from persistent
           queue
       expr: sum(increase(vm_persistentqueue_bytes_dropped_total[5m])) without(path)
         > 0
       for: 10m
       labels:
         severity: critical
     - alert: RejectedRemoteWriteDataBlocksAreDropped
       annotations:
-        dashboard: http:///d/G7Z9GzMGz?viewPanel=79&var-instance={{ $labels.instance
-          }}&var-cluster={{ $labels.cluster }}
+        dashboard: '{{ $externalURL }}/d/G7Z9GzMGz?viewPanel=79&var-instance={{ $labels.instance
+          }}'
         description: Job "{{ $labels.job }}" on instance {{ $labels.instance }} drops
           the rejected by remote-write server data blocks. Check the logs to find
           the reason for rejects.
         summary: Vmagent is dropping data blocks that are rejected by remote storage
       expr: sum(increase(vmagent_remotewrite_packets_dropped_total[5m])) without(url)
         > 0
       for: 15m
       labels:
         severity: warning
     - alert: TooManyScrapeErrors
       annotations:
-        dashboard: http:///d/G7Z9GzMGz?viewPanel=31&var-instance={{ $labels.instance
-          }}&var-cluster={{ $labels.cluster }}
+        dashboard: '{{ $externalURL }}/d/G7Z9GzMGz?viewPanel=31&var-instance={{ $labels.instance
+          }}'
         description: Job "{{ $labels.job }}" on instance {{ $labels.instance }} fails
           to scrape targets for last 15m
         summary: Vmagent fails to scrape one or more targets
       expr: increase(vm_promscrape_scrapes_failed_total[5m]) > 0
       for: 15m
       labels:
@@ -64,40 +64,40 @@

         0
       for: 30m
       labels:
         severity: warning
     - alert: TooManyWriteErrors
       annotations:
-        dashboard: http:///d/G7Z9GzMGz?viewPanel=77&var-instance={{ $labels.instance
-          }}&var-cluster={{ $labels.cluster }}
+        dashboard: '{{ $externalURL }}/d/G7Z9GzMGz?viewPanel=77&var-instance={{ $labels.instance
+          }}'
         description: Job "{{ $labels.job }}" on instance {{ $labels.instance }} responds
           with errors to write requests for last 15m.
         summary: Vmagent responds with too many errors on data ingestion protocols
       expr: (sum(increase(vm_ingestserver_request_errors_total[5m])) without(name,net,type)
         + sum(increase(vmagent_http_request_errors_total[5m])) without(path,protocol))
         > 0
       for: 15m
       labels:
         severity: warning
     - alert: TooManyRemoteWriteErrors
       annotations:
-        dashboard: http:///d/G7Z9GzMGz?viewPanel=61&var-instance={{ $labels.instance
-          }}&var-cluster={{ $labels.cluster }}
+        dashboard: '{{ $externalURL }}/d/G7Z9GzMGz?viewPanel=61&var-instance={{ $labels.instance
+          }}'
         description: |-
           Vmagent fails to push data via remote write protocol to destination "{{ $labels.url }}"
            Ensure that destination is up and reachable.
         summary: Job "{{ $labels.job }}" on instance {{ $labels.instance }} fails
           to push to remote storage
       expr: rate(vmagent_remotewrite_retries_count_total[5m]) > 0
       for: 15m
       labels:
         severity: warning
     - alert: RemoteWriteConnectionIsSaturated
       annotations:
-        dashboard: http:///d/G7Z9GzMGz?viewPanel=84&var-instance={{ $labels.instance
-          }}&var-cluster={{ $labels.cluster }}
+        dashboard: '{{ $externalURL }}/d/G7Z9GzMGz?viewPanel=84&var-instance={{ $labels.instance
+          }}'
         description: |-
           The remote write connection between vmagent "{{ $labels.job }}" (instance {{ $labels.instance }}) and destination "{{ $labels.url }}" is saturated by more than 90% and vmagent won't be able to keep up.
            There could be the following reasons for this:
            * vmagent can't send data fast enough through the existing network connections. Increase `-remoteWrite.queues` cmd-line flag value to establish more connections per destination.
            * remote destination can't accept data fast enough. Check if remote destination has enough resources for processing.
         summary: Remote write connection from "{{ $labels.job }}" (instance {{ $labels.instance
@@ -106,52 +106,52 @@

         > 0.9
       for: 15m
       labels:
         severity: warning
     - alert: PersistentQueueForWritesIsSaturated
       annotations:
-        dashboard: http:///d/G7Z9GzMGz?viewPanel=98&var-instance={{ $labels.instance
-          }}&var-cluster={{ $labels.cluster }}
+        dashboard: '{{ $externalURL }}/d/G7Z9GzMGz?viewPanel=98&var-instance={{ $labels.instance
+          }}'
         description: Persistent queue writes for vmagent "{{ $labels.job }}" (instance
           {{ $labels.instance }}) are saturated by more than 90% and vmagent won't
           be able to keep up with flushing data on disk. In this case, consider to
           decrease load on the vmagent or improve the disk throughput.
         summary: Persistent queue writes for instance {{ $labels.instance }} are saturated
       expr: rate(vm_persistentqueue_write_duration_seconds_total[5m]) > 0.9
       for: 15m
       labels:
         severity: warning
     - alert: PersistentQueueForReadsIsSaturated
       annotations:
-        dashboard: http:///d/G7Z9GzMGz?viewPanel=99&var-instance={{ $labels.instance
-          }}&var-cluster={{ $labels.cluster }}
+        dashboard: '{{ $externalURL }}/d/G7Z9GzMGz?viewPanel=99&var-instance={{ $labels.instance
+          }}'
         description: Persistent queue reads for vmagent "{{ $labels.job }}" (instance
           {{ $labels.instance }}) are saturated by more than 90% and vmagent won't
           be able to keep up with reading data from the disk. In this case, consider
           to decrease load on the vmagent or improve the disk throughput.
         summary: Persistent queue reads for instance {{ $labels.instance }} are saturated
       expr: rate(vm_persistentqueue_read_duration_seconds_total[5m]) > 0.9
       for: 15m
       labels:
         severity: warning
     - alert: SeriesLimitHourReached
       annotations:
-        dashboard: http:///d/G7Z9GzMGz?viewPanel=88&var-instance={{ $labels.instance
-          }}&var-cluster={{ $labels.cluster }}
+        dashboard: '{{ $externalURL }}/d/G7Z9GzMGz?viewPanel=88&var-instance={{ $labels.instance
+          }}'
         description: Max series limit set via -remoteWrite.maxHourlySeries flag is
           close to reaching the max value. Then samples for new time series will be
           dropped instead of sending them to remote storage systems.
         summary: Instance {{ $labels.instance }} reached 90% of the limit
       expr: (vmagent_hourly_series_limit_current_series / vmagent_hourly_series_limit_max_series)
         > 0.9
       labels:
         severity: critical
     - alert: SeriesLimitDayReached
       annotations:
-        dashboard: http:///d/G7Z9GzMGz?viewPanel=90&var-instance={{ $labels.instance
-          }}&var-cluster={{ $labels.cluster }}
+        dashboard: '{{ $externalURL }}/d/G7Z9GzMGz?viewPanel=90&var-instance={{ $labels.instance
+          }}'
         description: Max series limit set via -remoteWrite.maxDailySeries flag is
           close to reaching the max value. Then samples for new time series will be
           dropped instead of sending them to remote storage systems.
         summary: Instance {{ $labels.instance }} reached 90% of the limit
       expr: (vmagent_daily_series_limit_current_series / vmagent_daily_series_limit_max_series)
         > 0.9
@@ -186,7 +186,39 @@

           deduplicate smaller number of series; reduce samples'' ingestion rate.'
         summary: Deduplication "{{ $labels.job }}" (instance {{ $labels.instance }})
           can't be finished within configured deduplication interval.
       expr: increase(vm_streamaggr_dedup_flush_timeouts_total[5m]) > 0
       labels:
         severity: warning
+    - alert: PersistentQueueRunsOutOfSpaceIn12Hours
+      annotations:
+        description: RemoteWrite destination ({{ $labels.url }}) is unavailable or
+          unable to receive data in a timely manner, so the persistent queue size
+          is growing. Once the available space is exhausted, some samples will be
+          discarded and cause incident. Please check the health of remoteWrite destination
+          ({{ $labels.url }}).
+        summary: Persistent Queue (url {{ $labels.url }}) of {{ $labels.instance }}
+          (job:{{ $labels.job }}) will run out of space in 12 hours.
+      expr: (((sum(vm_persistentqueue_free_disk_space_bytes) by(job,instance,path,cluster)
+        / (sum(deriv(vm_persistentqueue_bytes_pending[1m])) by(job,instance,path,cluster)
+        > 0)) * on(job,instance,path,cluster) group_left(url) ((vmagent_remotewrite_pending_data_bytes
+        * 0) + 1)) < 43200) > 0
+      for: 10m
+      labels:
+        severity: warning
+    - alert: PersistentQueueRunsOutOfSpaceIn4Hours
+      annotations:
+        description: RemoteWrite destination ({{ $labels.url }}) is unavailable or
+          unable to receive data in a timely manner, so the persistent queue size
+          is growing. Once the available space is exhausted, some samples will be
+          discarded and cause incident. Please check the health of remoteWrite destination
+          ({{ $labels.url }}).
[Diff truncated by flux-local]
--- HelmRelease: observability/victoria-metrics VMRule: observability/vm-vmalert

+++ HelmRelease: observability/victoria-metrics VMRule: observability/vm-vmalert

@@ -23,43 +23,41 @@

           }}
       expr: vmalert_config_last_reload_successful != 1
       labels:
         severity: warning
     - alert: AlertingRulesError
       annotations:
-        dashboard: http:///d/LzldHAVnz?viewPanel=13&var-instance={{ $labels.instance
-          }}&var-file={{ $labels.file }}&var-group={{ $labels.group }}&var-cluster={{
-          $labels.cluster }}
+        dashboard: '{{ $externalURL }}/d/LzldHAVnz?viewPanel=13&var-instance={{ $labels.instance
+          }}&var-file={{ $labels.file }}&var-group={{ $labels.group }}'
         description: Alerting rules execution is failing for "{{ $labels.alertname
           }}" from group "{{ $labels.group }}" in file "{{ $labels.file }}". Check
           vmalert's logs for detailed error message.
         summary: Alerting rules are failing for vmalert instance {{ $labels.instance
           }}
       expr: sum(increase(vmalert_alerting_rules_errors_total[5m])) without(id) > 0
       for: 5m
       labels:
         severity: warning
     - alert: RecordingRulesError
       annotations:
-        dashboard: http:///d/LzldHAVnz?viewPanel=30&var-instance={{ $labels.instance
-          }}&var-file={{ $labels.file }}&var-group={{ $labels.group }}&var-cluster={{
-          $labels.cluster }}
+        dashboard: '{{ $externalURL }}/d/LzldHAVnz?viewPanel=30&var-instance={{ $labels.instance
+          }}&var-file={{ $labels.file }}&var-group={{ $labels.group }}'
         description: Recording rules execution is failing for "{{ $labels.recording
           }}" from group "{{ $labels.group }}" in file "{{ $labels.file }}". Check
           vmalert's logs for detailed error message.
         summary: Recording rules are failing for vmalert instance {{ $labels.instance
           }}
       expr: sum(increase(vmalert_recording_rules_errors_total[5m])) without(id) >
         0
       for: 5m
       labels:
         severity: warning
     - alert: RecordingRulesNoData
       annotations:
-        dashboard: http:///d/LzldHAVnz?viewPanel=33&var-file={{ $labels.file }}&var-group={{
-          $labels.group }}&var-cluster={{ $labels.cluster }}
+        dashboard: '{{ $externalURL }}/d/LzldHAVnz?viewPanel=33&var-file={{ $labels.file
+          }}&var-group={{ $labels.group }}'
         description: Recording rule "{{ $labels.recording }}" from group "{{ $labels.group
           }} in file "{{ $labels.file }}" produces 0 samples over the last 30min.
           It might be caused by a misconfiguration or incorrect query expression.
         summary: Recording rule {{ $labels.recording }} ({{ $labels.group }}) produces
           no data
       expr: sum(vmalert_recording_rules_last_evaluation_samples) without(id) < 1
--- HelmRelease: observability/victoria-metrics VMRule: observability/vm-vmcluster

+++ HelmRelease: observability/victoria-metrics VMRule: observability/vm-vmcluster

@@ -15,14 +15,14 @@

     interval: 30s
     name: vmcluster
     params: {}
     rules:
     - alert: DiskRunsOutOfSpaceIn3Days
       annotations:
-        dashboard: http:///d/oS7Bi_0Wz?viewPanel=20&var-instance={{ $labels.instance
-          }}&var-cluster={{ $labels.cluster }}
+        dashboard: '{{ $externalURL }}/d/oS7Bi_0Wz?viewPanel=20&var-instance={{ $labels.instance
+          }}'
         description: |-
           Taking into account current ingestion rate, free disk space will be enough only for {{ $value | humanizeDuration }} on instance {{ $labels.instance }}.
            Consider to limit the ingestion rate, decrease retention or scale the disk space up if possible.
         summary: Instance {{ $labels.instance }} will run out of disk space in 3 days
       expr: ((sum(vm_free_disk_space_bytes) without(path) / (((rate(vm_rows_added_to_storage_total[1d])
         - sum(rate(vm_deduplicated_samples_total[1d])) without(type)) * (sum(vm_data_size_bytes{type!~"indexdb.*"})
@@ -31,14 +31,14 @@

         without(type))))) < 259200) > 0
       for: 30m
       labels:
         severity: critical
     - alert: NodeBecomesReadonlyIn3Days
       annotations:
-        dashboard: http:///d/oS7Bi_0Wz?viewPanel=20&var-instance={{ $labels.instance
-          }}&var-cluster={{ $labels.cluster }}
+        dashboard: '{{ $externalURL }}/d/oS7Bi_0Wz?viewPanel=20&var-instance={{ $labels.instance
+          }}'
         description: |-
           Taking into account current ingestion rate, free disk space and -storage.minFreeDiskSpaceBytes instance {{ $labels.instance }} will remain writable for {{ $value | humanizeDuration }}.
            Consider to limit the ingestion rate, decrease retention or scale the disk space up if possible.
         summary: Instance {{ $labels.instance }} will become read-only in 3 days
       expr: ((sum(vm_free_disk_space_bytes - vm_free_disk_space_limit_bytes) without(path)
         / (((rate(vm_rows_added_to_storage_total[1d]) - sum(rate(vm_deduplicated_samples_total[1d]))
@@ -48,14 +48,14 @@

         without(type))))) < 259200) > 0
       for: 30m
       labels:
         severity: warning
     - alert: DiskRunsOutOfSpace
       annotations:
-        dashboard: http:///d/oS7Bi_0Wz?viewPanel=20&var-instance={{ $labels.instance
-          }}&var-cluster={{ $labels.cluster }}
+        dashboard: '{{ $externalURL }}/d/oS7Bi_0Wz?viewPanel=20&var-instance={{ $labels.instance
+          }}'
         description: |-
           Disk utilisation on instance {{ $labels.instance }} is more than 80%.
            Having less than 20% of free disk space could cripple merges processes and overall performance. Consider to limit the ingestion rate, decrease retention or scale the disk space if possible.
         summary: Instance {{ $labels.instance }} (job={{ $labels.job }}) will run
           out of disk space soon
       expr: (sum(vm_data_size_bytes) by(job,instance,cluster) / (sum(vm_free_disk_space_bytes)
@@ -63,27 +63,27 @@

         > 0.8
       for: 30m
       labels:
         severity: critical
     - alert: RequestErrorsToAPI
       annotations:
-        dashboard: http:///d/oS7Bi_0Wz?viewPanel=52&var-instance={{ $labels.instance
-          }}&var-cluster={{ $labels.cluster }}
+        dashboard: '{{ $externalURL }}/d/oS7Bi_0Wz?viewPanel=52&var-instance={{ $labels.instance
+          }}'
         description: Requests to path {{ $labels.path }} are receiving errors. Please
           verify if clients are sending correct requests.
         summary: Too many errors served for {{ $labels.job }} path {{ $labels.path
           }} (instance {{ $labels.instance }})
       expr: increase(vm_http_request_errors_total[5m]) > 0
       for: 15m
       labels:
         severity: warning
         show_at: dashboard
     - alert: RPCErrors
       annotations:
-        dashboard: http:///d/oS7Bi_0Wz?viewPanel=44&var-instance={{ $labels.instance
-          }}&var-cluster={{ $labels.cluster }}
+        dashboard: '{{ $externalURL }}/d/oS7Bi_0Wz?viewPanel=44&var-instance={{ $labels.instance
+          }}'
         description: |-
           RPC errors are interconnection errors between cluster components.
            Possible reasons for errors are misconfiguration, overload, network blips or unreachable components.
         summary: Too many RPC errors for {{ $labels.job }} (instance {{ $labels.instance
           }})
       expr: ((sum(increase(vm_rpc_connection_errors_total[5m])) by(job,instance,cluster)
@@ -93,55 +93,52 @@

       for: 15m
       labels:
         severity: warning
         show_at: dashboard
     - alert: TooHighChurnRate
       annotations:
-        dashboard: http:///d/oS7Bi_0Wz?viewPanel=102&var-cluster={{ $labels.cluster
-          }}
+        dashboard: '{{ $externalURL }}/d/oS7Bi_0Wz?viewPanel=102'
         description: |-
           VM constantly creates new time series.
            This effect is known as Churn Rate.
            High Churn Rate tightly connected with database performance and may result in unexpected OOM's or slow queries.
         summary: Churn rate is more than 10% for the last 15m
       expr: (sum(rate(vm_new_timeseries_created_total[5m])) by(job,cluster) / sum(rate(vm_rows_inserted_total[5m]))
         by(job,cluster)) > 0.1
       for: 15m
       labels:
         severity: warning
     - alert: TooHighChurnRate24h
       annotations:
-        dashboard: http:///d/oS7Bi_0Wz?viewPanel=102&var-cluster={{ $labels.cluster
-          }}
+        dashboard: '{{ $externalURL }}/d/oS7Bi_0Wz?viewPanel=102'
         description: |-
           The number of created new time series over last 24h is 3x times higher than current number of active series.
            This effect is known as Churn Rate.
            High Churn Rate tightly connected with database performance and may result in unexpected OOM's or slow queries.
         summary: Too high number of new series created over last 24h
       expr: sum(increase(vm_new_timeseries_created_total[24h])) by(job,cluster) >
         (sum(vm_cache_entries{type="storage/hour_metric_ids"}) by(job,cluster) * 3)
       for: 15m
       labels:
         severity: warning
     - alert: TooHighSlowInsertsRate
       annotations:
-        dashboard: http:///d/oS7Bi_0Wz?viewPanel=108&var-cluster={{ $labels.cluster
-          }}
+        dashboard: '{{ $externalURL }}/d/oS7Bi_0Wz?viewPanel=108'
         description: High rate of slow inserts may be a sign of resource exhaustion
           for the current load. It is likely more RAM is needed for optimal handling
           of the current number of active time series. See also https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3976#issuecomment-1476883183
         summary: Percentage of slow inserts is more than 5% for the last 15m
       expr: (sum(rate(vm_slow_row_inserts_total[5m])) by(job,cluster) / sum(rate(vm_rows_inserted_total[5m]))
         by(job,cluster)) > 0.05
       for: 15m
       labels:
         severity: warning
     - alert: VminsertVmstorageConnectionIsSaturated
       annotations:
-        dashboard: http:///d/oS7Bi_0Wz?viewPanel=139&var-instance={{ $labels.instance
-          }}&var-cluster={{ $labels.cluster }}
+        dashboard: '{{ $externalURL }}/d/oS7Bi_0Wz?viewPanel=139&var-instance={{ $labels.instance
+          }}'
         description: |-
           The connection between vminsert (instance {{ $labels.instance }}) and vmstorage (instance {{ $labels.addr }}) is saturated by more than 90% and vminsert won't be able to keep up.
            This usually means that more vminsert or vmstorage nodes must be added to the cluster in order to increase the total number of vminsert -> vmstorage links.
         summary: Connection between vminsert on {{ $labels.instance }} and vmstorage
           on {{ $labels.addr }} is saturated
       expr: rate(vm_rpc_send_duration_seconds_total[5m]) > 0.9
--- HelmRelease: observability/victoria-metrics VMRule: observability/vm-vmsingle

+++ HelmRelease: observability/victoria-metrics VMRule: observability/vm-vmsingle

@@ -15,14 +15,14 @@

     interval: 30s
     name: vmsingle
     params: {}
     rules:
     - alert: DiskRunsOutOfSpaceIn3Days
       annotations:
-        dashboard: http:///d/wNf0q_kZk?viewPanel=53&var-instance={{ $labels.instance
-          }}&var-cluster={{ $labels.cluster }}
+        dashboard: '{{ $externalURL }}/d/wNf0q_kZk?viewPanel=53&var-instance={{ $labels.instance
+          }}'
         description: |-
           Taking into account current ingestion rate, free disk space will be enough only for {{ $value | humanizeDuration }} on instance {{ $labels.instance }}.
            Consider to limit the ingestion rate, decrease retention or scale the disk space if possible.
         summary: Instance {{ $labels.instance }} will run out of disk space soon
       expr: ((sum(vm_free_disk_space_bytes) without(path) / (((rate(vm_rows_added_to_storage_total[1d])
         - sum(rate(vm_deduplicated_samples_total[1d])) without(type)) * (sum(vm_data_size_bytes{type!~"indexdb.*"})
@@ -31,14 +31,14 @@

         without(type))))) < 259200) > 0
       for: 30m
       labels:
         severity: critical
     - alert: NodeBecomesReadonlyIn3Days
       annotations:
-        dashboard: http:///d/oS7Bi_0Wz?viewPanel=53&var-instance={{ $labels.instance
-          }}&var-cluster={{ $labels.cluster }}
+        dashboard: '{{ $externalURL }}/d/oS7Bi_0Wz?viewPanel=53&var-instance={{ $labels.instance
+          }}'
         description: |-
           Taking into account current ingestion rate and free disk space instance {{ $labels.instance }} is writable for {{ $value | humanizeDuration }}.
            Consider to limit the ingestion rate, decrease retention or scale the disk space up if possible.
         summary: Instance {{ $labels.instance }} will become read-only in 3 days
       expr: ((sum(vm_free_disk_space_bytes - vm_free_disk_space_limit_bytes) without(path)
         / (((rate(vm_rows_added_to_storage_total[1d]) - sum(rate(vm_deduplicated_samples_total[1d]))
@@ -48,14 +48,14 @@

         without(type))))) < 259200) > 0
       for: 30m
       labels:
         severity: warning
     - alert: DiskRunsOutOfSpace
       annotations:
-        dashboard: http:///d/wNf0q_kZk?viewPanel=53&var-instance={{ $labels.instance
-          }}&var-cluster={{ $labels.cluster }}
+        dashboard: '{{ $externalURL }}/d/wNf0q_kZk?viewPanel=53&var-instance={{ $labels.instance
+          }}'
         description: |-
           Disk utilisation on instance {{ $labels.instance }} is more than 80%.
            Having less than 20% of free disk space could cripple merge processes and overall performance. Consider to limit the ingestion rate, decrease retention or scale the disk space if possible.
         summary: Instance {{ $labels.instance }} (job={{ $labels.job }}) will run
           out of disk space soon
       expr: (sum(vm_data_size_bytes) by(job,instance,cluster) / (sum(vm_free_disk_space_bytes)
@@ -63,26 +63,26 @@

         > 0.8
       for: 30m
       labels:
         severity: critical
     - alert: RequestErrorsToAPI
       annotations:
-        dashboard: http:///d/wNf0q_kZk?viewPanel=35&var-instance={{ $labels.instance
-          }}&var-cluster={{ $labels.cluster }}
+        dashboard: '{{ $externalURL }}/d/wNf0q_kZk?viewPanel=35&var-instance={{ $labels.instance
+          }}'
         description: Requests to path {{ $labels.path }} are receiving errors. Please
           verify if clients are sending correct requests.
         summary: Too many errors served for path {{ $labels.path }} (instance {{ $labels.instance
           }})
       expr: increase(vm_http_request_errors_total[5m]) > 0
       for: 15m
       labels:
         severity: warning
     - alert: TooHighChurnRate
       annotations:
-        dashboard: http:///d/wNf0q_kZk?viewPanel=66&var-instance={{ $labels.instance
-          }}&var-cluster={{ $labels.cluster }}
+        dashboard: '{{ $externalURL }}/d/wNf0q_kZk?viewPanel=66&var-instance={{ $labels.instance
+          }}'
         description: |-
           VM constantly creates new time series on "{{ $labels.instance }}".
            This effect is known as Churn Rate.
            High Churn Rate is tightly connected with database performance and may result in unexpected OOM's or slow queries.
         summary: Churn rate is more than 10% on "{{ $labels.instance }}" for the last
           15m
@@ -90,14 +90,14 @@

         sum(rate(vm_rows_inserted_total[5m])) by(instance,cluster)) > 0.1
       for: 15m
       labels:
         severity: warning
     - alert: TooHighChurnRate24h
       annotations:
-        dashboard: http:///d/wNf0q_kZk?viewPanel=66&var-instance={{ $labels.instance
-          }}&var-cluster={{ $labels.cluster }}
+        dashboard: '{{ $externalURL }}/d/wNf0q_kZk?viewPanel=66&var-instance={{ $labels.instance
+          }}'
         description: |-
           The number of created new time series over last 24h is 3x times higher than current number of active series on "{{ $labels.instance }}".
            This effect is known as Churn Rate.
            High Churn Rate is tightly connected with database performance and may result in unexpected OOM's or slow queries.
         summary: Too high number of new series on "{{ $labels.instance }}" created
           over last 24h
@@ -106,20 +106,35 @@

         * 3)
       for: 15m
       labels:
         severity: warning
     - alert: TooHighSlowInsertsRate
       annotations:
-        dashboard: http:///d/wNf0q_kZk?viewPanel=68&var-instance={{ $labels.instance
-          }}&var-cluster={{ $labels.cluster }}
+        dashboard: '{{ $externalURL }}/d/wNf0q_kZk?viewPanel=68&var-instance={{ $labels.instance
+          }}'
         description: High rate of slow inserts on "{{ $labels.instance }}" may be
           a sign of resource exhaustion for the current load. It is likely more RAM
           is needed for optimal handling of the current number of active time series.
           See also https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3976#issuecomment-1476883183
         summary: Percentage of slow inserts is more than 5% on "{{ $labels.instance
           }}" for the last 15m
       expr: (sum(rate(vm_slow_row_inserts_total[5m])) by(instance,cluster) / sum(rate(vm_rows_inserted_total[5m]))
         by(instance,cluster)) > 0.05
       for: 15m
       labels:
         severity: warning
+    - alert: MetadataCacheUtilizationIsTooHigh
+      annotations:
+        description: Metadata cache stores meta information about ingested time series
+          - see https://docs.victoriametrics.com/victoriametrics/#metrics-metadata.
+          When cache is overutilized, the oldest entries will be dropped out automatically.
+          It may result into incomplete response for /api/v1/metadata API calls. It
+          doesn't impact regular queries or alerts. Cache size is controlled via -storage.maxMetadataStorageSize
+          cmd-line flag.
+        summary: Metadata cache capacity on {{ $labels.instance }} (job={{ $labels.job
+          }}) is utilized for more than 95% for the last 15min
+      expr: (vm_metrics_metadata_storage_size_bytes / vm_metrics_metadata_storage_max_size_bytes)
+        > 0.95
+      for: 15m
+      labels:
+        severity: warning
 
--- HelmRelease: observability/victoria-metrics VMSingle: observability/vm

+++ HelmRelease: observability/victoria-metrics VMSingle: observability/vm

@@ -9,14 +9,13 @@

     app.kubernetes.io/instance: victoria-metrics
     app.kubernetes.io/managed-by: Helm
     app.kubernetes.io/name: victoria-metrics-k8s-stack
 spec:
   extraArgs: {}
   image:
-    tag: v1.135.0
-  license: {}
+    tag: v1.138.0
   port: '8428'
   replicaCount: 1
   resources:
     limits:
       memory: 2Gi
     requests:
--- HelmRelease: observability/victoria-metrics ValidatingWebhookConfiguration: observability/victoria-metrics-victoria-metrics-operator-admission

+++ HelmRelease: observability/victoria-metrics ValidatingWebhookConfiguration: observability/victoria-metrics-victoria-metrics-operator-admission

@@ -11,13 +11,13 @@

 - clientConfig:
     service:
       namespace: observability
       name: victoria-metrics-victoria-metrics-operator
       path: /validate-operator-victoriametrics-com-v1-vlagent
       port: 9443
-    caBundle: LS0tLS1CRUdJTiBDRVJUSUZJQ0FURS0tLS0tCk1JSURIVENDQWdXZ0F3SUJBZ0lRWXhMcHE3eXNTRXBrMk1zWldNVUc4REFOQmdrcWhraUc5dzBCQVFzRkFEQVoKTVJjd0ZRWURWUVFERXc1MmJTMXZjR1Z5WVhSdmNpMWpZVEFlRncweU5qQXpNVFl4TnpVMk1URmFGdzB6TmpBegpNVE14TnpVMk1URmFNQmt4RnpBVkJnTlZCQU1URG5adExXOXdaWEpoZEc5eUxXTmhNSUlCSWpBTkJna3Foa2lHCjl3MEJBUUVGQUFPQ0FROEFNSUlCQ2dLQ0FRRUFzamdMazU1U1hvdGpBTGF1Sy80RmpuQlZEVlRJRFRjOEVCclMKNTJwMmFqRlpDL3BTV0VqNWYvdGJkcFc2SDdKZVE4ZlFJWmdqYkdEMTMxeWpJajVEbnlIZ0Q4NnR3NHBHL3NDNwpRQ3NXYXJZV25DYUs5aVBMa2FoYWZVOTJqSU1FaTJvcTJUM0dUSUtvWHFrMmdkaGhHUmIyc2NZdWZqaExwVFdSCi9JT3pIeGxrWmFQZkhVc1RWbm1Bc3NxOEk4WDMzQnBzaXRSNy9YSVdKbU4vWjM1VlEyeXkxYkgydEFPQU9IK2MKK0J6aG13YmZmaWhvZTZlTWN2c29qa3YwaTQvSUNub2dXdUw1RVk0bGNwN0kzTWJWYWdIVTJIS09kS1BsTGp4Zgo2YU9hWFh0enNsdFZZS2s5ZGJNSXJJalJsY25MazhKZm1XeVdqU2F1d0FFaGpUczhDUUlEQVFBQm8yRXdYekFPCkJnTlZIUThCQWY4RUJBTUNBcVF3SFFZRFZSMGxCQll3RkFZSUt3WUJCUVVIQXdFR0NDc0dBUVVGQndNQ01BOEcKQTFVZEV3RUIvd1FGTUFNQkFmOHdIUVlEVlIwT0JCWUVGQjZJeVgxVFcvQXJLT1M3Z0NXZmkxVWloNnhETUEwRwpDU3FHU0liM0RRRUJDd1VBQTRJQkFRQWsxZ0ZxRHdyUEJ4UXhnU2VaR3poUmdXbzgyWTJGSHJoWFNzRFk1N2szClVHMlREUEVzOUpBNFcxNlE5WUUzT1VhRWViTjF5eHNXRmVHREFyRXQrRHlUcU8yUHNGSFoycjJveFBVcVBTZUgKdU1vRVFLcVJHcjg1L2VuZ3hseGVOeG9COVpKWmxLb2ljQzRKTG83aHlwMDBjQ3p1cFlSQjBDUVhUY3MyMFU3TQowN3hkVjdGVEtQelhzOVAwd1JuNXFqS3k5V0RjK21TYWE0SUpXeHUyODFIU3RwWE0vTDl1WlIyWlRTVTd0SnpoCi9DL1RkdEFOblAxM21KQXlHUjErZnI2bnVRRTVJTVdBRnVoeCs0YVpQTE1jRGJtYko1enlUZUJlSE05YW9yZjQKRHEyVWRGV0VnbFRoTjd0VG4vOTVtb0QzdjlCbExlYVI4QTJOTUtlOVZsY3MKLS0tLS1FTkQgQ0VSVElGSUNBVEUtLS0tLQo=
+    caBundle: LS0tLS1CRUdJTiBDRVJUSUZJQ0FURS0tLS0tCk1JSURIakNDQWdhZ0F3SUJBZ0lSQU5ZMFVjd3ZVejlpelNVQTh2OFVSSEl3RFFZSktvWklodmNOQVFFTEJRQXcKR1RFWE1CVUdBMVVFQXhNT2RtMHRiM0JsY21GMGIzSXRZMkV3SGhjTk1qWXdNekUyTVRjMU5qRXhXaGNOTXpZdwpNekV6TVRjMU5qRXhXakFaTVJjd0ZRWURWUVFERXc1MmJTMXZjR1Z5WVhSdmNpMWpZVENDQVNJd0RRWUpLb1pJCmh2Y05BUUVCQlFBRGdnRVBBRENDQVFvQ2dnRUJBTmZJTFV6YTdGTkJpd2hJK1hORXdhdGlrVFFXVjVTWUZicU4KSjZjUDdDTTk1WjhaczRpWGN2WTJoc2s0RnFUdVo0bHVoaDF0NnA1K1RVODZDaUppWW5mTmVWRFFKK2U1TlFvRwp6elpNajdISjFhSVdzUER0UHBFRTFYNmJLaE00ZWlVcGhpS2hnNXBaUVhvS3hzRWhldzJ2RzAzNXU2WWYrNUc5ClJUYjBWNmVRc1U0TXRzWjZpRVcyYjVFOTBjVUNEOFg1eWtKMmFqeHo4QjUrREk5RUMrSk9UQmlvaVRMbFo5a0QKck1RVHc0NGhidjBsd3FqazVrVjRPcWUvTnZ0MTRlTWxoSDlUUEJKYUtQSlU1akJ5cjFuaWNnV0VmVENwQ3RYbQp0RjJ1UzBrR2ZHVWZ2SURWTFBZNFppY29CSjBvRFdOZFg1WEtVVnlxdHlHTlNNUHVQTjhDQXdFQUFhTmhNRjh3CkRnWURWUjBQQVFIL0JBUURBZ0trTUIwR0ExVWRKUVFXTUJRR0NDc0dBUVVGQndNQkJnZ3JCZ0VGQlFjREFqQVAKQmdOVkhSTUJBZjhFQlRBREFRSC9NQjBHQTFVZERnUVdCQlRPdm9JM1ZVMzZoZ2tuejBTNVlKTDBxVnZzRGpBTgpCZ2txaGtpRzl3MEJBUXNGQUFPQ0FRRUFhRVkvR2luZC9WMElqaUsrQVVHbElGRG5tSms2VjJydWVIWnVJZEtwCnZ2SmxaaFdEQ0g3SDgxUE9pK0JJMEtSaTJob0NjV1BaMEJQQ1p5R1dBdFYvMmIxY3hXYUFHaTJRaGpadlBSMGUKc2RjbzRtNEtCNlJhWEkweTV6SnhleHRYVDZjNlV1c29tTklTeXFSLzA1aTZxdmEyN214Y08zREdmOVJnbTV0awpyMnV0dnlYdGkwK2JxbGF5eFNtNnc1SFpxWXU1SDBtdUdJcUszZ2FaaTdvTk1lbjNnR0ZVNXcxam1UWjlLN3RQCjY2NTkwKzV0dlBPVEVyZDkvWXdKWXI4R21LVzE0ZDN2VFFTbkZPeHNRK0pCaDg3c2FtazhJNmx1R3RQYzhIOE4KVFpTN1dYRGRmQUdGUi9wQnhISURWU0htajVvSG5paWpvcnZhdWphbnpQdVJxZz09Ci0tLS0tRU5EIENFUlRJRklDQVRFLS0tLS0K
   failurePolicy: Fail
   name: vlagents.operator.victoriametrics.com
   admissionReviewVersions:
   - v1
   sideEffects: None
   objectSelector:
@@ -39,13 +39,13 @@

 - clientConfig:
     service:
       namespace: observability
       name: victoria-metrics-victoria-metrics-operator
       path: /validate-operator-victoriametrics-com-v1-vlcluster
       port: 9443
-    caBundle: LS0tLS1CRUdJTiBDRVJUSUZJQ0FURS0tLS0tCk1JSURIVENDQWdXZ0F3SUJBZ0lRWXhMcHE3eXNTRXBrMk1zWldNVUc4REFOQmdrcWhraUc5dzBCQVFzRkFEQVoKTVJjd0ZRWURWUVFERXc1MmJTMXZjR1Z5WVhSdmNpMWpZVEFlRncweU5qQXpNVFl4TnpVMk1URmFGdzB6TmpBegpNVE14TnpVMk1URmFNQmt4RnpBVkJnTlZCQU1URG5adExXOXdaWEpoZEc5eUxXTmhNSUlCSWpBTkJna3Foa2lHCjl3MEJBUUVGQUFPQ0FROEFNSUlCQ2dLQ0FRRUFzamdMazU1U1hvdGpBTGF1Sy80RmpuQlZEVlRJRFRjOEVCclMKNTJwMmFqRlpDL3BTV0VqNWYvdGJkcFc2SDdKZVE4ZlFJWmdqYkdEMTMxeWpJajVEbnlIZ0Q4NnR3NHBHL3NDNwpRQ3NXYXJZV25DYUs5aVBMa2FoYWZVOTJqSU1FaTJvcTJUM0dUSUtvWHFrMmdkaGhHUmIyc2NZdWZqaExwVFdSCi9JT3pIeGxrWmFQZkhVc1RWbm1Bc3NxOEk4WDMzQnBzaXRSNy9YSVdKbU4vWjM1VlEyeXkxYkgydEFPQU9IK2MKK0J6aG13YmZmaWhvZTZlTWN2c29qa3YwaTQvSUNub2dXdUw1RVk0bGNwN0kzTWJWYWdIVTJIS09kS1BsTGp4Zgo2YU9hWFh0enNsdFZZS2s5ZGJNSXJJalJsY25MazhKZm1XeVdqU2F1d0FFaGpUczhDUUlEQVFBQm8yRXdYekFPCkJnTlZIUThCQWY4RUJBTUNBcVF3SFFZRFZSMGxCQll3RkFZSUt3WUJCUVVIQXdFR0NDc0dBUVVGQndNQ01BOEcKQTFVZEV3RUIvd1FGTUFNQkFmOHdIUVlEVlIwT0JCWUVGQjZJeVgxVFcvQXJLT1M3Z0NXZmkxVWloNnhETUEwRwpDU3FHU0liM0RRRUJDd1VBQTRJQkFRQWsxZ0ZxRHdyUEJ4UXhnU2VaR3poUmdXbzgyWTJGSHJoWFNzRFk1N2szClVHMlREUEVzOUpBNFcxNlE5WUUzT1VhRWViTjF5eHNXRmVHREFyRXQrRHlUcU8yUHNGSFoycjJveFBVcVBTZUgKdU1vRVFLcVJHcjg1L2VuZ3hseGVOeG9COVpKWmxLb2ljQzRKTG83aHlwMDBjQ3p1cFlSQjBDUVhUY3MyMFU3TQowN3hkVjdGVEtQelhzOVAwd1JuNXFqS3k5V0RjK21TYWE0SUpXeHUyODFIU3RwWE0vTDl1WlIyWlRTVTd0SnpoCi9DL1RkdEFOblAxM21KQXlHUjErZnI2bnVRRTVJTVdBRnVoeCs0YVpQTE1jRGJtYko1enlUZUJlSE05YW9yZjQKRHEyVWRGV0VnbFRoTjd0VG4vOTVtb0QzdjlCbExlYVI4QTJOTUtlOVZsY3MKLS0tLS1FTkQgQ0VSVElGSUNBVEUtLS0tLQo=
+    caBundle: LS0tLS1CRUdJTiBDRVJUSUZJQ0FURS0tLS0tCk1JSURIakNDQWdhZ0F3SUJBZ0lSQU5ZMFVjd3ZVejlpelNVQTh2OFVSSEl3RFFZSktvWklodmNOQVFFTEJRQXcKR1RFWE1CVUdBMVVFQXhNT2RtMHRiM0JsY21GMGIzSXRZMkV3SGhjTk1qWXdNekUyTVRjMU5qRXhXaGNOTXpZdwpNekV6TVRjMU5qRXhXakFaTVJjd0ZRWURWUVFERXc1MmJTMXZjR1Z5WVhSdmNpMWpZVENDQVNJd0RRWUpLb1pJCmh2Y05BUUVCQlFBRGdnRVBBRENDQVFvQ2dnRUJBTmZJTFV6YTdGTkJpd2hJK1hORXdhdGlrVFFXVjVTWUZicU4KSjZjUDdDTTk1WjhaczRpWGN2WTJoc2s0RnFUdVo0bHVoaDF0NnA1K1RVODZDaUppWW5mTmVWRFFKK2U1TlFvRwp6elpNajdISjFhSVdzUER0UHBFRTFYNmJLaE00ZWlVcGhpS2hnNXBaUVhvS3hzRWhldzJ2RzAzNXU2WWYrNUc5ClJUYjBWNmVRc1U0TXRzWjZpRVcyYjVFOTBjVUNEOFg1eWtKMmFqeHo4QjUrREk5RUMrSk9UQmlvaVRMbFo5a0QKck1RVHc0NGhidjBsd3FqazVrVjRPcWUvTnZ0MTRlTWxoSDlUUEJKYUtQSlU1akJ5cjFuaWNnV0VmVENwQ3RYbQp0RjJ1UzBrR2ZHVWZ2SURWTFBZNFppY29CSjBvRFdOZFg1WEtVVnlxdHlHTlNNUHVQTjhDQXdFQUFhTmhNRjh3CkRnWURWUjBQQVFIL0JBUURBZ0trTUIwR0ExVWRKUVFXTUJRR0NDc0dBUVVGQndNQkJnZ3JCZ0VGQlFjREFqQVAKQmdOVkhSTUJBZjhFQlRBREFRSC9NQjBHQTFVZERnUVdCQlRPdm9JM1ZVMzZoZ2tuejBTNVlKTDBxVnZzRGpBTgpCZ2txaGtpRzl3MEJBUXNGQUFPQ0FRRUFhRVkvR2luZC9WMElqaUsrQVVHbElGRG5tSms2VjJydWVIWnVJZEtwCnZ2SmxaaFdEQ0g3SDgxUE9pK0JJMEtSaTJob0NjV1BaMEJQQ1p5R1dBdFYvMmIxY3hXYUFHaTJRaGpadlBSMGUKc2RjbzRtNEtCNlJhWEkweTV6SnhleHRYVDZjNlV1c29tTklTeXFSLzA1aTZxdmEyN214Y08zREdmOVJnbTV0awpyMnV0dnlYdGkwK2JxbGF5eFNtNnc1SFpxWXU1SDBtdUdJcUszZ2FaaTdvTk1lbjNnR0ZVNXcxam1UWjlLN3RQCjY2NTkwKzV0dlBPVEVyZDkvWXdKWXI4R21LVzE0ZDN2VFFTbkZPeHNRK0pCaDg3c2FtazhJNmx1R3RQYzhIOE4KVFpTN1dYRGRmQUdGUi9wQnhISURWU0htajVvSG5paWpvcnZhdWphbnpQdVJxZz09Ci0tLS0tRU5EIENFUlRJRklDQVRFLS0tLS0K
   failurePolicy: Fail
   name: vlclusters.operator.victoriametrics.com
   admissionReviewVersions:
   - v1
   sideEffects: None
   objectSelector:
@@ -67,17 +67,17 @@

 - clientConfig:
     service:
       namespace: observability
       name: victoria-metrics-victoria-metrics-operator
       path: /validate-operator-victoriametrics-com-v1beta1-vlogs
       port: 9443
-    caBundle: LS0tLS1CRUdJTiBDRVJUSUZJQ0FURS0tLS0tCk1JSURIVENDQWdXZ0F3SUJBZ0lRWXhMcHE3eXNTRXBrMk1zWldNVUc4REFOQmdrcWhraUc5dzBCQVFzRkFEQVoKTVJjd0ZRWURWUVFERXc1MmJTMXZjR1Z5WVhSdmNpMWpZVEFlRncweU5qQXpNVFl4TnpVMk1URmFGdzB6TmpBegpNVE14TnpVMk1URmFNQmt4RnpBVkJnTlZCQU1URG5adExXOXdaWEpoZEc5eUxXTmhNSUlCSWpBTkJna3Foa2lHCjl3MEJBUUVGQUFPQ0FROEFNSUlCQ2dLQ0FRRUFzamdMazU1U1hvdGpBTGF1Sy80RmpuQlZEVlRJRFRjOEVCclMKNTJwMmFqRlpDL3BTV0VqNWYvdGJkcFc2SDdKZVE4ZlFJWmdqYkdEMTMxeWpJajVEbnlIZ0Q4NnR3NHBHL3NDNwpRQ3NXYXJZV25DYUs5aVBMa2FoYWZVOTJqSU1FaTJvcTJUM0dUSUtvWHFrMmdkaGhHUmIyc2NZdWZqaExwVFdSCi9JT3pIeGxrWmFQZkhVc1RWbm1Bc3NxOEk4WDMzQnBzaXRSNy9YSVdKbU4vWjM1VlEyeXkxYkgydEFPQU9IK2MKK0J6aG13YmZmaWhvZTZlTWN2c29qa3YwaTQvSUNub2dXdUw1RVk0bGNwN0kzTWJWYWdIVTJIS09kS1BsTGp4Zgo2YU9hWFh0enNsdFZZS2s5ZGJNSXJJalJsY25MazhKZm1XeVdqU2F1d0FFaGpUczhDUUlEQVFBQm8yRXdYekFPCkJnTlZIUThCQWY4RUJBTUNBcVF3SFFZRFZSMGxCQll3RkFZSUt3WUJCUVVIQXdFR0NDc0dBUVVGQndNQ01BOEcKQTFVZEV3RUIvd1FGTUFNQkFmOHdIUVlEVlIwT0JCWUVGQjZJeVgxVFcvQXJLT1M3Z0NXZmkxVWloNnhETUEwRwpDU3FHU0liM0RRRUJDd1VBQTRJQkFRQWsxZ0ZxRHdyUEJ4UXhnU2VaR3poUmdXbzgyWTJGSHJoWFNzRFk1N2szClVHMlREUEVzOUpBNFcxNlE5WUUzT1VhRWViTjF5eHNXRmVHREFyRXQrRHlUcU8yUHNGSFoycjJveFBVcVBTZUgKdU1vRVFLcVJHcjg1L2VuZ3hseGVOeG9COVpKWmxLb2ljQzRKTG83aHlwMDBjQ3p1cFlSQjBDUVhUY3MyMFU3TQowN3hkVjdGVEtQelhzOVAwd1JuNXFqS3k5V0RjK21TYWE0SUpXeHUyODFIU3RwWE0vTDl1WlIyWlRTVTd0SnpoCi9DL1RkdEFOblAxM21KQXlHUjErZnI2bnVRRTVJTVdBRnVoeCs0YVpQTE1jRGJtYko1enlUZUJlSE05YW9yZjQKRHEyVWRGV0VnbFRoTjd0VG4vOTVtb0QzdjlCbExlYVI4QTJOTUtlOVZsY3MKLS0tLS1FTkQgQ0VSVElGSUNBVEUtLS0tLQo=
[Diff truncated by flux-local]
--- HelmRelease: observability/victoria-metrics ClusterRole: observability/victoria-metrics-victoria-metrics-operator-cleanup-hook

+++ HelmRelease: observability/victoria-metrics ClusterRole: observability/victoria-metrics-victoria-metrics-operator-cleanup-hook

@@ -46,12 +46,15 @@

   - vmauths
   - vmauths/finalizers
   - vmauths/status
   - vmclusters
   - vmclusters/finalizers
   - vmclusters/status
+  - vmdistributed
+  - vmdistributed/finalizers
+  - vmdistributed/status
   - vmnodescrapes
   - vmnodescrapes/finalizers
   - vmnodescrapes/status
   - vmpodscrapes
   - vmpodscrapes/finalizers
   - vmpodscrapes/status
--- HelmRelease: observability/victoria-metrics Job: observability/victoria-metrics-victoria-metrics-operator-cleanup-hook

+++ HelmRelease: observability/victoria-metrics Job: observability/victoria-metrics-victoria-metrics-operator-cleanup-hook

@@ -40,11 +40,11 @@

             - ALL
           readOnlyRootFilesystem: true
         command:
         - kubectl
         args:
         - delete
-        - vlagent,vlcluster,vlogs,vlsingle,vmagent,vmalert,vmalertmanager,vmalertmanagerconfig,vmanomaly,vmauth,vmcluster,vmnodescrape,vmpodscrape,vmprobe,vmrule,vmscrapeconfig,vmservicescrape,vmsingle,vmstaticscrape,vmuser,vtcluster,vtsingle
+        - vlagent,vlcluster,vlogs,vlsingle,vmagent,vmalert,vmalertmanager,vmalertmanagerconfig,vmanomaly,vmauth,vmcluster,vmdistributed,vmnodescrape,vmpodscrape,vmprobe,vmrule,vmscrapeconfig,vmservicescrape,vmsingle,vmstaticscrape,vmuser,vtcluster,vtsingle
         - --all
         - --ignore-not-found=true
       restartPolicy: OnFailure

Signed-off-by: bot-akira[bot] <159718293+bot-akira[bot]@users.noreply.github.com>

bot-akira bot added renovate/helm type/minor labels Feb 16, 2026

github-actions bot added the area/kubernetes label Feb 16, 2026

bot-akira bot force-pushed the renovate/victoria-metrics-k8s-stack-0.x branch from 3979fb9 to ec48e83 Compare February 17, 2026 07:43

bot-akira bot changed the title ~~feat(helm): update chart victoria-metrics-k8s-stack (0.70.0 → 0.71.0)~~ feat(helm): update chart victoria-metrics-k8s-stack (0.70.0 → 0.71.1) Feb 17, 2026

bot-akira bot changed the title ~~feat(helm): update chart victoria-metrics-k8s-stack (0.70.0 → 0.71.1)~~ feat(helm): update chart victoria-metrics-k8s-stack (0.70.0 → 0.72.0) Feb 24, 2026

bot-akira bot force-pushed the renovate/victoria-metrics-k8s-stack-0.x branch from ec48e83 to 6881d5a Compare February 24, 2026 11:33

bot-akira bot changed the title ~~feat(helm): update chart victoria-metrics-k8s-stack (0.70.0 → 0.72.0)~~ feat(helm): update chart victoria-metrics-k8s-stack (0.70.0 → 0.72.1) Feb 24, 2026

bot-akira bot force-pushed the renovate/victoria-metrics-k8s-stack-0.x branch from 6881d5a to d195acb Compare February 24, 2026 21:26

bot-akira bot changed the title ~~feat(helm): update chart victoria-metrics-k8s-stack (0.70.0 → 0.72.1)~~ feat(helm): update chart victoria-metrics-k8s-stack (0.70.0 → 0.72.2) Feb 25, 2026

bot-akira bot force-pushed the renovate/victoria-metrics-k8s-stack-0.x branch 2 times, most recently from 2384e64 to 5106f1e Compare March 3, 2026 19:42

bot-akira bot changed the title ~~feat(helm): update chart victoria-metrics-k8s-stack (0.70.0 → 0.72.2)~~ feat(helm): update chart victoria-metrics-k8s-stack (0.70.0 → 0.72.3) Mar 4, 2026

bot-akira bot force-pushed the renovate/victoria-metrics-k8s-stack-0.x branch from 5106f1e to 0bbc090 Compare March 4, 2026 23:21

bot-akira bot changed the title ~~feat(helm): update chart victoria-metrics-k8s-stack (0.70.0 → 0.72.3)~~ feat(helm): update chart victoria-metrics-k8s-stack (0.70.0 → 0.72.4) Mar 5, 2026

bot-akira bot force-pushed the renovate/victoria-metrics-k8s-stack-0.x branch from 0bbc090 to 1ca91c9 Compare March 5, 2026 08:30

feat(helm): update chart victoria-metrics-k8s-stack (0.70.0 → 0.72.5)

7748bd8

Signed-off-by: bot-akira[bot] <159718293+bot-akira[bot]@users.noreply.github.com>

bot-akira bot force-pushed the renovate/victoria-metrics-k8s-stack-0.x branch from 1ca91c9 to 7748bd8 Compare March 16, 2026 17:55

bot-akira bot changed the title ~~feat(helm): update chart victoria-metrics-k8s-stack (0.70.0 → 0.72.4)~~ feat(helm): update chart victoria-metrics-k8s-stack (0.70.0 → 0.72.5) Mar 16, 2026

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

feat(helm): update chart victoria-metrics-k8s-stack (0.70.0 → 0.72.5)#3045

feat(helm): update chart victoria-metrics-k8s-stack (0.70.0 → 0.72.5)#3045
bot-akira[bot] wants to merge 1 commit intomainfrom
renovate/victoria-metrics-k8s-stack-0.x

bot-akira bot commented Feb 16, 2026 •

edited

Loading

Uh oh!

bot-akira bot commented Feb 16, 2026 •

edited

Loading

Uh oh!

bot-akira bot commented Feb 16, 2026 •

edited

Loading

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

0 participants

Conversation

bot-akira bot commented Feb 16, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Release Notes

Release notes for version 0.72.5

Release notes for version 0.72.4

Release notes for version 0.72.3

Release notes for version 0.72.2

Release notes for version 0.72.1

Release notes for version 0.72.0

Release notes for version 0.71.1

Release notes for version 0.71.0

Configuration

Uh oh!

bot-akira bot commented Feb 16, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

bot-akira bot commented Feb 16, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

0 participants

bot-akira bot commented Feb 16, 2026 •

edited

Loading

bot-akira bot commented Feb 16, 2026 •

edited

Loading

bot-akira bot commented Feb 16, 2026 •

edited

Loading