Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
43 changes: 43 additions & 0 deletions docs/Data-insights/Features/New-alerting/overview.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,43 @@
# Alerting

Alerting tells you when something needs attention - ideally before your users do. It covers the static rules you define, the AI detectors that learn what normal looks like for you, and the notification setup that decides who hears about it and when.

Navigate to **Alerting** in the left-hand menu to open it.

## What's in this section

| Page | Use it to |
|---|---|
| [Status](status.md) | See every alert rule's state at a glance, and find what needs attention first |
| [Alert Rules](rules.md) | Build, manage, and investigate static alert rules |
| [Annotation Templates](annotation-templates.md) | Write alert messages that carry live values from the query that fired |
| [Recording Rules](recording-rules.md) | Pre-compute expensive queries and save the result as a new metric |
| [Service Anomaly Detectors](service-anomaly-detectors.md) | Tune the detectors created automatically for each instrumented service |
| [Custom Anomaly Detectors](custom-anomaly-detectors.md) | Run anomaly detection against your own PromQL series |
| [Contact Points](contact-points.md) | Define where notifications are sent |
| [Notification Policy](notification-policy.md) | Route each alert to the right contact point |
| [Silences](silences.md) | Suppress notifications temporarily |
| [Time Intervals](time-intervals.md) | Suppress notifications on a schedule, such as outside working hours |
| [Troubleshooting](troubleshooting.md) | Diagnose why an alert did or did not fire |

## Rules or detectors?

**Rules** are static checks - they run on a fixed schedule against fixed thresholds, best for known conditions with clear boundaries, like system CPU or allocated memory.

**Detectors** use AI to learn normal behavior and flag anomalies automatically, so they adapt as your system changes. Use them where a fixed threshold would be guesswork - set too tight it produces false alarms, set too loose it misses real problems.

The two are not exclusive. Most environments use both.

## Common controls

Every alerting page shares the same set of controls:

- **Breadcrumbs** at the top of the page - **Alerting › Status**, for example - move between the alerting pages. Each breadcrumb has a dropdown, so you can go directly to another page.
- **An introductory banner** describes what the page does. Click **Dismiss** to hide it.
- **A refresh control** at the right of the toolbar sets how often the page updates (such as, **30s**), with a button beside it to refresh immediately.
- **An OpsPilot dropdown**, on most pages, offers AI shortcuts for the page you are viewing - help writing a rule, an explanation of what is firing, or suggestions for what to add.

---

!!! question "Need more help?"
Contact support in the chat bubble and let us know how we can assist.
30 changes: 17 additions & 13 deletions docs/Data-insights/Features/New-alerting/rules.md
Original file line number Diff line number Diff line change
@@ -1,32 +1,33 @@
# Rules
# Alert Rules

A well-configured alert rule is the difference between knowing about a problem before your users do and finding out from a support ticket. The Rules page is where you build, manage, and investigate every alert rule in your environment - with a live view of what's firing, what's pending, and what's healthy, all in one place.
A well-configured alert rule is the difference between knowing about a problem before your users do and finding out from a support ticket. The Alert Rules page is where you build, manage, and investigate every alert rule in your environment - with a live view of what's firing, what's pending, and what's healthy, all in one place.

Navigate to **Alerting > Rules** to open it.
Navigate to **Alerting > Alert Rules** to open it.

!!! info "Rules vs detectors"
**Rules** are static checks - they run on a fixed schedule against fixed thresholds, best for known conditions with clear boundaries (like system CPU or allocated memory). **[Detectors](service-anomaly-detectors.md)** use AI to learn normal behaviour and flag anomalies automatically, so they adapt as your system changes.
**Rules** are static checks - they run on a fixed schedule against fixed thresholds, best for known conditions with clear boundaries (like system CPU or allocated memory). **[Detectors](service-anomaly-detectors.md)** use AI to learn normal behavior and flag anomalies automatically, so they adapt as your system changes.

## The rules list

![Screenshot](/Data-insights/Features/images/Alerting/rule-table.png)

Rules are displayed with four state counters at the top:
Four state counters sit above the list:

| State | Description |
|---|---|
| **Firing** | The alert condition is currently met |
| **Pending** | The condition is met but the pending period has not yet elapsed |
| **Normal** | The rule is evaluating and its condition is not met |
| **Normal (Missingseries)** | The rule returned no data - the query matched no series. The rule remains Normal but flags that the data source returned nothing |
| **Paused** | The rule is paused and not being evaluated |

A state can carry a sub-reason, which appears in the rule's state history rather than in the counters - **Normal (Missingseries)**, for example, means the rule returned no data because the query matched no series. The rule remains Normal, but flags that the data source returned nothing.

### Views

Switch between **Table** and **Tree** view using the buttons in the toolbar:

- **Table** - a flat list of all rules with their state, folder, and group
- **Tree** - rules grouped by folder and evaluation group, useful for seeing how rules are organised
- **Table** - a flat list of all rules with their state, namespace, and group
- **Tree** - rules grouped by namespace and evaluation group, useful for seeing how rules are organized

Use **Collapse all** to fold all groups at once in Tree view.

Expand All @@ -38,11 +39,13 @@ The table has the following columns:
|---|---|
| **State** | Current state of the rule (Firing, Pending, Normal, Paused) |
| **Name** | The alert rule name |
| **Namespace** | The folder the rule belongs to |
| **Namespace** | The namespace the rule belongs to |
| **Group** | The evaluation group and its interval (such as, `auto-1m`) |
| **Last evaluation** | When the rule was last evaluated |
| **Last evaluation** | When the rule was last evaluated (such as, *29 Sept, 12:43*) |
| **Actions** | A notification count, an **Active** toggle to enable or pause the rule, and buttons to view (eye), edit, and open more options |

The **State**, **Name**, **Namespace**, and **Group** headers are sortable - click one to order the list by it.

### Expanding a rule

Click a rule row in the list to expand it in place. The left side shows:
Expand All @@ -60,7 +63,8 @@ The **Dashboard** and **Runbook** buttons (top right of the expanded view) open
| **Data source** | The data source the rule queries |
| **On no data** | What state the rule enters when the query returns no data |
| **On query error** | What state the rule enters when the query fails |
| **Notifies** | The contact points configured to receive notifications |
| **Labels** | The rule's labels as name/value chips (such as, `severity = warning`) - these are what [notification policies](notification-policy.md) route on. Only shown when the rule has labels |
| **Notifies** | The contact points configured to receive notifications, each shown as a card with its name and type. Reads *None configured* when the rule has none |
| **Instances** | The matched instances with a firing/pending breakdown and a **Firing only** toggle. Each row shows the instance's labels, its state and when it last fired, and a **Logs** button to jump to its logs |

![Screenshot](/Data-insights/Features/images/Alerting/rule-expanded.png)
Expand Down Expand Up @@ -112,7 +116,7 @@ A row of summary cards sits below the header:

- **Sort** - order rules by State, Name, or other fields. Toggle ascending/descending with the arrow button
- **Search** - find rules by name
- **Filters** - filter by folder, evaluation group, state, or label
- **Filters** - **All namespaces** and **All groups** dropdowns narrow the list to a single namespace or evaluation group
- **Hide anomaly detectors** - toggle on to show only static rules and hide anomaly detectors from the list

### OpsPilot
Expand Down Expand Up @@ -189,7 +193,7 @@ The **Evaluation** section controls how the rule runs:

#### Namespace

Expand **Namespace** and choose the **namespace** where the rule is stored. Namespaces keep rules organised and control access.
Expand **Namespace** and choose the **namespace** where the rule is stored. Namespaces keep rules organized and control access.

#### Rule name

Expand Down
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
# Service Anomaly Detectors

Picking the right threshold is hard - too tight and you drown in false alarms, too loose and you miss the real problem. **Anomaly detectors** flag unusual behaviour in your services automatically, so you get meaningful alerts without hand-tuning numbers.
Picking the right threshold is hard - too tight and you drown in false alarms, too loose and you miss the real problem. **Anomaly detectors** flag unusual behavior in your services automatically, so you get meaningful alerts without hand-tuning numbers.

**Service detectors** are auto-created for each instrumented service. To define detectors against your own PromQL queries instead, see [Custom Anomaly Detectors](custom-anomaly-detectors.md).

Expand All @@ -16,6 +16,8 @@ Each instrumented service automatically gets three detectors, shown as **R**, **
- **Error** (E) - error rate anomalies
- **Latency** (D) - response time anomalies (also shown as *Duration* on the Status page)

The badges are filled while the service's detectors are running, and grayed out while they are paused. **Last evaluation** reads a timestamp (such as, *29 Sept, 13:01*) for a running service, and **—** for a paused one.

Each row in the service list shows the service name, its **Detectors** (R/E/D), the **Last evaluation** time, and actions: a notification count, an **Active** toggle to enable or pause the service's detectors, an **Open dashboard** button, and a **delete** button. Expand a row to see and tune the individual detectors.

## Scanning for services
Expand Down
10 changes: 7 additions & 3 deletions docs/Data-insights/Features/New-alerting/status.md
Original file line number Diff line number Diff line change
Expand Up @@ -8,7 +8,9 @@ Navigate to **Alerting > Status** to open it.

## Needs attention

A summary band at the top leads with what matters: how many rules **need attention** out of the total, and how many are currently **firing** and **pending** (for example, *7 of 44 rules - 1 firing, 6 pending*). A health bar shows the split across states, with a count for each:
A summary band at the top leads with what matters: how many rules **need attention** out of the total, and which states they are in - for example, **NEEDS ATTENTION**, *1 of 70 rules — 1 pending*. When everything is healthy the band reads *0 of 70 rules need attention*; when something needs attention, the band is highlighted.

Below it, a health bar shows the split across states, with a chip and count for each. Only states that have rules in them appear - an account with nothing firing or pending shows just **Normal** and **Paused**.

| State | Meaning |
|---|---|
Expand All @@ -35,7 +37,9 @@ Controls across the top shape how the page is laid out:

## Groups and rules

Rules are organised into cards - by **namespace** or by **source**, depending on the toggle. Each group card shows its name, a health bar, and a count for each state, with its rules laid out inside.
Rules are organized into cards - by **namespace** or by **source**, depending on the toggle. Each group card shows its name, a health bar, and a chip with a count for each state present.

A card holding rules that need attention is **highlighted** and moved to the front of the page, and those rules are listed on the card without you expanding it. Healthy rules stay folded away behind **Show all N rules**, so what needs looking at is what you see first.

Each rule shows:

Expand All @@ -49,7 +53,7 @@ When a group has more rules than fit, click **Show all N rules** to expand it, a

Shift-click rules to select more than one at a time, then silence the whole selection in one go. See [Silences](silences.md).

The **List** view stacks the groups and shows their rule cards inline, while the **Grid** view lays the groups out as compact cards in a multi-column grid - each summarising its state at a glance, and expandable to reveal the rules inside.
The **List** view stacks the groups and shows their rule cards inline, while the **Grid** view lays the groups out as compact cards in a multi-column grid - each summarizing its state at a glance, and expandable to reveal the rules inside.

## Creating from Status

Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -23,7 +23,7 @@ This page covers common issues when setting up and operating OpsPilot alerting,

**Fix:**

- Navigate to the rule and check the **Pending period** value under **Set evaluation behaviour**.
- Navigate to the rule and check the **Pending period** value under **Set evaluation behavior**.
- If you want the alert to fire immediately, set the pending period to `0s`.
- If the metric is intermittently crossing the threshold, consider whether the pending period is appropriate or whether the condition itself needs adjusting.

Expand Down
Binary file modified docs/Data-insights/Features/images/Alerting/status.png
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
3 changes: 2 additions & 1 deletion mkdocs.yml
Original file line number Diff line number Diff line change
Expand Up @@ -127,8 +127,9 @@ nav:
- Overview: Monitor-your-data/Kubernetes-monitoring/overview.md
- Shipping data: Monitor-your-data/Kubernetes-monitoring/Shipping.md
- Alerting:
- Overview: Data-insights/Features/New-alerting/overview.md
- Status: Data-insights/Features/New-alerting/status.md
- Rules:
- Alert Rules:
- Overview: Data-insights/Features/New-alerting/rules.md
- Annotation Templates: Data-insights/Features/New-alerting/annotation-templates.md
- Recording Rules: Data-insights/Features/New-alerting/recording-rules.md
Expand Down
Loading