Repository navigation
Description
Summary
Add support for Service Level Objectives (SLOs), Service Level Indicators (SLIs), and Error Budgets to HyperDX, inspired by Honeycomb's SLO implementation.
Background
SLOs are a critical component of modern observability and Site Reliability Engineering (SRE) practices. They help teams:
- Define measurable reliability goals for services
- Balance feature development with infrastructure maintenance
- Prioritize incidents effectively based on error budget consumption
- Report on service quality with precision
Key Concepts to Implement
1. Service Level Indicator (SLI)
A per-event measurement that defines whether the system succeeded or failed.
Examples:
- Latency:
response_time < 500ms - Availability:
http_status_code < 500 - Error rate:
error == false
2. Service Level Objective (SLO)
The target proportion of successful SLIs over a rolling time window, expressed as a percentage.
Examples:
- "99.9% availability over 30 days"
- "99% of requests under 500ms latency over 7 days"
3. Error Budget
The allowable amount of failure within the SLO window.
Example calculation:
- At 99.9% target with 1 million events in 30 days
- Error budget = 1,000 failed events (0.1% of 1M)
- Or ~44 minutes of downtime in 30 days
4. Burn Rate
How quickly the error budget is being consumed compared to the target rate.
- Burn rate of
1.0= consuming budget evenly (will deplete exactly at window end) - Burn rate of
2.0= consuming twice as fast (will deplete in half the window)
Formula:
Burn rate = actual error rate / expected error rate
Proposed Features
Phase 1: Core SLO Infrastructure
-
SLI Definition
- Allow users to define SLIs using existing query builder
- Support for multiple SLI types:
- Latency-based (threshold comparison)
- Availability-based (error rate)
- Custom expressions
-
SLO Configuration
- Target percentage (e.g., 99.9%)
- Rolling time window (7d, 14d, 30d, 90d)
- Associated services/sources
- Name and description
-
Error Budget Tracking
- Calculate remaining error budget
- Budget burndown visualization
- Historical budget consumption
Phase 2: Alerting & Notifications
-
Burn Alerts
- Alert when error budget is depleting faster than expected
- Configurable burn rate thresholds
- Multiple alert severity levels based on burn rate
-
Integration with Existing Alert Channels
- Webhook notifications
- Future: Slack, PagerDuty, email
Phase 3: Visualization & Reporting
-
SLO Dashboard
- Current SLO compliance percentage
- Error budget remaining (events and time)
- Burn rate visualization
- Historical SLO performance
-
BubbleUp Integration
- Click through from SLO violations to investigate root causes
- Identify outliers contributing to budget burn
Phase 4: Advanced Features (Future)
-
Multi-Service SLOs
- Share error budget across multiple services
- Aggregate events from related services
- Support for up to 10 services per SLO
-
SLO Tags
- Organize SLOs by team, project, or service
- Filter and group SLOs
-
SLO Reporting
- Weekly/monthly SLO reports
- Trend analysis
Technical Considerations
Data Model
New MongoDB models needed:
// SLI Model
interface ISLI {
id: string;
name: string;
description?: string;
team: ObjectId;
source: ObjectId; // Reference to Source/Service
// Query definition
successCondition: {
field: string;
operator: '<' | '<=' | '>' | '>=' | '==' | '!=' | 'exists';
value: string | number | boolean;
};
createdBy: ObjectId;
createdAt: Date;
updatedAt: Date;
}
// SLO Model
interface ISLO {
id: string;
name: string;
description?: string;
team: ObjectId;
sli: ObjectId; // Reference to SLI
// Target configuration
targetPercentage: number; // e.g., 99.9
windowDays: number; // e.g., 30
// Current state
currentPercentage?: number;
errorBudgetRemaining?: number;
burnRate?: number;
// Alerting
burnAlerts?: {
enabled: boolean;
thresholds: Array<{
burnRate: number;
severity: 'warning' | 'critical';
channel: AlertChannel;
}>;
};
tags?: string[];
createdBy: ObjectId;
createdAt: Date;
updatedAt: Date;
}ClickHouse Queries
SLO calculations will require efficient ClickHouse queries:
-- Calculate SLI success rate over time window
SELECT
countIf(duration_ms < 500) as successful_events,
count(*) as total_events,
successful_events / total_events * 100 as success_rate
FROM events
WHERE
timestamp >= now() - INTERVAL 30 DAY
AND service_name = 'my-service'Background Tasks
New background task for SLO monitoring:
- Calculate SLO compliance periodically (every 1-5 minutes)
- Update burn rates
- Trigger burn alerts when thresholds exceeded
- Store historical SLO data for reporting
UI/UX Considerations
-
SLO List Page
- Overview of all SLOs with current status
- Quick view of error budget remaining
- Filter by tags, services, compliance status
-
SLO Detail Page
- Real-time compliance percentage
- Error budget burndown chart
- Burn rate over time
- Recent violations with drill-down capability
-
SLO Creation Wizard
- Step-by-step SLI and SLO configuration
- Preview of expected error budget
- Alert configuration
References
Acceptance Criteria
- Users can define SLIs based on telemetry data
- Users can create SLOs with target percentages and time windows
- Error budget is calculated and displayed in real-time
- Burn rate is tracked and visualized
- Burn alerts can be configured and trigger notifications
- SLO dashboard provides at-a-glance view of service reliability
- Historical SLO data is available for reporting
Labels
enhancement, feature-request, observability
Activity
teeohhem commentedon Dec 8, 2025
Thanks for the feature request. This is noted in our backlog!
alok87 commentedon Dec 11, 2025
@teeohhem Is it going to be build as part of open source here in this repo or going to be closed? and why when do u think this can come out?
kentan88 commentedon Dec 12, 2025
@alok87 Please don't mind me asking
For the SLO calculations, where do you get the "events" table or is the query you written purely inspired by honeycomb? i'm under the impression the HyperDX falls under under observability 1.5 where it still uses separate tables like otel_traces and otel_logs and not a single events table. i was wondering if i'm not using the latest version of HyperDX
alok87 commentedon Dec 12, 2025
@kentan88 why do you think separate table make hyperdx 1.5? Observaility 2.0 is about connecting traces and logs as one in the UX, does it necessarily mean in the backend to? Its about unifying o11y, having a single source of truth (may be in two tables joined via spanID), deriving metrics from traces, and avoiding pre-aggregating. Making trace as the SOT, and trace having logs connected to it(spanevents it was before).
Latest master as of today also have logs and traces as separate table.
Would want to know your views on same and learn.
alok87 commentedon Dec 15, 2025
@teeohhem Please suggest do you plan to build as open source or closed?
teeohhem commentedon Dec 15, 2025
Thanks for the discussion. This isn't on our immediate roadmap so it's unknown at this time.
alok87 commentedon Dec 18, 2025
@teeohhem do you have office hours wanted to participate on the design of this feature. I have some thoughts - does business o11y also in scope with SLO as clickhouse is an OLAP too! or only tech o11y. // @MikeShi42 @wrn14897
alok87 commentedon Jan 5, 2026
@brandon-pereira any thoughts on this? We desperately need this. We can help get the first version out if you want to work together. // @MikeShi42 @wrn14897 @teeohhem
MikeShi42 commentedon Jan 5, 2026
Hey @alok87 - sorry we haven't had time to evaluate this request due to the holidays. We'll try to find some time to discuss internally this week or next on how we want to implement this and get back to you on the design.
sgarfinkel commentedon Jan 11, 2026
Just wanted to chime in: I think apdex should be a priority here, since it’s an industry standard way of measuring satisfaction. We actually have it in our fork and I’ve been meaning to open a PR. I’ll try and do that next week.
dofinn commentedon Jan 15, 2026
@sgarfinkel APDEX solves a different problem. Native SLO Management and Alerting is found in most observability products today. SREs need this functionality to drive reliability in their orgs. I am a big +1 on this request. Suggest also looking into how Nobl9 does them as well. They have a very intuitive representation of SLOs, Burn rates and error budgets
alok87 commentedon Mar 12, 2026
Are we building this @MikeShi42 ?
alok87 commentedon Apr 2, 2026
@MikeShi42 @wrn14897 Can we prioritze this, we internally have few developers who can also work with you on this. We will share a spec/plan.
teeohhem commentedon Apr 13, 2026
@alok87 With the recent addition of raw-sql charting and alerting in open source, we may be able to accomplish at least a subset of this in the interim. Discussing this with the product team currently for broader feature acceptance/prioritization.
alok87 commentedon May 30, 2026
@teeohhem @MikeShi42 @wrn14897 Please priortize this, this is the next big thing missing and the design of this should keep the vendor agnostic thing still and not coupled the companies with a system what OTEL enabled.
@teeohhem are you developing this in open? do you have a RFC or some design doc for it now?