Skip to content

Feature Request: Service Level Objectives (SLOs), SLIs, and Error Budgets #1453

Open

Description

@alok87

Summary

Add support for Service Level Objectives (SLOs), Service Level Indicators (SLIs), and Error Budgets to HyperDX, inspired by Honeycomb's SLO implementation.

Background

SLOs are a critical component of modern observability and Site Reliability Engineering (SRE) practices. They help teams:

  • Define measurable reliability goals for services
  • Balance feature development with infrastructure maintenance
  • Prioritize incidents effectively based on error budget consumption
  • Report on service quality with precision

Key Concepts to Implement

1. Service Level Indicator (SLI)

A per-event measurement that defines whether the system succeeded or failed.

Examples:

  • Latency: response_time < 500ms
  • Availability: http_status_code < 500
  • Error rate: error == false

2. Service Level Objective (SLO)

The target proportion of successful SLIs over a rolling time window, expressed as a percentage.

Examples:

  • "99.9% availability over 30 days"
  • "99% of requests under 500ms latency over 7 days"

3. Error Budget

The allowable amount of failure within the SLO window.

Example calculation:

  • At 99.9% target with 1 million events in 30 days
  • Error budget = 1,000 failed events (0.1% of 1M)
  • Or ~44 minutes of downtime in 30 days

4. Burn Rate

How quickly the error budget is being consumed compared to the target rate.

  • Burn rate of 1.0 = consuming budget evenly (will deplete exactly at window end)
  • Burn rate of 2.0 = consuming twice as fast (will deplete in half the window)

Formula:

Burn rate = actual error rate / expected error rate

Proposed Features

Phase 1: Core SLO Infrastructure

  1. SLI Definition

    • Allow users to define SLIs using existing query builder
    • Support for multiple SLI types:
      • Latency-based (threshold comparison)
      • Availability-based (error rate)
      • Custom expressions
  2. SLO Configuration

    • Target percentage (e.g., 99.9%)
    • Rolling time window (7d, 14d, 30d, 90d)
    • Associated services/sources
    • Name and description
  3. Error Budget Tracking

    • Calculate remaining error budget
    • Budget burndown visualization
    • Historical budget consumption

Phase 2: Alerting & Notifications

  1. Burn Alerts

    • Alert when error budget is depleting faster than expected
    • Configurable burn rate thresholds
    • Multiple alert severity levels based on burn rate
  2. Integration with Existing Alert Channels

    • Webhook notifications
    • Future: Slack, PagerDuty, email

Phase 3: Visualization & Reporting

  1. SLO Dashboard

    • Current SLO compliance percentage
    • Error budget remaining (events and time)
    • Burn rate visualization
    • Historical SLO performance
  2. BubbleUp Integration

    • Click through from SLO violations to investigate root causes
    • Identify outliers contributing to budget burn

Phase 4: Advanced Features (Future)

  1. Multi-Service SLOs

    • Share error budget across multiple services
    • Aggregate events from related services
    • Support for up to 10 services per SLO
  2. SLO Tags

    • Organize SLOs by team, project, or service
    • Filter and group SLOs
  3. SLO Reporting

    • Weekly/monthly SLO reports
    • Trend analysis

Technical Considerations

Data Model

New MongoDB models needed:

// SLI Model
interface ISLI {
  id: string;
  name: string;
  description?: string;
  team: ObjectId;
  source: ObjectId; // Reference to Source/Service
  
  // Query definition
  successCondition: {
    field: string;
    operator: '<' | '<=' | '>' | '>=' | '==' | '!=' | 'exists';
    value: string | number | boolean;
  };
  
  createdBy: ObjectId;
  createdAt: Date;
  updatedAt: Date;
}

// SLO Model
interface ISLO {
  id: string;
  name: string;
  description?: string;
  team: ObjectId;
  sli: ObjectId; // Reference to SLI
  
  // Target configuration
  targetPercentage: number; // e.g., 99.9
  windowDays: number; // e.g., 30
  
  // Current state
  currentPercentage?: number;
  errorBudgetRemaining?: number;
  burnRate?: number;
  
  // Alerting
  burnAlerts?: {
    enabled: boolean;
    thresholds: Array<{
      burnRate: number;
      severity: 'warning' | 'critical';
      channel: AlertChannel;
    }>;
  };
  
  tags?: string[];
  createdBy: ObjectId;
  createdAt: Date;
  updatedAt: Date;
}

ClickHouse Queries

SLO calculations will require efficient ClickHouse queries:

-- Calculate SLI success rate over time window
SELECT
  countIf(duration_ms < 500) as successful_events,
  count(*) as total_events,
  successful_events / total_events * 100 as success_rate
FROM events
WHERE 
  timestamp >= now() - INTERVAL 30 DAY
  AND service_name = 'my-service'

Background Tasks

New background task for SLO monitoring:

  • Calculate SLO compliance periodically (every 1-5 minutes)
  • Update burn rates
  • Trigger burn alerts when thresholds exceeded
  • Store historical SLO data for reporting

UI/UX Considerations

  1. SLO List Page

    • Overview of all SLOs with current status
    • Quick view of error budget remaining
    • Filter by tags, services, compliance status
  2. SLO Detail Page

    • Real-time compliance percentage
    • Error budget burndown chart
    • Burn rate over time
    • Recent violations with drill-down capability
  3. SLO Creation Wizard

    • Step-by-step SLI and SLO configuration
    • Preview of expected error budget
    • Alert configuration

References

Acceptance Criteria

  • Users can define SLIs based on telemetry data
  • Users can create SLOs with target percentages and time windows
  • Error budget is calculated and displayed in real-time
  • Burn rate is tracked and visualized
  • Burn alerts can be configured and trigger notifications
  • SLO dashboard provides at-a-glance view of service reliability
  • Historical SLO data is available for reporting

Labels

enhancement, feature-request, observability

Activity

  1. teeohhem commented on Dec 8, 2025

    @teeohhem
    Contributor

    Thanks for the feature request. This is noted in our backlog!

  2. alok87 commented on Dec 11, 2025

    @alok87
    ContributorAuthor

    @teeohhem Is it going to be build as part of open source here in this repo or going to be closed? and why when do u think this can come out?

  3. kentan88 commented on Dec 12, 2025

    @kentan88

    @alok87 Please don't mind me asking
    For the SLO calculations, where do you get the "events" table or is the query you written purely inspired by honeycomb? i'm under the impression the HyperDX falls under under observability 1.5 where it still uses separate tables like otel_traces and otel_logs and not a single events table. i was wondering if i'm not using the latest version of HyperDX

  4. alok87 commented on Dec 12, 2025

    @alok87
    ContributorAuthor

    @kentan88 why do you think separate table make hyperdx 1.5? Observaility 2.0 is about connecting traces and logs as one in the UX, does it necessarily mean in the backend to? Its about unifying o11y, having a single source of truth (may be in two tables joined via spanID), deriving metrics from traces, and avoiding pre-aggregating. Making trace as the SOT, and trace having logs connected to it(spanevents it was before).

    Latest master as of today also have logs and traces as separate table.

    Would want to know your views on same and learn.

  5. alok87 commented on Dec 15, 2025

    @alok87
    ContributorAuthor

    @teeohhem Please suggest do you plan to build as open source or closed?

  6. teeohhem commented on Dec 15, 2025

    @teeohhem
    Contributor

    Thanks for the discussion. This isn't on our immediate roadmap so it's unknown at this time.

  7. alok87 commented on Dec 18, 2025

    @alok87
    ContributorAuthor

    @teeohhem do you have office hours wanted to participate on the design of this feature. I have some thoughts - does business o11y also in scope with SLO as clickhouse is an OLAP too! or only tech o11y. // @MikeShi42 @wrn14897

  8. alok87 commented on Jan 5, 2026

    @alok87
    ContributorAuthor

    @brandon-pereira any thoughts on this? We desperately need this. We can help get the first version out if you want to work together. // @MikeShi42 @wrn14897 @teeohhem

  9. MikeShi42 commented on Jan 5, 2026

    @MikeShi42
    Contributor

    Hey @alok87 - sorry we haven't had time to evaluate this request due to the holidays. We'll try to find some time to discuss internally this week or next on how we want to implement this and get back to you on the design.

    does business o11y also in scope with SLO as clickhouse is an OLAP too! or only tech o11y
    The lines are a bit blurry - the general thing is that we're not trying to cater to the BI/analyst space, but of course reliability metrics should be tied to business outcomes, so it makes sense that we can capture those from that perspective - if that makes sense.

  10. sgarfinkel commented on Jan 11, 2026

    @sgarfinkel
    Contributor

    Just wanted to chime in: I think apdex should be a priority here, since it’s an industry standard way of measuring satisfaction. We actually have it in our fork and I’ve been meaning to open a PR. I’ll try and do that next week.

  11. dofinn commented on Jan 15, 2026

    @dofinn

    @sgarfinkel APDEX solves a different problem. Native SLO Management and Alerting is found in most observability products today. SREs need this functionality to drive reliability in their orgs. I am a big +1 on this request. Suggest also looking into how Nobl9 does them as well. They have a very intuitive representation of SLOs, Burn rates and error budgets

  12. alok87 commented on Mar 12, 2026

    @alok87
    ContributorAuthor

    Are we building this @MikeShi42 ?

  13. alok87 commented on Apr 2, 2026

    @alok87
    ContributorAuthor

    @MikeShi42 @wrn14897 Can we prioritze this, we internally have few developers who can also work with you on this. We will share a spec/plan.

  14. teeohhem commented on Apr 13, 2026

    @teeohhem
    Contributor

    @alok87 With the recent addition of raw-sql charting and alerting in open source, we may be able to accomplish at least a subset of this in the interim. Discussing this with the product team currently for broader feature acceptance/prioritization.

  15. alok87 commented on May 30, 2026

    @alok87
    ContributorAuthor

    @teeohhem @MikeShi42 @wrn14897 Please priortize this, this is the next big thing missing and the design of this should keep the vendor agnostic thing still and not coupled the companies with a system what OTEL enabled.

    @teeohhem are you developing this in open? do you have a RFC or some design doc for it now?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions