Skip to content

CONFIGURATION.md

Latest commit

 

History

History
159 lines (121 loc) · 6 KB

CONFIGURATION.md

File metadata and controls

159 lines (121 loc) · 6 KB
Raw

Configuration Reference

Sentry-Node is configured via config.toml with environment variable overrides. Environment variables take precedence over the TOML file.

Config File

Default location: config.toml in the working directory.

[server]
port = 9100              # HTTP server port for /metrics and /health
hostname = ""            # Hostname label; empty = auto-detect from OS

[scraper]
interval_seconds = 5     # Metrics collection interval
top_n = 40               # Number of top processes to export per dimension
enable_disk_io = true    # Read /proc/[pid]/io (requires SYS_PTRACE)
enable_ports = true      # Resolve TCP listening ports (expensive)
procfs_path = "/proc"    # Override for Docker: /host/proc

[thresholds]
cpu_warning = 80.0       # Warning threshold (currently unused — reserved)
cpu_critical = 90.0      # CPU% that triggers a crash event
mem_warning = 80.0       # Warning threshold (reserved)
mem_critical = 90.0      # Memory% that triggers a crash event
disk_warning = 85.0      # Warning threshold (reserved)
disk_critical = 90.0     # Disk% that triggers a crash event
hysteresis = 5.0         # Delta below critical to resolve a crash event

[log]
watch_paths = [          # Log files to tail (inotify)
  "/var/log/syslog",
  "/var/log/auth.log"
]
keywords = [             # Keywords to match (Aho-Corasick, case-insensitive)
  "CRITICAL", "ERROR", "FATAL", "WARNING",
  "SEGFAULT", "OOM", "panic", "kill"
]

[loki]
url = ""                 # Loki push URL; empty = disabled
batch_size = 100         # Flush after this many entries
flush_interval_ms = 1000 # Reserved for timer-based flush (not yet implemented)

[correlation]
enabled = true           # Enable multi-signal correlation engine

Environment Variable Overrides

Variable Config Path Type Default Description
METRICS_PORT server.port u16 9100 HTTP server port
PROC_HOSTNAME server.hostname string auto Hostname label for all metrics
TOP_N scraper.top_n usize 40 Top-N processes per sort dimension
PROCFS_PATH scraper.procfs_path string /proc Path to mounted procfs
ENABLE_DISK_IO scraper.enable_disk_io bool true Enable disk I/O metrics
ENABLE_PORTS scraper.enable_ports bool true Enable TCP port resolution
LOKI_URL loki.url string "" Loki push endpoint
THRESHOLD thresholds.{cpu,mem,disk}_critical f64 90 Sets all three critical thresholds

Boolean parsing: "false" disables; any other value (including "true", "1", "yes") enables.

Validation Rules

The following values are automatically corrected at startup with a warning:

Condition Correction
interval_seconds = 0 Reset to 5
top_n = 0 Reset to 40
port = 0 Reset to 9100
cpu_critical <= hysteresis Reset hysteresis to 5.0

Threshold Behavior

The threshold engine uses a state machine with hysteresis to avoid flapping:

Event starts when:  cpu >= cpu_critical  OR  mem >= mem_critical  OR  disk >= disk_critical
Event resolves when: cpu < (cpu_critical - hysteresis)
                 AND mem < (mem_critical - hysteresis)
                 AND disk < (disk_critical - hysteresis)

With defaults (critical=90, hysteresis=5):

  • Event triggers at 90%
  • Event resolves when ALL metrics drop below 85%

Docker Configuration

When running in Docker, the host's procfs must be mounted and the path overridden:

volumes:
  - /proc:/host/proc:ro        # Mount host procfs
environment:
  - PROCFS_PATH=/host/proc     # Tell sentry-node where to find it

Required capabilities:

Capability Purpose
SYS_PTRACE Read /proc/[pid]/io and /proc/[pid]/fd
DAC_READ_SEARCH Read other users' process information
NET_ADMIN Read process network namespace

Additional settings:

Setting Value Purpose
pid: host Required Access host PID namespace
read_only: true Recommended Immutable container filesystem
oom_score_adj: -1000 Recommended Protect agent from OOM killer
apparmor=unconfined May be needed Bypass AppArmor restrictions

Keyword Severity Mapping

Keywords are case-insensitive. Severity is determined by the keyword itself:

Severity Keywords
Critical CRITICAL, FATAL, SEGFAULT, OOM
Error ERROR, PANIC, KILL
Warning WARNING, WARN
Info Any other custom keyword

When a log line matches multiple keywords, the highest severity is reported.

Correlation Rules

Built-in rules (not configurable via TOML — hardcoded):

Rule Metric Condition Log Condition Severity
DISK_IO_WRITE_ERROR disk > 90% "Write Error" critical
CPU_SPIKE_WITH_OOM cpu > 80% "OOM" critical
MEMORY_EXHAUSTION mem > 90% "kill" critical
AUTH_FAILURE_UNDER_LOAD cpu > 70% "FATAL" high

Loki Integration

When LOKI_URL is set (e.g., http://loki:3100):

  • Log entries are batched (default: 100 entries per push)
  • Entries are grouped by severity into separate Loki streams
  • Crash event start/resolution notifications are flushed immediately
  • Stream labels: {job="sentry-node", host="<hostname>", severity="<level>"}

Performance Tuning

Parameter Effect of increasing Recommendation
interval_seconds Lower CPU usage, less granular data 5s for production, 1s for debugging
top_n More processes exported, higher cardinality 20-40 for most deployments
enable_ports=false Saves ~2-5ms per cycle Disable if port info not needed
enable_disk_io=false Avoids /proc/[pid]/io reads (requires SYS_PTRACE) Disable if disk I/O not needed
batch_size (Loki) Fewer HTTP requests, higher latency to Loki 50-200 depending on log volume