METRICS.md

Metrics Reference

All metrics are exported via the Prometheus text format at GET /metrics on the configured port (default: 9100).

Process Metrics

Per-process gauges for the top-N processes (default 40) by each sort dimension. Labels identify the process and its context.

Labels

Label	Description	Example
pid	Process ID	1234
user	Username (resolved from UID)	root, www-data
command	Process name from /proc/[pid]/comm	nginx, python3
runtime	Container runtime	host, docker, kubernetes, containerd
rank	Position in sort order (1 = highest)	1, 2, ... 40
hostname	System hostname	web-01
container_id	Container ID (empty if host)	abc123def456
ports	Comma-separated listening TCP ports	80,443
full_path	Executable path	/usr/sbin/nginx

Gauges

Metric	Type	Unit	Description
sentry_process_cpu_percent	Gauge	%	CPU usage. 100% = 1 full core (matches top output)
sentry_process_memory_bytes	Gauge	bytes	Resident Set Size (RSS) in bytes
sentry_process_disk_read_bytes	Gauge	bytes	Cumulative disk read bytes from /proc/[pid]/io
sentry_process_disk_write_bytes	Gauge	bytes	Cumulative disk write bytes from /proc/[pid]/io

Note: Processes with a value of 0 for a metric are excluded from that metric's output to reduce cardinality.

Example Output

sentry_process_cpu_percent{pid="1234",user="www-data",command="nginx",runtime="docker",rank="1",hostname="web-01",container_id="abc123",ports="80,443",full_path="/usr/sbin/nginx"} 45.2
sentry_process_memory_bytes{pid="1234",user="www-data",command="nginx",runtime="docker",rank="1",hostname="web-01",container_id="abc123",ports="80,443",full_path="/usr/sbin/nginx"} 524288000

System Metrics

Metric	Type	Unit	Description
sentry_system_cpu_percent	Gauge	%	System-wide CPU usage (0-100%)
sentry_system_memory_percent	Gauge	%	System-wide memory usage
sentry_system_disk_percent	Gauge	%	Root filesystem (/) usage
sentry_processes_total	Gauge	count	Total number of processes scraped

Scrape Internals

Metric	Type	Unit	Description
sentry_scrape_duration_seconds	Histogram	seconds	Time taken for each scrape cycle. Buckets: exponential from 1ms to ~32s
sentry_scrape_errors_total	Counter	count	Number of failed scrape cycles (e.g., procfs read failure)

Crash Event Metrics

Metric	Type	Description
sentry_crash_events_total	Counter	Total crash events triggered (threshold breached)
sentry_critical_active	Gauge	1 if a crash event is currently active, 0 otherwise

Log Metrics

Metric	Type	Labels	Description
sentry_log_error_count	Counter	severity, source	Matched log events by severity and file source

Label values:

Label	Values
severity	critical, error, warning, info
source	File path (e.g., /var/log/syslog) or journald

Attack Detection Metrics

Metric	Type	Labels	Description
sentry_attack_detected_total	Counter	attack_type, severity	Detected security attack patterns in log lines

Label values:

Label	Values
attack_type	SQL Injection, XSS, Path Traversal, Scanner
severity	critical, high, medium

GPU Metrics

Only present when nvidia-smi is available. One time series per GPU.

Metric	Type	Labels	Unit	Description
sentry_gpu_utilization_percent	Gauge	gpu_id	%	GPU compute utilization
sentry_gpu_memory_used_mb	Gauge	gpu_id	MB	GPU memory in use
sentry_gpu_temperature_celsius	Gauge	gpu_id	C	GPU temperature

eBPF Metrics

Metric	Type	Description
sentry_ebpf_exec_events_total	Counter	Process execution events captured by eBPF or /proc fallback

Cardinality

With default settings (top_n=40), worst-case cardinality per scrape:

Metric Family	Max Series
Process metrics (4 families x 40 processes)	160
System metrics	4
Log counters (4 severities x N sources)	~12
Attack counters (4 types x 3 severities)	~12
GPU metrics (3 families x N GPUs)	~12
Other (scrape, crash, ebpf)	5
Total	~205

Old process label sets are cleared each cycle to prevent unbounded growth from PID recycling.

Prometheus Scrape Config

scrape_configs:
  - job_name: 'sentry-node'
    scrape_interval: 10s
    static_configs:
      - targets: ['localhost:9100']

Recommended scrape interval: 10s (agent collects every 5s, so 10s scrape gives fresh data without redundancy).

Useful PromQL Queries

# Top 5 CPU consumers right now
topk(5, sentry_process_cpu_percent)

# System CPU trend over 1 hour
rate(sentry_system_cpu_percent[1h])

# Memory usage by container
sum by (container_id) (sentry_process_memory_bytes{container_id!=""})

# Crash event rate
rate(sentry_crash_events_total[1h])

# Log error rate by severity
rate(sentry_log_error_count[5m])

# Attack detection rate
rate(sentry_attack_detected_total[1h])

# Scrape duration p95
histogram_quantile(0.95, rate(sentry_scrape_duration_seconds_bucket[5m]))

# Is a critical event active right now?
sentry_critical_active == 1