Skip to content

ARCHITECTURE.md

Latest commit

 

History

History
174 lines (140 loc) · 9.27 KB

ARCHITECTURE.md

File metadata and controls

174 lines (140 loc) · 9.27 KB
Raw

Architecture

Overview

Sentry-Node is a unified Linux observability agent. It replaces two separate Go services (proc-sentry + log-sentry) with a single high-performance Rust binary that scrapes process metrics, tails logs, detects attacks, correlates multi-signal events, and exports to Prometheus and Loki.

System Diagram

                ┌──────────────────────────────────────────────────────────┐
                │                     sentry-node                          │
                │                                                          │
  /proc/[pid]/* ─┤  ┌──────────┐     ┌──────────────┐    ┌─────────────┐  ├──► Prometheus
  /proc/stat    ─┤  │   proc   │────►│   Collector   │──►│   export/   │  │    /metrics
  /proc/meminfo ─┤  │  module  │     │   (5s loop)   │   │ prometheus  │  │
  statfs("/")  ──┤  └──────────┘     │               │   └─────────────┘  │
                │                    │  ┌──────────┐ │                     │
  /var/log/*   ──┤  ┌──────────┐    │  │  engine  │ │    ┌─────────────┐  ├──► Loki
  journald     ──┤  │   log    │───►│  │threshold │ │──►│   export/   │  │    /loki/api/
                │  │  module  │    │  │correlate │ │   │    loki     │  │    v1/push
                │  └──────────┘    │  │ lineage  │ │   └─────────────┘  │
                │                    │  └──────────┘ │                     │
  nvidia-smi   ──┤                   └───────────────┘                    ├──► /health
                │  ┌──────────┐                                           │
  /proc (poll) ──┤  │   ebpf   │  (future: aya tracepoints)              │
                │  └──────────┘                                           │
                └──────────────────────────────────────────────────────────┘

Module Breakdown

proc/ — System & Process Metrics

Reads Linux procfs to capture per-process and system-wide metrics.

File Responsibility Data Source
process.rs Per-PID stat/status/io/comm/exe/cmdline/oom_score/fd scanning /proc/[pid]/*
cpu.rs System-wide and per-process CPU via delta-based jiffy calculation /proc/stat
memory.rs System memory (total, available) /proc/meminfo
disk.rs Root filesystem usage via statfs() syscall /
network.rs TCP port resolution via socket inode cross-reference /proc/[pid]/fd, /proc/[pid]/net/tcp{,6}
container.rs Docker/K8s/containerd runtime detection from cgroup paths /proc/[pid]/cgroup
gpu.rs NVIDIA GPU utilization, memory, temperature nvidia-smi subprocess

Key design decisions:

  • Port resolution is expensive (fd enumeration + net/tcp parsing per PID). Only performed for top-N "winner" processes, not all PIDs.
  • CPU percentage uses delta-based calculation with per-core normalization: (proc_tick_delta / sys_total_delta) * 100 * num_cores. This matches top output where 100% = 1 full core.
  • User map (UID -> username) is loaded once from /etc/passwd via LazyLock and cached for the lifetime of the process.

log/ — Log Aggregation & Analysis

File Responsibility Mechanism
tailer.rs File tailing with inotify + polling fallback, log rotation handling inotify crate, inode tracking
journald.rs Systemd journal reading journalctl -f -o json subprocess
filter.rs Keyword matching with severity ranking Aho-Corasick automaton (O(n) multi-pattern)
attack.rs SQL injection, XSS, path traversal, scanner UA detection Pre-compiled regex patterns

Data flow:

tailer.rs ──────► mpsc::channel(1000) ──► Log Consumer Task
journald.rs ───►                          │
                                          ├─ filter.check() → severity match → Prometheus counter
                                          ├─ attack.detect_attack() → attack match → Prometheus counter
                                          ├─ recent_log_events buffer → Correlation engine
                                          └─ loki.push() → Loki batch buffer

engine/ — Intelligence Layer

File Responsibility
threshold.rs State machine monitoring CPU/mem/disk against critical thresholds with hysteresis
crash_event.rs Event lifecycle: create, accumulate snapshots, resolve with verdict and severity
correlation.rs Multi-signal rule engine combining metric thresholds with log keyword presence
lineage.rs PPID chain walker for process ancestry tracking

Threshold state machine:

                     metric >= critical
         ┌────────┐ ─────────────────► ┌──────────┐
         │  Idle  │                    │  Active  │
         │        │ ◄───────────────── │  Event   │
         └────────┘  all metrics <     └──────────┘
                     (critical -        │
                      hysteresis)       │ each cycle:
                                        │  add_snapshot()
                                        ▼

Correlation rules (built-in):

Rule Conditions Severity
DISK_IO_WRITE_ERROR disk > 90% AND log contains "Write Error" critical
CPU_SPIKE_WITH_OOM cpu > 80% AND log contains "OOM" critical
MEMORY_EXHAUSTION mem > 90% AND log contains "kill" critical
AUTH_FAILURE_UNDER_LOAD cpu > 70% AND log contains "FATAL" high

export/ — Data Export

File Responsibility
prometheus.rs Registry of all metrics, label management, stale label cleanup, text encoding
loki.rs Batching HTTP client for Loki push API, groups entries by severity

Metric label lifecycle: Each scrape cycle, update_top_processes() removes the previous cycle's label sets and creates new ones. This prevents stale PIDs from persisting in Prometheus after process death.

ebpf/ — Short-Lived Process Capture

Currently uses a /proc polling fallback (500ms interval) to detect new PIDs. The eBPF tracepoint attachment via aya is stubbed out behind a --features ebpf flag.

Concurrency Model

tokio runtime (multi-threaded)
│
├─ Task 1: Log file tailer (spawn_blocking → inotify blocking reads)
│
├─ Task 2: Journald reader (async subprocess stdout)
│
├─ Task 3: Log consumer (async mpsc::Receiver → filter + attack + loki)
│
├─ Task 4: Collector loop (5s interval → proc scan + threshold + correlation + GPU)
│
├─ Task 5: Signal handler (SIGTERM/SIGINT → survival mode)
│
├─ Task 6: eBPF/proc fallback (std::thread, 500ms poll)
│
└─ HTTP Server: axum on port 9100 (/metrics, /health)

Configuration Hierarchy

config.toml (defaults)
    ↓
Config::load() → TOML parse
    ↓
Environment variable overrides (METRICS_PORT, PROCFS_PATH, etc.)
    ↓
Validation (clamp invalid values)
    ↓
Auto-detect hostname if empty

Security Model

  • Requires Linux capabilities: SYS_PTRACE, DAC_READ_SEARCH, NET_ADMIN
  • Container runs read_only: true with distroless base image
  • oom_score_adj: -1000 protects the agent from OOM killer
  • No shell in container (distroless) — reduced attack surface
  • Secrets: None stored; Loki URL is the only credential-adjacent config value

Data Flow Summary

/proc/[pid]/* ──► ProcessInfo[] ──► sort by CPU/Mem/DiskIO ──► top-N ──► Prometheus gauges
                                                                       └► Snapshot (for crash events)

/proc/stat ──────► CpuSnapshot ──► delta calculation ──► sys_cpu_pct ──► Prometheus gauge
                                                                        └► Threshold check

/var/log/* ──────► TailedLine ──► LogFilter ──► severity ──► Prometheus counter
journald ────────►             └► AttackDetect ──────────► Prometheus counter
                               └► recent_logs buffer ──► CorrelationEngine ──► alerts (log)
                               └► LokiPusher ──────────► Loki HTTP push

nvidia-smi ──────► GpuSnapshot[] ──► Prometheus gauges

eBPF/proc poll ──► ExecEvent[] ──► Prometheus counter