🛡️ SentinelSH-AI

Self-Healing Server Monitoring & Auto-Remediation, Powered by Bash + AI

Most monitoring tools tell you something broke. SentinelSH-AI tells you why it broke, fixes it automatically when it's safe to do so, and writes you a postmortem about it — all in pure Bash, with an AI co-pilot, running for free on AWS or your own laptop.

🎯 Why This Project Exists

Every bash scripting tutorial ends the same way: a todo app, a backup script, a log parser. This project asks a different question —

What if a bash script could think like a junior SRE: notice something's wrong, reason about why, decide whether it's safe to fix, fix it, and then write up what happened?

SentinelSH-AI is that answer. It's a lightweight, dependency-free, fully autonomous monitoring daemon that combines classic shell scripting fundamentals with a real LLM (via the free Groq API) to bring genuine reasoning into the ops loop — not just if cpu > 80 then alert.

✨ What Makes It Different

Feature	Typical Monitoring Script	SentinelSH-AI
Detects issues	✅	✅
Auto-remediates	⚠️ Sometimes, blindly	✅ With AI safety review first
Explains why it happened	❌	✅ AI root-cause diagnosis
Generates incident postmortems	❌	✅ Auto-written Markdown reports
Learns "normal" for your server	❌	✅ Baseline learning + anomaly detection
Natural language log queries	❌	✅ `ask.sh "why was the server slow yesterday?"`
Weekly AI-written health summary	❌	✅
Extensible without touching core code	❌	✅ Drop-in plugin architecture
Real-time alerts	Maybe	✅ Discord webhook
Live dashboard	❌	✅ Auto-refreshing HTML dashboard

🧠 The standout idea: AI as a safety guardrail, not a blind trigger

Before executing any destructive remediation (clearing logs, restarting services), SentinelSH-AI asks the AI: "is this actually safe given the current context?" If the AI has concerns, it halts and flags for manual review instead of charging ahead. This mirrors how a cautious human engineer would actually operate — automation with judgment, not automation with a hair-trigger.

🏗️ Architecture

                     ┌─────────────────────┐
                     │   systemd.timer      │  (runs every 2 min)
                     └──────────┬───────────┘
                                │
                     ┌──────────▼───────────┐
                     │     sentinel.sh        │  (main orchestrator)
                     └──────────┬───────────┘
            ┌───────────────────┼───────────────────┐
            ▼                   ▼                   ▼
   ┌────────────────┐ ┌──────────────────┐ ┌──────────────────┐
   │  checks/*.sh     │ │  lib/baseline.sh   │ │  lib/dashboard.sh  │
   │  (plugin system)  │ │  (anomaly detect)  │ │  (status.html)     │
   └────────┬────────┘ └──────────────────┘ └──────────────────┘
            │ on FAILURE
            ▼
   ┌────────────────────┐        ┌─────────────────────┐
   │  ai/diagnose.sh       │──────▶│   Groq API (LLaMA 3.3) │
   │  (root cause analysis) │      └─────────────────────┘
   └────────┬────────────┘
            ▼
   ┌────────────────────┐        ┌─────────────────────┐
   │  ai/safety_check.sh   │──────▶│   Groq API (verdict)   │
   │  (SAFE / UNSAFE gate)  │      └─────────────────────┘
   └────────┬────────────┘
            │ if SAFE
            ▼
   ┌────────────────────┐    ┌────────────────────┐    ┌────────────────────┐
   │ remediate/*.sh        │    │ lib/report.sh         │    │ lib/notify.sh          │
   │ (auto-fix plugin)     │    │ (Markdown postmortem) │    │ (Discord webhook)     │
   └────────────────────┘    └────────────────────┘    └────────────────────┘

📂 Project Structure

sentinelsh-ai/
├── sentinel.sh              # Main orchestrator — auto-discovers and runs everything
├── install.sh                # One-command installer for fresh machines
├── config.conf.example        # Copy to config.conf and fill in your keys
├── Dockerfile                  # Run the whole thing in a container
├── docker-compose.yml          # One-command Docker deployment
│
├── checks/                    # 🔌 Plugin system — drop a new .sh here, it's auto-discovered
│   ├── cpu_check.sh
│   ├── disk_check.sh
│   ├── memory_check.sh
│   ├── load_avg_check.sh
│   └── service_check.sh
│
├── remediate/                  # 🔧 Auto-fix scripts, matched to checks by resource name
│   ├── clear_logs.sh
│   └── restart_service.sh
│
├── ai/                          # 🧠 Groq-powered intelligence layer
│   ├── ai_client.sh             # Core Groq API wrapper
│   ├── diagnose.sh               # Root cause analysis on incidents
│   ├── safety_check.sh            # SAFE/UNSAFE guardrail before remediation
│   ├── ask.sh                      # Natural language log queries
│   └── weekly_report.sh             # AI-written weekly health summary
│
├── lib/                         # Shared utilities
│   ├── logger.sh
│   ├── report.sh                 # Markdown incident report generator
│   ├── baseline.sh                # Metric history + anomaly detection
│   ├── notify.sh                  # Discord webhook integration
│   └── dashboard.sh               # Live HTML dashboard generator
│
├── deploy/                      # systemd + logrotate configs for production
│   ├── sentinelsh.service
│   ├── sentinelsh.timer
│   └── sentinelsh.logrotate
│
├── reports/                     # Auto-generated incident & weekly reports (gitignored)
└── logs/                        # Runtime logs (gitignored)

📸 Screenshots & Proof of Work

All features below were tested end-to-end on a live AWS EC2 instance. See Quick Start to reproduce these yourself.

✅ Clean Healthy Run All plugins auto-discovered and passing, dashboard auto-regenerated.	📊 Live Auto-Refreshing Dashboard Health banner, color-coded metrics, progress bars — updates every 30s.
🚨 Full Incident Pipeline Detection → AI diagnosis → incident report → Discord alert, fully automated.	💬 Real-Time Discord Alert AI's actual root-cause explanation delivered straight to chat.
📝 Auto-Generated Incident Report A real Markdown postmortem — timestamp, root cause, action taken, system snapshot.	🗣️ Natural Language Log Queries `ask.sh "what happened to my server this week?"` — conversational ops assistant.
📅 AI-Written Weekly Report Reads like a real SRE wrote it — summary, incidents, recommendations.	🛑 AI Safety Guardrail in Action AI reviewing a proposed remediation before allowing it to execute.
🔌 Plugin Architecture Proof Five independent check plugins, auto-discovered with zero hardcoding.	⏱️ Fully Autonomous via systemd Timer fires every 2 minutes without any manual trigger.
🩹 The Self-Healing Moment — Service Crash → Auto-Restart → Verified `cron` service killed manually → detected by `service_check` → AI diagnosed it → `restart_service.sh` auto-fired → confirmed back online, all without human intervention.

💡 Note for repo setup: create a /screenshots folder in your repo root and drop your PNG files there using the filenames referenced above, or update the paths to match your own filenames.

🚀 Quick Start

You have three ways to run SentinelSH-AI. Pick whichever fits you.

Option A — Docker (Recommended, runs anywhere)

The fastest way to see it working, no AWS account or systemd needed. Works on Windows/Mac/Linux as long as Docker is installed.

# 1. Clone the repo
git clone https://github.com/Adarsh73111/sentinelsh-ai.git
cd sentinelsh-ai

# 2. Set your environment variables
export GROQ_API_KEY="your_groq_api_key_here"
export WEBHOOK_URL="your_discord_webhook_url_here"   # optional

# 3. Build and run
docker compose up --build

Then open your browser to:

http://localhost:8080/status.html

To stop:

docker compose down

Get a free Groq API key at console.groq.com/keys — no credit card required.

Option B — Local Machine / WSL / Linux

Run it directly on any Linux machine (or WSL on Windows) without Docker.

# 1. Clone the repo
git clone https://github.com/Adarsh73111/sentinelsh-ai.git
cd sentinelsh-ai

# 2. Run the installer (installs jq, curl, sets up folders & permissions)
chmod +x install.sh
./install.sh

# 3. Configure your keys
cp config.conf.example config.conf
nano config.conf   # add your GROQ_API_KEY and WEBHOOK_URL

# 4. Run it manually
bash sentinel.sh

# 5. (Optional) View the dashboard
python3 -m http.server 8080
# visit http://localhost:8080/status.html

install.sh will also offer to set up a systemd timer for automatic recurring runs — skip this step if you're just testing on a personal laptop.

Option C — AWS EC2 Free Tier (Full Production Setup)

For the complete experience with autonomous systemd scheduling, a public dashboard URL, and Discord alerting from a real server.

Click to expand full AWS setup guide

1. Launch an EC2 instance

AMI: Ubuntu Server 24.04 LTS, Free Tier eligible
Instance type: t2.micro
Storage: 30 GB gp2
Security Group inbound rules:

Type Port Source

SSH 22 My IP

Custom TCP 8080 Anywhere

2. (Recommended) Allocate an Elastic IP

So your dashboard URL never changes after stop/start: EC2 → Elastic IPs → Allocate → Associate with your instance.

3. SSH in and set up the project

ssh -i "your-key.pem" ubuntu@<your-ec2-ip>

git clone https://github.com/Adarsh73111/sentinelsh-ai.git ~/sentinelsh
cd ~/sentinelsh
chmod +x install.sh
./install.sh

4. Configure

cp config.conf.example config.conf
nano config.conf

Add your GROQ_API_KEY and WEBHOOK_URL.

5. Verify the systemd timer is running

sudo systemctl status sentinelsh.timer

6. Serve the dashboard

cd ~/sentinelsh
nohup python3 -m http.server 8080 > /dev/null 2>&1 &

Visit http://<your-ec2-ip>:8080/status.html

⚙️ Configuration

All settings live in config.conf (copy from config.conf.example):

Variable	Description	Default
`GROQ_API_KEY`	Your free Groq API key	(required)
`AI_MODEL`	Groq model to use	`llama-3.3-70b-versatile`
`DISK_THRESHOLD`	Disk usage % that triggers an alert	`80`
`CPU_THRESHOLD`	CPU usage % that triggers an alert	`75`
`MEMORY_THRESHOLD`	Memory usage % that triggers an alert	`80`
`MONITORED_SERVICES`	Space-separated systemd service names to watch	`ssh cron`
`WEBHOOK_URL`	Discord (or Slack-compatible) webhook for alerts	(optional)
`AI_SAFETY_CHECK`	Require AI approval before destructive remediation	`yes`

🛡️ How the AI Safety Guardrail Works

Before any destructive remediation action executes (e.g. clearing logs, deleting temp files), SentinelSH-AI sends the proposed action and its context to the AI and requires an explicit verdict:

Proposed action: Clear compressed logs, journal logs older than 2 days, apt cache, and /tmp
Context: Disk usage exceeded threshold

→ AI Verdict: SAFE
   "These are routine, reversible cleanup actions unlikely to cause disruption."

If the AI responds UNSAFE — for example, flagging that a file might be in active use — the remediation halts and the incident is logged for manual review instead of executing blindly. This is the difference between automation and automation with judgment.

🔌 Extending SentinelSH-AI (Plugin System)

Adding a new health check requires zero changes to sentinel.sh. Drop a script into checks/ that follows this contract:

#!/bin/bash
source ~/sentinelsh/config.conf
source ~/sentinelsh/lib/logger.sh

check_something() {
    # your check logic here
    if [ "$condition_failed" ]; then
        echo "FAIL:resource_name:value"
        return 1
    else
        echo "OK"
        return 0
    fi
}

check_something

To add an auto-fix for it, create remediate/<resource_name>_fix.sh (or a service-style fix that accepts the resource as $1) — sentinel.sh will auto-discover and wire it in.

🧰 Tech Stack

Bash — 100% of the orchestration logic, no Python/Node dependency for core functionality
Groq API (LLaMA 3.3 70B) — free-tier LLM inference for diagnosis, safety checks, and reports
AWS EC2 — Free Tier deployment target
systemd — timers and services for autonomous scheduling
Discord Webhooks — real-time alerting
Docker — portable, zero-dependency deployment option
jq — JSON parsing for API responses
HTML/CSS — self-contained live dashboard, no frontend framework

🗺️ Roadmap / Stretch Goals

AWS CLI integration — auto-snapshot EBS volume on critical disk alerts
Multi-server support with aggregated dashboard
Slack slash-command interface for ask.sh
Historical trend analysis with monthly AI reports
Configurable severity thresholds via a whiptail/dialog TUI menu

🤝 Contributing

This started as a personal learning project to push bash scripting beyond tutorial-level exercises. Issues, forks, and pull requests are welcome — especially new checks/ or remediate/ plugins.

📄 License

This project is licensed under the MIT License.

Built with 🐚 Bash, 🧠 Groq AI, and ☁️ AWS Free Tier — proof that infrastructure automation doesn't need a heavyweight framework to be genuinely intelligent.

Navigation Menu

Repository navigation

About

Repository files navigation

🛡️ SentinelSH-AI

Self-Healing Server Monitoring & Auto-Remediation, Powered by Bash + AI

📖 Table of Contents

🎯 Why This Project Exists

✨ What Makes It Different

🧠 The standout idea: AI as a safety guardrail, not a blind trigger

🏗️ Architecture

📂 Project Structure

📸 Screenshots & Proof of Work

🚀 Quick Start

Option A — Docker (Recommended, runs anywhere)

Option B — Local Machine / WSL / Linux

Option C — AWS EC2 Free Tier (Full Production Setup)

1. Launch an EC2 instance

2. (Recommended) Allocate an Elastic IP

3. SSH in and set up the project

4. Configure

5. Verify the systemd timer is running

6. Serve the dashboard

⚙️ Configuration

🛡️ How the AI Safety Guardrail Works

🔌 Extending SentinelSH-AI (Plugin System)

🧰 Tech Stack

🗺️ Roadmap / Stretch Goals

🤝 Contributing

📄 License

About

Releases

Packages

Used by

Contributors

Languages

Name		Name	Last commit message	Last commit date
ai		ai
checks		checks
deploy		deploy
lib		lib
remediate		remediate
screenshots		screenshots
.dockerignore		.dockerignore
.gitignore		.gitignore
Dockerfile		Dockerfile
LICENSE		LICENSE
README.md		README.md
config.conf.example		config.conf.example
docker-compose.yml		docker-compose.yml
install.sh		install.sh
sentinel.sh		sentinel.sh
status.html		status.html

Type	Port	Source
SSH	22	My IP
Custom TCP	8080	Anywhere

Latest commit

History

Folders and files

Repository files navigation

🛡️ SentinelSH-AI

Self-Healing Server Monitoring & Auto-Remediation, Powered by Bash + AI

📖 Table of Contents

🎯 Why This Project Exists

✨ What Makes It Different

🧠 The standout idea: AI as a safety guardrail, not a blind trigger

🏗️ Architecture

📂 Project Structure

📸 Screenshots & Proof of Work

🚀 Quick Start

Option A — Docker (Recommended, runs anywhere)

Option B — Local Machine / WSL / Linux

Option C — AWS EC2 Free Tier (Full Production Setup)

1. Launch an EC2 instance

2. (Recommended) Allocate an Elastic IP

3. SSH in and set up the project

4. Configure

5. Verify the systemd timer is running

6. Serve the dashboard

⚙️ Configuration

🛡️ How the AI Safety Guardrail Works

🔌 Extending SentinelSH-AI (Plugin System)

🧰 Tech Stack

🗺️ Roadmap / Stretch Goals

🤝 Contributing

📄 License

About

Resources

Stars

Watchers

Forks

Releases

Packages

Used by

Contributors

Languages