🛡️ CTI Agent — Open Source Cyber Threat Intelligence Platform

Automated threat intelligence collection, enrichment, IOC management, and feed distribution. STIX 2.1 · TAXII 2.1 · REST API · Slack Alerts · Cloudflare Edge Caching

---

✨ Features

Capability	Details
11 Intelligence Sources	CISA KEV, NVD/CVE, AlienVault OTX, Abuse.ch (URLhaus + MalwareBazaar + ThreatFox), MISP Public Feeds, MITRE ATT&CK, PhishTank, Feodo Tracker, Emerging Threats
All IOC Types	IP addresses, domains, URLs, SHA256/SHA1/MD5 hashes, email addresses, CVE IDs, YARA rules
STIX 2.1 / TAXII 2.1	Industry-standard feed format, TAXII collections for IOC consumers
Enrichment	GeoIP + ASN (MaxMind GeoLite2 or ip-api.com), VirusTotal scoring, Shodan port/vuln data, MITRE ATT&CK technique mapping
REST API	JSON feeds, CSV/TXT exports, rate-limited (5 req/min per IP)
Dashboard	Live read-only UI with charts, DataTables, collection run status
Slack Alerts	Real-time alerts for critical/high threats, daily digest summary
Cloudflare Worker	Optional edge proxy with KV-based rate limiting and response caching
Cross-platform	macOS (Intel/Apple Silicon) + Linux (Ubuntu/Debian/Oracle/RHEL)

---

🚀 Quick Install

git clone https://github.com/tapasrc123/cti-agent.git
cd cti-agent
bash install.sh

The installer will:

1. Detect your OS (macOS / Debian / Oracle Linux / RHEL)
2. Install system dependencies (Python 3.9+, PostgreSQL)
3. Create a dedicated PostgreSQL database and user
4. Set up a Python virtualenv and install all dependencies
5. Generate a .env configuration file
6. Initialise the database schema
7. Register a system service (macOS LaunchAgent / systemd)

Options

bash install.sh --no-service          # Skip service installation
bash install.sh --port=9000           # Use a custom port
bash install.sh --db-url="postgresql://user:pass@host/db"  # Use existing DB

---

⚙️ Configuration

Edit .env after installation:

# Required for full functionality
OTX_API_KEY=your_otx_key          # https://otx.alienvault.com/api (free)
VIRUSTOTAL_API_KEY=your_vt_key    # https://virustotal.com (free 500 req/day)
PHISHTANK_API_KEY=your_pt_key     # https://phishtank.com (free)
NVD_API_KEY=your_nvd_key          # https://nvd.nist.gov/developers (free, higher rate)

# Alerting
SLACK_WEBHOOK_URL=https://hooks.slack.com/...
SLACK_ALERT_SEVERITY=HIGH         # Alert on HIGH and CRITICAL

# Optional: MaxMind GeoLite2 (better GeoIP accuracy)
MAXMIND_LICENSE_KEY=your_key      # https://maxmind.com/en/geolite2/signup

---

🖥️ Usage

# Activate virtualenv
source venv/bin/activate

# Start server + scheduler (hourly collection)
python main.py serve

# Run one collection cycle immediately
python main.py collect

# Initialise DB only
python main.py init-db

---

🌐 API Endpoints

All API endpoints are rate-limited to 5 requests/minute per IP.

Threat Intelligence

Method	Endpoint	Description
GET	/api/threats	List threats (filter: severity, source, CVE, days)
GET	/api/threats/{id}	Threat detail with associated IOCs
GET	/api/threats/stats	Severity/source/type breakdown + 7d trend
GET	/api/iocs	List IOCs (filter: type, threat_level, source)
GET	/api/iocs/search?value=x	Search for a specific indicator
GET	/api/iocs/{id}	IOC detail with enrichment data
GET	/api/iocs/stats	IOC statistics and top malware families

Feeds & Exports

Method	Endpoint	Description
GET	/api/stix/bundle	Full STIX 2.1 Bundle (IOCs + Vulnerabilities)
GET	/api/stix/indicators	STIX Indicator objects
GET	/api/stix/vulnerabilities	STIX Vulnerability objects
GET	/taxii/	TAXII 2.1 Discovery
GET	/taxii/cti/collections/	List TAXII collections
GET	/taxii/cti/collections/{id}/objects/	TAXII feed objects
GET	/api/export/iocs.csv	CSV export of IOCs
GET	/api/export/threats.csv	CSV export of threats
GET	/feeds/ips.txt	Plain-text IP blocklist (?min_level=HIGH&days=N)
GET	/feeds/domains.txt	Plain-text domain blocklist
GET	/feeds/urls.txt	Plain-text URL blocklist
GET	/feeds/hashes.txt	Plain-text SHA-256 hash list
GET	/feeds/md5.txt	Plain-text MD5 hash list
GET	/feeds/sha1.txt	Plain-text SHA-1 hash list
GET	/feeds/emails.txt	Plain-text malicious email list
GET	/feeds/cves.txt	Plain-text CVE list
GET	/feeds/{type}	Generic: any IOC type, .txt suffix optional

Dashboard

URL	Description
/dashboard	Main dashboard (KPIs, charts, recent threats)
/dashboard/threats	Full threats table
/dashboard/iocs	Full IOC table with enrichment
/docs	Swagger UI (interactive API docs)
/health	Health check

---

📡 TAXII 2.1 Collections

Three pre-defined collections:

Collection ID	Name	Contents
364ced75-0001-4cti-0000-000000000001	IOC Indicators	All active IOCs as STIX Indicators
364ced75-0001-4cti-0000-000000000002	Vulnerabilities	CVE-linked STIX Vulnerabilities
364ced75-0001-4cti-0000-000000000003	All Objects	Combined bundle

Subscribe from a TAXII client (e.g., OpenCTI, MISP):

TAXII Server: http://your-server:8000/taxii/
API Root:     http://your-server:8000/taxii/cti/

---

☁️ Cloudflare Worker Deployment

cd cloudflare

# Install Wrangler
npm install -g wrangler

# Authenticate
wrangler login

# Create KV namespace for rate limiting
wrangler kv:namespace create "CTI_RATE_LIMIT"
# Copy the ID into wrangler.toml

# Set secrets
wrangler secret put ORIGIN_URL     # e.g. http://152.x.x.x:8000
wrangler secret put CF_WORKER_TOKEN

# Deploy
wrangler deploy

The Worker provides:

• Edge-level rate limiting (5 req/min per IP via KV)
• Response caching for STIX/TAXII feeds (5 min TTL)
• CORS headers
• DDoS protection
• Custom domain routing

---

🏗️ Architecture

cti-agent/
├── install.sh              # Cross-platform installer
├── main.py                 # CLI entry point (serve|collect|init-db)
├── config/                 # Pydantic settings
├── storage/                # SQLAlchemy models + PostgreSQL
│   ├── models.py           # Threat, IOC, ThreatIOC, CollectionRun
│   └── database.py         # Async engine
├── collectors/             # 11 intelligence source collectors
│   ├── cisa_kev.py         # CISA Known Exploited Vulnerabilities
│   ├── nvd_cve.py          # NIST NVD
│   ├── otx.py              # AlienVault OTX
│   ├── abusech.py          # URLhaus + MalwareBazaar + ThreatFox
│   ├── misp_feeds.py       # MISP + Emerging Threats + IPSum
│   ├── shodan.py           # Shodan InternetDB enrichment
│   ├── virustotal.py       # VT enrichment
│   ├── mitre_attack.py     # MITRE ATT&CK
│   ├── phishtank.py        # PhishTank
│   └── feodo_tracker.py    # Feodo C2 tracker
├── enrichment/             # GeoIP/ASN + MITRE mapping
├── api/                    # FastAPI application
│   ├── routers/            # threats, iocs, stix, taxii, health
│   └── rate_limiter.py     # slowapi (5 req/min/IP)
├── dashboard/              # Jinja2 templates + Bootstrap 5
├── alerting/               # Slack webhook notifications
├── scheduler/              # APScheduler (hourly collection)
└── cloudflare/             # Worker JS + wrangler.toml

---

🔒 Security Notes

• All API endpoints are read-only (GET/HEAD only)
• Rate limited at 5 req/min/IP by default; set RATE_LIMIT_ENABLED=false in .env to disable
• Plain-text feed endpoints (/feeds/*) are exempt from rate limiting for automated consumers
• Cloudflare Worker adds additional edge protection
• No authentication required (designed as an internal/VPN-protected service)
• For production: place behind a reverse proxy (nginx/Caddy) with TLS

---

📦 Free API Keys Needed

Service	URL	Rate Limit
AlienVault OTX	https://otx.alienvault.com/api	Generous free tier
Abuse.ch	https://abuse.ch/ (register → My Account)	Free — enables URLhaus, MalwareBazaar, ThreatFox
VirusTotal	https://virustotal.com/gui/join-us	500 req/day (free)
PhishTank	https://phishtank.com/api_info.php	1000 req/hour with key
NVD	https://nvd.nist.gov/developers	50 req/30s with key
MaxMind GeoLite2	https://maxmind.com/en/geolite2/signup	Free (DB download)

All collectors work without API keys at reduced rate limits. Set ABUSE_CH_API_KEY to unlock SHA-256 / MD5 / SHA-1 malware hash feeds.

---

📄 License

MIT License — Free to use, modify, and distribute.

---

Built for defenders. Powered by open-source threat intelligence.

cti-agent