feat: Implement ML-based anomaly detection to identify unusual logging patterns for issue #143 by anshul23102 · Pull Request #153 · Dharanish-AM/Logara-AI

anshul23102 · 2026-06-12T13:34:25Z

Summary

Resolves #143 by implementing a comprehensive ML-based anomaly detection system that identifies unusual logging patterns, enabling proactive security monitoring and incident response.

Problem Addressed

No anomaly detection mechanism
Unusual patterns missed
Security incidents not detected
No early warning system
Reactive-only approach

Solution Implemented

1. Multi-Model Detection Engine

Ensemble approach combining:

Statistical anomaly detection (Z-scores)
Behavioral pattern analysis
Error rate tracking
Message pattern shifting

2. Statistical Detection

Z-score and IQR-based outlier detection:

Configurable threshold (default: 3.0 sigma)
IQR multiplier (default: 1.5x)
Percentile calculations
Dynamic baseline learning

3. Behavioral Detection

Service behavior monitoring:

Error rate spike detection
Pattern distribution changes
Deviation from baseline
Learning phase support

4. Pattern Analysis

Log pattern tracking:

Frequent pattern identification
Pattern frequency calculation
Pattern shift detection
Service-specific patterns

5. Severity Classification

Automatic severity assignment:

CRITICAL: score > 0.8
HIGH: score 0.6-0.8
MEDIUM: score 0.4-0.6
LOW: score 0.2-0.4
INFO: score < 0.2

Technical Details

Detection Process

Log ingestion
Pattern analysis and update
Run all detectors:
- Statistical outlier check
- Behavioral change detection
- Error rate analysis
Ensemble scoring
Severity classification
Alert generation

Baseline Training

Learn normal behavior from historical logs:

Minimum 100 logs required
Calculates: mean, std, min, max, quartiles
Per-service baselines
Automatic updates

API Endpoints

POST /api/anomalies/analyze - Analyze single log
GET /api/anomalies/history/{service_id} - Get history
GET /api/anomalies/statistics/{service_id} - Get statistics
POST /api/anomalies/train - Train baseline
GET /api/anomalies/alerts/{service_id} - Get anomalies
GET /api/anomalies/summary - System summary
POST /api/anomalies/cleanup - Clean old data
GET /api/anomalies/health - Health check

Features

Multi-detector ensemble voting
Service-specific baselines
Adaptive learning
Pattern history tracking
Configurable thresholds
Comprehensive statistics
Integration with vector embeddings
Anomaly severity classification

Performance

Sub-second detection
Minimal CPU overhead
Efficient memory usage
Scalable to multiple services
Real-time monitoring

Integration Points

Works with log ingestion pipeline
Compatible with embeddings
Integrates with alert system
Works with existing APIs
Non-blocking operation

Use Cases

Security incident detection
Performance anomaly identification
Error rate spikes
Unusual behavior patterns
Early warning system
Compliance monitoring

Closes #143

anshul23102 · 2026-06-12T13:34:32Z

@Dharanish-AM Please review this PR for the GSSoC 2026 program.

Suggested Labels

gssoc-approved (GSSoC 2026 program label)
feature (anomaly detection system)
ml (machine learning)
security (security monitoring)
detection (anomaly detection)
enhancement (monitoring enhancement)

This PR implements a comprehensive ML-based anomaly detection system addressing issue #143, enabling proactive identification of unusual logging patterns through ensemble detection methods including statistical analysis, behavioral tracking, and pattern learning.

Dharanish-AM · 2026-06-12T13:34:42Z

⚠️ Merge Conflict Detected! This PR cannot be merged automatically because it conflicts with the main branch.