feat: Add comprehensive logging system with VERBOSE level

[poiley]

Summary

This PR implements a comprehensive logging system with improved CLI architecture, structured logging, and the new VERBOSE log level for detailed API tracking.

Related Issue: #35

🎯 Major Features

Comprehensive Logging System

• Centralized logging module (logging_config.py) with JSON formatting
• Five log levels: ERROR, WARNING, INFO, VERBOSE, DEBUG
• Multiple log types: main, API, analysis, debug with automatic rotation
• Automatic log directory setup at ~/.rulectl/logs/

Enhanced CLI Architecture (BREAKING CHANGES)

• NEW: Direct rulectl logs command (replaces rulectl config logs)
• Intuitive command structure following industry standards (docker logs, kubectl logs)
• Clean separation: config=settings, logs=output, start=execution

VERBOSE Log Level

• Perfect balance: API details without debug noise
• Tracks: API calls, token usage, rate limiting, request patterns
• Level 15: Between INFO (20) and DEBUG (10)
• Replaces --verbose-logging flag with cleaner log level system

Comprehensive Test Suite

• 58 test cases covering all logging functionality
• Core tests: Log levels, JSON formatting, structured logging
• CLI tests: Options, commands, error handling, integration
• API tests: Rate limiter, token tracker, performance monitoring
• Test runner and complete documentation

🔧 New CLI Commands

Log Viewing (NEW Direct Command)

rulectl logs                          # Show recent main logs
rulectl logs --type api               # Show API logs  
rulectl logs --type analysis          # Show analysis summaries
rulectl logs --type debug             # Show debug information
rulectl logs --follow                 # Follow logs in real-time
rulectl logs --lines 100              # Show more lines

Log Configuration (Enhanced)

rulectl start --log-level VERBOSE     # API tracking level
rulectl start --log-level DEBUG       # Full debug level
rulectl start --log-dir ./logs        # Custom log location

📊 Enhanced API Monitoring

• Detailed API call tracking with timing and success/failure logging
• Rate limiting event logging with backoff strategies and delays
• Token usage monitoring with cost calculations per analysis phase
• Structured JSON logs for easy parsing and analysis

🗂️ Log Organization

• Main Log: ~/.rulectl/logs/rulectl.log - General application events
• API Log: ~/.rulectl/logs/api-calls-YYYY-MM.log - Detailed API tracking
• Analysis Log: ~/.rulectl/logs/analysis-YYYY-MM-DD.log - Daily summaries
• Debug Log: ~/.rulectl/logs/debug.log - Technical debugging (JSON)

🚨 Breaking Changes

CLI Command Changes

# Old (no longer works)
rulectl config logs --log-type api --follow

# New (required)
rulectl logs --type api --follow

Removed Options

• ❌ --verbose-logging flag (replaced by --log-level VERBOSE)
• ❌ rulectl config logs (moved to direct rulectl logs)

Migration Guide

• rulectl config logs → rulectl logs
• --log-type → --type
• --verbose-logging → --log-level VERBOSE

💡 Benefits

• 🔍 Full transparency: Complete visibility into rulectl operations
• 💰 Cost monitoring: Real-time API usage and expense tracking
• 🛠️ Easy debugging: Structured logs make troubleshooting simple
• 📊 Performance insights: Identify bottlenecks and optimization opportunities
• 🏗️ Better architecture: Intuitive CLI following industry standards
• 📝 Audit trail: Complete record of all analysis runs

📚 Documentation

• ✅ Complete logging guide in LOGGING.md
• ✅ Updated README with all new features and examples
• ✅ CLI help integration with clear option descriptions
• ✅ Comprehensive test documentation in tests/README.md
• ✅ Usage examples for common monitoring scenarios

🧪 Test Coverage

# Run all tests
python tests/run_logging_tests.py

# Run with pytest
pip install -r tests/test_requirements.txt
pytest tests/ -v

Test Results: 58 comprehensive test cases covering:

• ✅ Core logging functionality (VERBOSE level, JSON formatting)
• ✅ CLI integration (new commands, options, error handling)
• ✅ API monitoring (rate limiter, token tracker, performance)
• ✅ Real integration with actual CLI commands

⚡ Performance

• Efficient structured logging with minimal overhead
• Log rotation prevents disk space issues
• Conditional logging avoids unnecessary processing
• JSON format enables fast parsing and analysis

This implementation provides production-ready observability for rulectl with an intuitive CLI architecture that follows established patterns while maintaining comprehensive functionality.

🤖 Generated with Claude Code

[poiley]

I recommended merging the following PRs, rebasing this branch, and moving documentation to docs/ before merging this PR.

• ci: Add GitHub Actions test workflow and local CI testing #33
• docs: Initial docs folder creation and organization of existing documentation #30