Saga Pattern + Circuit Breakers + Caching + Agent Health Monitoring

[Mosss-OS]

Summary

Webscout's multi-agent architecture is already well-designed. It can benefit from improved resilience patterns.

Recommended Changes

1. Saga Pattern for Distributed Agent Transactions

• Agent workflows span multiple services (Discovery → Matching → Action):
  • Implement the Saga Pattern with compensating transactions
  • If Action Agent fails after Discovery succeeded, trigger compensation (mark opportunity as unscored)
  • Each agent reports completion/failure to the Orchestrator

2. Circuit Breakers for External APIs

• Add circuit breakers for all external dependencies:
  • Apify API: if rate limited, circuit opens → fallback to cached results
  • OpenAI/Groq API: if down, circuit opens → fallback to heuristic matching
  • Zynd/Superplane: if unavailable, continue with local logging
• Use a library like opossum for circuit breaker implementation

3. Caching Strategy

• Cache Apify results with TTL (opportunities change hourly, not minutely):
  • In-memory cache for hot opportunities (LRU, 15 min TTL)
  • Supabase persistent cache (1 hour TTL)
• Cache AI-generated application drafts (hash of opportunity+user → draft)

4. Agent Health Monitoring

• Add a health dashboard for all agents:
  • Last successful run timestamp
  • Error rate per agent
  • Queue depth (pending opportunities to process)
  • API credit consumption (Apify, OpenAI)
• Implement agent alerting: if an agent hasn't run in N minutes, notify

5. Rate Limiting

• Per-telegram-user rate limiting (prevent spam)
• Per-agent rate limiting to respect external API quotas
• Implement token bucket algorithm

6. Testing

• Unit tests for each agent in isolation
• Integration tests for agent → agent communication
• E2E tests for: user sets profile → /scout → opportunities returned → application drafted