Skip to content

feat: add retention policies and cleanup jobs for auth_nonces, DLQ, processed_events, agent_logs (#161)#182

Merged
robertocarlous merged 1 commit into
Neurowealth:mainfrom
Nanle-code:feature/retention-cleanup-jobs
Jun 18, 2026
Merged

feat: add retention policies and cleanup jobs for auth_nonces, DLQ, processed_events, agent_logs (#161)#182
robertocarlous merged 1 commit into
Neurowealth:mainfrom
Nanle-code:feature/retention-cleanup-jobs

Conversation

@Nanle-code

Copy link
Copy Markdown
Contributor

Closes #161

What changed

Added scheduled data retention jobs to prevent unbounded table growth across four tables, with Prometheus metrics and configurable retention periods via environment variables.

New files

  • src/jobs/dataRetention.ts — cleanup functions for all four tables + scheduler
  • prisma/migrations/20260618000000_add_retention_indexes/migration.sql — indexes to support efficient cleanup queries

Modified files

  • src/config/env.ts — added config.retention block with four env-configurable retention settings
  • src/utils/metrics.ts — added retention_deletes_total{table} and retention_last_run_timestamp_seconds{table}
  • src/index.ts — registers scheduleDataRetention() after server starts; clears handle on graceful shutdown
  • .env.example — documents all new retention env vars with defaults

Retention defaults (all overridable via env)

Table Env var Default
processed_events RETENTION_PROCESSED_EVENTS_DAYS 90 days
dead_letter_events (RESOLVED only) RETENTION_DEAD_LETTER_EVENTS_DAYS 30 days
agent_logs RETENTION_AGENT_LOGS_DAYS 60 days
auth_nonces (expires naturally via expiresAt) immediate
Run interval RETENTION_INTERVAL_MS 24h

Notes

  • DLQ cleanup only removes RESOLVED records — PENDING and RETRIED are left intact
  • Jobs run once on startup to clear any backlog, then on the configured interval
  • All jobs follow the existing sessionCleanup pattern: try/catch, structured logging, metrics recording

@robertocarlous robertocarlous merged commit 2d87a91 into Neurowealth:main Jun 18, 2026
1 of 4 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Add retention policies + cleanup jobs for auth_nonces, DLQ, processed events

2 participants