Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
17 changes: 15 additions & 2 deletions docs/queue-dead-letter-maintenance.md
Original file line number Diff line number Diff line change
@@ -1,18 +1,21 @@
# Queue dead-letter maintenance — changes made

Summary

- Bounded failed-job retention: default job options updated so failed jobs are retained only up to a bounded count instead of forever.
- Daily maintenance cron: a scheduled service prunes old failed jobs, persists a write-only record in Postgres, optionally reports to Sentry, and removes old failed jobs from Redis.
- Observability: a Prometheus gauge `bull_dead_letter_count{queue="..."}` is updated with the current failed-job count per queue.
- Persistence: added a Prisma model `DeadLetter` (mapped to table `dead_letters`) to store pruned failure records for audit.

Files added/modified

- Modified: [src/queue/queue.module.ts](src/queue/queue.module.ts) — set `removeOnFail: 1000`, registered `ScheduleModule`, and added the maintenance service
- Added: [src/queue/queue-maintenance.service.ts](src/queue/queue-maintenance.service.ts) — maintenance cron, Sentry integration, Prometheus metric
- Modified: [prisma/schema.prisma](prisma/schema.prisma) — added `DeadLetter` model
- Modified: [package.json](package.json) — added `@nestjs/schedule` and `prom-client` dependencies

Behavior details

- Default job options (applied to the main queues: `email`, `contract-events`, `analytics`, `export`):
- `attempts: 3`
- `backoff: { type: 'exponential', delay: 5000 }`
Expand All @@ -28,30 +31,40 @@ Behavior details
- Updates Prometheus gauge `bull_dead_letter_count` for each queue via `prom-client`.

Deployment & local steps

1. Install dependencies:

```bash
npm install
```

2. Generate Prisma client and run migrations (create migration to add `dead_letters`):

```bash
npx prisma generate
npx prisma migrate dev --name add_dead_letters
```

3. Set optional environment variables:
- `SENTRY_DSN` to enable Sentry reporting.
- `DATABASE_URL` for Postgres.

- `SENTRY_DSN` to enable Sentry reporting.
- `DATABASE_URL` for Postgres.

4. Build and run:

```bash
npm run build
npm start:dev
```

Notes & follow-ups

- I did not add an HTTP metrics endpoint — if you want `/metrics` exported for Prometheus scraping I can add a small controller that exposes `prom-client` metrics.
- The branch with these changes is: `feat/queue-dead-letter-maintenance`.
- Open PR URL: https://github.com/coderolisa/OrbitChain-API/pull/new/feat/queue-dead-letter-maintenance

If you'd like, I can also:

- Add an HTTP `/metrics` endpoint.
- Make the prune retention and retained-fail-count configurable via environment variables.
- Add unit tests around the maintenance service.
2 changes: 1 addition & 1 deletion eslint.config.mjs
Original file line number Diff line number Diff line change
Expand Up @@ -48,7 +48,7 @@ export default tseslint.config(
'@typescript-eslint/require-await': 'warn',
'@typescript-eslint/restrict-template-expressions': 'warn',
'@typescript-eslint/prefer-as-const': 'warn',
"prettier/prettier": ["error", { endOfLine: "lf" }],
'prettier/prettier': ['error', { endOfLine: 'lf' }],
},
},
);
Loading
Loading