feat: Add BullMQ background worker system (co-located now, split-server ready)

[Abhay2133]

Overview

We need to introduce a robust background task / job queue system using BullMQ.
Currently, background processing is minimal or synchronous. As we scale and add features like email dispatch, heavy computations, third-party integrations, or periodic syncs, we need a reliable, Redis-backed queue system with retries, delay support, and concurrency controls.

The architecture must support:

• Same-server execution (Now): Running the background worker on the same server as the primary Nuxt/Nitro application.
• Separate-server execution (Future): Offloading the background worker to a separate dedicated server/service/container without modifying the application code, simply by pointing it to the same shared Redis instance.

---

Design & Architecture

1. Separation of Concerns

To make the transition to a separate server seamless in the future, the background worker should run as a separate process rather than inline within the Nuxt/Nitro event loop.

• Producer (Nuxt/Nitro API): imports Queue from BullMQ, defines jobs, and adds them to Redis.
• Consumer (Worker Process): A standalone script (e.g., server/worker.ts or a top-level worker.ts run via Bun) that imports Worker and processes jobs.

2. PM2 Process Configuration (ecosystem.config.js)

We will configure PM2 to run both the API and the Worker as separate applications. This makes it trivial to split them later:

module.exports = {
  apps: [
    {
      name: "api21",
      script: ".output/server/index.mjs",
      interpreter: "bun",
      // ...
    },
    {
      name: "api21-worker",
      script: "server/worker.ts", // or compiled equivalent
      interpreter: "bun",
      instances: 1,
      autorestart: true,
      watch: false,
      env: {
        NODE_ENV: "production",
      }
    }
  ]
};

---

Tasks to Complete

Phase 1: Setup and Configuration

• Install BullMQ (bullmq) and any necessary Redis client dependencies (typically ioredis is preferred/used internally by BullMQ).
• Define shared connection configurations for Redis that both Nitro and the worker process can consume.
• Create a utility structure for defining queue names and job payloads in a type-safe manner.

Phase 2: Implement the Worker Process

• Create server/worker.ts (or src/worker.ts) to initialize BullMQ Worker instances.
• Set up job handlers (e.g., a sample/test job handler).
• Add graceful shutdown hooks to the worker process to ensure active jobs are not abruptly terminated.

Phase 3: Integration & Local Orchestration

• Update ecosystem.config.js to run the worker process alongside the Nuxt app.
• Create helper/plugin in Nitro to reference and publish to the BullMQ queues.
• Implement a test endpoint or a Nitro task to trigger a test job and verify successful end-to-end processing.

Phase 4: Monitoring (Optional but recommended)

• Integrate Bull Board (a UI dashboard for BullMQ) as an admin-only route under /admin/queues or as a separate microservice to monitor queue health and retry failed jobs.

Phase 5: Verification & Tests

• Write unit tests for queue publishers.
• Write integration/E2E tests to assert that a queued job is successfully handled by the worker.