fix: enable concurrent task processing and add retry with exponential backoff

[Copilot]

Worker processes translation tasks sequentially despite prefetch > 1, and silently discards failed tasks with no retry.

Concurrent processing

The channel.consume callback was async — amqplib ignores the returned Promise, making the concurrency behavior implicit and fragile. Changed to a synchronous callback with explicit .then()/.catch():

// Before: async callback, amqplib ignores the promise
async (message) => {
  await handler(task);
  channel.ack(message);
}

// After: synchronous callback, concurrent by design
(message) => {
  handler(task)
    .then(() => channel.ack(message))
    .catch((error) => channel.nack(message, false, !message.fields.redelivered));
}

Tasks now run concurrently up to the WORKER_PREFETCH limit (default 5).

Retry on error

• runWorkerTask retries up to WORKER_MAX_RETRIES (default 3) with exponential backoff (1s, 2s, 4s)
• Error events emitted to clients only on final failure, not intermediate attempts
• Active-run lock is renewed before each retry to prevent expiry
• Consumer requeues messages once on infrastructure-level failures (Redis/AMQP down); discards on second redelivery

Cleanup

• Removed duplicate releaseActiveRun in worker.ts — already called in worker/index.ts's handleTask finally block

[vercel]

The latest updates on your projects. Learn more about Vercel for GitHub.

Project	Deployment	Actions	Updated (UTC)
universal-translation	Ready	Preview, Comment	Mar 25, 2026 3:44pm

[Copilot]

Pull request overview

This PR updates the worker’s AMQP consumption and task execution flow to better utilize RabbitMQ prefetch concurrency and to add retry behavior with exponential backoff for task failures.

Changes:

• Switches the RabbitMQ consume callback to a synchronous function and explicitly ack/nack via .then()/.catch() to make concurrency behavior explicit.
• Adds retry-with-exponential-backoff logic to runWorkerTask, emitting client-visible errors only on final failure.
• Removes duplicated active-run lock release from lib/translation/worker.ts (lock lifecycle handled in worker/index.ts).

Reviewed changes

Copilot reviewed 2 out of 2 changed files in this pull request and generated 5 comments.

File	Description
lib/translation/worker.ts	Adds retry/backoff loop around task execution and removes redundant lock release.
lib/queue/consumer.ts	Refactors the consumer callback to non-async and handles ack/nack in promise handlers for explicit concurrency.

[Copilot]

WORKER_MAX_RETRIES is parsed with Number(...). If the env var is non-numeric, this becomes NaN, causing the retry loop condition (attempt <= WORKER_MAX_RETRIES) to be false and the task to be silently skipped after logging "task_runner_started". Consider parsing as an integer and falling back to the default when the value is not finite / < 0 (and optionally clamping to a reasonable max).

Suggested change

	const WORKER_MAX_RETRIES = Number(process.env.WORKER_MAX_RETRIES ?? "3");
	function getWorkerMaxRetries(): number {
	const raw = process.env.WORKER_MAX_RETRIES;
	const parsed =
	raw !== undefined ? Number.parseInt(raw, 10) : Number.NaN;
	if (!Number.isFinite(parsed) || parsed < 0) {
	return 3;
	}
	const MAX_RETRIES = 10;
	return Math.min(parsed, MAX_RETRIES);
	}
	const WORKER_MAX_RETRIES = getWorkerMaxRetries();

[Copilot]

renewActiveRun() can return false when the active-run lock is no longer owned by this runId. Right now the retry path ignores the return value, so the worker may keep retrying (and emitting events) even after losing the lock, potentially allowing concurrent processing for the same project. Consider checking the boolean result and aborting retries (and/or emitting the same lock-failure error used elsewhere) if the lock cannot be renewed.

Suggested change

	await renewActiveRun(task.projectId, task.runId);
	const renewed = await renewActiveRun(task.projectId, task.runId);
	if (!renewed) {
	logWorker(
	"task_runner_lock_lost",
	`${taskDetails(task)} attempt=${attempt + 1}/${WORKER_MAX_RETRIES + 1}`,
	);
	throw new Error("Active run lock lost");
	}

[Copilot]

This retry loop introduces new behavior (multiple attempts with backoff and conditional final-failure reporting) but isn’t covered by unit tests. Since this file already has tests (e.g., getChatPayload), consider adding tests for runWorkerTask retry/backoff and final-failure emission to prevent regressions.

[Copilot]

parseJson() can throw on invalid JSON. With the removal of the surrounding try/catch, a malformed message will now throw out of the consume callback and won’t be nacked/acked (and may crash the consumer). Consider wrapping parseJson(message) in a try/catch here (or making parseJson return null on JSON.parse failure) and nack/discard the message on parse errors.

[Copilot]

With the new retry loop, runTranslateTask/runChatTask can be executed multiple times for the same runId. Those functions have one-time side effects (e.g., translate clears the cancel flag and clears the run event cache) that may now run again on retries; in particular, a user cancellation that happens between attempts can be wiped by clearRunCancelled inside the task runner. Consider making the retried portion idempotent (or moving one-time initialization/clears outside the retry loop / behind an isRetry guard).

[sirily11]

@copilot rror: Failed to run "next" due to signal SIGILL
76.35 Illegal instruction (core dumped)

WARNING: No output specified with docker-container driver. Build result will only remain in the build cache. To push result image into registry use --push or to load image into docker use --load
Dockerfile:13

11 | COPY --from=deps /app/node_modules ./node_modules
12 | COPY . .
13 | >>> RUN bun next build
14 |
15 | FROM base AS runner

ERROR: failed to build: failed to solve: process "/bin/sh -c bun next build" did not complete successfully: exit code: 132
Reference
Check build summary support

Build error

[Copilot]

@copilot rror: Failed to run "next" due to signal SIGILL
76.35 Illegal instruction (core dumped)

...

The Docker build failure is a Bun runtime crash (panic: Segmentation fault / SIGILL) inside oven/bun:1 — not caused by the code changes in this PR. The crash report links to bun.report confirming it's a Bun bug. The changed files (lib/queue/consumer.ts, lib/translation/worker.ts) are only imported by the worker entry point (worker/index.ts) and are not part of the Next.js build graph.

In commit 5f578a1 I've also addressed all the automated review feedback (parseJson safety, retry NaN guard, lock renewal check, cancellation before retry).