Skip to content

TSA-126: Worker concurrency so the buffer does not drain - #303

Merged
arturobdev merged 12 commits into
mainfrom
feat/TSA-126-worker-concurrency
Sep 22, 2026
Merged

TSA-126: Worker concurrency so the buffer does not drain#303
arturobdev merged 12 commits into
mainfrom
feat/TSA-126-worker-concurrency

Conversation

@miraelno

@miraelno miraelno commented Sep 16, 2026

Copy link
Copy Markdown
Collaborator

Closed #126

  • Added configurable worker concurrency for page.transcribe (PAGE_TRANSCRIBE_CONCURRENCY, default 3), passed to the BullMQ worker through BaseQueue.
  • Provider rate limits (429) are now recognised and no longer fail pages: the page returns to queued without spending an attempt, and the job is retried after retry-after or 30 seconds.
  • Tokens already spent before the rate limit are still charged to spent_usd atomically, and each wait is recorded as a transcribe_rate_limited page event.

@miraelno miraelno added this to the transcripta-release-3 milestone Sep 16, 2026
@miraelno miraelno self-assigned this Sep 16, 2026
@miraelno miraelno added backend Backend application feature New feature (auto-applied by the feature template) labels Sep 16, 2026
@miraelno miraelno changed the title TSA-126: + ratelimit error handling, concurrency config TSA-126: Worker concurrency so the buffer does not drain Sep 16, 2026
arturobdev
arturobdev reviewed Sep 16, 2026
View reviewed changes
Comment thread apps/backend/src/modules/jobs/transcribe.ts
@MatiStb
MatiStb self-requested a review September 18, 2026 19:23
@MatiStb

MatiStb commented Sep 18, 2026

Copy link
Copy Markdown
Collaborator

Hi! @miraelno ! You have to resolve some conflicts before we can review this PR. Please take a look. Thanks!

arturobdev
arturobdev previously approved these changes Sep 21, 2026
View reviewed changes
MatiStb
MatiStb previously approved these changes Sep 21, 2026
View reviewed changes
@MatiStb

MatiStb commented Sep 21, 2026

Copy link
Copy Markdown
Collaborator

hi @Yehor-Makarenko ! Just left one conflict to resolve, and then you will be able to push.

arturobdev
arturobdev previously approved these changes Sep 21, 2026
View reviewed changes
@arturobdev
arturobdev merged commit f257ee4 into main Sep 22, 2026
2 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@Yehor-Makarenko Yehor-Makarenko Yehor-Makarenko left review comments
@arturobdev arturobdev arturobdev approved these changes
@RomanNabukhotniidev RomanNabukhotniidev RomanNabukhotniidev approved these changes
@MatiStb MatiStb MatiStb left review comments
@Dekshad Dekshad Awaiting requested review from Dekshad

Assignees

@miraelno miraelno

Labels

backend Backend application feature New feature (auto-applied by the feature template)

Projects

bsa-2026-transcripta
Status: No status

Milestone

transcripta-release-3

Development

Successfully merging this pull request may close these issues.

[BE] Worker concurrency so the buffer does not drain

5 participants

@miraelno @MatiStb @arturobdev @Yehor-Makarenko @RomanNabukhotniidev