Repository navigation
Add llm-inference job type with Gemini-backed semantic caching - #1
Merged
Merged
Conversation
pritamahanta
commented
Sep 4, 2026
pritamahanta
commented
Owner
- New job type routes through SemanticCacheService: embed prompt, check pgvector cache, call Gemini on miss, cache the result
- JobEntity gets a generic result column so job types can return structured output (not just llm-inference)
- executeWithTimeout made generic to carry that result through
- Fixed broken lint script (no eslint.config.js existed; switched to oxlint, which .oxlintrc.json was already configured for but nothing invoked)
- Added CI workflow: lint, build, test on push/PR
- New job type routes through SemanticCacheService: embed prompt, check pgvector cache, call Gemini on miss, cache the result - JobEntity gets a generic result column so job types can return structured output (not just llm-inference) - executeWithTimeout made generic to carry that result through - Fixed broken lint script (no eslint.config.js existed; switched to oxlint, which .oxlintrc.json was already configured for but nothing invoked) - Added CI workflow: lint, build, test on push/PR
There was a problem hiding this comment.
🟡 Changes recommended
It introduces correctness issues in the new rate limiter and cache maintenance logic (and includes an extra patch artifact) that should be fixed before merge.
Once you've addressed the issues Copilot identified, you can request another Copilot review.
Pull request overview
This PR introduces a new llm-inference job type backed by a Gemini-based provider and a semantic cache (pgvector) to reuse similar prompt completions, while also extending job persistence/API responses to store structured job results and adding CI + lint tooling.
Changes:
- Add
llm-inferencejob processing viaSemanticCacheService(embed → pgvector similarity lookup → Gemini completion on miss → cache write). - Extend
JobEntity/JobRepository/ API DTO mapping to persist and return a genericresultpayload for completed jobs. - Replace the lint script with
oxlintand add a CI workflow to lint/build/test on push/PR.
File summaries
| File | Description |
|---|---|
| src/worker/worker.module.ts | Wires LlmModule into the worker so jobs can call semantic caching/LLM. |
| src/worker/processors/job.processor.ts | Adds llm-inference processing and makes timeout wrapper generic to carry a job result through completion. |
| src/worker/processors/job.processor.spec.ts | Adds unit tests asserting llm-inference result persistence and prompt validation behavior. |
| src/llm/llm.module.ts | Introduces an LLM module that binds embedding + completion providers and exports semantic cache service. |
| src/llm/semantic-cache.service.ts | Implements semantic cache flow using embeddings + pgvector similarity search + LLM fallback. |
| src/llm/semantic-cache.service.spec.ts | Tests cache hit/miss behavior and error propagation for embedding failures. |
| src/llm/providers/gemini.provider.ts | Adds Gemini-backed embedding + completion provider with in-process rate limiting. |
| src/llm/providers/gemini.provider.spec.ts | Unit tests for Gemini provider request/response handling and error cases. |
| src/llm/interfaces/embedding-provider.interface.ts | Defines embedding provider contract + DI token. |
| src/llm/interfaces/llm-provider.interface.ts | Defines completion provider contract + DI token. |
| src/llm/rate-limiter.ts | Adds a minimal token-bucket rate limiter for provider calls. |
| src/llm/rate-limiter.spec.ts | Adds unit tests for rate limiter behavior. |
| src/database/repositories/llm-cache.repository.ts | Adds raw SQL repository for pgvector cache reads/writes/purge. |
| src/database/repositories/llm-cache.repository.spec.ts | Unit tests for cache repository query parameterization and inserts. |
| src/database/repositories/job.repository.ts | Extends markAsCompleted to optionally persist a result JSON payload. |
| src/database/entities/job.entity.ts | Adds result JSONB column to store structured job output. |
| src/api/jobs.service.ts | Includes result in API job response mapping. |
| src/api/dto/job-response.dto.ts | Exposes result in the job response DTO. |
| infra/init-db/01-pgvector.sql | Enables the pgvector extension in Postgres init scripts. |
| infra/init-db/02-llm-cache-table.sql | Creates llm_cache_entries table + ANN index for semantic cache. |
| infra/docker-compose.yml | Switches Postgres image to pgvector-enabled and mounts init-db scripts; adds LLM/cache env vars. |
| docker-compose.dev.yml | Switches dev Postgres image to pgvector-enabled and mounts init-db scripts. |
| .github/workflows/ci.yml | Adds CI workflow to lint/build/test on pushes and PRs. |
| .oxlintrc.json | Adds oxlint configuration used by the new lint script. |
| package.json | Switches lint to oxlint and adds oxlint dev dependency. |
| pnpm-lock.yaml | Locks oxlint dependency. |
| .env.example | Documents required environment variables for Gemini + cache tuning. |
| semantic-cache-feature.patch | Adds a patch artifact duplicating the PR’s changes. |
Review details
Files not reviewed (1)
- pnpm-lock.yaml: Generated file
- Files reviewed: 26/28 changed files
- Comments generated: 6
- Review effort level: Lite
💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.
Comment on lines
+38
to
+55
| async acquire(): Promise<void> { | ||
| this.refill(); | ||
|
|
||
| if (this.tokens > 0) { | ||
| this.tokens -= 1; | ||
| return; | ||
| } | ||
|
|
||
| const waitMs = | ||
| this.refillIntervalMs - (Date.now() - this.lastRefillAt); | ||
|
|
||
| await new Promise((resolve) => | ||
| setTimeout(resolve, Math.max(waitMs, 0)), | ||
| ); | ||
|
|
||
| this.refill(); | ||
| this.tokens = Math.max(this.tokens - 1, 0); | ||
| } |
Comment on lines
+91
to
+97
| async purgeExpired(): Promise<number> { | ||
| const result = await this.dataSource.query( | ||
| `DELETE FROM llm_cache_entries WHERE expires_at <= now()`, | ||
| ); | ||
|
|
||
| return result[1] ?? 0; | ||
| } |
Comment on lines
+35
to
+42
| this.similarityThreshold = parseFloat( | ||
| this.configService.get('CACHE_SIMILARITY_THRESHOLD', '0.92'), | ||
| ); | ||
|
|
||
| this.ttlMs = parseInt( | ||
| this.configService.get('CACHE_TTL_MS', '86400000'), | ||
| 10, | ||
| ); |
Comment on lines
+122
to
126
| private async executeWithTimeout<T>( | ||
| jobId: string, | ||
| work: Promise<void>, | ||
| ): Promise<void> { | ||
| work: Promise<T>, | ||
| ): Promise<T> { | ||
| let timeoutId: NodeJS.Timeout | undefined; |
Comment on lines
+1
to
+5
| diff --git a/.env.example b/.env.example | ||
| new file mode 100644 | ||
| index 0000000..d8656a4 | ||
| --- /dev/null | ||
| +++ b/.env.example |
Comment on lines
+19
to
+20
| // All three resolved without needing to advance fake timers. | ||
| expect(true).toBe(true); |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.