Skip to content

feat(extension-pgvector): support variable length embeddings - #30234

Open
RyanGarber wants to merge 2 commits into
prisma:mainfrom
RyanGarber:main
Open

RyanGarber wants to merge 2 commits into
prisma:mainfrom
RyanGarber:main

Conversation

@RyanGarber

@RyanGarber RyanGarber commented Sep 8, 2026

Copy link
Copy Markdown

Linked issue

n/a

Summary

The pgvector extension currently requires a fixed length for all pgvector columns, but many real-world use cases involve lengths that cannot be known. For example, many apps today allow users to supply their own API keys and models. The requirement of a length at contract time prevents that use case entirely.

This commit simply adds a second overload to the existing vector column type that cleanly maps to the non-dimensioned vector type in Postgres. All existing validation for fixed-length vectors remain in place, so existing behavior is completely unchanged. This change simply adds a new option.

Testing performed

  • pnpm typecheck && pnpm lint && pnpm test:packages (all code relevant to this commit green)
  • manual testing of the emitted contract

Skill update

  • Internal and external README updated to account for new optionality; all existing skills remain accurate

Checklist

  • All commits are signed off (git commit -s) per the DCO. The DCO status check will block merge if any commit is missing a Signed-off-by: trailer.
  • I read CONTRIBUTING.md and the change is scoped to one logical concern.
  • Tests are updated (or n/a if the change is doc-only / refactor with no behavioural delta).
  • The PR title is in TML-NNNN: <sentence-case title> form (Linear ticket prefix + concise title naming the concrete deliverable). See .claude/skills/create-pr/SKILL.md for the full convention.
  • The Skill update section above is filled in (or stated n/a — internal only).

Notes for the reviewer

This change is mission-critical for my work. Until it is released, I must use my own custom fork of this extension for migrations to work. Given it also just aligns the extension type closer to the official Postgres type and has zero breaking changes, I very much hope to see it in the coming RCs!

Summary by CodeRabbit

  • New Features

    • Added support for variable-dimension vectors without requiring an explicit length.
    • Existing fixed-dimension vectors continue to support dimension validation and typed output.
    • Variable-length vector values can be encoded, decoded, and round-tripped across different dimensions.
  • Documentation

    • Updated pgvector documentation to cover both vector() and vector(n) column definitions and PostgreSQL compatibility requirements.

@RyanGarber
RyanGarber requested a review from a team as a code owner September 8, 2026 21:03
Copilot AI lite review requested due to automatic review settings September 8, 2026 21:03
@CLAassistant

CLAassistant commented Sep 8, 2026

Copy link
Copy Markdown

CLA assistant check
All committers have signed the CLA.

@coderabbitai

coderabbitai Bot commented Sep 8, 2026

Copy link
Copy Markdown
Contributor

Review Change StackReview Change Stack

No actionable comments were generated in the recent review. 🎉

ℹ️ Recent review info
⚙️ Run configuration

Configuration used: Repository: prisma/orm/.coderabbit.yml

Review profile: CHILL

Plan: Advanced

Run ID: 04bb83a4-f252-4cdb-811e-0fbd889680ed

📥 Commits

Reviewing files that changed from the base of the PR and between ae0ead9 and 351be53.

📒 Files selected for processing (1)
  • test/integration/test/packaging/extension-tarball.test.ts

Included review availability: Your plan provides up to 10 included reviews per hour; 9 remain after this review.


📝 Walkthrough

Walkthrough

The pgvector extension now supports both undimensioned vector() and fixed-dimension vector(N) columns. Codecs, authoring types, rendering, documentation, and integration tests cover both forms.

Changes

Variable-dimension pgvector support

Layer / File(s) Summary
Authoring contracts
packages/3-extensions/pgvector/src/core/authoring.ts, packages/3-extensions/pgvector/src/exports/column-types.ts
The vector authoring type and vector factory accept an omitted dimension and produce empty type parameters.
Codec and column behavior
packages/3-extensions/pgvector/src/core/codecs.ts
Codec parameters and constructors accept undefined lengths. Variable-length vectors skip fixed-dimension validation and render as Vector. Fixed-dimension behavior remains available.
Runtime verification and published surface
packages/3-extensions/pgvector/test/*, test/integration/test/packaging/extension-tarball.test.ts, packages/3-extensions/pgvector/README.md, packages/9-public/@prisma/orm-extension-pgvector/README.md
Tests cover undimensioned authoring, rendering, codec round trips, conformance, packaging, and documentation for both vector forms.

Priority: ⬇️ Low

Estimated code review effort: 3 (Moderate) | ~25 minutes

Change: Feature

Sequence Diagram(s)

sequenceDiagram
  participant Application
  participant vector as vector()
  participant pgVectorColumn
  participant PgVectorCodec
  Application->>vector: Create an undimensioned descriptor
  vector->>pgVectorColumn: Provide empty typeParams
  pgVectorColumn->>PgVectorCodec: Create codec with undefined length
  Application->>PgVectorCodec: Encode and decode vectors of different lengths
  PgVectorCodec-->>Application: Return round-tripped vector values
Loading

Suggested reviewers: wmadden-electric

🚥 Pre-merge checks | ✅ 4 | ❌ 1

❌ Failed checks (1 warning)

Check name Status Explanation Resolution
Docstring Coverage ⚠️ Warning Docstring coverage is 0.00% which is insufficient. The required threshold is 80.00%. Docstring coverage is scoped to functions touched by this diff. Analyzed 5 functions across 8 files. Write docstrings for the functions missing them to satisfy the coverage threshold.
✅ Passed checks (4 passed)
Check name Status Explanation
Description Check ✅ Passed Check skipped - CodeRabbit’s high-level summary is enabled.
Title check ✅ Passed The title clearly and concisely describes the main change: adding support for variable-length embeddings in the pgvector extension.
Linked Issues check ✅ Passed Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check ✅ Passed Check skipped because no linked issues were found for this pull request.
  • Fix all pre-merge checks with AI
✨ Finishing Touches
🧪 Generate unit tests (beta)
  • Create a new PR

Comment @coderabbitai help to get the list of available commands.

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 Changes recommended

Variable-dimension vectors currently skip dimension bounds validation in the runtime codec, which can allow oversized/invalid payloads before Postgres rejects them.

Once you've addressed the issues Copilot identified, you can request another Copilot review.

Pull request overview

Adds variable-length pgvector column authoring to Prisma Next’s pgvector extension by introducing an overload that maps to Postgres’ undimensioned vector type, while preserving existing fixed-dimension behavior and validation.

Changes:

  • Add vector() / pgVectorColumn() authoring surfaces for undimensioned vectors (no length in typeParams), plus descriptor/runtime support for optional length.
  • Update codec typing/rendering so emitted contract.d.ts can reference Vector (undimensioned) or Vector<N> (dimensioned).
  • Expand coverage with new unit/integration tests and update READMEs for the new optionality.
File summaries
File Description
packages/9-public/@prisma/orm-extension-pgvector/test/extension-tarball.test.ts Adds an installed-package smoke test ensuring variable-dimension vectors work end-to-end.
packages/9-public/@prisma/orm-extension-pgvector/README.md Updates public entrypoint docs to mention vector() and variable dimensions.
packages/3-extensions/pgvector/test/variable-dimensions.test.ts New tests covering authoring, params validation, native type expansion, and encode/decode for variable dimensions.
packages/3-extensions/pgvector/test/codec-render-output-type.test.ts Adds coverage for renderOutputType when length is omitted (render Vector).
packages/3-extensions/pgvector/test/codec-conformance.integration.test.ts Adds integration conformance cases for {} typeParams (variable-dimension vectors).
packages/3-extensions/pgvector/src/exports/column-types.ts Introduces the vector() overload and keeps existing vector(n) validation for fixed dimensions.
packages/3-extensions/pgvector/src/core/codecs.ts Makes length optional in params schema and codec; renders Vector vs Vector<N> accordingly; adds variable-dim column helper.
packages/3-extensions/pgvector/src/core/authoring.ts Makes the PSL authoring length argument optional for pgvector.Vector.
packages/3-extensions/pgvector/README.md Updates internal docs to describe the new optional length behavior and contract typing output.
Review details

Suppressed comments (1)

packages/3-extensions/pgvector/src/core/codecs.ts:114

  • When length is omitted (variable-dimension vectors), the runtime codec no longer enforces pgvector’s dimension bounds. This makes it possible to encode/decode empty vectors or extremely large vectors (e.g. > VECTOR_MAX_DIM) that will likely be rejected by Postgres and can create very large wire payloads before failing. Consider validating value.length is within [1, VECTOR_MAX_DIM] even when this.length is undefined, while still skipping the exact-length match in that mode.
    if (this.length !== undefined && value.length !== this.length) {
      throw pgVectorError(
        code,
        `Vector length mismatch: expected ${this.length}, got ${value.length}`,
        {
  • Files reviewed: 9/9 changed files
  • Comments generated: 1
  • Review effort level: Lite

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

Comment thread packages/3-extensions/pgvector/README.md Outdated
@RyanGarber
RyanGarber force-pushed the main branch 2 times, most recently from a0a1227 to ae0ead9 Compare September 13, 2026 06:56
RyanGarber and others added 2 commits September 19, 2026 21:11
Signed-off-by: Ryan <ryanmichaelgarber@gmail.com>
…EADME

Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com>

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants