Repository navigation
feat: add dialectTypes column option for per-database column types - #12884
xue-ding-e wants to merge 17 commits into
Conversation
Column types are not portable across databases: tinyint exists on mysql
and sqlite but not on postgres, jsonb is postgres-only, and every driver
has a few types like these. A single `type` forces either a portable
lowest common denominator or hand-written migrations.
A column can now declare a type per driver:
\@column({ type: "tinyint", dialectTypes: { postgres: "smallint" } })
level: number
The override replaces the generic type when the matching driver builds
the schema; every other driver keeps using `type` (or the inferred
default). The new option is consulted in three places:
- TableUtils.createTableColumnOptions, where the metadata column becomes
DDL, so synchronize and schema builders pick it up
- EntityMetadataValidator, so a logical type unsupported by the current
driver (tinyint on postgres) no longer fails initialization when an
override covers that driver
- ColumnMetadata, which stores the option
Runtime value handling is unaffected: the column's logical type, value
transformers and query building stay exactly as before, and columns
without dialectTypes are processed identically to today.
Code Review by Qodo
1. MySQL defaults lose fractional digits
|
- resolve the effective physical type in one place: ColumnMetadata .resolveDriverType(driver) is now used by table column creation, the changed-column comparison of the postgres, mysql and sqlite drivers, and rename detection, so synchronize settles on columns whose override differs from their logical type - validate only the base name of an override (varchar in varchar(10)), since the value is written verbatim and driver lists hold unparameterized names - propagate dialectTypes to generated foreign key, junction and closure columns, so derived keys keep the physical type of the column they reference - declare the option on ColumnCommonOptions, so the type-first @column overloads accept it too - forward dialectTypes through EntitySchemaTransformer and add it to EntitySchemaColumnOptions - move the test from test/github-issues to test/functional/columns/dialect-types and add a case asserting a second synchronize produces no changes
|
Thanks for the review — pushed 4fd5664 addressing the findings:
Locally re-verified: |
Resolve physical type, length, precision and scale in one schema metadata projection. Use the existing driver normalization for validation, creation and change detection, while preserving logical value metadata and unique constraint column references. Reject blank and invalid parameter overrides, constrain driver keys, and document fallback and value-conversion requirements. Add functional coverage for type aliases, numeric modifiers, uniqueness, invalid values and an empty second synchronization query plan. Developed and validated with AI assistance.
|
Updated in 3c23f9e, with upstream master merged in 36e3c0a.
The second-sync test now asserts that both generated query lists are empty, in addition to comparing the resulting schema. This catches repeated alterations that leave the same final table definition. Type checking, compilation and lint passed. The full local suite on PostgreSQL 17.9 with PostGIS/pgvector and better-sqlite3 finished with 3091 passing, 176 pending and no failures. AI assistance was used for these changes and this follow-up. |
Clear inherited length, precision and scale when an unparameterized physical type cannot use them. Preserve compatible modifiers and leave the original logical column metadata unchanged. Cover varchar-to-text and decimal-to-double overrides, retained modifiers, and a second synchronization without DDL. Add SAP driver unit tests for plain timestamp and rejected timestamp parameters, consistent with the documented HANA column type syntax. Developed and validated with AI assistance.
|
Addressed the inherited-modifier finding in 072ba6e. An unparameterized override now keeps only compatible modifiers: For the SAP timestamp finding, the documented CREATE TABLE grammar lists Compilation, lint, formatting and commit hooks passed. The full local PostgreSQL 17.9/PostGIS/pgvector + better-sqlite3 suite finished with 3094 passing, 176 pending and no failures. AI assistance was used for the implementation, verification and this response. |
Resolve physical column metadata for typed parameters and OUTPUT table variable declarations. Include length, precision and scale while preserving explicit MssqlParameter values and logical entity metadata. Keep computed expressions out of the temporary table definition. Cover Unicode and decimal values through real INSERT, UPDATE and find operations, plus declaration and parameter unit tests. Developed and validated with AI assistance.
|
Fixed the SQL Server finding in 812ebd8. Both The regression tests exercise actual SQL Server INSERT/UPDATE OUTPUT with Unicode and decimal values, then ordinary parameterized writes and finds (including Compilation, lint, formatting and commit hooks passed. The full local suite with PostgreSQL 17.9/PostGIS/pgvector, SQL Server 2025 Express 17.0.5005.3 and better-sqlite3 finished with 3101 passing, 176 pending and no failures. AI assistance was used for implementation, verification and this response. |
Use physical column types for PostgreSQL extension installation and Oracle RETURNING out binds. Preserve UUID generation and exclusion constraint handling, and leave logical value metadata unchanged. Remove unreachable negative checks after unsigned parameter parsing while retaining overflow validation. Cover extension setup, live citext queries, Oracle bind types and rejected parameter inputs. Developed and validated with AI assistance.
|
Fixed PostgreSQL extension detection and removed the redundant negative checks in 2d99cb8. Extension setup now checks the resolved physical column types for citext, hstore, cube, spatial types, ltree and vector/halfvec. UUID generation and exclusion-constraint detection retain their existing behavior. The regression test records the extension-creation requests; a real PostgreSQL test also created a citext column from a logical varchar, verified case-insensitive lookup, and produced an empty second-sync query plan. The test database did not have citext installed before this run. The parser still rejects signed parameters and numeric overflow; only the unreachable checks after unsigned digit parsing were removed. The same audit found an Oracle RETURNING gap: OUT binds now use the physical type, with unit coverage for text-to-clob and bytea-to-blob plus unchanged fallback cases. Oracle was not tested against a live server. Final local suite with PostgreSQL, SQL Server and better-sqlite3 enabled: 3107 passing, 176 pending, no failures. Compilation, lint, formatting and commit hooks passed. AI assistance was used for implementation, verification and this response. |
Use Oracle RAW output bindings sized to the physical column. Propagate referenced type overrides to explicit join columns as well as virtual ones. Reject zero numeric precision and invalid SQL Server decimal and vector parameters. Preserve PostgreSQL scale greater than precision and zero time precision, with database and query-builder regression coverage. Remove redundant fixture comments. Developed and validated with AI assistance.
|
Addressed the new findings in 4f1727b:
The decimal check is driver-specific: PostgreSQL allows scale greater than precision, so Compilation, lint, formatting and commit hooks passed. The full local PostgreSQL/SQL Server/better-sqlite3 run had 3114 passing, 176 pending and no failures. After isolating the PostgreSQL 15+ case, the final focused run passed all 28 tests. Oracle RAW coverage uses driver/query-builder tests, not a live Oracle server. AI assistance was used for these changes, verification and this response. |
Recognize MAX only for SQL Server variable-length text and binary types and Spanner string and bytes. Keep numeric parameter validation and discard an inherited MAX length when the replacement type cannot use it. Cover metadata validation, schema declarations and long SQL Server parameter/OUTPUT values, plus Spanner emulator reads and synchronization. Document the supported types and link the functional suite to issue typeorm#12874. Developed and validated with AI assistance.
|
Fixed MAX-length overrides and added the linked issue comment in a6b87ed. The parser accepts SQL Server integration tests create the MAX columns, bind and return 9000-character ASCII text, 5000-character Unicode text and 9000-byte binary data, then read the saved values back. A new Spanner emulator test creates STRING(MAX)/BYTES(MAX), saves and reads values beyond the logical length, and checks that a second sync generates no DDL. Logical metadata remains unchanged. TypeScript compilation, lint, formatting and commit hooks passed. The full local PostgreSQL/SQL Server/better-sqlite3 run finished with 3138 passing, 176 pending and no failures. The final focused run with those databases plus the repository's Spanner emulator passed all 52 tests. AI assistance was used for implementation, verification and this response. |
Use the physical column type to select spatial write constructors, PostgreSQL casts and read expressions. Keep the original logical column metadata for value preparation and hydration. Cover geometry/geography, spatial/non-spatial replacements and unmodified columns across PostgreSQL, SQL Server and MySQL, with Aurora query-builder coverage. Scope a second-sync fixture to its own table. Developed and validated with AI assistance.
|
Addressed the spatial write finding in 38df651. INSERT and UPDATE now choose the spatial constructor and PostgreSQL cast from the resolved physical column. SELECT uses the same physical type for spatial read expressions. Value preparation, hydration and the original logical column metadata remain unchanged. The new unit cases cover 13 geometry/geography and spatial/non-spatial combinations across PostgreSQL, SQL Server, MySQL and Aurora MySQL, including legacy MySQL spatial functions. Real PostgreSQL, SQL Server and MySQL tests create the physical columns, insert, update and read compatible values, then check that the next schema sync emits no DDL. The affected dialect-types suite passed all 20 tests after its table comparison was scoped to the table it owns. TypeScript compilation, lint, formatting and commit hooks passed. A broader four-driver run had 3154 passing and 176 pending, with one failure in the prior all-tables test comparison; that test was narrowed and then passed in the focused run. The unrelated #4753 test targets fixed localhost:3306 and was excluded because that port belongs to another local service. AI assistance was used for the changes, verification and this response. |
A named logical enum mapped to PostgreSQL varchar or text retained its enum name in the physical schema projection. Introspection then reported a change on every sync and rebuilt associated constraints. Clear enum values and names only when the resolved physical type is not enum. Cover parameterized and plain overrides with a real PostgreSQL schema containing a foreign key, index and unique constraints. Verify the second schema plan is empty and logical metadata remains unchanged. Developed and validated with AI assistance.
When an unsigned numeric column is mapped to varchar, or a character column to int, the projected physical metadata retained clauses that MySQL rejects. Keep UNSIGNED only for numeric physical types and charset/collation only for character types across MySQL-family drivers. Cover incompatible parameterized and plain overrides, retained compatible modifiers, real MySQL table creation, and an empty second schema plan. Document the modifier behavior. Developed and validated with AI assistance.
DDL normalizes defaults from the resolved physical column, but INSERT fallbacks used logical metadata. A logical SQLite jsonb column overridden to text therefore wrote a JSONB blob when the value was omitted. Resolve the physical column before normalizing inlined defaults while leaving supplied value conversion unchanged. A SQLite regression failed with blob storage before the fix and now reads text JSON with an empty second schema plan. Existing SQLite JSONB tests also pass. Document the default behavior. Developed and validated with AI assistance.
A geometry column mapped to a nonspatial physical type retained feature type and SRID in its schema projection. PostgreSQL introspection omitted those fields, causing every subsequent synchronization to plan another alteration. Clear them only when the normalized physical type is not spatial. Extend the spatial integration fixture with a geometry-to-jsonb column carrying Point and SRID. The second schema plan contained one query before the fix and is empty afterward; logical metadata and spatial overrides remain intact. Developed and validated with AI assistance.
Keep schema default comparisons aligned with dialect-specific metadata. Reject incompatible generated types and temporal precision without rewriting expressions or evaluating default factories twice. Add regressions for metadata, DDL and repeated schema synchronization. AI-assisted implementation and validation.
|
Addressed the four remaining findings in this review with 5ed655a, after reproducing the failures against the preceding head. The follow-up filters incompatible ON UPDATE/collation modifiers, validates PostgreSQL generated types, and preserves explicit physical length/precision/scale during introspection. It also rejects bare MySQL CURRENT_TIMESTAMP/NOW precision conflicts without rewriting expressions or evaluating a default factory twice. DEFAULT validation is at DDL-option construction, not a guarantee that an entire synchronization has performed no earlier writes. The new unit cases went from 61 passing / 48 failing to 109 passing; 157 related unit cases passed. Both native DB regressions failed before the patch, and the complete dialect-types test file passed 9 tests with isolated PostgreSQL 17.11 and MySQL 8.4.11. Type checking passed; package ESLint completed with 0 errors and 7,751 warnings. Full-matrix and live MariaDB/Aurora validation remain unrun for this follow-up. Details and limits are in the updated PR description. AI-assisted implementation, testing and report. |
|
Additional validation is now available for product head The first MariaDB run caught an incorrect JSON introspection assertion. Commit The QA workflow is separate from this PR. This is focused fork evidence, not a full matrix or an upstream CI pass; upstream checks still require maintainer approval. The PR description records the exact scope and remaining untested drivers. AI assistance was used for these fixes and verification. |
|
Hi @xue-ding-e, thanks for this PR. I worked on fixes for a few of the open review findings. Would you like me to open a PR to your fork, or share the branch so you can pick what's useful? |
|
Thanks @noahisdai. Please open a PR against AI-assisted coordination. |
|
@xue-ding-e |
Description of change
Closes #12874.
A column's
typecurrently applies to every database. This addsdialectTypesso the same entity can use a different physical type on a particular driver:Decorators and EntitySchema accept the option. Missing driver entries fall back to
type. Relation keys inherit the referenced column's overrides, including explicitly declared join columns.Schema validation, creation and comparison use the same resolved type and modifiers. Driver normalization handles aliases, numeric length/precision parameters and supported
maxlengths; incompatible inherited modifiers, including MySQLunsigned,charsetandcollationand spatial feature type/SRID on nonspatial replacements, are removed. PostgreSQL extension detection, SQL Server parameter/OUTPUT declarations and Oracle RETURNING bindings also use the physical type. Spatial INSERT/UPDATE constructors, PostgreSQL casts, SELECT expressions and inlined omitted-value defaults follow the physical type. Value conversion, hydration and transformers retain the logical type, so overrides must be compatible with the field's values.Blank overrides and unsupported parameter syntax are rejected at initialization.
maxis accepted for SQL Server's variable-length text/binary types and Spanner's string/bytes types, and rejected for fixed-length types or precision/scale. SQL Server overrides also validate decimal precision/scale and required vector dimensions. PostgreSQL's valid scale-greater-than-precision declarations and zero fractional-second precision remain supported. Usage is documented in the entity guide and decorator reference.Earlier validation, before the 2026-10-11 follow-up: TypeScript compilation, lint, formatting and commit hooks passed. Spatial query-builder unit tests passed (13); PostgreSQL 17.9/PostGIS, SQL Server 2025 Express and MySQL 8 integration tests created the physical columns, wrote, updated and read values, and produced no second-sync DDL (3 passed). Focused dialect-types runs passed 23 tests with isolated PostgreSQL 17.9/better-sqlite3 and 22 with isolated MySQL 8/better-sqlite3. The PostgreSQL named-enum case covers unique constraints, an index and a foreign key, with an empty second schema plan (four DDL statements before the fix); two existing named-enum tests also passed. A spatial-to-nonspatial PostgreSQL override planned one repeated DDL operation before the fix and none afterward, while logical spatial metadata stayed intact. The MySQL case covers incompatible and retained
unsigned/character modifiers, actual table creation and an empty second schema plan; table creation failed before the fix. An in-memory SQLite regression showed a JSON default stored as BLOB before the fix and TEXT afterward; 12 related SQLite tests passed, including an empty second schema plan.A broader four-driver run (also including better-sqlite3) had 3154 passing and 176 pending, with one failure in that original all-tables comparison before it was narrowed and retested. The separate existing #4753 replication test, which connects to a fixed localhost:3306, was excluded because that port is owned by another local service. Aurora MySQL has query-builder unit coverage; no live Aurora, Oracle or SAP server was used.
AI assistants were used for implementation, review, testing and drafting this description.
2026-10-11: physical modifier follow-up
Commit 5ed655a addresses the four remaining findings in the review, after reproducing them against the preceding head.
onUpdatewhen the physical type is neither DATETIME nor TIMESTAMP.decimal(10,0)andvarchar(255)overrides no longer cause repeated schema changes.The DEFAULT check occurs during DDL-plan construction; it does not guarantee that an entire synchronize operation has made no earlier changes. Quoted strings and compound SQL expressions are not treated as bare timestamp functions. Logical column metadata and runtime conversion remain unchanged.
Validation of this exact commit, using isolated synthetic data on Linux:
dialect-types.test.tsfile passed 9 tests with PostgreSQL 17.11 and MySQL 8.4.11, including empty second-sync plans.tsc --noEmitpassed. Package ESLint completed with 0 errors and 7,751 warnings. Changed-file formatting andgit diff --checkpassed.The initial local run did not exercise real MariaDB/Aurora services or the full multi-database matrix. The additional fork verification below now includes MariaDB. Upstream CI is separate from these results. AI assistance was used for implementation, review, testing and this report.
Additional fork verification at de58583
The focused fork workflow passed, checking out the exact product commit
de58583be0efef3f7dba305b872fcfd490393261. The QA workflow lives on a separate branch and is not part of this PR.pnpm run format:ciand compilation with locked dependencies.Pull-Request Checklist
masterbranch