AI Refactor: Switch Database Migration Pipeline to Use Knex - #99
Draft
klondikemarlen wants to merge 11 commits into
Draft
klondikemarlen wants to merge 11 commits into
klondikemarlen wants to merge 11 commits into
Conversation
…Knex This investigation analyzes the feasibility and approach for migrating the database migration system from Umzug to Knex, based on patterns used in other icefoganalytics projects (vendor-portal, travel-authorization, digital-vault). Key findings: - Knex provides better schema builder API and native migration support - Recommends soft migration approach (keep Umzug, add Knex for new migrations) - Includes step-by-step migration strategy and implementation checklist - Documents code examples from other icefoganalytics repositories
…ach) This commit implements Knex.js v3.1.0 as a migration system that runs alongside the existing Umzug setup, following patterns from vendor-portal and travel-authorization repositories. Changes: - Add knex@^3.1.0 dependency to package.json - Create db-migration-client.ts with MSSQL configuration - Create knexfile.ts for CLI configuration - Add Knex migration and seed templates - Add initializers for running Knex migrations (21-) and seeds (31-) - Update package.json with Knex CLI scripts - Create comprehensive KNEX_MIGRATION_GUIDE.md documentation - Update KNEX_MIGRATION_INVESTIGATION.md with implementation status Implementation approach: - Soft migration: Both Umzug and Knex coexist - Umzug runs first (20-run-migrations.ts) - Knex runs second (21-run-knex-migrations.ts) - New migrations should use Knex going forward - Existing Umzug migrations remain functional New npm scripts: - npm run knex:migrate - Run pending migrations - npm run knex:migrate:make <name> - Create new migration - npm run knex:migrate:rollback - Rollback last batch - npm run knex:migrate:list - List migration status - npm run knex:seed:make <name> - Create seed file - npm run knex:seed:run - Run seed files Files created: - api/src/db/db-migration-client.ts - api/src/config.d/knexfile.ts - api/src/db/templates/sample-migration-knex.ts - api/src/db/templates/sample-seed-knex.ts - api/src/initializers/21-run-knex-migrations.ts - api/src/initializers/31-run-knex-seeds.ts - api/KNEX_MIGRATION_GUIDE.md
This commit removes all Umzug-related code and files, making Knex the sole migration system for the project. Removed files: - bin/migrate.ts - Old Umzug CLI - bin/seed.ts - Old Umzug seed CLI - src/db/umzug.ts - Umzug configuration - src/db/umzug-null-storage.ts - Custom Umzug storage - src/initializers/21-run-knex-migrations.ts - Renamed to 20- - src/initializers/31-run-knex-seeds.ts - Renamed to 30- - src/db/templates/sample-migration-knex.ts - Renamed - src/db/templates/sample-seed-knex.ts - Renamed Changed files: - package.json: Remove umzug dependency, simplify scripts - src/initializers/20-run-migrations.ts: Now runs Knex migrations - src/initializers/30-run-seeds.ts: Now runs Knex seeds - src/db/db-migration-client.ts: Update template paths - src/db/templates/sample-migration.ts: Now Knex template - src/db/templates/sample-seed.ts: Now Knex template Migration scripts now aliased directly (no "knex:" prefix): - npm run migrate - Run pending migrations - npm run migrate:make <name> - Create new migration - npm run migrate:rollback - Rollback last batch - npm run migrate:list - List migration status - npm run seed:make <name> - Create seed file - npm run seed:run - Run seed files
This commit adds 12 Knex migrations that recreate the complete database schema from the DDL. Migrations are ordered to respect foreign key dependencies. Base tables (no foreign keys): - 20241117000001_create-users-table - 20241117000002_create-centres-table - 20241117000003_create-fiscal-periods-table - 20241117000004_create-funding-periods-table - 20241117000005_create-funding-submission-lines-table - 20241117000006_create-logs-table Tables with foreign keys: - 20241117000007_create-employee-wage-tiers-table (FK to fiscal_periods) - 20241117000008_create-employee-benefits-table (FK to centres, fiscal_periods) - 20241117000009_create-funding-submission-line-jsons-table (FK to centres) - 20241117000010_create-payments-table (FK to centres, fiscal_periods) - 20241117000011_create-user-roles-table (FK to users with CASCADE delete) - 20241117000012_create-wage-enhancements-table (FK to centres, employee_wage_tiers) All migrations include: - Proper MSSQL DATETIME2(0) types for timestamps - GETDATE() defaults for created_at/updated_at - Appropriate unique constraints and indexes - Complete up() and down() functions for rollback capability
This commit updates all documentation to remove references to the soft migration approach and Umzug, reflecting that Knex is now the sole migration system. Changes to api/KNEX_MIGRATION_GUIDE.md: - Updated title and intro to remove "alongside Umzug" references - Changed all command examples from knex:* to simplified aliases (e.g., npm run migrate instead of npm run knex:migrate) - Updated Migration Strategy section to list the 12 base migrations - Removed references to Umzug and soft migration approach - Updated initializer references (21 -> 20, 31 -> 30) - Fixed all code examples and troubleshooting commands Changes to KNEX_MIGRATION_INVESTIGATION.md: - Updated status to "FULLY IMPLEMENTED" - Added completed actions checklist - Added note explaining this is historical reference - Clarified that full replacement was used instead of soft migration The documentation now accurately reflects the current Knex-only setup.
Following the wrap project pattern, migrations are now run manually rather than automatically on server start. This gives explicit control over when schema changes are applied. Changes: - Removed src/initializers/20-run-migrations.ts (no auto-migration) - Removed src/initializers/30-run-seeds.ts (no auto-seeding) - Removed src/config.d/knexfile.ts (nested config) - Created knexfile.ts at api root (simpler location) - Updated package.json knex script to use new knexfile location - Updated KNEX_MIGRATION_GUIDE.md to reflect manual migration approach Benefits: - Simpler, more explicit control over migrations - Matches proven pattern from wrap project - Migrations must be run manually: npm run migrate - Seeds must be run manually: npm run seed:run Developers must now run migrations explicitly before starting the server: cd api npm run migrate npm start
Moved knexfile from api root to api/src/config.d/knexfile.ts to match the wrap project's structure exactly. This is the correct location used by wrap and keeps configuration files organized in the src/config.d/ directory.
Documents the complete switch from Umzug to Knex for future reference and Jira ticket creation. Includes background, implementation details, acceptance criteria, and benefits.
Provides a ready-to-use prompt for creating the Jira ticket through another AI session that has the Atlassian connector properly configured.
klondikemarlen
added
the
♻️ refactor
Re-add automatic migration and seed running on server startup, but now using Knex db-migration-client instead of Umzug. Changes: - Added src/initializers/20-run-migrations.ts (runs Knex migrations) - Added src/initializers/30-run-seeds.ts (runs Knex seeds) - Updated seed template to use Sequelize Models instead of raw Knex Key points: - Migrations run automatically on server start via Knex - Seeds run automatically in dev/test via Knex - Seed files use Sequelize Models for type safety and consistency with application code (not raw Knex queries) - This maintains the automatic migration pattern while using Knex
klondikemarlen
changed the title
Implement all 16 tables from the updated DDL schema with proper foreign key dependencies and unique indexes: Base tables (no dependencies): - funding_periods (fiscal_year, dates, title) - funding_regions (region names) - funding_submission_lines (section/line data with age ranges) - logs (category, description, data) - users (email, status, org structure fields) Tables with foreign keys: - building_expense_categories (depends on funding_regions) - centres (depends on funding_regions; includes contact info, addresses) - fiscal_periods (depends on funding_periods; monthly periods) - funding_reconciliations (depends on centres, funding_periods, users) - funding_submission_line_jsons (depends on centres) - payments (depends on centres, fiscal_periods; estimate/actual) - building_expenses (depends on building_expense_categories, centres, fiscal_periods) - employee_benefits (depends on centres, fiscal_periods; CPP, EI, etc.) - employee_wage_tiers (depends on fiscal_periods) - funding_reconciliation_adjustments (depends on funding_reconciliations, fiscal_periods) - wage_enhancements (depends on centres, employee_wage_tiers) All migrations include: - MSSQL-specific DATETIME2(0) and DATETIMEOFFSET types - created_at/updated_at with GETUTCDATE() defaults - deleted_at for soft deletes - Unique indexes with WHERE deleted_at IS NULL predicates - Proper up/down functions for rollback capability Removed all old Umzug and initial Knex migrations.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Fixes TODO
Relates to:
Context
TODO
Implementation
TODO
Screenshots
TODO
Testing Instructions
dev test(ordev test_api)dev up