Switch to using goose for migrations

[iplay88keys]

Summary

• Replace golang-migrate with Goose for PostgreSQL migrations.
• Combine the existing core migrations into one version 1 baseline.
• Combine the existing vector migrations into one version 1 baseline.
• Keep separate migration state and locks for each migration source.
• Keep CLI down migrations within the Goose sequence.
• Require a fresh PostgreSQL database for the cutover.

Why this change

Kagent currently uses golang-migrate to apply PostgreSQL schema changes.
The runner records a dirty version before it applies a migration.
It clears that state after the migration transaction commits.

A failure between those operations can commit the schema while the migration table remains dirty.
The runner cannot use the dirty state to prove whether PostgreSQL committed the schema.
An operator must inspect the database and force a version before startup can continue.

Substrate PR #1196 moved Substrate to Goose for the same reason.
This review thread describes the commit and tracking failure.

Goose runs each normal SQL migration and its migration record in one PostgreSQL transaction.
PostgreSQL commits both changes or rolls back both changes.
This behavior removes the uncertain dirty state.

Benefits compared with golang-migrate

Area	golang-migrate behavior	Goose behavior in this PR
Migration transaction	Commits schema changes before the final clean version update	Commits schema changes and the migration record together
Failure state	Can leave a dirty version after schema commit	Leaves the previous version when the transaction fails
Recovery	Requires dirty state logic and version forcing	Retries the pending migration after restart
Migration files	Uses separate Up and Down files	Keeps both sections in one file
Runner behavior	Uses custom rollback and dirty recovery code	Uses the Goose provider and PostgreSQL transaction behavior
Operator commands	Exposes force for dirty state recovery	Removes force because Goose has no dirty state

This change keeps the existing PostgreSQL advisory locks.
Each migration source still has an independent lock and migration table.

Fresh database contract

This cutover supports only a fresh installation with a new PostgreSQL database.
It does not migrate PostgreSQL data from a release that uses golang-migrate.

The core source keeps the schema_migrations table name.
The vector source keeps the vector_schema_migrations table name.
The tables use the standard Goose column structure after the cutover.

The runner rejects a table that contains the old dirty column.
The error tells the operator to use a new PostgreSQL database.

This PR does not add a migration bridge.
It does not support a product downgrade across the cutover.
Upgrade tests skip releases that do not use Goose.

Later Goose releases can upgrade the database in place.
They will append migration files to the same source sequences.

Separate migration sources

The core and vector sources remain independent.
The core source always runs first.
The vector source runs second when database vectors are active.
Downstream sources run after the built-in sources in registration order.

Each source has these items:

• An embedded migration directory.
• A migration table.
• A PostgreSQL schema selection.
• An optional precheck.
• A PostgreSQL advisory lock.

The runner validates every source and migration file first.
It then runs all source prechecks before it applies a migration.

The runner applies sources in registration order.
It does not use one transaction across all sources.

If a later source fails, earlier sources remain committed.
The next startup skips completed sources and retries the failed source.

Migration files

The core source now contains core/000001_initial.sql.
The vector source now contains vector/000001_initial.sql.

Each file contains a -- +goose Up section and a -- +goose Down section.
Goose requires the Up section and makes the Down section optional.
Kagent requires both sections because the database CLI supports down migrations.

The baseline Up section combines the previous Up files in version order.
The baseline Down section combines the previous Down files in reverse order.

The runner and CI reject -- +goose NO TRANSACTION.
This rule preserves the atomic schema and migration record guarantee.

CI also protects merged Goose migration files from changes, renames, and deletions.
A later correction must use a new migration file.

Failure behavior

• An invalid source or migration file stops the run before migrations start.
• A failed precheck stops the run before migrations start.
• A SQL error rolls back the current migration and its migration record.
• A lost commit response cannot separate the schema from its migration record.
• A later source failure does not reverse a completed source.
• Concurrent controllers serialize each source with its advisory lock.
• A database version above the embedded version stops controller startup.
• An old migration table stops the source with the fresh database error.

A successful migration can still contain an application defect.
Fix an accepted migration with a new forward migration.
Do not change a migration file after it merges.

The CLI can run a controlled down migration when a forward fix is not suitable.
Operators must stop controllers before a manual down migration.
Otherwise, a controller can apply the migration again during startup.

CLI changes

The database migration CLI keeps these commands:

• up applies pending migrations for all registered sources.
• down N reverses migrations for one selected source.
• goto V moves one selected source to a migration version.
• status reports applied and pending Goose migrations.
• version reports the current version for each selected source.

The CLI removes the force command.
Status output also removes the dirty field.

Down migrations can delete data.
They support database operations within the Goose sequence.
They do not provide a product downgrade across the cutover.

Developer documentation

The Kagent development guide now defines the Goose migration rules.
It documents file names, required sections, transactions, immutability, source ownership, and PostgreSQL tests.

Testing

• go test ./core/pkg/migrations ./core/pkg/cli/db/migrate ./core/test/upgrade ./core/internal/database
• go vet ./core/pkg/migrations ./core/pkg/cli/db/migrate ./core/test/upgrade ./core/internal/database
• make lint
• make sqlc-generate
• git diff --check

The migration tests cover these cases:

• Fresh core and vector installation.
• Complete baseline Down and Up behavior.
• Concurrent controller startup.
• Atomic rollback after a SQL failure.
• Restart after a later source fails.
• Rejection of the old migration table format.
• CLI Up, Down, status, version, and goto behavior.