Skip to content

feat(cli): octop db migrate-to-pg — SQLite → PostgreSQL 数据迁移 - #3028

Open
labspc wants to merge 2 commits into
TencentCloud:mainfrom
labspc:feat/db-migrate-to-pg
Open

labspc wants to merge 2 commits into
TencentCloud:mainfrom
labspc:feat/db-migrate-to-pg

Conversation

@labspc

@labspc labspc commented Nov 18, 2026

Copy link
Copy Markdown

PR Note:feat(cli) octop db migrate-to-pg — SQLite → PostgreSQL 数据迁移

仓库:TencentCloud/Octop
分支:labspc:Octop feat/db-migrate-to-pg → main
提交:db99e8df test(cli): add P0 type-mapping unit tests for _pg0param and _convert_value
变更:3 个文件(新增 src/octop/cli/commands/db.py 342 行;src/octop/cli/registry.py +5 行;tests/unit/cli/test_cli_db.py +153 行)
2 个提交:b1781a4a feat(cli): add migrate-to-pg → db99e8df test(cli): add P0 unit tests

What

新增 octop db 命令组,首个子命令 migrate-to-pg:把存量 SQLite 控制面数据完整迁移到 PostgreSQL。

octop db migrate-to-pg --target-config /path/to/config.json [--source-home ~/.octop] [--dry-run] [--yes]
  • --target-config(必填):目标 PG 的 config.json 路径
  • --source-home:源 OCTOP_HOME,默认 $OCTOP_HOME 或 ~/.octop
  • --dry-run:只预检不落数据
  • --yes:跳过交互确认

Why

  • SQLite 单文件库无法支撑多实例 / 生产级部署,Octop 控制面需要向 PG 演进
  • 存量用户从 SQLite 升级到 PG 时没有官方迁移路径,手工搬库容易踩 FK 约束与类型不兼容的坑
  • 本 PR 与后续 HITL PG store(feat/hitl-postgres-store)同属控制面 PG 化方向,是该方向的基础设施建设

How

  1. 前置校验:源必须为 SQLite;目标必须已 octop init(只搬数据、不建 schema);目标 _schema_version 不得低于源版本
  2. FK 处理:目标用户是 SUPERUSER 时 SET session_replication_role = 'replica' 临时关闭 FK 检查;无 SUPERUSER 时按启发式排序(无 FK 表先迁)
  3. 类型映射:SQLite → psycopg 占位符 cast(_pg0param():REAL→%s::double precision、JSON→%s::jsonb、BLOB→%s::bytea),_convert_value() 逐单元格转换
  4. 迁移执行:逐表 DELETE FROM → 逐行 INSERT(事务包裹)→ 行数比对,不一致记 MISMATCH 并最终报错
  5. 跳过规则:_schema_version、sqlite_sequence(PG_NATIVE_TABLES)不迁移;源 0 行且目标非空的表 SKIP 保留

设计要点

  • 幂等可重跑:DELETE + INSERT 全量覆盖式,重复执行结果一致
  • 源库只读:迁移不修改源 SQLite,失败可回退(继续用 SQLite)
  • dry-run 先行:先跑一遍校验与行数预估,确认无误再正式迁移

Test

P0:类型映射单元测试 ✅

37 项新增测试已合入 tests/unit/cli/test_cli_db.py(commit db99e8df),覆盖:

  • _pg0param():22 组参数化用例(REAL/FLOAT/DOUBLE→%s::double precision、INTEGER/INT/BIGINT→%s::bigint、BLOB→%s::bytea、JSON→%s::jsonb、TEXT/VARCHAR/BOOLEAN/TIMESTAMP/空串→%s fallback),含大小写不敏感
  • _convert_value():NULL 透传(7 种类型)、浮点/整数类型派发(含跨类型输入 int→float)、BLOB→bytes 往返、JSON 五种场景(有效字符串往返、dict/list 序列化、空字符串→[]、空白字符串直接透传、非法 JSON 字符串→JSONDecodeError)、未知类型 fallback(按 Python 值类型推断)

全量 test_cli_db.py:43 passed(6 项原有测试零回归)

手工验证路径

  1. 准备含真实数据的 SQLite OCTOP_HOME
  2. 目标 PG 执行 octop init 建好 schema
  3. --dry-run 预检 → 正式迁移 → 逐表行数比对一致

P1:PG 集成冒烟测试(不阻塞本 PR 合入,后续补)

  • 需 Docker 或本地 PG 实例配合 OCTOP_TEST_DATABASE_URL
  • 建议覆盖:octop db migrate-to-pg 完整链路(含 FK 依赖表排序、类型映射端到端、行数校验 MISMATCH 告警)
  • 类型映射逻辑已被 P0 单元测试(43 项)充分覆盖,本 PR 可独立合入

Limitations

  • 全量覆盖式迁移,非增量;迁移期间建议停写源库
  • 无 SUPERUSER 时启发式排序不保证覆盖所有 FK 拓扑(如循环依赖场景)
  • 目标必须先 init,本命令不负责建 schema

Related

  • 配套基础设施:Octop feat/hitl-postgres-store(HITL pending 记录 PG 持久化,依赖 PG 控制面)

…L migration

Introduces a new CLI command group octop db with a migrate-to-pg
subcommand that copies all user-data tables from a SQLite source to
a PostgreSQL target in a single bulk operation.

Features:
- Automatic type conversion (REAL→double precision, BLOB→bytea,
  INTEGER→bigint, JSON→jsonb)
- Dual foreign-key strategy: disable session_replication_role when
  SUPERUSER is available, otherwise order tables by FK dependency
  (parent tables first) so the copy works without elevated privileges
- Per-table COUNT(*) verification with mismatch reporting
- --dry-run mode to preview row counts without writing
- Idempotent: target tables are DELETEd before INSERT, so the command
  can be re-run safely
- Lazy-loaded via the existing CLI registry (no import cost at launch)
…value

37 new tests covering:
- _pg0param(): 22 parametrized cases (REAL/FLOAT/DOUBLE/INTEGER/INT/BIGINT/BLOB/JSON + fallback)
- _convert_value(): NULL passthrough, numeric type dispatch (float/int), BLOB→bytes, JSON roundtrip/dict/list/empty-string, fallback inference

All 43 tests in test_cli_db.py pass (zero regression on 6 existing tests).
Copilot AI balanced review requested due to automatic review settings November 18, 2026 07:20

Copilot AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Warning

Copilot couldn't run its full agentic review because it didn't start before the timeout. Make sure your repository has a runner available, or add a copilot-code-review.yml file specifying one with the runs-on attribute. See the docs for more details.

Copilot review overview

19 open issues
What changed in this PR

Adds a new octop db CLI command group with a migrate-to-pg subcommand to migrate existing Octop control-plane data from SQLite to PostgreSQL, including type conversion and basic safety checks.

Changes:

  • Add octop db migrate-to-pg command implementing SQLite → PostgreSQL full-copy migration.
  • Register the new db CLI command group.
  • Add P0 unit tests for SQLite→PG type mapping helpers (_pg0param, _convert_value).
File Description
src/​octop/​cli/​commands/​db.py Implements the migration workflow, type mapping, and CLI options for migrate-to-pg.
src/​octop/​cli/​registry.py Registers the new db command group in the CLI registry.
tests/​unit/​cli/​test_cli_db.py Adds unit tests covering type-cast placeholder mapping and per-cell conversion behavior.

Review effort: Lite

def _source_row_count(sqlite_path: Path, table: str) -> int:
conn = sqlite3.connect(f"file:{sqlite_path}?mode=ro", uri=True)
try:
return conn.execute(f"SELECT COUNT(*) FROM {table}").fetchone()[0]
"""Return [(name, sqlite_type), ...] for a table."""
conn = sqlite3.connect(f"file:{sqlite_path}?mode=ro", uri=True)
try:
return [(r[1], (r[2] or "").upper()) for r in conn.execute(f"PRAGMA table_info({table})")]
Comment on lines +77 to +78
if t in ("JSON",):
return "%s::jsonb"
if isinstance(val, str) and val.strip():
return json.dumps(json.loads(val))
if isinstance(val, (dict, list)):
return json.dumps(val)
finally:
src_db.close()

tgt_conn = pg_connect(tgt_db.postgresql_conninfo(), autocommit=True, connect_timeout=10)
is_flag=True,
help="Skip the confirmation prompt.",
)
def migrate_to_pg(
Comment on lines +214 to +215
click.echo(f"Source : {sqlite_path}")
click.echo(f"Target : {tgt_db.postgresql_conninfo()}")
Comment on lines +301 to +312
sqlite_rows = conn.execute(f"SELECT * FROM {tbl}").fetchall()
finally:
conn.close()

with tgt_conn.transaction():
for row in sqlite_rows:
values = [
_convert_value(row[ci], col_types[ci])
for ci in range(len(row))
]
tgt_conn.execute(insert_sql, values)

if not sqlite_path.is_file():
raise click.UsageError(f"SQLite database not found: {sqlite_path}")

tgt_dir, tgt_db = _resolve_tgt_config(target_config)
Comment on lines +252 to +260
# Heuristic: tables whose name appears as a prefix in other FK
# constraint names are likely parent tables.
fk_tables: set[str] = set()
try:
for r in tgt_conn.execute(
"SELECT table_name FROM information_schema.table_constraints "
"WHERE constraint_type='FOREIGN KEY' AND table_schema='public'"
):
fk_tables.add(r[0])

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants