Repository navigation
Conversation
…L migration Introduces a new CLI command group octop db with a migrate-to-pg subcommand that copies all user-data tables from a SQLite source to a PostgreSQL target in a single bulk operation. Features: - Automatic type conversion (REAL→double precision, BLOB→bytea, INTEGER→bigint, JSON→jsonb) - Dual foreign-key strategy: disable session_replication_role when SUPERUSER is available, otherwise order tables by FK dependency (parent tables first) so the copy works without elevated privileges - Per-table COUNT(*) verification with mismatch reporting - --dry-run mode to preview row counts without writing - Idempotent: target tables are DELETEd before INSERT, so the command can be re-run safely - Lazy-loaded via the existing CLI registry (no import cost at launch)
…value 37 new tests covering: - _pg0param(): 22 parametrized cases (REAL/FLOAT/DOUBLE/INTEGER/INT/BIGINT/BLOB/JSON + fallback) - _convert_value(): NULL passthrough, numeric type dispatch (float/int), BLOB→bytes, JSON roundtrip/dict/list/empty-string, fallback inference All 43 tests in test_cli_db.py pass (zero regression on 6 existing tests).
There was a problem hiding this comment.
Warning
Copilot couldn't run its full agentic review because it didn't start before the timeout. Make sure your repository has a runner available, or add a copilot-code-review.yml file specifying one with the runs-on attribute. See the docs for more details.
Copilot review overview
19 open issues
Table/column identifiers are interpolated directly into SQL. Even if names are sourced from… · New Table/column identifiers are interpolated directly into SQL. Even if names are sourced from… · New For JSON columns you always cast tojsonb, but_convert_value()can return whitespace-only… · New For JSON columns you always cast tojsonb, but_convert_value()can return whitespace-only… · New Withautocommit=True,DELETE FROM {tbl}is committed immediately and is not protected by the… · New Withautocommit=True,DELETE FROM {tbl}is committed immediately and is not protected by the… · New Table/column identifiers are interpolated directly into SQL. Even if names are sourced from… · New Table/column identifiers are interpolated directly into SQL. Even if names are sourced from… · New Table/column identifiers are interpolated directly into SQL. Even if names are sourced from… · New Table/column identifiers are interpolated directly into SQL. Even if names are sourced from… · New Withautocommit=True,DELETE FROM {tbl}is committed immediately and is not protected by the… · New Withautocommit=True,DELETE FROM {tbl}is committed immediately and is not protected by the… · New Table/column identifiers are interpolated directly into SQL. Even if names are sourced from… · New The newmigrate-to-pgcommand adds significant behavior (schema checks, dry-run gating, FK… · New The newmigrate-to-pgcommand adds significant behavior (schema checks, dry-run gating, FK… · New Printingpostgresql_conninfo()may leak credentials (e.g., password) into terminal output and… · Newfetchall()loads the entire table into memory, and executing one INSERT per row will be very slow… · Newtgt_diris assigned but not used anywhere in this file. Remove it (return onlyDatabaseConfig)… · New The comment describes a heuristic based on constraint-name prefixes, but the code actually collects… · New
What changed in this PR
Adds a new octop db CLI command group with a migrate-to-pg subcommand to migrate existing Octop control-plane data from SQLite to PostgreSQL, including type conversion and basic safety checks.
Changes:
- Add
octop db migrate-to-pgcommand implementing SQLite → PostgreSQL full-copy migration. - Register the new
dbCLI command group. - Add P0 unit tests for SQLite→PG type mapping helpers (
_pg0param,_convert_value).
| File | Description |
|---|---|
src/octop/cli/commands/db.py |
Implements the migration workflow, type mapping, and CLI options for migrate-to-pg. |
src/octop/cli/registry.py |
Registers the new db command group in the CLI registry. |
tests/unit/cli/test_cli_db.py |
Adds unit tests covering type-cast placeholder mapping and per-cell conversion behavior. |
Review effort: Lite
| def _source_row_count(sqlite_path: Path, table: str) -> int: | ||
| conn = sqlite3.connect(f"file:{sqlite_path}?mode=ro", uri=True) | ||
| try: | ||
| return conn.execute(f"SELECT COUNT(*) FROM {table}").fetchone()[0] |
| """Return [(name, sqlite_type), ...] for a table.""" | ||
| conn = sqlite3.connect(f"file:{sqlite_path}?mode=ro", uri=True) | ||
| try: | ||
| return [(r[1], (r[2] or "").upper()) for r in conn.execute(f"PRAGMA table_info({table})")] |
Comment on lines
+77
to
+78
| if t in ("JSON",): | ||
| return "%s::jsonb" |
| if isinstance(val, str) and val.strip(): | ||
| return json.dumps(json.loads(val)) | ||
| if isinstance(val, (dict, list)): | ||
| return json.dumps(val) |
| finally: | ||
| src_db.close() | ||
|
|
||
| tgt_conn = pg_connect(tgt_db.postgresql_conninfo(), autocommit=True, connect_timeout=10) |
| is_flag=True, | ||
| help="Skip the confirmation prompt.", | ||
| ) | ||
| def migrate_to_pg( |
Comment on lines
+214
to
+215
| click.echo(f"Source : {sqlite_path}") | ||
| click.echo(f"Target : {tgt_db.postgresql_conninfo()}") |
Comment on lines
+301
to
+312
| sqlite_rows = conn.execute(f"SELECT * FROM {tbl}").fetchall() | ||
| finally: | ||
| conn.close() | ||
|
|
||
| with tgt_conn.transaction(): | ||
| for row in sqlite_rows: | ||
| values = [ | ||
| _convert_value(row[ci], col_types[ci]) | ||
| for ci in range(len(row)) | ||
| ] | ||
| tgt_conn.execute(insert_sql, values) | ||
|
|
| if not sqlite_path.is_file(): | ||
| raise click.UsageError(f"SQLite database not found: {sqlite_path}") | ||
|
|
||
| tgt_dir, tgt_db = _resolve_tgt_config(target_config) |
Comment on lines
+252
to
+260
| # Heuristic: tables whose name appears as a prefix in other FK | ||
| # constraint names are likely parent tables. | ||
| fk_tables: set[str] = set() | ||
| try: | ||
| for r in tgt_conn.execute( | ||
| "SELECT table_name FROM information_schema.table_constraints " | ||
| "WHERE constraint_type='FOREIGN KEY' AND table_schema='public'" | ||
| ): | ||
| fk_tables.add(r[0]) |
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.



PR Note:feat(cli)
octop db migrate-to-pg— SQLite → PostgreSQL 数据迁移What
新增
octop db命令组,首个子命令migrate-to-pg:把存量 SQLite 控制面数据完整迁移到 PostgreSQL。octop db migrate-to-pg --target-config /path/to/config.json [--source-home ~/.octop] [--dry-run] [--yes]--target-config(必填):目标 PG 的 config.json 路径--source-home:源 OCTOP_HOME,默认$OCTOP_HOME或~/.octop--dry-run:只预检不落数据--yes:跳过交互确认Why
feat/hitl-postgres-store)同属控制面 PG 化方向,是该方向的基础设施建设How
octop init(只搬数据、不建 schema);目标_schema_version不得低于源版本SET session_replication_role = 'replica'临时关闭 FK 检查;无 SUPERUSER 时按启发式排序(无 FK 表先迁)_pg0param():REAL→%s::double precision、JSON→%s::jsonb、BLOB→%s::bytea),_convert_value()逐单元格转换DELETE FROM→ 逐行 INSERT(事务包裹)→ 行数比对,不一致记 MISMATCH 并最终报错_schema_version、sqlite_sequence(PG_NATIVE_TABLES)不迁移;源 0 行且目标非空的表 SKIP 保留设计要点
Test
P0:类型映射单元测试 ✅
37 项新增测试已合入
tests/unit/cli/test_cli_db.py(commitdb99e8df),覆盖:_pg0param():22 组参数化用例(REAL/FLOAT/DOUBLE→%s::double precision、INTEGER/INT/BIGINT→%s::bigint、BLOB→%s::bytea、JSON→%s::jsonb、TEXT/VARCHAR/BOOLEAN/TIMESTAMP/空串→%sfallback),含大小写不敏感_convert_value():NULL 透传(7 种类型)、浮点/整数类型派发(含跨类型输入int→float)、BLOB→bytes 往返、JSON 五种场景(有效字符串往返、dict/list 序列化、空字符串→[]、空白字符串直接透传、非法 JSON 字符串→JSONDecodeError)、未知类型 fallback(按 Python 值类型推断)全量
test_cli_db.py:43 passed(6 项原有测试零回归)手工验证路径
octop init建好 schema--dry-run预检 → 正式迁移 → 逐表行数比对一致P1:PG 集成冒烟测试(不阻塞本 PR 合入,后续补)
OCTOP_TEST_DATABASE_URLoctop db migrate-to-pg完整链路(含 FK 依赖表排序、类型映射端到端、行数校验 MISMATCH 告警)Limitations
Related
feat/hitl-postgres-store(HITL pending 记录 PG 持久化,依赖 PG 控制面)