Skip to content

chore(db): PostgreSQL 迁移统一分支 - #299

Draft
cropflre wants to merge 696 commits into
mainfrom
pg-migration-unified
Draft

chore(db): PostgreSQL 迁移统一分支#299
cropflre wants to merge 696 commits into
mainfrom
pg-migration-unified

Conversation

@cropflre

@cropflre cropflre commented Jul 15, 2026

Copy link
Copy Markdown
Owner

目的

所有未完成的 PostgreSQL 迁移继续统一在 pg-migration-unified 推进,不拆分新的长期迁移分支。PR 继续保持 Draft;当前仍是 PostgreSQL runtime-only,不宣称可以生产切换。

主线同步状态

  • 最新主线:main@c2069b2971f23eca17ed49b9221a343ca6cc5fa3
  • 当前分支:pg-migration-unified@31bdeae22a7bef30e0c68fe49d9e697e33f3cede
  • 当前比较:behind_by = 0ahead_by = 695
  • 已带入主线最新 MiCloud 大批量导入加固及前后端回归;一次性同步工作流已从 main 清理。

当前已开放的 PostgreSQL Runtime

Notes HTTP Runtime

  • GET /api/notes
  • POST /api/notes
  • GET /api/notes/trash/summary
  • DELETE /api/notes/trash/empty
  • GET /api/notes/:id
  • GET /api/notes/:id?slim=1
  • PUT /api/notes/:id
    • Tiptap JSON / Markdown / HTML
    • title / content / contentFormat
    • pin / favorite / archive / lock
    • trash / restore / same-workspace move / sortOrder
  • PUT /api/notes/reorder/batch
  • DELETE /api/notes/:id

Realtime / Yjs

  • WS /ws
    • note/workspace room subscriptions
    • presence/editing/cursor
    • note/workspace mutation events
    • Yjs join/leave/read sync/state-vector diff
    • awareness relay
    • y:update PostgreSQL 持久化与广播
    • snapshot compaction / update GC
  • WS /ws/subdocuments
    • Tiptap subdocument manifest/state
    • 稳定章节边界 Yjs 写入
    • 章节新增、删除、移动、拆分、合并的结构变化事务
    • section/root manifest ACK、reload、幂等重试与断线清理

本轮完成:Subdocument 结构变化事务与 root manifest 重建

新协议

新增 WebSocket 消息:

  • y:subdoc:structure
  • y:subdoc:structure-ack

客户端提交完整 Tiptap 文档、当前 generation 和稳定 operationId。服务端只接受已加入 manifest 且拥有写权限的连接。

原子事务边界

一次结构变化在同一 PostgreSQL 事务内完成:

  • 乐观锁更新 notes.contentcontentTextversion
  • 保存旧正文到 note_versions
  • 重建 note_blocks_index
  • 清理旧 section update 与 section snapshot;
  • 生成全部新 section snapshot;
  • 原子重建 root snapshot / manifest;
  • generation 与 structureVersion 同步递增;
  • 清理不再可信的 legacy note_yupdates / note_ysnapshots
  • 写入结构操作账本。

migration 0014

新增 note_y_subdocument_structure_operations

  • (noteId, operationId) 主键;
  • 记录 actor、base/result generation、result structureVersion、note version、content hash;
  • 相同 operationId + content 重试直接重放成功结果;
  • 相同 operationId 搭配不同正文返回 SUBDOCUMENT_OPERATION_REUSED
  • generation/version 冲突 fail-closed,并返回最新 manifest/version 供客户端 reload。

安全与协议限制

  • 结构正文最大 8 MiB;
  • 仅接受完整、有效的 Tiptap JSON;
  • 要求已有稳定 block ID,不在结构事务中静默生成新 ID;
  • 章节边界未变化时拒绝结构接口,继续走稳定 section update;
  • ACK 只在数据库提交成功后发送;
  • 提交后清理旧 room membership 并广播 reload;
  • owner/admin/manage/editor 可写,viewer 保持只读。

前序已完成

  • PostgreSQL y:update 持久化、广播、版本冲突保护;
  • Yjs baseline snapshot、snapshot compaction 与 update GC;
  • Tiptap subdocument manifest/state;
  • 稳定章节边界 Yjs 写入;
  • /ws/ws/subdocuments upgrade dispatcher 共存;
  • JWT/session/tokenVersion 与动态 note 权限校验。

最终验证(当前提交 31bdeae2

PostgreSQL Runtime 30599921383

  • TypeScript ✅
  • Direct DB Audit ✅
  • PostgreSQL schema parity ✅
  • SQL dialect/runtime unit tests ✅
  • 完整 SQLite 串行回归 ✅
  • 真实 PostgreSQL adapter/migrations/repositories/Notes/Yjs/Subdocuments 全套回归 ✅
  • Production backend bundle ✅
  • migrations 0001~0014 打包 ✅
  • PostgreSQL runtime-only smoke ✅

PostgreSQL Yjs Subdocuments 30599921374

  • migrations 0001~0014 bootstrap / idempotency ✅
  • 稳定 section 真实 PostgreSQL 事务 ✅
  • 结构变化真实 PostgreSQL 事务 ✅
  • 新增、移动、删除/合并 section ✅
  • operation retry / reuse / stale generation ✅
  • 稳定 section 与结构变化 WebSocket 协议 ✅
  • Production backend bundle 与 00130014 打包 ✅

PostgreSQL Runtime Smoke 30599921355

  • PostgreSQL health ✅
  • 新 subdocument route / capability / idle stats ✅
  • 结构写入已从 pending capability 移除 ✅
  • 未授权 API 边界、优雅关机、无 SQLite 文件副作用 ✅

PostgreSQL Permissions 30599921401

  • TypeScript、Direct DB Audit、schema parity、SQL dialect ✅
  • 真实 PostgreSQL permission/access-session 回归 ✅
  • 完整 SQLite 串行回归、构建与 migration 打包 ✅

其他关键门禁

  • PostgreSQL Direct DB Audit 30599921406
  • PostgreSQL Note Runtime 30599921393
  • MiCloud Import CI 30599921362:前后端回归与类型检查 ✅

下一迁移切片

进入 Block Patch / noteTransfer / notebook / knowledge-tree 跨驱动事务收口

  • 把多表写入统一下沉到 Repository / transaction boundary;
  • SQLite 与 PostgreSQL 共用同一业务事务语义;
  • 为失败回滚、幂等重试和并发冲突增加真实 PostgreSQL 回归;
  • 优先处理会修改 note、tree、permission、attachment/link metadata 的复合操作。

仍待后续阶段

businessRoutesReady = false,PR 继续保持 Draft。

cropflre commented Jul 16, 2026

Copy link
Copy Markdown
Owner Author

✅ PostgreSQL access/session validation passed

Validated branch: pg-migration-unified
Validated head: 9085b5453e6bd012aa60aab8229b63b0e43a1622
Run: 29554768794

  • ✅ npm ci
  • ✅ TypeScript
  • ✅ SQLite direct-access audit
  • ✅ PostgreSQL schema parity
  • ✅ SQL dialect tests
  • ✅ Full SQLite regression
  • ✅ Real PostgreSQL access/session
  • ✅ Production backend bundle

@github-actions

Copy link
Copy Markdown
Contributor

❌ PostgreSQL unified validation failed

Validated branch: pg-migration-unified
Validated head: 1ccaabf709f98dd8bb4bec92fbe5bc976206f9b5
Run: 29510812222

  • ✅ npm ci
  • ✅ TypeScript
  • ✅ SQLite direct-access audit
  • ❌ PostgreSQL schema parity(exit 1)
  • ✅ Runtime / SQL dialect tests
  • ✅ Full SQLite regression
  • ✅ Real PostgreSQL tests
  • ✅ Production backend bundle
  • ✅ PostgreSQL runtime resources packaged

Full diagnostic log is attached to the workflow run artifact.

@github-actions

Copy link
Copy Markdown
Contributor

❌ PostgreSQL unified validation failed

Validated branch: pg-migration-unified
Validated head: 1ccaabf709f98dd8bb4bec92fbe5bc976206f9b5
Run: 29510954347

  • ✅ npm ci
  • ✅ TypeScript
  • ✅ SQLite direct-access audit
  • ❌ PostgreSQL schema parity(exit 1)
  • ✅ Runtime / SQL dialect tests
  • ✅ Full SQLite regression
  • ✅ Real PostgreSQL tests
  • ✅ Production backend bundle
  • ✅ PostgreSQL runtime resources packaged

Full diagnostic log is attached to the workflow run artifact.

Copy link
Copy Markdown
Owner Author

/validate-pg-unified

@github-actions

Copy link
Copy Markdown
Contributor

❌ PostgreSQL unified validation failed

Validated branch: pg-migration-unified
Validated head: 3291f44ead31429e8d91f6fe1114704f5b64f5c2
Run: 29511494621

  • ✅ npm ci
  • ✅ TypeScript
  • ✅ SQLite direct-access audit
  • ❌ PostgreSQL schema parity(exit 1)
  • ✅ Runtime / SQL dialect tests
  • ✅ Full SQLite regression
  • ✅ Real PostgreSQL tests
  • ✅ Production backend bundle
  • ✅ PostgreSQL runtime resources packaged

Full diagnostic log is attached to the workflow run artifact.

Copy link
Copy Markdown
Owner Author

/validate-pg-unified

@github-actions

Copy link
Copy Markdown
Contributor

✅ PostgreSQL unified validation passed

Validated branch: pg-migration-unified
Validated head: 542935f50c66255470d77c4d38495007ad970cd5
Run: 29511804247

  • ✅ npm ci
  • ✅ TypeScript
  • ✅ SQLite direct-access audit
  • ✅ PostgreSQL schema parity
  • ✅ Runtime / SQL dialect tests
  • ✅ Full SQLite regression
  • ✅ Real PostgreSQL tests
  • ✅ Production backend bundle
  • ✅ PostgreSQL runtime resources packaged

Full diagnostic log is attached to the workflow run artifact.

Copy link
Copy Markdown
Owner Author

/validate-pg-unified

@github-actions

Copy link
Copy Markdown
Contributor

✅ PostgreSQL unified validation passed

Validated branch: pg-migration-unified
Validated head: b828395c3fd29ae901d52039ebc07ba39a6fdc1c
Run: 29512099142

  • ✅ npm ci
  • ✅ TypeScript
  • ✅ SQLite direct-access audit
  • ✅ PostgreSQL schema parity
  • ✅ Runtime / SQL dialect tests
  • ✅ Full SQLite regression
  • ✅ Real PostgreSQL tests
  • ✅ Production backend bundle
  • ✅ PostgreSQL runtime resources packaged

Full diagnostic log is attached to the workflow run artifact.

Copy link
Copy Markdown
Owner Author

/validate-pg-unified-batch-b

@github-actions

Copy link
Copy Markdown
Contributor

✅ PostgreSQL Batch B validation passed

Validated branch: pg-migration-unified
Validated head: 6bba9a0da836c3fe1a4f3679aa78efbfaab2f231
Run: 29546608777

  • ✅ npm ci
  • ✅ TypeScript
  • ✅ SQLite direct-access audit
  • ✅ PostgreSQL schema parity
  • ✅ Runtime / SQL dialect tests
  • ✅ Full SQLite regression
  • ✅ Real PostgreSQL tests
  • ✅ Production backend bundle

Copy link
Copy Markdown
Owner Author

/validate-pg-unified-batch-b

Copy link
Copy Markdown
Owner Author

/validate-pg-unified-batch-b-final

@github-actions

Copy link
Copy Markdown
Contributor

✅ PostgreSQL Batch B validation passed

Validated branch: pg-migration-unified
Validated head: 2b6b705efecedc69222c1bb434eb4a5343d6ff31
Run: 29546701987

  • ✅ npm ci
  • ✅ TypeScript
  • ✅ SQLite direct-access audit
  • ✅ PostgreSQL schema parity
  • ✅ Runtime / SQL dialect tests
  • ✅ Full SQLite regression
  • ✅ Real PostgreSQL tests
  • ✅ Production backend bundle

Copy link
Copy Markdown
Owner Author

#249 Batch B 已落地

最终 Head:2b6b705efecedc69222c1bb434eb4a5343d6ff31
验证 Run:29546774716

本轮直接在 pg-migration-unified 完成:

  • userAISettingsRepository async API 接入 Runtime Provider,并改为惰性解析 Adapter,避免模块导入早于 Runtime 初始化;
  • taskDependenciesRepository async API 接入 Runtime Provider;
  • taskProjectsRepository async API 接入 Runtime Provider;
  • 统一 SQLite / PostgreSQL 下项目统计字段的 number 返回;
  • 项目移除流程通过 executeStatements 保证解除任务关联与项目变更位于同一事务;
  • 新增 repository-batch-b-pg.test.ts,覆盖 AI 设置 upsert、项目统计/排序、任务依赖和事务行为;
  • Batch B 已加入常驻 pg-runtime.yml

最终验证全部通过:npm ci、TypeScript、SQLite direct-access audit、PostgreSQL schema parity、Runtime/SQL dialect、完整 SQLite 回归、真实 PostgreSQL 回归、生产 backend bundle。

一次性验证工作流已从 main 清理。PR 继续保持 Draft;#249 其余 Repository、#250 深度 Schema parity、#251#254 仍待推进。

Copy link
Copy Markdown
Owner Author

/validate-pg-permissions

2 similar comments

Copy link
Copy Markdown
Owner Author

/validate-pg-permissions

Copy link
Copy Markdown
Owner Author

/validate-pg-permissions

Copy link
Copy Markdown
Owner Author

本轮已直接在 pg-migration-unified 完成权限域 PostgreSQL 迁移:note ACL、workspace members、notebook members 的 async API 已改用 Runtime Provider;新增 0004_notebook_members_unique.sql 和真实 PG 测试 repository-permissions-pg.test.ts;新增常驻 pg-permissions.yml 门禁。最终验证 Head 2c04bc531409334479834c897a6ac22f7f904958,Run 29547923197:TypeScript、SQLite 审计、Schema parity、完整 SQLite 回归、真实 PostgreSQL、生产构建和 migration 资源打包全部通过。一次性 main 验证工作流已删除。

Copy link
Copy Markdown
Owner Author

/validate-pg-access-session

2 similar comments

Copy link
Copy Markdown
Owner Author

/validate-pg-access-session

Copy link
Copy Markdown
Owner Author

/validate-pg-access-session

cropflre and others added 28 commits July 30, 2026 21:21
Merge main into pg-migration-unified before continuing PostgreSQL subdocument structure migration.
Merge the temporary command-workflow cleanup from main into pg-migration-unified.

Copy link
Copy Markdown
Owner Author

本轮迁移完成

  • 已将最新 main@c2069b2 完整同步到 pg-migration-unified@31bdeae2,当前 behind_by = 0ahead_by = 695
  • 已完成 Yjs Subdocument 结构变化写入:支持章节新增、删除、移动、拆分与合并。
  • 新增 migration 0014_yjs_subdocument_structure_operations.sql,提供 operationId 幂等账本、generation/version 乐观锁和冲突恢复。
  • 结构事务会原子更新 note 正文/版本、版本历史、block index、section snapshots、root manifest,并清理失效的 legacy Yjs 状态。
  • 最终提交的 PostgreSQL Runtime、Yjs Subdocuments、Runtime Smoke、Permissions、Direct DB Audit、Note Runtime、MiCloud 前后端 CI 均已通过。

下一切片进入 Block Patch / noteTransfer / notebook / knowledge-tree 跨驱动复合事务收口。PR 继续保持 Draft,businessRoutesReady = false

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant