diff --git a/docs/MSSQL_SQL_EXPORT.md b/docs/MSSQL_SQL_EXPORT.md new file mode 100644 index 0000000..9c768f3 --- /dev/null +++ b/docs/MSSQL_SQL_EXPORT.md @@ -0,0 +1,59 @@ +# MSSQL SQL logical export + +This exporter creates a local UTF-8 SQL script for explicitly selected SQL Server tables, optionally including supported dependent views. It is a logical export, not a native `.bak`, and does not change scheduled backups or other providers. + +## Source and restore requirements + +Data export uses one owned connection and one SNAPSHOT transaction. The source database must already have `ALLOW_SNAPSHOT_ISOLATION ON` with state `ON`; the exporter does not enable it or fall back to weaker isolation. Schema-only export does not require a data snapshot. The account needs SELECT plus readable definitions for the selected objects and dependencies. Missing prerequisites and unsupported features block preflight. + +Restore to a prepared, compatible database using **one sqlcmd process/session for the entire artifact**: + +```powershell +sqlcmd -S '' -d '' -E -b -x -f 65001 -i '' +``` + +`-b` stops on SQL errors; `-x` disables sqlcmd variable substitution; `-f 65001` specifies UTF-8 input. Do not split/reconnect between GO batches: large rows use session-local staging. The existing GUI SQL import splitter is not the validated import path. Review the destination before restoring: schema-and-data creates objects and does not drop/replace existing objects; data-only assumes compatible existing objects. Re-running a whole artifact is not an idempotent restore. + +For the tested official **Go sqlcmd v1.10.0**, input is natively UTF-8 and `-f` is not supported. Use its explicit alternative command: + +```powershell +& '' -S '' -d '' -E -b -x -i '' +``` + +The installed classic sqlcmd 15.0.1300.359 fails when this fixture's input crosses 2 GiB. Do not use that version for >2 GiB artifacts. The Go client's UTF-8 decoder and one retained database connection are documented in its [version-pinned source](https://github.com/microsoft/go-sqlcmd/blob/v1.10.0/pkg/sqlcmd/sqlcmd.go#L339-L373). Local compatibility tests check Unicode codepoints/catalog names, literal substitution syntax, GO parsing, same-session temporary state and fail-fast behavior. On this LocalDB environment Go requires the exact running instance pipe (`np:\\.\pipe\LOCALDB#\tsql\query`); the shorthand `(localdb)\name` did not resolve. Never substitute another instance. See verification for the large-file result and client provenance. +## Supported behavior and boundaries + +Native scalar types retain precision and temporal offsets, raw ANSI/Unicode/binary values retain bytes, and NULL differs from empty or trailing spaces. Supported schema includes identity seed/increment, computed columns, rowversion definitions, defaults, keys, supported indexes, checks, foreign keys, comments and supported dependent views. Computed and rowversion values are regenerated, not copied. Identity source allocation history/gaps are not promised. Empty tables retain their declared identity seed. Named filegroups must already exist on the destination. + +Large values are streamed in bounded chunks; a row may span multiple GO batches. The importer validates ordered chunks, totals and completion before its one target INSERT. Replaying a consumed row's terminal batch does not insert it twice. A failed row is not replaced with NULL, even if a client continues after errors. **Earlier successfully inserted rows may remain:** restoration of the whole file is not all-or-nothing. Stop and investigate any sqlcmd error. + +Legacy `text` rows use typed staging even for small values because direct varbinary assignment to text is rejected by SQL Server. This preserves the source codepage but increases script size and restore work for those rows. `ntext`, `image`, untyped XML and ordinary WKT text are supported within the documented tests; native SQL Server `sys.geometry` now uses `Serialize()` -> bounded binary chunks -> `geometry::Deserialize()`, with small values optionally inline. Native geometry DDL, NULL/empty, serialized bytes, SRID and Z/M are retained without WKT conversion, simplification or MakeValid. Geography, user-defined/alias CLR types, spatial indexes, typed XML, sql_variant, unsupported dependencies/table/column/index features and unreadable definitions are rejected rather than silently omitted. Tables with spatial indexes remain blocked in all modes, including data-only; geometry value support does not imply spatial-index scripting. Refer to the current preflight details for the affected object/column. + +The transaction remains open while source rows are exported, retaining snapshot versions. It is released before local hash verification and mirror work. Allow disk space for source version store, SQL hex expansion (often more than twice binary payload), partial output, verified local output and optional mirror copies. Progress reports actual source bytes/rows and separate file verification/mirror work; estimates can remain unknown. More bytes in an export do not mean rows have been successfully restored. + +Cancellation waits for owned reader/command/transaction/file cleanup. The final consent stamp detects destination changes observed before publication. Windows File.Replace does not provide an atomic compare-and-swap against unrelated external writers: a change in the narrow final-check/replace window can still be displaced. Recovery paths are retained and reported; after a partial replacement error, the previous file may be at a recovery path instead of its original location. Use a new filename for externally managed targets. Before replacement starts, failure or cancellation preserves the existing output. If the local file was published but the mirror fails or is canceled, the result is **local completed, mirror incomplete**, not complete success. Any retained recovery paths are reported. + +## Reproducing the isolated acceptance + +The opt-in runner defaults to instructions only: + +```powershell +.\tests\Run-MssqlExportIntegration.ps1 +.\tests\Run-MssqlExportIntegration.ps1 -Run -Server '(localdb)\mySQLPunkExportTest_20261012' -Profile Core -OutputDirectory '' +# Profiles Geometry, Large and Concurrency use the same explicit arguments. +# Official portable Go client is opt-in: -Client Go -SqlcmdPath . +# Profile ClientProbe verifies client semantics before a large restore. +# Large -RestoreArtifact validates prior SHA/token, then restores it in a new fixture. +``` + +This task's runner deliberately accepts only its authorized dedicated instance and the local authorization manifest at `.superpowers/sdd/implementation-plan/fixture-environment.json`. It requires that instance to be running, installed sqlcmd, Integrated Security and sufficient free C: space. It does not install/start databases, read saved connections/credentials, or accept a user database name. Each invocation generates GUID `_src`/`_dst` databases with an ownership token, bounded MDF/LDF sizes and exact file paths, then validates token/names/paths before DROP. Outputs and evidence remain local under the fixture root; no raw large artifact is committed. Never work around an antivirus block. + +Core uses actual service artifacts, catalog comparison and streaming SHA-256 row digests. Large uses independent 1×/10× processes, actual file length, single-session restore and 25 ms memory sampling by service phase. Concurrency uses actual source connections with controlled second-connection updates, DDL, blocking/cancellation, disconnect and filesystem faults. The verifier retains only per-row digests, never whole field values. SQL fixture setup batching is separate from actual artifact restoration. + +See [verification evidence](design/mssql-export/verification.md) for exact PASS/FAIL/PENDING scope. Native wizard/device sound/long-job UI, SSH lifecycle and GUI import require their own evidence; local engine acceptance does not establish those, production, IIS or user-account acceptance. + +## Geometry restore compatibility + +The Geometry profile verifies native geometry on SQL Server 2025 (17.0.4025.3), including a 17,600,048-byte serialized value, with real service output restored through one classic sqlcmd session. It compares streamed Serialize() hashes plus catalog, SRID, Z/M and empty/validity observations; invalid but storable shapes remain invalid and unchanged. Malformed serialized bytes or invalid chunk ordering fail the guarded row without inserting a partial value. + +This is SQL Server's internal spatial serialization format, not portable OGC WKB. Cross-version engine compatibility and theoretical maximum-size geometries require their own restore acceptance. The client remains bounded; source Serialize(), target Deserialize() and staging still require SQL Server memory/tempdb/log/disk resources. See [Geometry design](design/mssql-export/geometry-design.md) and [verification](design/mssql-export/verification.md) for exact tested boundaries. diff --git a/docs/design/mssql-export/README.md b/docs/design/mssql-export/README.md new file mode 100644 index 0000000..0431eb5 --- /dev/null +++ b/docs/design/mssql-export/README.md @@ -0,0 +1,44 @@ +# MSSQL SQL 匯出精靈：互動畫面原型 + +這是 2026-10-12 設計討論用的獨立 HTML 原型；正式 WinForms 精靈及匯出引擎另外實作。所有資料、速度、ETA、檢查結果、成功與取消狀態均為模擬；不連線資料庫，也不產生 SQL 檔案。 + +使用者已於 2026-10-12 同意此畫面及修訂後的正式功能規格：[design.md](design.md)。[實作計畫](implementation-plan.md) 已同意並執行。正式功能與實際驗證請看 [操作說明](../../MSSQL_SQL_EXPORT.md)、[驗證紀錄](verification.md)；下列紀錄只描述當時的 HTML 原型。 + +## 開啟 + +可直接開啟 `index.html`，或以 Python 3.8+ 啟動本機預覽： + +```powershell +python docs/design/mssql-export/serve-preview.py --port 17643 +``` + +網址：`http://127.0.0.1:17643/docs/design/mssql-export/index.html` + +預覽伺服器只綁定 loopback，僅提供此頁與三個明確允許的 Punky 素材，不提供資料夾索引。按 Ctrl+C 停止。 + +## 可以操作 + +- 上方原型導覽直接切換選表、輸出設定、確認、進度與完成。 +- 選表預設全選；可搜尋表名／註解，跨搜尋保留勾選，零選取停用下一步。 +- 選擇結構與資料／僅結構／僅資料；編輯示意輸出路徑。 +- 右下情境選單可模擬未啟用快照，查看確認頁的提示與停用開始按鈕。 +- 從確認頁按「開始模擬匯出」，約 40 秒內走過資料匯出、檔案驗證與完成；取消則經過停止與清理的模擬狀態。 +- 深淺色切換；進度頁 Punky 自動循環播放，只有靜音控制。實際開始模擬時嘗試播放一次原音；仍受瀏覽器播放政策與裝置音量影響。 +- 開啟資料夾是示意動作，會提示沒有實際輸出。 + +## 素材與邊界 + +直接引用既有 `mySQLPunk/image/punky_report_loading_atlas.png`（384px、73 格、83ms／格）、靜態圖與 WAV，不複製大型素材。正式版仍需以真正的 MSSQL 工作狀態、吞吐量與檔案驗證結果驅動畫面；本原型的數字與進度算法不可當成實作。 + +原型導覽是設計檢視工具，正式精靈執行中不能藉此切換設定。本 HTML 原型未實作 Windows 儲存檔案對話框、真實連線／SSH、快照條件、取消 SQL、串流匯出與還原驗證；正式版本的證據另列於驗證紀錄。 + +## 本機驗證紀錄（2026-10-12） + +- JavaScript 與 Python 語法檢查通過；本機 HTTP 回應 200。 +- 在 Brave 實際操作：預設 8/8 全選、清空停用下一步、註解「訂單」搜尋出 2 表、跨搜尋保留 7/8 選取，空路徑停用下一步。 +- 快照情境切為未啟用時，確認頁出現原因並停用開始；切回可用後可模擬匯出至完成。 +- 取消依序呈現「正在取消」與「本次匯出已取消」；深淺色、Punky loop 及畫面截圖已檢查。 +- 原型視窗限制高度、內容區捲動、固定底部操作列；目前瀏覽器尺寸下完整進度與三列工作狀態可見。其他 DPI／視窗尺寸的正式 WinForms 驗收尚未進行。 +- 未實際驗證裝置聽到的音效，也未執行應用程式／測試 EXE 或連線 DB。 + +已保存 [淺色進度圖](preview-progress-light.png) 與 [深色進度圖](preview-progress-dark.png)。 diff --git a/docs/design/mssql-export/design.md b/docs/design/mssql-export/design.md new file mode 100644 index 0000000..5f5f8a0 --- /dev/null +++ b/docs/design/mssql-export/design.md @@ -0,0 +1,224 @@ +# MSSQL 大量資料 SQL 匯出精靈設計 + +日期：2026-10-12 + +狀態：互動畫面、修訂規格與[實作計畫](implementation-plan.md) 已獲使用者同意；2026-10-12 使用者選擇「同意計畫，分工實作並逐項覆核（建議）」。產品實作、必要的隔離還原／容量驗證及最終覆核已完成；實際證據見 [verification.md](verification.md)。最後新增提示已於 2026-10-12 完成深／淺色原生 fixture 畫面重驗；SSH、實體 DPI／聲音及原生長作業仍待驗證；尚未合併主線或推送。 + +2026-10-12 修訂：使用者要求調整單筆大欄位限制，關心 geometry／長文字，並接受極端型別後續再擴充。取消原草案「單列 SQL 超過 16 MiB 就拒絕」規則；一般長文字／binary 改為可分段輸出，原生 spatial 型別仍列後續擴充。型別可容納的大小是設計目標，不代表所有理論上限已完成實機驗收。 + +## 1. 已確認的目標 + +使用者要升級 MSSQL 的「建立備份」及「匯出結構與資料」，解決大量資料時畫面卡住、記憶體不足與無法掌握進度的問題。第一版限定本機 `.sql` 邏輯匯出，來源通常仍持續新增、修改資料。 + +2026-10-12 使用者看過互動原型後回覆：「畫面很棒，可以同意」。此確認記錄為畫面與互動方向的驗收，不代表真正串流匯出、DB 一致性或還原已驗證。 + +- 原型：[index.html](index.html)；操作：[README.md](README.md)。 +- 淺色：[preview-progress-light.png](preview-progress-light.png)。 +- 深色：[preview-progress-dark.png](preview-progress-dark.png)。 +- 流程：選表 → 輸出設定 → 確認／前置檢查 → 執行／結果。 +- 保留雙層進度、逐表狀態、實際筆數與檔案大小、耗時與 ETA、右側 Punky、自動動畫與靜音、固定底部取消按鈕。 + +## 2. 範圍與實作方向 + +### 本版納入 + +1. 既有 WinForms MSSQL 互動式本機 SQL 匯出入口共用專用精靈及 worker；依入口預選結構與資料、僅結構或僅資料。資料庫入口預設全選，單表入口只預選指定資料表。 +2. 原生 SQL Server 型別與結構的正確腳本、逐表串流資料、大欄位分塊輸出、可取消查詢、串流檔案完整性驗證。 +3. 持續寫入來源採同一個 SNAPSHOT 交易；不自動變更 DB 設定。 +4. 「建立備份」保留目前選用的備份鏡像與保留份數功能；一般手動 SQL 匯出不因此新增鏡像。 +5. 沿用現有 Punky 素材、ThemeManager、UiKit、獨立連線及 SSH transport 的生命週期。 + +### 本版不納入 + +SQL Server 原生 `.bak`、還原工具重寫、排程／自動備份改造、其他 provider 重構、暫停或斷點續傳、平行多表讀取、壓縮／加密輸出，以及伺服器登入、權限、SQL Agent job 等完整災難復原能力。 + +此功能是所選物件的 SQL 邏輯匯出，不以「建立備份」舊選單名稱暗示等同完整 `.bak`。精靈確認與結果必須列出實際包含／未包含範圍。 + +### 選定的架構 + +新增 MSSQL 專用匯出服務與表單。另一方案是把所有 provider 的通用 dump API 全面改成串流，但會擴大 MySQL、PostgreSQL、SQLite 等既有行為的回歸範圍，因此留待有明確需求再做。 + +不以現有 `BuildDatabaseDump()`／`BuildTableDump()` 的字串結果餵給背景執行緒；那仍會累積整份 SQL。也不把報表 wizard 繼承成通用 wizard framework。 + +## 3. 已查到的根因與相容性注意 + +- `Form1.cs` 的非 MySQL 本機 dump 和 MSSQL「建立備份」同步執行，會占用 UI 執行緒。 +- `DatabaseDumpService.cs` 累積每表與整庫 StringBuilder，最後才寫檔；分頁查詢沒有解除輸出記憶體限制。 +- `BackupIntegrityService.cs` 對 SQL 使用 `File.ReadAllText`，鏡像後也可能再次整檔讀入。 +- `my_mssql.cs` 分頁使用 OFFSET；無 PK 時使用任意排序。新 worker 不沿用分頁路徑，每表只開一個連續 reader。 +- 現有建表語法使用跨庫 copy 型別映射，會改變 varchar、datetimeoffset、uniqueidentifier 等語意，也缺少 identity／computed 等必要 metadata。 +- `SecuredDatabase` 包裝 SSH lease。直接將 `IDatabase` 強制轉型成 `my_mssql` 在 SSH 路徑不可行。 +- 目前發布流程有 delete-then-move 缺口；新路徑不得先刪除舊目標檔案。 + +## 4. 畫面與操作契約 + +### 選擇資料表 + +- 顯示 schema、表名、comment、估計筆數與資料大小；以 schema/name/object_id 識別，不以顯示文字識別。 +- 可搜尋名稱及 comment。搜尋不清除其他資料表的選取；「全選全部」明確作用於全部資料表。 +- 零選取停用下一步。metadata 載入在背景執行，取消／失敗有明確狀態。 +- 資料大小是來源儲存量估計，不等同最後 SQL 大小；缺少估計資訊時顯示未知，不以 0 替代。 + +### 輸出設定 + +- 模式為結構與資料、僅結構、僅資料；使用 Windows 儲存檔案對話框選本機路徑，輸出 UTF-8 `.sql`。 +- 若目標已存在，確認覆寫後仍保留原檔直到新檔驗證與安全替換完成。 +- 庫級結構匯出須保留既有檢視定義能力，提供「包含檢視定義」選項，庫級入口預設勾選；僅資料不包含檢視定義。這是原型未展示、正式版需補上的選項。 +- 備份入口若有鏡像設定，確認頁顯示啟用狀態及目的地；不將登入密碼／連線字串放入畫面或日誌。 + +### 確認與前置檢查 + +列出來源、模式、表與檢視範圍、輸出路徑、估計工作量、一致性模式、鏡像，以及具體不支援項目。檢查失敗須指出物件及原因，保留選取供修正。 + +未啟用 SNAPSHOT 時，含資料模式停在此頁；不能自動降級成一般讀取或 NOLOCK。僅結構不需要跨表資料快照，但仍檢查 schema 變動與定義可讀性。 + +開始按鈕捕獲不可變的本次 request；執行期間不能改選取、路徑或模式。原型頂端的任意跳頁工具不進正式產品。 + +### 執行與結果 + +- 階段：準備 → 結構 → 逐表資料 → 檔案驗證 → 本機發布 → 選用鏡像 → 結果。 +- 同時顯示整體估計進度、目前表進度／已讀列數、表名與 comment、已完成表數、SQL 已寫 bytes、elapsed、速率與 ETA。 +- 詳細紀錄只留有上限的操作紀錄，不逐列寫 UI log，不記資料內容或秘密。 +- 取消按鈕固定在底部。按取消後顯示「正在取消…」，等 worker 清理結束才顯示已取消。 +- 結果區分成功、取消、失敗，以及本機成功但鏡像未完成；只有已發布產物提供開啟資料夾。 +- SQL 可能非常大，完成後不自動把整個檔案載入應用程式編輯器。 + +### Punky + +直接重用 `PunkyReportLoadingControl`。開始作業時 `BeginExport()` 自動 loop，原音只播放一次；靜音立即停聲、取消靜音不重播。只有靜音按鈕，無播放／暫停。完成、失敗、取消後 `EndExport()`；關閉／Dispose 釋放聲音、Timer 與素材資源。保持深淺色可讀性與 DPI／較小工作區的取消可達性。 + +## 5. Worker、連線與一致性 + +1. UI 只捕獲本次選擇及必要的連線設定，worker 經既有安全連線流程取得獨立連線及 SSH lease；不得借用主視窗 provider。 +2. 在 `ConnectionOpenService` 邊界提供最小的內部 MSSQL 工作連線存取方式，保持 wrapper 及 raw SqlConnection 同一擁有者。不改動所有 provider 的 `IDatabase` public contract，不以另開連線字串繞過 transport 生命週期。 +3. 前置檢查讀取目標 DB 的 `snapshot_isolation_state`，僅 ON 接受含資料快照；過渡狀態不當作可用。只開 `READ_COMMITTED_SNAPSHOT` 不等於這次跨表交易已使用 SNAPSHOT。 +4. 使用者選擇期間不持有快照。按開始後重做必要檢查，在指定資料庫開始一筆明確的 SNAPSHOT transaction，必要來源查詢均使用同一交易。 +5. 所有選取表依序用同一交易讀取；不能每表重新開交易，也不能中途重新連線續跑而仍宣稱同一時點。 +6. SNAPSHOT 不保證 metadata 可任意變更。預檢後凍結本次物件定義；物件變更、不可讀、schema 變動導致查詢失敗等情況中止本次工作，不跳過失敗表後顯示成功。 +7. 讀完最後來源資料及必要來源查核後立即結束交易並釋放 DB／SSH 資源；本機驗證、發布、mirror 不延長快照持有時間。 +8. 清楚提示長時間快照需要保留資料版本；不承諾零負載或完全無阻塞。取得不到版本儲存監控權限不應成為一般匯出的額外權限門檻。 + +參考：[Microsoft Snapshot isolation](https://learn.microsoft.com/en-us/dotnet/framework/data/adonet/sql/snapshot-isolation-in-sql-server)。 + +## 6. 有界記憶體與 SQL 正確性 + +### 讀寫方式 + +- 每表以明確欄位清單開啟 `SqlDataReader`／`SequentialAccess`，避免 OFFSET、整表 DataTable 或整表字串。 +- 不為取得精確進度先對每表完整 COUNT；使用 metadata 估計並保留未知狀態。 +- 文字與 binary 大欄位使用固定大小 buffer 分段轉義／十六進位寫出；不能先完整 GetString/GetValue/byte[] 後才分塊。 +- writer 可緩衝小段輸出，不累積整表、整份檔案或一個巨大 INSERT。一般小列走明確欄位 INSERT；整列的小資料緩衝預算最多 256 KiB，超過即轉分段路徑，不視為資料不能匯出。適當加 GO 分批，不把整個資料匯入綁成單一巨大 transaction。 +- 資料大小與 SQL 指令大小分開管理。大欄位以至多 64 KiB 原始 binary 或 16K 個 UTF-16 碼單元為起始 chunk 目標；轉義、hex 膨脹與字元邊界均算入實際大小。一般資料 statement 以 256 KiB 為生成預算，GO batch 目標不超過 4 MiB，必要時進一步縮小 chunk。這些是可調整的內部緩衝／批次參數，不是單筆資料的拒絕上限；不得在 INSERT／UPDATE 中間插 GO，亦不截斷欄位值。 +- bytes、筆數、時間計量使用可容納大檔案的型別；顯示及 I/O 不截成 32-bit int。 +- 查詢等待可取消，避免沿用一般查詢的短 timeout。worker 的長查詢 command 必須直接配置 timeout，不能依賴目前不涵蓋 local command 的 `setTimeout()`。 + +參考：[Microsoft SqlClient streaming](https://learn.microsoft.com/en-us/dotnet/framework/data/adonet/sqlclient-streaming-support)。CancellationToken 搭配所屬 SqlCommand.Cancel，不能只期待 token 自動中斷正在等待的 reader。 + +SQL Server 的 batch 長度受 network packet size 影響，總檔案可達數 GiB 不代表單一 statement 可以任意大。本版限制生成批次的大小，讓受支援的大欄位跨多個批次重建；不再以 16 MiB 定義通用單列上限。驗收使用支援 GO 的 Microsoft `sqlcmd`，記錄工具／伺服器版本與輸入設定。既有 mySQLPunk 匯入端仍需獨立驗證，不能假設它支援無界大檔。[Microsoft 容量限制](https://learn.microsoft.com/en-us/sql/sql-server/maximum-capacity-specifications-for-sql-server?view=sql-server-ver17) + +### 大欄位分段還原 + +1. 大列在同一個匯入 session 的私有 `#temp` staging 區組裝；不依賴目標表有 PK，不直接先插入空附件再反覆 UPDATE 目標列。先完整組裝及驗證，再一次 INSERT 到目標，避免約束／trigger 看到半列。 +2. `varchar(max)`、`nvarchar(max)`、`varbinary(max)` 使用對應 staging 型別與 `.WRITE(chunk, NULL, 0)` 追加。來源非 NULL 值先初始化空值，來源 NULL 另行標記，不能把 NULL 與空字串／空 binary 混為一談。varchar staging 明確使用來源 collation，不能繼承 tempdb 的預設定序而改變資料。 +3. 文字分段不切斷 surrogate pair，保留換行、尾端空白、單引號與字元語意；以 DATALENGTH 而非 LEN 核對 bytes。`text/ntext/image` 採相應的 max staging 與明確目標轉型，必須有還原測試，不能更改原表 DDL 型別來迴避問題。原生特殊型別使用獨立 codec，不假裝成一般文字。 +4. worker 按 reader 欄位次序讀取；小列緩衝超過門檻時，已讀的有界資料也轉入 staging 指令，之後繼續串流，不重新查來源列、不依賴 PK 或 OFFSET。 +5. staging 記錄工作／列識別、欄位、chunk 次序、已收 bytes、預期完成數與錯誤／完成狀態。只有所有欄位完整，最後短 transaction 才能執行目標 INSERT 並標記 consumed；缺段、重複、順序錯誤、轉型失敗、重複執行最後批次或 staging 不存在都明確失敗。 +6. 每個 batch 自己處理錯誤；TRY/CATCH 與區域變數不能跨 GO。不能只依靠 XACT_ABORT 或客戶端停止選項防止半列插入；即使後續批次被執行，最後完整性 guard 也須阻止錯誤資料進目標。SQL NULL 不能作為轉型失敗的替代結果。 +7. 匯入工具必須全檔維持同一 session，支援 GO，錯誤時停止，且不展開來源文字中的 sqlcmd `$(...)`。驗收使用單次 `sqlcmd -i`、`-b`、`-x` 及明確 UTF-8 輸入設定；不把必須同 session 的脚本分成多次連線執行。正式交付文件列出經驗證的實際參數；GUI 匯入只有通過同樣測試後才宣稱支援。 +8. 每列 INSERT 完就清掉該列 staging，不累積整張表。匯出端維持有界記憶體；還原端仍需足夠 tempdb／交易記錄與目標儲存空間容納當前大列，不能承諾無限資源。空間不足時，該列不得以殘缺值發布。 + +以下為設計依據；實際 SQL 驗證結果另記於交付證據，不以本節規格取代：[Microsoft UPDATE .WRITE](https://learn.microsoft.com/en-us/sql/t-sql/queries/update-transact-sql?view=sql-server-ver17)、[GO 與 batch 範圍](https://learn.microsoft.com/en-us/sql/t-sql/language-elements/sql-server-utilities-statements-go?view=sql-server-ver17)。 + +### 空間資料與後續擴充 + +- MSSQL 長文字主要是 `varchar(max)`／`nvarchar(max)`，亦可能有既有 `text/ntext`；`longtext` 是 MySQL 的型別名稱。本版只處理 MSSQL。 +- 若 WKT 或其他內容實際存在一般文字／binary 欄位，依原欄位型別串流保存，不解析、縮短或簡化其中內容。 +- 原生 `sys.geometry` 依 [Geometry 擴充規格](geometry-design.md) 支援 `Serialize → 分段 binary → Deserialize`，保留 native DDL、序列化 bytes、NULL／empty、SRID 與 Z/M；不轉成 WKT／nvarchar，也不修復或簡化圖形。`geography`、user CLR／alias 型別及 spatial index 仍在預檢明確阻擋；含 spatial index 的表連僅資料模式也維持拒絕。跨引擎版本與理論容量上限仍須額外還原驗證。 +- 使用者接受遇到極端型別再擴充，因此本版不要求同時支援所有空間／自訂型別，也不把未測試的多 GiB 邊界宣稱為已通過。真正超過型別容量、物件無法保真或環境資源不足時才明確停止，保留原正式檔案。 + +### 支援與預檢 + +v1 以一般 disk-based user tables 為主，從 `sys.*` 收集原生定義；不得沿用跨庫 type mapper。至少處理： + +- schema/name 正確引用，含空格、Unicode、點號、右方括號等名稱；不把整個 `sales.orders` 引用成單一識別字，也不將目標鎖死在來源 DB 名稱。 +- 原欄位型別、長度、precision/scale、nullability、collation、default 與表／欄位 comment。 +- Unicode／單引號／換行、NULL 與空值、binary；decimal(38,s) 不經不足精度的 CLR decimal；日期時間保留對應型別的精度與 datetimeoffset offset。 +- identity seed/increment 與既有欄位值，適當的 IDENTITY_INSERT；還原後新 identity 不與已匯入資料衝突。不承諾重建來源 identity 配號留下的所有間隙。 +- computed 定義／persisted；INSERT 排除 computed 與 rowversion，rowversion 由目標重生且在摘要說明。 +- 一般 rowstore PK/UQ/index，保留名稱、key 次序、ASC/DESC、INCLUDE、filter 與適用屬性。FK/CHECK 在資料之後建立，保留其有效／信任狀態。 +- 檢視定義用獨立 batch，保留必要 SET 選項、處理所選範圍內的依賴順序。無法讀取／加密／依賴未包含物件時預檢提出，不產生看似完整的結果。 + +第一版不支援的特殊型別／表／索引特性須在預檢明列並阻擋受影響選取，不可默默轉成其他型別或省略。包含 temporal/ledger、memory-optimized、FILESTREAM、user CLR/alias UDT、Always Encrypted、sql_variant、geography／spatial index、typed XML、partition／特殊索引等；原生 sys.geometry 值已由專用 binary codec 支援。新增支援必須有對應還原證據。 + +依賴自訂 function、sequence、trigger 或未匯出物件的定義需要具體列出；不猜測可用性。結構模式若選取不完整 FK／view 依賴，要求調整範圍，不用 NOCHECK 隱藏缺漏。僅資料模式明示需要相容且已建立的目標結構，不保證任意既有資料庫都能直接匯入。 + +## 7. 進度與 ETA 契約 + +- 以 immutable progress snapshot 回報工作 ID、階段、表、完成／估計列數、完成表數、已寫 bytes、elapsed、速率、可空的 ETA 及估計可信狀態。 +- UI Progress 在 UI context 接收；一般更新約 100–250ms 節流，階段／表切換、錯誤及完成立即回報。不每列塞入 message queue。 +- 開始與樣本不足時顯示「估算中」；網路／DB 等待或速率劇烈變動時退回估算中或顯示正在等待，不維持假倒數。 +- 用最近一段時間的平滑吞吐量估算。不同表列寬與 LOB 差異須納入估计工作量，不以完成表數等權平均。沒有足夠估計值時保持 indeterminate，仍顯示實際已完成值。 +- metadata row count 不必然等於快照實際列數；實際大於估計時更新說明，不越過 100% 或提早完成。 +- 整體百分比明標估計，包含後續檔案驗證／鏡像工作；階段切換不能重置整體為 0。驗證／鏡像可用實際 bytes 作分階段精確進度。 +- 完成前最多 99%；只有要求的步驟皆完成才為 100% 成功。鏡像失敗不得以整體成功覆蓋警告，但本機成功須保留。 +- 不保證 ETA 的固定誤差百分比；驗收其資料來源、變動／未知行為與不欺騙的完成判斷。 + +## 8. 取消、驗證與發布 + +### 狀態 + +Idle → Preflight → Reading/Writing → Verifying → Publishing → Mirroring（可選）→ Completed。 + +可中斷階段收到取消後轉 CancelRequested → 清理 → Canceled。不可重複的短發布區段使用明確互斥；若發布已完成，結果不得再假稱完全未輸出。 + +### 暫存與驗證 + +1. 在目標同目錄建立工作專屬的不可碰撞 `.partial` 檔，以 CreateNew 建立；不刪除其他工作或上次未知的暫存檔。 +2. writer 記錄 bytes、工作識別、已完成物件及列數，完整成功時才寫完成標记；關閉／flush 成功才進入驗證。 +3. 以固定 buffer 重新讀取，核對寫出時的 hash、長度與必要完整性標記。不得 ReadAllText、ReadToEnd，亦不可用可能整行巨大 INSERT 的無界 ReadLine。 +4. 驗證表示檔案與本次完整寫出結果相符，不等於 SQL 已在目標伺服器成功執行。真正語意正確性另由還原測試取得。 +5. 最後取消檢查之後，無目標時用不可覆寫 move；已有獲同意覆寫的目標時用同磁碟安全 replace。替換不支援或失敗時不退回先刪原檔；回報失敗及可恢復路徑。 +6. 確認覆寫後若目標又被其他工作變更，偵測衝突並停止發布，不能覆蓋未確認的新版本。 +7. 取消／失敗僅清理本工作暫存檔；清理失敗要列出殘留路徑，不把不完整檔案當成正式產物。日誌不含連線秘密或 SQL 資料值。 + +### UI 關閉 + +執行中關閉視窗視同取消請求，等 worker 清理完成再關閉。UI 不直接 Dispose 正在讀取的共享資源；operation ID 阻止舊工作回報污染新畫面。 + +## 9. 備份鏡像相容性 + +- 只對原本會 mirror 的備份入口保留鏡像；目的地及保留份數沿用設定，不擅自另建雲端／遠端服務。 +- 本機產物發布後，鏡像用工作專屬暫存檔分塊複製並回報 bytes；副本驗證完成才發布並執行既有保留規則。 +- 不能先清理舊備份再發現新副本無效。不得擴大既有清理規則至不相關檔案。 +- mirror 失敗、取消或目的地不可用：保留已成功本機檔案與既有遠端備份，結果明示「本機已完成，鏡像未完成」，附可採取的下一步。 +- 從 mirror 開始就已存在本機正式檔案；取消文案與結果須反映這個發布邊界。 + +## 10. 驗收與證據 + +| 項目 | 必須取得的證據 | +| --- | --- | +| 編譯 | 正式方案 Debug／Release 通過，既有 provider 路徑未被不必要變更 | +| 畫面 | WinForms 真正選表、查詢中不凍結、雙層進度、DPI／小工作區、深淺色、取消／關閉生命週期 | +| 大資料 | 超過 2 GiB 的實際產物；來源擴大至少 10 倍時記憶體不按匯出總量線性增加，记录 peak、rows、bytes、elapsed | +| 大欄位 | 跨過舊 16 MiB 門檻的文字／binary 必須走分段且可還原，核心案例至少 64 MiB binary 及超過 16 MiB 的 Unicode 文字；比對 bytes／雜湊、NULL／空值、尾空白、跨 chunk 引號與 surrogate、無 PK、多 LOB 同列。缺段／重複／故意轉型或空間失敗後不得插入半列。超過 1 GiB 與接近型別理論上限的壓力測試另列環境條件與結果，未測不得宣稱通過 | +| 空間欄位 | 原生 sys.geometry 走 Serialize／bounded binary／Deserialize，比對還原前後 raw bytes、SRID、Z/M、NULL／empty，含 >16 MiB 單值；geography 與 spatial index 仍明確預檢拒絕。一般文字中的大型 WKT 保留原文字，不自動轉型、截斷或簡化 | +| 型別與結構 | 在獨立測試 DB 匯入實際產物，驗證精度、offset、Unicode、binary、identity 後續新增、computed、rowversion 語意、constraints／indexes／view | +| 持續寫入 | 用另一連線交易性修改父子表，匯出仍對應同一快照，不混用不同時間點；版本條件不足時預檢停止 | +| 取消與競態 | 讀取／寫入／驗證／鏡像時取消、關閉窗、查詢等待、DB 中斷、磁碟滿、覆寫競態；原檔與結果狀態符合發布契約 | +| SSH | 透過工作自己的 transport 成功、取消、失敗後釋放；不影響主視窗連線 | +| 鏡像 | 成功驗證後保留清理；失敗／取消不清掉舊可用備份，本機產物仍可找到 | + +測試來源限明確的隔離／可拋棄 fixture，不在使用者正式 DB 建測試資料或擅改 SNAPSHOT 設定。沒有可用測試 DB 時，實機還原、持續寫入一致性、SSH 與大資料效能必須列 pending，不能由編譯或 fake-provider 測試代替。 + +保留既有 `ReportExportTests.exe` 防毒隔離邊界：不還原／執行、不變名重跑、不調整防毒。新增功能測試選用可用且未被隔離的正常測試管道；若環境阻擋就明列未驗證，不繞過。 + +## 11. 交付與流程狀態 + +- 已交付獨立互動原型；真實匯出程式與隔離 DB 驗證進行中，尚未完成整體交付。 +- 本文件已獲同意，具體實作步驟與測試安排見 [implementation-plan.md](implementation-plan.md)。規格核准不擴張為 merge、push 或 release 授權。 +- 舊版入口相容、metadata／SQL writer、worker／取消／發布、精靈與進度、實機 fixture 是同一功能的必要工作，不能只接好畫面就宣布完成。 +- 開發過程保留既有檔案 BOM／換行、未相關工作樹內容；重要結果追加 `history.md`。 + +## 2026-10-12 Geometry 擴充 + +使用者明確要求繼續推進 Geometry `Serialize → 分段 binary → Deserialize`，取代初版對 geometry 值的延後決策。範圍與驗收見 [Geometry 規格](geometry-design.md)／[實作計畫](geometry-implementation-plan.md)；實際 SQL2025 還原、完整回歸與 Debug／Release 編譯已通過；針對 5edf69d..e3d4998 的獨立完整擴充覆核亦通過，沒有需修正的問題。其餘 provider、geography、spatial index 與既有未驗證環境邊界維持不變。 diff --git a/docs/design/mssql-export/geometry-design.md b/docs/design/mssql-export/geometry-design.md new file mode 100644 index 0000000..95777dc --- /dev/null +++ b/docs/design/mssql-export/geometry-design.md @@ -0,0 +1,30 @@ +# MSSQL Geometry binary export extension + +Date: 2026-10-12. User authorization: continue `Geometry Serialize -> chunked binary -> Deserialize` after final native wizard verification. This extends the approved export design; the earlier v1 spatial rejection remains historical evidence, not the current intended geometry contract. + +## Scope and data flow + +Support SQL Server's built-in `sys.geometry` columns in the existing SchemaAndData, DataOnly and SchemaOnly workflows. Keep native `geometry` DDL. Project each non-computed source geometry as `CONVERT(varbinary(max),[column].Serialize())`, consume through the existing sequential bounded binary reader, and reconstruct with `geometry::Deserialize` in the target SQL. Small values may remain inline; larger values use the existing guarded varbinary(max) staging and chunk protocol. SQL NULL remains distinct from an empty geometry. + +Recognize only the native type (`type_schema=sys`, `type_name=geometry`, not user-defined); its assembly-type flag is expected. Do not grant a name-only exemption to aliases or user CLR types. Preserve existing computed/rowversion handling and all other preflight checks. Geography and spatial indexes remain unsupported and explicitly blocked, including data-only tables with spatial indexes under the existing conservative index policy. No spatial metadata/DDL is silently discarded. + +## Global constraints + +- Reuse the existing .NET Framework 4.7.2 exporter; no new client SQL CLR/geometry package, no public API or UI flow redesign. +- Same source connection/SNAPSHOT and SequentialAccess; no whole-field GetValue/GetSqlBytes, byte-array materialization, WKT normalization, source reread, or extra SELECT per value. +- Preserve existing bounds: binary chunks <=64 KiB, small-row buffer <=256 KiB, value statements <=256 KiB, batches <=4 MiB, metadata-only setup/final INSERT <=2 MiB. No 16 MiB value rejection. +- Retain length/order/completion/failed/consumed guards, single final target INSERT, cancellation, cleanup, and verify-before-publication contracts. +- Geometry must preserve serialized bytes, NULL/empty, SRID and Z/M. Do not simplify, repair, reproject, or convert native values to WKT. +- Geography, aliases/user CLR types and spatial indexes remain preflight failures; identify the object/column or index in the existing safe diagnostics. +- Use only the already authorized `(localdb)\mySQLPunkExportTest_20261012` and generated token-owned C: fixtures. The existing manifest, ownership checks and guarded cleanup remain mandatory. Never touch saved connections, user databases, quarantined ReportExportTests.exe, antivirus or account state. +- Preserve each edited file's encoding/BOM and line endings, unrelated files, and original master. No merge, push, release or large artifact in Git. + +## Acceptance + +Focused tests must prove native-type discrimination, SQL projection/escaping, inline NULL/small values, large-row retained/partial/unread handoffs, bounded reads, accurate progress bytes and cancellation. Real service export and one-session classic sqlcmd restore must preserve streamed Serialize() hashes and catalog definitions for Point, NULL, empty, SRID 0/4326/3826, Z/M, collection, polygon and supported curves. Include a mixed scalar/multi-geometry/ordinary binary row and a native geometry serialized value >16 MiB. Compare independent SRID/Z/M/type observations as well as bytes. Malformed serialized content or omitted/replayed/out-of-order chunks must not insert the failing row or allow a later protocol batch to revive it. + +The client verifier must identify CLR native types via sys.types joined on user_type_id, not TYPE_NAME(system_type_id). Hash geometry through Serialize() without materializing it. Server-side Serialize/Deserialize and restore staging necessarily consume SQL Server resources; bounded exporter memory does not imply bounded server memory. Actual SQL2025 is the available engine; cross-version internal-format compatibility, theoretical size limits, geography and spatial-index restoration remain outside this extension's acceptance. + +## References + +The user's [MSSQL dump article](https://3wa.tw/mypaper/?uid=shadow&mode=view&id=2760) supplies the serialization direction. Microsoft documents [Serialize](https://learn.microsoft.com/en-us/dotnet/api/microsoft.sqlserver.types.sqlgeometry.serialize?view=sql-dacfx-161) and [Deserialize](https://learn.microsoft.com/en-us/dotnet/api/microsoft.sqlserver.types.sqlgeometry.deserialize?view=sql-dacfx-161) as the paired SQL Server internal spatial binary format operations. Retain bounded streaming rather than adopting the article's whole-result SQL string accumulation. diff --git a/docs/design/mssql-export/geometry-implementation-plan.md b/docs/design/mssql-export/geometry-implementation-plan.md new file mode 100644 index 0000000..b90e1fc --- /dev/null +++ b/docs/design/mssql-export/geometry-implementation-plan.md @@ -0,0 +1,53 @@ +# MSSQL Geometry Export Implementation Plan + +> **For agentic workers:** REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development. The user has explicitly authorized this implementation direction; proceed without another approval gate. + +**Goal:** Restore native geometry exactly using the current bounded SQL export pipeline. + +**Architecture:** Add a native geometry codec to the existing binary projections, inline expressions and guarded large-row final projection. Preserve metadata/DDL and all current lifecycle/file guarantees. Extend the existing opt-in integration runner with a Geometry profile. + +**Tech Stack:** .NET Framework 4.7.2, System.Data.SqlClient, PowerShell, existing smoke runner, authorized SQL2025 LocalDB and classic sqlcmd. + +**Spec:** [geometry-design.md](geometry-design.md), extending [design.md](design.md). + +## Global Constraints + +- Reuse the existing .NET Framework 4.7.2 exporter; no new client SQL CLR/geometry package, no public API or UI flow redesign. +- Same source connection/SNAPSHOT and SequentialAccess; no whole-field GetValue/GetSqlBytes, byte-array materialization, WKT normalization, source reread, or extra SELECT per value. +- Preserve existing bounds: binary chunks <=64 KiB, small-row buffer <=256 KiB, value statements <=256 KiB, batches <=4 MiB, metadata-only setup/final INSERT <=2 MiB. No 16 MiB value rejection. +- Retain length/order/completion/failed/consumed guards, single final target INSERT, cancellation, cleanup, and verify-before-publication contracts. +- Geometry must preserve serialized bytes, NULL/empty, SRID and Z/M. Do not simplify, repair, reproject, or convert native values to WKT. +- Geography, aliases/user CLR types and spatial indexes remain preflight failures; identify the object/column or index in the existing safe diagnostics. +- Use only the already authorized `(localdb)\mySQLPunkExportTest_20261012` and generated token-owned C: fixtures. The existing manifest, ownership checks and guarded cleanup remain mandatory. Never touch saved connections, user databases, quarantined ReportExportTests.exe, antivirus or account state. +- Preserve each edited file's encoding/BOM and line endings, unrelated files, and original master. No merge, push, release or large artifact in Git. + +## Review Focus + +1. Built-in geometry has is_assembly_type=1; do not reject it or accidentally allow a user-defined same-name type. +2. Complete/partial/unread continuation prefixes and suffixes must not corrupt binary or double-count bytes. +3. Geometry NULL and empty values must survive both inline and staging without a deserialization error or NULL substitution. +4. Spatial indexes must remain blocked; CLR type IDs in the independent verifier must resolve by user_type_id. +5. A Deserialize failure must preserve protocol poisoning/rollback even if an importing client continues after the error. + +### Task 1: Native geometry codec and real restore acceptance + +**Files:** +- Modify `mySQLPunk/lib/MssqlExport/MssqlExportMetadataReader.cs`, `MssqlValueCodec.cs`, `MssqlLobScriptWriter.cs`; `MssqlDataScriptWriter.cs` or schema writer only if a verified seam requires it. +- Extend existing `tests/SmokeTests.MssqlExport.cs`, `.Values.cs`, `.Lob.cs`, `.Schema.cs` and their registrations in `tests/SmokeTests.cs` as needed. +- Modify `tests/SmokeTests.MssqlExport.Integration.cs`, `tests/Run-MssqlExportIntegration.ps1`; add `tests/MssqlExportFixtures/geometry.sql` if useful. Keep the original unsupported profile rejecting geography/UDT/spatial indexes. +- Controller owns public docs and history; implementer must not edit them. No csproj/dependency changes expected; discuss any required file outside this list first. + +**Interfaces:** Consume existing SourceExpression/IsBinaryProjected/BinaryPrefix/BinarySuffix and IMssqlLargeRowWriter continuation contracts. Produce no new public API. The Geometry opt-in profile must run using the unchanged explicit fixture server/root/ownership guards. + +- [x] **1. Baseline and RED.** Run focused existing metadata/value/LOB smoke groups before editing. Add meaningful native-vs-user-type, source-projection/inline codec, NULL/empty/retained/partial/unread continuation and cancel/bounds regressions using existing strict readers. Run them and record expected failure on current code. +- [x] **2. Implement.** Allow only native sys.geometry; retain geography/user types/index rejection. Reuse binary sequential reads. Inline and final staged expressions call geometry::Deserialize, handling NULL explicitly if SQL requires it. Keep other formats unchanged. +- [x] **3. GREEN.** Run affected metadata/value/LOB/schema smoke groups and confirm bounded read/statement/batch behavior, accurate byte progress, no premature final row on cancellation, and no dependency changes. +- [x] **4. Actual Geometry profile.** Generate small spatial fixtures and a >16 MiB serialized geometry without a whole large client string/byte array (a generated streaming WKB file/parameter is acceptable for fixture setup). Use actual service -> published SQL -> one classic sqlcmd session `-E -b -x -f 65001 -i`. Compare source/target geometry Serialize() raw streaming hashes, catalog types, and independently SRID/Z/M/type/empty; cover multiple geometry fields plus scalars/binary, NULL, points, polygon, collection and curves. Keep source file generation and DB resources recorded separately from exporter allocation. +- [x] **5. Failure and compatibility.** Exercise malformed geometry bytes in a guarded staged row and existing chunk fault variants with an importer that continues, assert zero failing rows and no revived protocol state. Verify actual geography/spatial-index preflight rejection. Update the generic comparator to resolve native type via user_type_id and explicit geometry Serialize; run the affected Core profile once to ensure existing nonspatial/unsupported checks still pass. +- [x] **6. Final verification and commit.** Run one complete existing smoke suite with live opt-in disabled, `build.bat Debug`, `build.bat Release`, and `git diff --check`, strictly sequentially with EXEs closed. Preserve/restore environment filters. Commit only owned product/test files. Report commands, RED/GREEN results, real fixture artifact/hash/size/cleanup, concerns and unverified boundaries in the assigned ignored report. No new UI preview or unrelated large-profile repeat needed. + +### Controller completion + +- [x] Review immutable extension diff against this spec once after implementation. Resolve material findings with one coordinated fix and scoped re-review; avoid duplicate task/final reviews for the same single-task diff. +- [x] Update current support docs/design matrix, history, and verification with actual evidence and residual boundaries. Existing prior v1 historical failures stay recorded. Preserve the original ignored fixture manifest under ruling13. +- [x] Keep feature branch/worktree for user acceptance; no merge/push/release in this task. diff --git a/docs/design/mssql-export/implementation-plan.md b/docs/design/mssql-export/implementation-plan.md new file mode 100644 index 0000000..807d08b --- /dev/null +++ b/docs/design/mssql-export/implementation-plan.md @@ -0,0 +1,425 @@ +# MSSQL 大量資料 SQL 匯出精靈 Implementation Plan + +> **For agentic workers:** REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development (recommended) or superpowers:executing-plans to implement this plan task-by-task. Steps use checkbox (`- [ ]`) syntax for tracking. + +**Goal:** 將 MSSQL 互動式本機 `.sql` 匯出接到專用精靈，以有界記憶體依序匯出一致快照，顯示進度／ETA／Punky，安全取消與發布，並以實際還原驗證資料。 + +**Architecture:** 新增 MSSQL 專用 metadata、SQL writer、worker 與 WinForms 精靈；不重寫其他 provider 的 dump API。worker 擁有獨立 SQL connection／SSH owner，以同一 SNAPSHOT 交易讀取選取表；小列 INSERT、大列在還原 session 的私有暫存表分段組回，再插入目標。檔案串流驗證、原子發布、鏡像在來源交易釋放後執行。 + +**Tech Stack:** 現有 .NET Framework 4.7.2／WinForms／System.Data.SqlClient／MSBuild；既有 ThemeManager、UiKit、PunkyReportLoadingControl、smoke harness；Microsoft sqlcmd 作還原驗收。第一版不增加 NuGet 依賴或換資料庫 driver。 + +**Spec:** [design.md](design.md)，2026-10-12 修訂版；使用者已回覆「照修訂版進入實作計畫（建議）」。 + +**Geometry 延伸：** 下列十個 task 記錄初版實作；使用者後續核准的 native Geometry Serialize／binary／Deserialize 擴充另見 [Geometry 計畫](geometry-implementation-plan.md)，取代下列初版對 geometry 值的拒絕範圍，geography 與 spatial index 仍不支援。 + +**Status:** 使用者於 2026-10-12 核准計畫並選擇「分工實作並逐項覆核」；十個工作單元已整合；合併後完整 smoke、Debug／Release 通過，最終整體覆核與六項修正複核已完成。checkbox 依各項實際證據更新。畫面原型及規格核准不表示 SQL 還原、效能或正式程式已完成。 + +## Global Constraints + +- 範圍是 MSSQL 互動式本機 SQL 邏輯匯出；原生 `.bak`、排程／自動備份改造、其他 provider 重構、暫停／續傳、壓縮／加密均不納入。 +- 持續寫入來源採同一個 SNAPSHOT 交易；不自動變更 DB 設定。`snapshot_isolation_state` 必須是 ON，僅結構模式例外；不降級為 NOLOCK／一般讀取。 +- 每表只開一個 SequentialAccess reader；不得整表 DataTable、OFFSET 分頁、先 COUNT 全表、完整 GetValue/GetString 大欄位或完整 SQL StringBuilder。 +- 整列的小資料緩衝預算最多 256 KiB；binary chunk 至多 64 KiB，文字起始 chunk 至多 16K 個 UTF-16 碼單元；一般資料 statement 生成預算 256 KiB，GO batch 目標不超過 4 MiB。計入實際編碼／hex／轉義 bytes；這些不是單列拒絕上限。 +- 跨 GO 組裝需要同一匯入 session；先完整組裝及驗證，再一次 INSERT 到目標。`sqlcmd -i`、`-b`、`-x`、明確 UTF-8 輸入設定為驗收契約。 +- 原生 geometry／geography、UDT 等 spec 所列特殊能力預檢阻擋受影響選取；一般文字中的 WKT 不解析、縮短或簡化。 +- 進度一般更新約 100–250ms；完成前最多 99%；未知 ETA 顯示「估算中」，不是 0 秒。鏡像未完成不可顯示整體成功。 +- 取消等 worker 清理；失敗不先刪原正式檔。鏡像驗證／發布後才清理保留份數，沿用原有分類規則。 +- Punky 使用現有素材及 BeginExport／EndExport／Muted；自動 loop、原音一次、只有靜音按鈕，不增加播放／暫停。 +- 保留既有 BOM／換行及無關工作，尤其 `mySQLPunk.Desktop/Properties/`；不以 `git add -A` 收檔。重要決策與證據追加既有 `history.md`。 +- 不還原／執行／重命名或重 host `ReportExportTests.exe`，不搬其測試到其他 runner，也不更改防毒。新 MSSQL 測試若受阻，停止並列 pending。 +- 本計畫的建庫、還原與大量資料測試只能在明確隔離 fixture 執行；不使用使用者正式 DB／已儲存帳密。沒有 fixture 就保留 DB 驗收 pending，不能用編譯代替。 + +## Review Focus + +以下五類容易落在主流程外，已各自納入後面測試步驟： + +1. 資料包含換行後的 `GO`、`:r`、`$(name)`、NUL／非 BMP 字元；匯入不能把資料當作工具命令或變數，值須完整還原（Task 4、5、10）。 +2. 兩個 schema 有同名表，或預檢後 drop/recreate；不能依顯示字串選錯表，也不能把新物件當原快照（Task 2、7、8）。 +3. 目標檔在覆寫同意後改變，或取消恰好遇到發布／鏡像；不得刪掉另一工作的檔案或把已發布結果說成未輸出（Task 6、7、10）。 +4. SQL Server 的 metadata 欄位因版本不同而不存在、權限不足、view 定義不可見；須區分不支援與讀取失敗，不能當成「沒有資料／沒有約束」（Task 2、3、10）。 +5. 無 PK、多個大欄位、來源空值、目標 tempdb 定序不同，或還原 client 在錯誤後仍繼續；不得插入半列、改變 bytes 或重複最後 INSERT（Task 4、5、10）。 + +## 檔案與接點 + +以下為計畫中的新檔，尚不存在。新引擎放 `mySQLPunk/lib/MssqlExport/`，namespace `mySQLPunk.lib.MssqlExport`，型別預設 internal；避免擴大公用 IDatabase contract。 + +| 檔案 | 責任／擁有者 | +| --- | --- | +| `MssqlExportModels.cs` | 不可變 request、metadata、progress、結果、限制值及來源測試介面 | +| `MssqlOwnedSession.cs` | owner 與原生 SqlConnection 成對持有，單一 Dispose | +| `MssqlExportMetadataReader.cs` | sys metadata、能力偵測、預檢及 schema fingerprint | +| `MssqlExportSource.cs` | 快照交易、command／reader／取消登記、來源生命週期 | +| `MssqlSchemaScriptWriter.cs` | 原生 DDL、依賴排序、index／constraint／view／comment | +| `MssqlDataScriptWriter.cs` | 逐列／逐欄 reader、有限小列緩衝與分段切換 | +| `MssqlValueCodec.cs` | 型別保真、identifier／scalar／資料 literal 編碼 | +| `MssqlLobScriptWriter.cs` | 大列 staging、chunk、完整性 guard 與最後 INSERT | +| `MssqlExportArtifact.cs` | 工作暫存、incremental hash、重讀驗證、發布／清理 | +| `MssqlExportMirror.cs` | 分段鏡像、驗證後發布及既有 retention | +| `MssqlExportProgressEstimator.cs` | 計量、節流、平滑 ETA／未知狀態 | +| `MssqlSqlExportService.cs` | 階段 orchestration、錯誤／取消與完整結果 | +| `mySQLPunk/MssqlExportWizardForm.cs` | 專用 WinForms 精靈，沿用樣式與 Punky | +| `mySQLPunk/Form1.MssqlExport.cs` | 既有四個互動入口共用的設定捕獲及開窗接點 | +| `tests/SmokeTests.MssqlExport.cs` | 新 MSSQL 純單元、fake-reader、檔案與 UI 狀態測試 | +| `tests/SmokeTests.MssqlExport.Integration.cs` | 明確啟用的隔離 DB／restore／stress 測試，不預設跑 | +| `tests/Run-MssqlExportIntegration.ps1` | 指定測試 server、profile、全新輸出目錄，保存證據 | +| `docs/MSSQL_SQL_EXPORT.md` | 實際支援範圍、操作、還原命令、資源與錯誤處理 | + +確定需要的既有修改：`lib/ConnectionOpenService.cs`、`Form1.cs`、`mySQLPunk.csproj`、`Properties/AssemblyInfo.cs`、`tests/SmokeTests.cs`、`tests/Run-SmokeTests.ps1`。`AssemblyInfo` 只新增測試 friend assembly，不改版本。沿用 `BackupRemoteMirrorService.PruneRemoteBackups`，不呼叫會提早 prune 的 `MirrorBackup`，不必改既有其他 provider 的方法。 + +不需要修改 `my_mssql.cs`、`ConnectionSecurityService.cs`、`IDatabase`、`DatabaseDumpService`、`BackupIntegrityService`、既有報表 writer／動畫素材或 `.bak` 表單。若後續證據要求修改，先說明必要性，不順手重構。 + +### 現有入口（2026-10-12 唯讀定位） + +- `Form1.cs:6634` `DumpSelectedTableSql(bool dataOnly)`：單表右鍵，另有自己的同步 dump。 +- `Form1.cs:6902` `DumpCurrentSelectionSqlWithDialog(bool dataOnlyForTable)`：工具列／群組；view selection 不可誤轉全庫。 +- `Form1.cs:7258` `DumpSelectedDatabaseSqlWithDialog()`：資料庫結構與資料。 +- `Form1.cs:8040` `BackupSelectedDatabaseWithDialog()`：建立備份；此入口捕獲 `BackupMirrorSettings.RemoteDirectory/RetainCount`。 +- `CreateDatabaseBackup` 還被還原前／刪除前備份等流程呼叫，本版不全面替換它。只在上述互動 handler 的 MSSQL 分支轉精靈，其他 caller 保持原流程並在操作文件註明範圍。 +- `ConnectionOpenService.Open(Dictionary, false)` 已持有 raw provider 與可能的 SSH wrapper；在這個邊界傳遞原生連線，不 unwrap private 欄位或另開 connection string。 + +## 共用契約與測試方式 + +### 跨 task 的型別 + +全部在 Models 檔定義，collection 在建構時複製成唯讀；不得讓 UI 在開始後改動 request。 + +- `MssqlObjectKey`：`int ObjectId`、`string Schema`、`string Name`；metadata identity 以 object_id 加原始 schema/name 為準，SQL 引用各段名稱。 +- `MssqlExportMode`：`SchemaAndData, SchemaOnly, DataOnly`。 +- `MssqlMirrorOptions`：`string Directory`、`int RetainCount`；null 表示未啟用。 +- `ExportDestinationStamp`：目標是否存在、正規路徑、檔案 identity／長度／最後寫入時間及 streaming SHA-256；在確認覆寫時捕獲，在發布前重新比對。不將 UI 同意當永久覆寫權。 +- `MssqlExportRequest(Guid jobId, string databaseName, IReadOnlyList tables, MssqlExportMode mode, bool includeViews, string targetPath, ExportDestinationStamp destination, MssqlMirrorOptions mirror)`；驗證非空選取、路徑、data-only 無 views。 +- `MssqlTableDefinition`／`MssqlColumnDefinition`／`MssqlIndexDefinition`／`MssqlConstraintDefinition`／`MssqlViewDefinition`：Task 2 依 spec 定義 native metadata；禁止沿用跨庫映射字串。Column 必有 ordinal、native type、length、precision/scale、collation、nullable、identity、computed、rowversion、comment；Table 含 key、comment、nullable estimated rows/bytes。 +- `MssqlExportCatalog`：tables、snapshot state、readability／capability issues；`MssqlExportIssue`：code、object key、optional column、可呈現原因。 +- `MssqlExportPreflight`：本次選取的schema摘要、snapshot state、`IReadOnlyList Issues`、`bool CanStart`。確認頁使用無長交易的Inspect；開始後必須重查，不能把UI預檢當成凍結來源。 +- `MssqlExportSchema`：凍結的 tables/views/dependencies、canonical definition fingerprint；`MssqlTableRowCount`：object key、`long Rows`。 +- `MssqlExportStage`：Preflight, Schema, Data, Verifying, Publishing, Mirroring, CancelRequested, Completed, Canceled, Failed, LocalCompletedMirrorIncomplete。 +- `MssqlExportProgress`：JobId、Stage、CurrentTable、long 實際 rows/bytes、nullable estimates、完成表數、elapsed、nullable ETA、nullable percent、估計狀態、限長訊息；immutable snapshot。 +- `MssqlExportResult`：Stage、PublishedPath、MirrorPath、row counts、elapsed、Warnings、ResidualPaths；只有已發布才有 PublishedPath。 +- `MssqlExportLimits`：`SmallRowBytes=262144`, `StatementBytes=262144`, `BatchBytes=4194304`, `BinaryChunkBytes=65536`, `TextChunkCodeUnits=16384`；限制算法的值，不用作資料拒絕上限。 + +### 測試命令 + +實作新增測試沿用 `SmokeTests` 的 `Assert`、`AssertEquals`、`AssertThrows`；case 名稱有 `MSSQL export ` 前綴。增加 `InternalsVisibleTo("mySQLPunk.SmokeTests")`，不把整個新引擎改為 public。每 task 使用以下命令，將 group 替換為其指定值，並保存原環境變數： + +```powershell +$savedMssqlSmokeFilter = $env:MYSQLPUNK_SMOKE_FILTER +try { + $env:MYSQLPUNK_SMOKE_FILTER = 'MSSQL export session' + .\tests\Run-SmokeTests.ps1 -Configuration Debug +} finally { + $env:MYSQLPUNK_SMOKE_FILTER = $savedMssqlSmokeFilter +} +``` + +紅燈必須是新 assertion／未實作介面造成，不把 NuGet／compiler／AV 失敗當成正確紅燈；綠燈為匹配 case 全 PASS、runner exit 0，並保留其既有 AutoReleasePolicy 檢查。新增 integration cases 必須在 `MYSQLPUNK_MSSQL_EXPORT_LIVE=1` 且 runner 驗證 fixture 時才註冊，預設 smoke 不連 DB、不產生大檔。 + +所有檔案測試只在以 CreateNew／GUID 建立的專屬新目錄內模擬舊檔；不碰使用者現有輸出。新 runner 只測 MSSQL 新功能，不參照 ReportExportTests source。測試支援物件放該 partial：可控 clock、可注入錯誤的 stream、拒絕 GetValue/GetString／逆序 ordinal 的 DbDataReader、記錄呼叫次序的 fake source。 + +## Task 1：獨立連線 owner 與不可變 request + +**Files:** Create `lib/MssqlExport/MssqlExportModels.cs`, `lib/MssqlExport/MssqlOwnedSession.cs`, `tests/SmokeTests.MssqlExport.cs`; Modify `lib/ConnectionOpenService.cs`, `Properties/AssemblyInfo.cs`, `mySQLPunk.csproj`, `tests/SmokeTests.cs`, `tests/Run-SmokeTests.ps1`。產品路徑皆在 `mySQLPunk/` 下。 + +**Interfaces:** + +```csharp +// ConnectionOpenService：既有 public overloads 保留。 +internal static MssqlOwnedSession OpenMssqlSession(Dictionary connection); +// MssqlOwnedSession : IDisposable +internal MssqlOwnedSession(IDatabase owner, SqlConnection connection); +internal SqlConnection Connection { get; } +public void Dispose(); +``` + +- [x] **1. 寫失敗測試，group `session`。** `Session_DisposesOwnerOnce` 以 fake IDatabase 計次；`Request_CopiesSelectionAndRejectsInvalidMode` 修改輸入 List 後 request 不變；測試 core assertions： + ```csharp + session.Dispose(); session.Dispose(); + Assert(owner.DisposeCount == 1, "SQL provider and SSH lease have one owner"); + Assert(request.Tables.Count == 1, "Request must not retain mutable selection"); + Assert(MssqlExportLimits.SmallRowBytes == 262144, "Row buffer budget"); + ``` +- [x] **2. 跑 session group，確認紅燈。** 編譯／註冊 wiring 與 friend assembly 併入這個可測 deliverable，不另做空 scaffolding task。 +- [x] **3. 實作上述介面與 Models。** `ConnectionOpenResult` 加 internal native SqlConnection property／constructor，在既有 raw provider 建立點傳入；public constructor 保留。factory 使用 `Open(settings,false)`，驗證 MSSQL；失敗 dispose owner。session 不獨立 Dispose 同一 SqlConnection，command／transaction 留來源層管理。不得使用 ConnectionString 另開第二條連線。 +- [x] **4. 跑 session group 為綠，確認既有 connection tests 仍過。** 只用 fake owner／未開啟連線完成生命週期單元測試；真 SSH 另列 Task 10。 +- [x] **5. 檢查 diff 與提交。** 僅 stage 本 task 明確檔案；建議訊息 `feat(mssql-export): own export session and immutable requests`。不 push。 + +## Task 2：原生 metadata、預檢與快照來源 + +**Files:** Create `lib/MssqlExport/MssqlExportMetadataReader.cs`, `lib/MssqlExport/MssqlExportSource.cs`; Extend Models、task 測試、csproj。 + +**Interfaces:** + +```csharp +internal interface IMssqlExportSource : IDisposable { + MssqlExportCatalog ReadCatalog(string databaseName, CancellationToken cancellation); + MssqlExportPreflight Inspect(MssqlExportRequest request, CancellationToken cancellation); + MssqlExportSchema BeginRead(MssqlExportRequest request, CancellationToken cancellation); + MssqlRowReader OpenRows(MssqlTableDefinition table, CancellationToken cancellation); + void CompleteRead(MssqlExportSchema schema, CancellationToken cancellation); +} +// MssqlRowReader : IDisposable owns Reader + command + cancel registration. +internal MssqlRowReader(DbDataReader reader, IDisposable commandOwner); +internal DbDataReader Reader { get; } +// MssqlExportSource : IMssqlExportSource +internal MssqlExportSource(MssqlOwnedSession session); +// MssqlExportMetadataReader +internal MssqlExportCatalog ReadCatalog(SqlConnection connection, string databaseName, CancellationToken cancellation); +internal MssqlExportSchema ReadSchema(SqlConnection connection, SqlTransaction transaction, MssqlExportRequest request, CancellationToken cancellation); +internal IReadOnlyList Validate(MssqlExportSchema schema, MssqlExportRequest request); +``` + +- [x] **1. 寫失敗測試，group `metadata`。** 覆蓋同名不同 schema／名稱含 `]`、spatial 明列、依賴缺漏、snapshot OFF／過渡狀態、estimate 缺失不是 0、typed XML／UDT 拒絕；assert： + ```csharp + Assert(catalog.Tables.Count == 2, "dbo.orders and sales.orders are distinct"); + Assert(catalog.Tables[0].EstimatedRows == null, "Unavailable estimate is unknown"); + Assert(issues.Any(x => x.Code == "UnsupportedSpatial" && x.ColumnName == "shape"), "Name rejected column"); + ``` +- [x] **2. 跑 metadata group 為紅。** metadata mapping 以 fixture rows 測純轉換；SQL查詢本身的證據留 Task 10，不能把 fake 當 live query 驗證。 +- [x] **3. 實作版本／能力感知 sys queries。** metadata 需涵蓋 tables/columns/types/schemas、extended properties、identity/computed/default、rowstore indexes/index_columns、FK/check、view definition/SET/dependencies。用可用 catalog 欄位／版本選查詢，舊版本缺少 ledger／temporal 欄位時不查不存在欄位。看不到 definition 或權限錯誤明確報錯，不以空集合替代。針對 spec 特殊表／型別／索引完整列出阻擋原因。 +- [x] **4. 實作 Source transaction／reader。** 改變 owned connection 的 database；Inspect以無transaction的ReadSchema供確認頁預檢，完畢dispose source，不在使用者思考時持有快照。含資料開始時再確認 snapshot ON，BeginRead 開一個 Snapshot transaction 並重新凍結／驗證 schema；SchemaOnly 不開資料快照。OpenRows 明確欄位、無排序／OFFSET需求、SequentialAccess；command timeout 直接設為 0 並登記 Cancel，dispose 登記先於 command。CompleteRead 核對 object IDs／完整 definition fingerprint，再結束交易；例外／取消 rollback 並 dispose。schema改變中途不得 retry 續跑。 +- [x] **5. 跑 metadata group 為綠並提交。** 加 reader/command/dispose 次序 fake 驗證；實際 snapshot／DDL race 在 Task 10。建議訊息 `feat(mssql-export): collect native metadata and preflight snapshot export`。 + +## Task 3：原生結構腳本與依賴順序 + +**Files:** Create `lib/MssqlExport/MssqlSchemaScriptWriter.cs`; Extend Models／測試／csproj。 + +**Interfaces:** + +```csharp +internal void WriteBeforeData(MssqlExportSchema schema, MssqlExportRequest request, TextWriter output, CancellationToken cancellation); +internal void WriteAfterData(MssqlExportSchema schema, MssqlExportRequest request, TextWriter output, CancellationToken cancellation); +``` + +- [x] **1. 寫失敗測試，group `schema`。** metadata fixture 含 native varchar／decimal(38,10)／datetimeoffset(7)、identity、computed persisted、rowversion、複合降冪索引／INCLUDE／filter、FK/CHECK trust 狀態、comment、多層 views。assert： + ```csharp + AssertContains(sql, "decimal(38,10)", "Keep native precision"); + AssertNotContains(sql, "USE [source_db]", "Restore target is chosen by caller"); + Assert(sql.IndexOf("CREATE TABLE") < sql.IndexOf("FOREIGN KEY"), "Create referenced tables first"); + AssertContains(sql, "GO", "View DDL has its own batch"); + ``` +- [x] **2. 跑 schema group 為紅。** 加 circular FK、未包含 view/function/sequence／trigger、不可見 definition 的失敗案例；不能生成成功卻遺漏物件。 +- [x] **3. 實作前後兩階段 writer。** 前段 schema/table、原生欄位與 default/computed；後段 keys/indexes、FK/CHECK、comments、依賴排序 views。自參照／循环 FK 留資料後建立。資料來源不含 computed／rowversion 值，摘要說明 rowversion 重生。僅資料兩段皆不建立物件；僅結構仍產生完整兩段。型別／identifier 不經 DatabaseDumpService mapper。 +- [x] **4. 明定 restore 語意與 identity 處理。** 結構模式預期空白／無衝突目標，存在物件應明確失敗，不產生 DROP DATABASE/TABLE；DataOnly 要求相容既有表。identity 的 seed/increment 與匯入值，以及空表、負 increment 的後續新增須能供 Task 4 資料段及 Task 10 驗證；依設計不承諾來源配號間隙或完整 last_value 歷史，避免「已插最大值」被誤當全部規則。 +- [x] **5. 跑 schema group 為綠並提交。** SQL tokens 測試不是還原證據；native execution 留 Task 10。建議訊息 `feat(mssql-export): script native schema and ordered dependencies`。 + +## Task 4：一般列串流、型別及批次預算 + +**Files:** Create `lib/MssqlExport/MssqlDataScriptWriter.cs`, `lib/MssqlExport/MssqlValueCodec.cs`; Extend source projection、測試、csproj。 + +**Interfaces:** + +```csharp +internal static string QuoteIdentifier(string identifier); // MssqlValueCodec +internal static string FormatScalar(object value, MssqlColumnDefinition column); +// Writer receives the already-open reader; does not query source again. +internal long WriteTable(DbDataReader reader, MssqlTableDefinition table, TextWriter output, + Action onRowsAndSourceBytes, CancellationToken cancellation); +``` + +- [x] **1. 寫失敗測試，group `values`。** decimal 38 不走 CLR decimal、offset與7位小數、GUID、money、float roundtrip、NULL/empty、Unicode／換行／NUL／`GO`／`$(name)`。以 `SqlDecimal` fixture、InvariantCulture、受限 DbDataReader 測： + ```csharp + AssertEquals("[a]]b]", MssqlValueCodec.QuoteIdentifier("a]b"), "Quote each identifier"); + AssertContains(decimalSql, "1234567890123456789012345678.1234567890", "Keep all 38 digits"); + Assert(maxObservedStatementBytes <= 262144, "Bound statement bytes"); + Assert(maxObservedBatchBytes <= 4194304, "Bound data batch bytes"); + ``` +- [x] **2. 跑 values group 為紅。** fake reader 對大值 GetString/GetValue、回頭讀 ordinal 立即 throw；偵測 row buffer 最大占用。不只測輸出相等，也測讀取方式。 +- [x] **3. 實作 scalar codec 與串流 projection。** LOB 用 GetBytes／GetChars 小段；對 varchar/text 投影原始 varbinary 保留 code-page bytes，回復時指定來源 collation；Unicode 使用不丟失 UTF-16 code units 的編碼。文字／binary資料 literal使用typed hex expression，避免行首 GO／sqlcmd directive／控制字元穿透；text chunk 不切 surrogate pair。decimal 使用 SqlDecimal；不依當前文化格式化日期／數字。untyped XML 需串流投影及明確 XML 轉型／還原驗證，不能不小心 GetValue 整欄。 +- [x] **4. 實作有限小列 INSERT 與計量。** 使用 source metadata 排除 computed／rowversion，identity 適當 ON/OFF。整列緩衝／encoded statement 接近預算時移交 Task 5，不回查來源。Task 5 完成前大列入口只能明確標為尚未實作並保持 UI 未接線，不能把暫時行為當正式支援上限。空表產生零筆INSERT；若無可插入欄位，以每來源列一個常數投影讀取並逐列INSERT DEFAULT VALUES，不把rowversion當持久值。 +- [x] **5. 跑 values group 為綠，完成小資料 reader cases 後提交。** 建議訊息 `feat(mssql-export): stream typed row values within batch budgets`。 + +## Task 5：大欄位 staging、分段與失敗保護 + +**Files:** Create `lib/MssqlExport/MssqlLobScriptWriter.cs`; Extend Task 4 writer、測試、csproj。 + +**Interfaces:** + +```csharp +internal void BeginRow(Guid jobId, long rowOrdinal, MssqlTableDefinition table, TextWriter output); +internal void WriteScalar(int columnOrdinal, string scalarExpression, TextWriter output); +internal void WriteChunk(int columnOrdinal, long chunkIndex, byte[] encodedBytes, int count, TextWriter output); +internal void CompleteColumn(int columnOrdinal, bool isNull, long chunkCount, long expectedBytes, TextWriter output); +internal void CommitRow(TextWriter output); +``` + +`encodedBytes` 是 codec 指定的原生 binary／varchar bytes／UTF-16 bytes；chunk writer 不保留呼叫者 buffer。型別、collation、欄位列表由 BeginRow 固定。小型非LOB值透過codec生成的scalarExpression交WriteScalar，一個受guard的batch設定typed欄位並標記完成；不能接受UI／外部任意SQL。先前small-row buffer內的scalar與文字／binary分別移交對應方法，無第二次來源讀取。CompleteColumn用於分段欄位，核對非NULL值的chunk／bytes並標記完成；NULL狀態與空值保持區別。 + +- [x] **1. 寫失敗測試，group `lob`。** 單元用惰性 reader 供 64 MiB binary、超過 16 MiB Unicode 與多欄位無 PK；不在 fixture 自己先配一整塊大字串。assert： + ```csharp + Assert(maxRawBinaryChunk <= 65536, "Bound raw binary chunk"); + Assert(maxTextCodeUnits <= 16384, "Bound UTF-16 chunk"); + Assert(maxSmallRowBuffer <= 262144, "Switch instead of growing row buffer"); + Assert(finalTargetInserts == 1, "Only insert the complete row"); + Assert(sourceReadPasses == 1, "No PK or row reread required"); + ``` +- [x] **2. 跑 lob group 為紅。** 測 NULL 與空值、尾空白、surrogate boundary、bytes 膨脹、small→large切換、text/ntext/image、untyped XML、cancel 中途。測試生成 guard 結構；SQL engine 實際拒絕缺段由 Task 10 驗證。 +- [x] **3. 實作同 session 私有 staging protocol。** 使用工作專屬 `#temp` 名稱；每列紀錄 job/row/column/chunk index/received bytes/expected totals/null/completed/failed/consumed。非NULL先空 max 值，`.WRITE(chunk,NULL,0)` 追加；每個 batch 驗證上一狀態，以短 transaction 原子更新值與計數，錯誤 rollback 後標記失敗再拋出。NULL 不用 `.WRITE(NULL,...)` 代替。 +- [x] **4. 實作最後 guard 與目標插入。** 同一短 transaction 驗證所有欄位完成、DATALENGTH、chunk數、順序及未 consumed，才 INSERT 並 consumed。staging缺失、缺段、重複、錯序、轉型失敗均無目標半列；不要用 TRY_CONVERT 的 NULL 吞錯。TRY/CATCH 每個 batch 完整閉合，變數不跨 GO；清除已插列內容但保留必要的消費狀態以拒絕重送最後批次。一般多列還原不承諾全檔單一 transaction。 +- [x] **5. 跑 lob／values groups 為綠並提交。** 檢查輸出每個 statement／batch實際UTF-8大小；不以擴高16MiB門檻充當分段。建議訊息 `feat(mssql-export): reconstruct large values with guarded SQL chunks`。 + +## Task 6：有界檔案驗證、安全發布與鏡像 + +**Files:** Create `lib/MssqlExport/MssqlExportArtifact.cs`, `lib/MssqlExport/MssqlExportMirror.cs`; Extend Models、測試、csproj。 + +**Interfaces:** + +```csharp +// MssqlExportArtifact : IDisposable owns only its job temp. +internal static ExportDestinationStamp CaptureDestination(string path, CancellationToken cancellation); +internal static MssqlExportArtifact Create(MssqlExportRequest request); +internal Stream Output { get; } +internal MssqlArtifactReceipt Complete(IReadOnlyList rows, CancellationToken cancellation); +internal void Verify(MssqlArtifactReceipt receipt, Action progress, CancellationToken cancellation); +internal string Publish(ExportDestinationStamp expectedDestination, CancellationToken cancellation); +// MssqlExportMirror +internal string CopyVerified(string publishedPath, MssqlArtifactReceipt receipt, MssqlMirrorOptions options, + Action progress, CancellationToken cancellation); +``` + +`MssqlArtifactReceipt` 定義 job ID、temp/published identity、long bytes、SHA-256、table/row totals、完成標記。完成標記不包含自己的 hash；receipt 保存包含標記在內的最終 hash。 + +- [x] **1. 寫失敗測試，group `artifact`。** 以全新 fixture 建「舊版」與「他人暫存檔」，注入 truncate／disk-full／rename失敗／cancel／同意後換檔： + ```csharp + Assert(File.ReadAllText(oldFixturePath) == "old fixture", "Failed publish keeps existing file"); + Assert(File.Exists(otherJobTemp), "Cleanup owns only this job temp"); + Assert(pruneCalls == 0, "Failed mirror never prunes old backups"); + Assert(streamReadMax <= 81920, "Verification uses fixed buffers"); + ``` + ReadAllText 只可用於這幾個 bytes 的測試哨兵檔；產品 verifier 及大檔測試禁用。 +- [x] **2. 跑 artifact group 為紅。** 增加同目錄mirror no-op、mirror target已存在不覆蓋、cleanup失敗殘留路徑、發布後cancel結果仍有PublishedPath。 +- [x] **3. 實作 temp／hash／驗證。** 目標同目錄 CreateNew，名稱為隨機 `.mysqlpunk-export-.partial`，不帶 `_backup_` 等 retention分類片段；UTF-8無BOM writer、incremental SHA-256與long計量、80KiB重讀。完成標記只在所有來源／DDL寫出成功才寫；關閉flush成功後才能verify。 +- [x] **4. 實作發布邊界。** 正規路徑同一工作／process互斥，最後取消與destination stamp核對後進短不可取消區；不存在用不可覆寫Move，存在用File.Replace，保留可恢復位置。目標變更、鎖定、FS不支援replace均fail closed，不退回delete-then-move。以 Windows 實測核對外部競態；不得把process內互斥宣稱成所有程序的檔案鎖。清理只刪本工作明確owned暫存；已發布／可恢復檔不誤刪。 +- [x] **5. 實作鏡像。** 串流copy到目的地獨有partial，驗證同receipt hash與長度，發布唯一正式名後才呼叫 `BackupRemoteMirrorService.PruneRemoteBackups(directory, retainCount)`。保留原分類規則，同時不可讓其他活躍工作partial被分類清掉；不呼叫舊MirrorBackup。錯誤／取消回worker以LocalCompletedMirrorIncomplete呈現。 +- [x] **6. 跑 artifact group 為綠並提交。** 建議訊息 `feat(mssql-export): verify and publish exports and mirrors safely`。真UNC/權限/磁碟滿與發布競態仍列Task10。 + +## Task 7：worker、進度／ETA 與取消 orchestration + +**Files:** Create `lib/MssqlExport/MssqlSqlExportService.cs`, `lib/MssqlExport/MssqlExportProgressEstimator.cs`; Extend Models、測試、csproj。 + +**Interfaces:** + +```csharp +internal MssqlSqlExportService(Func sourceFactory); +internal MssqlExportCatalog LoadCatalog(string databaseName, CancellationToken cancellation); +internal MssqlExportPreflight Preflight(MssqlExportRequest request, CancellationToken cancellation); +internal MssqlExportResult Export(MssqlExportRequest request, + IProgress progress, CancellationToken cancellation); +// MssqlExportProgressEstimator: caller supplies elapsed time, tests use a fake clock. +internal MssqlExportProgressEstimator(Guid jobId, MssqlExportSchema schema, bool includeMirror); +internal MssqlExportProgress Sample(MssqlExportStage stage, MssqlObjectKey currentTable, + long currentTableRows, long currentTableSourceBytes, long writtenBytes, TimeSpan elapsed, bool force); +``` + +`currentTableRows/currentTableSourceBytes`為目前表累計，`writtenBytes`為全檔累計；Sample未達節流時可回null，force必須回snapshot。estimator從schema取得nullable估計，切表時記錄已完成實際數；不可自行查DB。LoadCatalog／Preflight都各開短生命週期source並dispose。 + +- [x] **1. 寫失敗測試，group `worker`。** fake source記錄open/begin/read/complete/dispose與檔案步驟順序；估計用fake clock，不sleep。assert： + ```csharp + Assert(calls.IndexOf("source.dispose") < calls.IndexOf("artifact.verify"), "Release snapshot before local work"); + Assert(samples.Where(x => x.Stage != MssqlExportStage.Completed).All(x => x.Percent == null || x.Percent <= 99), "No early success"); + Assert(initial.Eta == null && stalled.Eta == null, "No fabricated countdown"); + Assert(result.PublishedPath != null && result.Stage == MssqlExportStage.LocalCompletedMirrorIncomplete, "Keep local outcome"); + ``` +- [x] **2. 跑 worker group 為紅。** 讀取／寫入／verify／mirror取消，發布區取消，舊operation回報、DB斷線、schema指紋改變、實際rows超estimate、不同列寬、缺失估計都覆蓋。 +- [x] **3. 實作階段順序。** 新source重做Inspect→BeginRead→DDL前段→逐表data→DDL後段／schema核對→CompleteRead與source Dispose→artifact Complete/Verify/Publish→選用mirror→結果。預檢失敗不建立partial；UI先前Preflight不跳過開始時的重查。SchemaOnly不讀rows；DataOnly不寫DDL。失敗停止整個工作，不跳表或重連續跑；finally reader/command/transaction/owner及本工作temp順序清理。 +- [x] **4. 實作進度。** 預設200ms節流，階段／表／terminal強制回報；使用最近10秒樣本，至少3個樣本且至少2秒才估ETA。連續5秒無實際進展時ETA回null並顯示等待。估計remaining work按表估計bytes及實測row-width調整，不按表數等權；估計失準或劇烈速率變化可退回未知。局部verify/mirror以實際bytes計算，整體百分比單調且成功前≤99，不重置0；terminal狀態立即回報。UI log採上限200條。 +- [x] **5. 跑worker group為綠並提交。** constructor default factory只建立自己的session；不讀全域backup設定、不觸碰共享provider。建議訊息 `feat(mssql-export): orchestrate cancellable export with measured progress`。 + +## Task 8：正式精靈與 Punky + +**Files:** Create `mySQLPunk/MssqlExportWizardForm.cs`; Extend 測試、csproj。重用現有 `PunkyReportLoadingControl`、ThemeManager、UiKit，不修改報表精靈。 + +**Interfaces:** + +```csharp +// MssqlExportWizardOptions (Models): mode, initial schema/name selections, +// includeViews default, default file name, optional mirror; immutable. +internal MssqlExportWizardForm(Dictionary connection, string databaseName, + MssqlExportWizardOptions options); +internal MssqlExportResult Result { get; } +internal MssqlExportWizardForm(Dictionary connection, string databaseName, + MssqlExportWizardOptions options, MssqlSqlExportService service); +``` + +- [x] **1. 寫失敗測試，group `wizard`。** 在STA透過fake service，不連DB；預設全選／單表預選、搜尋comment保持勾選、同名表不同schema、零選取／空路徑阻擋、mode切換views、snapshot不足、reset舊結果、關閉等cleanup。assert： + ```csharp + Assert(selectedObjectIds.SequenceEqual(expectedObjectIds), "Search keeps object identity"); + Assert(!startButton.Enabled, "Failed preflight blocks start"); + Assert(!form.IsDisposed, "Close waits for worker cleanup"); + Assert(soundButtons.Count == 1, "Mute only; no play or pause"); + ``` +- [x] **2. 跑wizard group為紅。** UI自動化以控件Name/AccessibleName與注入service驅動；不靠像素坐標斷言全部版面，也不把fake進度當真匯出。 +- [x] **3. 建立精靈。** 參照已驗收index.html：選表→輸出設定→確認／預檢→進度／結果；table list顯示schema/name/comment/nullable estimates。Windows SaveFileDialog、背景捕獲destination stamp、庫級結構views預設勾選；確認頁明列SNAPSHOT與logical export範圍／mirror。DPI、working-area clamp、可捲內容及固定footer沿用現有UI模式，中英文沿用ReportText.Get。 +- [x] **4. 接worker生命週期。** constructor只複製scalar connection settings並剔除pdo/isConnect/runtime state；Shown背景LoadCatalog，確認頁背景Preflight，改選項使預檢失效，通過才啟用開始；開始時freeze request。UI context建立Progress，Task.Run執行service；只接受同JobId且視窗存活的進度。UI timer只更新elapsed，距離最近實際進展5秒後將ETA顯示為等待／未知，不在阻塞reader時繼續假倒數，也不虛增bytes或百分比。執行禁改選項；取消／關閉等待worker，terminal才EndExport及disposeCTS。開始BeginExport一次、只有Muted控制，取消靜音不重播原音。 +- [x] **5. 跑wizard group為綠，做本機WinForms人工檢視再提交。** light/dark、最小可用工作區與100%/150%/200%DPI、長table/comment、未知ETA、cancel與關閉按鈕可達。實際聽到聲音另記證據；未可觀察不聲稱通過。建議訊息 `feat(mssql-export): add native export wizard and Punky progress`。 + +## Task 9：四個互動入口接線與 provider 回歸 + +**Files:** Create `mySQLPunk/Form1.MssqlExport.cs`; Modify `mySQLPunk/Form1.cs` 四個handler、csproj；Extend測試。 + +**Interfaces:** + +```csharp +// Form1 partial; TreeDatabaseTarget is the existing private nested type. +private void OpenMssqlExportWizard(TreeDatabaseTarget target, + MssqlExportMode mode, string initialSchema, string initialTable, bool backupEntry); +``` + +- [x] **1. 寫失敗測試，group `routing`。** 用mode/options捕獲 seam驗證四個handler在MSSQL時只開一次wizard、單表只預選指定schema/name、DataOnly保留、backup才帶mirror，view單物件不變成全庫；MySQL/SQLite/PostgreSQL仍走既有路徑。 +- [x] **2. 跑routing group為紅。** 測到真正handler分流／options，不只搜尋source中的方法名字。 +- [x] **3. 在既有SaveFileDialog／同步dump之前分流。** 使用target.ProviderName識別MSSQL，不能 `target.Database is my_mssql`，SSH會包wrapper。target.ConnectionInfo剔除runtime物件後交form，backup入口捕獲Directory/RetainCount並使用_backup_檔名；其餘export傳null mirror。指定單表按metadata解析原始schema/name，object_id由catalog確認，不從顯示comment反推。 +- [x] **4. 收斂完成訊息。** handler根據wizard.Result更新main status，不無條件再跳「SQL已匯出」；使用者取消設定時無成功訊息。本機完成但mirror未完成顯示精確狀態。維持view-only舊入口、非互動backup caller及其他provider不變。 +- [x] **5. 跑routing、wizard與既有dump/native-backup/connection groups，再Debug/Release build後提交。** 建議訊息 `feat(mssql-export): route MSSQL interactive SQL exports through wizard`。 + +## Task 10：隔離 DB 還原、大資料與交付證據 + +**Files:** Create `tests/SmokeTests.MssqlExport.Integration.cs`, `tests/Run-MssqlExportIntegration.ps1`, `docs/MSSQL_SQL_EXPORT.md`, `docs/design/mssql-export/verification.md`; Modify smoke registration/runner source list、必要的純fixture忽略項、現有history。不加入raw SQL大檔或credentials。 + +**Interfaces:** + +```powershell +# Runner defaults to instructions/no execution unless -Run is explicit. +param([switch]$Run, [string]$Server, + [ValidateSet('Core','Large','Concurrency','Ssh')][string]$Profile = 'Core', + [string]$OutputDirectory) +``` + +runner要求明確server及全新／空output目錄，Integrated Security為第一版fixture認證。若測試環境只有其他認證，執行時補非command-line秘密輸入方式；不讀使用者已存連線或把密碼寫入args/log。不自動下載容器、安裝／啟動DB。明確記錄server與sqlcmd版本／architecture／資料庫collation與profile，敏感資訊不輸出。 + +- [x] **1. 寫與執行Core restore失敗測試。** 真實產物由新service生成，透過一次 `sqlcmd -S -d -E -b -x -f 65001 -i ` 還原；不使用mySQLPunk舊script splitter。來源／目標DB名必須是此次runner產生的 `mysqlpunk_export__src/_dst`，清理前核對server、完整名稱及ownership token，絕不DROP傳入任意DB。runner預設只介紹用法、不建庫；-Run後只操作自己建的fixture。 +- [x] **2. Core矩陣。** 兩個schema同名表、Unicode／`]`識別字；SQL型別精度／offset／collation／NULL／empty／tailspaces；keys/constraints/indexes/views/comment/identity正負increment／空表與新增；computed/rowversion預期；geometry／geography和其他unsupported預檢拒絕。對reader不可見definition／snapshot off／版本缺欄位也保留案例。使用catalog比較及streaming value hash，不只比行數。 +- [x] **3. 大欄位restore矩陣。** 至少64MiBbinary、超過16MiBUnicode、多LOB/無PK/legacy text/ntext/image/untyped XML、不同tempdb/source collation及長WKT普通文字。assert還原DATALENGTH／streamingSHA-256相同；`GO`、`:r`、`$(name)`、NUL／surrogate不被工具展開。對fixture產物副本移除／重複／調換chunk，故意conversion error與錯誤後繼續的client，assert失敗列沒有新增／沒有NULL替代。測重複最後batch不再插入；成功先前列可存在，不把整份還原稱all-or-nothing。 +- [x] **4. Large profile。** 持續產生而非一次配置大buffer，輸出檔實際 `Length > 2147483648`；10倍來源對照記錄peak working set/private bytes、rows、source/export bytes、elapsed與versions。出口／verify／mirror各自sampling，確認無隨總量線性成長；若峰值差異異常需分析，不只用含糊「跑完」。>1GiB單欄／接近型別理論上限另列optional stress，未跑保留pending，不能阻擋使用者已同意的v1範圍。 +- [x] **5. Concurrency profile。** 第二連線以一個transaction同步修改父子表，驗證export同一快照；途中DDL變更須失敗；等待command取消／連線斷線、IO中斷／磁碟滿或受控故障stream、verify/mirror取消、Windows覆寫競態均核對舊檔與結果。來源交易必須在file verify前已釋放。設定snapshot只限自有fixture。 +- [ ] **6. Ssh與人工UI。** 明確測試SSH端點提供後驗證同job tunnel成功／取消／失敗釋放且main connection仍可用；缺環境保持pending。Native UI實際大資料作業、聲音一次／mute、長時間回應性另外列結果，不用HTML模擬截圖替代。 +- [x] **7. 正式本機檢查。** 所有定向cases通過後，保存／清除既有filter跑一次完整既有smoke、`build.bat Debug`、`build.bat Release`、`git diff --check`。檢查release既有CodexVerify／test EXE排除，fixture只在ignored全新目錄；必要時增filename正反案例，不為本功能執行packaging／release。任何AV攔截均停止該測試管道並記pending。 +- [x] **8. 文件與證據。** 操作文件寫實際已測sqlcmd命令、SNAPSHOT前提、支援／不支援能力、部分restore語意／資源成本、GUI匯入未驗證範圍、結果與取消文案。verification逐項填PASS/FAIL/PENDING、命令、版本、產物hash、記憶體記錄與未驗證理由；保留大檔local-only。history追加精準結果。 +- [x] **9. 最終review及提交。** 依選定執行方式進行一次整體review，修正實質問題後只重跑受影響檢查；不循環產生無實質發現的review。建議訊息 `test(mssql-export): verify isolated restore and large export behavior`。未完成的DB／SSH／UI驗收保持明列，不宣稱功能全數驗收；merge/push/tag/release需另外授權。 + +已完成的隔離還原與合併後檢查見 [verification.md](verification.md)。第 6 項的 SSH、實際長作業 UI／聲音及實體 DPI 仍待環境／人工驗收；第 9 項已完成整體覆核及單次修正複核（4 Important、2 Minor 均已修正）；最後新增提示的深／淺色原生 fixture 畫面檢視已於 2026-10-12 重驗通過，包含統計、停滯 ETA、取消結果及等待清理後關閉；模擬服務不代表真實長作業 UI 驗收。 + +### 驗收命令（實際執行結果見 verification.md） + +```powershell +.\build.bat Debug +.\build.bat Release +.\tests\Run-SmokeTests.ps1 -Configuration Release +# <...> 是執行時明確選定的隔離環境與全新目錄，不能原樣執行。 +.\tests\Run-MssqlExportIntegration.ps1 -Run -Server '' -Profile Core -OutputDirectory '' +.\tests\Run-MssqlExportIntegration.ps1 -Run -Server '' -Profile Large -OutputDirectory '' +.\tests\Run-MssqlExportIntegration.ps1 -Run -Server '' -Profile Concurrency -OutputDirectory '' +git diff --check +``` + +## 執行順序與 handoff + +- 主線順序：1→2→3→4→5→6→7→8→9→10。每task測試紅→綠與diffreview後形成可追溯提交；尚未接線時不對外宣稱完成。 +- 可並行部分：Task 6在Models／介面固定後，可與3–5的writer工作並行；不要同時修改Models、csproj、同一測試partial。其他task按依賴執行。Task10的fixture準備可以提前，但真restore證據必須使用最終writer。 +- 建議分工實作：此功能的錯誤可能造成無法還原的備份，適合每個可測交付獨立review；主agent管理shared介面與最終整合。使用者也可選擇主agent依序實作、完成後整體review。 +- 執行前檢查當時git status與既有worktree附件，依using-git-worktrees流程選隔離checkout；必須攜帶本spec／plan／既有新prototype檔，不能在新worktree漏掉未提交設計。不擅自stash/reset或帶入無關Desktop Properties。 +- 計畫審閱後才開始產品實作。本次只寫規格狀態、plan、README導航與history；沒有build、EXE、DB、commit或push。 + +## 自查（計畫撰寫時） + +- Spec 2–4／Punky對應Task8–9；spec5對應Task1–2、7；spec6對應Task2–5、10；spec7對應Task7；spec8–9對應Task6–7；spec10證據對應Task10。 +- 四條互動入口包含獨立的單表handler；view-only、非互動backup、其他provider保持原路。覆寫同意／partial ownership、mirror驗證後prune均有獨立測試。 +- Review Focus五項均已分配owner與測試。接口一致採Models／IMssqlExportSource／MssqlRowReader／receipt／request；沒有混用原shared IDatabase當worker connection。 +- 編譯與fake cases、native UI、實際SQLrestore／大資料／持續寫入／SSH證據分開。未執行步驟沒有勾選，理論型別上限沒有宣稱已驗證。 diff --git a/docs/design/mssql-export/implementation-rulings.md b/docs/design/mssql-export/implementation-rulings.md new file mode 100644 index 0000000..a1b27a2 --- /dev/null +++ b/docs/design/mssql-export/implementation-rulings.md @@ -0,0 +1,35 @@ +# MSSQL export implementation rulings + +These decisions resolve implementation details within the approved design. Listed in decision order, including costs and validation boundaries. Verification outcomes are recorded separately in verification.md. + +1. Use a CLI worktree for this repository after the native tool failed resolving the explicitly supplied repository commit — the app tool targeted a different repository and could not create this checkout — cost if wrong: worktree attachment/cleanup must be managed through Git, not the app artifact catalog. Main checkout and untracked user files stay unchanged; .git/info/exclude ignores .worktrees/. + +2. Interpret stale-overwrite protection as serialized participating jobs plus a final full consent-stamp comparison; do not claim atomic compare-and-swap against uncooperative external writers — File.Replace has no expected-identity parameter, and holding a deny-delete handle also prevents replacement — cost if wrong: an external rename in the final check/replace interval may displace a newer version, which must remain recoverable rather than silently deleted. Use a unique recovery path for every overwrite and surface ambiguous/partial replacement outcomes; recommend a new filename where exclusive destination ownership is unavailable. Document this limit and test the race deterministically, not just via random stress. + +3. On partial File.Replace failure, preserving verified prior bytes at an explicitly reported recovery pathname satisfies recoverability; do not universally assert preservation at the original pathname — Windows documents partial name changes on replacement errors — cost if wrong: a user may need manual recovery from the reported path. Never blindly roll back over a newly appeared destination, and preserve uncertain survivors. + +4. Preserve varchar/text source bytes via direct binary assignment into an exact-source-collated target/staging column, instead of CONVERT(varchar(max), hex) followed by COLLATE — the dedicated SQL Server probe proved the proposed cast loses bytes under differing code pages — cost if wrong: collation-specific roundtrips could corrupt values; retain byte/hash fixtures across code pages and UTF8. Task 4/5 briefs carry the measured correction. + +5. Apply the approved design's identity contract (seed/increment, preserved values, collision-free subsequent inserts), not exact restoration of the source allocation counter — Task 3 plan wording mentioned last_value more strongly than the design, and native counter allocation is mutable outside the data snapshot — cost if wrong: post-restore generated values can differ from source allocation gaps/reseed history. Task 3 must verify fresh empty and positive/negative arbitrary-order restoration plus safe next inserts; do not add last_value to the definition fingerprint. Native probe in fixture-prep-report.md passed both directions. Final exporter/sqlcmd evidence remains required. + +6. Run Task4 values concurrently with Task3 schema in a separate C: worktree after Task2 metadata contracts passed review — their code responsibilities are independent on the frozen native models, and isolation avoids shared mutable files — cost if wrong: additive registration/model conflicts or cross-writer assumptions require integration fixes and tests before Task7. Task4 owns codec/source projection, Task3 owns DDL and preflight gates; final combined review remains required. + +7. Explicitly block legacy ANSI_PADDING/ANSI_NULLS metadata that supported target engines cannot recreate faithfully, after primary-source confirmation — silently emitting ineffective SET options would claim fidelity not provided — cost if wrong: some otherwise exportable legacy tables are conservatively refused until a tested compatibility path is added. Only explicit facts and affected native types trigger the gate. + +8. Treat rare over-budget legal identifier lists as bounded metadata, using a separate narrowly scoped metadata-only statement path while preserving value-bearing statement and row-buffer budgets — SQL Server's maximum column count and identifier length impose a fixed bound independent of row payload, whereas rejecting legal tables would turn algorithm budgets into data limits — cost if wrong: an unusually wide-name table can consume more bounded metadata memory or reveal importer statement limits; add a pathological-name regression and keep the 4 MiB batch ceiling. + +9. After Task7 passes review, run Task10's engine/service live acceptance in an isolated C: worktree alongside Tasks8–9 native UI/routing — the final writers/artifact/service contracts are then stable, and DB fixtures and UI files have separate ownership — cost if wrong: shared test registration or a discovered engine defect may require an additive merge or coordinated fix; final combined builds/smoke and relevant reruns remain mandatory after integration. Task10 must not claim unimplemented UI/routing acceptance. + +10. Present ETA as the currently measurable phase's remaining time, and leave unmeasured future phases explicit — source-read throughput cannot truthfully predict future disk verification or remote mirror throughput — cost if wrong: users get a less complete total-finish forecast; label it 本階段預估 / Phase ETA and never imply whole-job completion from that value. Overall estimated percent still includes all required phases and never reaches100 before complete. + +11. Permit an isolated, official portable Microsoft sqlcmd client under the authorized C: fixture, without installing or replacing the system client — classic15.0.1300.359 actually truncates a valid statement crossing byte2147483648 — cost if wrong: client-specific import compatibility must be documented and validated; a newer downloaded client is checked for official release hash and Authenticode, with no AV bypass. + +12. For Go sqlcmd, validate native UTF-8 input and use its supported equivalent command instead of requiring the classic-only -f65001 switch — the binding contract is UTF-8, variable substitution disabled, fail-on-error and one session, not an unsupported flag spelling — cost if wrong: non-ASCII metadata or client parsing could differ; require native Unicode/parser/temp-session probes and full large restore before acceptance. Preserve classic client FAIL evidence. + +13. Retain this plan's ignored SDD workspace as local operational evidence until the authorized fixture is retired — the acceptance runner intentionally consumes its fixture-environment.json authorization manifest, so deleting the workspace would break repeatable isolated verification — cost if wrong: untracked local reports remain and require later explicit cleanup; none are published, no raw bulk artifact is added to Git. Public rulings and final verification still live in Git. + +## Geometry extension rulings (2026-10-12) + +1. Support native sys.geometry only, keeping geography and spatial indexes blocked in all modes — the requested extension concerns Geometry value serialization and no spatial-index DDL exists — cost if wrong: spatial-indexed tables require a later index extension rather than receiving incomplete scripts. +2. Use one consolidated implementation unit and one independent full extension review as both task and final gate — codec, continuation and restore acceptance form one coherent change — cost if wrong: no duplicate broad pass; the reviewer must receive the complete extension and actual evidence. +3. Retain the ignored extension ledger/report alongside the authorized fixture evidence under original ruling13 — preserves repeatable acceptance and continuity — cost if wrong: local reports remain for later explicit cleanup; bulk data stays outside Git. diff --git a/docs/design/mssql-export/index.html b/docs/design/mssql-export/index.html new file mode 100644 index 0000000..f490c16 --- /dev/null +++ b/docs/design/mssql-export/index.html @@ -0,0 +1,161 @@ + + + + + +mySQLPunk · MSSQL 匯出精靈預覽 + + + + +
+
mySQLPunk/互動原型 · 模擬資料
+ + +
+
+
匯出資料庫 · SQL Server
mySQLPunk
+
DATABASE EXPORT

資料庫匯出精靈

demo_shop MSSQL
DEMO-SQL / SQL Server
+
1選擇資料表
2輸出設定
3確認匯出
4執行與結果
+
+ + + +
+
資料匯出中一致性快照

正在匯出資料

+
43%
整體進度 · 估計
+
目前資料表
+
已經過
04:32
目前速度
18,420筆／秒
預估剩餘
約 6 分 10 秒

依目前吞吐量動態估算，包含後續檔案驗證。

+
正在努力整理你的資料！你可以查看下方的逐表進度。
+

本次工作

✓ 準備›✓ 結構›匯出資料›驗證檔案›完成
+
+ +
+
+
+
畫面方向預覽 · 沿用 mySQLPunk 精靈操作與 Punky 素材 · 資料、檢查結果、時間均為示意
+ + + + + diff --git a/docs/design/mssql-export/preview-progress-dark.png b/docs/design/mssql-export/preview-progress-dark.png new file mode 100644 index 0000000..57901a3 Binary files /dev/null and b/docs/design/mssql-export/preview-progress-dark.png differ diff --git a/docs/design/mssql-export/preview-progress-light.png b/docs/design/mssql-export/preview-progress-light.png new file mode 100644 index 0000000..027f32e Binary files /dev/null and b/docs/design/mssql-export/preview-progress-light.png differ diff --git a/docs/design/mssql-export/serve-preview.py b/docs/design/mssql-export/serve-preview.py new file mode 100644 index 0000000..c7d4fe8 --- /dev/null +++ b/docs/design/mssql-export/serve-preview.py @@ -0,0 +1,59 @@ +"""Local-only, allowlisted server for the MSSQL export UI prototype. + +No database access, file uploads, or directory listing. Stop with Ctrl+C. +""" +from http.server import BaseHTTPRequestHandler, ThreadingHTTPServer +from pathlib import Path +from urllib.parse import urlsplit +import argparse + +HERE = Path(__file__).resolve().parent +ROOT = HERE.parents[2] +ROUTES = { + "/": (HERE / "index.html", "text/html; charset=utf-8"), + "/docs/design/mssql-export/index.html": (HERE / "index.html", "text/html; charset=utf-8"), +} +for name, mime in ( + ("punky_report_loading_atlas.png", "image/png"), + ("punky_report_loading.png", "image/png"), + ("punky_report_loading.wav", "audio/wav"), +): + ROUTES["/mySQLPunk/image/" + name] = (ROOT / "mySQLPunk" / "image" / name, mime) + + +class Handler(BaseHTTPRequestHandler): + def do_GET(self): + path = urlsplit(self.path).path + if path == "/": + self.send_response(302) + self.send_header("Location", "/docs/design/mssql-export/index.html") + self.end_headers() + return + resource = ROUTES.get(path) + if resource is None or not resource[0].is_file(): + self.send_error(404) + return + file, mime = resource + self.send_response(200) + self.send_header("Content-Type", mime) + self.send_header("Content-Length", str(file.stat().st_size)) + self.send_header("Cache-Control", "no-store") + self.send_header("X-Content-Type-Options", "nosniff") + self.end_headers() + try: + with file.open("rb") as stream: + while chunk := stream.read(65536): + self.wfile.write(chunk) + except (BrokenPipeError, ConnectionResetError): + pass + + def log_message(self, *_): + pass + + +if __name__ == "__main__": + parser = argparse.ArgumentParser() + parser.add_argument("--port", type=int, default=17643) + args = parser.parse_args() + print(f"Preview: http://127.0.0.1:{args.port}/docs/design/mssql-export/index.html", flush=True) + ThreadingHTTPServer(("127.0.0.1", args.port), Handler).serve_forever() diff --git a/docs/design/mssql-export/verification.md b/docs/design/mssql-export/verification.md new file mode 100644 index 0000000..74acde2 --- /dev/null +++ b/docs/design/mssql-export/verification.md @@ -0,0 +1,188 @@ +# MSSQL export verification — 2026-10-12 + +This is service/engine evidence from an isolated C: worktree, separate from native wizard, routing, audio and production acceptance. Only the explicitly authorized `(localdb)\mySQLPunkExportTest_20261012` instance was used, with Integrated Security and generated ownership-token-checked `_src`/`_dst` databases. The instance had idled to Stopped; it was explicitly restarted under the existing authorization outside the runner. The runner itself never starts or installs a server. + +## Environment and commands + +- SQL Server LocalDB `17.0.4025.3` (SQL Server 2025). +- System.Data SqlClient, .NET Framework assembly `4.0.0.0`; smoke process x64. +- sqlcmd x64, file version `2019.0150.1300.359`, installed ODBC 170 client. PE architecture and help/version saved by the runner. +- Source/target database collation `Latin1_General_100_BIN2`; tempdb `SQL_Latin1_General_CP1_CI_AS`; ANSI columns additionally use `Chinese_Taiwan_Stroke_CI_AS`. +- Local-only evidence root: `C:\Users\johnho\AppData\Local\Temp\mysqlpunk-export-20261012-7cadb673332543c3b17c61924490a702`. + +```powershell +.\tests\Run-MssqlExportIntegration.ps1 -Run -Server '(localdb)\mySQLPunkExportTest_20261012' -Profile Core -OutputDirectory '\task10-core-final-eeffe3a4581f41ffa17bf4a545c0e8af' +.\tests\Run-MssqlExportIntegration.ps1 -Run -Server '(localdb)\mySQLPunkExportTest_20261012' -Profile Concurrency -OutputDirectory '\task10-concurrency-1221f034404f49ae8990486c49cb296e' +.\tests\Run-MssqlExportIntegration.ps1 -Run -Server '(localdb)\mySQLPunkExportTest_20261012' -Profile Large -OutputDirectory '\task10-large-5fbeb319575947a298118473ae3ea3b9' +``` + +The runner invokes the existing SmokeTests executable with one opt-in filter, never ReportExportTests or a new executable. Large launches two independent sequential processes (1× then 10×), and each stage uses the final service, not manually concatenated writers. Default invocation prints usage without building/connecting. `evidence.json` records exact generated names, sqlcmd arguments/exit, source row hashes, artifact SHA-256, file length, service elapsed time, phase memory peaks and token-checked cleanup. + +Classic client artifact restore uses exactly one command: + +```powershell +sqlcmd -S '(localdb)\mySQLPunkExportTest_20261012' -d 'mysqlpunk_export__dst' -E -b -x -f 65001 -i '' +``` + +## Core results + +| Status | Acceptance | +| --- | --- | +| PASS | Actual service-generated 178,948,508-byte Core artifact restored with sqlcmd exit 0. Mirror completed; real worker connections were closed before file verification/publication/mirror. | +| PASS | Catalog equality: native column type/length/precision/scale/collation/nullability, identity seed/increment, computed definitions/persistence, defaults, PK/UQ/index ordering/DESC/INCLUDE/filter/disabled state, FK/check state, view body/SET options and comments. Only outer whitespace around module definitions is normalized. | +| PASS | Streaming SHA-256 raw-value row multiset comparison, with explicit NULL and length markers; decimal38, temporal ticks/offsets, Unicode/unpaired surrogates/NUL, raw ANSI, empty and trailing spaces. Computed values compared; rowversion bytes excluded from equality and checked as regenerated 8-byte values. | +| PASS | Two schemas with the same Unicode/right-bracket table name; positive/negative identity subsequent inserts and fresh empty identity seed. Source allocation gaps are outside the contract. | +| PASS | 67,108,864-byte binary, 16,793,600-byte Unicode, multi-LOB/no-PK, large text/ntext/image, independently small ntext/image, untyped XML whitespace, >2 MiB ordinary WKT text and literal GO/:r/$(name) text. All raw value hashes match. | +| PASS | Existing destination CREATE collision returns nonzero sqlcmd exit, without an implicit replacement path. | +| PASS | Live preflight blocks native geometry/geography, sql_variant, alias type, typed XML, unselected sequence dependency, encrypted definition, SNAPSHOT OFF, and SELECT-only reader denied VIEW DEFINITION. | +| PASS | Actual service DataOnly artifact copies with missing, duplicated or swapped chunks, conversion error and repeated terminal batch fail under sqlcmd -b. A deliberately continuing same-session client also cannot insert the malformed row or substitute NULL; replay leaves the earlier row plus exactly one guarded row. | + +The first Core attempt exposed an actual product defect: small legacy `text` was emitted as direct varbinary assignment (SQL Msg 206). Commit `14ba2fc` routes these rows through existing typed raw-byte staging. A focused tiny-text test failed before the change and all four values tests passed afterward. The final Core run verifies preserved source codepage bytes; ordinary small `ntext` and `image` also pass independently. Staging small legacy text has additional script/restore cost. Other intermediate Core failures were harness expectation corrections (outer view whitespace; denied-definition Preflight throws rather than returns issues), not product data changes. + +## Concurrency and failure results + +| Status | Acceptance | +| --- | --- | +| PASS | A second source connection updates parent and child in one transaction between the export's table reads. Restored parent and child both contain the old generation; source contains the new generation. | +| PASS | Schema changed after snapshot start fails the export and preserves the prior file. Closing the owned source connection during the job fails and releases it. | +| PASS | Live read cancellation, controlled write/verify/mirror faults, actual verify/mirror-read cancellation, and a Windows concurrent target content replacement preserve the appropriate old/racing file or confirmed local result. Owned partials are cleaned. Controlled stream failure represents disk-full/IO behavior; the disk was not physically filled. | +| PASS | An uncommitted ALTER TABLE holds a real schema lock while the worker is waiting for metadata; cancellation completes in 633 ms with source disposal, old output preserved and blocker rollback. | +| PASS | Source connections are closed before local verification/mirror; mirror failure/cancel reports local completed, mirror incomplete. | +| PENDING | Real UNC/share removal or physical disk-full event; only controlled stream/filesystem fault coverage is claimed. | + +The controlled overwrite-race test changes the file before the final consent-stamp check. It does not prove atomic protection against unrelated external writers in the final check/File.Replace window. File.Replace has no compare-and-swap guarantee; displaced prior content can remain in a reported recovery path after a partial replacement error. A new filename is recommended for externally managed targets. + +## Capacity and memory + +**PASS:** 1x service, single-session restore, hash/catalog and cleanup. **PASS:** 10x actual >2 GiB artifact, export/verify/mirror and measured bounded process memory. **FAIL with installed classic client:** 10x restore crosses its 2 GiB input limit. **PASS with official Go v1.10.0:** the unchanged saved 10x artifact restored completely in one session; 10,240 rows, streaming hashes, catalog and exact-owned cleanup all passed. + +Classic sqlcmd15.0.1300.359 returned Msg102 at row7,961, after exactly7,960 committed rows. The affected132,302-byte statement crosses input byte2,147,483,648 (line95,541, offset2,147,365,047). After normalizing only the row sequence and consumed watermark, it is byte-identical to a chunk already accepted for row1; normalized SHA256 `02292c14e069f1e52d323046d1f11211c4cf59ea26145bd999bd00bfac9be15e`. This supports a client input-boundary defect, not a new writer change. Microsoft staff have also documented [classic sqlcmd truncation at2GB](https://learn.microsoft.com/en-us/answers/questions/248682/sqlcmd-breaks-input-insert-string). The failed run and diagnosis remain in Large-10; both owned databases were cleaned. The alternative official Go client subsequently restored the same saved artifact successfully without another exporter run. + +| Metric | 1x | 10x | +| --- | ---: | ---: | +| Rows | 1,024 | 10,240 | +| Source binary payload bytes | 134,217,728 | 1,342,177,280 | +| Actual SQL file bytes | 276,231,755 | 2,762,444,660 | +| Service elapsed including verify/publication/mirror | 8.789 s | 105.212 s | +| Data peak working set / private bytes | 62,398,464 / 47,181,824 | 63,320,064 / 47,919,104 | +| Verify peak working set / private bytes | 60,321,792 / 44,920,832 | 62,115,840 / 46,460,928 | +| Mirror peak working set / private bytes | 60,489,728 / 44,945,408 | 62,152,704 / 46,465,024 | + +Data peaks grew approximately 1.5%, verify/mirror about 3%, while source rows/payload grew exactly 10x. No proportional total-data memory growth was observed. File phase peak changes were under 2 MiB and consistent with process/runtime allocation variation. The elapsed difference includes local IO/cache/environment variability; this is not a controlled throughput benchmark. + +Artifact SHA-256 (final Core and original Large service output): + +- Core: `685148d4d9cab08c3eca6244b4b484e17efcb39525c02869eae26e0634869881` +- 1x: `49c54297650993a739c837b17531475f9c14d57c2d1388bf550229f5f2c2469f` +- 10x: `7679cf0f779eb9044a845d0f8dfdc32de14de89f73973ce52ca93b6d3adeb24b` + + The verifier hashes fields through a 64 KiB buffer; per-row digests are retained only for comparison, outside the service-memory measurement. Independent service processes are sampled every 25 ms, including periods when the worker is blocked. Phase peaks are working set/private bytes, not SQL Server or sqlcmd memory. These measurements establish this fixture's behavior, not a universal upper bound on all metadata or workloads. + +## Explicit pending boundaries + +- SSH endpoint unavailable: job tunnel success/cancel/failure cleanup and main-connection continuity remain PENDING. +- Native wizard long-job responsiveness, device sound exactly once/mute and human UI acceptance remain PENDING in this engine worktree; HTML or fake-source tests are not substituted. +- GUI import path, browser/IIS/production/user-account acceptance remain PENDING/not exercised. +- Older SQL engines with missing catalog columns and feature-specific CLR/FILESTREAM/AlwaysEncrypted/partition environments were not installed or enabled. Current-engine preflight/fake metadata coverage does not establish those environments. +- Single fields >1 GiB or near theoretical type limits remain optional PENDING stress, outside the approved v1 required gate. +- No packaging, release, merge, push or antivirus changes were performed. Large artifacts are local-only and are not in Git. + +## Official Go client compatibility + +Controller approved a portable official client only within the C: fixture root; no installer/system client replacement. [Microsoft v1.10.0 release](https://github.com/microsoft/go-sqlcmd/releases/tag/v1.10.0), `sqlcmd-windows-amd64.zip`, official release asset SHA-256 `a4c28332fcc6e497d655e53ac8f4939f4ab170f9cfd32d0fa5081b60f4d9d691` matched. Extracted EXE SHA-256 `d9dbd1a8bd26213747b246deba1e13663a7e1def530cd961d7582b41a5e27852`; Authenticode Valid. Provenance/help/log files remain in `sqlcmd-client-9ced1042941b441e88f61425c091e60f`. + +This client's `-f65001` fails as an unsupported flag. The agreed contract is UTF-8 input, substitution disabled, fail-fast and one connection; `-f` is classic-client syntax. The pinned [IncludeFile source](https://github.com/microsoft/go-sqlcmd/blob/v1.10.0/pkg/sqlcmd/sqlcmd.go#L339-L373) explicitly wraps input in a UTF-8 decoder with BOM override and reads input incrementally. The same module retains one `sql.Conn` across batches, bypasses variable replacement with `-x`, and exits on qualifying SQL errors with `-b`. + +`task10-go-probe-a846216a8dd943f0b0cca4d750b5c24d/ClientProbe-1`: PASS native UTF-8 Unicode codepoints/catalog identifier, literal `$(name)`, GO inside string versus batch delimiter, temporary table surviving GO, and SQL error preventing the following batch. Successful probe exit 0; intentional fail-fast probe exit 1. Both owned databases cleaned. + +Go did not resolve the shorthand `(localdb)\mySQLPunkExportTest_20261012`: the failed probe is retained under `task10-go-probe-a944d2473e664c7b8be2b4aa3766305b`. Resolving only the already authorized instance's running pipe via `sqllocaldb info` passed. The explicit equivalent command is: + +```powershell +& '' -S 'np:\\.\pipe\LOCALDB#\tsql\query' -d 'mysqlpunk_export__dst' -E -b -x -i '' +``` + +No input splitting or independent sessions are used. The opt-in runner exposes `-Client Go -SqlcmdPath `; `-Profile ClientProbe` checks compatibility. `-Profile Large -RestoreArtifact ` checks original token/evidence/SHA, copies unchanged bytes to a new owned fixture, regenerates deterministic expected rows and compares streaming hashes/catalog after restoration. It preserves the prior autogenerated PK name solely for catalog equality. + +Additional Go module-text parser probe: `go-parser-e8b7c0d85a264e8f84b7e65d9cc614bc`, **9/9 PASS**, using a separately generated token-owned small database. Matrix: multiline block comment, multiline N-string and single-line comment, each containing GO, `:r` to a confirmed nonexistent owned path, or literal `$(name)`. GO/:r/variable tokens begin at column 1 inside multiline contexts; line-comment tokens follow `--`. Every case exited 0 with its marker; string outputs retained literal tokens. The include path was absent before/after and never opened as a directive. Exact name/token/datafile checks passed before database DROP. Tiny SQL, logs, results and cleanup evidence remain local; `.superpowers/sdd/implementation-plan/task-10-go-parser-probe.ps1` reproduces the bounded diagnostic. The long restore session was not interrupted and no checkout build ran concurrently. This proves these SELECT-only valid quoted/comment contexts, not malformed modules or bare directives outside SQL syntax. + +## Standalone checks and review + +PASS `tests/Run-SmokeTests.ps1 -Configuration Release`: 133/133 plus AutoReleasePolicy, with prior filters/live opt-ins saved, cleared and restored. PASS `build.bat Debug`, `build.bat Release`, and `git diff --check`. Logs: `.superpowers/sdd/implementation-plan/task-10-full-smoke.log`, `task-10-build-debug.log`, `task-10-build-release.log`. No AV block occurred. Packaging was not run; existing smoke includes release exclusion/policy checks. The later Go harness extension compiled and its live client probe passed. Controller accepted checkpoint `c67e9b4752d5ab9eccd284a7cab76885061124d1` after read-only review; final combined branch tests remain controller-owned. + +## Final >2 GiB restore result + +**PASS** `task10-go-large-e61519c425e545e4b47c4ecfc00e3bca/Large-10`: the unchanged 2,762,444,660-byte final-service artifact (SHA-256 `7679cf0f779eb9044a845d0f8dfdc32de14de89f73973ce52ca93b6d3adeb24b`) restored with official Go v1.10.0, one process/connection, `-E -b -x -i`, and the authorized instance's `np:\\.\pipe\LOCALDB#D570B7E8\tsql\query` endpoint. Exit 0; restore elapsed **22:00.8247112**, separate from the original 105.212-second service export/verify/mirror measurement. + +All **10,240** rows match the independently regenerated source through streaming field hashes including lengths/null markers. Sorted-row digest SHA-256: `57B5ACA8A4E28FBDF32752E1483102512C93A0721C790BB2EADBED9383669407`. All catalog comparisons passed. Source/target `mysqlpunk_export_d415addfbd444f3ca3079283c636bff2_src/_dst` were checked against exact names, ownership token and file paths before DROP. Successful restore log, evidence JSON and catalog comparisons remain local. Existing SmokeTests integration filter and AutoReleasePolicy exited successfully; no new executable or alternate restore service was used. + +Reproduction command (the output directory must be fresh): + +```powershell +.\tests\Run-MssqlExportIntegration.ps1 -Run -Server '(localdb)\mySQLPunkExportTest_20261012' -Profile Large -Client Go -SqlcmdPath '\sqlcmd-client-9ced1042941b441e88f61425c091e60f\extracted\sqlcmd.exe' -RestoreArtifact '\task10-large-5fbeb319575947a298118473ae3ea3b9\Large-10\export.sql' -OutputDirectory '\' +``` + +Both client outcomes are retained: the tested classic version is unsuitable above this boundary; the tested official Go version meets the complete-file restore gate. This does not establish performance or compatibility for every sqlcmd/SQL Server version. Remaining SSH/native UI/older-engine/optional stress boundaries above are unchanged. + +## Combined branch and native fixture evidence + +Controller integration commit `bb38aa3` combines reviewed engine, native wizard, routing and acceptance harness. The only merge conflicts were additive smoke registrations; Wizard, Routing and Integration are all retained exactly once. + +- PASS Windows PowerShell `tests/Run-SmokeTests.ps1 -Configuration Release`, exit 0: runner reports 146 groups, including the live integration opt-in group intentionally SKIP without explicit `-Run`. This does not repeat or replace the separate Core/Large/Concurrency evidence above. All existing MYSQLPUNK process-environment filters/opt-ins were saved, cleared and restored. +- PASS AutoReleasePolicy, explicit `build.bat Debug`, explicit `build.bat Release`, and `git diff --check`. Existing `nuget.exe not found, packages directory detected` fallback warning remains; successful builds are not warning-free. No packaging/release or AV change occurred. Logs: `.superpowers/sdd/implementation-plan/final-combined-smoke.log`, `final-combined-build-debug.log`, `final-combined-build-release.log`. +- Native Computer Use observation, after explicit user retry: light/dark 1102x762 four-step fixture, schema/comments, large animated Punky on the right, one mute control, dual measured fixture bars, stale ETA becoming unknown, and delayed close awaiting cancellation cleanup. Terminal canceled state and Close/Reset accessible names were observed. Preview windows were closed through UI before builds. +- A real `ShowDialog` fake-service regression caught and verified the WinForms CancelButton modal-exit fix: cancel export keeps the result visible after cleanup; titlebar close waits and then exits. Root-column layout was exercised at 640x480 with 1/1.5/2 simulated control scaling. These are control tests, not physical monitor DPI observations. +- Still PENDING: actual audible playback, physical 100/150/200% DPI, native wizard driving a long real database job, SSH endpoint lifecycle, GUI import and production acceptance. No user saved connection or production database was accessed. + +Final whole-branch review and the single scoped fix re-review are complete; their verdict and amended-code checks are recorded below. The original master checkout and its pre-existing untracked files remain untouched; no master merge, push, tag or release has occurred. +## Final review and amended-code verification + +The broad review of `7507adc..bb38aa3` found 0 Critical, 4 Important and 2 Minor issues. One coordinated fix wave, `0c0009a777f43d96df2d77e5aeeb361674ff068e`, addressed all six. Independent scoped re-review reports all six ADDRESSED and no new Critical/Important breakage. Reports are retained locally as `final-review.md`, `final-fix-report.md` and `final-rereview.md` in the plan's ignored SDD workspace; public outcomes are below. + +| Resolved issue | Final evidence | +| --- | --- | +| Native sizes were always unknown | Bounded heap/clustered/LOB allocation metadata, optional-query fallback to null, no table COUNT or new DMV permission. Real tables returned 16,384 and 172,032 bytes and fed the weighted estimator. Mutable sizes/row estimates remain excluded from definition fingerprints. | +| Preflight lost object/column identity | Control regression displays the exact schema/object/column, safe issue code and reason, with a database-level fallback. | +| Missing old-engine index option became OFF | Missing capability remains null and the writer omits SQL2019-only syntax. Static capability-to-DDL regression passes; actual SQL2017 restore remains PENDING. | +| DataOnly lacked session settings | Common SET preamble, no object DDL. A real service artifact restores into an indexed computed-column target using the exact documented classic command without an added -I. | +| Scope/counters absent from UI | Control tests cover views, computed/rowversion regeneration, snapshot resource note, completed/total tables, row counters and estimated percent. Phase ETA remains explicitly phase-only. | +| Durations wrapped at 24 hours | Pure formatter regression preserves 25:02:03 for elapsed and ETA. | + +PASS amended-code focused groups: metadata5, schema6, worker4, wizard10; final Release metadata5. AutoReleasePolicy passed. Root separately ran explicit `build.bat Debug` and `build.bat Release` at final code head; both passed (`final-fixed-build-debug.log`, `final-fixed-build-release.log`). Existing NuGet cached-package fallback warning remains. The full146-group run above was before this narrow fix; it is not represented as a new full-suite run. No >2GiB exporter/restore repeat was needed for these metadata/UI/session-preamble changes; the earlier exact large-artifact evidence is retained with its original provenance. + +Final tiny live evidence: `/final-fix-verified-813444899eb0494fa7429419ea15435d/FinalFix-1/evidence.json`, engine17.0.4025.3. The actual DataOnly artifact is395bytes, SHA256 `30438aa2f50d52957dde52e90e5471960b62c61c788ed83c4cb958a8c7f190c9`. One classic sqlcmd process with `-E -b -x -f 65001 -i` (no -I) exited0 and restored both rows into the indexed computed-column table. Exact token `11ad52b49eee49d0b0603b820a013845` source/target DBs were checked and dropped. Reproduce with the opt-in runner's `-Profile FinalFix` and a new empty directory under the same authorized root; all standard ownership gates apply. + +A final-head native preview was launched with the existing fixture-only SmokeTests executable, then the user accidentally pressed physical Escape before the first window observation. The user explicitly resumed, but the tool immediately continued reporting its stopped state. No further window inputs, screenshots, process termination or executable rehosting occurred. At that point, actual final-head visual observation of the amended labels was PENDING, not passed. The earlier native fixture observations and the new control tests retained their separate scopes. The preview was not closed through a bypass. The subsequent user-requested retry below resolves this specific visual-observation gap. + +All of the review's listed unjudged environment/scope items were retained: SSH; physical DPI/audio and native real long-job; GUI import/production/IIS/browser/account; real UNC/disk-full/power-loss; optional >1GiB single-field/theoretical limits; older/feature-specific engine environments; future spatial codec and spatial indexes; File.Replace external races/recovery paths; limited native replacement-error layouts; inherited NuGet warning; identity allocation-history exclusion; compatible-target/non-idempotent/partial-restore and resource requirements; prototype-only evidence; other providers/native/scheduled backups and publication. Previously deferred continuation and 2MiB metadata-budget coverage is resolved. No unjudged environment is silently promoted to PASS. + +All required implementation work and mandatory isolated tests are complete. The explicitly pending acceptance boundaries above remain open. Feature branch `codex/mssql-export-wizard` is retained; no master merge, push, tag, packaging or release was performed. The ignored SDD authorization manifest/evidence is retained under ruling13 so the guarded runner stays reproducible. + +## Final amended-label native retry (2026-10-12) + +PASS user-requested Computer Use retry against the existing final-code fixture executable at `mySQLPunk/bin/CodexVerify/mySQLPunk.SmokeTests.exe`, product code `0c0009a`, documentation head `1410b25`. No rebuild or product edits were needed. Both windows were 1102x762; test title was `FIXTURE ONLY`. + +- Dark, schema and data: schema/comment/estimated-row/estimated-size columns and selection totals visible. Confirmation fully showed view scope, computed/rowversion regeneration, same SNAPSHOT transaction and version-store cost. No label overlap was observed. +- Running in both themes: estimated 43%, 12,345,678 bytes, tables 0/2, rows 50/100 and the separate 50% table bar were visible. Initial phase ETA 00:00:03 and speed 2,500 bytes/s changed to waiting/estimating and unknown speed after the fixture stopped reporting progress. These are synthetic values, not a new throughput measurement. +- Punky visibly advanced through multiple poses automatically in both themes. Only a mute control was present; the dark-theme click changed its caption to enable sound. This confirms UI behavior, not audible playback or audio-once timing. +- Dark cancel: disabled cancel and cleanup wording were observed before terminal canceled stage/footer and both canceled queue rows. The result could be scrolled fully into view, including 0/2 completed, zero result rows, view/computed/SNAPSHOT notes, no published file, and the complete restore-client notice. Close and Reset remained visible. Cancel correctly kept the result window open. +- Light, data only: the views checkbox became unchecked/disabled and confirmation stated all views excluded. During running, titlebar Close first showed cancellation/cleanup and then the window exited. Dark terminal Close also exited normally. Final window inventory contained no fixture windows, and both launched process IDs had exited; no process kill or executable rehosting was used. + +Scope: real native rendering/interaction with a fake service, not HTML screenshots. The fixture does not open a DB or write/publish SQL; its preflight can inspect the generated destination, which was kept unchanged. No database acceptance, build, full suite, release or production action was repeated. Actual audio, physical DPI, native long real-DB jobs, SSH and the other explicitly pending boundaries above remain unverified. The earlier Escape interruption remains recorded, but final amended-label visual observation is now PASS. + +## Native Geometry extension: isolated restore evidence (2026-10-12) + +This explicitly authorized extension supersedes only the earlier native geometry rejection. Geography, user-defined/alias CLR types and spatial indexes remain rejected. The existing wizard, source transaction, artifact ownership and publication boundaries are retained. + +PASS `Geometry` profile using the actual service and installed classic sqlcmd, SQL2025 `17.0.4025.3`, 64-bit .NET Framework. Evidence: `/geometry-9285b5a115c54830bfb449fef2697d95/Geometry-1/evidence.json`; ownership token `039c89c494de483190be66a93f0ea582`. Both exact token-owned source/target databases were checked and dropped. + +- Stream-generated WKB was uploaded using a stream parameter for fixture construction. The native large value serialized to **17,600,048 bytes**, exceeding 16 MiB. Production export uses server Serialize() and sequential bounded GetBytes; no client spatial package or whole-value materialization was introduced. +- Complete service artifact: **36,193,053 bytes**, SHA-256 `6e30d3b286eef4fddd34abc39c1d08f39c56e98a7b7d6609097f1b085babcf49`; 10 rows across 3 tables. Service export/verify/publish elapsed 1.562496 s; one classic `-E -b -x -f 65001 -i` restore exited 0 in 4.8491934 s. These are this fixture's measurements, not a throughput promise. +- Source/target raw Serialize() streaming hashes and catalog equality PASS. Cases include NULL, empty, points, polygon, collection, curves, invalid but storable geometry, multiple geometry fields with ordinary scalar/binary fields, SRID 0/4326/3826, Z/M and the large value. Independent spatial observations confirmed the tested semantics; no MakeValid, reprojection or simplification occurred. GeometryLarge row digest: `ED402280096B33C712A88EC55AFF3987A781A13AEB858ED2FCF7A1C343134083`. +- Actual SchemaOnly and DataOnly artifacts restored successfully. Missing, duplicate and reordered chunks, malformed serialized content, and replayed final batches were tested with fail-fast sqlcmd and a continuing importer; the failing row stayed absent, while replay retained one already committed row. All-mode geography and spatial-index preflight rejection PASS, with no published file. +- Export Data-phase sampled process peaks: working set 64,618,496 bytes, private bytes 51,924,992 bytes (42 samples). Fixture generation/upload was sampled separately. This single run is not a scaling benchmark or universal memory ceiling; SQL Server serialization/deserialization and staging resources are separate. + +The first live run had already passed artifact restoration, byte hashes and catalog, but its observation query failed with SQL24144 because STGeometryType() rejects invalid geometries. The verifier was corrected to use STIsValid/SRID and raw serialized bytes for that row, while valid values retain type/Z/M checks. The corrected full Geometry profile above passed; earlier failure evidence remains in the ignored extension workspace. This was a verifier correction, not geometry repair. + +Cross-version internal serialization compatibility, physical native UI/audio/DPI, long real-job UI, SSH, production, geography, spatial-index DDL and near-theoretical size limits remain unverified or unsupported as stated. Earlier >2GiB acceptance remains separate and was not rerun for this codec extension. Final product commit `92fda819f72b9307df5c37d7901d966b845dd1cf`: focused metadata6/values5/LOB4/schema6, full existing smoke151 (live integration explicitly SKIP in that run), AutoReleasePolicy, explicit Debug/Release builds and diff check PASS. Existing cached-package/NuGet-executable warning remains; no new dependency. Logs/report are in `.superpowers/sdd/geometry-implementation-plan/`. Independent review of immutable extension diff `5edf69d..e3d4998` is APPROVED with no actionable findings. The reviewer independently traced native metadata identity, same-SNAPSHOT projection, bounded continuation, final Deserialize rollback/poisoning and the actual malformed-payload restore log, without rerunning tests or DB work. Report: `.superpowers/sdd/geometry-implementation-plan/geometry-review.md`. Its unjudged cross-version, theoretical-capacity, server-resource, geometry-scaling and previous UI/SSH/production boundaries remain as stated; no additional acceptance is implied. + +The affected Core profile also passed after the verifier began resolving native types through user_type_id: `/geometry-core-9fc8f25c34d04967855d22531dfb9f65/Core-1`, artifact178,948,509bytes SHA256 `80afc52efd6cb8a00b4f312428f2bf91d2274f7e7b684e5bcb417005f4ce9159`, ownership token `2bcec8beffef4c2b95f693a047f47cd4`. Existing64MiB binary/>16MiB Unicode, catalog/raw hashes, identity/computed/rowversion, mirror, unsupported/visibility/SNAPSHOT gates, and all five chunk-fault/continuing-importer cases passed. Exact owned DBs were cleaned. This rerun replaces no earlier Large/Concurrency/native UI evidence. diff --git a/mySQLPunk/Form1.MssqlExport.cs b/mySQLPunk/Form1.MssqlExport.cs new file mode 100644 index 0000000..2ac428c --- /dev/null +++ b/mySQLPunk/Form1.MssqlExport.cs @@ -0,0 +1,114 @@ +using System; +using System.Collections.Generic; +using System.IO; +using System.Windows.Forms; +using mySQLPunk.lib; +using mySQLPunk.lib.MssqlExport; + +namespace mySQLPunk +{ + public partial class Form1 + { + // Instance boundaries keep modal routing testable without opening saved profiles or connections. + internal Func, string, MssqlExportWizardOptions, MssqlExportResult> MssqlExportWizardRunner = null; + internal Func SqlExportSaveDialogRunner = null; + internal Func MssqlExportMirrorCapture = null; + internal Action MssqlExportStatusSink = null; + + private static bool IsMssqlExportTarget(TreeDatabaseTarget target) + { + return target != null && (string.Equals(target.ProviderName, "mssql", StringComparison.OrdinalIgnoreCase) + || string.Equals(target.ProviderName, "sqlserver", StringComparison.OrdinalIgnoreCase)); + } + + private DialogResult ShowSqlExportSaveDialog(SaveFileDialog dialog) + { + return SqlExportSaveDialogRunner == null ? dialog.ShowDialog() : SqlExportSaveDialogRunner(dialog); + } + + private void SetMssqlExportStatus(string message) + { + if (MssqlExportStatusSink == null) UpdateMainStatus(message); + else MssqlExportStatusSink(message); + } + + private void OpenMssqlExportWizard(TreeDatabaseTarget target, MssqlExportMode mode, + string initialSchema, string initialTable, bool backupEntry) + { + try + { + var settings = new Dictionary(); + foreach (var pair in target.ConnectionInfo) + { + if (string.Equals(pair.Key, "pdo", StringComparison.OrdinalIgnoreCase) + || string.Equals(pair.Key, "isConnect", StringComparison.OrdinalIgnoreCase) + || string.Equals(pair.Key, "connString", StringComparison.OrdinalIgnoreCase)) continue; + if (pair.Value == null || pair.Value is string || pair.Value.GetType().IsPrimitive || pair.Value is decimal) + settings[pair.Key] = pair.Value; + } + settings["initial_database"] = target.DatabaseName; + + MssqlMirrorOptions mirror = null; + if (backupEntry) + { + if (MssqlExportMirrorCapture != null) mirror = MssqlExportMirrorCapture(); + else + { + string directory = BackupMirrorSettings.RemoteDirectory; + int retainCount = BackupMirrorSettings.RetainCount; + if (!string.IsNullOrWhiteSpace(directory)) mirror = new MssqlMirrorOptions(directory, retainCount); + } + } + + // Tree callers pass initialSchema=null and the provider's canonical raw ObjectName unchanged. + // Only the background native catalog may resolve that seed; never split on dots here. + string seed = initialSchema == null ? initialTable : initialSchema + "." + initialTable; + string fileName = seed == null + ? target.DatabaseName + (backupEntry ? "_backup_" : "_dump_") + DateTime.Now.ToString("yyyyMMdd_HHmmss") + ".sql" + : target.DatabaseName + "_" + seed + (mode == MssqlExportMode.DataOnly ? "_data" : "_structure_data") + ".sql"; + foreach (char invalid in Path.GetInvalidFileNameChars()) fileName = fileName.Replace(invalid, '_'); + var options = new MssqlExportWizardOptions(mode, null, seed == null, fileName, mirror, seed); + MssqlExportResult result; + if (MssqlExportWizardRunner != null) result = MssqlExportWizardRunner(settings, target.DatabaseName, options); + else + { + using (var wizard = new MssqlExportWizardForm(settings, target.DatabaseName, options)) + { + wizard.ShowDialog(this); + result = wizard.Result; + } + } + // Closing setup has no result and must not produce a success notification. + if (result != null) SetMssqlExportStatus(BuildMssqlExportStatus(result)); + } + catch (Exception ex) + { + SetMssqlExportStatus(ReportText.Get("SQL Server 匯出失敗", "SQL Server export failed") + " (" + ex.GetType().Name + ")"); + } + } + + private static string BuildMssqlExportStatus(MssqlExportResult result) + { + string message; + switch (result.Stage) + { + case MssqlExportStage.Completed: + message = ReportText.Get("SQL Server 匯出完成", "SQL Server export completed"); + break; + case MssqlExportStage.LocalCompletedMirrorIncomplete: + message = ReportText.Get("SQL Server 本機匯出完成；鏡像未完成", "SQL Server local export completed; mirror incomplete"); + break; + case MssqlExportStage.Canceled: + message = ReportText.Get("SQL Server 匯出已取消", "SQL Server export canceled"); + break; + default: + message = ReportText.Get("SQL Server 匯出失敗", "SQL Server export failed"); + break; + } + if (!string.IsNullOrWhiteSpace(result.PublishedPath)) message += " · " + result.PublishedPath; + if (!string.IsNullOrWhiteSpace(result.MirrorPath)) message += " · " + result.MirrorPath; + if (!string.IsNullOrWhiteSpace(result.ErrorMessage)) message += " · " + result.ErrorMessage; + return message; + } + } +} diff --git a/mySQLPunk/Form1.cs b/mySQLPunk/Form1.cs index 35e85b8..4617393 100644 --- a/mySQLPunk/Form1.cs +++ b/mySQLPunk/Form1.cs @@ -6662,6 +6662,12 @@ private void DumpSelectedTableSql(bool dataOnly) IDatabase db = (IDatabase)connInfo["pdo"]; string dbName = pathParts[1]; string tableName = pathParts[3]; + TreeDatabaseTarget target = BuildTargetFromNode(node); + if (IsMssqlExportTarget(target)) + { + OpenMssqlExportWizard(target, dataOnly ? lib.MssqlExport.MssqlExportMode.DataOnly : lib.MssqlExport.MssqlExportMode.SchemaAndData, null, tableName, false); + return; + } using (SaveFileDialog dialog = new SaveFileDialog()) { @@ -6669,7 +6675,7 @@ private void DumpSelectedTableSql(bool dataOnly) dialog.DefaultExt = "sql"; dialog.FileName = dbName + "_" + tableName + (dataOnly ? "_data" : "_structure_data") + ".sql"; - if (dialog.ShowDialog() != DialogResult.OK) + if (ShowSqlExportSaveDialog(dialog) != DialogResult.OK) { return; } @@ -6881,7 +6887,7 @@ private void DumpSelectedViewSql() dialog.DefaultExt = "sql"; dialog.FileName = selection.DatabaseName + "_" + selection.ObjectName + "_view.sql"; - if (dialog.ShowDialog() != DialogResult.OK) + if (ShowSqlExportSaveDialog(dialog) != DialogResult.OK) { return; } @@ -6909,6 +6915,13 @@ private async void DumpCurrentSelectionSqlWithDialog(bool dataOnlyForTable) } DatabaseObjectSelection selection = GetSelectedDatabaseObject(); + if (IsMssqlExportTarget(target) && (selection == null || selection.GroupName == "Tables")) + { + bool tableDataOnly = selection != null && dataOnlyForTable; + OpenMssqlExportWizard(target, tableDataOnly ? lib.MssqlExport.MssqlExportMode.DataOnly : lib.MssqlExport.MssqlExportMode.SchemaAndData, + null, selection == null ? null : selection.ObjectName, false); + return; + } MySqlExportOptions mysqlOptions = null; if (!dataOnlyForTable && selection == null && IsMySqlTarget(target)) { @@ -6922,7 +6935,7 @@ private async void DumpCurrentSelectionSqlWithDialog(bool dataOnlyForTable) dialog.DefaultExt = "sql"; dialog.FileName = BuildCurrentSelectionDumpFileName(dataOnlyForTable); - if (dialog.ShowDialog() != DialogResult.OK) + if (ShowSqlExportSaveDialog(dialog) != DialogResult.OK) { return; } @@ -7264,13 +7277,19 @@ private void DumpSelectedDatabaseSqlWithDialog() return; } + if (IsMssqlExportTarget(target)) + { + OpenMssqlExportWizard(target, lib.MssqlExport.MssqlExportMode.SchemaAndData, null, null, false); + return; + } + using (SaveFileDialog dialog = new SaveFileDialog()) { dialog.Filter = Localization.T("Common.SqlFilesFilter"); dialog.DefaultExt = "sql"; dialog.FileName = target.DatabaseName + "_dump_" + DateTime.Now.ToString("yyyyMMdd_HHmmss") + ".sql"; - if (dialog.ShowDialog() != DialogResult.OK) + if (ShowSqlExportSaveDialog(dialog) != DialogResult.OK) { return; } @@ -8046,6 +8065,12 @@ private void BackupSelectedDatabaseWithDialog() return; } + if (IsMssqlExportTarget(target)) + { + OpenMssqlExportWizard(target, lib.MssqlExport.MssqlExportMode.SchemaAndData, null, null, true); + return; + } + using (SaveFileDialog dialog = new SaveFileDialog()) { bool sqlite = target.Database is my_sqlite; @@ -8053,7 +8078,7 @@ private void BackupSelectedDatabaseWithDialog() dialog.DefaultExt = sqlite ? "sqlite" : "sql"; dialog.FileName = target.DatabaseName + "_backup_" + DateTime.Now.ToString("yyyyMMdd_HHmmss") + (sqlite ? ".sqlite" : ".sql"); - if (dialog.ShowDialog() != DialogResult.OK) + if (ShowSqlExportSaveDialog(dialog) != DialogResult.OK) { return; } diff --git a/mySQLPunk/MssqlExportWizardForm.cs b/mySQLPunk/MssqlExportWizardForm.cs new file mode 100644 index 0000000..d7f7418 --- /dev/null +++ b/mySQLPunk/MssqlExportWizardForm.cs @@ -0,0 +1,929 @@ +using System; +using System.Collections.Generic; +using System.Diagnostics; +using System.Drawing; +using System.IO; +using System.Linq; +using System.Threading; +using System.Threading.Tasks; +using System.Windows.Forms; +using mySQLPunk.lib; +using mySQLPunk.lib.MssqlExport; + +namespace mySQLPunk +{ + internal sealed class MssqlExportWizardForm : Form + { + private readonly string database; + private readonly MssqlExportWizardOptions options; + private readonly MssqlSqlExportService service; + private readonly HashSet selected = new HashSet(); + private readonly HashSet processed = new HashSet(); + private readonly Panel[] pages = new Panel[4]; + private readonly TextBox search = new TextBox(), output = new TextBox(); + private readonly DataGridView tables = new DataGridView(), queue = new DataGridView(); + private readonly CheckBox views = new CheckBox(); + private readonly RadioButton schemaData = new RadioButton(), schemaOnly = new RadioButton(), dataOnly = new RadioButton(); + private readonly Label steps = new Label(), status = new Label(), selection = new Label(), confirmation = new Label(); + private readonly Label stage = new Label(), current = new Label(), percent = new Label(), written = new Label(), tableRows = new Label(), elapsed = new Label(), speed = new Label(), eta = new Label(), resultText = new Label(); + private readonly Button next = new Button(), back = new Button(), cancel = new Button(), openFile = new Button(), openFolder = new Button(), sound = new Button(); + private readonly ProgressBar overall = new ProgressBar(), tableProgress = new ProgressBar(); + private readonly PunkyReportLoadingControl punky = new PunkyReportLoadingControl(); + private readonly ListBox log = new ListBox(); + private readonly System.Windows.Forms.Timer timer = new System.Windows.Forms.Timer + { + Interval = 500 + }; + private readonly Stopwatch clock = new Stopwatch(); + private IReadOnlyList catalog = new List(); + private CancellationTokenSource cancellation; + private MssqlExportRequest prepared; + private MssqlExportProgress last; + private bool busy, closing, rebuilding, loaded, exporting, cancelRequested; + private int step, generation, includedViews; + internal MssqlExportResult Result { get; private set; } + + private static string T(string zh, string en) + { + return ReportText.Get(zh, en); + } + + internal MssqlExportWizardForm(Dictionary connection, string databaseName, MssqlExportWizardOptions options) : this(connection, databaseName, options, null) + { + } + + internal MssqlExportWizardForm(Dictionary connection, string databaseName, MssqlExportWizardOptions options, MssqlSqlExportService service) + { + if (connection == null) + throw new ArgumentNullException(nameof(connection)); + if (string.IsNullOrWhiteSpace(databaseName)) + throw new ArgumentException("Database is required", nameof(databaseName)); + this.database = databaseName; + this.options = options ?? throw new ArgumentNullException(nameof(options)); + var captured = new Dictionary(); + foreach (var pair in connection) + if (!new[] + { + "pdo", + "isConnect", + "connString" + }.Contains(pair.Key, StringComparer.OrdinalIgnoreCase) && (pair.Value == null || pair.Value is string || pair.Value.GetType().IsPrimitive || pair.Value is decimal)) + captured[pair.Key] = pair.Value; + captured["initial_database"] = databaseName; + this.service = service ?? new MssqlSqlExportService(captured); + Name = "mssqlExportWizard"; + Text = T("匯出資料庫 · SQL Server", "Export database · SQL Server"); + AutoScaleMode = AutoScaleMode.Dpi; + Font = UiKit.Body; + ClientSize = new Size(1100, 730); + MinimumSize = new Size(640, 480); + StartPosition = FormStartPosition.CenterParent; + var root = new TableLayoutPanel + { + Dock = DockStyle.Fill, + Padding = new Padding(24), + ColumnCount = 1, + RowCount = 4 + }; + root.ColumnStyles.Add(new ColumnStyle(SizeType.Percent, 100)); + root.RowStyles.Add(new RowStyle(SizeType.AutoSize)); + root.RowStyles.Add(new RowStyle(SizeType.Percent, 100)); + root.RowStyles.Add(new RowStyle(SizeType.AutoSize)); + root.RowStyles.Add(new RowStyle(SizeType.AutoSize)); + Controls.Add(root); + var header = Stack(); + header.Controls.Add(Label(T("資料庫匯出精靈", "Database export wizard") + " · " + database + " / MSSQL", UiKit.Title)); + steps.AutoSize = true; + steps.Margin = new Padding(0, 16, 0, 20); + header.Controls.Add(steps); + root.Controls.Add(header, 0, 0); + var body = new Panel + { + Dock = DockStyle.Fill + }; + root.Controls.Add(body, 0, 1); + for (int i = 0; i < 4; i++) + { + pages[i] = new Panel + { + Name = "mssqlStep" + i, + Dock = DockStyle.Fill, + AutoScroll = true + }; + body.Controls.Add(pages[i]); + } + + BuildSelection(); + BuildOutput(); + BuildConfirmation(); + BuildProgress(); + status.Name = "mssqlStatus"; + status.AutoSize = true; + status.MaximumSize = new Size(950, 0); + status.Margin = new Padding(0, 12, 0, 10); + root.Controls.Add(status, 0, 2); + var footer = new FlowLayoutPanel + { + Name = "mssqlFooter", + AutoSize = true, + Dock = DockStyle.Fill, + FlowDirection = FlowDirection.RightToLeft, + WrapContents = true + }; + Configure(cancel, "mssqlCancel", T("取消", "Cancel"), (s, e) => + { + if (busy) + RequestCancel(); + else + Close(); + }); + Configure(next, "mssqlStart", T("下一步 →", "Next →"), async (s, e) => + { + if (step == 0) + { + step = 1; + ShowPage(); + } + else if (step == 1) + { + step = 2; + ShowPage(); + await PreflightAsync(); + } + else if (step == 2) + await ExportAsync(); + else + { + InvalidateRequest(); + step = 0; + ShowPage(); + } + }); + Configure(back, "mssqlBack", T("← 上一步", "← Back"), (s, e) => + { + InvalidateRequest(); + step--; + ShowPage(); + }); + Configure(openFile, "mssqlOpenFile", T("開啟檔案", "Open file"), (s, e) => Open(Result?.PublishedPath)); + Configure(openFolder, "mssqlOpenFolder", T("開啟資料夾", "Open folder"), (s, e) => Open(Result?.PublishedPath == null ? null : Path.GetDirectoryName(Result.PublishedPath))); + footer.Controls.AddRange(new Control[] { cancel, next, back, openFolder, openFile }); + root.Controls.Add(footer, 0, 3); + ThemeManager.MarkAsPrimary(next); + AcceptButton = next; + CancelButton = cancel; + // CancelButton assigns DialogResult.Cancel automatically. Keep cancellation + // inside the wizard so a modal caller can inspect the terminal result. + cancel.DialogResult = DialogResult.None; + Shown += async (s, e) => + { + var work = Screen.FromControl(Owner ?? this).WorkingArea; + MinimumSize = new Size(Math.Min(MinimumSize.Width, work.Width), Math.Min(MinimumSize.Height, work.Height)); + Size = new Size(Math.Min(Width, work.Width), Math.Min(Height, work.Height)); + Location = new Point(Math.Max(work.Left, Math.Min(Left, work.Right - Width)), Math.Max(work.Top, Math.Min(Top, work.Bottom - Height))); + await LoadAsync(); + }; + FormClosing += (s, e) => + { + if (busy) + { + e.Cancel = true; + closing = true; + RequestCancel(); + } + }; + timer.Tick += (s, e) => UpdateClock(); + ThemeManager.ApplyTo(this); + percent.ForeColor = ThemeManager.AccentColor; + ShowPage(); + } + + private static FlowLayoutPanel Stack() + { + return new FlowLayoutPanel + { + Dock = DockStyle.Top, + AutoSize = true, + FlowDirection = FlowDirection.TopDown, + WrapContents = false + }; + } + + private static Label Label(string text, Font font = null) + { + return new Label + { + Text = text, + AutoSize = true, + Font = font ?? UiKit.Body, + Margin = new Padding(0, 6, 0, 8) + }; + } + + private static void Configure(Button button, string name, string text, EventHandler handler) + { + button.Name = name; + button.Text = text; + button.AccessibleName = text; + button.AutoSize = true; + button.MinimumSize = new Size(90, 36); + button.Click += handler; + } + + private static void Grid(DataGridView grid) + { + grid.Dock = DockStyle.Fill; + grid.AllowUserToAddRows = false; + grid.AllowUserToDeleteRows = false; + grid.RowHeadersVisible = false; + grid.AutoSizeColumnsMode = DataGridViewAutoSizeColumnsMode.Fill; + grid.SelectionMode = DataGridViewSelectionMode.FullRowSelect; + grid.BackgroundColor = ThemeManager.SurfaceColor; + grid.BorderStyle = BorderStyle.None; + grid.AutoSizeRowsMode = DataGridViewAutoSizeRowsMode.AllCells; + } + + private void BuildSelection() + { + var layout = new TableLayoutPanel + { + Dock = DockStyle.Fill, + ColumnCount = 1, + RowCount = 3 + }; + layout.RowStyles.Add(new RowStyle(SizeType.AutoSize)); + layout.RowStyles.Add(new RowStyle(SizeType.Percent, 100)); + layout.RowStyles.Add(new RowStyle(SizeType.AutoSize)); + pages[0].Controls.Add(layout); + var tools = new FlowLayoutPanel + { + Dock = DockStyle.Fill, + AutoSize = true + }; + search.Name = "mssqlTableSearch"; + search.Width = 260; + search.AccessibleName = T("搜尋 schema、資料表與註解", "Search schema, table and comment"); + tools.Controls.Add(Label(T("搜尋資料表", "Search tables"))); + tools.Controls.Add(search); + var all = new Button(); + var none = new Button(); + Configure(all, "mssqlSelectAll", T("全選", "Select all"), (s, e) => + { + foreach (var t in catalog) + selected.Add(t.Key.ObjectId); + InvalidateRequest(); + RefreshTables(); + }); + Configure(none, "mssqlSelectNone", T("全部取消", "Clear all"), (s, e) => + { + selected.Clear(); + InvalidateRequest(); + RefreshTables(); + }); + tools.Controls.Add(all); + tools.Controls.Add(none); + layout.Controls.Add(tools, 0, 0); + Grid(tables); + tables.Name = "mssqlTableList"; + tables.Columns.Add(new DataGridViewCheckBoxColumn { Name = "selected", HeaderText = T("選取", "Select"), FillWeight = 35 }); + foreach (var text in new[] + { + "Schema", + T("資料表", "Table"), + T("註解", "Comment"), + T("估計筆數", "Est. rows"), + T("估計大小", "Est. bytes") + } + + ) + { + int index = tables.Columns.Add(text, text); + tables.Columns[index].ReadOnly = true; + } + + tables.CurrentCellDirtyStateChanged += (s, e) => + { + if (tables.IsCurrentCellDirty) + tables.CommitEdit(DataGridViewDataErrorContexts.Commit); + }; + tables.CellValueChanged += (s, e) => + { + if (rebuilding || e.RowIndex < 0 || e.ColumnIndex != 0) + return; + var key = (MssqlObjectKey)tables.Rows[e.RowIndex].Tag; + if (Convert.ToBoolean(tables.Rows[e.RowIndex].Cells[0].Value)) + selected.Add(key.ObjectId); + else + selected.Remove(key.ObjectId); + InvalidateRequest(); + UpdateButtons(); + }; + search.TextChanged += (s, e) => RefreshTables(); + layout.Controls.Add(tables, 0, 1); + selection.AutoSize = true; + layout.Controls.Add(selection, 0, 2); + } + + private void RefreshTables() + { + rebuilding = true; + tables.Rows.Clear(); + foreach (var t in catalog) + if ((t.Key.Schema + " " + t.Key.Name + " " + t.Comment).IndexOf(search.Text, StringComparison.OrdinalIgnoreCase) >= 0) + { + int i = tables.Rows.Add(selected.Contains(t.Key.ObjectId), t.Key.Schema, t.Key.Name, t.Comment, t.EstimatedRows?.ToString("N0") ?? "—", t.EstimatedBytes?.ToString("N0") ?? "—"); + tables.Rows[i].Tag = t.Key; + } + + rebuilding = false; + UpdateButtons(); + } + + private void BuildOutput() + { + var content = Stack(); + pages[1].Controls.Add(content); + content.Controls.Add(Label(T("輸出設定", "Output settings"), UiKit.Subtitle)); + var radios = new[] + { + schemaData, + schemaOnly, + dataOnly + }; + var names = new[] + { + "mssqlSchemaAndData", + "mssqlSchemaOnly", + "mssqlDataOnly" + }; + var titles = new[] + { + T("結構與資料", "Schema and data"), + T("僅結構", "Schema only"), + T("僅資料", "Data only") + }; + for (int i = 0; i < 3; i++) + { + var radio = radios[i]; + radio.Name = names[i]; + radio.Text = titles[i]; + radio.AutoSize = true; + content.Controls.Add(radio); + radio.CheckedChanged += (s, e) => + { + if (!radio.Checked) + return; + views.Enabled = !dataOnly.Checked; + if (dataOnly.Checked) + views.Checked = false; + InvalidateRequest(); + UpdateButtons(); + }; + } + + views.Name = "mssqlIncludeViews"; + views.Text = T("包含檢視表定義（結構模式）", "Include view definitions (schema modes)"); + views.AutoSize = true; + views.Checked = options.IncludeViews; + views.CheckedChanged += (s, e) => + { + InvalidateRequest(); + UpdateButtons(); + }; + content.Controls.Add(views); + radios[(int)options.Mode].Checked = true; + content.Controls.Add(Label(T("SQL 輸出路徑", "SQL output path"))); + var path = new FlowLayoutPanel + { + AutoSize = true, + Width = 850 + }; + output.Name = "mssqlOutputPath"; + output.ReadOnly = true; + output.Width = 580; + output.AccessibleName = T("SQL 輸出路徑", "SQL output path"); + output.Text = !string.IsNullOrWhiteSpace(options.DefaultFileName) && Path.IsPathRooted(options.DefaultFileName) ? options.DefaultFileName : ""; + output.TextChanged += (s, e) => + { + InvalidateRequest(); + UpdateButtons(); + }; + path.Controls.Add(output); + var browse = new Button(); + Configure(browse, "mssqlBrowse", T("瀏覽…", "Browse…"), (s, e) => + { + using (var dialog = new SaveFileDialog + { + Filter = "SQL (*.sql)|*.sql", + DefaultExt = "sql", + AddExtension = true, + OverwritePrompt = false, + FileName = string.IsNullOrWhiteSpace(output.Text) ? options.DefaultFileName ?? database + ".sql" : output.Text + } + + ) + if (dialog.ShowDialog(this) == DialogResult.OK) + output.Text = dialog.FileName; + }); + path.Controls.Add(browse); + content.Controls.Add(path); + content.Controls.Add(Label(options.Mirror == null ? T("鏡像：未啟用", "Mirror: disabled") : T("鏡像：", "Mirror: ") + options.Mirror.Directory + " · " + options.Mirror.RetainCount)); + content.Controls.Add(Label(T("SQL 邏輯匯出；不包含登入帳號、作業或原生 .bak 備份。", "Logical SQL export; excludes logins, jobs and native .bak backups."))); + } + + private void BuildConfirmation() + { + var content = Stack(); + pages[2].Controls.Add(content); + content.Controls.Add(Label(T("確認匯出與預檢", "Confirm export and preflight"), UiKit.Subtitle)); + confirmation.Name = "mssqlConfirmation"; + confirmation.AutoSize = true; + confirmation.MaximumSize = new Size(900, 0); + content.Controls.Add(confirmation); + } + + private void BuildProgress() + { + var content = new TableLayoutPanel + { + Dock = DockStyle.Top, + AutoSize = true, + ColumnCount = 2, + RowCount = 4 + }; + content.ColumnStyles.Add(new ColumnStyle(SizeType.Percent, 72)); + content.ColumnStyles.Add(new ColumnStyle(SizeType.Percent, 28)); + pages[3].Controls.Add(content); + var stats = new TableLayoutPanel + { + Dock = DockStyle.Top, + AutoSize = true, + ColumnCount = 1 + }; + content.Controls.Add(stats, 0, 0); + foreach (var label in new[] + { + stage, + current, + percent, + written, + tableRows, + elapsed, + speed, + eta + } + + ) + { + label.AutoSize = true; + label.Margin = new Padding(0, 7, 0, 7); + label.MaximumSize = new Size(720, 0); + } + + stage.Name = "mssqlStage"; + stage.Font = UiKit.Subtitle; + current.Name = "mssqlCurrentTable"; + percent.Name = "mssqlPercent"; + percent.Font = UiKit.Display; + written.Name = "mssqlWritten"; + elapsed.Name = "mssqlElapsed"; + speed.Name = "mssqlSpeed"; + eta.Name = "mssqlEta"; + overall.Name = "mssqlOverallProgress"; + overall.Dock = DockStyle.Top; + overall.Height = 10; + tableProgress.Name = "mssqlTableProgress"; + tableProgress.Dock = DockStyle.Top; + tableProgress.Height = 6; + foreach (Control c in new Control[] + { + stage, + current, + percent, + written, + overall, + tableRows, + tableProgress, + elapsed, + speed, + eta + } + + ) + stats.Controls.Add(c); + var mascot = Stack(); + mascot.Dock = DockStyle.Top; + punky.Name = "mssqlPunky"; + punky.Size = new Size(240, 240); + punky.Muted = false; + mascot.Controls.Add(punky); + Configure(sound, "mssqlToggleSound", T("靜音", "Mute"), (s, e) => + { + punky.Muted = !punky.Muted; + sound.Text = punky.Muted ? T("開啟聲音", "Unmute") : T("靜音", "Mute"); + sound.AccessibleName = sound.Text; + }); + sound.Enabled = punky.AudioAvailable; + sound.AccessibleDescription = T("原音只播放一次；取消靜音不會重播。", "Original audio plays once; unmuting does not replay it."); + mascot.Controls.Add(sound); + mascot.Controls.Add(Label(T("正在幫你整理資料！", "Organizing your data!"))); + content.Controls.Add(mascot, 1, 0); + var flow = Label(T("準備 › 結構 › 資料 › 驗證 › 發布 › 鏡像", "Prepare › Schema › Data › Verify › Publish › Mirror")); + content.Controls.Add(flow, 0, 1); + content.SetColumnSpan(flow, 2); + Grid(queue); + queue.Name = "mssqlTableQueue"; + queue.Height = 150; + queue.Dock = DockStyle.Top; + queue.ReadOnly = true; + foreach (var text in new[] + { + T("資料表", "Table"), + T("註解", "Comment"), + T("狀態", "Status") + } + + ) + queue.Columns.Add(text, text); + content.Controls.Add(queue, 0, 2); + content.SetColumnSpan(queue, 2); + var lower = Stack(); + lower.Dock = DockStyle.Top; + resultText.AutoSize = true; + resultText.MaximumSize = new Size(900, 0); + resultText.Name = "mssqlResult"; + lower.Controls.Add(resultText); + var detail = new Button(); + Configure(detail, "mssqlDetails", T("詳細記錄", "Details"), (s, e) => log.Visible = !log.Visible); + lower.Controls.Add(detail); + log.Name = "mssqlLog"; + log.Width = 850; + log.Height = 120; + log.Visible = false; + lower.Controls.Add(log); + content.Controls.Add(lower, 0, 3); + content.SetColumnSpan(lower, 2); + pages[3].Resize += (s, e) => + { + punky.Size = new Size(Math.Max(100, Math.Min(240, pages[3].ClientSize.Width / 4)), Math.Max(100, Math.Min(240, pages[3].ClientSize.Width / 4))); + }; + } + + private MssqlExportMode Mode => dataOnly.Checked ? MssqlExportMode.DataOnly : schemaOnly.Checked ? MssqlExportMode.SchemaOnly : MssqlExportMode.SchemaAndData; + + private void InvalidateRequest() + { + generation++; + prepared = null; + includedViews = 0; + Result = null; + last = null; + resultText.Text = ""; + openFile.Visible = openFolder.Visible = false; + } + + private void ShowPage() + { + for (int i = 0; i < pages.Length; i++) + { + pages[i].Visible = i == step; + if (i == step) + pages[i].BringToFront(); + } + + steps.Text = T("① 選擇資料表 → ② 輸出設定 → ③ 確認匯出 → ④ 執行與結果", "① Tables → ② Output → ③ Confirm → ④ Progress and result"); + UpdateButtons(); + } + + private void UpdateButtons() + { + selection.Text = T("已選取 ", "Selected ") + selected.Count + " / " + catalog.Count; + back.Visible = step > 0 && step < 3; + back.Enabled = !busy; + next.Visible = step < 3 || (!busy && Result != null); + next.Enabled = !busy && loaded && (step == 0 ? selected.Count > 0 : step == 1 ? !string.IsNullOrWhiteSpace(output.Text) : step == 2 ? prepared != null : true); + next.Text = step == 2 ? T("開始匯出", "Start export") : step == 3 ? T("重新設定", "Reset") : T("下一步 →", "Next →"); + cancel.Enabled = !cancelRequested; + cancel.Text = busy && exporting ? T("取消匯出", "Cancel export") : T("關閉", "Close"); + next.AccessibleName = next.Text; + cancel.AccessibleName = cancel.Text; + openFile.Visible = openFolder.Visible = !busy && Result?.PublishedPath != null; + for (int i = 0; i < 3; i++) + pages[i].Enabled = !busy; + } + + private void BeginOperation() + { + busy = true; + cancelRequested = false; + cancellation = new CancellationTokenSource(); + UpdateButtons(); + } + + private void EndOperation() + { + busy = false; + exporting = false; + cancelRequested = false; + cancellation.Dispose(); + cancellation = null; + UpdateButtons(); + if (closing) + Close(); + } + + private async Task LoadAsync() + { + BeginOperation(); + status.Text = T("讀取原生目錄…", "Loading native catalog…"); + try + { + var value = await Task.Run(() => service.LoadCatalog(database, cancellation.Token)); + if (closing) + return; + catalog = value.Tables; + selected.Clear(); + if (options.InitialObjectName != null) + { + var matches = catalog.Where(t => (t.Key.Schema == "dbo" ? t.Key.Name : t.Key.Schema + "." + t.Key.Name) == options.InitialObjectName).ToList(); + if (matches.Count == 1) + selected.Add(matches[0].Key.ObjectId); + } + else + foreach (var table in catalog) + if (options.InitialTables == null || options.InitialTables.Any(k => k.Schema == table.Key.Schema && k.Name == table.Key.Name)) + selected.Add(table.Key.ObjectId); + loaded = true; + status.Text = selected.Count == 0 ? T("請選取資料表；初始物件可能已移除或名稱不唯一。", "Select tables; the initial object may be missing or ambiguous.") : T("估計筆數與大小來自目錄，可能與實際值不同。", "Catalog row and size estimates may differ from actual values."); + RefreshTables(); + } + catch (OperationCanceledException) + { + status.Text = T("已取消", "Canceled"); + } + catch (Exception) + { + status.Text = T("無法讀取目錄，請關閉後重試。", "Cannot load catalog. Close and retry."); + } + finally + { + EndOperation(); + } + } + + private async Task PreflightAsync() + { + InvalidateRequest(); + int version = generation; + var keys = catalog.Where(t => selected.Contains(t.Key.ObjectId)).Select(t => t.Key).ToList().AsReadOnly(); + var mode = Mode; + var include = views.Checked; + var path = output.Text; + BeginOperation(); + confirmation.Text = T("正在捕獲目的檔案狀態並預檢…", "Capturing destination and checking preflight…"); + try + { + var stamp = await Task.Run(() => MssqlExportArtifact.CaptureDestination(path, cancellation.Token)); + if (closing || version != generation) + return; + if (stamp.Exists && MessageBox.Show(this, T("目的檔案已存在。確認覆寫這個檔案？", "Destination exists. Replace this file?"), Text, MessageBoxButtons.YesNo, MessageBoxIcon.Warning) != DialogResult.Yes) + { + confirmation.Text = T("未同意覆寫。", "Replacement not approved."); + return; + } + + var request = new MssqlExportRequest(Guid.NewGuid(), database, keys, mode, include, path, stamp, options.Mirror); + var check = await Task.Run(() => service.Preflight(request, cancellation.Token)); + if (closing || version != generation) + return; + includedViews = check.Schema.Views.Count; + confirmation.Text = T("SQL 邏輯匯出；不包含登入帳號、作業、原生備份。", "Logical SQL export; excludes logins, jobs and native backups.") + "\r\n\r\n" + keys.Count + T(" 張資料表", " tables") + " · " + ModeText(mode) + "\r\n" + request.TargetPath + "\r\n" + (options.Mirror == null ? T("鏡像：未啟用", "Mirror: disabled") : T("鏡像：", "Mirror: ") + options.Mirror.Directory) + "\r\n" + (mode == MssqlExportMode.SchemaOnly ? T("僅結構：不開啟資料快照交易。", "Schema only: no data snapshot transaction.") : "SNAPSHOT: " + (check.SnapshotState == 1 ? "ON" : "OFF") + T("；全部資料使用同一交易。", "; all data uses one transaction.")) + "\r\n" + ViewScope(mode, include, check.Schema.Views.Count) + "\r\n" + T("計算欄位與 rowversion 值由目標重新產生。", "Computed and rowversion values are regenerated by the target.") + (mode == MssqlExportMode.SchemaOnly ? "" : "\r\n" + T("長時間 SNAPSHOT 可能增加來源 version store 使用量。", "Long SNAPSHOT exports may increase source version-store usage.")) + "\r\n\r\n" + string.Join("\r\n", check.Issues.Select(FormatIssue)); + if (check.CanStart && (mode == MssqlExportMode.SchemaOnly || check.SnapshotState == 1)) + prepared = request; + status.Text = prepared != null ? T("預檢通過；開始時會再次檢查來源。", "Preflight passed; source is checked again at start.") : T("預檢未通過，無法開始。", "Preflight failed; start is blocked."); + } + catch (OperationCanceledException) + { + status.Text = T("已取消", "Canceled"); + } + catch (Exception) + { + status.Text = T("目的路徑或預檢失敗，請返回檢查設定。", "Destination or preflight failed. Go back and check settings."); + } + finally + { + EndOperation(); + } + } + + private async Task ExportAsync() + { + if (prepared == null) + return; + var request = prepared; + int version = generation; + Result = null; + last = null; + log.Items.Clear(); + queue.Rows.Clear(); + processed.Clear(); + foreach (var t in catalog.Where(t => selected.Contains(t.Key.ObjectId))) + { + int row = queue.Rows.Add(t.Key.Schema + "." + t.Key.Name, t.Comment, T("等待中", "Waiting")); + queue.Rows[row].Tag = t.Key.ObjectId; + } + stage.Text = StageText(MssqlExportStage.Preflight); + status.Text = stage.Text; + current.Text = ""; + percent.Text = T("估算中", "Estimating"); + written.Text = T("已寫入 0 bytes", "Written 0 bytes"); + overall.Value = tableProgress.Value = 0; + overall.Style = tableProgress.Style = ProgressBarStyle.Marquee; + tableRows.Text = ""; + resultText.Text = ""; + step = 3; + ShowPage(); + BeginOperation(); + exporting = true; + clock.Restart(); + timer.Start(); + punky.BeginExport(); + UpdateButtons(); + var progress = new Progress(p => + { + if (!IsDisposed && !closing && busy && exporting && version == generation && p.JobId == request.JobId) + RenderProgress(p); + }); + try + { + var result = await Task.Run(() => service.Export(request, progress, cancellation.Token)); + if (result.JobId == request.JobId) + { + Result = result; + stage.Text = StageText(result.Stage); + status.Text = StageText(result.Stage); + foreach (DataGridViewRow row in queue.Rows) + { + bool done = processed.Contains((int)row.Tag) || result.RowCounts.Any(r => r.Key.ObjectId == (int)row.Tag) || result.Stage == MssqlExportStage.Completed; + row.Cells[2].Value = done ? T("已處理", "Processed") : result.Stage == MssqlExportStage.Canceled ? T("已取消", "Canceled") : T("未完成", "Incomplete"); + } + resultText.Text = T("已完成資料表：", "Completed tables: ") + (result.Stage == MssqlExportStage.Completed ? request.Tables.Count : Math.Max(processed.Count, result.RowCounts.Count)) + " / " + request.Tables.Count + T("；匯出列數：", "; exported rows: ") + result.RowCounts.Sum(r => r.Rows).ToString("N0") + "\r\n" + ViewScope(request.Mode, request.IncludeViews, includedViews) + "\r\n" + T("計算欄位與 rowversion 值由目標重新產生。", "Computed and rowversion values are regenerated by the target.") + (request.Mode == MssqlExportMode.SchemaOnly ? "" : "\r\n" + T("長時間 SNAPSHOT 可能增加來源 version store 使用量。", "Long SNAPSHOT exports may increase source version-store usage.")) + "\r\n" + (result.PublishedPath == null ? T("沒有已發布的本機檔案。", "No local file was published.") : T("已發布：", "Published: ") + result.PublishedPath) + "\r\n" + (result.MirrorPath ?? "") + "\r\n" + string.Join("\r\n", result.Warnings.Concat(result.ResidualPaths)) + "\r\n" + (result.ErrorMessage ?? "") + "\r\n" + T("檔案驗證不等同還原驗收。請使用支援此檔案大小的客戶端，以 UTF-8、停用變數替代、遇錯停止，於同一工作階段還原。詳見 docs/MSSQL_SQL_EXPORT.md。", "File verification is not restore acceptance. Use a client supporting this file size: UTF-8, variable substitution disabled, stop on error, and one restore session. See docs/MSSQL_SQL_EXPORT.md."); + if (result.Stage == MssqlExportStage.Completed) + { + overall.Style = ProgressBarStyle.Continuous; + overall.Value = 100; + percent.Text = "100%"; + } + } + } + catch (Exception) + { + status.Text = T("匯出作業失敗。", "Export failed."); + stage.Text = StageText(MssqlExportStage.Failed); + } + finally + { + timer.Stop(); + clock.Stop(); + punky.EndExport(); + EndOperation(); + } + } + + private void RenderProgress(MssqlExportProgress p) + { + if (last?.CurrentTable != null && p.CompletedTables > last.CompletedTables) + processed.Add(last.CurrentTable.ObjectId); + last = p; + stage.Text = cancelRequested ? StageText(MssqlExportStage.CancelRequested) : StageText(p.Stage); + if (!cancelRequested) + status.Text = stage.Text; + current.Text = p.CurrentTable == null ? "" : p.CurrentTable.Schema + "." + p.CurrentTable.Name; + percent.Text = p.Percent.HasValue ? T("估計 ", "Estimated ") + Math.Min(99, p.Percent.Value).ToString("0") + "%" : T("估算中", "Estimating"); + overall.Style = p.Percent.HasValue ? ProgressBarStyle.Continuous : ProgressBarStyle.Marquee; + if (p.Percent.HasValue) + overall.Value = (int)Math.Max(0, Math.Min(99, p.Percent.Value)); + written.Text = T("已寫入 ", "Written ") + p.WrittenBytes.ToString("N0") + " bytes · " + T("資料表 ", "Tables ") + p.CompletedTables + " / " + p.TotalTables + " · " + T("列數 ", "Rows ") + p.Rows.ToString("N0") + " / " + (p.EstimatedRows?.ToString("N0") ?? "—") + T("（估計）", " (estimated)"); + var table = catalog.FirstOrDefault(t => t.Key.ObjectId == p.CurrentTable?.ObjectId); + var estimate = table?.EstimatedRows; + tableRows.Text = T("目前資料表：", "Current table: ") + p.CurrentTableRows.ToString("N0") + " / " + (estimate?.ToString("N0") ?? "—"); + tableProgress.Style = estimate > 0 ? ProgressBarStyle.Continuous : ProgressBarStyle.Marquee; + if (estimate > 0) + tableProgress.Value = (int)Math.Min(100, 100.0 * p.CurrentTableRows / estimate.Value); + if (p.CurrentTable == null && p.PhaseTotalBytes > 0) + { + tableRows.Text = T("本階段檔案處理：", "Current file phase: ") + p.PhaseProcessedBytes.ToString("N0") + " / " + p.PhaseTotalBytes.Value.ToString("N0") + " bytes"; + tableProgress.Style = ProgressBarStyle.Continuous; + tableProgress.Value = (int)Math.Min(100, 100.0 * p.PhaseProcessedBytes / p.PhaseTotalBytes.Value); + } + for (int i = 0; i < queue.Rows.Count; i++) + queue.Rows[i].Cells[2].Value = processed.Contains((int)queue.Rows[i].Tag) ? T("已處理", "Processed") : p.CurrentTable != null && (int)queue.Rows[i].Tag == p.CurrentTable.ObjectId ? T("匯出中", "Exporting") : T("等待中", "Waiting"); + if (!string.IsNullOrWhiteSpace(p.Message)) + { + log.Items.Add(p.Message); + while (log.Items.Count > MssqlExportLimits.ProgressLogLimit) + log.Items.RemoveAt(0); + log.TopIndex = log.Items.Count - 1; + } + + UpdateClock(); + } + + private void UpdateClock() + { + elapsed.Text = T("已經過：", "Elapsed: ") + FormatDuration(clock.Elapsed); + bool stale = last == null || clock.Elapsed - last.LastProgressElapsed >= TimeSpan.FromSeconds(5) || cancelRequested; + speed.Text = T("本階段速度：", "Phase rate: ") + (!stale && last.BytesPerSecond.HasValue ? last.BytesPerSecond.Value.ToString("N0") + " bytes/s" : "—"); + eta.Text = T("本階段預估：", "Phase ETA: ") + (!stale && last.Eta.HasValue ? FormatDuration(last.Eta.Value) : T("等待／估算中", "Waiting / estimating")); + } + + internal static string FormatDuration(TimeSpan duration) + { + return ((long)Math.Floor(duration.TotalHours)).ToString("00") + ":" + duration.Minutes.ToString("00") + ":" + duration.Seconds.ToString("00"); + } + + private static string FormatIssue(MssqlExportIssue issue) + { + string scope = issue.ObjectKey == null ? T("資料庫", "Database") : issue.ObjectKey.Schema + "." + issue.ObjectKey.Name; + if (issue.ColumnName != null) scope += "." + issue.ColumnName; + return scope + " [" + issue.Code + "]: " + issue.Message; + } + + private static string ViewScope(MssqlExportMode mode, bool include, int included) + { + return mode == MssqlExportMode.DataOnly || !include + ? T("檢視表：包含 0；全部排除（僅匯出資料表）。", "Views: 0 included; all excluded (tables only).") + : T("檢視表：包含 ", "Views: ") + included + T("；排除 0（僅定義，不含資料）。", " included; 0 excluded (definitions only, no data)."); + } + + private void RequestCancel() + { + if (cancellation == null || cancelRequested) + return; + cancelRequested = true; + cancellation.Cancel(); + status.Text = T("正在取消，等待查詢停止及清理完成…", "Canceling; waiting for query and cleanup…"); + stage.Text = StageText(MssqlExportStage.CancelRequested); + UpdateButtons(); + UpdateClock(); + } + + private static string ModeText(MssqlExportMode mode) + { + return mode == MssqlExportMode.SchemaOnly ? T("僅結構", "Schema only") : mode == MssqlExportMode.DataOnly ? T("僅資料", "Data only") : T("結構與資料", "Schema and data"); + } + + private static string StageText(MssqlExportStage value) + { + switch (value) + { + case MssqlExportStage.Preflight: + return T("準備與預檢", "Preparing and checking"); + case MssqlExportStage.Schema: + return T("正在匯出結構", "Exporting schema"); + case MssqlExportStage.Data: + return T("正在匯出資料", "Exporting data"); + case MssqlExportStage.Verifying: + return T("正在驗證檔案", "Verifying file"); + case MssqlExportStage.Publishing: + return T("正在發布檔案", "Publishing file"); + case MssqlExportStage.Mirroring: + return T("正在複製並驗證鏡像", "Copying and verifying mirror"); + case MssqlExportStage.CancelRequested: + return T("正在取消，等待清理", "Canceling; waiting for cleanup"); + case MssqlExportStage.Completed: + return T("匯出完成", "Export completed"); + case MssqlExportStage.Canceled: + return T("匯出已取消", "Export canceled"); + case MssqlExportStage.LocalCompletedMirrorIncomplete: + return T("本機完成；鏡像未完成", "Local completed; mirror incomplete"); + default: + return T("匯出失敗", "Export failed"); + } + } + + private void Open(string path) + { + if (path == null) + return; + try + { + Process.Start(new ProcessStartInfo(path) { UseShellExecute = true }); + } + catch (Exception) + { + status.Text = T("無法開啟路徑。", "Cannot open path."); + } + } + + protected override void Dispose(bool disposing) + { + if (disposing) + { + timer.Dispose(); + } + + base.Dispose(disposing); + } + } +} diff --git a/mySQLPunk/Properties/AssemblyInfo.cs b/mySQLPunk/Properties/AssemblyInfo.cs index cb14305..63985c6 100644 --- a/mySQLPunk/Properties/AssemblyInfo.cs +++ b/mySQLPunk/Properties/AssemblyInfo.cs @@ -5,6 +5,7 @@ // 組件的一般資訊是由下列的屬性集控制。 // 變更這些屬性的值即可修改組件的相關 // 資訊。 +[assembly: InternalsVisibleTo("mySQLPunk.SmokeTests")] [assembly: AssemblyTitle("mySQLPunk")] [assembly: AssemblyDescription("免費開源的多資料庫管理工具：MySQL / MariaDB、PostgreSQL、SQL Server、Oracle、SQLite")] [assembly: AssemblyConfiguration("")] diff --git a/mySQLPunk/lib/ConnectionOpenService.cs b/mySQLPunk/lib/ConnectionOpenService.cs index 0ffdf60..fedfba0 100644 --- a/mySQLPunk/lib/ConnectionOpenService.cs +++ b/mySQLPunk/lib/ConnectionOpenService.cs @@ -2,6 +2,8 @@ using System.Collections.Generic; using System.IO; using MySqlConnector; +using System.Data.SqlClient; +using mySQLPunk.lib.MssqlExport; namespace mySQLPunk.lib { @@ -11,8 +13,16 @@ public sealed class ConnectionOpenResult public List Databases { get; private set; } public string ConnectionString { get; private set; } + internal SqlConnection NativeMssqlConnection { get; private set; } + public ConnectionOpenResult(IDatabase database, List databases, string connectionString = null) + : this(database, databases, connectionString, null) + { + } + + internal ConnectionOpenResult(IDatabase database, List databases, string connectionString, SqlConnection nativeMssqlConnection) { + NativeMssqlConnection = nativeMssqlConnection; Database = database; Databases = databases ?? new List(); ConnectionString = connectionString ?? string.Empty; @@ -68,7 +78,7 @@ public static ConnectionOpenResult Open(Dictionary connection, b database.SetConn(connectionString); database.Open(); List databases = loadDatabases ? database.GetDatabases() : new List(); - return new ConnectionOpenResult(database, databases, connectionString); + return new ConnectionOpenResult(database, databases, connectionString, (rawDatabase as my_mssql)?.MCT); } catch { @@ -78,6 +88,24 @@ public static ConnectionOpenResult Open(Dictionary connection, b } } + internal static MssqlOwnedSession OpenMssqlSession(Dictionary connection) + { + if (connection == null) throw new ArgumentNullException(nameof(connection)); + string provider = ConnectionConfigurationService.NormalizeProvider(ConnectionConfigurationService.GetValue(connection, "db_kind")); + if (provider != "mssql") throw new ArgumentException("MSSQL export requires a SQL Server connection", nameof(connection)); + ConnectionOpenResult result = Open(connection, false); + try + { + if (result.NativeMssqlConnection == null) throw new InvalidOperationException("SQL Server provider connection is unavailable"); + return new MssqlOwnedSession(result.Database, result.NativeMssqlConnection); + } + catch + { + try { result.Database.Dispose(); } catch { } + throw; + } + } + private static void ValidateSecurityCombination(Dictionary connection) { if (!ConnectionSecuritySettingsService.IsTrue(connection, "ssh_enabled")) return; diff --git a/mySQLPunk/lib/MssqlExport/MssqlDataScriptWriter.cs b/mySQLPunk/lib/MssqlExport/MssqlDataScriptWriter.cs new file mode 100644 index 0000000..16c4412 --- /dev/null +++ b/mySQLPunk/lib/MssqlExport/MssqlDataScriptWriter.cs @@ -0,0 +1,175 @@ +using System; +using System.Collections.Generic; +using System.Data.Common; +using System.IO; +using System.Linq; +using System.Text; +using System.Threading; + +namespace mySQLPunk.lib.MssqlExport +{ + // Consumes this same reader synchronously, finishes exactly one row, and returns + // additional source bytes consumed. It must not Read(), reopen, rewind, or own the reader. + // The callback reports cumulative additional bytes after actual reads, excluding the + // retained prefix; WriteTable keeps completed rows unchanged until CommitRow succeeds. + internal interface IMssqlLargeRowWriter + { + long WriteRow(DbDataReader reader, MssqlTableDefinition table, IReadOnlyList columns, + MssqlRowContinuation row, MssqlScriptBatchWriter output, CancellationToken cancellation, Action onAdditionalBytes); + } + internal sealed class MssqlRowContinuation + { + private readonly char[] buffer; + private readonly int[] starts, lengths; + internal int Ordinal { get; } + internal long CurrentByteOffset { get; } + internal bool CurrentValueStarted { get; } + internal bool CurrentValueComplete { get; } + internal int BufferBytes { get { return buffer.Length * sizeof(char); } } + internal MssqlRowContinuation(char[] buffer, int[] starts, int[] lengths, int ordinal, long offset, bool started, bool complete) + { this.buffer = buffer; this.starts = starts; this.lengths = lengths; Ordinal = ordinal; CurrentByteOffset = offset; CurrentValueStarted = started; CurrentValueComplete = complete; } + // Completed columns contain full expressions; an open binary column contains prefix+hex, + // without suffix. No source bytes are pending outside this buffer. Valid only during callback. + internal int ValueLength(int ordinal) + { CheckOrdinal(ordinal); return lengths[ordinal]; } + internal int CopyValueChars(int ordinal, int sourceOffset, char[] destination, int destinationOffset, int count) + { + CheckOrdinal(ordinal); + if (sourceOffset < 0 || sourceOffset > lengths[ordinal]) throw new ArgumentOutOfRangeException(nameof(sourceOffset)); + int copied = Math.Min(count, lengths[ordinal] - sourceOffset); + Array.Copy(buffer, starts[ordinal] + sourceOffset, destination, destinationOffset, copied); + return copied; + } + private void CheckOrdinal(int ordinal) + { if (ordinal < 0 || ordinal > Ordinal) throw new ArgumentOutOfRangeException(nameof(ordinal)); } + } + internal sealed class MssqlScriptBatchWriter + { + private readonly TextWriter output; + private readonly Encoding encoding; + private int batchBytes; + internal MssqlScriptBatchWriter(TextWriter output) + { this.output = output ?? throw new ArgumentNullException(nameof(output)); encoding = output is StreamWriter ? output.Encoding : new UTF8Encoding(false, true); } + internal int Bytes(string text) { return encoding.GetByteCount(text); } + internal void WriteStatement(string statement) + { WriteBoundedStatement(statement, MssqlExportLimits.StatementBytes); } + // Only LOB setup/final INSERT metadata may use this path: never inline row values. + // 1024 columns with 128-character escaped Unicode identifiers plus guards fit 2 MiB. + internal void WriteMetadataStatement(string statement) + { WriteBoundedStatement(statement, 2 * 1024 * 1024); } + private void WriteBoundedStatement(string statement, int limit) + { + int bytes = Bytes(statement) + Bytes("\r\n"); + if (bytes > limit) throw new InvalidOperationException("Statement exceeds the bounded script budget."); + if (batchBytes + bytes + Bytes("GO\r\n") > MssqlExportLimits.BatchBytes) Flush(); + output.Write(statement); output.Write("\r\n"); batchBytes += bytes; + } + internal bool TryWriteRow(string header, char[] values, int[] starts, int[] lengths) + { + int bytes = Bytes(header) + Bytes(lengths.Length == 0 ? "\r\n" : ");\r\n"); + for (int i = 0; i < lengths.Length; i++) bytes += encoding.GetByteCount(values, starts[i], lengths[i]) + (i == 0 ? 0 : Bytes(",")); + if (bytes > MssqlExportLimits.StatementBytes) return false; + if (batchBytes + bytes + Bytes("GO\r\n") > MssqlExportLimits.BatchBytes) Flush(); + output.Write(header); + for (int i = 0; i < lengths.Length; i++) { if (i > 0) output.Write(","); output.Write(values, starts[i], lengths[i]); } + output.Write(lengths.Length == 0 ? "\r\n" : ");\r\n"); + batchBytes += bytes; + return true; + } + internal void Flush() { if (batchBytes == 0) return; output.Write("GO\r\n"); batchBytes = 0; } + } + internal sealed class MssqlDataScriptWriter + { + private readonly IMssqlLargeRowWriter largeRows; + // A worker reuses new MssqlDataScriptWriter(new MssqlLobScriptWriter(request.JobId)) + // across every table in one output. The default also supports large values safely. + internal MssqlDataScriptWriter(IMssqlLargeRowWriter largeRows = null) { this.largeRows = largeRows ?? new MssqlLobScriptWriter(Guid.NewGuid()); } + internal long WriteTable(DbDataReader reader, MssqlTableDefinition table, TextWriter output, + Action onRowsAndSourceBytes, CancellationToken cancellation) + { + if (reader == null) throw new ArgumentNullException(nameof(reader)); + if (table == null) throw new ArgumentNullException(nameof(table)); + var columns = table.Columns.Where(c => !c.Computed && !c.RowVersion).OrderBy(c => c.Ordinal).ToList().AsReadOnly(); + string target = MssqlValueCodec.QuoteIdentifier(table.Key.Schema) + "." + MssqlValueCodec.QuoteIdentifier(table.Key.Name); + string header = "INSERT INTO " + target + (columns.Count == 0 ? " DEFAULT VALUES;" : " (" + string.Join(",", columns.Select(c => MssqlValueCodec.QuoteIdentifier(c.Name))) + ") VALUES ("); + var batches = new MssqlScriptBatchWriter(output); + // Buffer plus scratch and column offsets stay under SmallRowBytes. The encoded + // statement is separately measured; conservative capacity is not a source limit. + int capacity = (MssqlExportLimits.SmallRowBytes - 8192 - columns.Count * 8) / 2; + if (capacity < 1024) throw new NotSupportedException("Column metadata exceeds row buffer budget."); + var buffer = new char[capacity]; var chunk = new byte[4096]; + var starts = new int[columns.Count]; var lengths = new int[columns.Count]; + // Legacy text rejects direct varbinary assignment even for tiny values. + // Typed staging preserves its source codepage without a default-collation cast. + bool requiresTypedStage = columns.Any(c => c.NativeType == "text"); + long rows = 0, sourceBytes = 0; bool identity = columns.Any(c => c.Identity), identityOn = false; + cancellation.ThrowIfCancellationRequested(); + while (reader.Read()) + { + cancellation.ThrowIfCancellationRequested(); + if (identity && !identityOn) { batches.WriteStatement("SET IDENTITY_INSERT " + target + " ON;"); identityOn = true; } + int used = 0; bool handedOff = false; + if (columns.Count > 0 && (requiresTypedStage || batches.Bytes(header) + 4 > MssqlExportLimits.StatementBytes)) + { + sourceBytes += Handoff(reader, table, columns, new MssqlRowContinuation(buffer, starts, lengths, 0, 0, false, false), batches, cancellation, rows, sourceBytes, onRowsAndSourceBytes); + handedOff = true; + } + for (int i = 0; i < columns.Count && !handedOff; i++) + { + cancellation.ThrowIfCancellationRequested(); + starts[i] = used; lengths[i] = 0; + var column = columns[i]; + bool binary = MssqlValueCodec.IsBinaryProjected(column); + // Fixed-width scalar expressions are bounded (<100 chars). Do not consume + // the next column when there is insufficient room; Task5 starts it itself. + if (capacity - used < 128) + { sourceBytes += Handoff(reader, table, columns, new MssqlRowContinuation(buffer, starts, lengths, i, 0, false, false), batches, cancellation, rows, sourceBytes, onRowsAndSourceBytes); handedOff = true; break; } + if (reader.IsDBNull(i)) { Append(buffer, ref used, "NULL"); lengths[i] = used - starts[i]; continue; } + if (!binary) + { + Append(buffer, ref used, MssqlValueCodec.FormatScalar(MssqlValueCodec.ReadScalar(reader, i, column), column)); + sourceBytes += MssqlValueCodec.ScalarBytes(column); lengths[i] = used - starts[i]; continue; + } + string suffix = MssqlValueCodec.BinarySuffix(column); + Append(buffer, ref used, MssqlValueCodec.BinaryPrefix(column)); + long offset = 0; + while (true) + { + cancellation.ThrowIfCancellationRequested(); + int count = Math.Min(chunk.Length, (capacity - used - suffix.Length) / 2); + if (count == 0) + { + lengths[i] = used - starts[i]; + sourceBytes += Handoff(reader, table, columns, new MssqlRowContinuation(buffer, starts, lengths, i, offset, true, false), batches, cancellation, rows, sourceBytes, onRowsAndSourceBytes); handedOff = true; break; + } + int read = checked((int)reader.GetBytes(i, offset, chunk, 0, count)); + if (read == 0) { Append(buffer, ref used, suffix); break; } + for (int j = 0; j < read; j++) { buffer[used++] = "0123456789ABCDEF"[chunk[j] >> 4]; buffer[used++] = "0123456789ABCDEF"[chunk[j] & 15]; } + offset += read; sourceBytes += read; + } + lengths[i] = used - starts[i]; + if (handedOff) break; + } + if (!handedOff) + { + if (!batches.TryWriteRow(header, buffer, starts, lengths)) + { + int last = columns.Count - 1; + sourceBytes += Handoff(reader, table, columns, new MssqlRowContinuation(buffer, starts, lengths, last, 0, true, true), batches, cancellation, rows, sourceBytes, onRowsAndSourceBytes); + } + } + rows++; onRowsAndSourceBytes?.Invoke(rows, sourceBytes); + cancellation.ThrowIfCancellationRequested(); + } + if (identityOn) batches.WriteStatement("SET IDENTITY_INSERT " + target + " OFF;"); + batches.Flush(); + return rows; + } + private long Handoff(DbDataReader reader, MssqlTableDefinition table, IReadOnlyList columns, MssqlRowContinuation row, MssqlScriptBatchWriter output, CancellationToken cancellation, long completedRows, long priorBytes, Action onProgress) + { + return largeRows.WriteRow(reader, table, columns, row, output, cancellation, added => onProgress?.Invoke(completedRows, priorBytes + added)); + } + private static void Append(char[] buffer, ref int used, string value) + { value.CopyTo(0, buffer, used, value.Length); used += value.Length; } + } +} diff --git a/mySQLPunk/lib/MssqlExport/MssqlExportArtifact.cs b/mySQLPunk/lib/MssqlExport/MssqlExportArtifact.cs new file mode 100644 index 0000000..f9b5a06 --- /dev/null +++ b/mySQLPunk/lib/MssqlExport/MssqlExportArtifact.cs @@ -0,0 +1,210 @@ +using System; +using System.Collections.Concurrent; +using System.Collections.Generic; +using System.ComponentModel; +using System.Globalization; +using System.IO; +using System.Runtime.InteropServices; +using System.Security.Cryptography; +using System.Text; +using System.Threading; +using Microsoft.Win32.SafeHandles; + +namespace mySQLPunk.lib.MssqlExport +{ + // These gates coordinate this process only, not external writers or path aliases. + internal static class MssqlArtifactGates + { + private static readonly ConcurrentDictionary Gates = new ConcurrentDictionary(StringComparer.OrdinalIgnoreCase); + internal static SemaphoreSlim Get(string path) => Gates.GetOrAdd(Path.GetFullPath(path).TrimEnd('\\', '/'), _ => new SemaphoreSlim(1, 1)); + } + internal class MssqlArtifactFiles + { + internal virtual Stream Create(string path) => new FileStream(path, FileMode.CreateNew, FileAccess.Write, FileShare.None, 81920); + internal virtual void Move(string source, string target) => File.Move(source, target); + internal virtual void Replace(string source, string target, string recovery) => File.Replace(source, target, recovery, false); + internal virtual void Delete(string path) => File.Delete(path); + internal virtual void Prune(string directory, int count) => BackupRemoteMirrorService.PruneRemoteBackups(directory, count); + } + // Cancellation and entry to the short commit region have a single lock order. + // A request arriving after entry cannot turn a successful publication into "nothing output". + internal sealed class MssqlPublicationCommit : IDisposable + { + private readonly object sync = new object(); + private readonly CancellationToken token; + private readonly CancellationTokenRegistration registration; + private bool canceled, entered; + internal MssqlPublicationCommit(CancellationToken token) + { + this.token = token; + registration = token.Register(() => { lock (sync) { if (!entered) canceled = true; } }); + } + internal void Enter() { lock (sync) { if (canceled || token.IsCancellationRequested) throw new OperationCanceledException(token); entered = true; } } + public void Dispose() => registration.Dispose(); + } + internal sealed class MssqlExportArtifact : IDisposable + { + internal const int BufferSize = 81920; + private readonly MssqlExportRequest request; + private readonly MssqlArtifactFiles files; + private readonly HashingOutput output; + private readonly List residuals = new List(); + private readonly List warnings = new List(); + private MssqlArtifactReceipt receipt; + private bool verified, preserveCandidates, disposed; + internal string TemporaryPath { get; } + internal string RecoveryPath { get; private set; } + internal string PublishedPath { get; private set; } + internal IReadOnlyList ResidualPaths => new List(residuals).AsReadOnly(); + internal IReadOnlyList Warnings => new List(warnings).AsReadOnly(); + internal Stream Output => output; + private MssqlExportArtifact(MssqlExportRequest request, MssqlArtifactFiles files) + { + this.request = request ?? throw new ArgumentNullException(nameof(request)); this.files = files; + TemporaryPath = Path.Combine(Path.GetDirectoryName(request.TargetPath), ".mysqlpunk-export-" + request.JobId.ToString("N") + "-" + Guid.NewGuid().ToString("N") + ".partial"); + output = new HashingOutput(files.Create(TemporaryPath)); + } + internal static MssqlExportArtifact Create(MssqlExportRequest request) => Create(request, new MssqlArtifactFiles()); + internal static MssqlExportArtifact Create(MssqlExportRequest request, MssqlArtifactFiles files) => new MssqlExportArtifact(request, files); + internal static ExportDestinationStamp CaptureDestination(string path, CancellationToken cancellation) + { + string full = Path.GetFullPath(path); var gate = MssqlArtifactGates.Get(full); gate.Wait(cancellation); + try { return CaptureCore(full, cancellation); } finally { gate.Release(); } + } + private static ExportDestinationStamp CaptureCore(string full, CancellationToken cancellation) + { + cancellation.ThrowIfCancellationRequested(); + FileStream stream; + try { stream = new FileStream(full, FileMode.Open, FileAccess.Read, FileShare.Read, BufferSize); } + catch (FileNotFoundException) { return new ExportDestinationStamp(false, full, null, 0, DateTime.MinValue, null); } + using (stream) + { + FileInformation info; + if (!GetFileInformationByHandle(stream.SafeFileHandle, out info)) throw new Win32Exception(Marshal.GetLastWin32Error()); + string identity = info.Volume.ToString("X8") + ":" + info.IndexHigh.ToString("X8") + info.IndexLow.ToString("X8"); + long length; string hash = Hash(stream, null, cancellation, out length); + long timestamp = ((long)info.WriteHigh << 32) | info.WriteLow; + return new ExportDestinationStamp(true, full, identity, length, DateTime.FromFileTimeUtc(timestamp), hash); + } + } + internal MssqlArtifactReceipt Complete(IReadOnlyList rows, CancellationToken cancellation) + { + if (disposed || receipt != null) throw new InvalidOperationException("Artifact is already closed"); + if (rows == null) throw new ArgumentNullException(nameof(rows)); + long total = 0; foreach (var row in rows) { if (row == null) throw new ArgumentException("Null row count"); total = checked(total + row.Rows); } + cancellation.ThrowIfCancellationRequested(); + string marker = "\r\n-- mysqlpunk-export-complete job=" + request.JobId.ToString("N") + " tables=" + rows.Count.ToString(CultureInfo.InvariantCulture) + " rows=" + total.ToString(CultureInfo.InvariantCulture) + "\r\n"; + byte[] bytes = new UTF8Encoding(false, true).GetBytes(marker); output.Write(bytes, 0, bytes.Length); + output.Finish(); + receipt = new MssqlArtifactReceipt(request.JobId, TemporaryPath, request.TargetPath, output.Bytes, output.Digest, rows, marker); + return receipt; + } + internal void Verify(MssqlArtifactReceipt candidate, Action progress, CancellationToken cancellation) + { + verified = false; + if (disposed || receipt == null || !ReferenceEquals(candidate, receipt)) throw new InvalidOperationException("Complete this artifact before verifying"); + VerifyFile(TemporaryPath, candidate, progress, cancellation); verified = true; + } + internal static void VerifyFile(string path, MssqlArtifactReceipt expected, Action progress, CancellationToken cancellation) + { + using (var stream = new FileStream(path, FileMode.Open, FileAccess.Read, FileShare.Read, BufferSize)) + { + long length; string hash = Hash(stream, progress, cancellation, out length); + if (length != expected.Bytes || !string.Equals(hash, expected.Sha256, StringComparison.Ordinal)) throw new InvalidDataException("Export length or SHA-256 mismatch"); + } + } + internal static string Hash(Stream stream, Action progress, CancellationToken cancellation, out long length) + { + byte[] buffer = new byte[BufferSize]; length = 0; long total = stream.Length; + using (var hash = SHA256.Create()) + { + int read; + while (true) + { + cancellation.ThrowIfCancellationRequested(); read = stream.Read(buffer, 0, buffer.Length); if (read == 0) break; + hash.TransformBlock(buffer, 0, read, null, 0); length = checked(length + read); progress?.Invoke(length, total); + } + cancellation.ThrowIfCancellationRequested(); hash.TransformFinalBlock(buffer, 0, 0); return BitConverter.ToString(hash.Hash).Replace("-", ""); + } + } + internal string Publish(ExportDestinationStamp expectedDestination, CancellationToken cancellation) + { + if (disposed || !verified || PublishedPath != null) throw new InvalidOperationException("Artifact must be verified and unpublished"); + var approved = request.Destination; + if (approved == null || !string.Equals(approved.FullPath, request.TargetPath, StringComparison.OrdinalIgnoreCase) || !SameDestination(expectedDestination, approved)) throw new ArgumentException("Consent stamp does not match the immutable request"); + var gate = MssqlArtifactGates.Get(request.TargetPath); gate.Wait(cancellation); + try + { + VerifyFile(TemporaryPath, receipt, null, cancellation); + var actual = CaptureCore(request.TargetPath, cancellation); + if (!SameDestination(actual, approved)) throw new IOException("Destination changed after overwrite consent"); + using (var commit = new MssqlPublicationCommit(cancellation)) + { + commit.Enter(); + if (actual.Exists) + { + RecoveryPath = Path.Combine(Path.GetDirectoryName(request.TargetPath), ".mysqlpunk-recovery-" + Guid.NewGuid().ToString("N") + ".recovery"); + try { files.Replace(TemporaryPath, request.TargetPath, RecoveryPath); } + catch { preserveCandidates = true; ObserveCandidate(TemporaryPath); ObserveCandidate(RecoveryPath); ObserveCandidate(request.TargetPath); throw; } + } + else files.Move(TemporaryPath, request.TargetPath); + PublishedPath = request.TargetPath; + } + // A successful replace retains the old version for explicit recovery, not retention pruning. + if (RecoveryPath != null) ObserveCandidate(RecoveryPath); + return PublishedPath; + } + finally { gate.Release(); } + } + private static bool SameDestination(ExportDestinationStamp left, ExportDestinationStamp right) + { + return left != null && right != null && string.Equals(left.FullPath, right.FullPath, StringComparison.OrdinalIgnoreCase) + && left.Exists == right.Exists && left.FileIdentity == right.FileIdentity && left.Length == right.Length + && left.LastWriteTimeUtc == right.LastWriteTimeUtc && left.Sha256 == right.Sha256; + } + private void ObserveCandidate(string path) + { + try { File.GetAttributes(path); if (!residuals.Contains(path)) residuals.Add(path); } + catch (FileNotFoundException) { } + catch (DirectoryNotFoundException) { } + catch (Exception ex) { if (!residuals.Contains(path)) residuals.Add(path); warnings.Add("Candidate state unknown: " + path + ": " + ex.Message); } + } + public void Dispose() + { + if (disposed) return; disposed = true; + try { output.Dispose(); } catch (Exception ex) { warnings.Add("Close failed: " + ex.Message); } + if (PublishedPath != null || preserveCandidates) return; + try { files.Delete(TemporaryPath); } catch (Exception ex) { residuals.Add(TemporaryPath); warnings.Add("Temporary cleanup failed: " + ex.Message); } + } + [StructLayout(LayoutKind.Sequential)] private struct FileInformation + { + internal uint Attributes, CreateLow, CreateHigh, AccessLow, AccessHigh, WriteLow, WriteHigh, Volume, SizeHigh, SizeLow, Links, IndexHigh, IndexLow; + } + [DllImport("kernel32.dll", SetLastError = true)] private static extern bool GetFileInformationByHandle(SafeFileHandle handle, out FileInformation info); + private sealed class HashingOutput : Stream + { + private readonly Stream inner; private readonly SHA256 hash = SHA256.Create(); private bool closed, faulted; + internal long Bytes { get; private set; } internal string Digest { get; private set; } + internal HashingOutput(Stream inner) { this.inner = inner; } + public override void Write(byte[] buffer, int offset, int count) + { + if (closed || faulted) throw new InvalidOperationException("Output is closed or faulted"); + try { inner.Write(buffer, offset, count); hash.TransformBlock(buffer, offset, count, null, 0); Bytes = checked(Bytes + count); } + catch { faulted = true; throw; } + } + internal void Finish() + { + if (closed || faulted) throw new InvalidOperationException("Cannot complete faulted output"); + try { inner.Flush(); inner.Dispose(); closed = true; hash.TransformFinalBlock(new byte[0], 0, 0); Digest = BitConverter.ToString(hash.Hash).Replace("-", ""); } + catch { faulted = true; throw; } + } + public override void Flush() { try { inner.Flush(); } catch { faulted = true; throw; } } + protected override void Dispose(bool disposing) { if (disposing) { try { if (!closed) inner.Dispose(); } finally { closed = true; hash.Dispose(); } } base.Dispose(disposing); } + public override bool CanRead => false; public override bool CanSeek => false; public override bool CanWrite => !closed && !faulted; + public override long Length => Bytes; public override long Position { get => Bytes; set => throw new NotSupportedException(); } + public override int Read(byte[] buffer, int offset, int count) => throw new NotSupportedException(); + public override long Seek(long offset, SeekOrigin origin) => throw new NotSupportedException(); + public override void SetLength(long value) => throw new NotSupportedException(); + } + } +} diff --git a/mySQLPunk/lib/MssqlExport/MssqlExportMetadataReader.cs b/mySQLPunk/lib/MssqlExport/MssqlExportMetadataReader.cs new file mode 100644 index 0000000..868b78d --- /dev/null +++ b/mySQLPunk/lib/MssqlExport/MssqlExportMetadataReader.cs @@ -0,0 +1,226 @@ +using System; +using System.Collections.Generic; +using System.Data; +using System.Data.SqlClient; +using System.Globalization; +using System.Linq; +using System.Threading; + +namespace mySQLPunk.lib.MssqlExport +{ + internal sealed class MssqlExportMetadataReader + { + internal static string Quote(string name) { return "[" + name.Replace("]", "]]") + "]"; } + internal static string Qualified(MssqlObjectKey key) { return Quote(key.Schema) + "." + Quote(key.Name); } + internal MssqlExportCatalog ReadCatalog(SqlConnection connection, string databaseName, CancellationToken cancellation) + { + cancellation.ThrowIfCancellationRequested(); + connection.ChangeDatabase(databaseName); + EnsureVisibility(connection, null, cancellation); + var tables = ReadTables(connection, null, cancellation); + var issues = tables.Where(t => !t.Flag("can_select")).Select(t => new MssqlExportIssue("UnreadableTable", t.Key, null, "SELECT permission is required.")); + return new MssqlExportCatalog(tables, Snapshot(connection, null, cancellation), issues); + } + internal MssqlExportSchema ReadSchema(SqlConnection connection, SqlTransaction transaction, MssqlExportRequest request, CancellationToken cancellation) + { + EnsureVisibility(connection, transaction, cancellation); + var issues = new List(); + var selected = new List(); + var catalog = ReadTables(connection, transaction, cancellation); + foreach (var key in request.Tables) + { + var table = catalog.SingleOrDefault(t => t.Key.ObjectId == key.ObjectId && t.Key.Schema == key.Schema && t.Key.Name == key.Name); + if (table == null) { issues.Add(new MssqlExportIssue("ObjectChanged", key, null, "Selected table is missing, renamed or replaced.")); continue; } + string id = key.ObjectId.ToString(CultureInfo.InvariantCulture); + var columnSql = @"SELECT c.column_id,c.name,ty.name type_name,SCHEMA_NAME(ty.schema_id) type_schema,ty.is_user_defined,ty.is_assembly_type, +c.max_length,c.precision,c.scale,c.collation_name,c.is_nullable,c.is_ansi_padded,c.is_identity,c.is_computed,c.is_filestream,c.is_sparse,c.is_column_set,c.is_rowguidcol,c.xml_collection_id,c.is_xml_document,c.rule_object_id, +CONVERT(nvarchar(100),ic.seed_value) seed_value,CONVERT(nvarchar(100),ic.increment_value) increment_value,ic.is_not_for_replication identity_not_for_replication, +cc.definition computed_definition,cc.is_persisted,dc.name default_name,dc.definition default_definition, +c.default_object_id,CONVERT(nvarchar(max),ep.value) comment," + Optional(connection, transaction, "sys.columns", "c", cancellation, "generated_always_type", "encryption_type", "is_hidden", "vector_dimensions") + "," + (HasCatalog(connection, transaction, "masked_columns", cancellation) ? "(SELECT MAX(CONVERT(int,mc.is_masked)) FROM sys.masked_columns mc WHERE mc.object_id=c.object_id AND mc.column_id=c.column_id)" : "0") + " is_masked" + @" +FROM sys.columns c JOIN sys.types ty ON c.user_type_id=ty.user_type_id +LEFT JOIN sys.identity_columns ic ON ic.object_id=c.object_id AND ic.column_id=c.column_id +LEFT JOIN sys.computed_columns cc ON cc.object_id=c.object_id AND cc.column_id=c.column_id +LEFT JOIN sys.default_constraints dc ON dc.object_id=c.default_object_id +LEFT JOIN sys.extended_properties ep ON ep.class=1 AND ep.major_id=c.object_id AND ep.minor_id=c.column_id AND ep.name=N'MS_Description' +WHERE c.object_id=" + id + " ORDER BY c.column_id"; + var columns = Query(connection, transaction, columnSql, cancellation).Select(r => new MssqlColumnDefinition(r)).ToList(); + var indexes = new List(); + foreach (var r in Query(connection, transaction, @"SELECT i.index_id,i.name,i.type,i.is_unique,i.is_primary_key,i.is_unique_constraint,i.is_disabled,i.is_hypothetical, +i.fill_factor,i.is_padded,i.ignore_dup_key,i.allow_row_locks,i.allow_page_locks,i.has_filter,i.filter_definition, +ds.name data_space_name,ds.type data_space_type,s.no_recompute," + Optional(connection, transaction, "sys.indexes", "i", cancellation, "optimize_for_sequential_key", "suppress_dup_key_messages") + @", +(SELECT MAX(p.data_compression) FROM sys.partitions p WHERE p.object_id=i.object_id AND p.index_id=i.index_id) data_compression +FROM sys.indexes i LEFT JOIN sys.data_spaces ds ON ds.data_space_id=i.data_space_id +LEFT JOIN sys.stats s ON s.object_id=i.object_id AND s.stats_id=i.index_id +WHERE i.object_id=" + id + " ORDER BY i.index_id", cancellation)) + { + var f = new MssqlNativeFacts(r); + var cols = Query(connection, transaction, "SELECT ic.index_column_id,ic.column_id,c.name,ic.key_ordinal,ic.is_descending_key,ic.is_included_column,ic.partition_ordinal FROM sys.index_columns ic JOIN sys.columns c ON c.object_id=ic.object_id AND c.column_id=ic.column_id WHERE ic.object_id=" + id + " AND ic.index_id=" + f.Number("index_id") + " ORDER BY ic.index_column_id", cancellation).Select(x => new MssqlNativeFacts(x)); + indexes.Add(new MssqlIndexDefinition(r, cols)); + } + var constraints = new List(); + foreach (var r in Query(connection, transaction, "SELECT object_id,name,'CHECK' kind,definition,is_disabled,is_not_trusted,is_not_for_replication,parent_column_id,uses_database_collation FROM sys.check_constraints WHERE parent_object_id=" + id + " ORDER BY object_id", cancellation)) constraints.Add(new MssqlConstraintDefinition(r, null)); + foreach (var r in Query(connection, transaction, @"SELECT fk.object_id,fk.name,'FK' kind,fk.referenced_object_id,SCHEMA_NAME(o.schema_id) referenced_schema,o.name referenced_table, +fk.delete_referential_action_desc,fk.update_referential_action_desc,fk.is_disabled,fk.is_not_trusted,fk.is_not_for_replication +FROM sys.foreign_keys fk JOIN sys.objects o ON o.object_id=fk.referenced_object_id WHERE fk.parent_object_id=" + id + " ORDER BY fk.object_id", cancellation)) + { + var f = new MssqlNativeFacts(r); + var cols = Query(connection, transaction, @"SELECT f.constraint_column_id,pc.name parent_column,rc.name referenced_column FROM sys.foreign_key_columns f +JOIN sys.columns pc ON pc.object_id=f.parent_object_id AND pc.column_id=f.parent_column_id +JOIN sys.columns rc ON rc.object_id=f.referenced_object_id AND rc.column_id=f.referenced_column_id WHERE f.constraint_object_id=" + f.Number("object_id") + " ORDER BY f.constraint_column_id", cancellation).Select(x => new MssqlNativeFacts(x)); + constraints.Add(new MssqlConstraintDefinition(r, cols)); + } + selected.Add(new MssqlTableDefinition(key, table.Facts.ToDictionary(x => x.Key, x => x.Value), columns, indexes, constraints)); + } + var views = new List(); + if (request.IncludeViews) + foreach (var r in Query(connection, transaction, @"SELECT v.object_id,SCHEMA_NAME(v.schema_id) schema_name,v.name,m.definition,m.uses_ansi_nulls,m.uses_quoted_identifier,m.is_schema_bound,v.with_check_option, +(SELECT COUNT(*) FROM sys.indexes i WHERE i.object_id=v.object_id AND i.index_id>0) index_count, +(SELECT COUNT(*) FROM sys.triggers t WHERE t.parent_id=v.object_id) trigger_count, +(SELECT COUNT(*) FROM sys.extended_properties e WHERE e.class=1 AND e.major_id=v.object_id) unsupported_properties +FROM sys.views v LEFT JOIN sys.sql_modules m ON m.object_id=v.object_id WHERE v.is_ms_shipped=0 ORDER BY v.object_id", cancellation)) + { var f = new MssqlNativeFacts(r); views.Add(new MssqlViewDefinition(new MssqlObjectKey(f.Number("object_id"), f.Text("schema_name"), f.Text("name")), r)); } + var objectIds = new HashSet(selected.Select(x => x.Key.ObjectId).Concat(views.Select(x => x.Key.ObjectId))); + var dependencies = new List(); + foreach (var r in Query(connection, transaction, @"SELECT CASE WHEN o.parent_object_id<>0 THEN o.parent_object_id ELSE d.referencing_id END object_id, +d.referenced_id,d.referenced_server_name,d.referenced_database_name,d.referenced_schema_name,d.referenced_entity_name +FROM sys.sql_expression_dependencies d JOIN sys.objects o ON o.object_id=d.referencing_id WHERE d.referenced_class=1", cancellation)) + { var f = new MssqlNativeFacts(r); if (objectIds.Contains(f.Number("object_id"))) dependencies.Add(new MssqlDependency(f.Number("object_id"), f.Text("referenced_id") == null ? (int?)null : f.Number("referenced_id"), f.Text("referenced_server_name"), f.Text("referenced_database_name"), f.Text("referenced_schema_name"), f.Text("referenced_entity_name"))); } + return new MssqlExportSchema(selected, views, dependencies, Snapshot(connection, transaction, cancellation), issues); + } + internal IReadOnlyList Validate(MssqlExportSchema schema, MssqlExportRequest request) + { + var issues = new List(schema.Issues); + if (request.Mode != MssqlExportMode.SchemaOnly && schema.SnapshotState != 1) issues.Add(new MssqlExportIssue("SnapshotUnavailable", null, null, "ALLOW_SNAPSHOT_ISOLATION must be ON (state 1); OFF and transitions cannot export data.")); + var ids = new HashSet(schema.Tables.Select(x => x.Key.ObjectId).Concat(schema.Views.Select(x => x.Key.ObjectId))); + foreach (var t in schema.Tables) + { + Action add = (code, col, reason) => issues.Add(new MssqlExportIssue(code, t.Key, col, reason)); + if (t.Text("can_select") != null && !t.Flag("can_select")) add("UnreadableTable", null, "SELECT permission is required."); + foreach (var flag in new[] { "temporal_type", "ledger_type", "is_memory_optimized", "is_filetable", "is_node", "is_edge", "is_external", "is_remote_data_archive_enabled", "trigger_count", "fulltext_count", "security_policy_count", "unsupported_properties" }) + if (t.Number(flag) != 0) add("UnsupportedTableFeature", null, "Unsupported table feature: " + flag); + if (request.Mode != MssqlExportMode.DataOnly && t.Text("uses_ansi_nulls") != null && !t.Flag("uses_ansi_nulls")) add("UnsupportedAnsiNulls", null, "Legacy ANSI_NULLS OFF cannot be recreated on modern SQL Server."); + foreach (var c in t.Columns) + { + if (request.Mode != MssqlExportMode.DataOnly && c.Text("is_ansi_padded") != null && !c.Flag("is_ansi_padded") && new[] { "char", "varchar", "binary", "varbinary" }.Contains(c.NativeType)) add("UnsupportedAnsiPadding", c.Name, "Legacy ANSI_PADDING OFF cannot be recreated on modern SQL Server."); + bool nativeGeometry = c.NativeType == "geometry" && c.TypeSchema == "sys" && !c.Flag("is_user_defined"); + if (c.NativeType == "geography" || (c.NativeType == "geometry" && !nativeGeometry)) add("UnsupportedSpatial", c.Name, "Only native sys.geometry has a supported spatial codec."); + if (!nativeGeometry && (c.Flag("is_user_defined") || c.Flag("is_assembly_type"))) add("UnsupportedUdt", c.Name, "Alias and CLR user types are not supported."); + if (c.Number("xml_collection_id") != 0) add("UnsupportedTypedXml", c.Name, "Typed XML schema collections are not exported."); + if (c.NativeType == "sql_variant" || !SupportedTypes.Contains(c.NativeType ?? "")) add("UnsupportedType", c.Name, "Unsupported native type: " + c.NativeType); + foreach (var flag in new[] { "is_filestream", "is_sparse", "is_column_set", "generated_always_type", "encryption_type", "is_hidden", "is_masked", "rule_object_id", "vector_dimensions" }) + if (c.Flag(flag) || c.Number(flag) != 0) add("UnsupportedColumnFeature", c.Name, "Unsupported column feature: " + flag); + if (c.Computed && c.ComputedDefinition == null) add("UnreadableDefinition", c.Name, "Computed definition is unavailable."); + if (c.Number("default_object_id") != 0 && c.DefaultDefinition == null) add("UnreadableDefinition", c.Name, "Default is hidden or is a legacy bound default."); + } + foreach (var i in t.Indexes) + { + if (i.Number("type") > 2 || i.Text("data_space_type") == "PS" || i.Number("data_compression") != 0 || i.Flag("is_hypothetical") || i.Flag("suppress_dup_key_messages")) add("UnsupportedIndex", null, "Unsupported index/partition/compression: " + i.Name); + if (i.Flag("has_filter") && i.Text("filter_definition") == null) add("UnreadableDefinition", null, "Index filter unavailable: " + i.Name); + } + foreach (var c in t.Constraints) + { + if (c.Kind == "CHECK" && c.Text("definition") == null) add("UnreadableDefinition", null, "CHECK definition unavailable: " + c.Name); + if (c.Kind == "FK" && !ids.Contains(c.Number("referenced_object_id")) && request.Mode != MssqlExportMode.DataOnly) add("MissingDependency", null, "Foreign key " + c.Name + " requires " + c.Text("referenced_schema") + "." + c.Text("referenced_table")); + } + } + foreach (var v in schema.Views) + { + if (v.Text("uses_ansi_nulls") != null && !v.AnsiNulls) issues.Add(new MssqlExportIssue("UnsupportedAnsiNulls", v.Key, null, "Legacy view ANSI_NULLS OFF cannot be recreated on modern SQL Server.")); + if (string.IsNullOrEmpty(v.Definition)) issues.Add(new MssqlExportIssue("UnreadableDefinition", v.Key, null, "View definition is encrypted or unreadable.")); + if (v.Number("index_count") != 0 || v.Number("trigger_count") != 0 || v.Number("unsupported_properties") != 0) issues.Add(new MssqlExportIssue("UnsupportedViewFeature", v.Key, null, "Indexed views, view triggers and view extended properties are unsupported.")); + } + if (request.Mode != MssqlExportMode.DataOnly) + foreach (var d in schema.Dependencies) + if (d.Server != null || d.Database != null || !d.ReferencedId.HasValue || !ids.Contains(d.ReferencedId.Value)) + issues.Add(new MssqlExportIssue("MissingDependency", schema.Tables.Select(x => x.Key).Concat(schema.Views.Select(x => x.Key)).FirstOrDefault(x => x.ObjectId == d.ObjectId), null, "Definition requires unexported object: " + string.Join(".", new[] { d.Server, d.Database, d.Schema, d.Name }.Where(x => x != null)))); + if (schema.Tables.GroupBy(x => x.Key.ObjectId).Any(g => g.Count() > 1)) + issues.Add(new MssqlExportIssue("DuplicateSelection", null, null, "Each table may be selected only once.")); + var pending = new HashSet(schema.Views.Select(x => x.Key.ObjectId)); + while (pending.Count != 0) + { + var ready = pending.Where(id => !schema.Dependencies.Any(d => d.ObjectId == id && d.ReferencedId.HasValue && pending.Contains(d.ReferencedId.Value))).ToList(); + if (ready.Count == 0) { issues.Add(new MssqlExportIssue("DependencyCycle", null, null, "View dependencies contain a cycle and cannot be scripted in dependency order.")); break; } + foreach (int id in ready) pending.Remove(id); + } + return issues.AsReadOnly(); + } + private static readonly HashSet SupportedTypes = new HashSet("bigint int smallint tinyint bit decimal numeric money smallmoney float real date time datetime datetime2 smalldatetime datetimeoffset char varchar text nchar nvarchar ntext binary varbinary image uniqueidentifier xml timestamp rowversion geometry".Split(' '), StringComparer.OrdinalIgnoreCase); + private static void EnsureVisibility(SqlConnection c, SqlTransaction t, CancellationToken token) + { + var row = Query(c, t, "SELECT HAS_PERMS_BY_NAME(DB_NAME(),'DATABASE','VIEW DEFINITION') allowed", token)[0]; + if (!new MssqlNativeFacts(row).Flag("allowed")) throw new InvalidOperationException("Database VIEW DEFINITION permission is required to detect hidden dependencies and definitions."); + } + private static int Snapshot(SqlConnection c, SqlTransaction t, CancellationToken token) { return new MssqlNativeFacts(Query(c, t, "SELECT snapshot_isolation_state FROM sys.databases WHERE database_id=DB_ID()", token).Single()).Number("snapshot_isolation_state"); } + private static bool HasCatalog(SqlConnection c, SqlTransaction t, string name, CancellationToken token) + { return Query(c, t, "SELECT object_id FROM sys.all_objects WHERE name='" + name + "' AND schema_id=SCHEMA_ID('sys')", token).Count != 0; } + private static List ReadTables(SqlConnection c, SqlTransaction t, CancellationToken token) + { + string optional = Optional(c, t, "sys.tables", "t", token, "temporal_type", "ledger_type", "is_memory_optimized", "is_filetable", "is_node", "is_edge", "is_external", "is_remote_data_archive_enabled"); + bool security = Query(c, t, "SELECT object_id FROM sys.all_objects WHERE name='security_predicates' AND schema_id=SCHEMA_ID('sys')", token).Count != 0; + string sql = @"SELECT t.object_id,SCHEMA_NAME(t.schema_id) schema_name,t.name,t.uses_ansi_nulls,t.lock_escalation_desc,t.text_in_row_limit,t.large_value_types_out_of_row,CONVERT(nvarchar(max),ep.value) comment, +HAS_PERMS_BY_NAME(QUOTENAME(SCHEMA_NAME(t.schema_id))+'.'+QUOTENAME(t.name),'OBJECT','SELECT') can_select, +(SELECT SUM(p.rows) FROM sys.partitions p WHERE p.object_id=t.object_id AND p.index_id IN(0,1)) estimated_rows, +(SELECT COUNT(*) FROM sys.triggers tr WHERE tr.parent_id=t.object_id) trigger_count, +(SELECT COUNT(*) FROM sys.fulltext_indexes f WHERE f.object_id=t.object_id) fulltext_count, +(SELECT COUNT(*) FROM sys.extended_properties e WHERE e.class=1 AND e.major_id=t.object_id AND e.name<>N'MS_Description') unsupported_properties," + +(security ? "(SELECT COUNT(*) FROM sys.security_predicates sp WHERE sp.target_object_id=t.object_id)" : "0") + " security_policy_count," + optional + @" +FROM sys.tables t LEFT JOIN sys.extended_properties ep ON ep.class=1 AND ep.major_id=t.object_id AND ep.minor_id=0 AND ep.name=N'MS_Description' +WHERE t.is_ms_shipped=0 ORDER BY t.object_id"; + var rows = Query(c, t, sql, token); + // Allocation metadata is a bounded catalog read. It is advisory and must not + // make catalog loading depend on a new permission or a table scan. + IReadOnlyList> storage = null; + try + { + storage = Query(c, t, @"SELECT p.object_id, +SUM(CONVERT(bigint, a.used_pages)) * CONVERT(bigint, 8192) estimated_bytes +FROM sys.partitions p JOIN sys.allocation_units a +ON (a.type IN (1,3) AND a.container_id=p.hobt_id) +OR (a.type=2 AND a.container_id=p.partition_id) +WHERE p.index_id IN (0,1) +GROUP BY p.object_id", token); + } + catch (SqlException) { /* Optional estimate unavailable; actual export can continue. */ } + return MapTables(rows, storage); + } + internal static List MapTables(IReadOnlyList> rows, IReadOnlyList> storage) + { + var sizes = (storage ?? new Dictionary[0]) + .Where(r => r.ContainsKey("object_id") && r.ContainsKey("estimated_bytes")) + .ToDictionary(r => Convert.ToInt32(r["object_id"], CultureInfo.InvariantCulture), r => r["estimated_bytes"]); + return rows.Select(row => + { + var values = new Dictionary(row, StringComparer.Ordinal); + int id = Convert.ToInt32(values["object_id"], CultureInfo.InvariantCulture); + object size; + values["estimated_bytes"] = sizes.TryGetValue(id, out size) ? size : null; + var facts = new MssqlNativeFacts(values); + return new MssqlTableDefinition(new MssqlObjectKey(id, facts.Text("schema_name"), facts.Text("name")), values, null, null, null); + }).ToList(); + } + // Probe catalog columns first: SQL Server still compiles a missing column inside CASE. + private static string Optional(SqlConnection c, SqlTransaction t, string catalog, string alias, CancellationToken token, params string[] names) + { + var available = new HashSet(Query(c, t, "SELECT name FROM sys.all_columns WHERE object_id=OBJECT_ID(N'" + catalog + "')", token).Select(r => Convert.ToString(r["name"], CultureInfo.InvariantCulture)), StringComparer.OrdinalIgnoreCase); + return string.Join(",", names.Select(n => OptionalProjection(alias, n, available))); + } + internal static string OptionalProjection(string alias, string name, IEnumerable available) + { + return available.Contains(name, StringComparer.OrdinalIgnoreCase) + ? alias + "." + Quote(name) + : "CAST(" + (name == "optimize_for_sequential_key" || name == "suppress_dup_key_messages" ? "NULL" : "0") + " AS int) " + Quote(name); + } + internal static List> Query(SqlConnection connection, SqlTransaction transaction, string sql, CancellationToken token) + { + token.ThrowIfCancellationRequested(); + var result = new List>(); + using (var owner = new MssqlCommandOwner(new SqlCommand(sql, connection, transaction) { CommandTimeout = 0 }, token)) + using (var reader = owner.ExecuteReader(CommandBehavior.Default)) + { + while (reader.Read()) + { token.ThrowIfCancellationRequested(); var row = new Dictionary(StringComparer.Ordinal); for (int i = 0; i < reader.FieldCount; i++) row.Add(reader.GetName(i), reader.IsDBNull(i) ? null : reader.GetValue(i)); result.Add(row); } + } + token.ThrowIfCancellationRequested(); return result; + } + } +} diff --git a/mySQLPunk/lib/MssqlExport/MssqlExportMirror.cs b/mySQLPunk/lib/MssqlExport/MssqlExportMirror.cs new file mode 100644 index 0000000..88e4495 --- /dev/null +++ b/mySQLPunk/lib/MssqlExport/MssqlExportMirror.cs @@ -0,0 +1,110 @@ +using System; +using System.Collections.Generic; +using System.IO; +using System.Threading; + +namespace mySQLPunk.lib.MssqlExport +{ + internal sealed class MssqlExportMirror + { + private readonly MssqlArtifactFiles files; + private readonly List residuals = new List(); + private readonly List warnings = new List(); + internal string MirrorPublishedPath { get; private set; } + internal IReadOnlyList ResidualPaths => new List(residuals).AsReadOnly(); + internal IReadOnlyList Warnings => new List(warnings).AsReadOnly(); + internal MssqlExportMirror() : this(new MssqlArtifactFiles()) { } + internal MssqlExportMirror(MssqlArtifactFiles files) { this.files = files; } + internal string CopyVerified(string publishedPath, MssqlArtifactReceipt receipt, MssqlMirrorOptions options, Action progress, CancellationToken cancellation) + { + if (MirrorPublishedPath != null) throw new InvalidOperationException("Mirror already published"); + if (receipt == null || options == null) throw new ArgumentNullException(); + string source = Path.GetFullPath(publishedPath); + string directory = options.Directory; + if (directory.Length > Path.GetPathRoot(directory).Length) directory = directory.TrimEnd(Path.DirectorySeparatorChar, Path.AltDirectorySeparatorChar); + cancellation.ThrowIfCancellationRequested(); + if (string.Equals(Path.GetDirectoryName(source), directory, StringComparison.OrdinalIgnoreCase)) return string.Empty; + Directory.CreateDirectory(directory); + string temporary = Path.Combine(directory, ".mysqlpunk-export-" + receipt.JobId.ToString("N") + "-" + Guid.NewGuid().ToString("N") + ".partial"); + bool owned = false; + try + { + using (var input = new FileStream(source, FileMode.Open, FileAccess.Read, FileShare.Read, MssqlExportArtifact.BufferSize)) + using (var output = files.Create(temporary)) + { + owned = true; + byte[] buffer = new byte[MssqlExportArtifact.BufferSize]; int read; long copied = 0; + while (true) + { + cancellation.ThrowIfCancellationRequested(); read = input.Read(buffer, 0, buffer.Length); if (read == 0) break; + output.Write(buffer, 0, read); copied = checked(copied + read); progress?.Invoke(copied, receipt.Bytes); + } + output.Flush(); + } + MssqlExportArtifact.VerifyFile(temporary, receipt, progress, cancellation); + var gate = MssqlArtifactGates.Get(directory); gate.Wait(cancellation); + try + { + string target = Path.Combine(directory, Path.GetFileNameWithoutExtension(source) + "_" + receipt.JobId.ToString("N") + "_" + Guid.NewGuid().ToString("N") + Path.GetExtension(source)); + File.SetLastWriteTimeUtc(temporary, DateTime.UtcNow); + using (var commit = new MssqlPublicationCommit(cancellation)) + { + commit.Enter(); files.Move(temporary, target); MirrorPublishedPath = target; owned = false; + } + if (cancellation.IsCancellationRequested) warnings.Add("Mirror published; retention skipped after cancellation"); + else + { + try + { + // Future-dated backups must not make the just-published file the prune victim. + if (WouldPrunePublished(target, options.RetainCount)) warnings.Add("Mirror published; retention skipped because newer timestamps would remove it"); + else + { + files.Prune(directory, options.RetainCount); + if (RetentionExceeded(directory, options.RetainCount)) warnings.Add("Mirror published; some managed backups exceed the retention limit"); + } + } + catch (Exception ex) { warnings.Add("Mirror published; retention incomplete: " + ex.Message); } + } + return MirrorPublishedPath; + } + finally { gate.Release(); } + } + finally + { + if (owned) + { + try { files.Delete(temporary); } + catch (Exception ex) { residuals.Add(temporary); warnings.Add("Mirror temporary cleanup failed: " + ex.Message); } + } + } + } + // Matches BackupRemoteMirrorService's existing classification and precedence. + private static string Kind(string name) + { + string lower = name.ToLowerInvariant(); + if (lower.Contains("_before_delete_")) return "before_delete"; + if (lower.Contains("_before_restore_")) return "before_restore"; + if (lower.Contains("_backup_")) return "backup"; + return string.Empty; + } + private static bool WouldPrunePublished(string path, int retain) + { + string kind = Kind(Path.GetFileName(path)); if (kind.Length == 0) return false; + DateTime stamp = File.GetLastWriteTimeUtc(path); int newerOrEqual = 0; + foreach (var file in new DirectoryInfo(Path.GetDirectoryName(path)).GetFiles()) + if (!string.Equals(file.FullName, path, StringComparison.OrdinalIgnoreCase) && Kind(file.Name) == kind && file.LastWriteTimeUtc >= stamp) newerOrEqual++; + return newerOrEqual >= retain; + } + private static bool RetentionExceeded(string directory, int retain) + { + var counts = new Dictionary(); + foreach (var file in new DirectoryInfo(directory).GetFiles()) + { + string kind = Kind(file.Name); if (kind.Length == 0) continue; + int count; counts.TryGetValue(kind, out count); counts[kind] = ++count; if (count > retain) return true; + } + return false; + } + } +} diff --git a/mySQLPunk/lib/MssqlExport/MssqlExportModels.cs b/mySQLPunk/lib/MssqlExport/MssqlExportModels.cs new file mode 100644 index 0000000..5847d7d --- /dev/null +++ b/mySQLPunk/lib/MssqlExport/MssqlExportModels.cs @@ -0,0 +1,336 @@ +using System; +using System.Collections.Generic; +using System.IO; +using System.Linq; +using System.Globalization; +using System.Collections.ObjectModel; +using System.Security.Cryptography; +using System.Text; + +namespace mySQLPunk.lib.MssqlExport +{ + internal enum MssqlExportMode { SchemaAndData, SchemaOnly, DataOnly } + + internal sealed class MssqlObjectKey + { + public int ObjectId { get; } + public string Schema { get; } + public string Name { get; } + public MssqlObjectKey(int objectId, string schema, string name) + { + if (objectId <= 0) throw new ArgumentOutOfRangeException(nameof(objectId)); + if (string.IsNullOrWhiteSpace(schema)) throw new ArgumentException("Schema is required", nameof(schema)); + if (string.IsNullOrWhiteSpace(name)) throw new ArgumentException("Name is required", nameof(name)); + ObjectId = objectId; Schema = schema; Name = name; + } + } + + internal sealed class MssqlMirrorOptions + { + public string Directory { get; } + public int RetainCount { get; } + public MssqlMirrorOptions(string directory, int retainCount) + { + if (string.IsNullOrWhiteSpace(directory)) throw new ArgumentException("Mirror directory is required", nameof(directory)); + if (retainCount < 1) throw new ArgumentOutOfRangeException(nameof(retainCount)); + Directory = Path.GetFullPath(directory); RetainCount = retainCount; + } + } + + // Captured destination facts; capture/revalidation belongs to the publication task. + internal sealed class ExportDestinationStamp + { + public bool Exists { get; } + public string FullPath { get; } + public string FileIdentity { get; } + public long Length { get; } + public DateTime LastWriteTimeUtc { get; } + public string Sha256 { get; } + public ExportDestinationStamp(bool exists, string fullPath, string fileIdentity, long length, DateTime lastWriteTimeUtc, string sha256) + { + if (string.IsNullOrWhiteSpace(fullPath)) throw new ArgumentException("Destination path is required", nameof(fullPath)); + if (length < 0) throw new ArgumentOutOfRangeException(nameof(length)); + Exists = exists; FullPath = Path.GetFullPath(fullPath); FileIdentity = fileIdentity; + Length = length; LastWriteTimeUtc = lastWriteTimeUtc; Sha256 = sha256; + } + } + + // Null selection and null seed mean database scope. An explicit empty or unresolved + // selection never broadens to all tables. Raw seeds use the existing provider tree + // representation (dbo => name, other schemas => schema.name), not SQL parsing. + internal sealed class MssqlExportWizardOptions + { + public MssqlExportMode Mode { get; } + public IReadOnlyList InitialTables { get; } + public string InitialObjectName { get; } + public bool IncludeViews { get; } + public string DefaultFileName { get; } + public MssqlMirrorOptions Mirror { get; } + internal MssqlExportWizardOptions(MssqlExportMode mode, IReadOnlyList initialTables = null, bool includeViews = true, string defaultFileName = null, MssqlMirrorOptions mirror = null, string initialObjectName = null) + { + if (!Enum.IsDefined(typeof(MssqlExportMode), mode)) throw new ArgumentOutOfRangeException(nameof(mode)); + Mode = mode; InitialTables = initialTables == null ? null : new List(initialTables).AsReadOnly(); + if (InitialTables != null && InitialTables.Any(x => x == null)) throw new ArgumentException("Null initial table", nameof(initialTables)); + InitialObjectName = initialObjectName; IncludeViews = includeViews; DefaultFileName = defaultFileName; Mirror = mirror; + } + } + + internal sealed class MssqlExportRequest + { + public Guid JobId { get; } + public string DatabaseName { get; } + public IReadOnlyList Tables { get; } + public MssqlExportMode Mode { get; } + public bool IncludeViews { get; } + public string TargetPath { get; } + public ExportDestinationStamp Destination { get; } + public MssqlMirrorOptions Mirror { get; } + public MssqlExportRequest(Guid jobId, string databaseName, IReadOnlyList tables, MssqlExportMode mode, bool includeViews, string targetPath, ExportDestinationStamp destination, MssqlMirrorOptions mirror) + { + if (jobId == Guid.Empty) throw new ArgumentException("Job identity is required", nameof(jobId)); + if (string.IsNullOrWhiteSpace(databaseName)) throw new ArgumentException("Database is required", nameof(databaseName)); + if (tables == null || tables.Count == 0) throw new ArgumentException("Select at least one table", nameof(tables)); + if (!Enum.IsDefined(typeof(MssqlExportMode), mode)) throw new ArgumentException("Unknown export mode", nameof(mode)); + if (mode == MssqlExportMode.DataOnly && includeViews) throw new ArgumentException("Data-only export cannot include views", nameof(includeViews)); + if (string.IsNullOrWhiteSpace(targetPath)) throw new ArgumentException("Output path is required", nameof(targetPath)); + var selection = new List(tables); + if (selection.Contains(null)) throw new ArgumentException("Table selection contains null", nameof(tables)); + JobId = jobId; DatabaseName = databaseName; Tables = selection.AsReadOnly(); + Mode = mode; IncludeViews = includeViews; TargetPath = Path.GetFullPath(targetPath); + Destination = destination; Mirror = mirror; + } + } + + internal static class MssqlExportLimits + { + public const int ProgressLogLimit = 200; + public const int SmallRowBytes = 262144; + public const int StatementBytes = 262144; + public const int BatchBytes = 4194304; + public const int BinaryChunkBytes = 65536; + public const int TextChunkCodeUnits = 16384; + } + // Native catalog facts are copied once. Extra SQL Server properties stay accessible + // without lossy cross-provider type mapping; writers use the typed accessors below. + internal class MssqlNativeFacts + { + public IReadOnlyDictionary Facts { get; } + internal MssqlNativeFacts(IDictionary facts) + { + Facts = new ReadOnlyDictionary(new Dictionary(facts ?? new Dictionary(), StringComparer.Ordinal)); + } + public string Text(string name) { object value; return Facts.TryGetValue(name, out value) && value != null && value != DBNull.Value ? Convert.ToString(value, CultureInfo.InvariantCulture) : null; } + public int Number(string name) { var value = Text(name); return value == null || value == "False" ? 0 : value == "True" ? 1 : int.Parse(value, CultureInfo.InvariantCulture); } + public bool Flag(string name) { var value = Text(name); return value == "1" || string.Equals(value, "True", StringComparison.OrdinalIgnoreCase); } + public long? Estimate(string name) { var value = Text(name); return value == null ? (long?)null : long.Parse(value, CultureInfo.InvariantCulture); } + internal void Canonical(StringBuilder output) + { + foreach (var fact in Facts.Where(x => x.Key != "estimated_rows" && x.Key != "estimated_bytes").OrderBy(x => x.Key, StringComparer.Ordinal)) + { Part(output, fact.Key); Part(output, Text(fact.Key)); } + } + internal static void Part(StringBuilder output, string text) { output.Append(text == null ? -1 : text.Length).Append(':').Append(text); } + internal static IReadOnlyList Copy(IEnumerable values) { return new List(values ?? Enumerable.Empty()).AsReadOnly(); } + } + internal sealed class MssqlColumnDefinition : MssqlNativeFacts + { + internal MssqlColumnDefinition(IDictionary facts) : base(facts) { } + public int Ordinal => Number("column_id"); + public string Name => Text("name"); + public string NativeType => Text("type_name"); + public string TypeSchema => Text("type_schema"); + public int Length => Number("max_length"); + public int Precision => Number("precision"); + public int Scale => Number("scale"); + public string Collation => Text("collation_name"); + public bool Nullable => Flag("is_nullable"); + public bool Identity => Flag("is_identity"); + public string IdentitySeed => Text("seed_value"); + public string IdentityIncrement => Text("increment_value"); + public bool Computed => Flag("is_computed"); + public string ComputedDefinition => Text("computed_definition"); + public bool Persisted => Flag("is_persisted"); + public bool RowVersion => NativeType == "timestamp" || NativeType == "rowversion"; + public string DefaultName => Text("default_name"); + public string DefaultDefinition => Text("default_definition"); + public string Comment => Text("comment"); + } + internal sealed class MssqlIndexDefinition : MssqlNativeFacts + { + public IReadOnlyList Columns { get; } + internal MssqlIndexDefinition(IDictionary facts, IEnumerable columns) : base(facts) { Columns = Copy(columns); } + public string Name => Text("name"); + public int IndexId => Number("index_id"); + } + internal sealed class MssqlConstraintDefinition : MssqlNativeFacts + { + public IReadOnlyList Columns { get; } + internal MssqlConstraintDefinition(IDictionary facts, IEnumerable columns) : base(facts) { Columns = Copy(columns); } + public string Name => Text("name"); + public string Kind => Text("kind"); + } + internal sealed class MssqlTableDefinition : MssqlNativeFacts + { + public MssqlObjectKey Key { get; } + public IReadOnlyList Columns { get; } + public IReadOnlyList Indexes { get; } + public IReadOnlyList Constraints { get; } + internal MssqlTableDefinition(MssqlObjectKey key, IDictionary facts, IEnumerable columns, IEnumerable indexes, IEnumerable constraints) : base(facts) + { Key = key; Columns = Copy(columns); Indexes = Copy(indexes); Constraints = Copy(constraints); } + public string Comment => Text("comment"); + public long? EstimatedRows => Estimate("estimated_rows"); + public long? EstimatedBytes => Estimate("estimated_bytes"); + } + internal sealed class MssqlViewDefinition : MssqlNativeFacts + { + public MssqlObjectKey Key { get; } + internal MssqlViewDefinition(MssqlObjectKey key, IDictionary facts) : base(facts) { Key = key; } + public string Definition => Text("definition"); + public bool AnsiNulls => Flag("uses_ansi_nulls"); + public bool QuotedIdentifier => Flag("uses_quoted_identifier"); + } + internal sealed class MssqlDependency + { + public int ObjectId { get; } + public int? ReferencedId { get; } + public string Server { get; } + public string Database { get; } + public string Schema { get; } + public string Name { get; } + internal MssqlDependency(int objectId, int? referencedId, string server, string database, string schema, string name) + { ObjectId = objectId; ReferencedId = referencedId; Server = server; Database = database; Schema = schema; Name = name; } + } + internal sealed class MssqlExportIssue + { + public string Code { get; } + public MssqlObjectKey ObjectKey { get; } + public string ColumnName { get; } + public string Message { get; } + internal MssqlExportIssue(string code, MssqlObjectKey key, string column, string message) { Code = code; ObjectKey = key; ColumnName = column; Message = message; } + } + internal sealed class MssqlExportCatalog + { + public IReadOnlyList Tables { get; } + public int SnapshotState { get; } + public IReadOnlyList Issues { get; } + internal MssqlExportCatalog(IEnumerable tables, int snapshotState, IEnumerable issues) + { Tables = MssqlNativeFacts.Copy(tables); SnapshotState = snapshotState; Issues = MssqlNativeFacts.Copy(issues); } + } + internal sealed class MssqlExportSchema + { + public IReadOnlyList Tables { get; } + public IReadOnlyList Views { get; } + public IReadOnlyList Dependencies { get; } + public IReadOnlyList Issues { get; } + public int SnapshotState { get; } + public string Fingerprint { get; } + internal MssqlExportSchema(IEnumerable tables, IEnumerable views, IEnumerable dependencies, int snapshotState, IEnumerable issues) + { + Tables = MssqlNativeFacts.Copy(tables); Views = MssqlNativeFacts.Copy(views); Dependencies = MssqlNativeFacts.Copy(dependencies); Issues = MssqlNativeFacts.Copy(issues); SnapshotState = snapshotState; + var text = new StringBuilder(); + foreach (var t in Tables.OrderBy(x => x.Key.ObjectId)) + { + Key(text, t.Key); t.Canonical(text); + foreach (var c in t.Columns.OrderBy(x => x.Ordinal)) c.Canonical(text); + foreach (var i in t.Indexes.OrderBy(x => x.IndexId)) { i.Canonical(text); foreach (var c in i.Columns) c.Canonical(text); } + foreach (var c in t.Constraints.OrderBy(x => x.Name, StringComparer.Ordinal)) { c.Canonical(text); foreach (var column in c.Columns) column.Canonical(text); } + } + foreach (var v in Views.OrderBy(x => x.Key.ObjectId)) { Key(text, v.Key); v.Canonical(text); } + foreach (var d in Dependencies.OrderBy(x => x.ObjectId).ThenBy(x => x.ReferencedId).ThenBy(x => x.Name, StringComparer.Ordinal)) + { MssqlNativeFacts.Part(text, d.ObjectId.ToString(CultureInfo.InvariantCulture)); MssqlNativeFacts.Part(text, d.ReferencedId?.ToString(CultureInfo.InvariantCulture)); MssqlNativeFacts.Part(text, d.Server); MssqlNativeFacts.Part(text, d.Database); MssqlNativeFacts.Part(text, d.Schema); MssqlNativeFacts.Part(text, d.Name); } + using (var sha = SHA256.Create()) Fingerprint = BitConverter.ToString(sha.ComputeHash(Encoding.UTF8.GetBytes(text.ToString()))).Replace("-", ""); + } + private static void Key(StringBuilder text, MssqlObjectKey key) { MssqlNativeFacts.Part(text, key.ObjectId.ToString(CultureInfo.InvariantCulture)); MssqlNativeFacts.Part(text, key.Schema); MssqlNativeFacts.Part(text, key.Name); } + } + internal sealed class MssqlExportPreflight + { + public MssqlExportSchema Schema { get; } + public int SnapshotState => Schema.SnapshotState; + public IReadOnlyList Issues { get; } + public bool CanStart => Issues.Count == 0; + internal MssqlExportPreflight(MssqlExportSchema schema, IEnumerable issues) { Schema = schema; Issues = MssqlNativeFacts.Copy(issues); } + } + internal sealed class MssqlTableRowCount + { + public MssqlObjectKey Key { get; } + public long Rows { get; } + public MssqlTableRowCount(MssqlObjectKey key, long rows) + { + if (key == null) throw new ArgumentNullException(nameof(key)); + if (rows < 0) throw new ArgumentOutOfRangeException(nameof(rows)); + Key = key; Rows = rows; + } + } + internal sealed class MssqlArtifactReceipt + { + public Guid JobId { get; } + public string TemporaryPath { get; } + public string IntendedTargetPath { get; } + public long Bytes { get; } + public string Sha256 { get; } + public IReadOnlyList Tables { get; } + public long TotalRows { get; } + public string CompletionMarker { get; } + internal MssqlArtifactReceipt(Guid jobId, string temporaryPath, string publishedPath, long bytes, string sha256, IReadOnlyList tables, string marker) + { + JobId = jobId; TemporaryPath = temporaryPath; IntendedTargetPath = publishedPath; Bytes = bytes; Sha256 = sha256; + Tables = new List(tables).AsReadOnly(); CompletionMarker = marker; + long total = 0; foreach (var table in Tables) total = checked(total + table.Rows); TotalRows = total; + } + } + + internal enum MssqlExportStage { Preflight, Schema, Data, Verifying, Publishing, Mirroring, CancelRequested, Completed, Canceled, Failed, LocalCompletedMirrorIncomplete } + internal enum MssqlExportEstimationState { Estimating, Measuring, Waiting, Unreliable, Complete } + internal sealed class MssqlExportProgress + { + public Guid JobId { get; } + public MssqlExportStage Stage { get; } + public MssqlObjectKey CurrentTable { get; } + public long CurrentTableRows { get; } + public long CurrentTableSourceBytes { get; } + public long Rows { get; } + public long SourceBytes { get; } + public long WrittenBytes { get; } + public long? EstimatedRows { get; } + public long? EstimatedBytes { get; } + public int CompletedTables { get; } + public int TotalTables { get; } + public TimeSpan Elapsed { get; } + public TimeSpan LastProgressElapsed { get; } + // Source bytes/s during Data; processed local phase bytes/s during verification/mirroring. + public double? BytesPerSecond { get; } + public TimeSpan? Eta { get; } + public double? Percent { get; } + public MssqlExportEstimationState EstimationState { get; } + public long PhaseProcessedBytes { get; } + public long? PhaseTotalBytes { get; } + public string Message { get; } + internal MssqlExportProgress(Guid jobId, MssqlExportStage stage, MssqlObjectKey table, long tableRows, long tableBytes, + long rows, long sourceBytes, long writtenBytes, long? estimatedRows, long? estimatedBytes, int completed, int total, + TimeSpan elapsed, TimeSpan lastProgressElapsed, double? bytesPerSecond, TimeSpan? eta, double? percent, MssqlExportEstimationState estimation, long phaseBytes, long? phaseTotal, string message) + { + JobId = jobId; Stage = stage; CurrentTable = table; CurrentTableRows = tableRows; CurrentTableSourceBytes = tableBytes; + Rows = rows; SourceBytes = sourceBytes; WrittenBytes = writtenBytes; EstimatedRows = estimatedRows; EstimatedBytes = estimatedBytes; + CompletedTables = completed; TotalTables = total; Elapsed = elapsed; LastProgressElapsed = lastProgressElapsed; BytesPerSecond = bytesPerSecond; Eta = eta; Percent = percent; EstimationState = estimation; + PhaseProcessedBytes = phaseBytes; PhaseTotalBytes = phaseTotal; Message = (message ?? string.Empty).Substring(0, Math.Min(512, (message ?? string.Empty).Length)); + } + } + internal sealed class MssqlExportResult + { + public Guid JobId { get; } + public MssqlExportStage Stage { get; } + public string PublishedPath { get; } + public string MirrorPath { get; } + public IReadOnlyList RowCounts { get; } + public TimeSpan Elapsed { get; } + public IReadOnlyList Warnings { get; } + public IReadOnlyList ResidualPaths { get; } + public string ErrorMessage { get; } + internal MssqlExportResult(Guid jobId, MssqlExportStage stage, string publishedPath, string mirrorPath, + IEnumerable rows, TimeSpan elapsed, IEnumerable warnings, IEnumerable residuals, string errorMessage) + { + JobId = jobId; Stage = stage; PublishedPath = publishedPath; MirrorPath = mirrorPath; RowCounts = MssqlNativeFacts.Copy(rows); + Elapsed = elapsed; Warnings = MssqlNativeFacts.Copy(warnings); ResidualPaths = MssqlNativeFacts.Copy(residuals); ErrorMessage = errorMessage; + } + } + +} diff --git a/mySQLPunk/lib/MssqlExport/MssqlExportProgressEstimator.cs b/mySQLPunk/lib/MssqlExport/MssqlExportProgressEstimator.cs new file mode 100644 index 0000000..1ad0e01 --- /dev/null +++ b/mySQLPunk/lib/MssqlExport/MssqlExportProgressEstimator.cs @@ -0,0 +1,115 @@ +using System; +using System.Collections.Generic; +using System.Linq; + +namespace mySQLPunk.lib.MssqlExport +{ + // All time and work are supplied by the caller. This class neither polls nor queries SQL. + internal sealed class MssqlExportProgressEstimator + { + private readonly Guid jobId; + private readonly MssqlExportSchema schema; + private readonly bool mirror; + private readonly HashSet completed = new HashSet(); + private readonly Queue> samples = new Queue>(); + private MssqlObjectKey current; + private long tableRows, tableBytes, completedRows, completedBytes, lastRows; + private MssqlExportStage lastStage; + private bool emitted; + private TimeSpan lastReport, lastAdvance; + private double lastWork; + private double? lastPercent; + internal MssqlExportProgressEstimator(Guid jobId, MssqlExportSchema schema, bool includeMirror) + { this.jobId = jobId; this.schema = schema ?? throw new ArgumentNullException(nameof(schema)); mirror = includeMirror; } + internal MssqlExportProgress Sample(MssqlExportStage stage, MssqlObjectKey currentTable, long currentTableRows, + long currentTableSourceBytes, long writtenBytes, TimeSpan elapsed, bool force) + { return Build(stage, currentTable, currentTableRows, currentTableSourceBytes, writtenBytes, 0, null, elapsed, force); } + internal MssqlExportProgress SamplePhase(MssqlExportStage stage, long processedBytes, long totalBytes, long writtenBytes, TimeSpan elapsed, bool force) + { return Build(stage, null, 0, 0, writtenBytes, processedBytes, totalBytes, elapsed, force || processedBytes == totalBytes); } + private MssqlExportProgress Build(MssqlExportStage stage, MssqlObjectKey table, long rows, long bytes, long written, + long phaseBytes, long? phaseTotal, TimeSpan elapsed, bool force) + { + bool changedTable = !Same(current, table), changedStage = !emitted || lastStage != stage; + bool terminal = stage == MssqlExportStage.Completed || stage == MssqlExportStage.Canceled || stage == MssqlExportStage.Failed || stage == MssqlExportStage.LocalCompletedMirrorIncomplete; + if (changedTable) + { + if (current != null) + { + // A terminal failure does not claim that its interrupted table completed. + completedRows += tableRows; completedBytes += tableBytes; + if (!terminal || stage == MssqlExportStage.Completed) completed.Add(current.ObjectId); + } + current = table; tableRows = 0; tableBytes = 0; + } + tableRows = Math.Max(tableRows, rows); tableBytes = Math.Max(tableBytes, bytes); + long actualRows = completedRows + tableRows, actualBytes = completedBytes + tableBytes; + double work = stage == MssqlExportStage.Data ? actualBytes : phaseBytes; + if (changedStage) { samples.Clear(); lastWork = work; lastAdvance = elapsed; } + else if (work > lastWork || stage == MssqlExportStage.Data && actualRows > lastRows) { lastAdvance = elapsed; lastWork = work; } + lastRows = actualRows; + // Actual progress time is updated even for suppressed UI samples. + if (!force && !changedStage && !changedTable && !terminal && elapsed - lastReport < TimeSpan.FromMilliseconds(200)) return null; + if (changedStage || samples.Count == 0 || elapsed > samples.Last().Item1) samples.Enqueue(Tuple.Create(elapsed, work)); + while (samples.Count > 1 && elapsed - samples.Peek().Item1 > TimeSpan.FromSeconds(10)) samples.Dequeue(); + lastStage = stage; lastReport = elapsed; emitted = true; + long? estimatedRows = schema.Tables.All(t => t.EstimatedRows.HasValue) ? (long?)schema.Tables.Sum(t => t.EstimatedRows.Value) : null; + long? estimatedBytes = schema.Tables.All(t => t.EstimatedBytes.HasValue) ? (long?)schema.Tables.Sum(t => t.EstimatedBytes.Value) : null; + double? remaining = null, candidate = null; + bool unreliable = false; + double dataCeiling = mirror ? 75 : 85; + if (stage == MssqlExportStage.Data) + { + double rest = 0; bool known = true; + foreach (var t in schema.Tables) + { + if (completed.Contains(t.Key.ObjectId)) continue; + if (!t.EstimatedRows.HasValue || !t.EstimatedBytes.HasValue) { known = false; continue; } + if (Same(t.Key, current)) + { + if (tableRows > t.EstimatedRows || tableBytes > t.EstimatedBytes * 4.0) unreliable = true; + double width = tableRows > 0 ? tableBytes / (double)tableRows : t.EstimatedBytes.Value / (double)Math.Max(1, t.EstimatedRows.Value); + rest += Math.Max(0, Math.Max(t.EstimatedBytes.Value - tableBytes, (t.EstimatedRows.Value - tableRows) * width)); + } + else rest += t.EstimatedBytes.Value; + } + if (known && !unreliable) { remaining = rest; candidate = dataCeiling * actualBytes / Math.Max(1, actualBytes + rest); } + } + else if (stage == MssqlExportStage.Verifying || stage == MssqlExportStage.Mirroring) + { + double start = stage == MssqlExportStage.Verifying ? dataCeiling : 90; + double span = stage == MssqlExportStage.Verifying ? 10 : 9; + if (phaseTotal.HasValue) { remaining = Math.Max(0, phaseTotal.Value - phaseBytes); candidate = start + span * Math.Min(1, phaseBytes / (double)Math.Max(1, phaseTotal.Value)); } + else candidate = start; + } + else if (stage == MssqlExportStage.Publishing) candidate = mirror ? 85 : 95; + if (candidate.HasValue) lastPercent = Math.Max(lastPercent ?? 0, Math.Min(99, candidate.Value)); + TimeSpan? eta = null; double? bytesPerSecond = null; var state = unreliable ? MssqlExportEstimationState.Unreliable : MssqlExportEstimationState.Estimating; + if (elapsed - lastAdvance >= TimeSpan.FromSeconds(5)) state = MssqlExportEstimationState.Waiting; + else if (!unreliable && samples.Count >= 3 && elapsed - samples.Peek().Item1 >= TimeSpan.FromSeconds(2)) + { + var points = samples.ToArray(); double low = double.MaxValue, high = 0; + for (int i = 1; i < points.Length; i++) + { + double dt = (points[i].Item1 - points[i - 1].Item1).TotalSeconds; + double rate = dt > 0 ? (points[i].Item2 - points[i - 1].Item2) / dt : 0; + if (rate > 0) { low = Math.Min(low, rate); high = Math.Max(high, rate); } + } + double speed = (work - points[0].Item2) / (elapsed - points[0].Item1).TotalSeconds; + if (speed > 0 && high <= low * 4) + { + bytesPerSecond = speed; + double? seconds = remaining / speed; + if (seconds > 0 && seconds < TimeSpan.MaxValue.TotalSeconds) { eta = TimeSpan.FromSeconds(seconds.Value); state = MssqlExportEstimationState.Measuring; } + } + else state = MssqlExportEstimationState.Unreliable; + } + if (terminal || stage == MssqlExportStage.Publishing) { eta = null; bytesPerSecond = null; } + if (stage == MssqlExportStage.Completed) { lastPercent = 100; state = MssqlExportEstimationState.Complete; } + string message = state == MssqlExportEstimationState.Waiting ? "Waiting for progress" : stage.ToString(); + return new MssqlExportProgress(jobId, stage, current, tableRows, tableBytes, actualRows, actualBytes, written, + estimatedRows, estimatedBytes, completed.Count, schema.Tables.Count, elapsed, lastAdvance, bytesPerSecond, eta, lastPercent, state, phaseBytes, phaseTotal, message); + } + private static bool Same(MssqlObjectKey a, MssqlObjectKey b) + { return ReferenceEquals(a, b) || a != null && b != null && a.ObjectId == b.ObjectId && a.Schema == b.Schema && a.Name == b.Name; } + } +} diff --git a/mySQLPunk/lib/MssqlExport/MssqlExportSource.cs b/mySQLPunk/lib/MssqlExport/MssqlExportSource.cs new file mode 100644 index 0000000..b4104a6 --- /dev/null +++ b/mySQLPunk/lib/MssqlExport/MssqlExportSource.cs @@ -0,0 +1,142 @@ +using System; +using System.Data; +using System.Data.Common; +using System.Data.SqlClient; +using System.Linq; +using System.Threading; + +namespace mySQLPunk.lib.MssqlExport +{ + internal interface IMssqlExportSource : IDisposable + { + MssqlExportCatalog ReadCatalog(string databaseName, CancellationToken cancellation); + MssqlExportPreflight Inspect(MssqlExportRequest request, CancellationToken cancellation); + MssqlExportSchema BeginRead(MssqlExportRequest request, CancellationToken cancellation); + MssqlRowReader OpenRows(MssqlTableDefinition table, CancellationToken cancellation); + void CompleteRead(MssqlExportSchema schema, CancellationToken cancellation); + } + internal sealed class MssqlRowReader : IDisposable + { + private IDisposable commandOwner; + public DbDataReader Reader { get; } + internal bool IsDisposed { get; private set; } + internal MssqlRowReader(DbDataReader reader, IDisposable commandOwner) + { Reader = reader ?? throw new ArgumentNullException(nameof(reader)); this.commandOwner = commandOwner ?? throw new ArgumentNullException(nameof(commandOwner)); } + public void Dispose() + { + if (IsDisposed) return; + IsDisposed = true; + try { Reader.Dispose(); } finally { var owner = commandOwner; commandOwner = null; owner.Dispose(); } + } + } + internal sealed class MssqlExportSource : IMssqlExportSource + { + private readonly MssqlOwnedSession session; + private readonly MssqlExportMetadataReader metadata = new MssqlExportMetadataReader(); + private SqlTransaction transaction; + private MssqlExportRequest request; + private MssqlExportSchema frozen; + private MssqlRowReader activeReader; + private bool disposed; + internal MssqlExportSource(MssqlOwnedSession session) { this.session = session ?? throw new ArgumentNullException(nameof(session)); } + public MssqlExportCatalog ReadCatalog(string databaseName, CancellationToken cancellation) + { CheckIdle(); try { return metadata.ReadCatalog(session.Connection, databaseName, cancellation); } catch { Dispose(); throw; } } + public MssqlExportPreflight Inspect(MssqlExportRequest value, CancellationToken cancellation) + { + CheckIdle(); + try { cancellation.ThrowIfCancellationRequested(); session.Connection.ChangeDatabase(value.DatabaseName); var schema = metadata.ReadSchema(session.Connection, null, value, cancellation); return new MssqlExportPreflight(schema, metadata.Validate(schema, value)); } + catch { Dispose(); throw; } + } + public MssqlExportSchema BeginRead(MssqlExportRequest value, CancellationToken cancellation) + { + CheckIdle(); + try + { + var preflight = Inspect(value, cancellation); + RequireValid(preflight); + request = value; + cancellation.ThrowIfCancellationRequested(); + if (value.Mode != MssqlExportMode.SchemaOnly) transaction = session.Connection.BeginTransaction(IsolationLevel.Snapshot); + frozen = metadata.ReadSchema(session.Connection, transaction, value, cancellation); + RequireValid(new MssqlExportPreflight(frozen, metadata.Validate(frozen, value))); + return frozen; + } + catch { Dispose(); throw; } + } + public MssqlRowReader OpenRows(MssqlTableDefinition table, CancellationToken cancellation) + { + CheckAlive(); + try + { + if (frozen == null || transaction == null) throw new InvalidOperationException("A data snapshot must be started first."); + if (activeReader != null && !activeReader.IsDisposed) throw new InvalidOperationException("Dispose the current row reader first."); + if (!frozen.Tables.Any(t => ReferenceEquals(t, table))) throw new ArgumentException("Table must belong to this frozen schema.", nameof(table)); + cancellation.ThrowIfCancellationRequested(); + var columns = table.Columns.Where(c => !c.Computed && !c.RowVersion).OrderBy(c => c.Ordinal).ToList(); + // A rowversion/computed-only table still needs a row count for DEFAULT VALUES. + string projection = columns.Count == 0 ? "CAST(1 AS bit) AS [__mysqlpunk_row]" : string.Join(",", columns.Select(MssqlValueCodec.SourceExpression)); + var command = new SqlCommand("SELECT " + projection + " FROM " + MssqlExportMetadataReader.Qualified(table.Key), session.Connection, transaction) { CommandTimeout = 0 }; + var owner = new MssqlCommandOwner(command, cancellation); + try { activeReader = new MssqlRowReader(owner.ExecuteReader(CommandBehavior.SequentialAccess), owner); return activeReader; } + catch { owner.Dispose(); throw; } + } + catch { Dispose(); throw; } + } + public void CompleteRead(MssqlExportSchema schema, CancellationToken cancellation) + { + CheckAlive(); + try + { + if (frozen == null || !ReferenceEquals(schema, frozen)) throw new InvalidOperationException("Schema does not belong to this read."); + if (activeReader != null && !activeReader.IsDisposed) throw new InvalidOperationException("Close the row reader before completing the read."); + cancellation.ThrowIfCancellationRequested(); + var current = metadata.ReadSchema(session.Connection, transaction, request, cancellation); + RequireValid(new MssqlExportPreflight(current, metadata.Validate(current, request))); + if (current.Fingerprint != frozen.Fingerprint) throw new InvalidOperationException("Source definitions changed during export; discard this export."); + cancellation.ThrowIfCancellationRequested(); + if (transaction != null) { transaction.Commit(); transaction.Dispose(); transaction = null; } + } + finally { Dispose(); } + } + private static void RequireValid(MssqlExportPreflight preflight) + { if (!preflight.CanStart) throw new InvalidOperationException(string.Join(Environment.NewLine, preflight.Issues.Select(x => x.Code + ": " + x.Message))); } + private void CheckAlive() { if (disposed) throw new ObjectDisposedException(nameof(MssqlExportSource)); } + private void CheckIdle() { CheckAlive(); if (request != null) throw new InvalidOperationException("A read is already active."); } + public void Dispose() + { + if (disposed) return; disposed = true; + try { if (activeReader != null) activeReader.Dispose(); } + finally + { + try { if (transaction != null) { try { transaction.Rollback(); } catch (InvalidOperationException) { } finally { transaction.Dispose(); transaction = null; } } } + finally { session.Dispose(); } + } + } + } + internal sealed class MssqlCommandOwner : IDisposable + { + private DbCommand command; + private CancellationTokenRegistration registration; + private readonly CancellationToken cancellation; + internal MssqlCommandOwner(DbCommand command, CancellationToken cancellation) + { + this.command = command; + this.cancellation = cancellation; + registration = cancellation.Register(() => { try { command.Cancel(); } catch (ObjectDisposedException) { } }); + } + internal DbDataReader ExecuteReader(CommandBehavior behavior) + { + // The provider must observe cancellation during startup, not just receive + // an idle Cancel() before synchronous execution starts. Keep our separate + // registration alive for subsequent synchronous reads of the returned reader. + return command.ExecuteReaderAsync(behavior, cancellation).GetAwaiter().GetResult(); + } + public void Dispose() + { + var value = command; + if (value == null) return; + command = null; + try { registration.Dispose(); } finally { value.Dispose(); } + } + } +} diff --git a/mySQLPunk/lib/MssqlExport/MssqlLobScriptWriter.cs b/mySQLPunk/lib/MssqlExport/MssqlLobScriptWriter.cs new file mode 100644 index 0000000..8ba0c5c --- /dev/null +++ b/mySQLPunk/lib/MssqlExport/MssqlLobScriptWriter.cs @@ -0,0 +1,161 @@ +using System; +using System.Collections.Generic; +using System.Data.Common; +using System.Globalization; +using System.Linq; +using System.Text; +using System.Threading; + +namespace mySQLPunk.lib.MssqlExport +{ + // One instance per export job, reused across tables. Projection ordinals are zero-based; + // the monotonically increasing protocol sequence is independent of source row/column IDs. + internal sealed class MssqlLobScriptWriter : IMssqlLargeRowWriter + { + private readonly string state, values, fields; + private long sequence; + private MssqlTableDefinition table; + private IReadOnlyList columns; + private string Identity { get { return "[Sequence]=" + N(sequence) + " AND [TableId]=" + N(table.Key.ObjectId); } } + internal MssqlLobScriptWriter(Guid jobId) + { + if(jobId==Guid.Empty) throw new ArgumentException("A job identity is required.",nameof(jobId)); + string root="#mp_"+jobId.ToString("N");state=root+"_s";values=root+"_v";fields=root+"_c"; + } + public long WriteRow(DbDataReader reader, MssqlTableDefinition table, IReadOnlyList columns, + MssqlRowContinuation row, MssqlScriptBatchWriter output, CancellationToken cancellation, Action onAdditionalBytes) + { + cancellation.ThrowIfCancellationRequested(); + BeginRow(table,columns,output); + long additional=0; + for(int i=0;i=2 && buffer[emit-1]>=0xD8 && buffer[emit-1]<=0xDB)emit-=2; + } + if(emit==0)throw new InvalidOperationException("Cannot advance LOB chunk."); + WriteChunk(i,index++,buffer,emit,output);total+=emit; + Buffer.BlockCopy(buffer,emit,buffer,0,used-emit);used-=emit; + } + CompleteColumn(i,false,index,total,output); + } + cancellation.ThrowIfCancellationRequested();CommitRow(output);return additional; + } + internal void BeginRow(MssqlTableDefinition table,IReadOnlyList columns,MssqlScriptBatchWriter output) + { + this.table=table;this.columns=columns;sequence=checked(sequence+1); + output.Flush(); + if(sequence==1) { + output.WriteStatement("IF OBJECT_ID('tempdb.."+state+"') IS NOT NULL THROW 51000,'LOB job already exists',1; CREATE TABLE "+state+" ([Sequence] bigint NOT NULL,[TableId] int NOT NULL,[ConsumedThrough] bigint NOT NULL,[Consumed] bit NOT NULL,[Failed] bit NOT NULL); INSERT INTO "+state+" VALUES(0,0,0,1,0);");output.Flush(); + } + string definition=string.Join(",",columns.Select((c,i)=>Field(i)+" "+StageType(c)+" NULL")); + string initial=string.Join(",",columns.Select(c=>MssqlValueCodec.IsBinaryProjected(c) ? (Unicode(c) ? "CONVERT(nvarchar(max),0x)" : "0x") : "NULL")); + // Compile the initial INSERT after CREATE: the reusable temp table may still + // have the previous table's shape when this batch is first compiled. Only + // generated empty/NULL constants enter EXEC, never source values or SQL names. + string body="IF NOT EXISTS(SELECT 1 FROM "+state+" WHERE [Failed]=0 AND [Consumed]=1 AND [ConsumedThrough]="+N(sequence-1)+") THROW 51000,'LOB prior row not consumed',1; "+ + "UPDATE "+state+" SET [Sequence]="+N(sequence)+",[TableId]="+N(table.Key.ObjectId)+",[Consumed]=0; "+ + "IF OBJECT_ID('tempdb.."+values+"') IS NOT NULL DROP TABLE "+values+"; IF OBJECT_ID('tempdb.."+fields+"') IS NOT NULL DROP TABLE "+fields+"; "+ + "CREATE TABLE "+values+" ("+definition+"); EXEC(N'INSERT INTO "+values+" VALUES("+initial+");'); "+ + "CREATE TABLE "+fields+" ([Ordinal] int NOT NULL PRIMARY KEY,[Chunks] bigint NOT NULL,[Bytes] bigint NOT NULL,[ExpectedChunks] bigint NULL,[ExpectedBytes] bigint NULL,[IsNull] bit NOT NULL,[Completed] bit NOT NULL); "+ + string.Join(" ",columns.Select((c,i)=>"INSERT INTO "+fields+" VALUES("+N(i)+",0,0,NULL,NULL,0,0);")); + Guarded(body,output,false,true); + } + internal void WriteScalar(int ordinal,string scalarExpression,MssqlScriptBatchWriter output) + { + Guarded(OpenField(ordinal)+"UPDATE "+values+" SET "+Field(ordinal)+"="+scalarExpression+"; UPDATE "+fields+" SET [Completed]=1,[IsNull]="+(scalarExpression=="NULL" ? "1" : "0")+",[ExpectedChunks]=0,[ExpectedBytes]=COALESCE((SELECT CONVERT(bigint,DATALENGTH("+Field(ordinal)+")) FROM "+values+"),0),[Bytes]=COALESCE((SELECT CONVERT(bigint,DATALENGTH("+Field(ordinal)+")) FROM "+values+"),0) WHERE [Ordinal]="+N(ordinal)+";",output); + } + internal void WriteChunk(int ordinal,long chunkIndex,byte[] encodedBytes,int count,MssqlScriptBatchWriter output) + { + if(count<=0 || count>encodedBytes.Length || count>MssqlExportLimits.BinaryChunkBytes || (Unicode(columns[ordinal]) && ((count&1)!=0 || count>MssqlExportLimits.TextChunkCodeUnits*2)))throw new ArgumentOutOfRangeException(nameof(count)); + var hex=new StringBuilder(2+count*2);hex.Append("0x");for(int j=0;j0 OR "+N(expectedBytes)+"<>0 THROW 51000,'LOB null totals',1; UPDATE "+values+" SET "+Field(ordinal)+"=NULL; " : LengthGuard(ordinal))+ + "UPDATE "+fields+" SET [Completed]=1,[IsNull]="+(isNull ? "1" : "0")+",[ExpectedChunks]="+N(chunkCount)+",[ExpectedBytes]="+N(expectedBytes)+" WHERE [Ordinal]="+N(ordinal)+";",output); + } + internal void CommitRow(MssqlScriptBatchWriter output) + { + string checks="IF (SELECT COUNT(*) FROM "+fields+")<>"+N(columns.Count)+" OR EXISTS(SELECT 1 FROM "+fields+" WHERE [Completed]<>1 OR [ExpectedChunks] IS NULL OR [ExpectedBytes] IS NULL OR [Chunks]<>[ExpectedChunks] OR [Bytes]<>[ExpectedBytes]) THROW 51000,'LOB incomplete row',1; "; + string lengths=string.Join(",",columns.Select((c,i)=>"("+N(i)+",CONVERT(bigint,DATALENGTH("+Field(i)+")))")); + checks+="IF EXISTS(SELECT 1 FROM "+values+" CROSS APPLY(VALUES "+lengths+") AS actual([Ordinal],[Bytes]) LEFT JOIN "+fields+" AS expected ON actual.[Ordinal]=expected.[Ordinal] WHERE expected.[Ordinal] IS NULL OR COALESCE(actual.[Bytes],0)<>expected.[Bytes] OR expected.[IsNull]<>CASE WHEN actual.[Bytes] IS NULL THEN 1 ELSE 0 END) THROW 51000,'LOB final length or null mismatch',1; "; + string target=MssqlValueCodec.QuoteIdentifier(table.Key.Schema)+"."+MssqlValueCodec.QuoteIdentifier(table.Key.Name); + Guarded(checks+"INSERT INTO "+target+" ("+string.Join(",",columns.Select(c=>MssqlValueCodec.QuoteIdentifier(c.Name)))+") SELECT "+string.Join(",",columns.Select((c,i)=>c.NativeType=="xml" ? "CONVERT(xml,"+Field(i)+",1)" : c.NativeType=="geometry" ? "geometry::Deserialize("+Field(i)+")" : Field(i)))+" FROM "+values+"; "+ + "UPDATE "+state+" SET [Consumed]=1,[ConsumedThrough]="+N(sequence)+"; DELETE FROM "+values+"; DELETE FROM "+fields+";",output,true,true); + } + private string OpenField(int ordinal) + { return "IF NOT EXISTS(SELECT 1 FROM "+fields+" WHERE [Ordinal]="+N(ordinal)+" AND [Completed]=0) THROW 51000,'LOB column completed or missing',1; "; } + private string LengthGuard(int ordinal) + { return "IF NOT EXISTS(SELECT 1 FROM "+fields+" WHERE [Ordinal]="+N(ordinal)+" AND [Bytes]=COALESCE((SELECT CONVERT(bigint,DATALENGTH("+Field(ordinal)+")) FROM "+values+"),0)) THROW 51000,'LOB byte length',1; "; } + private void Guarded(string body,MssqlScriptBatchWriter output,bool active=true,bool metadataOnly=false) + { + string check=active ? "IF NOT EXISTS(SELECT 1 FROM "+state+" WHERE "+Identity+" AND [Failed]=0 AND [Consumed]=0 AND [ConsumedThrough]="+N(sequence-1)+") THROW 51000,'LOB inactive or consumed row',1; IF (SELECT COUNT(*) FROM "+values+")<>1 THROW 51000,'LOB missing stage',1; " : ""; + string batch="BEGIN TRY BEGIN TRANSACTION; "+check+body+" COMMIT TRANSACTION; END TRY BEGIN CATCH IF XACT_STATE()<>0 ROLLBACK TRANSACTION; UPDATE "+state+" SET [Failed]=1; THROW; END CATCH;"; + if(metadataOnly)output.WriteMetadataStatement(batch);else output.WriteStatement(batch); + // Each short transaction is a self-contained batch. Failed clients that continue at + // the next GO encounter the durable Failed bit before any further target mutation. + output.Flush(); + } + private static string StageType(MssqlColumnDefinition c) + { + if(Unicode(c))return "nvarchar(max)"; + if(c.NativeType=="varchar" || c.NativeType=="char" || c.NativeType=="text") { + if(string.IsNullOrEmpty(c.Collation) || c.Collation.Any(ch=>!(char.IsLetterOrDigit(ch) && ch<128) && ch!='_'))throw new InvalidOperationException("Native ANSI collation is required."); + return "varchar(max) COLLATE "+c.Collation; + } + if(MssqlValueCodec.IsBinaryProjected(c))return "varbinary(max)"; + switch(c.NativeType) { + case "decimal":case "numeric":return c.NativeType+"("+N(c.Precision)+","+N(c.Scale)+")"; + case "time":case "datetime2":case "datetimeoffset":return c.NativeType+"("+N(c.Scale)+")"; + case "bit":case "tinyint":case "smallint":case "int":case "bigint":case "money":case "smallmoney":case "real":case "float":case "date":case "datetime":case "smalldatetime":case "uniqueidentifier":return c.NativeType; + default:throw new NotSupportedException("No LOB staging type for "+c.NativeType); + } + } + private static bool Unicode(MssqlColumnDefinition c) { return c.NativeType=="nvarchar" || c.NativeType=="nchar" || c.NativeType=="ntext" || c.NativeType=="xml"; } + private static bool IsNull(MssqlRowContinuation row,int ordinal) { if(row.ValueLength(ordinal)!=4)return false;var b=new char[4];row.CopyValueChars(ordinal,0,b,0,4);return new string(b)=="NULL"; } + private static int Nibble(char c) { return c<='9' ? c-'0' : c-'A'+10; } + private static string Field(int ordinal) { return "[c"+N(ordinal)+"]"; } + private static string N(long value) { return value.ToString(CultureInfo.InvariantCulture); } + } +} diff --git a/mySQLPunk/lib/MssqlExport/MssqlOwnedSession.cs b/mySQLPunk/lib/MssqlExport/MssqlOwnedSession.cs new file mode 100644 index 0000000..8fc77ca --- /dev/null +++ b/mySQLPunk/lib/MssqlExport/MssqlOwnedSession.cs @@ -0,0 +1,27 @@ +using System; +using System.Data.SqlClient; +using System.Threading; + +namespace mySQLPunk.lib.MssqlExport +{ + // The provider owner also owns any SSH lease; never dispose its connection separately. + internal sealed class MssqlOwnedSession : IDisposable + { + private IDatabase owner; + internal SqlConnection Connection { get; private set; } + + internal MssqlOwnedSession(IDatabase owner, SqlConnection connection) + { + if (owner == null) throw new ArgumentNullException(nameof(owner)); + if (connection == null) throw new ArgumentNullException(nameof(connection)); + this.owner = owner; + Connection = connection; + } + + public void Dispose() + { + IDatabase owned = Interlocked.Exchange(ref owner, null); + if (owned != null) owned.Dispose(); + } + } +} diff --git a/mySQLPunk/lib/MssqlExport/MssqlSchemaScriptWriter.cs b/mySQLPunk/lib/MssqlExport/MssqlSchemaScriptWriter.cs new file mode 100644 index 0000000..b53a438 --- /dev/null +++ b/mySQLPunk/lib/MssqlExport/MssqlSchemaScriptWriter.cs @@ -0,0 +1,268 @@ +using System; +using System.Collections.Generic; +using System.Globalization; +using System.IO; +using System.Linq; +using System.Text; +using System.Threading; + +namespace mySQLPunk.lib.MssqlExport +{ + // Native catalog DDL only. Data/IDENTITY_INSERT belong to the streaming data writer. + internal sealed class MssqlSchemaScriptWriter + { + internal void WriteBeforeData(MssqlExportSchema schema, MssqlExportRequest request, TextWriter output, CancellationToken cancellation) + { + Validate(schema, request, output, cancellation); + Settings(output); + if (request.Mode == MssqlExportMode.DataOnly) return; + output.WriteLine("-- Restore into an empty, compatible target selected by the caller; existing objects fail."); + output.WriteLine("-- Named filegroups must already exist. Rowversion values are regenerated."); + output.WriteLine("-- Identity preserves seed/increment and inserted values, not historical allocation gaps."); + foreach (string name in schema.Tables.Select(t => t.Key.Schema).Concat(schema.Views.Select(v => v.Key.Schema)).Distinct(StringComparer.Ordinal).OrderBy(x => x, StringComparer.Ordinal)) + { + cancellation.ThrowIfCancellationRequested(); + output.WriteLine("IF SCHEMA_ID(" + Literal(name) + ") IS NULL EXEC(" + Literal("CREATE SCHEMA " + Quote(name)) + ");"); + } + Batch(output); + foreach (var table in schema.Tables.OrderBy(t => t.Key.ObjectId)) + { + cancellation.ThrowIfCancellationRequested(); + output.WriteLine("CREATE TABLE " + Qualified(table.Key) + " ("); + var columns = table.Columns.OrderBy(c => c.Ordinal).ToArray(); + for (int i = 0; i < columns.Length; i++) + { + cancellation.ThrowIfCancellationRequested(); + output.Write(" " + Column(columns[i])); + output.WriteLine(i + 1 == columns.Length ? "" : ","); + } + var heap = table.Indexes.FirstOrDefault(i => i.Number("type") == 0); + output.WriteLine(")" + Placement(heap) + ";"); + if (table.Text("lock_escalation_desc") != null) + output.WriteLine("ALTER TABLE " + Qualified(table.Key) + " SET (LOCK_ESCALATION = " + table.Text("lock_escalation_desc") + ");"); + TableOption(output, table, "text_in_row_limit", "text in row"); + TableOption(output, table, "large_value_types_out_of_row", "large value types out of row"); + Batch(output); + } + } + + internal void WriteAfterData(MssqlExportSchema schema, MssqlExportRequest request, TextWriter output, CancellationToken cancellation) + { + Validate(schema, request, output, cancellation); + if (request.Mode == MssqlExportMode.DataOnly) return; + Settings(output); + // All candidate keys precede any FK, including cross-table cycles. + foreach (var table in schema.Tables.OrderBy(t => t.Key.ObjectId)) + foreach (var index in table.Indexes.Where(i => i.Number("type") != 0).OrderBy(i => i.Number("type")).ThenBy(i => i.IndexId)) + { + cancellation.ThrowIfCancellationRequested(); + WriteIndex(table, index, output); + } + foreach (var table in schema.Tables.OrderBy(t => t.Key.ObjectId)) + { + foreach (var constraint in table.Constraints.OrderBy(c => c.Name, StringComparer.Ordinal)) + { + cancellation.ThrowIfCancellationRequested(); + WriteConstraint(table, constraint, output); + } + Comment(output, table.Key, null, table.Comment); + foreach (var column in table.Columns) { cancellation.ThrowIfCancellationRequested(); Comment(output, table.Key, column.Name, column.Comment); } + } + var pending = schema.Views.ToDictionary(v => v.Key.ObjectId); + while (pending.Count != 0) + { + var ready = pending.Values.Where(v => !schema.Dependencies.Any(d => d.ObjectId == v.Key.ObjectId && d.ReferencedId.HasValue && pending.ContainsKey(d.ReferencedId.Value))).OrderBy(v => v.Key.ObjectId).ToArray(); + if (ready.Length == 0) throw new InvalidOperationException("View dependency cycle."); + foreach (var view in ready) + { + cancellation.ThrowIfCancellationRequested(); + output.WriteLine("SET ANSI_NULLS " + OnOff(view.AnsiNulls) + ";"); + output.WriteLine("SET QUOTED_IDENTIFIER " + OnOff(view.QuotedIdentifier) + ";"); + Batch(output); + output.WriteLine(ViewDefinition(view)); + Batch(output); + pending.Remove(view.Key.ObjectId); + } + } + // Disabling a clustered index first makes its table inaccessible and can disable FKs. + // Delay all requested disabled states until every object is in place. + Settings(output); + foreach (var table in schema.Tables) + foreach (var index in table.Indexes.Where(i => i.Flag("is_disabled")).OrderByDescending(i => i.Number("type"))) + { + cancellation.ThrowIfCancellationRequested(); + output.WriteLine("ALTER INDEX " + Quote(index.Name) + " ON " + Qualified(table.Key) + " DISABLE;"); + } + Batch(output); + } + + private static void Validate(MssqlExportSchema schema, MssqlExportRequest request, TextWriter output, CancellationToken cancellation) + { + if (schema == null) throw new ArgumentNullException(nameof(schema)); + if (request == null) throw new ArgumentNullException(nameof(request)); + if (output == null) throw new ArgumentNullException(nameof(output)); + cancellation.ThrowIfCancellationRequested(); + var issues = new MssqlExportMetadataReader().Validate(schema, request); + if (issues.Count != 0) throw new InvalidOperationException(string.Join("; ", issues.Select(i => i.Code + ": " + i.Message))); + if (request.Mode == MssqlExportMode.DataOnly) return; + foreach (var view in schema.Views) ViewDefinition(view); // Fail before any partial DDL. + foreach (var table in schema.Tables) + { + if (table.Columns.Count == 0) throw new InvalidOperationException("Table has no visible columns: " + Qualified(table.Key)); + foreach (var c in table.Constraints) + if (c.Kind != "FK" && c.Kind != "CHECK") throw new InvalidOperationException("Unsupported constraint: " + c.Kind); + } + } + private static string Column(MssqlColumnDefinition c) + { + var sql = new StringBuilder(Quote(c.Name)); + if (c.Computed) + return sql.Append(" AS ").Append(c.ComputedDefinition).Append(c.Persisted ? " PERSISTED" : "").Append(c.Persisted && !c.Nullable ? " NOT NULL" : "").ToString(); + sql.Append(' ').Append(NativeType(c)); + if (c.Collation != null) sql.Append(" COLLATE ").Append(c.Collation); + if (c.Identity) sql.Append(" IDENTITY(").Append(c.IdentitySeed).Append(',').Append(c.IdentityIncrement).Append(')').Append(c.Flag("identity_not_for_replication") ? " NOT FOR REPLICATION" : ""); + if (c.Flag("is_rowguidcol")) sql.Append(" ROWGUIDCOL"); + sql.Append(c.Nullable ? " NULL" : " NOT NULL"); + if (c.DefaultDefinition != null) sql.Append(" CONSTRAINT ").Append(Quote(c.DefaultName)).Append(" DEFAULT ").Append(c.DefaultDefinition); + return sql.ToString(); + } + private static string NativeType(MssqlColumnDefinition c) + { + string type = c.NativeType.ToLowerInvariant(); + if (c.RowVersion) return "rowversion"; + switch (type) + { + case "varchar": case "char": case "binary": case "varbinary": + return type + "(" + (c.Length == -1 ? "max" : Number(c.Length)) + ")"; + case "nvarchar": case "nchar": + return type + "(" + (c.Length == -1 ? "max" : Number(c.Length / 2)) + ")"; + case "decimal": case "numeric": return type + "(" + Number(c.Precision) + "," + Number(c.Scale) + ")"; + case "datetime2": case "datetimeoffset": case "time": return type + "(" + Number(c.Scale) + ")"; + case "float": return type + "(" + Number(c.Precision) + ")"; + default: return type; + } + } + private static void WriteIndex(MssqlTableDefinition table, MssqlIndexDefinition index, TextWriter output) + { + string keys = string.Join(", ", index.Columns.Where(c => c.Number("key_ordinal") > 0).OrderBy(c => c.Number("key_ordinal")).Select(c => Quote(c.Text("name")) + (c.Flag("is_descending_key") ? " DESC" : " ASC"))); + string kind = index.Number("type") == 1 ? "CLUSTERED" : "NONCLUSTERED"; + bool constraint = index.Flag("is_primary_key") || index.Flag("is_unique_constraint"); + string sql = constraint + ? "ALTER TABLE " + Qualified(table.Key) + " ADD CONSTRAINT " + Quote(index.Name) + (index.Flag("is_primary_key") ? " PRIMARY KEY " : " UNIQUE ") + kind + : "CREATE " + (index.Flag("is_unique") ? "UNIQUE " : "") + kind + " INDEX " + Quote(index.Name) + " ON " + Qualified(table.Key); + sql += " (" + keys + ")"; + var includes = index.Columns.Where(c => c.Flag("is_included_column")).OrderBy(c => c.Number("index_column_id")).Select(c => Quote(c.Text("name"))).ToArray(); + if (includes.Length != 0) sql += " INCLUDE (" + string.Join(", ", includes) + ")"; + if (index.Flag("has_filter")) sql += " WHERE " + index.Text("filter_definition"); + var options = new List(); + foreach (var pair in new[] { new[] {"is_padded","PAD_INDEX"}, new[] {"ignore_dup_key","IGNORE_DUP_KEY"}, new[] {"allow_row_locks","ALLOW_ROW_LOCKS"}, new[] {"allow_page_locks","ALLOW_PAGE_LOCKS"}, new[] {"no_recompute","STATISTICS_NORECOMPUTE"}, new[] {"optimize_for_sequential_key","OPTIMIZE_FOR_SEQUENTIAL_KEY"} }) + if (index.Text(pair[0]) != null) options.Add(pair[1] + " = " + OnOff(index.Flag(pair[0]))); + if (index.Number("fill_factor") != 0) options.Add("FILLFACTOR = " + Number(index.Number("fill_factor"))); + if (options.Count != 0) sql += " WITH (" + string.Join(", ", options) + ")"; + output.WriteLine(sql + Placement(index) + ";"); + Batch(output); + } + private static void WriteConstraint(MssqlTableDefinition table, MssqlConstraintDefinition c, TextWriter output) + { + string sql = "ALTER TABLE " + Qualified(table.Key) + (c.Flag("is_not_trusted") || c.Flag("is_disabled") ? " WITH NOCHECK" : " WITH CHECK") + " ADD CONSTRAINT " + Quote(c.Name) + " "; + if (c.Kind == "CHECK") sql += "CHECK " + (c.Flag("is_not_for_replication") ? "NOT FOR REPLICATION " : "") + c.Text("definition"); + else + { + var columns = c.Columns.OrderBy(x => x.Number("constraint_column_id")).ToArray(); + sql += "FOREIGN KEY (" + string.Join(", ", columns.Select(x => Quote(x.Text("parent_column")))) + ") REFERENCES " + Quote(c.Text("referenced_schema")) + "." + Quote(c.Text("referenced_table")) + " (" + string.Join(", ", columns.Select(x => Quote(x.Text("referenced_column")))) + ")"; + foreach (string action in new[] { "delete", "update" }) + if (c.Text(action + "_referential_action_desc") != null) sql += " ON " + action.ToUpperInvariant() + " " + c.Text(action + "_referential_action_desc").Replace('_', ' '); + if (c.Flag("is_not_for_replication")) sql += " NOT FOR REPLICATION"; + } + output.WriteLine(sql + ";"); + output.WriteLine("ALTER TABLE " + Qualified(table.Key) + (c.Flag("is_disabled") ? " NOCHECK" : " CHECK") + " CONSTRAINT " + Quote(c.Name) + ";"); + Batch(output); + } + private static void Comment(TextWriter output, MssqlObjectKey key, string column, string value) + { + if (value == null) return; + output.WriteLine("EXEC sys.sp_addextendedproperty @name=N'MS_Description', @value=" + Literal(value) + ", @level0type=N'SCHEMA', @level0name=" + Literal(key.Schema) + ", @level1type=N'TABLE', @level1name=" + Literal(key.Name) + (column == null ? "" : ", @level2type=N'COLUMN', @level2name=" + Literal(column)) + ";"); + Batch(output); + } + private static void TableOption(TextWriter output, MssqlTableDefinition table, string fact, string option) + { + if (table.Number(fact) != 0) output.WriteLine("EXEC sys.sp_tableoption @TableNamePattern=" + Literal(Qualified(table.Key)) + ", @OptionName=" + Literal(option) + ", @OptionValue=" + Literal(table.Text(fact) == "True" ? "1" : table.Text(fact) == "False" ? "0" : table.Text(fact)) + ";"); + } + // Read only the module header. Comments and quoted identifiers may contain keywords. + // Rewrite ALTER/CREATE OR ALTER to CREATE so an existing target cannot be overwritten. + // The catalog name wins over stale module text left by sp_rename. + private static string ViewDefinition(MssqlViewDefinition view) + { + string text = view.Definition; int position = 0; + string token = Token(text, ref position); + if (string.Equals(token, "CREATE", StringComparison.OrdinalIgnoreCase)) + { + token = Token(text, ref position); + if (string.Equals(token, "OR", StringComparison.OrdinalIgnoreCase)) + { + if (!string.Equals(Token(text, ref position), "ALTER", StringComparison.OrdinalIgnoreCase)) throw new InvalidOperationException("Invalid view header."); + token = Token(text, ref position); + } + } + else if (string.Equals(token, "ALTER", StringComparison.OrdinalIgnoreCase)) token = Token(text, ref position); + else throw new InvalidOperationException("Unsupported view module header: " + Qualified(view.Key)); + if (!string.Equals(token, "VIEW", StringComparison.OrdinalIgnoreCase)) throw new InvalidOperationException("Invalid view header."); + if (Token(text, ref position).Length == 0) throw new InvalidOperationException("Missing view name."); + int end = position; int look = position; + if (Token(text, ref look) == ".") { if (Token(text, ref look).Length == 0) throw new InvalidOperationException("Missing view name."); end = look; } + return "CREATE VIEW " + Qualified(view.Key) + text.Substring(end); + } + private static string Token(string text, ref int position) + { + while (position < text.Length) + { + if (char.IsWhiteSpace(text[position])) { position++; continue; } + if (position + 1 < text.Length && text.Substring(position, 2) == "--") + { while (position < text.Length && text[position] != '\n') position++; continue; } + if (position + 1 < text.Length && text.Substring(position, 2) == "/*") + { + position += 2; int depth = 1; + while (position < text.Length && depth != 0) + { + if (position + 1 < text.Length && text.Substring(position, 2) == "/*") { depth++; position += 2; } + else if (position + 1 < text.Length && text.Substring(position, 2) == "*/") { depth--; position += 2; } + else position++; + } + if (depth != 0) throw new InvalidOperationException("Unclosed view header comment."); + continue; + } + break; + } + int start = position; + if (position == text.Length) return ""; + char first = text[position++]; + if (first == '[' || first == '"') + { + char closing = first == '[' ? ']' : '"'; bool closed = false; + while (position < text.Length) + { + if (text[position++] != closing) continue; + if (position < text.Length && text[position] == closing) { position++; continue; } + closed = true; break; + } + if (!closed) throw new InvalidOperationException("Unclosed view identifier."); + } + else if (char.IsLetterOrDigit(first) || first == '_' || first == '@' || first == '#') + while (position < text.Length && (char.IsLetterOrDigit(text[position]) || "_@$#".IndexOf(text[position]) >= 0)) position++; + return text.Substring(start, position - start); + } + private static string Placement(MssqlIndexDefinition index) => index?.Text("data_space_name") == null ? "" : " ON " + Quote(index.Text("data_space_name")); + private static string Number(int value) => value.ToString(CultureInfo.InvariantCulture); + private static string OnOff(bool value) => value ? "ON" : "OFF"; + private static string Quote(string value) => MssqlExportMetadataReader.Quote(value); + private static string Qualified(MssqlObjectKey key) => MssqlExportMetadataReader.Qualified(key); + private static string Literal(string value) => "N'" + value.Replace("'", "''") + "'"; + private static void Batch(TextWriter output) { output.WriteLine("GO"); } + private static void Settings(TextWriter output) + { + output.WriteLine("SET ANSI_NULLS ON; SET QUOTED_IDENTIFIER ON; SET ANSI_PADDING ON; SET ANSI_WARNINGS ON;"); + output.WriteLine("SET ARITHABORT ON; SET CONCAT_NULL_YIELDS_NULL ON; SET NUMERIC_ROUNDABORT OFF;"); + Batch(output); + } + } +} diff --git a/mySQLPunk/lib/MssqlExport/MssqlSqlExportService.cs b/mySQLPunk/lib/MssqlExport/MssqlSqlExportService.cs new file mode 100644 index 0000000..f58a35f --- /dev/null +++ b/mySQLPunk/lib/MssqlExport/MssqlSqlExportService.cs @@ -0,0 +1,147 @@ +using System; +using System.Collections.Generic; +using System.Diagnostics; +using System.IO; +using System.Text; +using System.Threading; + +namespace mySQLPunk.lib.MssqlExport +{ + internal class MssqlSqlExportService + { + private readonly Func sourceFactory; + private readonly MssqlArtifactFiles files; + private readonly Action localPhase; + internal MssqlSqlExportService(Func sourceFactory) : this(sourceFactory, new MssqlArtifactFiles(), null) { } + internal MssqlSqlExportService(Dictionary connection) : this(OwnedFactory(connection)) { } + // Existing file operations plus a local phase boundary are sufficient for deterministic faults. + internal MssqlSqlExportService(Func sourceFactory, MssqlArtifactFiles files, Action localPhase) + { this.sourceFactory = sourceFactory ?? throw new ArgumentNullException(nameof(sourceFactory)); this.files = files ?? throw new ArgumentNullException(nameof(files)); this.localPhase = localPhase; } + private static Func OwnedFactory(Dictionary connection) + { + if (connection == null) throw new ArgumentNullException(nameof(connection)); + var settings = new Dictionary(connection); + return () => new MssqlExportSource(ConnectionOpenService.OpenMssqlSession(new Dictionary(settings))); + } + internal virtual MssqlExportCatalog LoadCatalog(string databaseName, CancellationToken cancellation) + { + try { cancellation.ThrowIfCancellationRequested(); using (var source = sourceFactory()) return source.ReadCatalog(databaseName, cancellation); } + catch (Exception ex) { throw SafeInspectionFailure("catalog", ex, cancellation); } + } + internal virtual MssqlExportPreflight Preflight(MssqlExportRequest request, CancellationToken cancellation) + { + try { cancellation.ThrowIfCancellationRequested(); using (var source = sourceFactory()) return source.Inspect(request, cancellation); } + catch (Exception ex) { throw SafeInspectionFailure("preflight", ex, cancellation); } + } + private static Exception SafeInspectionFailure(string operation, Exception error, CancellationToken cancellation) + { + if (cancellation.IsCancellationRequested) return new OperationCanceledException(cancellation); + // No original exception as InnerException: even ToString must remain safe for a UI log. + return new InvalidOperationException("Export " + operation + " failed (" + error.GetType().Name + ")."); + } + internal virtual MssqlExportResult Export(MssqlExportRequest request, IProgress progress, CancellationToken cancellation) + { + if (request == null) throw new ArgumentNullException(nameof(request)); + var clock = Stopwatch.StartNew(); var counts = new List(); + var warnings = new List(); var residuals = new List(); + MssqlExportArtifact artifact = null; MssqlExportMirror mirror = null; + var estimator = new MssqlExportProgressEstimator(request.JobId, new MssqlExportSchema(null, null, null, 0, null), request.Mirror != null); + var stage = MssqlExportStage.Preflight; bool succeeded = false; bool canceled = false; string error = null; + long written = 0; + Action report = p => { if (p != null && progress != null) { try { progress.Report(p); } catch { /* Observer failure must not change publication truth. */ } } }; + Action phase = s => { + stage = s; report(estimator.Sample(s, null, 0, 0, written, clock.Elapsed, true)); + localPhase?.Invoke(s); cancellation.ThrowIfCancellationRequested(); + }; + try + { + report(estimator.Sample(stage, null, 0, 0, 0, clock.Elapsed, true)); + cancellation.ThrowIfCancellationRequested(); + using (var source = sourceFactory()) + { + var preflight = source.Inspect(request, cancellation); + if (!preflight.CanStart) throw new InvalidOperationException("Export preflight did not pass."); + var schema = source.BeginRead(request, cancellation); + estimator = new MssqlExportProgressEstimator(request.JobId, schema, request.Mirror != null); + cancellation.ThrowIfCancellationRequested(); + artifact = MssqlExportArtifact.Create(request, files); + // Wrapper releases before Complete, underlying artifact remains owned by this job. + using (var writer = new StreamWriter(artifact.Output, new UTF8Encoding(false, true), 8192, true)) + { + var ddl = new MssqlSchemaScriptWriter(); + var data = new MssqlDataScriptWriter(new MssqlLobScriptWriter(request.JobId)); + stage = MssqlExportStage.Schema; + report(estimator.Sample(stage, null, 0, 0, artifact.Output.Position, clock.Elapsed, true)); + ddl.WriteBeforeData(schema, request, writer, cancellation); + foreach (var table in schema.Tables) + { + long rows = 0, sourceBytes = 0; + stage = request.Mode == MssqlExportMode.SchemaOnly ? MssqlExportStage.Schema : MssqlExportStage.Data; + report(estimator.Sample(stage, table.Key, 0, 0, artifact.Output.Position, clock.Elapsed, true)); + if (request.Mode != MssqlExportMode.SchemaOnly) + { + using (var reader = source.OpenRows(table, cancellation)) + { + rows = data.WriteTable(reader.Reader, table, writer, (r, b) => { + rows = r; sourceBytes = b; written = artifact.Output.Position; + report(estimator.Sample(stage, table.Key, r, b, written, clock.Elapsed, false)); + }, cancellation); + } + } + counts.Add(new MssqlTableRowCount(table.Key, rows)); + written = artifact.Output.Position; + report(estimator.Sample(stage, table.Key, rows, sourceBytes, written, clock.Elapsed, true)); + } + stage = MssqlExportStage.Schema; + report(estimator.Sample(stage, null, 0, 0, artifact.Output.Position, clock.Elapsed, true)); + ddl.WriteAfterData(schema, request, writer, cancellation); + writer.Flush(); written = artifact.Output.Position; + } + source.CompleteRead(schema, cancellation); + } + // No snapshot/reader/command/connection is alive during these local phases. + var receipt = artifact.Complete(counts.AsReadOnly(), cancellation); written = receipt.Bytes; + phase(MssqlExportStage.Verifying); + artifact.Verify(receipt, (done, total) => report(estimator.SamplePhase(stage, done, total, written, clock.Elapsed, false)), cancellation); + phase(MssqlExportStage.Publishing); // Includes cancelable integrity/consent preparation. + artifact.Publish(request.Destination, cancellation); // Only the artifact's actual commit defers cancellation. + if (request.Mirror != null) + { + mirror = new MssqlExportMirror(files); + phase(MssqlExportStage.Mirroring); + long last = 0; bool verifyPass = false; + mirror.CopyVerified(artifact.PublishedPath, receipt, request.Mirror, (done, total) => { + // CopyVerified reports a copy pass followed by a verification pass (which restarts at zero). + if (!verifyPass && last == total && done <= last) verifyPass = true; + last = done; + report(estimator.SamplePhase(stage, (verifyPass ? total : 0) + done, checked(total * 2), written, clock.Elapsed, false)); + }, cancellation); + } + succeeded = true; + } + catch (Exception ex) + { + canceled = cancellation.IsCancellationRequested; + // SQL/provider messages can contain credentials or row values. Return only a safe category. + error = canceled ? "Export canceled." : "Export failed during " + stage + " (" + ex.GetType().Name + ")."; + } + finally + { + if (artifact != null) + { + written = artifact.Output.Position; + artifact.Dispose(); + warnings.AddRange(artifact.Warnings); residuals.AddRange(artifact.ResidualPaths); + } + // Mirror owns its temporary lifetime inside CopyVerified's finally. + if (mirror != null) { warnings.AddRange(mirror.Warnings); residuals.AddRange(mirror.ResidualPaths); } + } + string published = artifact?.PublishedPath, mirrorPath = mirror?.MirrorPublishedPath; + if (succeeded) stage = MssqlExportStage.Completed; + else if (published != null) stage = MssqlExportStage.LocalCompletedMirrorIncomplete; + else stage = canceled ? MssqlExportStage.Canceled : MssqlExportStage.Failed; + report(estimator.Sample(stage, null, 0, 0, written, clock.Elapsed, true)); + return new MssqlExportResult(request.JobId, stage, published, mirrorPath, counts, clock.Elapsed, warnings, residuals, error); + } + } +} diff --git a/mySQLPunk/lib/MssqlExport/MssqlValueCodec.cs b/mySQLPunk/lib/MssqlExport/MssqlValueCodec.cs new file mode 100644 index 0000000..5ce5bb2 --- /dev/null +++ b/mySQLPunk/lib/MssqlExport/MssqlValueCodec.cs @@ -0,0 +1,97 @@ +using System; +using System.Data.Common; +using System.Data.SqlTypes; +using System.Globalization; +using System.Linq; + +namespace mySQLPunk.lib.MssqlExport +{ + internal static class MssqlValueCodec + { + private static readonly CultureInfo Invariant = CultureInfo.InvariantCulture; + internal static string QuoteIdentifier(string identifier) + { + if (identifier == null) throw new ArgumentNullException(nameof(identifier)); + return "[" + identifier.Replace("]", "]]") + "]"; + } + internal static bool IsBinaryProjected(MssqlColumnDefinition column) + { + switch (column.NativeType) { + case "varchar": case "char": case "text": case "nvarchar": case "nchar": case "ntext": + case "xml": case "binary": case "varbinary": case "image": case "geometry": return true; + default: return false; + } + } + internal static string SourceExpression(MssqlColumnDefinition column) + { + string name = QuoteIdentifier(column.Name); + if (!IsBinaryProjected(column)) return name; + if (column.NativeType == "geometry") name += ".Serialize()"; + if (column.NativeType == "text") name = "CONVERT(varchar(max)," + name + ")"; + if (column.NativeType == "ntext" || column.NativeType == "xml") name = "CONVERT(nvarchar(max)," + name + ")"; + return "CONVERT(varbinary(max)," + name + ")"; + } + internal static string BinaryPrefix(MssqlColumnDefinition column) + { + switch (column.NativeType) { + case "geometry": return "geometry::Deserialize(0x"; + case "xml": return "CONVERT(xml,CONVERT(nvarchar(max),0x"; + case "nvarchar": case "nchar": case "ntext": return "CONVERT(nvarchar(max),0x"; + default: return "0x"; // Direct binary assignment preserves ANSI bytes in the target collation. + } + } + internal static string BinarySuffix(MssqlColumnDefinition column) + { + switch (column.NativeType) { + case "geometry": return ")"; + case "xml": return "),1)"; + case "nvarchar": case "nchar": case "ntext": return ")"; + default: return ""; + } + } + internal static object ReadScalar(DbDataReader reader, int ordinal, MssqlColumnDefinition column) + { + if (column.NativeType == "decimal" || column.NativeType == "numeric") + return reader.GetProviderSpecificValue(ordinal); // SqlDecimal supports precision 38. + return reader.GetValue(ordinal); // Only fixed-width types reach this method. + } + internal static int ScalarBytes(MssqlColumnDefinition column) + { + if (column.Length > 0) return column.Length; + switch (column.NativeType) { + case "bit": case "tinyint": return 1; + case "smallint": return 2; + case "int": case "real": case "smallmoney": case "smalldatetime": return 4; + case "uniqueidentifier": return 16; + case "decimal": case "numeric": return 17; + case "datetimeoffset": return 10; + case "date": return 3; + case "time": return 5; + default: return 8; + } + } + internal static string FormatScalar(object value, MssqlColumnDefinition column) + { + if (value == null || value == DBNull.Value || (value is INullable && ((INullable)value).IsNull)) return "NULL"; + switch (column.NativeType) { + case "decimal": case "numeric": + // SqlDecimal.ToString is invariant and retains the declared scale without CLR decimal conversion. + if (value is SqlDecimal) return ((SqlDecimal)value).ToString(); + if (value is decimal) return ((decimal)value).ToString(Invariant); + throw new InvalidOperationException("Expected provider SqlDecimal."); + case "money": case "smallmoney": + return (value is SqlMoney ? ((SqlMoney)value).Value : (decimal)value).ToString("0.0000", Invariant); + case "bit": return (bool)value ? "1" : "0"; + case "tinyint": case "smallint": case "int": case "bigint": return Convert.ToString(value, Invariant); + case "float": return ((double)value).ToString("R", Invariant); + case "real": return ((float)value).ToString("R", Invariant); + case "uniqueidentifier": return "'" + ((Guid)value).ToString("D") + "'"; + case "date": return "'" + ((DateTime)value).ToString("yyyy-MM-dd", Invariant) + "'"; + case "datetime": case "datetime2": case "smalldatetime": return "'" + ((DateTime)value).ToString("yyyy-MM-dd'T'HH:mm:ss.fffffff", Invariant) + "'"; + case "datetimeoffset": return "'" + ((DateTimeOffset)value).ToString("yyyy-MM-dd'T'HH:mm:ss.fffffffzzz", Invariant) + "'"; + case "time": return "'" + ((TimeSpan)value).ToString(@"hh\:mm\:ss\.fffffff", Invariant) + "'"; + default: throw new NotSupportedException("No scalar codec for " + column.NativeType + "."); + } + } + } +} diff --git a/mySQLPunk/mySQLPunk.csproj b/mySQLPunk/mySQLPunk.csproj index d6cea87..4d64dca 100644 --- a/mySQLPunk/mySQLPunk.csproj +++ b/mySQLPunk/mySQLPunk.csproj @@ -297,6 +297,8 @@ + + @@ -359,6 +361,18 @@ + + + + + + + + + + + + diff --git a/tests/MssqlExportFixtures/core-lob.sql b/tests/MssqlExportFixtures/core-lob.sql new file mode 100644 index 0000000..61eca8b --- /dev/null +++ b/tests/MssqlExportFixtures/core-lob.sql @@ -0,0 +1,24 @@ +-- Required Core-profile LOB phase, same explicitly owned source database. +-- Small reusable chunks; no whole-LOB client allocation and no multiGB value. +SET NOCOUNT ON; +INSERT dbo.MultiLob(tag,b,n,v,wkt) VALUES (4,0x,N'','',N'LINESTRING('); +DECLARE @i int=0,@b varbinary(max)=CONVERT(varbinary(max),REPLICATE(CONVERT(varchar(max),'Z'),65536)), + @n nvarchar(max)=REPLICATE(CONVERT(nvarchar(max),N'中'),8192), + @v varbinary(max)=0xA4A4002020; +WHILE @i<1024 BEGIN + UPDATE dbo.MultiLob SET b.WRITE(@b,NULL,0) WHERE tag=4; + IF @i<1025 UPDATE dbo.MultiLob SET n.WRITE(@n,NULL,0) WHERE tag=4; + UPDATE dbo.MultiLob SET v.WRITE(@v,NULL,0),wkt.WRITE(REPLICATE(N'0 0,',256),NULL,0) WHERE tag=4; + SET @i+=1; +END; +-- 1024 * 16384 = exactly 16 MiB: one more chunk crosses the acceptance threshold. +UPDATE dbo.MultiLob SET n.WRITE(@n,NULL,0),wkt.WRITE(N'1 1)',NULL,0) WHERE tag=4; +SELECT tag,DATALENGTH(b) binary_bytes,DATALENGTH(n) unicode_bytes FROM dbo.MultiLob WHERE tag=4; +-- Expected b=67108864, n=16793600. + +-- Large legacy values exercise the same raw staging with a different table shape. +DECLARE @legacy TABLE(v varchar(max) COLLATE Chinese_Taiwan_Stroke_CI_AS); +INSERT @legacy VALUES(''); +DECLARE @legacyCount int=0; +WHILE @legacyCount<70000 BEGIN UPDATE @legacy SET v.WRITE(0xA4A4002020,NULL,0); SET @legacyCount+=1; END; +INSERT dbo.MultiLob(tag,t,nt,i) SELECT 5,v,REPLICATE(CONVERT(nvarchar(max),0x3DD800DE0000200000D8),100000),CONVERT(varbinary(max),REPLICATE(CONVERT(varchar(max),'z'),1048576)) FROM @legacy; diff --git a/tests/MssqlExportFixtures/core.sql b/tests/MssqlExportFixtures/core.sql new file mode 100644 index 0000000..b0c3a90 --- /dev/null +++ b/tests/MssqlExportFixtures/core.sql @@ -0,0 +1,76 @@ +-- Run only after the caller validates its generated *_src DB and ownership token. +-- Database default must be Latin1_General_100_BIN2 (different from tempdb). +SET NOCOUNT ON; +SET XACT_ABORT ON; +GO +CREATE SCHEMA [一] AUTHORIZATION dbo; +GO +CREATE SCHEMA [二] AUTHORIZATION dbo; +GO +CREATE TABLE [一].[同名]]表] ( + [id] int IDENTITY(3,2) NOT NULL CONSTRAINT [PK_same] PRIMARY KEY, + [unique] int NOT NULL CONSTRAINT [UQ_same] UNIQUE, + [decimal38] decimal(38,18) NULL, + [offset] datetimeoffset(7) NULL, + [date] date NULL, [time] time(7) NULL, [dt2] datetime2(7) NULL, + [raw] varchar(max) COLLATE Chinese_Taiwan_Stroke_CI_AS NULL, + [unicode] nvarchar(max) NULL, + [fixed] char(5) COLLATE Chinese_Taiwan_Stroke_CI_AS NULL, + [binary] varbinary(max) NULL, + [defaulted] int NOT NULL CONSTRAINT [DF_same] DEFAULT (7), + [calc] AS ([unique]+[defaulted]) PERSISTED, + [rv] rowversion, + CONSTRAINT [CK_same] CHECK ([unique]>0) +); +CREATE TABLE [二].[同名]]表] ([id] int NOT NULL PRIMARY KEY,[parent] int NULL, + CONSTRAINT [FK_same] FOREIGN KEY ([parent]) REFERENCES [一].[同名]]表]([id])); +CREATE INDEX [IX_filtered] ON [一].[同名]]表]([unique] DESC,[id] ASC) + INCLUDE ([defaulted]) WHERE [unique]>1; +INSERT [一].[同名]]表]([unique],[decimal38],[offset],[date],[time],[dt2],[raw],[unicode],[fixed],[binary]) +VALUES (1,12345678901234567890.123456789012345678,'2026-10-12T12:34:56.1234567+05:45', + '0001-01-01','23:59:59.9999999','9999-12-31T23:59:59.9999999', + '',CONVERT(nvarchar(max),0x41003DD800DE000020002000),NULL,0x0001FF), + (2,-99999999999999999999.999999999999999999,'2026-10-12T00:00:00.0000001-03:30', + '9999-12-31','00:00:00.0000001','0001-01-01T00:00:00.0000001','',N'','',0x), + (3,NULL,NULL,NULL,NULL,NULL,NULL,NULL,NULL,NULL); +UPDATE [一].[同名]]表] SET [raw].WRITE(0xA4A4002020,NULL,0) WHERE [unique]=1; +UPDATE [一].[同名]]表] SET [fixed]=0xA4A4202020 WHERE [unique]=1; +INSERT [二].[同名]]表] VALUES (1,3); +EXEC sys.sp_addextendedproperty @name=N'MS_Description',@value=N'table comment: ] 中文', + @level0type=N'SCHEMA',@level0name=N'一',@level1type=N'TABLE',@level1name=N'同名]表'; +EXEC sys.sp_addextendedproperty @name=N'MS_Description',@value=N'column comment: '' and trailing ', + @level0type=N'SCHEMA',@level0name=N'一',@level1type=N'TABLE',@level1name=N'同名]表', + @level2type=N'COLUMN',@level2name=N'raw'; +CREATE TABLE dbo.IdentityEmpty ([id] int IDENTITY(11,3),v int); +CREATE TABLE dbo.IdentityNegative ([id] int IDENTITY(-5,-2),v int); +INSERT dbo.IdentityNegative(v) VALUES (1),(2); +CREATE TABLE dbo.IdentityReseed ([id] int IDENTITY(1,1),v int); +INSERT dbo.IdentityReseed(v) VALUES (1); +DBCC CHECKIDENT ('dbo.IdentityReseed',RESEED,100) WITH NO_INFOMSGS; +CREATE TABLE dbo.IdentityDeleted ([id] int IDENTITY(10,2),v int); +INSERT dbo.IdentityDeleted(v) VALUES (1); +DELETE dbo.IdentityDeleted; +CREATE TABLE dbo.IdentityTruncated ([id] int IDENTITY(20,4),v int); +INSERT dbo.IdentityTruncated(v) VALUES (1); +TRUNCATE TABLE dbo.IdentityTruncated; +-- No PK: row ordinal is test-side identity; compare rows as a multiset. +CREATE TABLE dbo.MultiLob (tag int, b varbinary(max),n nvarchar(max),v varchar(max) + COLLATE Chinese_Taiwan_Stroke_CI_AS,t text COLLATE Chinese_Taiwan_Stroke_CI_AS, + nt ntext,i image,x xml,wkt nvarchar(max)); +DECLARE @legacy TABLE(v varchar(max) COLLATE Chinese_Taiwan_Stroke_CI_AS); +INSERT @legacy VALUES (''); +UPDATE @legacy SET v.WRITE(0xA4A4002020,NULL,0); +INSERT dbo.MultiLob VALUES (1,0x0001FF,CONVERT(nvarchar(max),0x00D800DC00D8),0xA4A4002020, + (SELECT v FROM @legacy),CONVERT(nvarchar(max),0x41003DD800DE000020002000),0x0001FF, + CONVERT(xml,N' x ',1),N'LINESTRING(0 0,1 1)'); +INSERT dbo.MultiLob VALUES (2,0x,N'','', '',N'',0x,CONVERT(xml,N''),N'GO'+NCHAR(10)+N':r ignored'+NCHAR(10)+N'$(name)'); +INSERT dbo.MultiLob(tag) VALUES (3); +GO +SET ANSI_NULLS ON; +SET QUOTED_IDENTIFIER ON; +GO +CREATE VIEW [一].[view]]中文] AS SELECT [id],[unique],[calc] FROM [一].[同名]]表]; +GO + +CREATE TABLE dbo.LegacySmall(n ntext,i image); +INSERT dbo.LegacySmall VALUES(CONVERT(nvarchar(max),0x00D800DC00D800002000),0x00FF0020),(N'',0x),(NULL,NULL); diff --git a/tests/MssqlExportFixtures/geometry.sql b/tests/MssqlExportFixtures/geometry.sql new file mode 100644 index 0000000..0879409 --- /dev/null +++ b/tests/MssqlExportFixtures/geometry.sql @@ -0,0 +1,13 @@ +-- Native Geometry fixtures; no client geometry library or normalization. +CREATE TABLE dbo.GeometryCases(id int NOT NULL PRIMARY KEY,label nvarchar(40),g geometry NULL,h geometry NULL,b varbinary(max)); +CREATE TABLE dbo.GeometryLarge(id int NOT NULL PRIMARY KEY,g geometry NOT NULL); +CREATE TABLE dbo.GeometryGuard(id int NOT NULL,g geometry NULL); +INSERT dbo.GeometryCases VALUES +(1,N'null and empty',NULL,geometry::STGeomFromText('GEOMETRYCOLLECTION EMPTY',0),0x), +(2,N'point SRID zero',geometry::Point(1,2,0),NULL,0x00FF), +(3,N'point ZM',geometry::STGeomFromText('POINT(1 2 3 4)',3826),geometry::Point(120,24,4326),0x010203), +(4,N'polygon',geometry::STGeomFromText('POLYGON((0 0,0 10,10 10,10 0,0 0))',4326),geometry::STGeomFromText('POINT EMPTY',3826),NULL), +(5,N'collection',geometry::STGeomFromText('GEOMETRYCOLLECTION(POINT(3 4),LINESTRING(0 0,2 2))',0),geometry::Point(5,6,0),0x7F), +(6,N'circularstring',geometry::STGeomFromText('CIRCULARSTRING(0 0,1 1,2 0)',0),NULL,0x), +(7,N'compoundcurve',geometry::STGeomFromText('COMPOUNDCURVE(CIRCULARSTRING(0 0,1 1,2 0),(2 0,3 0))',4326),geometry::STGeomFromText('CURVEPOLYGON(CIRCULARSTRING(0 0,1 1,2 0,1 -1,0 0))',0),0x42), +(8,N'invalid self intersection',geometry::STGeomFromText('POLYGON((0 0,2 2,0 2,2 0,0 0))',0),NULL,0xFF); diff --git a/tests/MssqlExportFixtures/large.sql b/tests/MssqlExportFixtures/large.sql new file mode 100644 index 0000000..1bb6bfb --- /dev/null +++ b/tests/MssqlExportFixtures/large.sql @@ -0,0 +1,14 @@ +-- Caller substitutes only an integer @Rows: baseline 1024; expanded 10240. +-- Execute in separate owned empty source DBs, or DROP only this owned table between runs. +-- 128 KiB per row; 128 MiB baseline / 1280 MiB expanded payload; hex >2560 MiB SQL. +SET NOCOUNT ON; +CREATE TABLE dbo.LargeBinary(id int NOT NULL PRIMARY KEY,payload varbinary(max) NOT NULL); +DECLARE @Rows int=1024; -- replace with 10240 for expanded run, record actual parameter +DECLARE @i int=1,@chunk varbinary(max)=CONVERT(varbinary(max),REPLICATE(CONVERT(varchar(max),'Z'),65536)); +WHILE @i<=@Rows BEGIN + INSERT dbo.LargeBinary VALUES (@i,0x); + UPDATE dbo.LargeBinary SET payload.WRITE(@chunk,NULL,0) WHERE id=@i; + UPDATE dbo.LargeBinary SET payload.WRITE(@chunk,NULL,0) WHERE id=@i; + SET @i+=1; +END; +SELECT COUNT_BIG(*) rows,SUM(CONVERT(bigint,DATALENGTH(payload))) source_payload_bytes FROM dbo.LargeBinary; diff --git a/tests/MssqlExportFixtures/unsupported.sql b/tests/MssqlExportFixtures/unsupported.sql new file mode 100644 index 0000000..c499132 --- /dev/null +++ b/tests/MssqlExportFixtures/unsupported.sql @@ -0,0 +1,19 @@ +-- Separate unsupported selection; never mix into the positive Core selection. +CREATE TABLE dbo.RejectSpatial(g geometry,h geography); +CREATE TABLE dbo.RejectVariant(v sql_variant); +CREATE TYPE dbo.RejectAlias FROM int NULL; +GO +CREATE TABLE dbo.RejectAliasTable(v dbo.RejectAlias); +GO +CREATE XML SCHEMA COLLECTION dbo.RejectXsd AS N''; +GO +CREATE TABLE dbo.RejectTypedXml(x xml(dbo.RejectXsd)); +GO +CREATE VIEW dbo.RejectEncrypted WITH ENCRYPTION AS SELECT 1 AS v; +GO +CREATE SEQUENCE dbo.RejectSequence AS int START WITH 1; +CREATE TABLE dbo.RejectSequenceDependency(v int DEFAULT NEXT VALUE FOR dbo.RejectSequence); +-- Separate fresh fixture: SNAPSHOT OFF is a DB setting, not a table in Core. +-- Separate minimally privileged reader: DENY VIEW DEFINITION must yield preflight failure. +-- Temporal, special indexes, CLR/FILESTREAM/encryption/version compatibility are pending +-- capability-specific phases; do not enable server features to fabricate these here. diff --git a/tests/Run-MssqlExportIntegration.ps1 b/tests/Run-MssqlExportIntegration.ps1 new file mode 100644 index 0000000..e6ddc55 --- /dev/null +++ b/tests/Run-MssqlExportIntegration.ps1 @@ -0,0 +1,94 @@ +param( + [switch]$Run, + [string]$Server, + [ValidateSet('Core','Large','Concurrency','ClientProbe','FinalFix','Geometry','Ssh')][string]$Profile = 'Core', + [string]$OutputDirectory, + [ValidateSet('Classic','Go')][string]$Client = 'Classic', + [string]$SqlcmdPath, + [string]$RestoreArtifact +) +$ErrorActionPreference = 'Stop' +if (!$Run) { + Write-Host 'Opt-in isolated MSSQL acceptance. Supply -Run -Server "(localdb)\mySQLPunkExportTest_20261012" -Profile Core|Large|Concurrency|Geometry -OutputDirectory .' + Write-Host 'Uses Integrated Security; creates only token-owned GUID source/target databases; requires an already running authorized fixture and sqlcmd. Never reads saved connections or installs/starts a server.' + return +} +if ($Profile -eq 'Ssh') { throw 'PENDING: an explicitly authorized SSH fixture and host-key policy are required. No connection attempted.' } +if ($Server -cne '(localdb)\mySQLPunkExportTest_20261012') { throw 'This acceptance runner is restricted to the explicitly authorized dedicated LocalDB instance.' } +$fixture = Get-Content -LiteralPath "$PSScriptRoot/../.superpowers/sdd/implementation-plan/fixture-environment.json" -Raw | ConvertFrom-Json +if (!$fixture.CreatedByTask -or $fixture.Instance -cne 'mySQLPunkExportTest_20261012') { throw 'Fixture authorization manifest mismatch.' } +$root = [IO.Path]::GetFullPath($fixture.FixtureRoot).TrimEnd('\') + '\' +if (!$root.StartsWith('C:\', [StringComparison]::OrdinalIgnoreCase)) { throw 'Bulk fixtures must remain on C:.' } +if (!$OutputDirectory) { throw 'Supply an explicit new/empty OutputDirectory.' } +$output = [IO.Path]::GetFullPath($OutputDirectory) +if (!$output.StartsWith($root, [StringComparison]::OrdinalIgnoreCase)) { throw 'OutputDirectory must be a descendant of the authorized FixtureRoot.' } +if ((Test-Path -LiteralPath $output) -and @(Get-ChildItem -LiteralPath $output -Force).Count) { throw 'OutputDirectory must be new or empty.' } +$ancestor = $output +while ($ancestor) { + if ((Test-Path -LiteralPath $ancestor) -and ((Get-Item -LiteralPath $ancestor).Attributes -band [IO.FileAttributes]::ReparsePoint)) { throw 'Reparse-point fixture paths are not accepted.' } + $ancestor = Split-Path -Parent $ancestor +} +$state = & sqllocaldb info 'mySQLPunkExportTest_20261012' +if ($LASTEXITCODE -ne 0 -or !($state -match 'LOCALDB#')) { throw 'Authorized LocalDB must already be running; runner does not start it.' } +$sqlcmd = if ($SqlcmdPath) { (Get-Item -LiteralPath $SqlcmdPath -ErrorAction Stop).FullName } else { (Get-Command sqlcmd -CommandType Application -ErrorAction Stop).Source } +if ($Client -eq 'Go' -and !$SqlcmdPath) { throw 'Go requires an explicit verified official SqlcmdPath; no automatic download.' } +if ($RestoreArtifact) { + if ($Profile -ne 'Large') { throw 'RestoreArtifact is limited to the Large factor 10 profile.' } + $RestoreArtifact = [IO.Path]::GetFullPath($RestoreArtifact) + if (!$RestoreArtifact.StartsWith($root,[StringComparison]::OrdinalIgnoreCase)) { throw 'RestoreArtifact must belong to the authorized fixture root.' } + $artifactAncestor = $RestoreArtifact + while ($artifactAncestor) { + if ((Test-Path -LiteralPath $artifactAncestor) -and ((Get-Item -LiteralPath $artifactAncestor).Attributes -band [IO.FileAttributes]::ReparsePoint)) { throw 'Reparse-point restore artifacts are not accepted.' } + $artifactAncestor = Split-Path -Parent $artifactAncestor + } + $artifactDir = Split-Path -Parent $RestoreArtifact + $oldEvidence = Get-Content -LiteralPath (Join-Path $artifactDir 'evidence.json') -Raw | ConvertFrom-Json + if ((Get-Content -LiteralPath (Join-Path $artifactDir 'owner.token') -Raw) -cne $oldEvidence.owner) { throw 'Original artifact ownership evidence mismatch.' } + $oldArtifact = $oldEvidence.('artifact-' + [IO.Path]::GetFileName($RestoreArtifact)) + if (!$oldArtifact -or $oldArtifact.bytes -le 2147483648 -or (Get-FileHash -LiteralPath $RestoreArtifact -Algorithm SHA256).Hash -ine $oldArtifact.sha256) { throw 'Original large service artifact hash/length evidence mismatch.' } +} +if ((Get-PSDrive C).Free -lt 20GB) { throw 'At least 20 GiB free C: space is required.' } +[IO.Directory]::CreateDirectory($output) | Out-Null +& $sqlcmd -? | Out-File (Join-Path $output 'sqlcmd-version.txt') -Encoding utf8 +$peReader = [IO.BinaryReader]::new([IO.File]::OpenRead($sqlcmd)) +try { + $peReader.BaseStream.Position = 60 + $peOffset = $peReader.ReadInt32() + $peReader.BaseStream.Position = $peOffset + 4 + $machine = $peReader.ReadUInt16() + $architecture = switch ($machine) { 0x8664 { 'x64' } 0x014c { 'x86' } 0xAA64 { 'ARM64' } default { "PE-machine-$machine" } } + @{ path=$sqlcmd; architecture=$architecture; fileVersion=(Get-Item -LiteralPath $sqlcmd).VersionInfo.FileVersion } | ConvertTo-Json | Out-File (Join-Path $output 'sqlcmd-identity.json') -Encoding utf8 +} finally { $peReader.Dispose() } +$envNames = @('MYSQLPUNK_SMOKE_FILTER','MYSQLPUNK_MSSQL_INTEGRATION','MYSQLPUNK_MSSQL_INTEGRATION_SERVER','MYSQLPUNK_MSSQL_INTEGRATION_OUTPUT','MYSQLPUNK_MSSQL_INTEGRATION_FIXTURES','MYSQLPUNK_MSSQL_INTEGRATION_FACTOR','MYSQLPUNK_MSSQL_INTEGRATION_SQLCMD','MYSQLPUNK_MSSQL_INTEGRATION_CLIENT','MYSQLPUNK_MSSQL_INTEGRATION_RESTORE','MYSQLPUNK_MSSQL_INTEGRATION_PIPE') +$saved = @{} +foreach ($name in $envNames) { $saved[$name] = [Environment]::GetEnvironmentVariable($name) } +try { + $env:MYSQLPUNK_SMOKE_FILTER = 'MSSQL export integration opt-in' + $env:MYSQLPUNK_MSSQL_INTEGRATION = $Profile + $env:MYSQLPUNK_MSSQL_INTEGRATION_SERVER = $Server + $env:MYSQLPUNK_MSSQL_INTEGRATION_FIXTURES = Join-Path $PSScriptRoot 'MssqlExportFixtures' + $env:MYSQLPUNK_MSSQL_INTEGRATION_SQLCMD = $sqlcmd + $env:MYSQLPUNK_MSSQL_INTEGRATION_CLIENT = $Client + $pipe = [regex]::Match(($state -join ' '), '\\\\\.\\pipe\\LOCALDB#[A-Fa-f0-9]+\\tsql\\query').Value + if ($Client -eq 'Go' -and !$pipe) { throw 'Exact authorized instance pipe could not be resolved.' } + $env:MYSQLPUNK_MSSQL_INTEGRATION_PIPE = $pipe + $env:MYSQLPUNK_MSSQL_INTEGRATION_RESTORE = $RestoreArtifact + $factors = @(1) + if ($Profile -eq 'Large') { $factors = if ($RestoreArtifact) { @(10) } else { @(1,10) } } + foreach ($factor in $factors) { + $phaseDir = Join-Path $output ("{0}-{1}" -f $Profile,$factor) + [IO.Directory]::CreateDirectory($phaseDir) | Out-Null + $env:MYSQLPUNK_MSSQL_INTEGRATION_OUTPUT = $phaseDir + $env:MYSQLPUNK_MSSQL_INTEGRATION_FACTOR = "$factor" + # Independent process peaks; builds and executions are strictly sequential. + # PowerShell 5 surfaces child stderr as ErrorRecord; retain it in the log before checking native exit. + $ErrorActionPreference = 'Continue' + try { & powershell.exe -NoProfile -File "$PSScriptRoot/Run-SmokeTests.ps1" -Configuration Release *> (Join-Path $output "run-$factor.log") } + finally { $ErrorActionPreference = 'Stop' } + $exitCode = $LASTEXITCODE + Get-Content -LiteralPath (Join-Path $output "run-$factor.log") -Tail 12 + if ($exitCode -ne 0) { throw "Integration $Profile factor=$factor failed (exit $exitCode); inspect run-$factor.log. An AV block must be reported pending, never bypassed." } + } +} finally { + foreach ($name in $envNames) { [Environment]::SetEnvironmentVariable($name,$saved[$name]) } +} diff --git a/tests/Run-SmokeTests.ps1 b/tests/Run-SmokeTests.ps1 index c58deaa..7a4d4a3 100644 --- a/tests/Run-SmokeTests.ps1 +++ b/tests/Run-SmokeTests.ps1 @@ -143,7 +143,7 @@ if (!(Test-Path -LiteralPath $mysqlConnector)) { /r:System.Core.dll ` /r:System.IO.Compression.dll ` /r:System.IO.Compression.FileSystem.dll ` - $source $preferencesSource $schemaSyncSource $erModelSource + $source $preferencesSource $schemaSyncSource $erModelSource (Join-Path $PSScriptRoot "SmokeTests.MssqlExport.cs") (Join-Path $PSScriptRoot "SmokeTests.MssqlExport.Schema.cs") (Join-Path $PSScriptRoot "SmokeTests.MssqlExport.Artifact.cs") (Join-Path $PSScriptRoot "SmokeTests.MssqlExport.Values.cs") (Join-Path $PSScriptRoot "SmokeTests.MssqlExport.Lob.cs") (Join-Path $PSScriptRoot "SmokeTests.MssqlExport.Worker.cs") (Join-Path $PSScriptRoot "SmokeTests.MssqlExport.Wizard.cs") (Join-Path $PSScriptRoot "SmokeTests.MssqlExport.Routing.cs") (Join-Path $PSScriptRoot "SmokeTests.MssqlExport.Integration.cs") if ($LASTEXITCODE -ne 0) { exit $LASTEXITCODE } diff --git a/tests/SmokeTests.MssqlExport.Artifact.cs b/tests/SmokeTests.MssqlExport.Artifact.cs new file mode 100644 index 0000000..de8296d --- /dev/null +++ b/tests/SmokeTests.MssqlExport.Artifact.cs @@ -0,0 +1,234 @@ +using System; +using System.IO; +using System.Text; +using System.Threading; +using mySQLPunk.lib.MssqlExport; +public static partial class SmokeTests +{ + // Removing hash checks, commit ordering or ownership cleanup must break these cases. + private static void Artifact_Pipeline() + { + string root = Path.Combine(Path.GetTempPath(), "mssql-artifact-" + Guid.NewGuid().ToString("N")); + Directory.CreateDirectory(root); + try + { + string target = Path.Combine(root, "fixture_backup_current.sql"); + File.WriteAllText(target, "old fixture"); + string other = Path.Combine(root, ".mysqlpunk-export-other.partial"); + File.WriteAllText(other, "other"); + var stamp = MssqlExportArtifact.CaptureDestination(target, CancellationToken.None); + var request = ArtifactRequest(target, stamp); + using (var artifact = MssqlExportArtifact.Create(request)) + { + var receipt = FinishArtifact(artifact); + AssertThrows(() => artifact.Publish(stamp, CancellationToken.None), "Unverified output cannot publish"); + using (var truncate = new FileStream(artifact.TemporaryPath, FileMode.Open, FileAccess.Write)) truncate.SetLength(receipt.Bytes - 1); + AssertThrows(() => artifact.Verify(receipt, null, CancellationToken.None), "Truncated output fails verification"); + } + Assert(File.ReadAllText(target) == "old fixture" && File.Exists(other), "Failure preserves old and unrelated partial"); + using (var artifact = MssqlExportArtifact.Create(request)) + { + var receipt = FinishArtifact(artifact); + artifact.Verify(receipt, null, CancellationToken.None); + var time = File.GetLastWriteTimeUtc(target); + File.WriteAllText(target, "new fixture"); File.SetLastWriteTimeUtc(target, time); + AssertThrows(() => artifact.Publish(stamp, CancellationToken.None), "Same size/time changes fail consent recheck"); + } + stamp = MssqlExportArtifact.CaptureDestination(target, CancellationToken.None); + request = ArtifactRequest(target, stamp); + using (var artifact = MssqlExportArtifact.Create(request)) + { + var receipt = FinishArtifact(artifact); + long last = 0; + artifact.Verify(receipt, (done, total) => { Assert(done - last <= 81920, "Verification bounded read"); last = done; }, CancellationToken.None); + Assert(last == receipt.Bytes, "Entire final marker is verified"); + var canceled = new CancellationTokenSource(); canceled.Cancel(); + AssertThrows(() => artifact.Publish(stamp, canceled.Token), "Cancel before commit"); + Assert(artifact.Publish(stamp, CancellationToken.None) == target, "Published path retained"); + canceled.Cancel(); Assert(artifact.PublishedPath == target, "Late cancel does not erase publication"); + var mirror = new MssqlExportMirror(); + Assert(mirror.CopyVerified(target, receipt, new MssqlMirrorOptions(root, 1), null, CancellationToken.None) == string.Empty, "Same directory is noop"); + string remote = Path.Combine(root, "remote"); Directory.CreateDirectory(remote); + string old = Path.Combine(remote, "old_backup_fixture.sql"); File.WriteAllText(old, "old fixture"); File.SetLastWriteTimeUtc(old, DateTime.UtcNow.AddDays(-1)); + string mirrorPath = mirror.CopyVerified(target, receipt, new MssqlMirrorOptions(remote, 1), null, CancellationToken.None); + Assert(File.Exists(mirrorPath) && !File.Exists(old), "Verified mirror published before retention"); + var stampMirror = MssqlExportArtifact.CaptureDestination(mirrorPath, CancellationToken.None); + Assert(stampMirror.Sha256 == receipt.Sha256 && stampMirror.Length == receipt.Bytes, "Mirror matches final receipt"); + AssertThrows(() => new MssqlExportMirror().CopyVerified(target, receipt, new MssqlMirrorOptions(remote, 1), null, canceled.Token), "Canceled mirror must not prune"); + Assert(File.Exists(mirrorPath), "Canceled mirror preserves old backups"); + } + string consentPath = Path.Combine(root,"consent.sql"); + File.WriteAllText(consentPath,"approved fixture A"); + var approved = MssqlExportArtifact.CaptureDestination(consentPath,CancellationToken.None); + var approvedRequest = ArtifactRequest(consentPath,approved); + File.WriteAllText(consentPath,"changed fixture B"); + var refreshed = MssqlExportArtifact.CaptureDestination(consentPath,CancellationToken.None); + using(var artifact=MssqlExportArtifact.Create(approvedRequest)) + { + var receipt=FinishArtifact(artifact);artifact.Verify(receipt,null,CancellationToken.None); + AssertThrows(()=>artifact.Publish(refreshed,CancellationToken.None),"Recaptured stamp cannot refresh immutable request consent"); + Assert(File.ReadAllText(consentPath)=="changed fixture B"&&artifact.PublishedPath==null,"Rejected refreshed consent preserves newer destination"); + } + string corrupt = Path.Combine(root,"postverify.sql"); + var absent = MssqlExportArtifact.CaptureDestination(corrupt,CancellationToken.None); + using(var artifact=MssqlExportArtifact.Create(ArtifactRequest(corrupt,absent))) + { + var receipt=FinishArtifact(artifact);artifact.Verify(receipt,null,CancellationToken.None); + using(var tamper=new FileStream(artifact.TemporaryPath,FileMode.Open,FileAccess.Write)) tamper.WriteByte(0); + AssertThrows(()=>artifact.Publish(absent,CancellationToken.None),"Mutation after verification cannot publish corrupt output"); + Assert(!File.Exists(corrupt),"Invalid output leaves absent destination absent"); + } + ArtifactFailures(root); + } + finally { Directory.Delete(root, true); } + } + private static MssqlExportRequest ArtifactRequest(string path, ExportDestinationStamp stamp) + { + return new MssqlExportRequest(Guid.NewGuid(), "fixture", new[] {new MssqlObjectKey(1,"dbo","T")}, MssqlExportMode.SchemaAndData, false, path, stamp, null); + } + private static MssqlArtifactReceipt FinishArtifact(MssqlExportArtifact artifact) + { + byte[] bytes = Encoding.UTF8.GetBytes("-- fixture\r\n" + new string('x', 180000)); + artifact.Output.Write(bytes, 0, bytes.Length); + return artifact.Complete(new[] {new MssqlTableRowCount(new MssqlObjectKey(1,"dbo","T"), 3000000000L)}, CancellationToken.None); + } + private static void ArtifactFailures(string root) + { + for (int mode = 0; mode < 3; mode++) + { + string path = Path.Combine(root, "replace" + mode + ".sql"); File.WriteAllText(path, "old fixture"); + var stamp = MssqlExportArtifact.CaptureDestination(path, CancellationToken.None); + var ops = new ArtifactFaultFiles { ReplaceFailure = mode + 1 }; + var artifact = MssqlExportArtifact.Create(ArtifactRequest(path, stamp), ops); + var receipt = FinishArtifact(artifact); artifact.Verify(receipt, null, CancellationToken.None); + AssertThrows(() => artifact.Publish(stamp, CancellationToken.None), "Documented partial replace failure"); + artifact.Dispose(); + Assert(artifact.ResidualPaths.Count > 0, "Ambiguous candidates reported and preserved"); + string oldPath = mode == 2 ? artifact.RecoveryPath : path; + Assert(File.ReadAllText(oldPath) == "old fixture", "Old bytes recoverable after replacement failure"); + } + string cleanupPath = Path.Combine(root, "cleanup.sql"); + var cleanup = MssqlExportArtifact.Create(ArtifactRequest(cleanupPath, MssqlExportArtifact.CaptureDestination(cleanupPath, CancellationToken.None)), new ArtifactFaultFiles { FailDelete = true }); + cleanup.Dispose(); Assert(cleanup.ResidualPaths.Count == 1 && File.Exists(cleanup.TemporaryPath), "Failed cleanup reports owned residual"); + string local = Path.Combine(root,"mirror-source.sql"); + using(var artifact = MssqlExportArtifact.Create(ArtifactRequest(local, MssqlExportArtifact.CaptureDestination(local,CancellationToken.None)))) + { + var receipt=FinishArtifact(artifact); artifact.Verify(receipt,null,CancellationToken.None); artifact.Publish(MssqlExportArtifact.CaptureDestination(local,CancellationToken.None),CancellationToken.None); + var ops=new ArtifactFaultFiles { FailMove=true }; + var mirror=new MssqlExportMirror(ops); + AssertThrows(()=>mirror.CopyVerified(local,receipt,new MssqlMirrorOptions(Path.Combine(root,"failed-remote"),1),null,CancellationToken.None),"Move failure"); + Assert(ops.PruneCalls==0,"Failed mirror never prunes"); + var cancel=new CancellationTokenSource(); ops=new ArtifactFaultFiles { AfterMove=()=>cancel.Cancel() }; mirror=new MssqlExportMirror(ops); + string published=mirror.CopyVerified(local,receipt,new MssqlMirrorOptions(Path.Combine(root,"late-cancel"),1),null,cancel.Token); + Assert(File.Exists(published)&&mirror.MirrorPublishedPath==published&&ops.PruneCalls==0,"Late cancellation retains mirror and skips retention"); + ops=new ArtifactFaultFiles { FailPrune=true }; mirror=new MssqlExportMirror(ops); + published=mirror.CopyVerified(local,receipt,new MssqlMirrorOptions(Path.Combine(root,"prune-fail"),1),null,CancellationToken.None); + Assert(File.Exists(published)&&mirror.Warnings.Count>0,"Retention failure does not erase mirror publication"); + } + } + private static void Artifact_Adversarial() + { + string root=Path.Combine(Path.GetTempPath(),"mssql-fault-"+Guid.NewGuid().ToString("N")); Directory.CreateDirectory(root); + try + { + string path=Path.Combine(root,"source_backup_fixture.sql"); File.WriteAllText(path,"old fixture"); + var stamp=MssqlExportArtifact.CaptureDestination(path,CancellationToken.None); + foreach(bool flush in new[]{false,true}) + { + var ops=new ArtifactFaultFiles { FailWrite=!flush, FailFlush=flush }; + using(var artifact=MssqlExportArtifact.Create(ArtifactRequest(path,stamp),ops)) + { + AssertThrows(()=>FinishArtifact(artifact),"Disk write/flush failure cannot create completed receipt"); + AssertThrows(()=>artifact.Publish(stamp,CancellationToken.None),"Faulted artifact cannot publish"); + } + Assert(File.ReadAllText(path)=="old fixture","Disk failure leaves old version intact"); + } + using(var locked=new FileStream(path,FileMode.Open,FileAccess.ReadWrite,FileShare.None)) + AssertThrows(()=>MssqlExportArtifact.CaptureDestination(path,CancellationToken.None),"Locked target is not absence"); + using(var artifact=MssqlExportArtifact.Create(ArtifactRequest(path,stamp))) + { + var receipt=FinishArtifact(artifact); + using(var cancel=new CancellationTokenSource()) + AssertThrows(()=>artifact.Verify(receipt,(done,total)=>cancel.Cancel(),cancel.Token),"Cancel while hashing"); + AssertThrows(()=>artifact.Publish(stamp,CancellationToken.None),"Canceled verifier cannot authorize publish"); + artifact.Verify(receipt,null,CancellationToken.None); artifact.Publish(stamp,CancellationToken.None); + string remote=Path.Combine(root,"remote");Directory.CreateDirectory(remote); + string prior=Path.Combine(remote,"old_backup_fixture.sql");File.WriteAllText(prior,"old fixture"); + var ops=new ArtifactFaultFiles { CollideMove=true }; var mirror=new MssqlExportMirror(ops); + AssertThrows(()=>mirror.CopyVerified(path,receipt,new MssqlMirrorOptions(remote,1),null,CancellationToken.None),"Mirror collision never overwrites"); + Assert(File.ReadAllText(ops.CollisionPath)=="collision fixture"&&File.Exists(prior)&&ops.PruneCalls==0,"Collision preserves other output and skips prune"); + ops=new ArtifactFaultFiles { FailWrite=true };mirror=new MssqlExportMirror(ops); + AssertThrows(()=>mirror.CopyVerified(path,receipt,new MssqlMirrorOptions(remote,1),null,CancellationToken.None),"Mirror disk full"); + Assert(ops.PruneCalls==0&&File.Exists(prior),"Copy failure never prunes"); + string active=Path.Combine(remote,".mysqlpunk-export-active.partial");File.WriteAllText(active,"other"); + string future=Path.Combine(remote,"future_backup_fixture.sql");File.WriteAllText(future,"old fixture");File.SetLastWriteTimeUtc(future,DateTime.UtcNow.AddDays(2)); + mirror=new MssqlExportMirror();string published=mirror.CopyVerified(path,receipt,new MssqlMirrorOptions(remote,1),null,CancellationToken.None); + Assert(File.Exists(published)&&File.Exists(active)&&mirror.Warnings.Count>0,"Future timestamps cannot prune newly published mirror or active partial"); + File.WriteAllText(path,"changed fixture");ops=new ArtifactFaultFiles();mirror=new MssqlExportMirror(ops); + AssertThrows(()=>mirror.CopyVerified(path,receipt,new MssqlMirrorOptions(remote,1),null,CancellationToken.None),"Changed published source cannot be mirrored as old receipt"); + Assert(ops.PruneCalls==0&&File.Exists(published),"Invalid mirror never prunes"); + } + } + finally { Directory.Delete(root,true); } + } + private static void Artifact_CooperativeGates() + { + string root=Path.Combine(Path.GetTempPath(),"mssql-gate-"+Guid.NewGuid().ToString("N"));Directory.CreateDirectory(root); + try + { + string path=Path.Combine(root,"target.sql");File.WriteAllText(path,"old fixture"); + var stamp=MssqlExportArtifact.CaptureDestination(path,CancellationToken.None); + using(var entered=new ManualResetEventSlim()) + using(var release=new ManualResetEventSlim()) + using(var captureStarted=new ManualResetEventSlim()) + using(var cancel=new CancellationTokenSource()) + { + var ops=new ArtifactFaultFiles { BeforeReplace=()=> { entered.Set(); if(!release.Wait(10000)) throw new TimeoutException(); } }; + using(var first=MssqlExportArtifact.Create(ArtifactRequest(path,stamp),ops)) + using(var second=MssqlExportArtifact.Create(ArtifactRequest(path,stamp))) + { + var receipt=FinishArtifact(first);first.Verify(receipt,null,CancellationToken.None); + receipt=FinishArtifact(second);second.Verify(receipt,null,CancellationToken.None); + var publish=System.Threading.Tasks.Task.Run(()=>first.Publish(stamp,CancellationToken.None)); + Assert(entered.Wait(10000),"Publisher enters replacement"); + var capture=System.Threading.Tasks.Task.Run(()=> {captureStarted.Set();return MssqlExportArtifact.CaptureDestination(path,cancel.Token);}); + try + { + Assert(captureStarted.Wait(10000),"Capture started");cancel.Cancel(); + AssertThrows(()=>capture.GetAwaiter().GetResult(),"Consent capture shares cancellable publication gate"); + } + finally { release.Set(); } + Assert(publish.GetAwaiter().GetResult()==path,"First publisher completes"); + AssertThrows(()=>second.Publish(stamp,CancellationToken.None),"Second cooperating job rejects stale consent"); + } + } + } + finally { Directory.Delete(root,true); } + } + private sealed class ArtifactFailureStream : Stream + { + private readonly Stream inner; private readonly bool write,flush; + internal ArtifactFailureStream(Stream inner,bool write,bool flush){this.inner=inner;this.write=write;this.flush=flush;} + public override void Write(byte[] b,int o,int c){if(write)throw new IOException("Injected disk full");inner.Write(b,o,c);} + public override void Flush(){if(flush)throw new IOException("Injected flush full");inner.Flush();} + protected override void Dispose(bool disposing){if(disposing)inner.Dispose();base.Dispose(disposing);} + public override bool CanRead=>false;public override bool CanSeek=>false;public override bool CanWrite=>true; + public override long Length=>inner.Length;public override long Position{get=>inner.Position;set=>throw new NotSupportedException();} + public override int Read(byte[] b,int o,int c)=>throw new NotSupportedException();public override long Seek(long o,SeekOrigin k)=>throw new NotSupportedException();public override void SetLength(long l)=>throw new NotSupportedException(); + } + private sealed class ArtifactFaultFiles : MssqlArtifactFiles + { + internal int ReplaceFailure; internal bool FailMove, FailDelete, FailPrune; internal int PruneCalls; internal Action AfterMove, BeforeReplace; internal bool FailWrite, FailFlush, CollideMove; internal string CollisionPath; + internal override Stream Create(string path) { var stream=base.Create(path); return FailWrite||FailFlush ? new ArtifactFailureStream(stream,FailWrite,FailFlush) : stream; } + internal override void Replace(string source,string target,string recovery) + { + BeforeReplace?.Invoke(); + if(ReplaceFailure==3) File.Move(target,recovery); + if(ReplaceFailure>0) throw new IOException("Injected documented replace failure " + ReplaceFailure); + base.Replace(source,target,recovery); + } + internal override void Move(string source,string target) { if(FailMove) throw new IOException("Injected rename failure"); if(CollideMove) {CollisionPath=target;File.WriteAllText(target,"collision fixture");} base.Move(source,target); AfterMove?.Invoke(); } + internal override void Delete(string path) { if(FailDelete) throw new IOException("Injected cleanup failure"); base.Delete(path); } + internal override void Prune(string directory,int count) { PruneCalls++; if(FailPrune) throw new IOException("Injected retention failure"); base.Prune(directory,count); } + } +} diff --git a/tests/SmokeTests.MssqlExport.Integration.cs b/tests/SmokeTests.MssqlExport.Integration.cs new file mode 100644 index 0000000..ce8d1e5 --- /dev/null +++ b/tests/SmokeTests.MssqlExport.Integration.cs @@ -0,0 +1,598 @@ +using System; +using System.Collections.Generic; +using System.Data; +using System.Data.SqlClient; +using System.Diagnostics; +using System.Globalization; +using System.IO; +using System.Linq; +using System.Security.Cryptography; +using System.Text; +using System.Text.RegularExpressions; +using System.Threading; +using mySQLPunk.lib; +using mySQLPunk.lib.MssqlExport; +using Newtonsoft.Json; + +public static partial class SmokeTests +{ + // This entry is inert in ordinary smoke runs. No saved connection configuration is read. + private static void Integration_Run() + { + string profile = Environment.GetEnvironmentVariable("MYSQLPUNK_MSSQL_INTEGRATION"); + if (string.IsNullOrEmpty(profile)) { Console.WriteLine("SKIP live integration (explicit runner -Run required)"); return; } + Assert(new[] { "Core", "Large", "Concurrency", "ClientProbe", "FinalFix", "Geometry" }.Contains(profile), "Known integration profile"); + using (var fixture = new IntegrationFixture(profile)) + { + if (profile == "ClientProbe") IntegrationClientProbe(fixture); + else if (profile == "FinalFix") IntegrationFinalFix(fixture); + else if (profile == "Geometry") IntegrationGeometry(fixture); + else if (profile == "Core") IntegrationCore(fixture); + else if (profile == "Large") IntegrationLarge(fixture); + else IntegrationConcurrency(fixture); + } + } + + private sealed class IntegrationFixture : IDisposable + { + internal readonly string Server, Root, Token, Source, Target, Profile; + internal readonly List Created = new List(); + internal readonly Dictionary Evidence = new Dictionary(); + internal readonly List WorkerConnections = new List(); + internal Action AfterBegin; + internal Action BeforeRows; + internal bool RestrictedReader; + private int restoreSequence; + internal IntegrationFixture(string profile) + { + Profile = profile; + Server = Environment.GetEnvironmentVariable("MYSQLPUNK_MSSQL_INTEGRATION_SERVER"); + Assert(Server == @"(localdb)\mySQLPunkExportTest_20261012", "Explicit authorized fixture server only"); + Root = Path.GetFullPath(Environment.GetEnvironmentVariable("MYSQLPUNK_MSSQL_INTEGRATION_OUTPUT")); + Assert(Root.StartsWith(@"C:\", StringComparison.OrdinalIgnoreCase) && Directory.Exists(Root) && !Directory.EnumerateFileSystemEntries(Root).Any(), "New empty C: fixture directory"); + Token = Guid.NewGuid().ToString("N"); Source = "mysqlpunk_export_" + Token + "_src"; Target = "mysqlpunk_export_" + Token + "_dst"; + File.WriteAllText(Path.Combine(Root,"owner.token"),Token); + Evidence["server"] = Server; Evidence["profile"] = profile; Evidence["source"] = Source; Evidence["target"] = Target; Evidence["owner"] = Token; + Evidence["processBits"] = IntPtr.Size * 8; Evidence["provider"] = typeof(SqlConnection).Assembly.FullName; + Save(); + try + { + using (var c = Open("master")) + { + Evidence["engine"] = Scalar(c,"SELECT CONVERT(nvarchar(128),SERVERPROPERTY('ProductVersion'))"); + Evidence["tempdbCollation"] = Scalar(c,"SELECT CONVERT(nvarchar(128),DATABASEPROPERTYEX('tempdb','Collation'))"); + foreach (var db in new[] { Source, Target }) + { + string data = Path.Combine(Root, db + ".mdf"), log = Path.Combine(Root, db + ".ldf"); + Exec(c,"CREATE DATABASE " + Q(db) + " ON PRIMARY (NAME=N'data',FILENAME=" + Lit(data) + ",SIZE=16MB,MAXSIZE=3072MB,FILEGROWTH=64MB) LOG ON(NAME=N'log',FILENAME=" + Lit(log) + ",SIZE=16MB,MAXSIZE=512MB,FILEGROWTH=64MB) COLLATE Latin1_General_100_BIN2"); + Created.Add(db); + Exec(c,"USE " + Q(db) + "; EXEC sys.sp_addextendedproperty @name=N'mysqlpunk_export_owner',@value=" + Lit(Token) + ";"); + Exec(c,"USE master; ALTER DATABASE " + Q(db) + " SET RECOVERY SIMPLE;"); + } + Exec(c,"ALTER DATABASE " + Q(Source) + " SET ALLOW_SNAPSHOT_ISOLATION ON;"); + } + Evidence["sourceCollation"] = "Latin1_General_100_BIN2"; + Save(); + } + catch { Dispose(); throw; } + } + internal static string Q(string s) { return "[" + s.Replace("]", "]]") + "]"; } + internal static string Lit(string s) { return "N'" + s.Replace("'", "''") + "'"; } + internal SqlConnection Open(string db) + { + Assert(db == "master" || db == Source || db == Target, "Only own databases"); + var c = new SqlConnection(new SqlConnectionStringBuilder { DataSource = Server, InitialCatalog = db, IntegratedSecurity = true, Pooling = false, TrustServerCertificate = true, ApplicationName = "mySQLPunk-owned-integration" }.ConnectionString); + c.Open(); return c; + } + internal static void Exec(SqlConnection c,string sql) + { using (var cmd = new SqlCommand(sql,c) { CommandTimeout = 0 }) cmd.ExecuteNonQuery(); } + internal static object Scalar(SqlConnection c,string sql) + { using (var cmd = new SqlCommand(sql,c) { CommandTimeout = 0 }) return cmd.ExecuteScalar(); } + internal void Setup(string name, int factor = 1) + { + string sql = File.ReadAllText(Path.Combine(Environment.GetEnvironmentVariable("MYSQLPUNK_MSSQL_INTEGRATION_FIXTURES"),name)); + if (name == "large.sql") sql = sql.Replace("@Rows int=1024", "@Rows int=" + (1024 * factor).ToString(CultureInfo.InvariantCulture)); + using (var c = Open(Source)) + foreach (var batch in Regex.Split(sql,@"(?im)^GO\s*\r?$")) if (!string.IsNullOrWhiteSpace(batch)) Exec(c,batch); + } + internal IMssqlExportSource NewSource() + { + var provider = new my_mssql(); + provider.SetConn(new SqlConnectionStringBuilder { DataSource = Server, InitialCatalog = Source, IntegratedSecurity = true, Pooling = false, TrustServerCertificate = true }.ConnectionString); + provider.Open(); WorkerConnections.Add(provider.MCT); + if (RestrictedReader) Exec(provider.MCT,"EXECUTE AS USER = N'fixture_reader';"); + return new IntegrationSource(new MssqlExportSource(new MssqlOwnedSession(provider,provider.MCT)),this); + } + internal MssqlSqlExportService Service(MssqlArtifactFiles files = null, Action phase = null) + { + return new MssqlSqlExportService(NewSource,files ?? new MssqlArtifactFiles(),stage => { + Assert(WorkerConnections.All(c => c.State == ConnectionState.Closed), "Real source connections released before file " + stage); + Evidence["sourceReleasedBeforeLocalPhases"] = true; + if (phase != null) phase(stage); + }); + } + internal MssqlExportRequest Request(string filename = "export.sql", bool views = true, MssqlExportMode mode = MssqlExportMode.SchemaAndData, Func select = null, bool mirror = false) + { + var tables = Service().LoadCatalog(Source,CancellationToken.None).Tables.Where(t => select == null || select(t)).Select(t => t.Key).ToArray(); + string path = Path.Combine(Root,filename); + return new MssqlExportRequest(Guid.NewGuid(),Source,tables,mode,views,path,MssqlExportArtifact.CaptureDestination(path,CancellationToken.None),mirror ? new MssqlMirrorOptions(Path.Combine(Root,"mirror"),2) : null); + } + internal MssqlExportResult Export(MssqlExportRequest request, CancellationToken cancellation = default(CancellationToken), MssqlArtifactFiles files = null, Action phase = null, Action progress = null) + { + using (var memory = new IntegrationMemory()) + { + var result = Service(files,phase).Export(request,new WorkerProgress(p => { memory.Stage = p.Stage.ToString(); if (progress != null) progress(p); }),cancellation); + Evidence["export-" + Path.GetFileName(request.TargetPath)] = new { stage=result.Stage.ToString(), result.Elapsed, result.ErrorMessage, result.PublishedPath, result.MirrorPath, rows = result.RowCounts.Sum(x => x.Rows), memory = memory.Snapshot() }; + Save(); return result; + } + } + internal void Restore(string file, bool expectSuccess = true) + { + Assert(File.Exists(file),"Restore artifact exists"); + bool go = Environment.GetEnvironmentVariable("MYSQLPUNK_MSSQL_INTEGRATION_CLIENT") == "Go"; + string endpoint = go ? "np:"+Environment.GetEnvironmentVariable("MYSQLPUNK_MSSQL_INTEGRATION_PIPE") : Server; + string args = "-S \"" + endpoint + "\" -d " + Target + " -E -b -x " + (go ? "" : "-f 65001 ") + "-i \"" + file + "\""; + var start = new ProcessStartInfo(Environment.GetEnvironmentVariable("MYSQLPUNK_MSSQL_INTEGRATION_SQLCMD"),args) { UseShellExecute=false,CreateNoWindow=true,RedirectStandardOutput=true,RedirectStandardError=true }; + var elapsed = Stopwatch.StartNew(); + using (var p = Process.Start(start)) + { + var stdout = p.StandardOutput.ReadToEndAsync(); var stderr = p.StandardError.ReadToEndAsync(); p.WaitForExit(); + int attempt = ++restoreSequence; + File.WriteAllText(file + ".restore-" + attempt + ".log",stdout.Result + stderr.Result); + elapsed.Stop(); + Evidence["restore-" + Path.GetFileName(file) + "-" + attempt] = new { command = "sqlcmd " + args, exitCode = p.ExitCode, elapsed=elapsed.Elapsed }; + Save(); Assert(expectSuccess ? p.ExitCode == 0 : p.ExitCode != 0,"sqlcmd single-session expected exit for " + file + ": " + p.ExitCode); + } + } + internal void RecordArtifact(string file) + { + using (var sha = SHA256.Create()) using (var stream = File.OpenRead(file)) + Evidence["artifact-" + Path.GetFileName(file)] = new { bytes=stream.Length, sha256=BitConverter.ToString(sha.ComputeHash(stream)).Replace("-","").ToLowerInvariant() }; + Save(); + } + internal void Pass(string message) { Console.WriteLine("PASS " + message); Evidence[message] = "PASS"; Save(); } + internal void Save() { File.WriteAllText(Path.Combine(Root,"evidence.json"),JsonConvert.SerializeObject(Evidence,Formatting.Indented)); } + public void Dispose() + { + Assert(File.ReadAllText(Path.Combine(Root,"owner.token")) == Token,"File ownership matches before cleanup"); + foreach (string db in Created.ToArray()) + { + Assert(db == "mysqlpunk_export_" + Token + "_src" || db == "mysqlpunk_export_" + Token + "_dst","Exact generated DB before DROP"); + using (var c = Open(db)) + { + Assert(Convert.ToString(Scalar(c,"SELECT value FROM sys.extended_properties WHERE class=0 AND name=N'mysqlpunk_export_owner'")) == Token,"DB token before DROP"); + using (var cmd = new SqlCommand("SELECT physical_name FROM sys.database_files",c)) using (var r = cmd.ExecuteReader()) + while (r.Read()) Assert(Path.GetDirectoryName(Path.GetFullPath(r.GetString(0))) == Root,"Owned database file path before DROP"); + } + using (var c = Open("master")) Exec(c,"ALTER DATABASE " + Q(db) + " SET SINGLE_USER WITH ROLLBACK IMMEDIATE; DROP DATABASE " + Q(db)); + Created.Remove(db); Console.WriteLine("CLEANED " + db); + } + Evidence["cleanup"] = "All exact token-owned databases dropped; artifacts retained"; Save(); + } + } + + private sealed class IntegrationSource : IMssqlExportSource + { + private readonly IMssqlExportSource inner; private readonly IntegrationFixture fixture; + internal IntegrationSource(IMssqlExportSource inner, IntegrationFixture fixture) { this.inner=inner; this.fixture=fixture; } + public MssqlExportCatalog ReadCatalog(string d,CancellationToken c) { return inner.ReadCatalog(d,c); } + public MssqlExportPreflight Inspect(MssqlExportRequest r,CancellationToken c) { return inner.Inspect(r,c); } + public MssqlExportSchema BeginRead(MssqlExportRequest r,CancellationToken c) { var schema=inner.BeginRead(r,c); if(fixture.AfterBegin!=null)fixture.AfterBegin(); return schema; } + public MssqlRowReader OpenRows(MssqlTableDefinition t,CancellationToken c) { if(fixture.BeforeRows!=null)fixture.BeforeRows(t); return inner.OpenRows(t,c); } + public void CompleteRead(MssqlExportSchema s,CancellationToken c) { inner.CompleteRead(s,c); } + public void Dispose() { inner.Dispose(); } + } + + private sealed class IntegrationMemory : IDisposable + { + internal volatile string Stage="Preflight"; + private readonly Dictionary peaks = new Dictionary(); + private readonly Timer timer; + internal IntegrationMemory() { timer=new Timer(_=>Sample(),null,0,25); } + private void Sample() { using(var p=Process.GetCurrentProcess()) { p.Refresh(); lock(peaks) { long[] pair; if(!peaks.TryGetValue(Stage,out pair))peaks[Stage]=pair=new long[3]; pair[0]=Math.Max(pair[0],p.WorkingSet64);pair[1]=Math.Max(pair[1],p.PrivateMemorySize64);pair[2]++; } } } + internal object Snapshot() { Sample(); lock(peaks)return peaks.ToDictionary(x=>x.Key,x=>new {workingSet=x.Value[0],privateBytes=x.Value[1],samples=x.Value[2]}); } + public void Dispose() { using(var done=new ManualResetEvent(false)){timer.Dispose(done);done.WaitOne();} } + } + + private static void IntegrationFinalFix(IntegrationFixture f) + { + const string table = "dbo.IndexedComputed"; + const string ddl = "CREATE TABLE dbo.IndexedComputed(id int NOT NULL, value int NOT NULL, doubled AS (value * 2) PERSISTED); CREATE UNIQUE INDEX IX_IndexedComputed_doubled ON dbo.IndexedComputed(doubled);"; + using (var c = f.Open(f.Source)) + IntegrationFixture.Exec(c, ddl + " INSERT dbo.IndexedComputed(id,value) VALUES(1,7),(2,19);"); + using (var c = f.Open(f.Source)) + IntegrationFixture.Exec(c, "CREATE TABLE dbo.WideEstimate(id int NOT NULL, pad char(4000) NOT NULL); INSERT dbo.WideEstimate SELECT TOP (40) ROW_NUMBER() OVER (ORDER BY object_id), REPLICATE('X',4000) FROM sys.all_objects;"); + using (var c = f.Open(f.Target)) IntegrationFixture.Exec(c, ddl); + var catalog = f.Service().LoadCatalog(f.Source, CancellationToken.None); + var selected = catalog.Tables.Single(t => t.Key.Schema == "dbo" && t.Key.Name == "IndexedComputed"); + var wide = catalog.Tables.Single(t => t.Key.Schema == "dbo" && t.Key.Name == "WideEstimate"); + Assert(selected.EstimatedRows.HasValue && selected.EstimatedBytes.HasValue && selected.EstimatedBytes.Value > 0 && wide.EstimatedBytes > selected.EstimatedBytes, "Real native catalog differentiates table storage"); + var estimateSchema = new MssqlExportSchema(new[] { selected, wide }, null, null, 1, null); + var progress = new MssqlExportProgressEstimator(Guid.NewGuid(), estimateSchema, false).Sample(MssqlExportStage.Data, selected.Key, 1, 100, 1000, TimeSpan.FromSeconds(3), true); + Assert(progress.EstimatedBytes == selected.EstimatedBytes + wide.EstimatedBytes && progress.Percent.HasValue, "Actual metadata flows into weighted estimator"); + f.Evidence["actualStorageEstimate"] = new { selected.EstimatedRows, selected.EstimatedBytes, wideBytes = wide.EstimatedBytes, progress.Percent }; + var request = f.Request("data-only-indexed-computed.sql", false, MssqlExportMode.DataOnly, t => t.Key.Name == "IndexedComputed"); + var result = f.Export(request); + Assert(result.Stage == MssqlExportStage.Completed, "DataOnly service artifact: " + result.ErrorMessage); + string script = File.ReadAllText(result.PublishedPath); + AssertContains(script, "SET QUOTED_IDENTIFIER ON;", "DataOnly artifact carries session preamble"); + AssertNotContains(script, "CREATE TABLE", "DataOnly artifact has no object creation"); + AssertNotContains(script, "ALTER TABLE", "DataOnly artifact has no object alteration"); + Assert(Environment.GetEnvironmentVariable("MYSQLPUNK_MSSQL_INTEGRATION_CLIENT") == "Classic", "Exact classic sqlcmd required"); + f.RecordArtifact(result.PublishedPath); + f.Restore(result.PublishedPath); + using (var c = f.Open(f.Target)) + { + Assert(Convert.ToInt32(IntegrationFixture.Scalar(c, "SELECT COUNT(*) FROM " + table + " WHERE doubled=value*2")) == 2, "Indexed computed target contains both restored rows"); + Assert(Convert.ToInt32(IntegrationFixture.Scalar(c, "SELECT COUNT(*) FROM sys.indexes WHERE object_id=OBJECT_ID(N'dbo.IndexedComputed') AND name=N'IX_IndexedComputed_doubled'")) == 1, "Compatible computed-column index remains"); + } + f.Pass("Real DataOnly service artifact restored through documented classic one-session sqlcmd -E -b -x -f 65001 -i"); + } + + private static void IntegrationGeometry(IntegrationFixture f) + { + Assert(Environment.GetEnvironmentVariable("MYSQLPUNK_MSSQL_INTEGRATION_CLIENT") == "Classic", "Geometry requires documented classic sqlcmd"); + f.Setup("geometry.sql"); + // Fixture creation is independent from exporter allocation; no large string/byte array. + using(var setupMemory=new IntegrationMemory()) { + setupMemory.Stage="FixtureGenerationAndUpload"; + IntegrationGeometryLine(f,"large-geometry.wkb","dbo.GeometryLarge",1100001); + IntegrationGeometryLine(f,"guard-geometry.wkb","dbo.GeometryGuard",20001); + f.Evidence["geometryFixtureMemorySeparateFromExporter"]=setupMemory.Snapshot();f.Save(); + } + using(var c=f.Open(f.Source)) { + Assert(Convert.ToInt32(IntegrationFixture.Scalar(c,"SELECT CASE WHEN geometry::Deserialize(NULL) IS NULL THEN 1 ELSE 0 END"))==1,"Real Deserialize(NULL) returns SQL NULL"); + long length=Convert.ToInt64(IntegrationFixture.Scalar(c,"SELECT DATALENGTH(g.Serialize()) FROM dbo.GeometryLarge")); + Assert(length>16L*1024*1024,"Actual serialized geometry exceeds 16MiB");f.Evidence["largeGeometrySerializedBytes"]=length; + Assert(Convert.ToInt32(IntegrationFixture.Scalar(c,"SELECT g.STIsValid() FROM dbo.GeometryCases WHERE id=8"))==0,"Constructible invalid polygon stays invalid"); + Assert(Convert.ToInt32(IntegrationFixture.Scalar(c,"SELECT COUNT(*) FROM dbo.GeometryCases WHERE id=3 AND g.STSrid=3826 AND g.Z=3 AND g.M=4"))==1,"Fixture has independent point SRID/Z/M"); + } + var req=f.Request("geometry.sql",false); + Assert(f.Service().Preflight(req,CancellationToken.None).CanStart,"Native geometry actual preflight"); + var result=f.Export(req);Assert(result.Stage==MssqlExportStage.Completed,"Geometry service artifact: "+result.ErrorMessage); + f.RecordArtifact(result.PublishedPath);f.Restore(result.PublishedPath);IntegrationCompare(f,req); + using(var a=f.Open(f.Source))using(var b=f.Open(f.Target)) { + string observations="SELECT id,g.STSrid AS srid,g.STGeometryType() AS kind,g.STIsEmpty() AS empty,g.STIsValid() AS valid,g.STNumPoints() AS points,g.STPointN(1).Z AS z,g.STPointN(1).M AS m,h.STSrid AS secondSrid,h.STGeometryType() AS secondKind,h.STIsEmpty() AS secondEmpty FROM dbo.GeometryCases WHERE id<>8"; + string invalid="SELECT id,g.STSrid AS srid,g.STIsValid() AS valid FROM dbo.GeometryCases WHERE id=8"; + string left=IntegrationCatalog(a,observations)+"\n"+IntegrationCatalog(a,invalid),right=IntegrationCatalog(b,observations)+"\n"+IntegrationCatalog(b,invalid); + Assert(left==right,"Independent SRID/Z/M/type/empty/validity/point observations preserved"); + File.WriteAllText(Path.Combine(f.Root,"geometry-observations-source.txt"),left);File.WriteAllText(Path.Combine(f.Root,"geometry-observations-target.txt"),right); + Assert(Convert.ToInt32(IntegrationFixture.Scalar(b,"SELECT COUNT(*) FROM dbo.GeometryCases WHERE id=1 AND g IS NULL AND h.STIsEmpty()=1"))==1,"SQL NULL distinct from valid empty geometry"); + Assert(Convert.ToInt32(IntegrationFixture.Scalar(b,"SELECT g.STIsValid() FROM dbo.GeometryCases WHERE id=8"))==0,"Invalid geometry bytes never repaired"); + } + f.Pass("Geometry real service classic sqlcmd restore preserves Serialize hashes native catalog SRID Z M empty invalid and curves"); + // All supported modes use the same native type/projection contract. + var schemaReq=f.Request("geometry-schema-only.sql",false,MssqlExportMode.SchemaOnly,t=>t.Key.Name=="GeometryCases"); + result=f.Export(schemaReq);Assert(result.Stage==MssqlExportStage.Completed,"Geometry SchemaOnly service"); + using(var c=f.Open(f.Target))IntegrationFixture.Exec(c,"DROP TABLE dbo.GeometryCases;"); + f.RecordArtifact(result.PublishedPath);f.Restore(result.PublishedPath); + using(var c=f.Open(f.Target))Assert(Convert.ToInt32(IntegrationFixture.Scalar(c,"SELECT COUNT(*) FROM dbo.GeometryCases"))==0,"SchemaOnly creates empty native geometry table"); + var dataReq=f.Request("geometry-data-only.sql",false,MssqlExportMode.DataOnly,t=>t.Key.Name=="GeometryCases"); + result=f.Export(dataReq);Assert(result.Stage==MssqlExportStage.Completed,"Geometry DataOnly service");f.RecordArtifact(result.PublishedPath);f.Restore(result.PublishedPath); + IntegrationCompare(f,req); + IntegrationGeometryGuards(f); + using(var c=f.Open(f.Source))IntegrationFixture.Exec(c,"CREATE TABLE dbo.GeometryRejectedGeography(g geography); CREATE TABLE dbo.GeometryRejectedIndex(id int NOT NULL PRIMARY KEY,g geometry); CREATE SPATIAL INDEX IX_GeometryRejected ON dbo.GeometryRejectedIndex(g) USING GEOMETRY_GRID WITH (BOUNDING_BOX=(0,0,100,100));"); + foreach(var name in new[]{"GeometryRejectedGeography","GeometryRejectedIndex"}) + foreach(var mode in new[]{MssqlExportMode.SchemaAndData,MssqlExportMode.DataOnly,MssqlExportMode.SchemaOnly}) { + var rejected=f.Request("reject-"+name+"-"+mode+".sql",false,mode,t=>t.Key.Name==name); + var pre=f.Service().Preflight(rejected,CancellationToken.None);string code=name=="GeometryRejectedGeography"?"UnsupportedSpatial":"UnsupportedIndex"; + Assert(!pre.CanStart && pre.Issues.Any(x=>x.Code==code),"Actual unsupported preflight "+name+" "+mode); + var failed=f.Export(rejected);Assert(failed.Stage==MssqlExportStage.Failed && !File.Exists(rejected.TargetPath),"Actual unsupported geometry selection publishes nothing"); + } + f.Pass("Geometry all modes and retained geography spatial-index preflight rejection"); + } + private static void IntegrationGeometryLine(IntegrationFixture f,string filename,string table,int points) + { + string file=Path.Combine(f.Root,filename); + using(var stream=new FileStream(file,FileMode.CreateNew,FileAccess.Write,FileShare.None,65536))using(var writer=new BinaryWriter(stream)) { + writer.Write((byte)1);writer.Write((uint)2);writer.Write((uint)points); + for(int i=0;it.Key.Name=="GeometryGuard"); + var result=f.Export(req);Assert(result.Stage==MssqlExportStage.Completed,"Geometry guarded service artifact");f.RecordArtifact(req.TargetPath); + var batches=Regex.Split(File.ReadAllText(req.TargetPath),@"(?m)^GO\r?$").Where(s=>!string.IsNullOrWhiteSpace(s)).ToArray(); + var chunks=Enumerable.Range(0,batches.Length).Where(i=>batches[i].Contains(".WRITE(")).ToArray(); + int final=Array.FindLastIndex(batches,s=>s.Contains("INSERT INTO [dbo].[GeometryGuard]")); + Assert(chunks.Length>2 && final>=0,"Actual geometry uses chunked stage"); + foreach(var fault in new[]{"missing","duplicate","outoforder","malformed","replay"}) { + using(var c=f.Open(f.Target))IntegrationFixture.Exec(c,"TRUNCATE TABLE dbo.GeometryGuard;"); + var altered=new List(); + for(int i=0;i17 && Uri.IsHexDigit(sql[at]),"Serialized geometry header mutation located"); + sql=sql.Substring(0,at)+"FF"+sql.Substring(at+2); + } + altered.Add(sql);if(fault=="duplicate" && i==chunks[0])altered.Add(sql); + } + altered.Add(batches[final]); + // A second attempted row cannot revive Failed; replay cannot insert twice either. + altered.AddRange(batches.Where(s=>!s.Contains("CREATE TABLE #mp_") || !s.Contains("LOB job already exists"))); + string copy=Path.Combine(f.Root,"geometry-guard-"+fault+".sql");File.WriteAllText(copy,string.Join("\r\nGO\r\n",altered)+"\r\nGO\r\n",new UTF8Encoding(false)); + f.Restore(copy,false); + using(var c=f.Open(f.Target)) { + Assert(Convert.ToInt32(IntegrationFixture.Scalar(c,"SELECT COUNT(*) FROM dbo.GeometryGuard"))==(fault=="replay"?1:0),"Fail-fast geometry failed row absent "+fault); + IntegrationFixture.Exec(c,"TRUNCATE TABLE dbo.GeometryGuard;");int errors=0; + foreach(string batch in altered)try{IntegrationFixture.Exec(c,batch);}catch(SqlException){errors++;} + Assert(errors>0 && Convert.ToInt32(IntegrationFixture.Scalar(c,"SELECT COUNT(*) FROM dbo.GeometryGuard"))==(fault=="replay"?1:0),"Continuing importer cannot revive geometry protocol "+fault); + } + f.Pass("Geometry "+fault+" fail-fast and continuing importer guards"); + } + } + + private static void IntegrationCore(IntegrationFixture f) + { + f.Setup("core.sql"); f.Setup("core-lob.sql"); + var request=f.Request(mirror:true); + Assert(f.Service().Preflight(request,CancellationToken.None).CanStart,"Core preflight passes"); + var result=f.Export(request); Assert(result.Stage==MssqlExportStage.Completed,"Core service: "+result.ErrorMessage); + f.RecordArtifact(result.PublishedPath); f.Restore(result.PublishedPath); + IntegrationCompare(f,request); + using(var c=f.Open(f.Target)) + { + Assert(Convert.ToInt64(IntegrationFixture.Scalar(c,"SELECT DATALENGTH(b) FROM dbo.MultiLob WHERE tag=4"))==67108864,"64MiB binary restore"); + Assert(Convert.ToInt64(IntegrationFixture.Scalar(c,"SELECT DATALENGTH(n) FROM dbo.MultiLob WHERE tag=4"))==16793600,"Unicode >16MiB restore"); + Assert(Convert.ToInt32(IntegrationFixture.Scalar(c,"INSERT dbo.IdentityEmpty(v) VALUES(9); SELECT SCOPE_IDENTITY();"))==11,"Empty identity seed"); + Assert(Convert.ToInt32(IntegrationFixture.Scalar(c,"INSERT dbo.IdentityNegative(v) VALUES(9); SELECT SCOPE_IDENTITY();"))==-9,"Descending identity next"); + Assert(Convert.ToInt32(IntegrationFixture.Scalar(c,"INSERT [一].[同名]]表]([unique]) VALUES(4); SELECT SCOPE_IDENTITY();"))==9,"Ascending identity next"); + Assert(Convert.ToInt32(IntegrationFixture.Scalar(c,"SELECT COUNT(*) FROM [一].[同名]]表] WHERE DATALENGTH(rv)=8 AND calc=[unique]+defaulted"))==4,"Regenerated rowversion and computed values"); + } + f.Pass("Core full service sqlcmd restore, metadata and streaming raw hashes, identity, computed and rowversion"); + f.Restore(result.PublishedPath,false); // Existing CREATE must fail without replacing destination. + IntegrationUnsupported(f); + IntegrationGuards(f); + } + + private static void IntegrationClientProbe(IntegrationFixture f) + { + string probe=Path.Combine(f.Root,"client-probe.sql"); + File.WriteAllText(probe,"CREATE TABLE #probe([中文]]欄] nvarchar(max));\r\nINSERT #probe VALUES(N'中文 $(name) :r literal\r\nGO inside string');\r\nGO\r\nIF NOT EXISTS(SELECT 1 FROM #probe WHERE [中文]]欄]=N'中文 $(name) :r literal\r\nGO inside string') THROW 51000,'UTF8/session/parser mismatch',1;\r\nIF NOT EXISTS(SELECT 1 FROM #probe WHERE UNICODE([中文]]欄])=20013) THROW 51000,'UTF8 codepoint mismatch',1;\r\nIF NOT EXISTS(SELECT 1 FROM tempdb.sys.columns WHERE object_id=OBJECT_ID('tempdb..#probe') AND name=NCHAR(20013)+NCHAR(25991)+N']'+NCHAR(27396)) THROW 51000,'UTF8 identifier mismatch',1;\r\nPRINT 'UTF8_SESSION_VARIABLE_GO_PASS';\r\nDROP TABLE #probe;\r\nGO\r\n",new UTF8Encoding(false)); + f.Restore(probe); + Assert(File.ReadAllText(probe+".restore-1.log").Contains("UTF8_SESSION_VARIABLE_GO_PASS"),"Native UTF8 literal/identifier, disabled substitution, GO and same-session temp table"); + string failure=Path.Combine(f.Root,"client-failfast.sql"); + File.WriteAllText(failure,"THROW 51000,'EXPECTED_FAILFAST',1;\r\nGO\r\nPRINT 'UNEXPECTED_AFTER_ERROR';\r\nGO\r\n",new UTF8Encoding(false)); + f.Restore(failure,false); + Assert(!File.ReadAllText(failure+".restore-2.log").Contains("UNEXPECTED_AFTER_ERROR"),"Client -b stops after SQL error"); + f.Pass("Native UTF8 Unicode identifiers/literals, literal dollar-parenthesis, GO parser, one session and fail-fast"); + } + + private static void IntegrationLarge(IntegrationFixture f) + { + int factor=int.Parse(Environment.GetEnvironmentVariable("MYSQLPUNK_MSSQL_INTEGRATION_FACTOR"),CultureInfo.InvariantCulture); + Assert(factor==1 || factor==10,"Known large factor"); + string prior=Environment.GetEnvironmentVariable("MYSQLPUNK_MSSQL_INTEGRATION_RESTORE"); + f.Setup("large.sql",factor); + if(!string.IsNullOrEmpty(prior)) + { + Assert(factor==10 && new FileInfo(prior).Length>2147483648L,"Reuse actual >2GiB service artifact"); + string artifact=Path.Combine(f.Root,"export.sql"); File.Copy(prior,artifact,false); + string tail; + using(var stream=File.OpenRead(artifact)) + { stream.Seek(Math.Max(0,stream.Length-16384),SeekOrigin.Begin); using(var reader=new StreamReader(stream,Encoding.UTF8))tail=reader.ReadToEnd(); } + var pk=Regex.Match(tail,@"ADD CONSTRAINT \[(?PK__[^\]]+)\] PRIMARY KEY"); + Assert(pk.Success,"Original generated Large PK name"); + using(var c=f.Open(f.Source)) + { + string old=Convert.ToString(IntegrationFixture.Scalar(c,"SELECT name FROM sys.key_constraints WHERE parent_object_id=OBJECT_ID('dbo.LargeBinary')")); + IntegrationFixture.Exec(c,"EXEC sys.sp_rename "+IntegrationFixture.Lit("dbo."+old)+","+IntegrationFixture.Lit(pk.Groups["name"].Value)+",N'OBJECT';"); + f.Evidence["sourcePayloadBytes"]=IntegrationFixture.Scalar(c,"SELECT SUM(CONVERT(bigint,DATALENGTH(payload))) FROM dbo.LargeBinary"); + } + var reusedRequest=f.Request("unused.sql",views:false); + f.Evidence["reusedServiceArtifact"]=prior; f.RecordArtifact(artifact); f.Restore(artifact); IntegrationCompare(f,reusedRequest); + f.Pass("Reused actual >2GiB final-service artifact single-session restore and streaming hashes"); + return; + } + var request=f.Request(views:false,mirror:true); + using(var c=f.Open(f.Source)) f.Evidence["sourcePayloadBytes"]=IntegrationFixture.Scalar(c,"SELECT SUM(CONVERT(bigint,DATALENGTH(payload))) FROM dbo.LargeBinary"); + var result=f.Export(request); Assert(result.Stage==MssqlExportStage.Completed,"Large service: "+result.ErrorMessage); + long length=new FileInfo(result.PublishedPath).Length; + if(factor==10)Assert(length>2147483648L,"Actual artifact exceeds 2GiB"); + f.RecordArtifact(result.PublishedPath); f.Restore(result.PublishedPath); IntegrationCompare(f,request); + Assert(result.RowCounts.Sum(x=>x.Rows)==1024L*factor,"Actual large row count"); + f.Pass("Large factor="+factor+" actual bytes="+length+" full service restore and hashes"); + } + + private static void IntegrationCompare(IntegrationFixture f,MssqlExportRequest request) + { + using(var a=f.Open(f.Source))using(var b=f.Open(f.Target)) + { + foreach(var table in request.Tables) + { + string qualified=IntegrationFixture.Q(table.Schema)+"."+IntegrationFixture.Q(table.Name); + var expressions=new List(); + using(var cmd=new SqlCommand("SELECT c.name,ty.name FROM sys.columns c JOIN sys.types ty ON c.user_type_id=ty.user_type_id WHERE c.object_id=OBJECT_ID("+IntegrationFixture.Lit(qualified)+") ORDER BY column_id",a))using(var r=cmd.ExecuteReader()) + while(r.Read()) + { + string name=IntegrationFixture.Q(r.GetString(0)), type=r.GetString(1); + if(type=="timestamp")continue; + if(type=="geometry")name+=".Serialize()"; + if(type=="text")name="CONVERT(varchar(max),"+name+")"; + if(type=="ntext"||type=="xml")name="CONVERT(nvarchar(max),"+name+")"; + expressions.Add("CONVERT(varbinary(max),"+name+")"); + } + string sql="SELECT "+string.Join(",",expressions)+" FROM "+qualified; + var ha=IntegrationRowHashes(a,sql);var hb=IntegrationRowHashes(b,sql); + Assert(ha.SequenceEqual(hb),"Streaming SHA256 rows including NULL/length: "+qualified); + using(var sha=SHA256.Create())f.Evidence["values-"+qualified]=new {rows=ha.Count,sha256=BitConverter.ToString(sha.ComputeHash(Encoding.ASCII.GetBytes(string.Join("",ha)))).Replace("-","")}; + } + foreach(var item in IntegrationMetadataQueries()) + { + string left=IntegrationCatalog(a,item.Value),right=IntegrationCatalog(b,item.Value); + File.WriteAllText(Path.Combine(f.Root,"catalog-"+item.Key+"-source.txt"),left); + File.WriteAllText(Path.Combine(f.Root,"catalog-"+item.Key+"-target.txt"),right); + Assert(left==right,"Catalog equality "+item.Key); + } + } + f.Save(); + } + private static List IntegrationRowHashes(SqlConnection c,string sql) + { + var result=new List(); var buffer=new byte[65536]; + using(var cmd=new SqlCommand(sql,c){CommandTimeout=0})using(var r=cmd.ExecuteReader(CommandBehavior.SequentialAccess)) + while(r.Read())using(var hash=SHA256.Create()) + { + for(int i=0;i0){hash.TransformBlock(buffer,0,count,null,0);at+=count;} + Assert(at==len,"Streaming verifier DATALENGTH"); + } + hash.TransformFinalBlock(new byte[0],0,0);result.Add(BitConverter.ToString(hash.Hash)); + } + result.Sort(StringComparer.Ordinal);return result; + } + private static string IntegrationCatalog(SqlConnection c,string sql) + { + var rows=new List();using(var cmd=new SqlCommand(sql,c))using(var r=cmd.ExecuteReader())while(r.Read()) + {var fields=new List();for(int i=0;i":(r.GetName(i)=="definition" ? Convert.ToString(r.GetValue(i),CultureInfo.InvariantCulture).Trim() : Convert.ToString(r.GetValue(i),CultureInfo.InvariantCulture)));rows.Add(JsonConvert.SerializeObject(fields));} + rows.Sort(StringComparer.Ordinal);return string.Join("\n",rows); + } + private static Dictionary IntegrationMetadataQueries() + { + return new Dictionary { + {"columns", "SELECT s.name,t.name,c.column_id,c.name,ty.name,c.max_length,c.precision,c.scale,c.collation_name,c.is_nullable,c.is_identity,CONVERT(nvarchar(60),ic.seed_value),CONVERT(nvarchar(60),ic.increment_value),cc.definition,cc.is_persisted,dc.name,dc.definition FROM sys.tables t JOIN sys.schemas s ON t.schema_id=s.schema_id JOIN sys.columns c ON t.object_id=c.object_id JOIN sys.types ty ON c.user_type_id=ty.user_type_id LEFT JOIN sys.identity_columns ic ON c.object_id=ic.object_id AND c.column_id=ic.column_id LEFT JOIN sys.computed_columns cc ON c.object_id=cc.object_id AND c.column_id=cc.column_id LEFT JOIN sys.default_constraints dc ON c.default_object_id=dc.object_id WHERE t.is_ms_shipped=0"}, + {"indexes", "SELECT s.name,t.name,i.name,i.type,i.is_unique,i.is_primary_key,i.is_unique_constraint,i.is_disabled,i.filter_definition,c.name,ic.key_ordinal,ic.is_descending_key,ic.is_included_column FROM sys.tables t JOIN sys.schemas s ON t.schema_id=s.schema_id JOIN sys.indexes i ON t.object_id=i.object_id JOIN sys.index_columns ic ON i.object_id=ic.object_id AND i.index_id=ic.index_id JOIN sys.columns c ON c.object_id=ic.object_id AND c.column_id=ic.column_id WHERE t.is_ms_shipped=0"}, + {"checks", "SELECT OBJECT_SCHEMA_NAME(parent_object_id),OBJECT_NAME(parent_object_id),name,definition,is_disabled,is_not_trusted,is_not_for_replication FROM sys.check_constraints"}, + {"foreignKeys", "SELECT OBJECT_SCHEMA_NAME(f.parent_object_id),OBJECT_NAME(f.parent_object_id),f.name,OBJECT_SCHEMA_NAME(f.referenced_object_id),OBJECT_NAME(f.referenced_object_id),COL_NAME(fc.parent_object_id,fc.parent_column_id),COL_NAME(fc.referenced_object_id,fc.referenced_column_id),fc.constraint_column_id,f.is_disabled,f.is_not_trusted,f.delete_referential_action,f.update_referential_action FROM sys.foreign_keys f JOIN sys.foreign_key_columns fc ON f.object_id=fc.constraint_object_id"}, + {"views", "SELECT s.name,v.name,m.definition,m.uses_ansi_nulls,m.uses_quoted_identifier FROM sys.views v JOIN sys.schemas s ON v.schema_id=s.schema_id JOIN sys.sql_modules m ON v.object_id=m.object_id"}, + {"comments", "SELECT OBJECT_SCHEMA_NAME(major_id),OBJECT_NAME(major_id),minor_id,name,CONVERT(nvarchar(max),value) FROM sys.extended_properties WHERE class=1 AND name=N'MS_Description'"} + }; + } + + private static void IntegrationUnsupported(IntegrationFixture f) + { + f.Setup("unsupported.sql"); + foreach(string table in new[]{"RejectSpatial","RejectVariant","RejectAliasTable","RejectTypedXml","RejectSequenceDependency"}) + { + var req=f.Request("reject-"+table+".sql",false,select:t=>t.Key.Name==table); + var pre=f.Service().Preflight(req,CancellationToken.None); + string expected=table=="RejectSpatial"?"UnsupportedSpatial":table=="RejectVariant"?"UnsupportedType":table=="RejectAliasTable"?"UnsupportedUdt":table=="RejectTypedXml"?"UnsupportedTypedXml":"MissingDependency"; + Assert(!pre.CanStart && pre.Issues.Any(x=>x.Code==expected),"Reject "+table+" with "+expected);var res=f.Export(req);Assert(res.Stage==MssqlExportStage.Failed&&!File.Exists(req.TargetPath),"Rejected selection creates no artifact"); + f.Evidence["preflight-"+table]=pre.Issues.Select(x=>x.Code).ToArray(); + } + var encrypted=f.Request("encrypted.sql",true,select:t=>t.Key.Name=="IdentityEmpty"); + var encryptedCheck=f.Service().Preflight(encrypted,CancellationToken.None); + Assert(!encryptedCheck.CanStart && encryptedCheck.Issues.Any(x=>x.Code=="UnreadableDefinition" && x.ObjectKey!=null && x.ObjectKey.Name=="RejectEncrypted"),"Encrypted definition preflight refusal"); + f.Evidence["encryptedDefinitionIssues"]=encryptedCheck.Issues.Select(x=>x.Code+":"+x.ObjectKey?.Name).ToArray(); + using(var c=f.Open("master"))IntegrationFixture.Exec(c,"ALTER DATABASE "+IntegrationFixture.Q(f.Source)+" SET ALLOW_SNAPSHOT_ISOLATION OFF;"); + var off=f.Request("snapshot-off.sql",false,select:t=>t.Key.Name=="IdentityEmpty"); + Assert(f.Service().Preflight(off,CancellationToken.None).Issues.Any(x=>x.Code=="SnapshotUnavailable"),"Snapshot OFF rejected"); + using(var c=f.Open("master"))IntegrationFixture.Exec(c,"ALTER DATABASE "+IntegrationFixture.Q(f.Source)+" SET ALLOW_SNAPSHOT_ISOLATION ON;"); + f.Pass("Live unsupported spatial variant alias typed XML sequence encrypted definition and SNAPSHOT OFF preflight"); + using(var c=f.Open(f.Source))IntegrationFixture.Exec(c,"CREATE USER fixture_reader WITHOUT LOGIN; GRANT SELECT TO fixture_reader; DENY VIEW DEFINITION TO fixture_reader;"); + var hidden=f.Request("hidden.sql",false,select:t=>t.Key.Schema=="一"); + f.RestrictedReader=true; + try { AssertThrows(()=>f.Service().Preflight(hidden,CancellationToken.None),"Denied definitions rejected"); Assert(f.Export(hidden).Stage==MssqlExportStage.Failed && !File.Exists(hidden.TargetPath),"Denied definition creates no artifact"); } + finally { f.RestrictedReader=false; } + f.Pass("Live SELECT reader denied VIEW DEFINITION blocked"); + } + + private static void IntegrationGuards(IntegrationFixture f) + { + using(var c=f.Open(f.Source))IntegrationFixture.Exec(c,"CREATE TABLE dbo.GuardRow(id int,b varbinary(max),x xml); INSERT dbo.GuardRow VALUES(0,0x,CONVERT(xml,N'')),(1,CONVERT(varbinary(max),REPLICATE(CONVERT(varchar(max),'x'),300000)),CONVERT(xml,N'')); "); + using(var c=f.Open(f.Target))IntegrationFixture.Exec(c,"CREATE TABLE dbo.GuardRow(id int,b varbinary(max),x xml);"); + var req=f.Request("guards.sql",false,MssqlExportMode.DataOnly,t=>t.Key.Name=="GuardRow"); + var result=f.Export(req);Assert(result.Stage==MssqlExportStage.Completed,"Guard service artifact"); + var batches=Regex.Split(File.ReadAllText(req.TargetPath),@"(?m)^GO\r?$").Where(s=>!string.IsNullOrWhiteSpace(s)).ToArray(); + var chunks=Enumerable.Range(0,batches.Length).Where(i=>batches[i].Contains(".WRITE(")).ToArray(); + int final=Array.FindLastIndex(batches,s=>s.Contains("INSERT INTO [dbo].[GuardRow]")); + Assert(chunks.Length>=2&&final>=0,"Guard artifact contains staged chunks and final insert"); + foreach(var fault in new[]{"missing","duplicate","outoforder","conversion","replay"}) + { + using(var c=f.Open(f.Target))IntegrationFixture.Exec(c,"TRUNCATE TABLE dbo.GuardRow;"); + var altered=new List(); + for(int i=0;i0&&Convert.ToInt32(IntegrationFixture.Scalar(c,"SELECT COUNT(*) FROM dbo.GuardRow"))==(fault=="replay"?2:1),"Continue-after-error no NULL substitute/duplicate "+fault); + } + f.Pass("Artifact mutation "+fault+" fail-fast sqlcmd and adversarial continuing client; earlier successful row retained"); + } + } + + private static void IntegrationConcurrency(IntegrationFixture f) + { + using(var c=f.Open(f.Source))IntegrationFixture.Exec(c,"CREATE TABLE dbo.Parent(id int PRIMARY KEY,v int,payload varbinary(max)); CREATE TABLE dbo.Child(id int PRIMARY KEY,parent int REFERENCES dbo.Parent(id),v int); INSERT dbo.Parent VALUES(1,1,CONVERT(varbinary(max),REPLICATE(CONVERT(varchar(max),'x'),1048576))); INSERT dbo.Child VALUES(1,1,1);"); + bool updated=false; + f.BeforeRows=t=>{ + if(t.Key.Name!="Child")return; + using(var c=f.Open(f.Source))IntegrationFixture.Exec(c,"BEGIN TRAN; UPDATE dbo.Parent SET v=2; UPDATE dbo.Child SET v=2; COMMIT;"); + updated=true; + }; + var req=f.Request(views:false); var result=f.Export(req); + Assert(updated&&result.Stage==MssqlExportStage.Completed,"Live snapshot export completes");f.BeforeRows=null; + f.RecordArtifact(req.TargetPath);f.Restore(req.TargetPath); + using(var c=f.Open(f.Target))Assert(Convert.ToInt32(IntegrationFixture.Scalar(c,"SELECT COUNT(*) FROM dbo.Parent p JOIN dbo.Child c ON c.parent=p.id WHERE p.v=1 AND c.v=1"))==1,"Parent/child both from pre-update snapshot"); + using(var c=f.Open(f.Source))Assert(Convert.ToInt32(IntegrationFixture.Scalar(c,"SELECT COUNT(*) FROM dbo.Parent p JOIN dbo.Child c ON c.parent=p.id WHERE p.v=2 AND c.v=2"))==1,"Second connection committed both changes"); + f.Pass("Concurrent second connection parent-child transaction restores one consistent snapshot"); + + foreach(string fault in new[]{"ddl","disconnect","cancel-read","write","verify","mirror","cancel-verify-read","cancel-mirror-read","overwrite-race"}) + using(var cancel=new CancellationTokenSource()) + { + string filename=fault+"_backup_export.sql",path=Path.Combine(f.Root,filename);File.WriteAllText(path,"original"); + bool mirrored=fault.Contains("mirror");req=f.Request(filename,false,mirror:mirrored); + f.AfterBegin=fault=="ddl"?(Action)(()=>{using(var c=f.Open(f.Source))IntegrationFixture.Exec(c,"ALTER TABLE dbo.Child ADD extra int NULL;");}):null; + f.BeforeRows=fault=="disconnect"?(Action)(_=>f.WorkerConnections.Last().Close()):null; + bool triggered=false; + result=f.Export(req,cancel.Token,new WorkerFiles(fault,cancel),stage=>{ + if(fault=="verify"&&stage==MssqlExportStage.Verifying)throw new IOException("controlled verify fault"); + if(fault=="overwrite-race"&&stage==MssqlExportStage.Publishing){File.WriteAllText(path,"racing replacement");triggered=true;} + },p=>{ + if(fault=="cancel-read"&&p.Stage==MssqlExportStage.Data&&p.SourceBytes>0 || fault=="cancel-verify-read"&&p.Stage==MssqlExportStage.Verifying&&p.PhaseProcessedBytes>0 || fault=="cancel-mirror-read"&&p.Stage==MssqlExportStage.Mirroring&&p.PhaseProcessedBytes>0) + {triggered=true;cancel.Cancel();} + }); + f.AfterBegin=null;f.BeforeRows=null; + if(mirrored)Assert(result.Stage==MssqlExportStage.LocalCompletedMirrorIncomplete&&result.PublishedPath==path&&File.ReadAllText(path)!="original","Mirror failure retains confirmed local success "+fault); + else + { + Assert(result.Stage==(fault.StartsWith("cancel")?MssqlExportStage.Canceled:MssqlExportStage.Failed)&&result.PublishedPath==null,"Expected failure/cancel outcome "+fault+": "+result.Stage); + Assert(File.ReadAllText(path)==(fault=="overwrite-race"?"racing replacement":"original"),"Prior/racing target preserved "+fault); + } + if(fault.StartsWith("cancel")||fault=="overwrite-race")Assert(triggered,"Fault actually reached "+fault); + Assert(f.WorkerConnections.All(c=>c.State==ConnectionState.Closed),"Live source disposed "+fault); + Assert(!Directory.GetFiles(f.Root,"*.partial",SearchOption.AllDirectories).Any(),"Owned partials cleaned "+fault); + f.Pass("Live service "+fault+" outcome, old-file preservation and source cleanup"); + } + + // Block metadata with an uncommitted schema lock, then cancel the actual native command. + req=f.Request("blocked.sql",false);File.WriteAllText(req.TargetPath,"original");req=f.Request("blocked.sql",false); + using(var blocker=f.Open(f.Source))using(var transaction=blocker.BeginTransaction())using(var cmd=new SqlCommand("ALTER TABLE dbo.Parent ADD blocked int NULL",blocker,transaction))using(var cancel=new CancellationTokenSource()) + { + cmd.ExecuteNonQuery();cancel.CancelAfter(600);var clock=Stopwatch.StartNew();result=f.Export(req,cancel.Token);clock.Stop(); + Assert(result.Stage==MssqlExportStage.Canceled&&clock.Elapsed.TotalSeconds<15,"Blocked actual command cancels promptly"); + Assert(File.ReadAllText(req.TargetPath)=="original","Blocked cancel preserves target"); + Assert(f.WorkerConnections.All(c=>c.State==ConnectionState.Closed),"Blocked command source released"); + transaction.Rollback();f.Evidence["blockedCancelMilliseconds"]=clock.ElapsedMilliseconds; + } + f.Pass("Live schema-lock command wait cancellation and rollback cleanup"); + } +} diff --git a/tests/SmokeTests.MssqlExport.Lob.cs b/tests/SmokeTests.MssqlExport.Lob.cs new file mode 100644 index 0000000..e73a44e --- /dev/null +++ b/tests/SmokeTests.MssqlExport.Lob.cs @@ -0,0 +1,188 @@ +using System; +using System.Collections; +using System.Collections.Generic; +using System.Data.Common; +using System.Globalization; +using System.IO; +using System.Linq; +using System.Text; +using System.Threading; +using mySQLPunk.lib.MssqlExport; + +public static partial class SmokeTests +{ + // Regressions: default missing implementation, backward handoff reads, duplicate/lost + // prefix bytes, unbounded chunks, and premature target INSERT. + private static void Lob_Geometry() + { + // First geometry retained, second partial, following NULL/empty/unread geometry forward only. + using (var reader = new LobReader(new long[] { 30, 18L*1024*1024+3, -1, 0, 300000 })) + using (var sink = new LobSink(true)) { + long bytes=0; + new MssqlDataScriptWriter().WriteTable(reader, ValueTable(Enumerable.Range(0,5).Select(i=>ValueColumn("geometry",i+1,"g"+i)).ToArray()), sink, (r,b)=>bytes=b, CancellationToken.None); + Assert(bytes==30+18L*1024*1024+3+300000 && reader.TotalBytes==bytes, "Retained/partial/unread geometry bytes counted exactly once"); + Assert(sink.BinaryBytes==bytes && sink.Inserts==1 && sink.GeometryInserts==1, "All geometry stage fields reconstructed in one final row"); + } + // Typed text requires staging before any geometry field is read. + var text = new MssqlColumnDefinition(new Dictionary{{"type_name","text"},{"column_id",1},{"name","a"},{"collation_name","Latin1_General_100_BIN2"}}); + using(var reader=new ValueReader(new[]{new object[]{new byte[0],new byte[]{1,2}}})) using(var output=new StringWriter()) { + new MssqlDataScriptWriter().WriteTable(reader,ValueTable(text,ValueColumn("geometry",2,"g")),output,null,CancellationToken.None); + Assert(output.ToString().Contains("geometry::Deserialize([c1])"), "Untouched geometry uses binary staging"); + } + // A completed small geometry must survive metadata-only handoff without reread. + var columns=Enumerable.Range(0,512).Select(i=>ValueColumn(i==0?"geometry":"int",i+1,new string('中',123)+i.ToString("D4"))).ToArray(); + using(var reader=new ValueReader(new[]{Enumerable.Range(0,512).Select(i=>i==0?(object)new byte[40000]:i).ToArray()})) using(var output=new StringWriter()) { + var observer=new LobContinuationObserver();new MssqlDataScriptWriter(observer).WriteTable(reader,ValueTable(columns),output,null,CancellationToken.None); + Assert(observer.Completed && output.ToString().Contains("geometry::Deserialize([c0])"), "Completed geometry suffix stripped before staged reconstruction"); + } + using(var cancel=new CancellationTokenSource()) using(var reader=new LobReader(new long[]{18L*1024*1024},cancel)) using(var sink=new LobSink()) { + AssertThrows(()=>new MssqlDataScriptWriter().WriteTable(reader,ValueTable(ValueColumn("geometry")),sink,null,cancel.Token), "Geometry cancellation during bounded read"); + Assert(sink.Inserts==0 && reader.TotalBytes<18L*1024*1024, "Canceled geometry never inserts target"); + } + } + private static void Lob_LazyLargeRow() + { + var lengths = new long[] { 64L*1024*1024, 18L*1024*1024+6, 0, -1 }; + using(var reader = new LobReader(lengths)) using(var sink = new LobSink(true)) { + long source=0; int during=0; + var table=ValueTable(ValueColumn("image",2,"b"),ValueColumn("ntext",7,"u"),ValueColumn("varbinary",11,"empty"),ValueColumn("nvarchar",19,"nil")); + Assert(new MssqlDataScriptWriter().WriteTable(reader,table,sink,(r,b)=>{Assert(b>=source,"Monotonic source progress");source=b;if(r==0 && b>0 && b10,"Progress while the first large row is still incomplete"); + Assert(source==lengths[0]+lengths[1],"Exact source-byte count across handoff"); + Assert(reader.ReadCalls==2 && reader.TotalBytes==source,"One sequential source pass"); + Assert(sink.Inserts==1 && sink.MaxStatement<=262144 && sink.MaxBatch<=4194304,"Bounded statements and exactly one target insert"); + Assert(sink.BinaryBytes==lengths[0] && sink.UnicodeBytes==lengths[1],"Every retained and new byte emitted exactly once"); + } + } + private static void Lob_ContinuationBranches() + { + var adapter=new LobContinuationObserver(); + using(var reader=new ValueReader(new[]{new object[]{new byte[63460],9}})) using(var sink=new LobSink()) { + new MssqlDataScriptWriter(adapter).WriteTable(reader,ValueTable(ValueColumn("varbinary"),ValueColumn("int",7,"tail")),sink,null,CancellationToken.None); + Assert(adapter.Untouched && sink.Inserts==1 && sink.BinaryBytes==63460,"Untouched current field is consumed once"); + } + // SQL Server permits 1024 columns and 128 Unicode characters per identifier. + // The legal column list alone exceeds the data statement limit; values stay staged. + foreach(int count in new[]{512,1024}) { + var columns=Enumerable.Range(0,count).Select(i=>ValueColumn(i==0 ? "varbinary" : "int",i*2+1,new string('\u4e2d',123)+i.ToString("D4"))).ToArray(); + var data=Enumerable.Range(0,count).Select(i=>i==0 ? (object)new byte[40000] : i).ToArray(); + adapter=new LobContinuationObserver(); + using(var reader=new ValueReader(new[]{data})) using(var sink=new LobSink()) { + new MssqlDataScriptWriter(adapter).WriteTable(reader,ValueTable(columns),sink,null,CancellationToken.None); + Assert((count==512 ? adapter.Completed : adapter.Untouched) && sink.Inserts==1 && sink.BinaryBytes==40000,"Completed or oversized header handoff never rereads source"); + Assert(sink.MaxBatch<=4194304,"Metadata-only insert fits bounded batch"); + } + } + } + private sealed class LobContinuationObserver : IMssqlLargeRowWriter + { + private readonly MssqlLobScriptWriter writer=new MssqlLobScriptWriter(Guid.NewGuid()); + internal bool Untouched,Completed; + public long WriteRow(DbDataReader reader,MssqlTableDefinition table,IReadOnlyList columns,MssqlRowContinuation row,MssqlScriptBatchWriter output,CancellationToken cancellation,Action onAdditionalBytes) { + Untouched=!row.CurrentValueStarted;Completed=row.CurrentValueComplete; + Assert(row.BufferBytes<=262144,"Small row buffer bounded"); + return writer.WriteRow(reader,table,columns,row,output,cancellation,onAdditionalBytes); + } + } + private static void Lob_Utf16AndCancel() + { + using(var reader=new LobReader(new long[]{2000078})) using(var sink=new LobSink(true)) { + new MssqlDataScriptWriter().WriteTable(reader,ValueTable(ValueColumn("nvarchar")),sink,null,CancellationToken.None); + Assert(sink.UnicodeBytes==2000078 && sink.Inserts==1,"Partial UTF16 handoff has no data loss"); + } + using(var cancel=new CancellationTokenSource()) using(var reader=new LobReader(new long[]{64L*1024*1024},cancel)) using(var sink=new LobSink()) { + AssertThrows(()=>new MssqlDataScriptWriter().WriteTable(reader,ValueTable(ValueColumn("varbinary")),sink,null,cancel.Token),"Cancel stops streaming current LOB"); + Assert(sink.Inserts==0 && reader.TotalBytes<64L*1024*1024,"Canceled row cannot be inserted"); + } + } + // Keep only one bounded generated statement, never the large SQL document. + private sealed class LobSink : TextWriter + { + private readonly StringBuilder line=new StringBuilder(); private int batch; + private readonly bool verifyPattern; private readonly Dictionary offsets=new Dictionary(); + private readonly Dictionary lastUnit=new Dictionary(); + internal LobSink(bool verifyPattern=false){this.verifyPattern=verifyPattern;} + internal long BinaryBytes,UnicodeBytes; internal int Inserts,GeometryInserts,MaxStatement,MaxBatch; + public override Encoding Encoding { get { return new UTF8Encoding(false,true); } } + public override void Write(char value) { if(value=='\n') { Consume(line.ToString().TrimEnd('\r'));line.Clear(); } else line.Append(value); } + public override void Write(string value) { foreach(char c in value) Write(c); } + public override void Write(char[] value,int start,int count) { for(int i=0;i=count-2)last|=value<<((j-(count-2))*8); + } + int prior; + if(unicode && lastUnit.TryGetValue(field,out prior))Assert(!(prior>=0xd800 && prior<=0xdbff && first>=0xdc00 && first<=0xdfff),"No split surrogate pair"); + lastUnit[field]=last;offsets[field]=offset+count; + } + } + private static int HexDigit(char value){return value<='9' ? value-'0' : value-'A'+10;} + } + private sealed class LobReader : DbDataReader + { + private readonly long[] lengths; private readonly CancellationTokenSource cancel; + private int ordinal=-1; private long offset; internal long TotalBytes; internal int ReadCalls; + internal LobReader(long[] lengths,CancellationTokenSource cancel=null){this.lengths=lengths;this.cancel=cancel;} + private void At(int i){if(iordinal){ordinal=i;offset=0;}} + public override bool Read(){ReadCalls++;ordinal=-1;offset=0;return ReadCalls==1;} + public override bool IsDBNull(int i){At(i);return lengths[i]<0;} + public override long GetBytes(int i,long start,byte[] buffer,int index,int count){ + At(i);if(buffer==null || start!=offset || count>65536)throw new InvalidOperationException("Nonsequential or unbounded read"); + int n=(int)Math.Min(count,lengths[i]-start); + // Repeated valid surrogate pair, NUL, control, spaces, then unpaired high surrogate. + byte[] pattern={0x3d,0xd8,0x00,0xde,0,0,0x0a,0,0x20,0,0,0xd8,0x41,0}; + for(int j=0;j200000)cancel.Cancel();return n; + } + public override object GetValue(int i){throw new InvalidOperationException("Whole LOB");} + public override string GetString(int i){throw new InvalidOperationException("Whole string");} + public override long GetChars(int i,long a,char[] b,int c,int d){throw new NotSupportedException();} + public override int GetInt32(int i){throw new NotSupportedException();} + public override bool GetBoolean(int i){throw new NotSupportedException();} + public override byte GetByte(int i){throw new NotSupportedException();} + public override short GetInt16(int i){throw new NotSupportedException();} + public override long GetInt64(int i){throw new NotSupportedException();} + public override decimal GetDecimal(int i){throw new NotSupportedException();} + public override float GetFloat(int i){throw new NotSupportedException();} + public override double GetDouble(int i){throw new NotSupportedException();} + public override Guid GetGuid(int i){throw new NotSupportedException();} + public override DateTime GetDateTime(int i){throw new NotSupportedException();} + public override char GetChar(int i){throw new NotSupportedException();} + public override int FieldCount{get{return lengths.Length;}} + public override bool HasRows{get{return true;}} + public override bool IsClosed{get{return false;}} + public override int RecordsAffected{get{return -1;}} + public override int Depth{get{return 0;}} + public override object this[int i]{get{return GetValue(i);}} + public override object this[string name]{get{throw new NotSupportedException();}} + public override string GetName(int i){throw new NotSupportedException();} + public override int GetOrdinal(string name){throw new NotSupportedException();} + public override string GetDataTypeName(int i){throw new NotSupportedException();} + public override Type GetFieldType(int i){throw new NotSupportedException();} + public override int GetValues(object[] values){throw new NotSupportedException();} + public override IEnumerator GetEnumerator(){throw new NotSupportedException();} + public override bool NextResult(){return false;} + } +} diff --git a/tests/SmokeTests.MssqlExport.Routing.cs b/tests/SmokeTests.MssqlExport.Routing.cs new file mode 100644 index 0000000..0139e15 --- /dev/null +++ b/tests/SmokeTests.MssqlExport.Routing.cs @@ -0,0 +1,197 @@ +using System; +using System.Collections.Generic; +using System.IO; +using System.Linq; +using System.Reflection; +using System.Runtime.Serialization; +using System.Windows.Forms; +using mySQLPunk; +using mySQLPunk.lib.MssqlExport; + +public static partial class SmokeTests +{ + // Removing the MSSQL branches from any real handler makes these cases fail. + private sealed class RoutingFixture : IDisposable + { + internal readonly object Form = FormatterServices.GetUninitializedObject(typeof(Form1)); + internal readonly TreeView Tree = new TreeView(); + internal readonly Dictionary Connection; + internal readonly List Options = new List(); + internal readonly List> Settings = new List>(); + internal readonly List Status = new List(); + internal readonly List LegacyFiles = new List(); + internal MssqlExportResult Result; + internal int MirrorReads; + internal MssqlMirrorOptions Mirror; + internal Action Captured; + + internal RoutingFixture(string provider, string group = null, string rawName = "sales.orders") + { + Connection = new Dictionary + { + { "conn_name", "fixture" }, { "provider", provider }, { "isConnect", "T" }, + { "pdo", new FakeDumpDatabase { Provider = provider, ThrowOnGetTables = true } }, + { "CONNSTRING", "must not cross boundary" }, { "host", "fixture.invalid" }, + { "ssh_enabled", "T" }, { "ssh_host", "ssh.fixture.invalid" }, + { "ssh_port", 2222 }, { "runtime", new object() } + }; + var mainField = typeof(Form1).GetField("myN", BindingFlags.Instance | BindingFlags.NonPublic); + var main = FormatterServices.GetUninitializedObject(mainField.FieldType); + mainField.FieldType.GetField("connections").SetValue(main, new List> { Connection }); + mainField.SetValue(Form, main); + SetField("db_tree", Tree); + var root = Tree.Nodes.Add("fixture"); root.Tag = 0; + var database = root.Nodes.Add("fixture_db"); + TreeNode selected = database; + if (group != null) + { + var folder = database.Nodes.Add(group); folder.Name = group; + selected = folder.Nodes.Add(rawName + " (decorated comment must not be identity)"); + var tagType = typeof(Form1).GetNestedType("TreeObjectNodeTag", BindingFlags.NonPublic); + selected.Tag = Activator.CreateInstance(tagType, new object[] { group, rawName, "" }); + } + Tree.SelectedNode = selected; + SetField("MssqlExportWizardRunner", new Func, string, MssqlExportWizardOptions, MssqlExportResult>((settings, db, options) => + { + AssertEquals("fixture_db", db, "Handler passes selected database"); + Options.Add(options); Settings.Add(settings); Captured?.Invoke(); return Result; + })); + SetField("SqlExportSaveDialogRunner", new Func(dialog => { LegacyFiles.Add(dialog.FileName); return DialogResult.Cancel; })); + SetField("MssqlExportMirrorCapture", new Func(() => { MirrorReads++; return Mirror; })); + SetField("MssqlExportStatusSink", new Action(text => Status.Add(text))); + } + + private void SetField(string name, object value) + { + var field = typeof(Form1).GetField(name, BindingFlags.Instance | BindingFlags.NonPublic); + Assert(field != null, "MSSQL routing invocation boundary must exist: " + name); + field.SetValue(Form, value); + } + + internal void Invoke(string handler, params object[] args) + { + typeof(Form1).GetMethod(handler, BindingFlags.Instance | BindingFlags.NonPublic).Invoke(Form, args); + } + public void Dispose() { Tree.Dispose(); } // Never construct/dispose an uninitialized main form. + } + + private static void Routing_FourHandlers() + { + foreach (string provider in new[] { "mssql", "SQLSERVER" }) + foreach (string handler in new[] { "DumpSelectedTableSql", "DumpCurrentSelectionSqlWithDialog", "DumpSelectedDatabaseSqlWithDialog", "BackupSelectedDatabaseWithDialog" }) + { + bool table = handler == "DumpSelectedTableSql" || handler == "DumpCurrentSelectionSqlWithDialog"; + using (var fixture = new RoutingFixture(provider, table ? "Tables" : null)) + { + fixture.Mirror = new MssqlMirrorOptions(Path.Combine(Path.GetTempPath(), "fixture-mirror"), 3); + fixture.Captured = () => fixture.Connection["host"] = "changed.invalid"; + fixture.Invoke(handler, table ? new object[] { true } : new object[0]); + Assert(fixture.Options.Count == 1 && fixture.LegacyFiles.Count == 0, handler + " opens only wizard for wrapper-like provider"); + var options = fixture.Options.Single(); + Assert(options.Mode == (table ? MssqlExportMode.DataOnly : MssqlExportMode.SchemaAndData), "Actual handler retains data-only/table mode"); + AssertEquals(table ? "sales.orders" : null, options.InitialObjectName, "Original tagged seed survives decorated text"); + Assert(options.InitialTables == null && options.IncludeViews == !table, "Table seed selects one; database default includes views"); + bool backup = handler == "BackupSelectedDatabaseWithDialog"; + Assert(fixture.MirrorReads == (backup ? 1 : 0) && (options.Mirror != null) == backup, "Only backup captures mirror once"); + AssertContains(options.DefaultFileName, backup ? "_backup_" : (table ? "_data" : "_dump_"), "Preserve entry filename intent"); + var captured = fixture.Settings.Single(); + Assert(!ReferenceEquals(captured, fixture.Connection), "Connection handoff is detached"); + Assert(!captured.Keys.Any(k => new[] { "pdo", "isConnect", "connString", "runtime" }.Contains(k, StringComparer.OrdinalIgnoreCase)), "No live/runtime settings cross boundary"); + fixture.Connection["ssh_host"] = "changed.invalid"; + fixture.Mirror = null; + AssertEquals("ssh.fixture.invalid", Convert.ToString(captured["ssh_host"]), "SSH facts are retained and copied"); + AssertEquals("fixture.invalid", Convert.ToString(captured["host"]), "Settings mutation during modal handoff cannot affect captured endpoint"); + if (backup) Assert(options.Mirror.RetainCount == 3, "Captured mirror policy survives later settings changes"); + Assert(fixture.Status.Count == 0, "Setup cancellation produces no success status"); + } + } + foreach (string handler in new[] { "DumpSelectedTableSql", "DumpCurrentSelectionSqlWithDialog" }) + using (var fixture = new RoutingFixture("mssql", "Tables", "dbo.name.with.[brackets]")) + { + fixture.Invoke(handler, false); + Assert(fixture.Options.Single().Mode == MssqlExportMode.SchemaAndData, "Table structure and data mode retained"); + AssertEquals("dbo.name.with.[brackets]", fixture.Options.Single().InitialObjectName, "Do not split dots or brackets"); + } + using (var fixture = new RoutingFixture("mssql")) + { + fixture.Invoke("DumpCurrentSelectionSqlWithDialog", true); + Assert(fixture.Options.Single().Mode == MssqlExportMode.SchemaAndData, "Database scope ignores table-only data flag"); + } + } + + private static void Routing_LegacyBranches() + { + foreach (string provider in new[] { "mysql", "sqlite", "postgresql" }) + foreach (string handler in new[] { "DumpSelectedTableSql", "DumpCurrentSelectionSqlWithDialog", "DumpSelectedDatabaseSqlWithDialog", "BackupSelectedDatabaseWithDialog" }) + using (var fixture = new RoutingFixture(provider, "Tables")) + { + fixture.Invoke(handler, handler == "DumpSelectedTableSql" || handler == "DumpCurrentSelectionSqlWithDialog" ? new object[] { false } : new object[0]); + Assert(fixture.Options.Count == 0 && fixture.LegacyFiles.Count == 1, "Other providers retain legacy dialog: " + provider + "/" + handler); + } + foreach (string handler in new[] { "DumpCurrentSelectionSqlWithDialog", "DumpSelectedViewSql" }) + using (var fixture = new RoutingFixture("mssql", "Views", "sales.report")) + { + fixture.Invoke(handler, handler == "DumpCurrentSelectionSqlWithDialog" ? new object[] { false } : new object[0]); + Assert(fixture.Options.Count == 0 && fixture.LegacyFiles.Count == 1, "View-only selection stays on single-view branch"); + AssertContains(fixture.LegacyFiles.Single(), "sales.report_view.sql", "Legacy view filename retained"); + } + } + + private static void Routing_SeedResolution() + { + var catalog = new[] + { + WizardService.Table(1, "dbo", ""), WizardService.Table(2, "sales", ""), + new MssqlTableDefinition(new MssqlObjectKey(3, "dbo", "sales.orders"), null, null, null, null), + new MssqlTableDefinition(new MssqlObjectKey(4, "odd.schema]", "table.name]"), null, null, null, null) + }; + foreach (var test in new[] + { + new { Seed = "orders", ExpectedId = 1, Catalog = catalog }, + new { Seed = "sales.orders", ExpectedId = 2, Catalog = catalog.Take(2).ToArray() }, + new { Seed = "odd.schema].table.name]", ExpectedId = 4, Catalog = catalog }, + new { Seed = "sales.orders", ExpectedId = 0, Catalog = catalog }, + new { Seed = "missing", ExpectedId = 0, Catalog = catalog } + }) + using (var fixture = new RoutingFixture("mssql", "Tables", test.Seed)) + { + fixture.Invoke("DumpSelectedTableSql", false); + var service = new WizardService { CatalogOverride = test.Catalog }; + using (var wizard = new MssqlExportWizardForm(fixture.Settings.Single(), "fixture_db", fixture.Options.Single(), service)) + { + wizard.Show(); + var grid = WControl(wizard, "mssqlTableList"); + WPump(() => grid.Rows.Count == test.Catalog.Length); + var selected = grid.Rows.Cast().Where(row => (bool)row.Cells[0].Value).ToList(); + Assert(selected.Count == (test.ExpectedId == 0 ? 0 : 1), "Captured raw seed resolves uniquely or selects none"); + if (test.ExpectedId != 0) + { + var key = test.Catalog.Single(t => t.Key.ObjectId == test.ExpectedId).Key; + AssertEquals(key.Schema, Convert.ToString(selected.Single().Cells[1].Value), "Native schema identity is preserved"); + AssertEquals(key.Name, Convert.ToString(selected.Single().Cells[2].Value), "Native table identity is preserved"); + } + else Assert(!string.IsNullOrWhiteSpace(WControl