diff --git a/AGENTS.md b/AGENTS.md index 1814fa5..91319f4 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -174,8 +174,69 @@ enum index → `InvalidCastException` (`Slots` vs `Slots`), N on the 2nd distinct key, and any **Guid schema field** → `AmbiguousMatchException` (.NET 10 added a `Guid.ToByteArray(bool)` overload; the unqualified `GetMethod("ToByteArray")` in `Persist`/`Transformer` now passes `Type.EmptyTypes`). Known gap: multi-column composite **prefix** query values are not per-slot -normalized (enum/Nullable only supported for the LEADING prefix field); Guid AS an index field still maps to -`byte[]` (no sentinel → filtered scan). +normalized (enum/Nullable only supported for the LEADING prefix field). + +**Guid / Guid? end-to-end (2026-07, remote-crash chain):** a Guid maps to `DataType.ByteArray` → +portable CLR `byte[]`; `Guid?` → `Slots(ByteArray)` → wrapper `Slots`. Five stacked bugs fixed +(each unmasked the next; covered by `RemoteIndexTests` Guid tests): +1. `DataTransformer.CreateToMethod` pre-`Expression.New(_type2)`d the target — `byte[]` has no default + ctor. Now delegates allocation+conversion to `TransformerHelper.BuildBody` (mirrors `CreateFromMethod`). +2. `BuildBody`'s `Nullable` branch used raw `Expression.Convert` on the inner value (no byte[]↔Guid + coercion) — now recurses through `BuildBody` for the inner value. +3. `TableIndexManager.GetMemberType` returned the `Slots` wrapper for a `Guid?` slot + (`TryGetPortableNullableType` only handles value-type inners) → remote filter literal decoded as the + wrapper → index-seek cast crash. Now flattens via `NormalizeStorageType` (encodings match: + `DataPersist` for Guid / Guid? / byte[] / single-slot Slots all emit the same + length-prefixed bytes). +4. A null wrapper normalized to `default(byte[])` = NULL index key → `GetHashCodeEx` NRE on leaf apply. + `SlotAccessor.NormalizeStorageValue` now emits an EMPTY byte[]/"" for null reference-storage values + (`NullStorageDefault`) — the byte[] analogue of null→default(T). +5. Equality seek on a byte[] field matched 0 rows: `Sentinels` had no byte[] entry (now Min=empty + array, no Max — like string); `BuildScanFromKeyBuilder`/`BuildPrefixSeekKeyBuilder` filled trailing + slots with `Expression.Default` (NULL for byte[]/string PKs, unorderable — now the min sentinel); + `BuildFieldEqualityComparer` used `EqualityComparer.Default` = REFERENCE equality (now + `BigEndianByteArrayEqualityComparer` content equality, scalar + per-slot). Remaining: byte[] has no + MAX sentinel → equality/upper bounds use the min-seek + stop-on-field-change fallback (still a seek). + +**Remote `GetIndex` was session-cache-only (2026-07):** `RemoteTableIndexManager.GetIndex` consulted +only its local cache → a fresh connection saw `null` for an existing server-side index, so an +"ensure" pattern (`GetIndex ?? CreateIndex`) re-created it → server "already exists" throw. Now +refreshes via `ListIndexes()` on a cache miss. (`HasIndexes` is still local-cache-only.) + +## Automatic Schema Migration (2026-07) + +Reopening a table whose RECORD type gained/lost/reordered properties **migrates automatically** +instead of throwing. `StorageEngine.Obtain` detects `recordDataType != locator.RecordDataType` and +calls `MigrateRecordSchema` ([StorageEngine.Migration.cs](src/CatDb/Database/StorageEngine.Migration.cs)): + +- **Mapping is name-based** (the locator persists top-level name→slot member maps; new names come + from the typed record class or the remote client's member map): same name ⇒ copy (slot DataType + must be identical), new member ⇒ default value, removed member ⇒ dropped. Works for slot SHIFTS + (property inserted in the middle). Without names (untyped portable open) it falls back to + positional-prefix (append/truncate only). Values are converted per-slot via + `TransformerHelper.BuildBody` when CLR representations differ (POCO Guid ↔ portable byte[], enum…). +- **Mechanics = table rewrite into a fresh locator** (same name, new id): read all rows in the old + layout, transform, `Clear` + soft-delete the old locator AND the table's `__idx_…` index tables + (their composite keys embed slots positionally — stale), reinsert. Rows are materialized in RAM + during the rewrite (v1); streaming rewrite is the follow-up for huge tables. +- **NOT migratable (diagnostic `ArgumentException`):** key type changes (keys order the tree — + forbidden), primitive (non-Slots) record schemas, and a same-named member whose type changed. +- **Caveats:** table handles obtained BEFORE the migration (same process) are stale — they point at + the soft-deleted locator; reopen to get the live table. A remote client holding the old locator id + from before another client's migration is likewise stale until it reopens. +- Tests: `SchemaMigrationTests` (add-at-end, add-in-middle/slot-shift, remove+add, type-change + diagnostic, index rebuild, remote end-to-end). + +**Index lifecycle fixes that migration depends on (same change):** +- `TableIndexManager.BuildEntry` now obtains its backing table via + `StorageEngine.ObtainInternalTable` which **REUSES the persisted `__idx_…` locator** when its + schema matches → **index data survives process restarts** (previously every `CreateIndex` made a + brand-new empty locator: after any restart, EnsureIndex-style code silently produced EMPTY + indexes — queries matched nothing). Stale-layout index tables are dropped+recreated. +- Index tables are enrolled in the engine `_map` → **`Commit` now flushes their buffered ops** + (previously they bypassed the map and unflushed index writes could be lost). +- `CreateIndex` **backfills** from existing main-table rows when the backing table is new + (`BackfillIfNeeded` → `RebuildEntry`); a reused persisted index table skips the rebuild. **Query-value `null` literal desynced the plan-cache parameter slots (2026-07, found by `TypedIndexStressService` stress churn):** `QueryCompiler.Signature` (the plan-cache key) is diff --git a/Directory.Build.props b/Directory.Build.props index b5919c9..a3c238a 100644 --- a/Directory.Build.props +++ b/Directory.Build.props @@ -21,13 +21,13 @@ The release workflow (.github/workflows/release.yml) requires the GitHub release tag to equal "v$(VersionPrefix)" (or "v$(VersionPrefix)-$(VersionSuffix)" when VersionSuffix is set) and fails the release otherwise — bump this FIRST, then tag/publish the matching release. --> - 2.0.2 + 2.0.3 - 2.0.2 - 2.0.2 + 2.0.3 + 2.0.3 Omid Mafakher diff --git a/examples/CatDb.StressTest/Services.cs b/examples/CatDb.StressTest/Services.cs index 9ea3a88..daf7399 100644 --- a/examples/CatDb.StressTest/Services.cs +++ b/examples/CatDb.StressTest/Services.cs @@ -2163,6 +2163,12 @@ protected override async Task ExecuteAsync(CancellationToken ct) EnsureIndex("Status", () => _items.CreateIndex("Status", r => r.Status, IndexType.NonUnique)); EnsureIndex("ExpiresAt", () => _items.CreateIndex("ExpiresAt", r => r.ExpiresAt, IndexType.NonUnique)); + // Seed the shadow with rows persisted by previous runs: index tables are REUSED across + // restarts now (and CreateIndex backfills), so index counts cover the WHOLE table. The + // old behavior (fresh empty index each start) let an empty shadow pass by accident. + foreach (var kv in _items) + _shadow[kv.Key] = kv.Value; + _nextId = _items.LastRow?.Key ?? 0; // continue ids past any persisted rows for (int i = 0; i < 500; i++) Insert(); diff --git a/src/CatDb/Data/DataTransformer.cs b/src/CatDb/Data/DataTransformer.cs index 9753434..c596d38 100644 --- a/src/CatDb/Data/DataTransformer.cs +++ b/src/CatDb/Data/DataTransformer.cs @@ -43,11 +43,12 @@ public Expression> CreateToMethod() } else { - // Different types — allocate T2, copy fields, return as object + // Different types — build T2 from the value (BuildBody handles conversion + // pairs like Guid↔byte[]/enum↔int and allocates POCOs itself; do NOT + // pre-New _type2 — byte[] has no default ctor), return as object. var t2 = Expression.Variable(_type2); var list = new List { - Expression.Assign(t2, Expression.New(_type2)), TransformerHelper.BuildBody(t2, value, _membersOrder1, _membersOrder2), Expression.Convert(t2, typeof(object)) }; diff --git a/src/CatDb/Data/Transformer.cs b/src/CatDb/Data/Transformer.cs index 956cea9..6095130 100644 --- a/src/CatDb/Data/Transformer.cs +++ b/src/CatDb/Data/Transformer.cs @@ -165,21 +165,22 @@ public static Expression BuildBody(Expression value1, Expression value2, Func(); - - var constructParam = Expression.PropertyOrField(data2Var, type2.IsNullable() ? "Value" : DataTypeUtils.GetPublicMembers(type2, membersOrder2).First().Name); - - var block = Expression.Block(new[] { data1Var, data2Var }, + var data2Var = Expression.Variable(value2.Type); + var innerType = type1.GetGenericArguments()[0]; + var innerVar = Expression.Variable(innerType); + + // Source of the wrapped value: Value for a Nullable, else the single slot. + var srcInner = Expression.PropertyOrField(data2Var, + type2.IsNullable() ? "Value" : DataTypeUtils.GetPublicMembers(type2, membersOrder2).First().Name); + + // Convert the inner value through BuildBody so conversion pairs (Guid↔byte[], + // enum↔integral, numeric widening, nested objects) are honored — a raw + // Expression.Convert has no coercion between e.g. byte[] and Guid. + var block = Expression.Block(new[] { data2Var, innerVar }, Expression.Assign(data2Var, value2), - Expression.Assign(data1Var, Expression.New( - type1.GetConstructor(new[] { type1.GetGenericArguments()[0] })!, - constructParam.GetType() == type1.GetGenericArguments()[0] ? - constructParam : - Expression.Convert(constructParam, type1.GetGenericArguments()[0]))), - Expression.Assign(value1, data1Var) + BuildBody(innerVar, srcInner, membersOrder1, membersOrder2), + Expression.Assign(value1, Expression.New( + type1.GetConstructor(new[] { innerType })!, innerVar)) ); return Expression.IfThenElse(Expression.NotEqual(value2, Expression.Constant(null, type2)), diff --git a/src/CatDb/Database/Indexing/Sentinels.cs b/src/CatDb/Database/Indexing/Sentinels.cs index 1fa5ba8..19b75f8 100644 --- a/src/CatDb/Database/Indexing/Sentinels.cs +++ b/src/CatDb/Database/Indexing/Sentinels.cs @@ -36,6 +36,8 @@ internal static class Sentinels [typeof(TimeSpan)] = (TimeSpan.MinValue, TimeSpan.MaxValue), // string: ordinal-minimum is "", but there is no maximum string. [typeof(string)] = ("", null), + // byte[] (Guid fields normalize to it): minimum is the empty array, no maximum. + [typeof(byte[])] = (Array.Empty(), null), }; /// diff --git a/src/CatDb/Database/Indexing/SlotAccessor.cs b/src/CatDb/Database/Indexing/SlotAccessor.cs index 4ee3b11..cc8cee6 100644 --- a/src/CatDb/Database/Indexing/SlotAccessor.cs +++ b/src/CatDb/Database/Indexing/SlotAccessor.cs @@ -172,7 +172,12 @@ internal static Func BuildScanFromKeyBuilder( } } - ctorArgs[^1] = Expression.Default(primaryKeyType); + // Min sentinel, not Default: for a reference-type PK (byte[] ← Guid, string) default is + // NULL, which the composite-key comparer/hasher cannot order — the seek silently matched + // nothing. Fall back to Default only for types without a sentinel. + ctorArgs[^1] = Sentinels.TryGet(primaryKeyType, max: false, out var pkMin) + ? Expression.Constant(pkMin, primaryKeyType) + : Expression.Default(primaryKeyType); var compositeCtor = compositeKeyType.GetConstructors() .First(c => c.GetParameters().Length == totalSlots); @@ -203,30 +208,46 @@ internal static Func BuildFieldEqualityComparer(Type fieldTy Expression? combined = null; foreach (var member in members) { - var memberType = GetMemberType(member); - var accessA = AccessMember(valueA, member); - var accessB = AccessMember(valueB, member); - var slotComparerType = typeof(System.Collections.Generic.EqualityComparer<>).MakeGenericType(memberType); - var slotDefaultProp = slotComparerType.GetProperty("Default", BindingFlags.Static | BindingFlags.Public)!; - var slotEqualsMethod = slotComparerType.GetMethod("Equals", [memberType, memberType])!; - var slotComparer = Expression.Property(null, slotDefaultProp); - var slotEquals = Expression.Call(slotComparer, slotEqualsMethod, accessA, accessB); + var memberType = GetMemberType(member); + var accessA = AccessMember(valueA, member); + var accessB = AccessMember(valueB, member); + var slotEquals = BuildEqualsCall(memberType, accessA, accessB); combined = combined == null ? slotEquals : Expression.AndAlso(combined, slotEquals); } equalsExpr = combined ?? Expression.Constant(true); } else { - var comparerType = typeof(System.Collections.Generic.EqualityComparer<>).MakeGenericType(fieldType); - var defaultProp = comparerType.GetProperty("Default", BindingFlags.Static | BindingFlags.Public)!; - var equalsMethod = comparerType.GetMethod("Equals", [fieldType, fieldType])!; - var comparer = Expression.Property(null, defaultProp); - equalsExpr = Expression.Call(comparer, equalsMethod, valueA, valueB); + equalsExpr = BuildEqualsCall(fieldType, valueA, valueB); } return Expression.Lambda>(equalsExpr, aParam, bParam).Compile(); } + /// + /// Equality call for one (slot) value pair. byte[] (a Guid field's storage type) needs CONTENT + /// equality — EqualityComparer<byte[]>.Default is reference equality, which made a + /// re-materialized query value never match a stored index key. + /// + private static Expression BuildEqualsCall(Type type, Expression a, Expression b) + { + if (type == typeof(byte[])) + { + var cmp = Expression.Field(null, typeof(General.Comparers.BigEndianByteArrayEqualityComparer) + .GetField(nameof(General.Comparers.BigEndianByteArrayEqualityComparer.Instance))!); + return Expression.Call(cmp, + typeof(General.Comparers.BigEndianByteArrayEqualityComparer) + .GetMethod(nameof(General.Comparers.BigEndianByteArrayEqualityComparer.Equals), [typeof(byte[]), typeof(byte[])])!, + a, b); + } + + var comparerType = typeof(System.Collections.Generic.EqualityComparer<>).MakeGenericType(type); + var defaultProp = comparerType.GetProperty("Default", BindingFlags.Static | BindingFlags.Public)!; + var equalsMethod = comparerType.GetMethod("Equals", [type, type])!; + var comparer = Expression.Property(null, defaultProp); + return Expression.Call(comparer, equalsMethod, a, b); + } + internal static Type GetSlotType(Type compositeKeyType, int slotIndex) => GetMemberType(GetSlotMember(compositeKeyType, slotIndex)); @@ -255,7 +276,10 @@ internal static Func BuildPrefixSeekKeyBuilder(Type compositeKeyTy args[i] = AccessMember(prefixValue, GetSlotMember(prefixType, i)); } for (var i = prefixLen; i < ps.Length; i++) - args[i] = Expression.Default(ps[i].ParameterType); + // Min sentinel over Default: default(byte[])/default(string) is NULL and unorderable. + args[i] = Sentinels.TryGet(ps[i].ParameterType, max: false, out var min) + ? Expression.Constant(min, ps[i].ParameterType) + : Expression.Default(ps[i].ParameterType); var newComposite = Expression.New(ctor, args); var castResult = Expression.Convert(newComposite, typeof(object)); @@ -314,7 +338,7 @@ private static Expression NormalizeStorageValue(Expression access) if (!t.IsValueType) return Expression.Condition( Expression.ReferenceEqual(access, Expression.Constant(null, t)), - Expression.Default(inner.Type), + NullStorageDefault(inner.Type), inner); return inner; } @@ -322,6 +346,21 @@ private static Expression NormalizeStorageValue(Expression access) return access; } + /// + /// Storage stand-in for a null wrapper: default(T) for value types, but a NON-null empty + /// value for reference storage types — a null byte[]/string index key would NRE the + /// key hash/comparer on leaf apply (portable Guid? normalizes to byte[], so a null + /// Guid must become an empty array, the byte[] analogue of default(T)). + /// + private static Expression NullStorageDefault(Type t) + { + if (t == typeof(byte[])) + return Expression.Constant(Array.Empty(), typeof(byte[])); + if (t == typeof(string)) + return Expression.Constant(string.Empty, typeof(string)); + return Expression.Default(t); + } + /// /// True when is the portable representation of a /// (or any single-field composite): a one-slot ISlots. Multi-slot composites (real composite diff --git a/src/CatDb/Database/Indexing/TableIndexManager.Query.cs b/src/CatDb/Database/Indexing/TableIndexManager.Query.cs index 49e9bc5..014db95 100644 --- a/src/CatDb/Database/Indexing/TableIndexManager.Query.cs +++ b/src/CatDb/Database/Indexing/TableIndexManager.Query.cs @@ -146,7 +146,16 @@ internal Type GetMemberType(string member) // same CLR type the client encoded it with (typeof(TField)); otherwise `.PrimitiveType` throws // "The type (T) is not primitive" on the wrapper. var clr = DataTypeUtils.BuildType(slotDt); - return SlotAccessor.TryGetPortableNullableType(clr, out var nullable) ? nullable : clr; + if (SlotAccessor.TryGetPortableNullableType(clr, out var nullable)) + return nullable; + + // Reference-type inner (Guid? → Slots(ByteArray) → Slots): TryGetPortableNullableType + // only handles value-type inners, so flatten the wrapper to its inner storage type here. + // The client encodes the FLAT value (DataPersist(Guid)/(Guid?) and (byte[]) share the same + // length-prefixed layout), and every downstream consumer (value normalizer, seek bounds, + // residual/sort comparers) works on the flat storage type — decoding as the Slots wrapper + // instead produced a Slots instance that the index seek then failed to cast to byte[]. + return SlotAccessor.NormalizeStorageType(clr); } /// CLR type of the primary key (for remote key-range decoding). diff --git a/src/CatDb/Database/Indexing/TableIndexManager.cs b/src/CatDb/Database/Indexing/TableIndexManager.cs index 80dfea8..64c9884 100644 --- a/src/CatDb/Database/Indexing/TableIndexManager.cs +++ b/src/CatDb/Database/Indexing/TableIndexManager.cs @@ -43,14 +43,15 @@ public IndexDefinition CreateIndex(string indexName, int[] slotIndices, IndexTyp throw new ArgumentException("Index name cannot be empty.", nameof(indexName)); if (slotIndices == null || slotIndices.Length == 0) throw new ArgumentException("Must specify at least one slot index.", nameof(slotIndices)); - if (_indexes.ContainsKey(indexName)) - throw new InvalidOperationException($"Index '{indexName}' already exists on table '{_tableName}'."); + if (TryGetIdenticalOrThrow(indexName, slotIndices, type) is { } same) + return same; var memberNames = ResolveMemberNames(slotIndices); var def = new IndexDefinition(indexName, slotIndices, memberNames, type); - var entry = BuildEntry(def); + var entry = BuildEntry(def, out var preexisting); _indexes[indexName] = entry; _planCache.Clear(); // index set changed → cached plans may now be stale + BackfillIfNeeded(entry, preexisting); return def; } @@ -60,17 +61,70 @@ public IndexDefinition CreateIndex(string indexName, string[] memberNames, Index throw new ArgumentException("Index name cannot be empty.", nameof(indexName)); if (memberNames == null || memberNames.Length == 0) throw new ArgumentException("Must specify at least one member name.", nameof(memberNames)); - if (_indexes.ContainsKey(indexName)) - throw new InvalidOperationException($"Index '{indexName}' already exists on table '{_tableName}'."); var slotIndices = ResolveSlotIndices(memberNames); + if (TryGetIdenticalOrThrow(indexName, slotIndices, type) is { } same) + return same; + var def = new IndexDefinition(indexName, slotIndices, memberNames, type); - var entry = BuildEntry(def); + var entry = BuildEntry(def, out var preexisting); _indexes[indexName] = entry; _planCache.Clear(); + BackfillIfNeeded(entry, preexisting); return def; } + /// + /// Re-registering an index that already exists with the IDENTICAL definition is a no-op + /// (SQL "IF NOT EXISTS" semantics) — the natural "ensure" pattern, and what a retried startup + /// does. A DIFFERENT definition under the same name still throws. + /// + private IndexDefinition? TryGetIdenticalOrThrow(string indexName, int[] slotIndices, IndexType type) + { + if (!_indexes.TryGetValue(indexName, out var existing)) + return null; + + var def = existing.Definition; + if (def.Type == type && def.SlotIndices.AsSpan().SequenceEqual(slotIndices)) + return def; + + throw new InvalidOperationException( + $"Index '{indexName}' already exists on table '{_tableName}' with a different definition " + + $"(existing: slots [{string.Join(",", def.SlotIndices)}] {def.Type}; " + + $"requested: slots [{string.Join(",", slotIndices)}] {type}). Drop it first to redefine."); + } + + /// + /// A NEWLY created backing table for a main table that already holds rows must be built now — + /// without this, creating an index on existing data (or after a schema migration dropped the + /// stale index tables) left the index silently EMPTY: queries through it matched nothing. + /// A REUSED (persisted) index table is verified instead of trusted: every index (unique or + /// composite non-unique) holds exactly one entry per row, so an entry-count/row-count mismatch + /// means the persisted index is not in step with the table (e.g. written by a version that did + /// not flush index tables at commit) — rebuild it. Cost: one streaming count of each table at + /// index registration; a persisted sync stamp is the follow-up optimization. + /// + private void BackfillIfNeeded(IndexEntry entry, bool preexistingIndexTable) + { + _table.Flush(); + + if (preexistingIndexTable) + { + entry.IndexTable.Flush(); + if (entry.IndexTable.Count() == _table.Count()) + return; // in step — reuse as-is + } + + if (_table.Forward().Any()) + RebuildEntry(entry); + else + { + // Empty main table: make sure a stale persisted index doesn't resurrect ghost rows. + entry.IndexTable.Clear(); + entry.IndexTable.Flush(); + } + } + public void DropIndex(string indexName) { if (!_indexes.TryGetValue(indexName, out var entry)) @@ -81,7 +135,10 @@ public void DropIndex(string indexName) // Mark the index table's locator as deleted var indexTableName = entry.Definition.GetTableName(_tableName); - DeleteIndexTable(indexTableName); + if (_tree is StorageEngine engine) + engine.RemoveInternalTable(indexTableName); // clears + soft-deletes + unmaps + else + DeleteIndexTable(indexTableName); _indexes.Remove(indexName); _planCache.Clear(); @@ -531,7 +588,30 @@ internal void OnClear() private static readonly IData _dummyValue = (object)(byte)0; - private IndexEntry BuildEntry(IndexDefinition def) + /// + /// Opens the backing index table. Through a this REUSES the + /// persisted locator when its schema still matches (index data survives restarts, and the + /// table is enrolled in the engine map so commits flush it); + /// then tells whether a backfill is needed. + /// + private XTablePortable ObtainIndexTable( + string indexTableName, DataType keyDataType, DataType recordDataType, Type keyType, Type recordType, + out bool preexisting) + { + if (_tree is StorageEngine engine) + return engine.ObtainInternalTable(indexTableName, keyDataType, recordDataType, keyType, recordType, out preexisting); + + // Raw WTree (no engine surface): legacy path — always a fresh locator. + preexisting = false; + var idxLocator = _tree.CreateLocator(indexTableName, StructureType.XTABLE, + keyDataType, recordDataType, keyType, recordType); + if (!idxLocator.IsReady) idxLocator.Prepare(); + return new XTablePortable(_tree, idxLocator); + } + + private IndexEntry BuildEntry(IndexDefinition def) => BuildEntry(def, out _); + + private IndexEntry BuildEntry(IndexDefinition def, out bool preexistingIndexTable) { var recordDataType = _locator.RecordDataType; var recordType = _locator.RecordType @@ -592,15 +672,11 @@ private IndexEntry BuildEntry(IndexDefinition def) Func? prefixSeekBuilder = null; IComparer? prefixComparer = null; + var preexisting = false; if (def.Type == IndexType.Unique) { // Unique: index table key = field value, value = primary key - var idxLocator = _tree.CreateLocator( - indexTableName, StructureType.XTABLE, - fieldDataType, keyDataType, - fieldType, keyType); - if (!idxLocator.IsReady) idxLocator.Prepare(); - indexTable = new XTablePortable(_tree, idxLocator); + indexTable = ObtainIndexTable(indexTableName, fieldDataType, keyDataType, fieldType, keyType, out preexisting); } else { @@ -612,12 +688,7 @@ private IndexEntry BuildEntry(IndexDefinition def) compositeKeyType = SlotsBuilder.BuildType(compositeStorageTypes); var compositeKeyDataType = DataTypeUtils.BuildDataType(compositeKeyType); - var idxLocator = _tree.CreateLocator( - indexTableName, StructureType.XTABLE, - compositeKeyDataType, DataType.Byte, - compositeKeyType, typeof(byte)); - if (!idxLocator.IsReady) idxLocator.Prepare(); - indexTable = new XTablePortable(_tree, idxLocator); + indexTable = ObtainIndexTable(indexTableName, compositeKeyDataType, DataType.Byte, compositeKeyType, typeof(byte), out preexisting); nonUniqueKeyBuilder = SlotAccessor.BuildNonUniqueKeyBuilder( recordType, def.SlotIndices, keyType); @@ -638,6 +709,7 @@ private IndexEntry BuildEntry(IndexDefinition def) } } + preexistingIndexTable = preexisting; return new IndexEntry( def, indexTable, fieldExtractor, fieldEquals, fieldComparer, nonUniqueKeyBuilder, scanFromKeyBuilder, diff --git a/src/CatDb/Database/StorageEngine.Migration.cs b/src/CatDb/Database/StorageEngine.Migration.cs new file mode 100644 index 0000000..37a4f03 --- /dev/null +++ b/src/CatDb/Database/StorageEngine.Migration.cs @@ -0,0 +1,321 @@ +// Copyright (c) 2024-2026 CatDb (https://github.com/OmidID/CatDb) +// Licensed under the MIT License. See LICENSE in the project root for license information. + +using System.Linq.Expressions; +using CatDb.Data; +using CatDb.WaterfallTree; + +namespace CatDb.Database; + +public partial class StorageEngine +{ + /// + /// Automatic record-schema migration: called from (under the engine lock) + /// when a table is reopened with a record DataType different from the stored one. + /// + /// Strategy — rewrite into a fresh locator (same name, new id), like a SQL table rewrite: + /// 1. Build a slot mapping old→new. Name-based when both sides have member names (the locator + /// persists name→slot maps; the new names come from the caller's record type / member map): + /// same name ⇒ copy (slot DataType must match), new name ⇒ default value, removed ⇒ dropped. + /// Without names, positional-prefix: slots are matched by position; appended slots get + /// defaults, truncated slots are dropped. + /// 2. Read every row with the OLD layout, transform, buffer. + /// 3. Clear + soft-delete the old locator (and this table's now-stale secondary index tables). + /// 4. Create the new locator under the same name and reinsert the transformed rows. + /// + /// Not migratable (throws a diagnostic ArgumentException): non-Slots (primitive) record schemas, + /// and a same-named member whose slot DataType changed. + /// + /// Note: rows are materialized in memory during the rewrite (v1) — fine for the entity-table + /// sizes this targets; a streaming rewrite through a temp locator is the follow-up for huge tables. + /// Secondary indexes are NOT re-created here: their definitions live with the caller + /// (CreateIndex/EnsureIndex on next use), and CreateIndex backfills from existing rows. + /// + private Item1 MigrateRecordSchema( + string name, Item1 item, DataType newRecordDataType, Type? newRecordType, + IReadOnlyDictionary? newRecordNames) + { + var oldLocator = item.Locator; + var oldDt = oldLocator.RecordDataType; + + if (!oldDt.IsSlots || !newRecordDataType.IsSlots) + throw new ArgumentException( + $"Record schema mismatch for table '{name}': stored record schema is {oldDt}, " + + $"but it was opened with {newRecordDataType}. Only composite (multi-field) record schemas " + + "can be migrated automatically; a primitive record type cannot change. " + + "Open with the original type, or delete/recreate the table."); + + // Resolve CLR types. The old CLR class may no longer exist (renamed/reshaped entity) — + // fall back to the portable Slots type; its persist layout is identical. + var oldRecordType = oldLocator.RecordType ?? DataTypeUtils.BuildType(oldDt); + newRecordType ??= DataTypeUtils.BuildType(newRecordDataType); + + // Member names: old side from the persisted locator map; new side from the caller + // (typed open → record type's public members; portable open → provided member map). + var oldNames = oldLocator.RecordMembers; + newRecordNames ??= SchemaMigrator.NamesFromType(newRecordType); + + var transform = SchemaMigrator.BuildRowTransformer( + name, oldRecordType, oldDt, oldNames, newRecordType, newRecordDataType, newRecordNames); + + // 2. Read all rows in the old layout. + if (oldLocator.RecordType is null) + oldLocator.RecordType = oldRecordType; + if (!oldLocator.IsReady) + oldLocator.Prepare(); + item.Table ??= new XTablePortable(this, oldLocator); + item.Table.Flush(); + + var rows = new List>(); + foreach (var kv in item.Table.Forward()) + rows.Add(new KeyValuePair(kv.Key, transform(kv.Value))); + + // 3. Retire the old locator + this table's secondary index tables (their composite keys + // embed record slots by position — stale after the migration; recreated+backfilled by + // the next CreateIndex). + item.Table.Clear(); + item.Table.Flush(); + oldLocator.IsDeleted = true; + _map.Remove(name); + + var idxPrefix = $"{InternalNaming.ReservedPrefix}idx_{name}_"; + foreach (var idxName in _map.Keys.Where(k => k.StartsWith(idxPrefix, StringComparison.Ordinal)).ToList()) + RemoveInternalTable(idxName); + + // 4. New locator, same name; carry over the key member map (key schema is unchanged). + // Record map: only from a typed record class here. For portable (Slots) records the map is + // left NULL on purpose — the open overload calls SetMembers with the client's full recursive + // map right after Obtain (SetMembers is only-if-null, so setting a flat map here would block + // it; and MemberMap.Build on a Slots type would persist useless "Slot0"/"Slot1" names). + MemberMap? newRecordMap = null; + if (!DataTypeUtils.IsAnonymousType(newRecordType) && !typeof(ISlots).IsAssignableFrom(newRecordType)) + newRecordMap = MemberMap.Build(newRecordDataType, newRecordType); + + var newLocator = CreateLocator(name, oldLocator.StructureType, + oldLocator.KeyDataType, newRecordDataType, oldLocator.KeyType, newRecordType); + newLocator.SetMembers(oldLocator.KeyMemberMap, newRecordMap); + if (!newLocator.IsReady) + newLocator.Prepare(); + + var newItem = new Item1(newLocator, new XTablePortable(this, newLocator)); + _map[name] = newItem; + + foreach (var kv in rows) + newItem.Table.Replace(kv.Key, kv.Value); + newItem.Table.Flush(); + + return newItem; + } + + /// + /// Opens (creating if absent) an engine-internal backing table, REUSING the persisted locator + /// when its schema still matches — this is what makes secondary-index data survive a process + /// restart. On a schema mismatch (main table migrated ⇒ the index composite key changed) the + /// stale table is dropped and a fresh one is created. Routing internal tables through + /// also enrolls them in _map, so their buffered operations are + /// flushed by like any public table. + /// + internal XTablePortable ObtainInternalTable( + string name, DataType keyDataType, DataType recordDataType, Type keyType, Type recordType, + out bool preexisting) + { + if (!InternalNaming.IsInternal(name)) + throw new ArgumentException( + $"Internal table name must start with '{InternalNaming.ReservedPrefix}'.", nameof(name)); + + _syncRoot.Enter(); + try + { + if (_map.TryGetValue(name, out var existing)) + { + if (existing.Locator.KeyDataType == keyDataType && + existing.Locator.RecordDataType == recordDataType) + { + preexisting = true; + existing.Locator.KeyType ??= keyType; + existing.Locator.RecordType ??= recordType; + if (!existing.Locator.IsReady) + existing.Locator.Prepare(); + existing.Table ??= new XTablePortable(this, existing.Locator); + return existing.Table; + } + + // Stale layout — drop and recreate below. + RemoveInternalTable(name); + } + + preexisting = false; + var item = Obtain(name, StructureType.XTABLE, keyDataType, recordDataType, keyType, recordType, allowInternal: true); + return item.Table; + } + finally { _syncRoot.Exit(); } + } + + /// Clears + soft-deletes an engine-internal table and forgets it. No-op when absent. + internal void RemoveInternalTable(string name) + { + _syncRoot.Enter(); + try + { + if (!_map.TryGetValue(name, out var item)) + return; + + if (!item.Locator.IsReady) + item.Locator.Prepare(); + item.Table ??= new XTablePortable(this, item.Locator); + item.Table.Clear(); + item.Table.Flush(); + item.Locator.IsDeleted = true; + _map.Remove(name); + } + finally { _syncRoot.Exit(); } + } +} + +/// +/// Builds the old→new record transformation for an automatic schema migration +/// (see ). +/// +internal static class SchemaMigrator +{ + /// Top-level member name → slot index for a record CLR type (null for anonymous/Slots types). + internal static IReadOnlyDictionary? NamesFromType(Type recordType) + { + if (DataTypeUtils.IsAnonymousType(recordType) || typeof(ISlots).IsAssignableFrom(recordType)) + return null; + + var names = new Dictionary(StringComparer.Ordinal); + var i = 0; + foreach (var member in DataTypeUtils.GetPublicMembers(recordType)) + names[member.Name] = i++; + return names.Count > 0 ? names : null; + } + + /// + /// Compiles a row transformer old-record → new-record according to the slot mapping. + /// Mapping is name-based when both name sets are known, else positional-prefix. + /// Throws a diagnostic ArgumentException for non-migratable changes. + /// + internal static Func BuildRowTransformer( + string tableName, + Type oldRecordType, DataType oldDt, IReadOnlyDictionary? oldNames, + Type newRecordType, DataType newDt, IReadOnlyDictionary? newNames) + { + var mapping = BuildSlotMapping(tableName, oldDt, oldNames, newDt, newNames); + + var input = Expression.Parameter(typeof(object), "oldRecord"); + var oldVar = Expression.Variable(oldRecordType, "old"); + var newVar = Expression.Variable(newRecordType, "new"); + + var body = new List + { + Expression.Assign(oldVar, Expression.Convert(input, oldRecordType)), + }; + + if (typeof(ISlots).IsAssignableFrom(newRecordType)) + { + // Portable record: single ctor taking all slots in order. + var ctor = newRecordType.GetConstructors() + .OrderByDescending(c => c.GetParameters().Length).First(); + var ps = ctor.GetParameters(); + var args = new Expression[ps.Length]; + for (var i = 0; i < ps.Length; i++) + args[i] = SlotValue(oldVar, oldRecordType, mapping[i], ps[i].ParameterType); + body.Add(Expression.Assign(newVar, Expression.New(ctor, args))); + } + else + { + // POCO record: parameterless ctor + member assignments. + body.Add(Expression.Assign(newVar, Expression.New(newRecordType))); + var members = DataTypeUtils.GetPublicMembers(newRecordType).ToArray(); + for (var i = 0; i < members.Length; i++) + { + var target = Expression.PropertyOrField(newVar, members[i].Name); + body.Add(Expression.Assign(target, SlotValue(oldVar, oldRecordType, mapping[i], target.Type))); + } + } + + body.Add(Expression.Convert(newVar, typeof(object))); + var block = Expression.Block(typeof(object), new[] { oldVar, newVar }, body); + return Expression.Lambda>(block, input).Compile(); + } + + /// Per NEW slot: the OLD slot index that feeds it, or null → default value. + private static int?[] BuildSlotMapping( + string tableName, + DataType oldDt, IReadOnlyDictionary? oldNames, + DataType newDt, IReadOnlyDictionary? newNames) + { + var oldCount = oldDt.TypesCount; + var newCount = newDt.TypesCount; + var mapping = new int?[newCount]; + + if (oldNames is { Count: > 0 } && newNames is { Count: > 0 }) + { + var newByIndex = newNames.ToDictionary(kv => kv.Value, kv => kv.Key); + for (var i = 0; i < newCount; i++) + { + if (!newByIndex.TryGetValue(i, out var memberName) || + !oldNames.TryGetValue(memberName, out var oldSlot)) + { + mapping[i] = null; // new member → default value + continue; + } + + if (oldDt[oldSlot] != newDt[i]) + throw new ArgumentException( + $"Cannot migrate table '{tableName}': member '{memberName}' changed type " + + $"from {oldDt[oldSlot]} to {newDt[i]}. Changing an existing member's type is " + + "not supported — rename the new member (old data is dropped) or migrate manually."); + + mapping[i] = oldSlot; + } + return mapping; + } + + // No member names (untyped portable open) — positional prefix match. + var shared = Math.Min(oldCount, newCount); + for (var i = 0; i < shared; i++) + { + if (oldDt[i] != newDt[i]) + throw new ArgumentException( + $"Cannot migrate table '{tableName}': no member names are stored for the old schema and " + + $"the layouts diverge at slot {i} ({oldDt[i]} vs {newDt[i]}). Positional migration only " + + "supports appending new fields at the end or truncating trailing fields. Reopen the " + + "table with a typed record class (so member names persist), or migrate manually."); + mapping[i] = i; + } + for (var i = shared; i < newCount; i++) + mapping[i] = null; + return mapping; + } + + /// Expression producing the NEW slot value: mapped old slot (converted if the CLR + /// representation differs, e.g. POCO Guid ↔ portable byte[]) or the type's default. + private static Expression SlotValue(Expression oldVar, Type oldRecordType, int? oldSlot, Type targetType) + { + if (oldSlot is null) + return Expression.Default(targetType); + + var source = SlotAccess(oldVar, oldRecordType, oldSlot.Value); + if (source.Type == targetType) + return source; + + // Same slot DataType, different CLR representation (enum↔integral, Guid↔byte[], POCO↔Slots + // nesting) — route through the transformer body, which handles all conversion pairs. + var slotVar = Expression.Variable(targetType); + return Expression.Block(targetType, new[] { slotVar }, + TransformerHelper.BuildBody(slotVar, source, null, null), + slotVar); + } + + private static Expression SlotAccess(Expression instance, Type recordType, int slot) + { + var slotField = recordType.GetField($"Slot{slot}"); + if (slotField != null) + return Expression.Field(instance, slotField); + + var member = DataTypeUtils.GetPublicMembers(recordType).ElementAt(slot); + return Expression.PropertyOrField(instance, member.Name); + } +} diff --git a/src/CatDb/Database/StorageEngine.cs b/src/CatDb/Database/StorageEngine.cs index 82e522b..07f719c 100644 --- a/src/CatDb/Database/StorageEngine.cs +++ b/src/CatDb/Database/StorageEngine.cs @@ -10,7 +10,7 @@ namespace CatDb.Database; -public class StorageEngine : WTree, IStorageEngine +public partial class StorageEngine : WTree, IStorageEngine { private readonly Dictionary _map = new(); private readonly Dictionary _transformerCache = new(); @@ -33,7 +33,7 @@ public StorageEngine(IHeap heap, DatabaseOptions? options = null, Storage.Operat } } - private Item1 Obtain(string name, int structureType, DataType keyDataType, DataType recordDataType, Type keyType, Type recordType, bool allowInternal = false) + private Item1 Obtain(string name, int structureType, DataType keyDataType, DataType recordDataType, Type keyType, Type recordType, bool allowInternal = false, IReadOnlyDictionary? recordMemberNames = null) { Debug.Assert(keyDataType != null); Debug.Assert(recordDataType != null); @@ -60,19 +60,29 @@ private Item1 Obtain(string name, int structureType, DataType keyDataType, DataT if (locator.StructureType != structureType) throw new ArgumentException($"Invalid structure type for '{name}'"); if (keyDataType != locator.KeyDataType) - throw new ArgumentException(nameof(keyDataType)); + throw new ArgumentException( + $"Key schema mismatch for table '{name}': stored key type is {locator.KeyDataType}, " + + $"but it was opened with {keyDataType}. The key type of an existing table cannot change; " + + "open with the original type, or delete/recreate the table."); if (recordDataType != locator.RecordDataType) - throw new ArgumentException(nameof(recordDataType)); - - locator.KeyType ??= DataTypeUtils.BuildType(keyDataType); - if (keyType != null && keyType != locator.KeyType) - throw new ArgumentException($"Invalid keyType for table '{name}'"); + { + // Record schema changed (entity gained/lost/reordered properties): migrate the + // stored rows to the new layout instead of refusing to open. Throws a diagnostic + // ArgumentException when the change is not migratable (see SchemaMigrator). + item = MigrateRecordSchema(name, item, recordDataType, recordType, recordMemberNames); + } + else + { + locator.KeyType ??= DataTypeUtils.BuildType(keyDataType); + if (keyType != null && keyType != locator.KeyType) + throw new ArgumentException($"Invalid keyType for table '{name}'"); - locator.RecordType ??= DataTypeUtils.BuildType(recordDataType); - if (recordType != null && recordType != locator.RecordType) - throw new ArgumentException($"Invalid recordType for table '{name}'"); + locator.RecordType ??= DataTypeUtils.BuildType(recordDataType); + if (recordType != null && recordType != locator.RecordType) + throw new ArgumentException($"Invalid recordType for table '{name}'"); - locator.AccessTime = DateTime.Now; + locator.AccessTime = DateTime.Now; + } } if (!item.Locator.IsReady) @@ -104,7 +114,8 @@ public ITable OpenXTablePortable( _syncRoot.Enter(); try { - var item = Obtain(name, StructureType.XTABLE, keyDataType, recordDataType, null, null); + var item = Obtain(name, StructureType.XTABLE, keyDataType, recordDataType, null, null, + recordMemberNames: recordMembers); item.Locator.SetMembers(keyMembers, recordMembers); return item.Table; } @@ -121,7 +132,8 @@ public ITable OpenXTablePortable( _syncRoot.Enter(); try { - var item = Obtain(name, StructureType.XTABLE, keyDataType, recordDataType, null, null); + var item = Obtain(name, StructureType.XTABLE, keyDataType, recordDataType, null, null, + recordMemberNames: recordMemberMap?.Names); item.Locator.SetMembers(keyMemberMap, recordMemberMap); return item.Table; } diff --git a/src/CatDb/Remote/RemoteTableIndexManager.cs b/src/CatDb/Remote/RemoteTableIndexManager.cs index ada87c3..860f9c1 100644 --- a/src/CatDb/Remote/RemoteTableIndexManager.cs +++ b/src/CatDb/Remote/RemoteTableIndexManager.cs @@ -31,6 +31,7 @@ public IndexDefinition CreateIndex(string indexName, int[] slotIndices, IndexTyp var cmd = new IndexCreateCommand(indexName, slotIndices, memberNames, type); _table.Execute(cmd); var def = new IndexDefinition(indexName, slotIndices, memberNames, type); + _localCache.RemoveAll(d => d.Name == indexName); // server create is idempotent — don't double-cache _localCache.Add(def); return def; } @@ -41,6 +42,7 @@ public IndexDefinition CreateIndex(string indexName, string[] memberNames, Index var cmd = new IndexCreateCommand(indexName, [], memberNames, type); _table.Execute(cmd); var def = new IndexDefinition(indexName, [], memberNames, type); + _localCache.RemoveAll(d => d.Name == indexName); // server create is idempotent — don't double-cache _localCache.Add(def); return def; } @@ -54,6 +56,16 @@ public void DropIndex(string indexName) public IndexDefinition? GetIndex(string indexName) { + var found = _localCache.Find(d => d.Name == indexName); + if (found is not null) + return found; + + // The local cache is only seeded by THIS session's CreateIndex/ListIndexes calls. + // On a fresh connection it is empty even when the index already exists server-side, + // so a plain cache lookup would wrongly report "not found" and a subsequent + // CreateIndex would throw "already exists". Refresh from the server before concluding + // the index is absent. + ListIndexes(); return _localCache.Find(d => d.Name == indexName); } diff --git a/src/CatDb/Remote/StorageEngineServer.cs b/src/CatDb/Remote/StorageEngineServer.cs index e4b92ce..123aed6 100644 --- a/src/CatDb/Remote/StorageEngineServer.cs +++ b/src/CatDb/Remote/StorageEngineServer.cs @@ -677,13 +677,15 @@ private ICommand StorageEngineRename(IStorageEngine storageEngine, ICommand comm private ICommand StorageEngineOpenXIndex(IStorageEngine storageEngine, ICommand command) { var cmd = (StorageEngineOpenXIndexCommand)command; - storageEngine.OpenXTablePortable(cmd.Name, cmd.KeyType, cmd.RecordType); + // Route the client's member maps INTO the open: schema migration (record shape changed) + // resolves old→new slots by member NAME, so the names must be available at Obtain time — + // not stamped on afterwards. + if (cmd.KeyMembers != null || cmd.RecordMembers != null) + storageEngine.OpenXTablePortable(cmd.Name, cmd.KeyType, cmd.RecordType, cmd.KeyMembers, cmd.RecordMembers); + else + storageEngine.OpenXTablePortable(cmd.Name, cmd.KeyType, cmd.RecordType); var loc = storageEngine[cmd.Name]; - // If the client sent member names, store them in the locator so they persist. - if (loc is Locator locator && (cmd.KeyMembers != null || cmd.RecordMembers != null)) - locator.SetMembers(cmd.KeyMembers, cmd.RecordMembers); - return new StorageEngineOpenXIndexCommand(loc.Id); } diff --git a/tests/CatDb.Tests/Database/KeyValueTypeTests.cs b/tests/CatDb.Tests/Database/KeyValueTypeTests.cs index 02c9c65..d47f8f6 100644 --- a/tests/CatDb.Tests/Database/KeyValueTypeTests.cs +++ b/tests/CatDb.Tests/Database/KeyValueTypeTests.cs @@ -178,4 +178,31 @@ public void Table_ByteArrayValue_Works() t[1] = data; t[1].Should().BeEquivalentTo(data); } + + public class RecordV1 { public string Name { get; set; } = ""; public int Age { get; set; } } + public class RecordV2 { public string Name { get; set; } = ""; public int Age { get; set; } public bool Active { get; set; } } + + // Reopening with a changed record shape now MIGRATES automatically (name-based slot remap; + // see SchemaMigrationTests for the full matrix). It must not throw, and data must survive. + [Fact] + public void Table_ReopenWithChangedRecordSchema_Migrates() + { + var t1 = _engine.OpenXTable("t"); + t1[1] = new RecordV1 { Name = "Ada", Age = 36 }; + + var t2 = _engine.OpenXTable("t"); + t2[1].Name.Should().Be("Ada"); + t2[1].Age.Should().Be(36); + t2[1].Active.Should().BeFalse(); + } + + [Fact] + public void Table_ReopenWithChangedKeyType_ThrowsDiagnosticMessage() + { + _engine.OpenXTable("t"); + + var act = () => _engine.OpenXTable("t"); + act.Should().Throw() + .WithMessage("*Key schema mismatch for table 't'*"); + } } diff --git a/tests/CatDb.Tests/Database/RemoteIndexTests.cs b/tests/CatDb.Tests/Database/RemoteIndexTests.cs index 8cff279..2083846 100644 --- a/tests/CatDb.Tests/Database/RemoteIndexTests.cs +++ b/tests/CatDb.Tests/Database/RemoteIndexTests.cs @@ -383,4 +383,153 @@ public async Task Remote_SecondaryIndex_And_Ordering_EndToEnd() await server.StopAsync(); } } + + public class GuidRecord + { + public Guid Id { get; set; } + public Guid? OwnerId { get; set; } + public string Name { get; set; } = ""; + } + + // Regression: a Guid key/record over the remote client used to throw + // "Type 'System.Byte[]' does not have a default constructor" in + // DataTransformer.CreateToMethod — the remote descriptor regenerates the CLR + // type from the DataType (Guid → ByteArray → byte[]), so the transformer maps + // Guid ↔ byte[]; CreateToMethod pre-New'd byte[] instead of using BuildBody. + [Fact] + public async Task Remote_GuidKeyAndRecord_RoundTrip() + { + var port = FreePort(); + using var serverEngine = CatDb.Database.CatDb.FromMemory(); + await using var tcp = new TcpServer(port); + var server = new StorageEngineServer(serverEngine, tcp, accessPolicy: null); + await server.StartAsync(); + + var key = Guid.NewGuid(); + var owner = Guid.NewGuid(); + try + { + using (var client = CatDb.Database.CatDb.FromNetwork("localhost", port, "default", "u", "p")) + { + var table = client.OpenXTable("guids"); + table.Replace(key, new GuidRecord { Id = key, OwnerId = owner, Name = "Omid" }); + client.Commit(); + + var back = table.Find(key); + back.Should().NotBeNull(); + back!.Id.Should().Be(key); + back.OwnerId.Should().Be(owner); + back.Name.Should().Be("Omid"); + } + } + finally + { + await server.StopAsync(); + } + } + + // Regression: querying an indexed Guid?/Guid field over the remote client threw + // "Unable to cast object of type 'CatDb.Data.Slots`1[System.Byte[]]' to type 'System.Byte[]'" + // server-side: GetMemberType returned the portable wrapper Slots for a Guid? slot + // (TryGetPortableNullableType only handles value-type inners), so the filter literal was + // decoded as the wrapper and the index seek failed to cast it to the flat byte[] storage key. + [Fact] + public async Task Remote_QueryOnGuidNullableIndexedField_Works() + { + var port = FreePort(); + using var serverEngine = CatDb.Database.CatDb.FromMemory(); + await using var tcp = new TcpServer(port); + var server = new StorageEngineServer(serverEngine, tcp, accessPolicy: null); + await server.StartAsync(); + + var owner = Guid.NewGuid(); + var other = Guid.NewGuid(); + try + { + using var client = CatDb.Database.CatDb.FromNetwork("localhost", port, "default", "u", "p"); + var table = client.OpenXTable("apps"); + table.CreateIndex("OwnerId", r => r.OwnerId, IndexType.NonUnique); + + for (var i = 0; i < 20; i++) + { + var id = Guid.NewGuid(); + table.Replace(id, new GuidRecord + { + Id = id, + OwnerId = i % 4 == 0 ? null : (i % 2 == 0 ? owner : other), + Name = $"app{i}", + }); + } + client.Commit(); + + table.Count().Should().Be(20); + table.Indexes.FindByIndexRange("OwnerId", null, false, true, null, false, true, backward: false) + .Count().Should().Be(20, "full index range = every row"); + table.Indexes.CountByIndex("OwnerId", owner).Should().Be(5, "direct index count"); + table.Indexes.FindByIndex("OwnerId", owner).Count().Should().Be(5, "direct index find"); + + // The exact repo pattern: fetch all records for one owner via the indexed Guid? field. + var mine = table.Query(x => x.OwnerId).Equal(owner).ToList(); + mine.Should().HaveCount(5); // i ∈ {2,6,10,14,18} + mine.Should().OnlyContain(kv => kv.Value.OwnerId == owner); + + table.Query(x => x.OwnerId).Equal(owner).Count().Should().Be(5); + table.Query(x => x.OwnerId).Equal(other).Count().Should().Be(10); + + // Exact repo shape: Equal(guid) + ORDER BY another field (drives a different plan). + var ordered = table.Query(x => x.OwnerId).Equal(owner).OrderBy(x => x.Name).ToList(); + ordered.Should().HaveCount(5); + ordered.Select(kv => kv.Value.Name).Should().BeInAscendingOrder(); + } + finally + { + await server.StopAsync(); + } + } + + // Regression: RemoteTableIndexManager.GetIndex only consulted a session-local cache, + // so a FRESH connection reported an existing index as null → an EnsureIndex-style + // "GetIndex ?? CreateIndex" pattern re-created it → server threw "already exists". + // GetIndex must refresh from the server on a cache miss. + [Fact] + public async Task Remote_GetIndex_FindsIndexCreatedOnEarlierConnection() + { + var port = FreePort(); + using var serverEngine = CatDb.Database.CatDb.FromMemory(); + await using var tcp = new TcpServer(port); + var server = new StorageEngineServer(serverEngine, tcp, accessPolicy: null); + await server.StartAsync(); + + try + { + // Connection 1: create the index. + using (var client = CatDb.Database.CatDb.FromNetwork("localhost", port, "default", "u", "p")) + { + var table = client.OpenXTable("customers"); + table.CreateIndex("City", c => c.City, IndexType.NonUnique); + client.Commit(); + } + + // Connection 2 (fresh cache): GetIndex must see the server-side index. + using (var client = CatDb.Database.CatDb.FromNetwork("localhost", port, "default", "u", "p")) + { + var table = client.OpenXTable("customers"); + + table.Indexes.GetIndex("City").Should().NotBeNull(); + table.Indexes.GetIndex("Nonexistent").Should().BeNull(); + + // The EnsureIndex pattern must now be a no-op (must NOT throw "already exists"). + var act = () => + { + if (table.Indexes.GetIndex("City") is null) + table.CreateIndex("City", c => c.City, IndexType.NonUnique); + }; + act.Should().NotThrow(); + } + } + finally + { + await server.StopAsync(); + } + } } diff --git a/tests/CatDb.Tests/Database/SchemaMigrationTests.cs b/tests/CatDb.Tests/Database/SchemaMigrationTests.cs new file mode 100644 index 0000000..59290f7 --- /dev/null +++ b/tests/CatDb.Tests/Database/SchemaMigrationTests.cs @@ -0,0 +1,314 @@ +// Copyright (c) 2024-2026 CatDb (https://github.com/OmidID/CatDb) +// Licensed under the MIT License. See LICENSE in the project root for license information. + +using System.Net; +using System.Net.Sockets; +using CatDb.Database; +using CatDb.Database.Indexing; +using CatDb.Extensions; +using CatDb.General.Communication; +using CatDb.Remote; +using FluentAssertions; + +namespace CatDb.Tests.Database; + +/// +/// Automatic record-schema migration: reopening a table with a record type that gained, lost or +/// reordered properties must migrate the stored rows (name-based slot remap; new members default, +/// removed members drop) instead of throwing. Covers local file reopen, remote open, index +/// survival via CreateIndex backfill, and the non-migratable diagnostics. +/// +public class SchemaMigrationTests : IDisposable +{ + private readonly string _filePath; + + public SchemaMigrationTests() + { + _filePath = Path.Combine(Path.GetTempPath(), $"catdb_migr_{Guid.NewGuid():N}.db"); + } + + public void Dispose() + { + if (File.Exists(_filePath)) + File.Delete(_filePath); + } + + public class PersonV1 + { + public string Name { get; set; } = ""; + public int Age { get; set; } + } + + // V1 + appended property + public class PersonV2Added + { + public string Name { get; set; } = ""; + public int Age { get; set; } + public bool Active { get; set; } + } + + // V1 + property added in the MIDDLE (shifts slots — only name-based mapping survives this) + public class PersonV2Middle + { + public string Name { get; set; } = ""; + public string City { get; set; } = ""; + public int Age { get; set; } + } + + // V1 with a property removed and another added + public class PersonV2Mixed + { + public int Age { get; set; } + public Guid? Tag { get; set; } + } + + // V1 with Age's type changed — NOT migratable + public class PersonV2TypeChanged + { + public string Name { get; set; } = ""; + public string Age { get; set; } = ""; + } + + [Fact] + public void AddProperty_AtEnd_MigratesAndKeepsData() + { + using (var engine = CatDb.Database.CatDb.FromFile(_filePath)) + { + var t = engine.OpenXTable("people"); + t[1L] = new PersonV1 { Name = "Ada", Age = 36 }; + t[2L] = new PersonV1 { Name = "Alan", Age = 41 }; + engine.Commit(); + } + + using (var engine = CatDb.Database.CatDb.FromFile(_filePath)) + { + var t = engine.OpenXTable("people"); + t.Count().Should().Be(2); + t[1L].Name.Should().Be("Ada"); + t[1L].Age.Should().Be(36); + t[1L].Active.Should().BeFalse(); // new member → default + engine.Commit(); + } + + // And the migrated data persists across another reopen. + using (var engine = CatDb.Database.CatDb.FromFile(_filePath)) + { + var t = engine.OpenXTable("people"); + t[2L].Name.Should().Be("Alan"); + } + } + + [Fact] + public void AddProperty_InMiddle_RemapsByName() + { + using (var engine = CatDb.Database.CatDb.FromFile(_filePath)) + { + var t = engine.OpenXTable("people"); + t[1L] = new PersonV1 { Name = "Ada", Age = 36 }; + engine.Commit(); + } + + using (var engine = CatDb.Database.CatDb.FromFile(_filePath)) + { + var t = engine.OpenXTable("people"); + var p = t[1L]; + p.Name.Should().Be("Ada"); + p.City.Should().BeNullOrEmpty(); // inserted member → default + p.Age.Should().Be(36); // slot SHIFTED but name-mapped + } + } + + [Fact] + public void RemoveAndAddProperties_DropsOldKeepsMatched() + { + using (var engine = CatDb.Database.CatDb.FromFile(_filePath)) + { + var t = engine.OpenXTable("people"); + t[1L] = new PersonV1 { Name = "Ada", Age = 36 }; + engine.Commit(); + } + + using (var engine = CatDb.Database.CatDb.FromFile(_filePath)) + { + var t = engine.OpenXTable("people"); + var p = t[1L]; + p.Age.Should().Be(36); // matched by name + p.Tag.Should().BeNull(); // new Guid? member → default + } + } + + [Fact] + public void ChangeMemberType_ThrowsDiagnostic() + { + using (var engine = CatDb.Database.CatDb.FromFile(_filePath)) + { + var t = engine.OpenXTable("people"); + t[1L] = new PersonV1 { Name = "Ada", Age = 36 }; + engine.Commit(); + } + + using (var engine = CatDb.Database.CatDb.FromFile(_filePath)) + { + var act = () => engine.OpenXTable("people"); + act.Should().Throw() + .WithMessage("*member 'Age' changed type*"); + } + } + + [Fact] + public void Migration_WithIndexes_IndexRebuiltViaCreateIndex() + { + using (var engine = CatDb.Database.CatDb.FromFile(_filePath)) + { + var t = engine.OpenXTable("people"); + t.CreateIndex("Name", x => x.Name, IndexType.NonUnique); + for (var i = 0; i < 50; i++) + t[i] = new PersonV1 { Name = $"p{i % 5}", Age = i }; + engine.Commit(); + } + + using (var engine = CatDb.Database.CatDb.FromFile(_filePath)) + { + var t = engine.OpenXTable("people"); // migrates + t.CreateIndex("Name", x => x.Name, IndexType.NonUnique); // recreates + backfills + t.Query(x => x.Name).Equal("p3").Count().Should().Be(10); + t.Query(x => x.Name).Equal("p3").ToList().Should().OnlyContain(kv => kv.Value.Name == "p3"); + } + } + + [Fact] + public void IndexData_SurvivesReopen_WithoutMigration() + { + // Index-table locators are now reused across restarts (and enrolled in the engine map for + // commit flushing): re-registering the index on reopen must NOT wipe it, and it must work + // immediately — this used to leave a silently EMPTY index after every process restart. + using (var engine = CatDb.Database.CatDb.FromFile(_filePath)) + { + var t = engine.OpenXTable("people"); + t.CreateIndex("Name", x => x.Name, IndexType.NonUnique); + for (var i = 0; i < 30; i++) + t[i] = new PersonV1 { Name = $"p{i % 3}", Age = i }; + engine.Commit(); + } + + using (var engine = CatDb.Database.CatDb.FromFile(_filePath)) + { + var t = engine.OpenXTable("people"); + t.CreateIndex("Name", x => x.Name, IndexType.NonUnique); + t.Query(x => x.Name).Equal("p1").Count().Should().Be(10); + } + } + + [Fact] + public void CreateIndex_OnExistingRows_Backfills() + { + using var engine = CatDb.Database.CatDb.FromMemory(); + var t = engine.OpenXTable("people"); + for (var i = 0; i < 20; i++) + t[i] = new PersonV1 { Name = $"p{i % 2}", Age = i }; + + // Index created AFTER the data exists must see it. + t.CreateIndex("Name", x => x.Name, IndexType.NonUnique); + t.Query(x => x.Name).Equal("p0").Count().Should().Be(10); + } + + [Fact] + public void CreateIndex_SameDefinitionTwice_IsIdempotent() + { + using var engine = CatDb.Database.CatDb.FromMemory(); + var t = engine.OpenXTable("people"); + t[1] = new PersonV1 { Name = "Ada", Age = 36 }; + + t.CreateIndex("Name", x => x.Name, IndexType.NonUnique); + // Second identical registration (retried startup / ensure pattern) must be a no-op… + var act = () => t.CreateIndex("Name", x => x.Name, IndexType.NonUnique); + act.Should().NotThrow(); + t.Query(x => x.Name).Equal("Ada").Count().Should().Be(1); + + // …but a CONFLICTING definition under the same name still throws. + var conflict = () => t.CreateIndex("Name", x => x.Age, IndexType.NonUnique); + conflict.Should().Throw() + .WithMessage("*different definition*"); + } + + [Fact] + public void ReusedIndexTable_OutOfStepWithTable_IsRebuilt() + { + // Out-of-step persisted index: rows written while NO index was registered (run 2 below — + // the manager only maintains indexes it knows about), then the index re-registered on a + // later open. The reuse path must detect the entry-count/row-count drift and rebuild — + // trusting the stale index would silently hide the run-2 rows from index queries. + using (var engine = CatDb.Database.CatDb.FromFile(_filePath)) + { + var t = engine.OpenXTable("people"); + t.CreateIndex("Name", x => x.Name, IndexType.NonUnique); + for (var i = 0; i < 20; i++) + t[i] = new PersonV1 { Name = $"p{i % 2}", Age = i }; + engine.Commit(); + } + + using (var engine = CatDb.Database.CatDb.FromFile(_filePath)) + { + var t = engine.OpenXTable("people"); + for (var i = 20; i < 30; i++) // no index registered → index table not maintained + t[i] = new PersonV1 { Name = $"p{i % 2}", Age = i }; + engine.Commit(); + } + + using (var engine = CatDb.Database.CatDb.FromFile(_filePath)) + { + var t = engine.OpenXTable("people"); + t.CreateIndex("Name", x => x.Name, IndexType.NonUnique); + t.Query(x => x.Name).Equal("p1").Count().Should().Be(15); // 10 + 5 from run 2 + } + } + + private static int FreePort() + { + var l = new TcpListener(IPAddress.Loopback, 0); + l.Start(); + var port = ((IPEndPoint)l.LocalEndpoint).Port; + l.Stop(); + return port; + } + + [Fact] + public async Task Remote_SchemaChange_MigratesOnOpen() + { + var port = FreePort(); + using var serverEngine = CatDb.Database.CatDb.FromMemory(); + await using var tcp = new TcpServer(port); + var server = new StorageEngineServer(serverEngine, tcp, accessPolicy: null); + await server.StartAsync(); + + try + { + using (var client = CatDb.Database.CatDb.FromNetwork("localhost", port, "default", "u", "p")) + { + var t = client.OpenXTable("people"); + t.Replace(1L, new PersonV1 { Name = "Ada", Age = 36 }); + client.Commit(); + } + + // New client, evolved entity: server must migrate, not throw. + using (var client = CatDb.Database.CatDb.FromNetwork("localhost", port, "default", "u", "p")) + { + var t = client.OpenXTable("people"); + var p = t.Find(1L); + p.Should().NotBeNull(); + p!.Name.Should().Be("Ada"); + p.Age.Should().Be(36); + p.City.Should().BeNullOrEmpty(); + + t.Replace(2L, new PersonV2Middle { Name = "Alan", City = "London", Age = 41 }); + client.Commit(); + + t.Find(2L)!.City.Should().Be("London"); + } + } + finally + { + await server.StopAsync(); + } + } +}