fix(duckdb): three DuckDB 2.0 drift defects — cursor row loss, false schema drift, invalid discovered contracts

.github/workflows/ci.yml

@@ -683,6 +683,9 @@ jobs:
             tests/providers/test_duckdb_compat.py \
             tests/providers/test_duckdb_sandbox.py \
             tests/providers/test_duckdb_sandbox_workspace_root.py \
+            tests/build_runners/test_duckdb_cursor_canonical.py \
+            tests/build_runners/test_schema_evolution_type_family.py \
+            tests/cli/discover/test_discover_type_canonicalization.py \
             tests/output_ports/test_drivers_duckdb.py \
             tests/build_runners/test_duckdb_runner.py \
             tests/build_runners/test_duckdb_runner_mysql_attach.py \

CHANGELOG.md

@@ -7,6 +7,53 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
 
 ## [Unreleased]
 
+### Fixed
+
+- **An incremental cursor written on DuckDB 1.5 and read on 2.0 silently dropped or
+  duplicated rows.** The cursor is stored as text and compared as an untyped SQL literal,
+  and the two versions disagree about the column type it is coerced against:
+  `read_json_auto` over a timestamp carrying `Z` gives TIMESTAMP on 1.5 and TIMESTAMP WITH
+  TIME ZONE on 2.0, where a naive literal is read in the session time zone -- the host's.
+  On a host with a non-zero UTC offset the `>` boundary therefore moved: an end-to-end run
+  that wrote its cursor on 1.5 and continued on 2.0 landed 5 of 6 rows and advanced the
+  cursor past the row it skipped, at exit code 0 with the stream recorded `succeeded`.
+  Both DuckDB versions resolve to `duckdb>=1.5.0`, so one install can write state another
+  reads.
+  A timestamp cursor now compares INSTANTS (`epoch_us`), which no session time zone can
+  shift, and the stored text is normalised to a naive UTC instant. **No state file needs
+  rewriting**: a cursor a shipped 2.0 install already wrote carries an explicit offset and
+  is normalised when read. Integer, date-as-integer and opaque string cursors keep the
+  plain comparison. CSV sources were never affected -- `read_csv_auto` already sniffed
+  TIMESTAMPTZ on 1.5.
+  Pinning the session zone to UTC would also have worked and is deliberately not used:
+  `TimeZone` is ICU-backed, the sandbox forbids autoloading extensions, and without ICU
+  DuckDB does not recognise the option at all, so a pin would fail the connection in
+  exactly the hardened environments the sandbox exists for. The accepted cost of the
+  instant comparison is that wrapping the column in a function can stop the predicate
+  being pushed into the Parquet or JSON reader.
+- **A pipeline discovered on DuckDB 1.5 failed `SchemaDriftError` on 2.0.** The drift gate
+  compared type *names*, and 2.0 infers HUGEINT where 1.5 infers DOUBLE for the same
+  integer literal, and TIMESTAMP WITH TIME ZONE where 1.5 infers TIMESTAMP. Upgrading
+  DuckDB — no contract edit, no new data — broke a working pipeline on a false positive.
+  Those pairs are now treated as one physical type under two names. The widening/narrowing
+  classification is unchanged: the family check runs after it, so `type_widened` and
+  `type_narrowed` still fire for every ordered numeric pair. Note DOUBLE and HUGEINT are
+  not value-identical — 1.5 landed a lossy 17-significant-digit float where 2.0 lands the
+  exact 128-bit integer — so the non-failing path says the physical type moved rather than
+  passing silently.
+- **`fluid init --discover` emitted contracts its own `fluid validate` rejected.** The
+  emitter wrote the source's type name straight through, so a discover on DuckDB 2.0
+  produced `type: hugeint` and `type: '"null"'` and `fluid validate` then exited 1 on the
+  file the tool had just written. Validating every name in DuckDB's own `duckdb_types()`
+  against the bundled schema showed **35 rejected names**, not the two reported: `uinteger`
+  and `integer[]` were rejected on 1.5 as well, so this was never only a 2.0 problem.
+  Discovered types are now canonicalised at the emitter, mapping each rejected name to the
+  nearest one the schema accepts and leaving every already-valid name untouched. An
+  unsigned or 128-bit type maps to a width that holds its range (`uinteger` → `bigint`,
+  `hugeint` → `numeric`) rather than to the similar-sounding name. Contracts already emitted
+  with an invalid type stay invalid — the fix is emit-time — but those already failed
+  validation, so nothing working changes.
+
 ### Security
 
 - A contract repository could hand the whole host to its own SQL through DuckDB's spill

fluid_build/build_runners/_acquisition_common.py

@@ -38,7 +38,7 @@
 import time
 from datetime import datetime, timezone
 from pathlib import Path
-from typing import Any, Callable, Dict, List, Optional, Tuple
+from typing import Any, Callable, Dict, List, Optional, Sequence, Tuple
 
 LOG = logging.getLogger("fluid.build_runners.acquisition")
 
@@ -717,6 +717,7 @@ def enforce_schema_policy_or_raise(ctx: Any, runner: Any) -> None:
         overrides=contract_block.get("evolutionOverrides") or {},
         is_first_run=False,
     )
+    _log_family_renames(baseline, current)
     if not plan.has_failure:
         return
 
@@ -736,6 +737,39 @@ def enforce_schema_policy_or_raise(ctx: Any, runner: Any) -> None:
     )
 
 
+def _log_family_renames(
+    baseline: Sequence[Any],
+    current: Sequence[Any],
+) -> None:
+    """Tell the operator about a type change the drift gate deliberately allowed.
+
+    ``diff_columns`` emits no event when a column's type changed only between
+    two spellings of one physical type (DuckDB 1.5's DOUBLE vs 2.0's HUGEINT,
+    TIMESTAMP vs TIMESTAMP WITH TIME ZONE), so that upgrading DuckDB does not
+    raise a false ``SchemaDriftError``. Suppressing the FAILURE is the intent;
+    suppressing the NOTICE as well is not, because the two types are not always
+    value-identical -- 1.5 landed a lossy 17-significant-digit float where 2.0
+    lands the exact 128-bit integer.
+
+    Logged whatever the policy decides, and before the gate's early return, so
+    a contract that declared ``schemaPolicy: strict`` still hears that the
+    physical type moved even though the run is allowed to continue.
+    """
+    from fluid_build.build_runners._schema_evolution import family_rename_note
+
+    by_name = {getattr(c, "name", None): c for c in baseline}
+    for col in current:
+        was = by_name.get(getattr(col, "name", None))
+        if was is None:
+            continue
+        note = family_rename_note(getattr(was, "type", "") or "", getattr(col, "type", "") or "")
+        if note:
+            LOG.warning(
+                "acquisition_schema_family_rename",
+                extra={"column": getattr(col, "name", None), "detail": note},
+            )
+
+
 def write_run_record(
     *,
     state_store: Any,

fluid_build/build_runners/_schema_evolution.py

@@ -66,6 +66,109 @@ def _is_narrowing(old: str, new: str) -> bool:
         return False
 
 
+#: Type names that are two spellings of one physical type, so a change between
+#: them is a RENAME rather than drift. Each entry was measured on duckdb 1.5.6
+#: and 2.0.0 over the same input bytes; nothing is here on the strength of the
+#: name alone.
+#:
+#: DuckDB 2.0 infers HUGEINT where 1.5 infers DOUBLE for the same integer
+#: literal, and TIMESTAMP WITH TIME ZONE where 1.5 infers TIMESTAMP for the same
+#: JSON timestamp. Without this, upgrading DuckDB -- no contract edit, no new
+#: data -- raises SchemaDriftError on a pipeline that was working, which is a
+#: false positive: the inferrer drifted, the data did not.
+#:
+#: HONEST CAVEAT, and the reason :func:`family_rename_note` exists: DOUBLE and
+#: HUGEINT are not value-identical. For the same CSV literal 1.5 landed
+#: 1.7014118346046924e+35 (17 significant digits, lossy) where 2.0 lands the
+#: exact 128-bit integer. Collapsing the pair keeps the pipeline running, and
+#: the note says the physical type moved, so the precision change is reported
+#: rather than hidden. Founder decision, 2026-10-26.
+#: Each group also has to contain the name ``cli/discover/emitter.py``
+#: canonicalises its members TO, or the fix for one bug would cause another:
+#: discover writes the canonical name into the contract, DuckDB keeps inferring
+#: the source name, and the gate would then fail at apply what it had just
+#: stopped failing at validate. ``tests/.../test_schema_evolution_type_family.py``
+#: pins that every alias in the emitter's table is same-family as its target,
+#: so the two tables cannot drift apart silently.
+_TYPE_FAMILIES: Tuple[Tuple[str, ...], ...] = (
+    # 128-bit and unsigned-64 integers, and the floats they are inferred as.
+    # `numeric` is the emitter's canonical name for all of them.
+    (
+        "double",
+        "numeric",
+        "hugeint",
+        "bignum",
+        "uhugeint",
+        "int128",
+        "uint128",
+        "uint64",
+        "ubigint",
+        "varint",
+    ),
+    # unsigned 32-bit does not fit `integer`, so the emitter writes `bigint`
+    ("bigint", "uinteger", "uint32", "oid"),
+    ("integer", "uint16", "usmallint", "integral", "signed"),
+    ("smallint", "uint8", "utinyint", "short"),
+    ("tinyint", "int1"),
+    (
+        "timestamp",
+        "timestamptz",
+        "timestamp with time zone",
+        "timestamp_tz",
+        "timestamp_s",
+        "timestamp_ms",
+        "timestamp_us",
+        "timestamp_ns",
+        "timestamptz_ns",
+    ),
+    ("time", "time_ns", "timetz", "time with time zone"),
+    # Non-numeric spellings of one physical type. `string`/`null` covers an
+    # all-null column: discover writes `string` for it and DuckDB keeps
+    # inferring the quoted `"NULL"`. It deliberately does NOT include a real
+    # type, so `string` -> `integer` once data arrives is still drift.
+    ("varchar", "bpchar"),
+    ("bit", "bitstring"),
+    ("boolean", "logical"),
+    ("array", "list"),
+    ("struct", "tuple"),
+    ("variant", "union"),
+    ("string", "null", "type"),
+)
+
+_FAMILY_OF: Dict[str, int] = {
+    name: index for index, names in enumerate(_TYPE_FAMILIES) for name in names
+}
+
+
+def _same_type_family(old: str, new: str) -> bool:
+    """Whether ``old`` and ``new`` are two spellings of one physical type.
+
+    Quotes are stripped because ``typeof(NULL)`` is the quoted string
+    ``"NULL"``, which is how an all-null column reaches here.
+    """
+    a = _FAMILY_OF.get(old.strip().strip('"').lower())
+    b = _FAMILY_OF.get(new.strip().strip('"').lower())
+    return a is not None and a == b
+
+
+def family_rename_note(old: str, new: str) -> Optional[str]:
+    """A line for the operator when a type change was only a spelling change.
+
+    Returned so the caller can SAY the physical type moved while not failing on
+    it. ``None`` when the two are not one family.
+    """
+    if old.strip().lower() == new.strip().lower():
+        return None  # nothing changed: a rename note for X -> X is noise
+    if not _same_type_family(old, new):
+        return None
+    return (
+        f"column type {old!r} -> {new!r}: the same physical type under two names "
+        "(DuckDB 1.5 and 2.0 infer these differently for identical input), so "
+        "this is not schema drift. The stored width does change, so a value's "
+        "precision may differ between the two."
+    )
+
+
 @dataclass
 class EvolutionPlan:
     decisions: List[SchemaEvolutionDecision]
@@ -96,6 +199,12 @@ def diff_columns(
             events.append(("type_widened", name, base, cur))
         elif _is_narrowing(base.type, cur.type):
             events.append(("type_narrowed", name, base, cur))
+        elif _same_type_family(base.type, cur.type):
+            # Last, never before the ladder above: every ordered pair in
+            # _NUMERIC_ORDER is also same-family, so testing this first
+            # collapses all 56 of them and type_widened / type_narrowed
+            # would never fire again.
+            continue
         else:
             events.append(("type_changed", name, base, cur))
     for name, base in by_baseline.items():