fix: parse nested and list temporal values with the inferred candidates

[teaguesterling]

Closes #159.

The report

Under datetime_format := 'us', a top-level <date>03/15/2024</date> parses correctly, but the same value nested one level deeper does not:

-- works
SELECT date FROM parse_xml('<events><event><date>03/15/2024</date></event></events>',
                           datetime_format := 'us');                      -- 2024-03-15

-- nested: Conversion Error, or {'date': NULL} with ignore_errors
SELECT meta FROM parse_xml('<events><event><meta><date>03/15/2024</date></meta></event></events>',
                           datetime_format := 'us', ignore_errors := true);

-- repeated -> LIST: [NULL, NULL]
SELECT date FROM parse_xml('<events><event><date>03/15/2024</date><date>12/25/2024</date></event></events>',
                           datetime_format := 'us', ignore_errors := true);

Cause

Inference and extraction disagreed about what a temporal value looks like.

InferTypeFromSamples decides a field is DATE by eliminating candidate formats against the samples — that is why typeof already answered STRUCT(date DATE) and DATE[]. The winning format, however, was only ever recorded for a top-level column:

• InferColumnType computes a nested field's winning format into nested_winning_fmt and then discards it (the long-standing TODO(#38)).
• LIST elements never carried a format at all.

Extraction then reached ConvertToValue with an empty datetime_format and fell back to default ISO parsing, which rejects 03/15/2024 — the very value inference had just accepted. The column's type and the column's parser came from two different decisions.

Fix

Two halves, both required — neither alone fixes both reported shapes:

1. Thread the format. ExtractValueFromNode, ExtractStructFromNode and ExtractListFromNode now take the column's winning format and pass it down, so a LIST column's elements convert with the column's own format. This fixes the DATE[] case, where the format was known and simply dropped.

2. Fall back to the candidates, but decline when they disagree. Where no format was ever recorded — nested fields — a failed default parse is retried against the same candidate list inference used. This fixes the nested STRUCT case, which threading alone cannot reach because there is nothing to thread.

The candidate list now lives once, at file scope, in EffectiveDatetimeCandidates(). Both InferTypeFromSamples and the new TryParseTemporalWithCandidates call it, so the two sides cannot drift apart again.

The fallback is consulted only after default parsing has failed, so every value that parses today keeps its current meaning.

The ambiguity rule, and why it is not "first match wins"

An earlier revision of this PR took the first candidate that parsed. Review caught that this is a silent-corruption bug, and it reproduced exactly as predicted:

data: 25/12/2024 (unambiguously day-first) + 01/02/2024 (ambiguous), auto mode

top level (format threaded):   2024-12-25, 2024-02-01   <- 1 February
nested   (per-value fallback): 2024-12-25, 2024-01-02   <- 2 January

Same extension, same data, contradictory answers. Inference eliminates candidates using every sample in a column, so one unambiguous 25/12/2024 settles that column on %d/%m/%Y. A fallback looking at a single value cannot repeat that elimination, and the auto list holds both %m/%d/%Y and %d/%m/%Y with month-first first.

So the fallback now declines when two candidates disagree about what the value means, letting the original conversion error stand. A silently transposed date is worse than the NULL this fix set out to remove.

This costs nothing for a user who names a format: the presets resolve to a single candidate ('us' is exactly {%m/%d/%Y}, 'eu' is {%d/%m/%Y}), so the rule cannot fire for them.

case	before this PR	first-match revision	as merged
nested ambiguous, auto	NULL / error	2024-01-02 (wrong)	Conversion Error
nested ambiguous, auto, ignore_errors	NULL	2024-01-02 (wrong)	NULL
nested ambiguous, explicit 'eu'	NULL	2024-02-01	2024-02-01
nested unambiguous day-first	NULL	2024-12-25	2024-12-25
top level (reference)	2024-02-01	2024-02-01	2024-02-01

Declining stays narrow: an unambiguous nested value is still parsed, so this is not a retreat to NULL.

Not limited to an explicit format

Measured — in auto mode, with no datetime_format at all:

SELECT typeof(meta), meta.date
FROM parse_xml('<events><event><meta><date>03/15/2024</date></meta></event></events>');
-- before: STRUCT(date DATE) | NULL
-- after:  STRUCT(date DATE) | 2024-03-15

The built-in candidates include %m/%d/%Y, which is why inference types the field DATE in the first place.

Covered

shape	before	after
nested STRUCT field	{'date': NULL} / Conversion Error	{'date': 2024-03-15}
repeated element → DATE[]	[NULL, NULL]	[2024-03-15, 2024-12-25]
STRUCT inside STRUCT	NULL	2024-03-15
LIST inside STRUCT	[NULL, NULL]	[2024-03-15, 2024-12-25]
nested TIMESTAMP	NULL	2024-03-15 14:30:00
nested, explicit %d-%m-%Y	NULL	2024-03-15
nested, auto mode (no format given)	NULL	2024-03-15

Controls that must not move, and do not: ISO input under a custom format still parses (default parsing runs first); a non-temporal nested field stays STRUCT("label" VARCHAR); all_varchar := true still wins over temporal typing. All five temporal branches (DATE, TIMESTAMP, TIMESTAMP_TZ, TIME, TIME_TZ) consult the fallback.

Verification

Built against the pinned duckdb submodule (d8cdaa33fd, v1.5.5).

• test/sql/github_issue_159_datetime_format_nested.test: all passing, now including the ambiguity cases and the top-level reference they must not contradict. Before the fix it failed on the nested case while its top-level control passed, so it distinguishes the two paths rather than failing everywhere.
• Full suite: 3996 assertions in 103 test cases, all passing.

Every expectation was measured against a built binary before being written — including the wrong 2024-01-02 above, which is why the ambiguity rule exists.

🤖 Generated with Claude Code

https://claude.ai/code/session_01XoHU19qnngy1CJnXyX1csz

[Copilot]

🟢 Approval recommended

The change is localized, aligns inference and extraction behavior, and includes targeted regression tests covering the reported failure modes and key controls.

Pull request overview

This PR fixes inconsistent temporal parsing in the parse_xml extraction path by ensuring nested STRUCT fields and LIST elements are converted using the same effective datetime candidates that schema inference used, resolving issue #159.

Changes:

• Centralized the effective datetime candidate list (EffectiveDatetimeCandidates) so inference and extraction consult the same formats.
• Threaded the column’s winning datetime format through recursive extraction helpers so LIST/STRUCT descendants can convert with the correct format when available.
• Added a fallback in temporal conversion: when default parsing fails and no explicit per-field format is available, retry parsing against the effective candidate list; added regression tests covering nested + repeated temporal shapes.

File summaries

File	Description
test/sql/github_issue_159_datetime_format_nested.test	Adds regression coverage for nested STRUCT and repeated/LIST temporal parsing under both explicit and auto datetime modes.
src/xml_schema_inference.cpp	Unifies datetime candidates across inference/extraction, threads winning format through recursive extraction, and adds candidate-based fallback parsing for temporal conversions.
src/include/xml_schema_inference.hpp	Updates extractor helper signatures/docs to accept and propagate the threaded datetime format.

Review details

• Files reviewed: 3/3 changed files
• Comments generated: 0
• Review effort level: Lite

---

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.