[datafusion-spark] parse_url incompatibilities

[andygrove]

Describe the bug

I'm working on enabling parse_url in Comet in apache/datafusion-comet#4152 and found some incompatibilities.

  SELECT parse_url('', 'PATH');                                                                                                                                                                        
  -- Spark: ""    Comet: NULL
                                                                                                                                                                                                       
  SELECT parse_url('http://example.com?foo=bar', 'FILE');                                                                                                                                              
  -- Spark: "?foo=bar"    Comet: "/?foo=bar"
                                                                                                                                                                                                       
  SELECT parse_url('http://example.com#fragment', 'FILE');                                                                                                                                             
  -- Spark: ""    Comet: "/"

To Reproduce

No response

Expected behavior

No response

Additional context

No response

[andygrove]

While auditing Comet's parse_url integration in apache/datafusion-comet#4152 we surfaced two more divergences from Spark beyond the ones in the original issue body:

3. PATH on a URL with a bare trailing slash

parse_url('http://example.com/', 'PATH')

• Spark (URI.getRawPath) returns "/".
• DataFusion (spark_parse_url in datafusion-spark) explicitly maps path == "/" to "" (see parse_url.rs::ParseUrl::parse, the unit test test_parse_path_root_is_empty_string asserts this behavior).

4. 3-arg parse_url(_, 'QUERY', key) decodes percent-encoded values

parse_url('https://use%20r:pas%20s@example.com/x?query=x%20y', 'QUERY', 'query')

• Spark returns the raw value x%20y (uses regex over getRawQuery).
• DataFusion calls Url::query_pairs() which form-decodes values, returning x y.

This is a subtler divergence than the others because the 2-arg parse_url(_, 'QUERY') does match Spark (both return the raw query string), but the 3-arg form with a key diverges.

For reference, Spark's own UrlFunctionsSuite exercises both behaviors in a single row (https://use%20r:pas%20s@example.com/dir%20/pa%20th.HTML?query=x%20y&q2=2#Ref%20two expects x%20y as the QUERY-with-key answer).