Add support for use assertDataFrameDataEquals with map types

[iburquijo]

References issue #444

This changes propuse convert Map type in dataframe schema to array of struct for having the possibility for compare two dataframes. Due to MapType is not orderable.

I added the private method convertMapToArrayStruct which is compatible with all current Spark version, using sql expressions instead of Spark API. (We could move to Spark API in versions greater than 3.0.0)

I also added two new test.

• One is for see that the method converts properly the types into array of struct
• Second one is for see the function assertDataFrameDataEquals works properly, compares the data and not raise and excepcion when map types is present into the dataframe.

All unit test are passed in my laptop, with differents Java version and in Github Actions.

[holdenk]

Thanks for working on this @iburquijo

[holdenk]

@CodeRabbit review

[Copilot]

Pull request overview

This PR adds support for comparing DataFrames containing Map types in the assertDataFrameDataEquals method by converting Maps to arrays of structs before comparison. This is necessary because MapType is not orderable in Spark.

Key Changes:

• Introduced convertMapToArrayStruct method that recursively converts Map types to arrays of structs using SQL expressions
• Modified assertDataFrameDataEquals to apply map conversion before comparison
• Added comprehensive tests for the new functionality including schema validation and data comparison

Reviewed changes

Copilot reviewed 2 out of 2 changed files in this pull request and generated 12 comments.

File	Description
core/src/main/2.4/scala/com/holdenkarau/spark/testing/DataFrameSuiteBase.scala	Added private convertMapToArrayStruct method and integrated it into assertDataFrameDataEquals to handle Map types by converting them to arrays of structs
core/src/test/2.4/scala/com/holdenkarau/spark/testing/SampleDataFrameTest.scala	Added test case classes with nested Maps and two test cases: one verifying schema conversion and another validating end-to-end comparison with Maps

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

[iburquijo]

@holdenk already reviewed an implemented changes propused by Copilot. Thank you and sorry for my late review.

[holdenk]

@iburquijo I think this largely looks good, the one trick is I've done update on the main branch so we support doing DataFrame equality tests on SparkConnect, so there's some conflicts with the RDD code path. If you've got the time to update this that would be awesome, if not no worries I can take a crack at it.

[iburquijo]

@holdenk sure no problem. I'll take a look and resolve the conflicts!!

[iburquijo]

@holdenk already resolved conflicts!!!!