Support for converting case class to Row

[jonas]

Motivation

The library currently offer the possibility of reading a Spark Row and convert it to a case class (or custom structure - but optimized for case classes). What's missing is going the other way around, from the case class to the Row, to be able to serialize case classes and load them back. A collateral product of this is likely be to produce a Spark StructType for a case class.

When we have bidirectional conversion Row <==> case class, then we can use Parquet to take case of the Row <==> Array[Byte] part to get full serialization of case classes and get Spark's SQL support on that data.

Input

• a failure case as captured in MPN-1558: Create minimal failing test for DataFrameReader.toRDD #15 which may suggest to possible failures/bugs in the parquet <-> binary serialization layer
  which would have a direct impact on the motivation for this issue.

Output

• an assessment of what the above failure examples mean.
  • is parquet serialization lossy? if so, can it be fixed? is it recoverable?
  • if parquet serialization to binary isn't lossy, then working code for converting case classes to Row with some of the following cases covered
    • custom field types with private/non-public constructor
    • nested field types
    • case classes with non-case class fields in them. a good example is joda DataTime
      • it's possible that the above may not be doable, or feasible, but in that case documentation regarding how to tackle such scenarios is to be added (eg. is the library user expected to define their own serializer/deserializers to the subset support by us, and if so what is the subset of cases our library would cover?)

Test

• CI (with unit tests covering the above) passes

[jonas]

@gpoirier I was lazy and just created this placeholder issue when I changed the project title. Can you give a bit more context what this feature would involve?

[gpoirier]

I filled in the empty description. Does that make sense?

[jonas]

Yes, thanks.

[suhailshergill]

@gpoirier @jonas a possible thing that we should investigate is what happens at the Row => Binary level and whether or not spark's parquet serializer discards some information and is lossy in that regard. hopefully it's not, but looking at some of the failures @ldacosta seemed to face today (he's in the process of minimizing them) suggests that there may be some upstream bugs we may need to address here.

we should take a dip into this sooner rather than later. @ldacosta let me know if you need the assistance of anybody here to minimize the examples, or do your best and put something up and we'll go from there

[jonas]

I am curious how this will compare performance wise to Spark's built-in reflection-based converter.

[gpoirier]

@jonas Reflection is used to create the schema, but I think the data is all fetched without reflection. Case classes are read through the Product interface I think. So performance might be similar, unless the schema creation is a significant part of the time spent on serialization.

[gpoirier]

@suhailshergill from the few tests I did with parquet to serialize a case class, we didn't lose any metadata required to read back to a case class.

[gpoirier]

Also, the failure examples that @ldacosta showed me this afternoon were all related to mismatch between the RowConverter structure and Parquet's, they weren't really Parquet bugs. Or was there other bugs @ldacosta ?

[suhailshergill]

to provide an update, @yawaramin is currently working on minimizing the failures that @ldacosta was experiencing. @yawaramin will make a PR with the minimal failing examples as tests, after which we can dig into what impact, if any, is wrt this issue.

@gpoirier fyi

[suhailshergill]

@gpoirier @jonas #15 demonstrates a small failing example