Hudi 1.1 MDT col-stats generation is failing for array and map types.

[vinishjail97]

Search before asking

• I had searched in the issues and found no similar issues.

Please describe the bug 🐞

There are failures when MDT col-stats are enabled for tables having array/map types in the schema for hudi 1.1

org.apache.hudi.exception.HoodieException: Failed to generate column stats records for metadata table

	at org.apache.hudi.metadata.HoodieTableMetadataUtil.convertMetadataToColumnStatsRecords(HoodieTableMetadataUtil.java:1594)
	at org.apache.hudi.metadata.HoodieMetadataWriteUtils.convertMetadataToRecords(HoodieMetadataWriteUtils.java:387)
	at org.apache.hudi.metadata.HoodieBackedTableMetadataWriter$BatchMetadataConversionFunction.convertMetadata(HoodieBackedTableMetadataWriter.java:1460)
	at org.apache.hudi.metadata.HoodieBackedTableMetadataWriter.processAndCommit(HoodieBackedTableMetadataWriter.java:1165)
	at org.apache.hudi.metadata.HoodieBackedTableMetadataWriter.update(HoodieBackedTableMetadataWriter.java:1399)
	at org.apache.hudi.client.BaseHoodieClient.writeTableMetadata(BaseHoodieClient.java:285)
	at org.apache.hudi.client.BaseHoodieWriteClient.writeToMetadataTable(BaseHoodieWriteClient.java:339)
	at org.apache.hudi.client.BaseHoodieWriteClient.commit(BaseHoodieWriteClient.java:320)
	at org.apache.hudi.client.BaseHoodieWriteClient.commitStats(BaseHoodieWriteClient.java:276)
	at org.apache.hudi.client.BaseHoodieWriteClient.commitStats(BaseHoodieWriteClient.java:244)
	at org.apache.hudi.client.HoodieJavaWriteClient.commit(HoodieJavaWriteClient.java:97)
	at org.apache.hudi.client.HoodieJavaWriteClient.commit(HoodieJavaWriteClient.java:52)
	at org.apache.hudi.client.BaseHoodieWriteClient.commit(BaseHoodieWriteClient.java:226)
	at org.apache.hudi.client.BaseHoodieWriteClient.commit(BaseHoodieWriteClient.java:221)
	at org.apache.hudi.client.BaseHoodieWriteClient.commit(BaseHoodieWriteClient.java:211)
	at org.apache.xtable.TestJavaHudiTable.insertRecordsWithCommitAlreadyStarted(TestJavaHudiTable.java:198)
	at org.apache.xtable.TestAbstractHudiTable.insertRecords(TestAbstractHudiTable.java:272)
	at org.apache.xtable.hudi.TestHudiFileStatsExtractor.columnStatsWithMetadataTable(TestHudiFileStatsExtractor.java:145)
	at java.base/java.lang.reflect.Method.invoke(Method.java:566)
	at java.base/java.util.concurrent.ForkJoinTask.doExec$$$capture(ForkJoinTask.java:290)
	at java.base/java.util.concurrent.ForkJoinTask.doExec(ForkJoinTask.java)
	at java.base/java.util.concurrent.ForkJoinPool$WorkQueue.topLevelExec(ForkJoinPool.java:1020)
	at java.base/java.util.concurrent.ForkJoinPool.scan(ForkJoinPool.java:1656)
	at java.base/java.util.concurrent.ForkJoinPool.runWorker(ForkJoinPool.java:1594)
	at java.base/java.util.concurrent.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:183)
Caused by: org.apache.avro.AvroRuntimeException: Not a record: {"type":"map","values":{"type":"record","name":"Nested","namespace":"test.nested_record","fields":[{"name":"nested_int","type":"int","default":0}]}}
	at org.apache.avro.Schema.getField(Schema.java:275)
	at org.apache.hudi.avro.HoodieAvroUtils.getSchemaForField(HoodieAvroUtils.java:1652)
	at org.apache.hudi.avro.HoodieAvroUtils.getSchemaForField(HoodieAvroUtils.java:1656)
	at org.apache.hudi.avro.HoodieAvroUtils.getSchemaForField(HoodieAvroUtils.java:1642)
	at org.apache.hudi.metadata.HoodieTableMetadataUtil.lambda$getColumnsToIndexWithoutRequiredMetaFields$48(HoodieTableMetadataUtil.java:1696)
	at java.base/java.util.stream.ReferencePipeline$3$1.accept(ReferencePipeline.java:195)
	at java.base/java.util.stream.ReferencePipeline$2$1.accept(ReferencePipeline.java:177)
	at java.base/java.util.ArrayList$ArrayListSpliterator.forEachRemaining(ArrayList.java:1655)
	at java.base/java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:484)
	at java.base/java.util.stream.AbstractPipeline.wrapAndCopyInto(AbstractPipeline.java:474)
	at java.base/java.util.stream.ForEachOps$ForEachOp.evaluateSequential(ForEachOps.java:150)
	at java.base/java.util.stream.ForEachOps$ForEachOp$OfRef.evaluateSequential(ForEachOps.java:173)
	at java.base/java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:234)
	at java.base/java.util.stream.ReferencePipeline.forEach(ReferencePipeline.java:497)
	at org.apache.hudi.metadata.HoodieTableMetadataUtil.getColumnsToIndexWithoutRequiredMetaFields(HoodieTableMetadataUtil.java:1698)
	at org.apache.hudi.metadata.HoodieTableMetadataUtil.getColumnsToIndex(HoodieTableMetadataUtil.java:1655)
	at org.apache.hudi.metadata.HoodieTableMetadataUtil.getColumnsToIndex(HoodieTableMetadataUtil.java:1615)
	at org.apache.hudi.metadata.HoodieTableMetadataUtil.convertMetadataToColumnStatsRecords(HoodieTableMetadataUtil.java:1583)
	... 24 more

Are you willing to submit PR?

• I am willing to submit a PR!
• I am willing to submit a PR but need help getting started!

Code of Conduct

• I agree to follow this project's Code of Conduct

[vinishjail97]

PR to fix nested array and map columns in MDT in hudi 1.x.
apache/hudi#17694