Empty Strings Not treated as null

[HugoLinton]

You're missing the actual code to handle empty strings being treated as null

[HugoLinton]

StaxXmlGenerator - Line 45

  case (_, _, "") if options.treatEmptyValuesAsNulls =>
  // If the string is empty and the user has set to treat empty strings as null
  // Then don't output anything

[HyukjinKwon]

Yea, thank you for pointing this out. Actually, I am waiting for apache/spark#14118 . I would like to match this up with CSV (and deprecating treatEmptyValuesAsNulls because nullValue set to "" is equivalent with this).

[Sokrates80]

Hello I'm experiencing a weird behaviour with the version 0.3.4 of the library (I'm using spark 1.6.2) and I think it might be connected to this issue.
Basically I get an exception when trying to parse an xml containing a row with only a whitespace in the tag (in this case " "). I attached a the test file I used.

The exception is:
scala.MatchError: </HNAME> (of class com.sun.xml.internal.stream.events.EndElementEvent)

I tried to set the option but it didn't solve the problem:
.option("treatEmptyValuesAsNulls", "true")

I used the following custom schema:

            StructType customSchema = new StructType(new StructField[] {
            new StructField("@type", DataTypes.StringType, true, Metadata.empty()),
            new StructField("@mytype", DataTypes.StringType, true, Metadata.empty()),
            new StructField("@myversion", DataTypes.StringType, true, Metadata.empty()),
            new StructField("@id", DataTypes.StringType, true, Metadata.empty()),
            new StructField("module", DataTypes.createStructType(new StructField[] {
                            new StructField("@type", DataTypes.StringType, true, Metadata.empty()),
                            new StructField("item",     DataTypes.createArrayType(DataTypes.createStructType(
                                    new StructField[] {
                                            new StructField("@type", DataTypes.StringType, true, Metadata.empty())  ,
                                            new StructField("attributes", DataTypes.StringType, true, Metadata.empty())
                                    }
                            ), false), true, Metadata.empty())
                    }
            ), true, Metadata.empty())
            });

And below you find the code to perform the parsing. Is that linked to this bug or there is any available workaround?
Thanks In advance

   `DataFrame myDF = context.read()
            .format("com.databricks.spark.xml")
            .option("rowTag", "NK")
            .option("treatEmptyValuesAsNulls", "true")
            .schema(customSchema)
            .load("/spark_data/test/test_whitespace_parsing.xml");

    myDF.printSchema();

    myDF.select("@id","module.item.@type","module.item.attributes").show(10);

    System.out.println("Test Task Completed");`

test_whitespace.xml.txt

[Sokrates80]

Hello, any help on that?

[HyukjinKwon]

Oh, it's fine. I will merge the PR #182 after double checking. I appreciate for your help.

[KenpachiRules]

If all the values are null in a row , is there a way to represent it with self closing xml tags For example

<xs:schema xmlns:xs="http://www.w3.org/2001/XMLSchema">
<xs:element name="part">
xs:complexType
xs:sequence
<xs:element name="P_PARTKEY" type="xs:int" />
<xs:element name="P_NAME" type="xs:string" />
<xs:element name="P_MFGR" type="xs:string" />
<xs:element name="P_BRAND" type="xs:string" />
<xs:element name="P_TYPE" type="xs:string" />
</xs:sequence>
</xs:complexType>
</xs:element>
</xs:schema>

and if this is the input

<P_PARTKEY></P_PARTKEY>
<P_NAME></P_NAME>
<P_BRAND></P_BRAND>
<P_TYPE></P_TYPE>

I currently get the output as

Is there a way I can get a self closing tag for each element like this in case of empty row?

</P_PARTKEY>
</P_NAME>
</P_BRAND>
</P_TYPE>

[BioQwer]

If problem in self-closing tag, it fix in #285

[nikilkatturi]

spark scala: if i have an empty XML tag , returning null while parsing XML using com.databricks.spark.xml

I am getting null values when i have an empty XML tag .

<TransactionMessage><TransactionGuid> </TransactionGuid><MessageMsmqId><MessageMsmqId> <TransactionXmlMessage><Transaction><Items></Items><Discounts/></Transaction></TransactionXmlMessage></TransactionMessage>
Since Discount is empty , the whole XML parser is null as below :

MachineName| MessageMsmqId| TransactionGuid|TransactionXmlMessage| +-----------+--------------------+--------------------+---------------------+ | null| null| null| null|

val pomXML = spark.sqlContext.read.format("com.databricks.spark.xml").option("rootTag", "TransactionMessage"). load(s"pathpofthye file")