Multiple ways to set configs leads to madness

[ericdill]

    def test_config_priority():
        c = sparklauncher.SparkConfiguration()
        c.driver_memory = "4g"
        c.conf.spark.driver.memory = "5g"
        c._set_environment_variables()
>       assert '--driver-memory 5g' in os.environ['PYSPARK_SUBMIT_ARGS']
E       AssertionError: assert '--driver-memory 5g' in '--archives /path/to/some/archive.zip,/path/to/someother/archive.zip --driver-memory 4g pyspark-shell'

tests/test_spark_launcher.py:52: AssertionError

Seems that setting any of the special cased SparkConfiguration._spark_launcher_arg_names in tandem with the fully dotted variable leads to the fully dotted one being superseded by the special cased one. Yuck