first party allocates all GPU for itself

[omarelnakib]

Hi,
when I try running using GPU, the first party allocates all the memory of the GPU, and when the other parties are created they throw exception out of memory as they cannot allocate anymore from the memory.
I use:
Tensorflow 1.15.0
keras 2.11.0
python 3.6
i tried adding this code in the keras_fl_model and they run successfully but no change happens, still allocates the whole memory
```
config = tf.compat.v1.ConfigProto()
config.gpu_options.allow_growth=True
config.gpu_options.per_process_gpu_memory_fraction = 0.1
self.sess = tf.compat.v1.Session(config=config)
set_session(self.sess)