Causality of bert encoder in PAST streamable model

[GaddamHiranmai]

Hi @Or-Tal @nadav366,
Thank you for providing no transformer version of PAST model.

While trying to train PAST streamable model, I have noticed that causal mask is not used on bert transformer encoder. Does that experiment was tried already, if so is it reducing the performance?

Thanks

[Or-Tal]

Hello,
In the paper experiments we used a causal mask for the transformer encoder for the streamable version of PAST.
Performance is as reported in the paper for the causal version.
We hope to get to adding this fix to the release code soon.