It is a very good job! But I am confused about the derivation in Equation 10 of the appendix. How can we apply the REINFORCE equation to achieve Equation 10? 
It is a very good job!
But I am confused about the derivation in Equation 10 of the appendix. How can we apply the REINFORCE equation to achieve Equation 10?