To further reduce the number of parameters and improve the computational efficiency, we adopt the factorized embedding parameterization proposed recently (Lan et al., 2019).
we decompose a word embedding matrix of size V × D into the product of two matrices of sizes V × H and H × D, respectively.