We also use a separate output embeddings in addition to input embeddings, similar to word2vec implementation (Mikolov et al., 2013a,b).
each word has two sets of distributions qI and qO, each of which is a Gaussian mixture.