By carefully examining the training procedure, we observe that the softmax layer, which predicts a distribution of the target word, often induces significant overhead, especially when the vocabulary size is large.
we revisit the design of the output layer and consider directly predicting the pre-trained embedding of the target word for a given context.