To see the reason, assuming that the vocabulary size is V and the number of neurons in the first hidden layer l 1 is L 1 , the size of the weight matrices W 1 and W ′ 1 is V * L 1 , which in our dataset is around two million while the number of training examples is much smaller.
instead of using one-hot vectors, we employ embedding features.