By appropriate representation learning (usually by back-propagations in neural network), these embeddings can replace traditional sparse features and perform quite well together with neural network.
it is not the case for our neural network as the output layer of our network only has two neurons.