Our model can predict the next word using the embeddings of the words in the current n-gram window, in addition to the hidden state of the LSTM layer.
we also use a gate to control how much word-level information should be taken before injecting it into the softmax function.