Our model can predict the next word using the embeddings of the words in the current n-gram window, in addition to the hidden state of the LSTM layer.
we found a relatively small frequency threshold θ = 5 works quite effectively.