One simple way to overcome this issue is to use a truncated BPTT (Mikolov, 2012) for restricting the backpropagation to only few steps like 4 or 5.
to give their network a better initialization, they learn word embeddings using a nonprobabilistic language model, which was trained on English Wikipedia for about 2 months.