One simple way to overcome this issue is to use a truncated BPTT (Mikolov, 2012) for restricting the backpropagation to only few steps like 4 or 5.
this solution limits the RNN to capture long-range dependencies.