We hypothesise that this is because it is better to build up representations with local context before exploiting long-range correlations.
this gives evidence that we do not need to store many separate representations for long-range memory to perform well at test time, but the approach does require storing them during training -and incurs significant slowdown to the model.