In order to achieve this goal, we deliberately choose a dataset that does not require longterm dependency, so that any improvement from establishing the recurrence can be attributed to solving the context fragmentation.
we perform this controlled experiment on the One Billion Word dataset, which can only benefit from removing the context fragmentation.