We hypothesize that the main determining factor in this behavior is the relatively small size of the data and its distribution, which does not require using long-range context in most examples.
we leverage a strong language model that was pre-trained on large corpora as a fixed encoder which supplies additional contextualized token representations.