Prior studies reveal that modeling locality in lower layers can achieve better performance (Shen et al., 2018;Yu et al., 2018;Yang et al., 2018).
we merely apply the locality model at the lowest two layers of the encoder.