This is consistent with the prior findings that the lower layers tend to learn more word-and phrase-level properties than the higher layers, while the top layers of SANs seek more global information (Peters et al., 2018;Yang et al., 2018;Devlin et al., 2019).
we also find that other function words (e.g, "ADP", "NUM","SYM") pay more attention on neighboring signals.