For each query q h i , we restrict its attention region (e.g., to a local scope with a fixed size M + 1 (M ≤ I) centered at the position i: Accordingly, the calculation of corresponding output in Equation 2is modified as: As seen, SANs are only allowed to attend to the neighboring tokens (e.g., K h , V h ), instead of all the tokens in the sequence (e.g., The SAN-based models are generally implemented as multiple layers, in which higher layers tend to learn semantic information while lower layers capture surface and lexical information (Peters et al., 2018;Raganato and Tiedemann, 2018).
we merely apply locality modeling to the lower layers, which same to the configuration in Yu et al.