To this end, we propose novel convolutional self-attention networks (CSANs), which model locality for self-attention model and interactions between features learned by different attention heads in an unified framework.
in order to pay more attention to a local part of the input sequence, we restrict the attention scope to a window of neighboring elements.