Although it is feasible to encode such information using positional encoding, our experiments have shown that this will lead to bad performance in our case.
we adopt a more straightforward positionwise attention, by multiplying the representation at pos by a positional attention digit a pos .