At the same time, it is easy to show that if cos(nπ), which has a period of two is used as positional encoding, the self-attention mechanism can easily achieve the task which we also observe when we empirically evaluated with such an encoding.
the same encoding would not work for a language such as (aaaa) * , which has a periodicity of four.