Because the information on proximity between tokens is not considered in self-attention itself, the information on a token's position is embedded by using positional encoding (PE).
pE provides a matrix that represents the absolute position information of tokens in a sentence, and Transformer adds pE to the embedding matrix of the input tokens.