Following (Vaswani et al., 2017), we encode the positional information of a sequence of length T via the sin and cos functions with frequencies dictated by the feature index.
we define the positional embedding (PE) of a sequence X ∈ R T ×d (where T is length) as a matrix where: for i = 1, .