In the Transformer network architecture, positional embeddings are used to encode order dependencies into the input representation.
(5), the final sentence representation S M is represented as: Note that for both RPEHead and MPRHead models, the RPEs will be jointly learned with the existing Transformer architecture.