In this work, we study an alternative of the relative position representations that ignores the directional information (Ahmad et al., 2019).
In our experiments, we observe that a deeper model (more layers) performs better than a wider model (larger d model).