As an example, a recent RE method called BRAN (Verga et al., 2018) proposed to use encoder of Transformer (Vaswani et al., 2017) for obtaining token representations and then used these representations for RE.
we experimented with different ablations of BRAN and noticed an improvement in results for DCN dataset upon removing multi-head selfattention layer.