This type of models, including Transformer (Vaswani et al., 2017) and "Bidirectional Encoder Representations from Transformers," shortened as BERT (Devlin et al., 2019), rely on the attention mechanism (Luong et al., 2015) to learn a context-dependent representation; compared to recurrent neural networks (RNN), these self-attention-based models have faster encoding speed and the capacity of modeling a wider context.
bERT is recently proposed to extend the directionality of the Transformer model, and "pre-trained" using multiple objectives to strengthen its encoding capability.