Both models are implemented on top of BERT encoders (Devlin et al., 2019).
we use BERT-Base, with the first 8 of the 12 layers frozen, only fine-tuning the last 4 transformer layers and the final projection layer.