Our model outperforms BERT BASE by a stable margin especially in SST-2, SenTube-A, SenTube-T, and SciTail, with the improvements of 1.2%, 2.1%, 1.7%, and 1.5%,   respectively.
the attention matrices of models with different setups are visualized in Figure 1.