An auxiliary task (next sentence prediction) is used to enhance the representations of the LM.
a comparison of the P-LM + aux model and the P-LM model shows that the performance of SiATL on classification tasks is improved by the auxiliary objective.