Furthermore, by comparing with prior work, we argue that learning from unlabelled corpora indeed helps to improve the sentence representations, although the current way of utilising corpora might not be optimal.
in one of the two classes of models of this type, an encoder-decoder model is learnt using a corpus of contiguous sentences Gan et al., 2017;Tang et al., 2018) to make predictions of the words in the next sentence given the words in the current one.