In many cases, emojis serve as a proxy for the emotional contents of a text.
pretraining on the classification task of predicting which emoji were initially part of a text can improve performance on the target task (see §5.3 for an analysis of why our pretraining helps).