Subsequent work has homed in on language modeling (LM) pretraining, finding that such mod-els can be productively fine-tuned on intermediate tasks like natural language inference before transferring to downstream tasks (Phang et al., 2018).
we identify two open questions: (1) How effective are tasks beyond language modeling in training reusable sentence encoders (2) Given the recent successes of LMs with intermediate-task training, which tasks can be effectively combined with language modeling and each other.