In light of the benefits of pretraining (Erhan et al., 2010), we should be able to do better than randomly initializing the remaining parameters of our models.
inductive transfer via finetuning has been unsuccessful for NLP (Mou et al., 2016).