Our results confirm that task-trained contextualization is important when the end task requires specific information that may not be captured by the pretraining task ( §4).
such end-taskspecific contextualization can come from either fine-tuning CWRs or using fixed output features as inputs to a task-trained contextualizer; Peters et al.