These pre-trained layers brought significant improvements to various NLP benchmarks, yielding  up to 30% relative error reductions.
due to high variability of language, gigantic NNs (e.g., LSTMs with 8,192 hidden states) are preferred to construct informative LMs and extract multifarious linguistic information (Peters et al., 2017).