However in practice, we have to make a trade-off between performances and the number of parameters (model complexity).
given that deep layers are more task-specific than shallow ones (Peters et al., 2018;Mou et al., 2016), and that word embeddings (shallow layers) contain the majority of parameters, we choose to expand only the top layers.