(Meftah et al., 2017) showed that an initialization with a combination of several pre-trained embedding vectors (from different pre-trained models) improves significantly the performances.
for English experiments, we initialize word embedding with a concatenation of four pre-trained models: 1.