(Meftah et al., 2017) showed that an initialization with a combination of several pre-trained embedding vectors (from different pre-trained models) improves significantly the performances.
word vectors (the combination between character level embedding and word level embedding CNN) are fed into a 100 dimension Gated Recurrent Units (GRUs) layer, a variant of RNNs.