As an alternative to learning a single model, we can use the text representation layer of the aforementioned single model in the word expert model.
after training the single model with the whole Wikipedia, we use the learned model of the LSTM as the text representation layer of the word expert models.