The language models initialized with pretrained CLWEs are significantly better than their un-pre-trained counterpart on small amounts of data, reaching par performance with MKN at somewhere just past 4k sentences of training data.
it takes more than 16k sentences of training data before the plain LSTM language model began to outperform MKN.