It is also worth mentioning that the proposed LMs are trained from scratch on ID training data, but adapting knowledge from the GD word vector model.
the hidden state in Equation 6is not strictly the "GD hidden state"; it uses the word embeddings from the GD pre-trained word vector model, but the inputs are still ID data.