We note that this trick comes at a cost of worse result on the density estimation task, since part of the parameters of the full model are trained to optimize an objective that does not capture all the dependencies that exist in the textual data.
the gap between purely deterministic LM and our model is small and easily controllable by the α hyperparameter.