In practice, probability vectors of LMs tend to be sparse (Kim et al., 2016).
a commonly discussed drawback of such LM-based text generation is exposure bias (Ranzato et al., 2015): during training, the model predicts the next token conditioned on the ground truth history, while at test time prediction is based on predicted tokens, causing a train-test mismatch.