Finally, we find that our method tends to achieve a superior language modeling results in terms of perplexity but an inferior ELBO value, and we use these results to argue that ELBO is suboptimal for language modeling even though it provides a formal lower bound on log marginal likelihood.
we suggest that future research in this direction should be careful to monitor the gap between ELBO and log marginal likelihood, and reconsider using ELBO as the surrogate, especially for evaluation.