In addition to its conceptual simplicity, our experiments show that ILM enables off-the-shelf LMs to infill effectively.
then, we compute PPL only on the tokens which comprise the masked span, i.e., PPL is computed for all models on exactly the same set of tokens.