 Language models are (1) capable of generatingremarkably coherent text (Zellers et al., 2019; See et al., 2019), (2) efficient at generating text, and (3) conceptually simple, but cannot infill effectively as they can only leverage context in a single direction (usually the past).
we can thus cast infilling as learning p(y |x) without loss of generality.