Formally, assume that we have an input sequence of w0, w1, . . . , wn−1, and we generate their corrupted positions p0, p1, . . . , pn−1 with our local bag strategy.
it may suffer from the same problem as unidirectional LMs: at one time, contexts from only one direction can be utilized instead of from both directions.