During pre-training, they are fed with complete sequences.
with a left-context-only decoder, these pretrained language models will suffer from incomplete and inconsistent context and thus cannot generate good enough context-aware word representations, especially during the inference process.