However, the model has access to the full context on each side of the target token, and predicting information that is already explicit in the input would not incentivise the model to learn about composition and semantics.
we must design the loss objective so that only sections of the model that have not yet observed the next word are optimised to perform the prediction.