For example, it is relatively easy to guess the "comma" by using the history words in language model task, where there are no bilingual information at all.
during the training, the model tries to adjust the parameters for highly frequent words from y i−1 and s i while neglecting the attention model.