The memory usage is also dominated by the attention layer as the attention span increase.
the span increases at the beginning of words and in the middle of composed words, e.g., to predict the "l" in "overlook".