The results of our analysis imply that early in training, representing part of speech is the natural way to get initial high performance.
as training progresses, it increasingly benefits the model to represent categories with longer-range dependencies, such as topic.