The results of our analysis imply that early in training, representing part of speech is the natural way to get initial high performance.
3 If the LM begins by maximizing mutual information with input, because the input is identical for the LM and tag models it may lead to these similar initial representations, followed by a decline in similarity as the compression narrows to properties specific to each task.