(2018) show that up to a certain layer performance of representations obtained from a deep LM improves on a constituent labeling task, but then decreases, while with representations obtained from an MT encoder performance continues to improve up to the highest layer.
for MLM, representations initially acquire information about the context around the token, partially forgetting the token identity and producing a more generalized token representation.