Rather than measuring the quality of representations obtained from a particular model on some auxiliary task, we characterize how the learning objective determines the information flow in the model.
we consider how the representations of individual tokens in the Transformer evolve between layers under different learning objectives.