We demonstrate the effectiveness of our attention guidance loss through several empirical studies.
we 1) report convergence results on masked language modeling, 2) evaluate trained language models on downstream tasks, and 3) analyze the learned attention representations using probes.