In cases where the dataset contains, apart from snippetlevel, also token-level annotations, the attention module can be directly supervised: we add a term ... to the model's loss that minimizes the error of the attention scores, with respect to token-level annotations.
we consider a case where the token-level annotations are of three different types and we create a supervised attention head for each type, averaging their outputs.