We do not view the teacher predicted distribution for all indices with importance and just take the top few predicted indices.
the loss on a single training sample is given below e ) + D(y e ||p S e ) (2) where p T b , p T e gives just the (normalised) probability of the top-k predictions of teacher model on the begin and end indices.