Another possible way to take advantage of teacher models is to extract more information than simply the predicted spans for each training example from the teacher models.
one easy way to extract piece of information is the gradient of the model output with respect to the input for the teacher model.