When the teacher changes from BERT 12 to BERT 24 (i.e., Setting #1 vs. #2), there is not much difference between the students' performance.
bERT 12 teacher performs better on SST-2, QQP and QNLI, while bERT 24 performs better on MNLI-m, MNLI-mm and RTE.