We also experiment with combining these approaches using a two-step pre-training regime.
we fine-tune the pre-trained BERT L on MultiNLI, and then fine-tune the resulting model again on the BoolQ train set.