We allow gradients to be passed through all layers of BERT, and finetune the initialized weights with the SQuAD 2.0 dataset.
for future work, we plan to generalize the relation module to other aspects of question answering, including span prediction or multi-hop reasoning.