Despite these drawbacks, PQA-A has substantial training instances so models could still benefit from it as a pre-training step.
in Phase I of multi-phase fine-tuning, we initialize BioBERT with θ 0 , and fine-tune it on PQA-A using question and context as input: Phase II Fine-tuning on Bootstrapped PQA-U: To fully utilize the unlabeled instances in PQA-U, we exploit the easiness of reasoning-free setting to pseudo-label these instances with a bootstrapping strategy: first, we initialize BioBERT with θ 0 , and fine-tune it on PQA-A using question and long answer (reasoning-free), then we further fine-tune BioBERT θ B 1 on PQA-L, also under the reasoning-free setting: We pseudo-label PQA-U instances using the most confident predictions of BioBERT θ B 2 for each class.