The LM updated with textbook data (BERT+Textbook), improves performance on the domains included in additional pre-training (Phy and Gov).
• RQ3 can be answered as the updated BERT LM model does not appear to generalize well on unseen domains, as the evidence suggests that LM becomes more domain-specific.