The LM updated with textbook data (BERT+Textbook), improves performance on the domains included in additional pre-training (Phy and Gov).
we suspect that the updated model becomes more specialized towards seen domains, which leads to performance degradation on the unseen domain of Psychology.