To address above problems, we design a novel pre-training task to make BERT domain-aware and then improve the BERT's fine-tuning procedure by adversarial training.
a novel post-training procedure is implemented that adapts BERT with unlabeled data from different domains to enhance the domain-awareness.