BIOBERT is trained on PubMed abstracts and PMC full text articles, and CLIN-ICALBERT is trained on clinical text from the MIMIC-III database .
sCIBERT is trained on the full text of 1.14M biomedical and computer science papers from the semantic scholar corpus (Ammar et al., 2018).