BIOBERT is trained on PubMed abstracts and PMC full text articles, and CLIN-ICALBERT is trained on clinical text from the MIMIC-III database .
this corpus consists of 18% papers from the computer science domain and 82% from the broad biomedical domain.