Although ROBERTA's pretraining corpus is derived from multiple sources, it has not yet been established if these sources are diverse enough to generalize to most of the variation in the English language.
we would like to understand what is out of ROBERTA's domain.