Many pre-trained language models, such as ELMO, BERT or GPT, have been produced by taking Web resources, such as the 1B Word Benchmark for ELMO, Wikipedia with the Toronto Book Corpus for BERT or WebText for GPT.
the way of dealing with this issue is by using robust methods to detect outliers within non-zero frequency documents and to use the traditional mean of the discounted frequencies (Winsorisation).