To further investigate the phenomenon of neural toxic degeneration, and partially motivated by the surprising effectiveness of domain-adaptive pretraining on non-toxic data, we turn our focus to two corpora used to pretrain several language models.
we quantify the toxicity in OPENAI-WT (GPT-2's training data; Radford et al., 2019) and its open-source replica OWTC (Gokaslan and Cohen, 2019), inspired by previous work in analyzing social biases in large text corpora (Fast et al., 2016).