A wealth of work has shown that toxicity and social biases in training data are acquired by large pretrained sentence encoders (e.g., gender bias in BERT; May et al., 2019;Zhao et al., 2019;Basta et al., 2019;Kurita et al., 2019).
fewer studies have investigated toxicity in autoregressive language models, whose generations also suffer from incoherence, blandness, and repetitiveness (Holtzman et al., 2020;Welleck et al., 2019).