The synthetic data is generated from English Wikipedia 5 , English Gigaword (Parker et al., 2011) and Newscrawl 6 as the previous work (Ge et    2018a; Zhang et al., 2019;Kiyono et al., 2019;Grundkiewicz et al., 2019) did, using back translation and sentence corruption.
we train a transformer (base) model (Vaswani et al., 2017) for back translation using the training data of the restricted track in the BEA-19 shared task.