This is analogous to recent work (Jia and Liang, 2017;Burlot and Yvon, 2017) on developing automated adversarial evaluation schemes for reading comprehension and machine translation.
unlike these efforts, our stress tests allow us to study model performance on a range of linguistic phenomena.