Given a pretrained language model, we perturb the prior context in various ways at test time, to study how much the perturbed information affects model performance.
we alter the context length to study how many tokens are used, permute tokens to see if LSTMs care about word order in both local and global contexts, and drop and replace target words to test the copying abilities of LSTMs with and without an external copy mechanism, such as the neural cache (Grave et al., 2017b).