To study the effect of content and function words on model perplexity, we drop them from different regions of the context and compare the resulting change in loss.
we perturb the context as follows, where f pos (y, span) is a function that drops all words with POS tag y in a given span.