Our results suggest that different pretraining objectives give rise to differences in function word comprehension; for instance, we see that natural language inference helps understanding negation, and CCG supertagging helps recognizing meaningful sentence boundaries.
overall, we find that the observed differences are not always straightforwardly interpretable, and further investigation is needed to determine which specific aspects of pretraining tasks yield good representations of function words.