 Finally, Kim et al. (2019) explore pre-training schemes' abilities to learn prepositions and wh-words with syntactic transformations (two kinds of closedclass words which our work does not address).
our probes combine novel open-class words with familiar closed-class words, to test whether the closed-class words are treated systematically by the network.