Neural network-based language models (LMs) have been shown to learn relevant properties of language without being explicitly trained for them.
recent work suggests that they are able to capture syntactic relations to a large extent (Gulordava et al., 2018; Kuncoro et al., 2018; Wilcox et al., 2018)