tain quantitative correlations between parts of the neural network and linguistic properties, in both speech (Wu and King, 2016;Alishahi et al., 2017;Wang et al., 2017) and language processing models (Köhn, 2015;Qian et al., 2016a;Adi et al., 2016;Linzen et al., 2016;Qian et al., 2016b).
we also find that representations from higher layers are better at capturing semantics, even though these are word-level labels.