In this paper, we investigate the Knowledge Distillation (KD) method, which has been shown to be effective for knowledge ensembling (Hinton et al., 2015;Kim and Rush, 2016;Furlanello et al., 2018), for heterogeneous structure integration.
we employ a sequential LSTM as the student for distilling heterogeneous syntactic structures from various teacher tree encoders, such as GCN (Kipf and Welling, 2017) and TreeLSTM (Tai et al., 2015a).