Methodologically, our approach can be seen as a combination of the knowledge distillation (Hinton et al., 2015;Bucilu et al., 2006) and the posterior regularization (PR) method (Ganchev et al., 2010).
at each iteration we adapt the posterior constraint principle from PR to construct a rule-regularized teacher, and train the student network of interest to imitate the predictions of the teacher network.