One principled way of doing this is to use knowledge distillation (Hinton et al., 2015) where the simple model acts as a student and learns to mimic the probability distributions predicted by a teacher.
instead of simply maximizing the log likelihood of the training data, the simple model now gets additional signals from the teacher which act as hints while training.