We propose a middle ground by training a simple model to mimic the characteristics of a complex model.
we make observations from a complex model which learns very diverse representations for different inputs and suitably modify the simple model to learn similar diverse representations.