In order to overcome this, we use standard simple models which do not capture all pairwise interactions, but learn to emulate certain characteristics of a complex teacher network.
we first investigate the conicity of representations learned by a complex model and observe that it is significantly lower than that of simpler models.