Our experiments, using the Holl-E dataset show that by (i) improving the conicity (Chandrahas et al., 2018) of the representations learned by the simple model and (ii) mimicking the outputs of the complex teacher model the simple model can give improved performance with fewer compute and memory requirements.
when compared to a standalone simple model the student model shows an improvement of 3.4% (compare SAM-mul-train (LD) and SAM-add-topk (LD) entries in Table 2 and Table 3 respectively).