Notice that some previous methods compress model directly during training phase (Khrulkov et al., 2019;Wen et al., 2017).
our problem setup follows (Chen et al., 2018a;Shu and Nakayama, 2017;Chen et al., 2018b) that given a pre-trained model, we want to compress the model with limited fine-tuning.