Secondly, the Transformer models are getting bigger and bigger (Sanh et al., 2019), resulting in a fact that it is almost impossible to pre-train such a big model on a personal computer.
models with plug-andplay property are attractive to Transformer-based models (Dathathri et al., 2019).