Recent advances in deep learning enabled a wide variety of NLP models to achieve impressive performance, thanks in part to the availability of large-scale annotated datasets.
we adopt a MoE module in C, which provides more flexibility than using a single MLP 3 .