The model architecture of CoMMA is shown in Figure 1.
coMMA differs from standard Transformer (Vaswani et al., 2017) as follows: 1) Some tokens are randomly replaced by a special mask token M with probability p, and the model is trained to predict original unmasked tokens.