We use the attention based Transformer (Vaswani et al., 2017) architecture as our baseline.
bERT denoise the 15% of the tokens at random by replacing 80% of them with [MASK], 10% of them with a random word and 10% of them unchanged.