Our implementation is based on the released code of XLM 1 (Paszke et al., 2017).
we use a Transformer architecture with 1024 hidden units, 8 heads, GELU activations (Hendrycks and Gimpel, 2016), with a dropout rate of 0.1.