In all of our multilingual models we use heterogeneous batching, where each batch contains examples which are uniformly sampled from a concatenation of all the language pairs the model is trained on.
we use batches of 64 examples for sequences shorter than 69 tokens and batches of 16 examples for longer sequences.