As mentioned in §3, orthogonal linear mappings are the preferred choice when learning transformations between the embedding spaces of different languages (Xing et al., 2015;Smith et al., 2017).
we perform an orthogonalization update (Cisse et al., 2017) after each training step to ensure that our mappings M are (approximately) orthogonal: where β is set to 0.001.