We use Adam (Kingma and Ba, 2015) for optimization.
2 Here the behavior is similar, but unlike the Swedish model which got somewhat better when closely related languages were added, the increase in cross-entropy is monotone.