To that end, we conduct a multifaceted evaluation in which we compare outputs produced by state-of-the-art neural machine translation and phrase-based machine translation systems for 9 language directions across a number of dimensions.
we measure the similarity of the outputs, their fluency and amount of reordering, the effect of sentence length and performance across different error categories.