For each sentence, the raters were to determine which of the two automatically translated sentences was the better translation of the (unseen) source sentence, assuming that the reference sentence was a perfect translation, with the option of choosing "neither" if the differences were negligible.
when we take 400 random sentences, the difference between the two versions only affects 30% of the sentences (133 or thereabouts) and therefore does not result in a significant difference (p=0.13 in a one tailed t-test).