Interestingly, previous work (Bojar et al., 2016a) has shown that a higher correlation can be achieved when comparing similar systems than when comparing different types of systems, e.g. phrase-based vs neural vs rulebased.
rule-based systems can be penalized as they produce less common translations, even when such translations are fluent and adequate.