Automatic metrics have been criticized for a variety of reasons (Babych and Hartley, 2004;Callison-Burch et al., 2006;Chiang et al., 2008), and it is clear that they only loosely approximate human judgments.
having people evaluate translation output would be preferable, if it were more practical.