We therefore suggest: • Replication be adopted as standard practice in MT experimental methodology, especially in reporting results; 7 • Replication of optimization (MERT) and test set evaluation be performed at least three times; more replications may be necessary for experimental manipulations with more subtle effects; • Use of the median system according to a trusted metric when manually analyzing system output; preferably, the median should be determined based on one test set and a second test set should be manually analyzed.
the first system pair contrasts a baseline phrasebased system (Moses) and experimental hierarchical phrase-based system (Hiero), which were constructed from the Chinese-English BtEC corpus (0.7M words), the later of which was decoded with the cdec decoder (Koehn et al., 2007;Chiang, 2007;Dyer et al., 2010).