Possible fixes to this problem include using a proper sentence level metric such a METEOR (Denkowski and Lavie, 2011) or a pseudo-corpus from the last few updates (Chiang et al., 2008).
in light of the result from section 3.1 that tuning on the dev set is still better than tuning on a held-out portion of the training data, we observe that tuning a corpus level metric on a highquality dev set from the same domain as the test set probably leads to the best translation quality.