Le's model does obtain an impressive +1.7 BLEU gain on top of a baseline without an NNLM (25.8 vs. 27.5).
when compared to the strongest baseline which includes an NNLM, Le's best models (S2T + T2S) only obtain an +0.6 BLEU improvement (26.9 vs. 27.5).