This as- Time (h) BLEU CE Baseline 51.3 25.1 47.31 Async (4) + τ = 4 39.7 25.07 46.59   pect of our work is similar to the work of Aji and Heafield (2017) where they drop the lower 99% of the gradient updates based on absolute value thus reducing the memory traffic.
larger batches have two well-known impacts on speed: making more use of GPU parallelism and communicating less often.