Although the above is the essence of our parallelization, invoking MPI Allreduce in the inner-most loop is very expensive because the communication setup is not so cheap.
our implementation calculates p(c l |vt i , n i ) in batches of B examples and calls MPI Allreduce at every B examples.