Whereas computations run in parallel in GPU, the number of operations for the output layer in the language model is proportional to the vocabulary size in CPU.
to deal with issues coming from introducing subword language model, we develop a retrace algorithm and a reranking method by approximate marginalization.