Our implementation is based on fairseq, 3 and we make use of its multi-GPU support to train on 16 NVIDIA V100 GPUs with a total batch size of 128,000 tokens.
these sentence embeddings are used to initialize the decoder LStM through a linear transformation, and are also concatenated to its input embeddings at every time step.