Pre-trained representations mostly impacts the training time of the sequence to sequence model (see Appendix B): SRC-ELMO slows throughput during training by about 5.3x and SRC-FT is even slower because of the need to backpropagate through the LM for fine-tuning (9.2x).
inference is only 12-14% slower than the baseline when adding pre-trained embeddings to the encoder (SRC-ELMO, SRC-FT).