Although Transformer-XL is mainly designed to better capture longer-term dependency, it dramatically improves the single-model SoTA from 23.7 to 21.8.
transformer-XL significantly outperforms a contemporary method using vanilla transformers (Baevski and Auli, 2018), suggesting the advantage of transformer-XL is generalizable to modeling short sequences.