We propose a new transformer pattern that adheres to this property, the sandwich transformer, and show that it improves perplexity on multiple word-level and character-level language modeling benchmarks, at no cost in parameters, memory, or training time.
the sandwich reordering pattern does not guarantee performance gains across every task, as we demonstrate on machine translation models.