While this layer allows for information to propagate across long distances, it has a computational and memory cost that scales quadratically with the size of the input sequence.
transformers hardly scale to sequences of more than a thousand tokens.