In their work, there is a single attention mechanism which is controlled by a fixed α.
we use a mini-batch size of 8192 tokens and warm up the learning rate linearly until 20k steps, after which it decays according to an inverse square root schedule.