And the query x is also encoded by a similar self-attention mechanism.
the attention processing units (APU) we employ is based on the Transformer model (Vaswani et al., 2017), which has achieved great success in natural language processing.