For clarification, we are not attacking the claim made by the prior work (Vaswani et al., 2017;Shaw et al., 2018;Huang et al., 2018b;Dai et al., 2019;Child et al., 2019), but we aim at providing a new look at the order-invariance problem when considering the attention mechanism with masks (masks refer to the set filtering function in our kernel formulation).
previous work did not consider the Table 3: Order-Invariance in Attention.