We hope our empirical study may potentially allow others to design better attention mechanisms given their particular applications.
for SP, removing PE from our proposed attention variant dramatically degrades the performance (from 24.28 to 30.92).