Based on these observations we hypothesize that decreasing the conicity of the vectors h R and h P would improve the performance of the simple attention model.
we propose to change the multiplicative method of combining vectors into an additive method instead.