The attention mechanism is similar to the one proposed by Yang et al.
(2016), which is defined as follows: to say, we first compute the u i as a hidden representation of the hidden states of LSTM h i for i th input token, i.e., Then, we measure the importance of the i th token with a trainable vector u w and get a normalized importance weight α i through a softmax function.