Hence, we introduce attention mechanism to extract such words that are important to the meaning of the sentence and aggregate the representation of those informative words to form a sentence vector.
that is, we first feed the word annotation h it through a one-layer MLP to get u it as a hidden representation of h it , then we measure the importance of the word as the similarity of u it with a word level context vector u w and get a normalized importance weight α it through a softmax function.