Suppose we have the sentence attention score A i for each sentence s i ∈ x, and the word attention score a i,j for each word w i,j ∈ s i , both of the scores are normalized which satisfy the following equations, The sentence attention measures which sentence is more important for the overall sentiment while the word attention captures sentiment signals such as sentiment words in each sentence.
the document representation r for document x is calculated as follows, Note that many LSTM based models represent the word sequences only using the hidden layer at the final node.