Given a sequence [x 1 , x 2 , ..., x T ] where x t is the input embedding of element t, the state of Bi-GRU at position t is: where h f t and h bt are the states of the forward and backward GRU at position t. The final sequence embedding is either the concatenation of h f T and h b1 or simply the average of h t .
attentive Sequence Encoder directly using [h f T , h b1 ] for sequence encoding often fails to capture all the information when the sequence is long, while using the average of h t also has the drawback of treating useless elements equally with informative ones.